Interrupt context can't sleep. Drivers like Panfrost and MSM are taking mutex when job is released, and thus, that code can sleep. This results into "BUG: scheduling while atomic" if locks are contented while job is freed. There is no good reason for releasing scheduler's jobs in IRQ context, hence use normal context to fix the trouble.
Cc: stable@vger.kernel.org Fixes: 542cff7893a3 ("drm/sched: Avoid lockdep spalt on killing a processes") Signed-off-by: Dmitry Osipenko dmitry.osipenko@collabora.com --- drivers/gpu/drm/scheduler/sched_entity.c | 6 +++--- include/drm/gpu_scheduler.h | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index 191c56064f19..6b25b2f4f5a3 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -190,7 +190,7 @@ long drm_sched_entity_flush(struct drm_sched_entity *entity, long timeout) } EXPORT_SYMBOL(drm_sched_entity_flush);
-static void drm_sched_entity_kill_jobs_irq_work(struct irq_work *wrk) +static void drm_sched_entity_kill_jobs_work(struct work_struct *wrk) { struct drm_sched_job *job = container_of(wrk, typeof(*job), work);
@@ -207,8 +207,8 @@ static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f, struct drm_sched_job *job = container_of(cb, struct drm_sched_job, finish_cb);
- init_irq_work(&job->work, drm_sched_entity_kill_jobs_irq_work); - irq_work_queue(&job->work); + INIT_WORK(&job->work, drm_sched_entity_kill_jobs_work); + schedule_work(&job->work); }
static struct dma_fence * diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 0fca8f38bee4..addb135eeea6 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -28,7 +28,7 @@ #include <linux/dma-fence.h> #include <linux/completion.h> #include <linux/xarray.h> -#include <linux/irq_work.h> +#include <linux/workqueue.h>
#define MAX_WAIT_SCHED_ENTITY_Q_EMPTY msecs_to_jiffies(1000)
@@ -295,7 +295,7 @@ struct drm_sched_job { */ union { struct dma_fence_cb finish_cb; - struct irq_work work; + struct work_struct work; };
uint64_t id;
On 2022-04-11 18:15, Dmitry Osipenko wrote:
Interrupt context can't sleep. Drivers like Panfrost and MSM are taking mutex when job is released, and thus, that code can sleep. This results into "BUG: scheduling while atomic" if locks are contented while job is freed. There is no good reason for releasing scheduler's jobs in IRQ context, hence use normal context to fix the trouble.
I am not sure this is the beast Idea to leave job's sw fence signalling to be executed in system_wq context which is prone to delays of executing various work items from around the system. Seems better to me to leave the fence signaling within the IRQ context and offload only the job freeing or, maybe handle rescheduling to thread context within drivers implemention of .free_job cb. Not really sure which is the better.
Andrey
Cc: stable@vger.kernel.org Fixes: 542cff7893a3 ("drm/sched: Avoid lockdep spalt on killing a processes") Signed-off-by: Dmitry Osipenko dmitry.osipenko@collabora.com
drivers/gpu/drm/scheduler/sched_entity.c | 6 +++--- include/drm/gpu_scheduler.h | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index 191c56064f19..6b25b2f4f5a3 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -190,7 +190,7 @@ long drm_sched_entity_flush(struct drm_sched_entity *entity, long timeout) } EXPORT_SYMBOL(drm_sched_entity_flush);
-static void drm_sched_entity_kill_jobs_irq_work(struct irq_work *wrk) +static void drm_sched_entity_kill_jobs_work(struct work_struct *wrk) { struct drm_sched_job *job = container_of(wrk, typeof(*job), work);
@@ -207,8 +207,8 @@ static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f, struct drm_sched_job *job = container_of(cb, struct drm_sched_job, finish_cb);
- init_irq_work(&job->work, drm_sched_entity_kill_jobs_irq_work);
- irq_work_queue(&job->work);
INIT_WORK(&job->work, drm_sched_entity_kill_jobs_work);
schedule_work(&job->work); }
static struct dma_fence *
diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 0fca8f38bee4..addb135eeea6 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -28,7 +28,7 @@ #include <linux/dma-fence.h> #include <linux/completion.h> #include <linux/xarray.h> -#include <linux/irq_work.h> +#include <linux/workqueue.h>
#define MAX_WAIT_SCHED_ENTITY_Q_EMPTY msecs_to_jiffies(1000)
@@ -295,7 +295,7 @@ struct drm_sched_job { */ union { struct dma_fence_cb finish_cb;
struct irq_work work;
struct work_struct work;};
uint64_t id;
On 4/12/22 19:51, Andrey Grodzovsky wrote:
On 2022-04-11 18:15, Dmitry Osipenko wrote:
Interrupt context can't sleep. Drivers like Panfrost and MSM are taking mutex when job is released, and thus, that code can sleep. This results into "BUG: scheduling while atomic" if locks are contented while job is freed. There is no good reason for releasing scheduler's jobs in IRQ context, hence use normal context to fix the trouble.
I am not sure this is the beast Idea to leave job's sw fence signalling to be executed in system_wq context which is prone to delays of executing various work items from around the system. Seems better to me to leave the fence signaling within the IRQ context and offload only the job freeing or, maybe handle rescheduling to thread context within drivers implemention of .free_job cb. Not really sure which is the better.
We're talking here about killing jobs when driver destroys context, which doesn't feel like it needs to be a fast path. I could move the signalling into drm_sched_entity_kill_jobs_cb() and use unbound wq, but do we really need this for a slow path?
On 2022-04-12 14:20, Dmitry Osipenko wrote:
On 4/12/22 19:51, Andrey Grodzovsky wrote:
On 2022-04-11 18:15, Dmitry Osipenko wrote:
Interrupt context can't sleep. Drivers like Panfrost and MSM are taking mutex when job is released, and thus, that code can sleep. This results into "BUG: scheduling while atomic" if locks are contented while job is freed. There is no good reason for releasing scheduler's jobs in IRQ context, hence use normal context to fix the trouble.
I am not sure this is the beast Idea to leave job's sw fence signalling to be executed in system_wq context which is prone to delays of executing various work items from around the system. Seems better to me to leave the fence signaling within the IRQ context and offload only the job freeing or, maybe handle rescheduling to thread context within drivers implemention of .free_job cb. Not really sure which is the better.
We're talking here about killing jobs when driver destroys context, which doesn't feel like it needs to be a fast path. I could move the signalling into drm_sched_entity_kill_jobs_cb() and use unbound wq, but do we really need this for a slow path?
You can't move the signaling back to drm_sched_entity_kill_jobs_cb since this will bring back the lockdep splat that 'drm/sched: Avoid lockdep spalt on killing a processes' was fixing.
I see your point and i guess we can go this way too. Another way would be to add to panfrost and msm job a work_item and reschedule to thread context from within their .free_job callbacks but that probably to cumbersome to be justified here.
Andrey
Reviewed-by: Andrey Grodzovsky andrey.grodzovsky@amd.com
On 4/12/22 22:40, Andrey Grodzovsky wrote:
On 2022-04-12 14:20, Dmitry Osipenko wrote:
On 4/12/22 19:51, Andrey Grodzovsky wrote:
On 2022-04-11 18:15, Dmitry Osipenko wrote:
Interrupt context can't sleep. Drivers like Panfrost and MSM are taking mutex when job is released, and thus, that code can sleep. This results into "BUG: scheduling while atomic" if locks are contented while job is freed. There is no good reason for releasing scheduler's jobs in IRQ context, hence use normal context to fix the trouble.
I am not sure this is the beast Idea to leave job's sw fence signalling to be executed in system_wq context which is prone to delays of executing various work items from around the system. Seems better to me to leave the fence signaling within the IRQ context and offload only the job freeing or, maybe handle rescheduling to thread context within drivers implemention of .free_job cb. Not really sure which is the better.
We're talking here about killing jobs when driver destroys context, which doesn't feel like it needs to be a fast path. I could move the signalling into drm_sched_entity_kill_jobs_cb() and use unbound wq, but do we really need this for a slow path?
You can't move the signaling back to drm_sched_entity_kill_jobs_cb since this will bring back the lockdep splat that 'drm/sched: Avoid lockdep spalt on killing a processes' was fixing.
Indeed
I see your point and i guess we can go this way too. Another way would be to add to panfrost and msm job a work_item and reschedule to thread context from within their .free_job callbacks but that probably to cumbersome to be justified here.
Yes, there is no clear justification for doing that.
Andrey
Reviewed-by: Andrey Grodzovsky andrey.grodzovsky@amd.com
Thank you!
On Tue, Apr 12, 2022 at 9:41 PM Andrey Grodzovsky andrey.grodzovsky@amd.com wrote:
On 2022-04-12 14:20, Dmitry Osipenko wrote:
On 4/12/22 19:51, Andrey Grodzovsky wrote:
On 2022-04-11 18:15, Dmitry Osipenko wrote:
Interrupt context can't sleep. Drivers like Panfrost and MSM are taking mutex when job is released, and thus, that code can sleep. This results into "BUG: scheduling while atomic" if locks are contented while job is freed. There is no good reason for releasing scheduler's jobs in IRQ context, hence use normal context to fix the trouble.
I am not sure this is the beast Idea to leave job's sw fence signalling to be executed in system_wq context which is prone to delays of executing various work items from around the system. Seems better to me to leave the fence signaling within the IRQ context and offload only the job freeing or, maybe handle rescheduling to thread context within drivers implemention of .free_job cb. Not really sure which is the better.
We're talking here about killing jobs when driver destroys context, which doesn't feel like it needs to be a fast path. I could move the signalling into drm_sched_entity_kill_jobs_cb() and use unbound wq, but do we really need this for a slow path?
You can't move the signaling back to drm_sched_entity_kill_jobs_cb since this will bring back the lockdep splat that 'drm/sched: Avoid lockdep spalt on killing a processes' was fixing.
I see your point and i guess we can go this way too. Another way would be to add to panfrost and msm job a work_item and reschedule to thread context from within their .free_job callbacks but that probably to cumbersome to be justified here.
FWIW since this mentioned individual drivers, commit 'drm/sched: Avoid lockdep spalt on killing a processes' also introduced problems for lima. There were some occurrences in our CI https://gitlab.freedesktop.org/mesa/mesa/-/jobs/20980982/raw . Later I found it also reproducible on normal usage when just closing applications, so it may be affecting users too.
I tested this patch and looks like it fixes things for lima.
Thanks
Erico
On 4/13/22 01:59, Erico Nunes wrote:
On Tue, Apr 12, 2022 at 9:41 PM Andrey Grodzovsky andrey.grodzovsky@amd.com wrote:
On 2022-04-12 14:20, Dmitry Osipenko wrote:
On 4/12/22 19:51, Andrey Grodzovsky wrote:
On 2022-04-11 18:15, Dmitry Osipenko wrote:
Interrupt context can't sleep. Drivers like Panfrost and MSM are taking mutex when job is released, and thus, that code can sleep. This results into "BUG: scheduling while atomic" if locks are contented while job is freed. There is no good reason for releasing scheduler's jobs in IRQ context, hence use normal context to fix the trouble.
I am not sure this is the beast Idea to leave job's sw fence signalling to be executed in system_wq context which is prone to delays of executing various work items from around the system. Seems better to me to leave the fence signaling within the IRQ context and offload only the job freeing or, maybe handle rescheduling to thread context within drivers implemention of .free_job cb. Not really sure which is the better.
We're talking here about killing jobs when driver destroys context, which doesn't feel like it needs to be a fast path. I could move the signalling into drm_sched_entity_kill_jobs_cb() and use unbound wq, but do we really need this for a slow path?
You can't move the signaling back to drm_sched_entity_kill_jobs_cb since this will bring back the lockdep splat that 'drm/sched: Avoid lockdep spalt on killing a processes' was fixing.
I see your point and i guess we can go this way too. Another way would be to add to panfrost and msm job a work_item and reschedule to thread context from within their .free_job callbacks but that probably to cumbersome to be justified here.
FWIW since this mentioned individual drivers, commit 'drm/sched: Avoid lockdep spalt on killing a processes' also introduced problems for lima. There were some occurrences in our CI https://gitlab.freedesktop.org/mesa/mesa/-/jobs/20980982/raw . Later I found it also reproducible on normal usage when just closing applications, so it may be affecting users too.
I tested this patch and looks like it fixes things for lima.
This patch indeed should fix that lima bug. Feel free to give yours tested-by :)
On Wed, Apr 13, 2022 at 8:05 AM Dmitry Osipenko dmitry.osipenko@collabora.com wrote:
On 4/13/22 01:59, Erico Nunes wrote:
On Tue, Apr 12, 2022 at 9:41 PM Andrey Grodzovsky andrey.grodzovsky@amd.com wrote:
On 2022-04-12 14:20, Dmitry Osipenko wrote:
On 4/12/22 19:51, Andrey Grodzovsky wrote:
On 2022-04-11 18:15, Dmitry Osipenko wrote:
Interrupt context can't sleep. Drivers like Panfrost and MSM are taking mutex when job is released, and thus, that code can sleep. This results into "BUG: scheduling while atomic" if locks are contented while job is freed. There is no good reason for releasing scheduler's jobs in IRQ context, hence use normal context to fix the trouble.
I am not sure this is the beast Idea to leave job's sw fence signalling to be executed in system_wq context which is prone to delays of executing various work items from around the system. Seems better to me to leave the fence signaling within the IRQ context and offload only the job freeing or, maybe handle rescheduling to thread context within drivers implemention of .free_job cb. Not really sure which is the better.
We're talking here about killing jobs when driver destroys context, which doesn't feel like it needs to be a fast path. I could move the signalling into drm_sched_entity_kill_jobs_cb() and use unbound wq, but do we really need this for a slow path?
You can't move the signaling back to drm_sched_entity_kill_jobs_cb since this will bring back the lockdep splat that 'drm/sched: Avoid lockdep spalt on killing a processes' was fixing.
I see your point and i guess we can go this way too. Another way would be to add to panfrost and msm job a work_item and reschedule to thread context from within their .free_job callbacks but that probably to cumbersome to be justified here.
FWIW since this mentioned individual drivers, commit 'drm/sched: Avoid lockdep spalt on killing a processes' also introduced problems for lima. There were some occurrences in our CI https://gitlab.freedesktop.org/mesa/mesa/-/jobs/20980982/raw . Later I found it also reproducible on normal usage when just closing applications, so it may be affecting users too.
I tested this patch and looks like it fixes things for lima.
This patch indeed should fix that lima bug. Feel free to give yours tested-by :)
Sure: Tested-by: Erico Nunes nunes.erico@gmail.com
Thanks
Erico
On 11/04/2022 23:15, Dmitry Osipenko wrote:
Interrupt context can't sleep. Drivers like Panfrost and MSM are taking mutex when job is released, and thus, that code can sleep. This results into "BUG: scheduling while atomic" if locks are contented while job is freed. There is no good reason for releasing scheduler's jobs in IRQ context, hence use normal context to fix the trouble.
Cc: stable@vger.kernel.org Fixes: 542cff7893a3 ("drm/sched: Avoid lockdep spalt on killing a processes") Signed-off-by: Dmitry Osipenko dmitry.osipenko@collabora.com
Reviewed-by: Steven Price steven.price@arm.com
drivers/gpu/drm/scheduler/sched_entity.c | 6 +++--- include/drm/gpu_scheduler.h | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index 191c56064f19..6b25b2f4f5a3 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -190,7 +190,7 @@ long drm_sched_entity_flush(struct drm_sched_entity *entity, long timeout) } EXPORT_SYMBOL(drm_sched_entity_flush);
-static void drm_sched_entity_kill_jobs_irq_work(struct irq_work *wrk) +static void drm_sched_entity_kill_jobs_work(struct work_struct *wrk) { struct drm_sched_job *job = container_of(wrk, typeof(*job), work);
@@ -207,8 +207,8 @@ static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f, struct drm_sched_job *job = container_of(cb, struct drm_sched_job, finish_cb);
- init_irq_work(&job->work, drm_sched_entity_kill_jobs_irq_work);
- irq_work_queue(&job->work);
- INIT_WORK(&job->work, drm_sched_entity_kill_jobs_work);
- schedule_work(&job->work);
}
static struct dma_fence * diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 0fca8f38bee4..addb135eeea6 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -28,7 +28,7 @@ #include <linux/dma-fence.h> #include <linux/completion.h> #include <linux/xarray.h> -#include <linux/irq_work.h> +#include <linux/workqueue.h>
#define MAX_WAIT_SCHED_ENTITY_Q_EMPTY msecs_to_jiffies(1000)
@@ -295,7 +295,7 @@ struct drm_sched_job { */ union { struct dma_fence_cb finish_cb;
struct irq_work work;
struct work_struct work;};
uint64_t id;
On Wed, Apr 13, 2022 at 12:05 PM Steven Price steven.price@arm.com wrote:
On 11/04/2022 23:15, Dmitry Osipenko wrote:
Interrupt context can't sleep. Drivers like Panfrost and MSM are taking mutex when job is released, and thus, that code can sleep. This results into "BUG: scheduling while atomic" if locks are contented while job is freed. There is no good reason for releasing scheduler's jobs in IRQ context, hence use normal context to fix the trouble.
Cc: stable@vger.kernel.org Fixes: 542cff7893a3 ("drm/sched: Avoid lockdep spalt on killing a processes") Signed-off-by: Dmitry Osipenko dmitry.osipenko@collabora.com
Reviewed-by: Steven Price steven.price@arm.com
Is there something blocking this patch? Mesa CI is still hitting the issue and I have been waiting for it to be applied/backported to update CI with it. Thanks
Erico
On 5/17/22 10:40, Erico Nunes wrote:
On Wed, Apr 13, 2022 at 12:05 PM Steven Price steven.price@arm.com wrote:
On 11/04/2022 23:15, Dmitry Osipenko wrote:
Interrupt context can't sleep. Drivers like Panfrost and MSM are taking mutex when job is released, and thus, that code can sleep. This results into "BUG: scheduling while atomic" if locks are contented while job is freed. There is no good reason for releasing scheduler's jobs in IRQ context, hence use normal context to fix the trouble.
Cc: stable@vger.kernel.org Fixes: 542cff7893a3 ("drm/sched: Avoid lockdep spalt on killing a processes") Signed-off-by: Dmitry Osipenko dmitry.osipenko@collabora.com
Reviewed-by: Steven Price steven.price@arm.com
Is there something blocking this patch? Mesa CI is still hitting the issue and I have been waiting for it to be applied/backported to update CI with it. Thanks
If this patch won't be picked up anytime soon, then I'll include it into my "memory shrinker" patchset together with the rest of the fixes, so it won't get lost.
Let me push it into drm-misc-next.
Andrey
On 2022-05-17 05:03, Dmitry Osipenko wrote:
On 5/17/22 10:40, Erico Nunes wrote:
On Wed, Apr 13, 2022 at 12:05 PM Steven Price steven.price@arm.com wrote:
On 11/04/2022 23:15, Dmitry Osipenko wrote:
Interrupt context can't sleep. Drivers like Panfrost and MSM are taking mutex when job is released, and thus, that code can sleep. This results into "BUG: scheduling while atomic" if locks are contented while job is freed. There is no good reason for releasing scheduler's jobs in IRQ context, hence use normal context to fix the trouble.
Cc: stable@vger.kernel.org Fixes: 542cff7893a3 ("drm/sched: Avoid lockdep spalt on killing a processes") Signed-off-by: Dmitry Osipenko dmitry.osipenko@collabora.com
Reviewed-by: Steven Price steven.price@arm.com
Is there something blocking this patch? Mesa CI is still hitting the issue and I have been waiting for it to be applied/backported to update CI with it. Thanks
If this patch won't be picked up anytime soon, then I'll include it into my "memory shrinker" patchset together with the rest of the fixes, so it won't get lost.
Done.
Andrey
On 2022-05-17 10:03, Andrey Grodzovsky wrote:
Let me push it into drm-misc-next.
Andrey
On 2022-05-17 05:03, Dmitry Osipenko wrote:
On 5/17/22 10:40, Erico Nunes wrote:
On Wed, Apr 13, 2022 at 12:05 PM Steven Price steven.price@arm.com wrote:
On 11/04/2022 23:15, Dmitry Osipenko wrote:
Interrupt context can't sleep. Drivers like Panfrost and MSM are taking mutex when job is released, and thus, that code can sleep. This results into "BUG: scheduling while atomic" if locks are contented while job is freed. There is no good reason for releasing scheduler's jobs in IRQ context, hence use normal context to fix the trouble.
Cc: stable@vger.kernel.org Fixes: 542cff7893a3 ("drm/sched: Avoid lockdep spalt on killing a processes") Signed-off-by: Dmitry Osipenko dmitry.osipenko@collabora.com
Reviewed-by: Steven Price steven.price@arm.com
Is there something blocking this patch? Mesa CI is still hitting the issue and I have been waiting for it to be applied/backported to update CI with it. Thanks
If this patch won't be picked up anytime soon, then I'll include it into my "memory shrinker" patchset together with the rest of the fixes, so it won't get lost.
dri-devel@lists.freedesktop.org
-
Andrey Grodzovsky -
Dmitry Osipenko -
Erico Nunes -
Steven Price