Am 25.02.21 um 22:27 schrieb Andrey Grodzovsky:
On 2021-02-25 1:42 p.m., Christian König wrote:
Am 25.02.21 um 17:03 schrieb Andrey Grodzovsky:
On 2021-02-25 2:53 a.m., Christian König wrote:
Am 24.02.21 um 16:13 schrieb Andrey Grodzovsky:
Ping
Sorry, I've been on vacation this week.
Andrey
On 2021-02-20 7:12 a.m., Andrey Grodzovsky wrote:
On 2/20/21 3:38 AM, Christian König wrote: > > > Am 18.02.21 um 17:41 schrieb Andrey Grodzovsky: >> >> On 2/18/21 10:15 AM, Christian König wrote: >>> Am 18.02.21 um 16:05 schrieb Andrey Grodzovsky: >>>> >>>> On 2/18/21 3:07 AM, Christian König wrote: >>>>> >>>>> >>>>> Am 17.02.21 um 22:59 schrieb Andrey Grodzovsky: >>>>>> Problem: If scheduler is already stopped by the time >>>>>> sched_entity >>>>>> is released and entity's job_queue not empty I encountred >>>>>> a hang in drm_sched_entity_flush. This is because >>>>>> drm_sched_entity_is_idle >>>>>> never becomes false. >>>>>> >>>>>> Fix: In drm_sched_fini detach all sched_entities from the >>>>>> scheduler's run queues. This will satisfy >>>>>> drm_sched_entity_is_idle. >>>>>> Also wakeup all those processes stuck in sched_entity flushing >>>>>> as the scheduler main thread which wakes them up is stopped >>>>>> by now. >>>>>> >>>>>> v2: >>>>>> Reverse order of drm_sched_rq_remove_entity and marking >>>>>> s_entity as stopped to prevent reinserion back to rq due >>>>>> to race. >>>>>> >>>>>> Signed-off-by: Andrey Grodzovsky andrey.grodzovsky@amd.com >>>>>> --- >>>>>> drivers/gpu/drm/scheduler/sched_main.c | 31 >>>>>> +++++++++++++++++++++++++++++++ >>>>>> 1 file changed, 31 insertions(+) >>>>>> >>>>>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c >>>>>> b/drivers/gpu/drm/scheduler/sched_main.c >>>>>> index 908b0b5..c6b7947 100644 >>>>>> --- a/drivers/gpu/drm/scheduler/sched_main.c >>>>>> +++ b/drivers/gpu/drm/scheduler/sched_main.c >>>>>> @@ -897,9 +897,40 @@ EXPORT_SYMBOL(drm_sched_init); >>>>>> */ >>>>>> void drm_sched_fini(struct drm_gpu_scheduler *sched) >>>>>> { >>>>>> + int i; >>>>>> + struct drm_sched_entity *s_entity; >>>>> >>>>> BTW: Please order that so that i is declared last. >>>>> >>>>>> if (sched->thread) >>>>>> kthread_stop(sched->thread); >>>>>> + /* Detach all sched_entites from this scheduler once >>>>>> it's stopped */ >>>>>> + for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= >>>>>> DRM_SCHED_PRIORITY_MIN; i--) { >>>>>> + struct drm_sched_rq *rq = &sched->sched_rq[i]; >>>>>> + >>>>>> + if (!rq) >>>>>> + continue; >>>>>> + >>>>>> + /* Loop this way because rq->lock is taken in >>>>>> drm_sched_rq_remove_entity */ >>>>>> + spin_lock(&rq->lock); >>>>>> + while ((s_entity = >>>>>> list_first_entry_or_null(&rq->entities, >>>>>> + struct drm_sched_entity, >>>>>> + list))) { >>>>>> + spin_unlock(&rq->lock); >>>>>> + >>>>>> + /* Prevent reinsertion and remove */ >>>>>> + spin_lock(&s_entity->rq_lock); >>>>>> + s_entity->stopped = true; >>>>>> + drm_sched_rq_remove_entity(rq, s_entity); >>>>>> + spin_unlock(&s_entity->rq_lock); >>>>> >>>>> Well this spin_unlock/lock dance here doesn't look correct >>>>> at all now. >>>>> >>>>> Christian. >>>> >>>> >>>> In what way ? It's in the same same order as in other call >>>> sites (see drm_sched_entity_push_job and >>>> drm_sched_entity_flush). >>>> If i just locked rq->lock and did list_for_each_entry_safe >>>> while manually deleting entity->list instead of calling >>>> drm_sched_rq_remove_entity this still would not be possible >>>> as the order of lock acquisition between s_entity->rq_lock >>>> and rq->lock would be reverse compared to the call sites >>>> mentioned above. >>> >>> Ah, now I understand. You need this because >>> drm_sched_rq_remove_entity() will grab the rq lock again! >>> >>> Problem is now what prevents the entity from being destroyed >>> while you remove it? >>> >>> Christian. >> >> Right, well, since (unfortunately) sched_entity is part of >> amdgpu_ctx_entity and amdgpu_ctx_entity is refcounted >> there is a problem here that we don't increment >> amdgpu_ctx.refcount when assigning sched_entity >> to new rq (e.g. before drm_sched_rq_add_entity) and not >> decrement before removing. We do it for >> amdgpu_cs_parser.entity for example (in amdgpu_cs_parser_init >> and amdgpu_cs_parser_fini by >> calling amdgpu_ctx_get and amdgpu_ctx_put). But this seems a >> bit tricky due to all the drm_sched_entity_select_rq >> logic. >> >> Another, kind of a band aid fix, would probably be just locking >> amdgpu_ctx_mgr.lock around drm_sched_fini >> when finalizing the fence driver and around idr iteration in >> amdgpu_ctx_mgr_fini (which should be lock protected >> anyway as I see from other idr usages in the code) ... This >> should prevent this use after free. > > Puh, that's rather complicated as well. Ok let's look at it from > the other side for a moment. > > Why do we have to remove the entities from the rq in the first > place? > > Wouldn't it be sufficient to just set all of them to stopped? > > Christian.
And adding it as another condition in drm_sched_entity_is_idle ?
Yes, I think that should work.
In this case reverse locking order is created, In drm_sched_entity_push_job and drm_sched_entity_flush lock entity->rq_lock locked first and rq->lock locked second. In drm_sched_fini on the other hand, I need to lock rq->lock first to iterate rq->entities and then lock s_entity->rq_lock for each entity to modify s_entity->stopped. I guess we could change s_entity->stopped to atomic ?
Good point. But I think the memory barrier inserted by wake_up_all(&sched->job_scheduled); should be sufficient.
If I see this correctly we have the entity->rq_lock mainly to protect concurrent changes of entity->rq.
But when two CPUs both set entity->stopped to true at the same time we don't really care about it as long drm_sched_entity_is_idle() sees it.
Regards, Christian.
I was more thinking about integrity of reading/writing entity->stopped from different threads. I guess since it's bool it's guaranteed to be atomic from HW perspective anyway ?
More or less yes. The key point is that we only change it from false -> true and never the other way around. All that you need then for other CPUs to see the value is a barrier.
Christian.
Will send V3 soon.
Andrey
Andrey
Christian.
Andrey
> >> >> Andrey >> >> >>> >>>> >>>> Andrey >>>> >>>> >>>> >>>>> >>>>>> + >>>>>> + spin_lock(&rq->lock); >>>>>> + } >>>>>> + spin_unlock(&rq->lock); >>>>>> + >>>>>> + } >>>>>> + >>>>>> + /* Wakeup everyone stuck in drm_sched_entity_flush for >>>>>> this scheduler */ >>>>>> + wake_up_all(&sched->job_scheduled); >>>>>> + >>>>>> /* Confirm no work left behind accessing device >>>>>> structures */ >>>>>> cancel_delayed_work_sync(&sched->work_tdr); >>>>> >>> >