On Mon, Jun 30, 2014 at 09:25:10PM +0200, David Herrmann wrote:
Hi
On Mon, Jun 30, 2014 at 9:04 PM, Jerome Glisse j.glisse@gmail.com wrote:
On Mon, Jun 30, 2014 at 08:47:31PM +0200, David Herrmann wrote:
Additionally to what AIO and Direct-IO do, intel userptr adds the range_start callback to release pinned pages whenever the pages are unmapped. However, anyone who truncates inode pages, schedules writeback, etc., has to lock the page. Thus, any following GUP-fast from userptr will fail and the slowpath will wait on mmap_sem. So I'd really prefer if you could elaborate on your race?
Some writback code path (and other cpu page table modificiation) will not call range_start but only invalidate_page. More over once the range_start is call a GUP that is done before range_end is call will return what ever it sees inside the cpu page table at the time which might be new pages or old pages.
range_start/end are usually called by unmap_*() functions, which normally are called under page-lock. Therefore, _no_ new PTEs can be established as long as the page is locked. This means, once range_end is called, you're guaranteed any racing GUP-fast will fail and any racing GUP will wait on the page-lock.
Page-lock does not protect cpu page table update. It happens that some code path take it but not all of them. Just grep for all the call site to mmu_notifier_invalidate_range_start they are quite few of them.
None the less, the slow GUP does not care about page-lock so again the race i describe does exist:
Time Function being call 1 - range_start 2 - some i915 cs submission making use of user ptr and calling GUP 3 - cpu page table update 4 - range_end
What ever happen at 2 return pages that are replaced by others at 3.
However, I wonder why i915 uses range_start instead of range_end. The PTEs are still in place when range_start is called, therefore a racing GUP-fast will still succeed.
Yes GUP-fast can succeed right after a range_start ends but before the cpu page table is updated in anyway.
Regarding "invalidate_page": This is called _after_ the PTE has been removed (see rmap and try_unmap_page()), also with the page-locked. Same scenario as range_end.
I am well aware of all that code. Yet nothing forbids the GPU to perform write access to the page while there is a writeback meaning that the wb semantic is badly violated by the GPU driver. The whole writeback code is full of assumption about when and how you can set the dirty bit and when its clear i am pretty sure that current code can race and violate that too ie i915 set dirty bit right before core code clears it and you end up with stall data inside the page that might never be written back to disk because kernel lost the fact that the page is now dirty.
However, I also wonder why i915 doesn't have a callback for that? They definitely need to, afaics.
On invalidate_page you must abide by the new access protection for the page if it becomes read only then secondary pte (in this case gpu page table entry) must be set to write only too.
Writeback code races with parallel GUP writes and keeps pages in place. They rely on GUP-writers to call SetDirty() once they're done. I've never liked that, but I don't see any code protecting writeback against GUP writes, do you?
GUP is a bad API, i am pretty sure all mm guys hate it, i think they were too forgiving to its user. The fact that it does exist does not means that we can rely on it. Especialy we should not assume things about it that are not true and the comment above GUP clearly tell you that whatever GUP returns might be a lie and you have been warn.
So once again the write case is not doable period. This would be a violation of GUP and several other core kernel mm code. So unlike trying to provide and API that lies and works only by luck stop pretending it is sane and ok.
Again i am not opose to an API that would steal page from a vma. Stealing the car is ok, riding in the trunk hopping the vma will not change car is not.
Thus you can imagine i915 trying to use an userptr object right after a range_start but before a range_end, the i915 will read the page table (GUP is not serializing anything here) and will assume that whatever it got from there is current while it might just be soon to be discarded/replaced pages. Hence you can not garanty that pages you use are the same backing the user space range. Note that the mmap_sem does not protect page migration or thing like that. It only protect vma modifications.
As i said for the gpu only accessing those buffer in read mode is fine but i am sure userspace will start relying on the gpu writting to those page and being able to read back what the gpu wrote through the user space mapping. This will work often but this can only work because you are lucky and there is no single way to make it work reliably.
i915 userptr is a 3.16 feature, right? So we can still revert it if it's really broken.
It is broken by all means revert.
However, I'd still like to see an example _call-trace_ with a *real race*. I cannot see any writeback code that replaces pages despite elevated page-refs. page-migration is quite strict about page-refs and fails in those cases. truncate() is the only place I can see, but this should be fixed by using range_end() instead of range_start(), right?
__replace_page in kernel/event/uprobes.c is an example i was looking at recently. I am sure they are others and there is nothing stoping core kernel mm guys to add more and i expect a lot more will come soon for valid core mm reasons (numa is king of the hill when it comes to mm code and whatever gpu driver rely on especialy if it is through GUP will be laughted at).
Cheers, Jérôme Glisse
Thanks David