Trying to start back up the conversation about multiple address spaces for IOMMU devices. If you will remember Jean-Philippe posted some patches back in February for SVM on arm-smmu-v3.
For quite some time the downstream Snapdragon kernels have supported something we call "per-process" page tables for the GPU. As with full SVM this involves creating a virtual address space per task but unlike SVM we don't automatically share the page table from the CPU. Instead we want to create a new page table and explicitly map/unmap address ranges into it. We provide the physical address of the page table to the GPU and it goes through the mechanics of programming the TTBR0 and invalidating the TLB when starts executing a submission for a given task.
As with all things IOMMU this discussion needs to be split into two parts - the API and the implementation. I want to focus on the generic API for this email. Shortly after Jean-Philippe posted his patches I sent out a rough prototype of how the downstream solution worked [1]:
+-----------------+ +------------------+ | "master" domain | ---> | "dynamic" domain | +-----------------+ \ +------------------+ \ \ +------------------+ - | "dynamic" domain | +------------------+
Given a "master" domain (created in the normal way) we can create any number of "dynamic" domains which share the same configuration as the master (table format, context bank, quirks, etc). When the dynamic domain is allocated/ attached it creates a new page table - for all intents and purposes this is a "real" domain except that it doesn't actually touch the hardware. We can use this domain with iommu_map() / iommu_unmap() as usual and then pass the physical address (acquired through a IOMMU domain attribute) to the GPU and everything works.
The main goal for this approach was to try to use the iommu API instead of teaching the GPU driver how to deal with several generations of page table formats and IOMMU devices. Shoehorning it into the domain struct was a path of least resistance that has served Snapdragon well but it was never really anything we considered to be a generic solution.
In the SVM patches, Jean-Philippe introduces iommu_bind_task(): https://patchwork.codeaurora.org/patch/184777/. Given a struct task and a bit of other stuff it goes off and does SVM magic.
My proposal would be to extend this slightly and return a iommu_task struct from iommu_bind_task:
struct iommu_task *iommu_bind_task(struct device *dev, strut task_strut *task, int *pasid, int flags, void *priv);
For implementations that did real SVM the rest would behave the same, but for targets with explicit map requirements the iommu_task token could then be used for map/unmap:
iommu_task_map(struct iommu_task *task, unsigned long iova, phys_addr_t paddr, size_t size, int prot);
iommu_task_unmap(struct iommu_task *task, unsigned long iova, size_t size);
int iommu_unbind_task(struct device *dev, struct iommu_task *task, int pasid, int flags);
(Note that I'm typing these up on the fly - don't take these prototypes as gospel. This is mostly just a representation of the hooks we would need).
Internally this would come down into the device drivers with code similar to https://patchwork.codeaurora.org/patch/184751/. At the device level the biggest question figuring out if this is the "full SVM" or "explicit" model - we could handle that with compatible strings or dt quirks or even a flag to iommu_bind_task(). On Snapdragon we have the additional requirement to get the physical address for the page table. That could be encoded into pasid or returned in the iommu_task struct or a task attribute function.
Comments welcome of course. I think that if we can focus on getting a good generic API nailed down we can drill into the specifics. Between the 410c and the 820c we have two good examples that we can rapidly prototype. I would like to get this on the radar so we can get it merged and stabilized with enough time to hit a LTS release.
Thanks, Jordan
[1] https://patchwork.codeaurora.org/patch/192573/ https://patchwork.codeaurora.org/patch/192571/ https://patchwork.codeaurora.org/patch/192577/ https://patchwork.codeaurora.org/patch/192579/