On 28/03/18 10:02 AM, Christian König wrote:
Yeah, that looks very similar to what I picked up from the older patches, going to read up on that after my vacation.
Yeah, I was just reading through your patchset and there are a lot of similarities. Though, I'm not sure what you're trying to accomplish as I could not find a cover letter and it seems to only enable one driver. Is it meant to enable DMA transactions only between two AMD GPUs?
I also don't see where you've taken into account the PCI bus address. On some architectures this is not the same as the CPU physical address.
Just in general why are you interested in the "distance" of the devices?
We've taken a general approach where some drivers may provide p2p memory (ie. an NVMe card or an RDMA NIC) and other drivers make use of it (ie. the NVMe-of driver). The orchestrator driver needs to find the most applicable provider device for a transaction in a situation that may have multiple providers and multiple clients. So the most applicable provider is the one that's closest ("distance"-wise) to all the clients for the P2P transaction.
And BTW: At least for writes that Peer 2 Peer transactions between different root complexes work is actually more common than the other way around.
Maybe on x86 with hardware made in the last few years. But on PowerPC, ARM64, and likely a lot more the chance of support is *much* less. Also, hardware that only supports P2P stores is hardly full support and is insufficient for our needs.
So I'm a bit torn between using a blacklist or a whitelist. A whitelist is certainly more conservative approach, but that could get a bit long.
I think a whitelist approach is correct. Given old hardware and other architectures, a black list is going to be too long and too difficult to comprehensively populate.
Logan