On Tue, Mar 07, 2023 at 11:13:36AM +0530, Viresh Kumar wrote:
On 06-03-23, 10:34, Stefan Hajnoczi wrote:
On Mon, Mar 06, 2023 at 04:40:24PM +0530, Viresh Kumar wrote:
+Xen mmap description +^^^^^^^^^^^^^^^^^^^^
++-------+-------+ +| flags | domid | ++-------+-------+
+:flags: 64-bit bit field
+- Bit 0 is set for Xen foreign memory memory mapping. +- Bit 1 is set for Xen grant memory memory mapping. +- Bit 2 is set if the back-end can directly map additional memory (like
- descriptor buffers or indirect descriptors, which aren't part of already
- shared memory regions) without the need of front-end sending an additional
- memory region first.
I don't understand what Bit 2 does. Can you rephrase this? It's unclear to me how additional memory can be mapped without a memory region (especially the fd) is sent?
I (somehow) assumed we will be able to use the same file descriptor that was shared for the virtqueues memory regions and yes I can see now why it wouldn't work or create problems.
And I need suggestion now on how to make this work.
With Xen grants, the front end receives grant address from the from guest kernel, they aren't physical addresses, kind of IOMMU stuff.
The back-end gets access for memory regions of the virtqueues alone initially. When the back-end gets a request, it reads the descriptor and finds the buffer address, which isn't part of already shared regions. The same happens for descriptor addresses in case indirect descriptor feature is negotiated.
At this point I was thinking maybe the back-end can simply call the mmap/ioctl to map the memory, using the file descriptor used for the virtqueues.
How else can we make this work ? We also need to unmap/remove the memory region, as soon as the buffer is processed as the grant address won't be relevant for any subsequent request.
Should I use VHOST_USER_IOTLB_MSG for this ? I did look at it and I wasn't convinced if it was an exact fit. For example it says that a memory address reported with miss/access fail should be part of an already sent memory region, which isn't the case here.
VHOST_USER_IOTLB_MSG probably isn't necessary because address translation is not required. It will also reduce performance by adding extra communication.
Instead, you could change the 1 memory region : 1 mmap relationship that existing non-Xen vhost-user back-end implementations have. In Xen vhost-user back-ends, the memory region details (including the file descriptor and Xen domain id) would be stashed away in back-end when the front-end adds memory regions. No mmap would be performed upon VHOST_USER_ADD_MEM_REG or VHOST_USER_SET_MEM_TABLE.
Whenever the back-end needs to do DMA, it looks up the memory region and performs the mmap + Xen-specific calls: - A long-lived mmap of the vring is set up when VHOST_USER_SET_VRING_ENABLE is received. - Short-lived mmaps of the indirect descriptors and memory pointed to by the descriptors is set up by the virtqueue processing code.
Does this sound workable to you?
Stefan