On Wed, Jan 13, 2021 at 12:21 AM Stefano Stabellini via Stratos-dev stratos-dev@op-lists.linaro.org wrote:
On Tue, 12 Jan 2021, Alex Bennée wrote:
I wanted to bounce some ideas around about our aim of limited memory sharing.
At the end of last year Windriver presented their shared memory approach for hypervisor-less virto. We have also discussed QC's iotlb approach. So far there is no proposed draft the Virtio spec and there are questions about how these shared memory approaches fit within the existing Virtio memory model and how they would interact with a Linux guest driver API to minimise the amount of copying needed as data moves from a primary guest to a back-end.
Given the performance requirements for high bandwidth multimedia devices it feels like we need to get some working code published so we can compare behaviour and implementations details. I think we are still a fair way off in being able to propose any updates to the standard until we can see the changes needed across guest APIs and get some measure of performance and bottlenecks.
However there are a range of devices we are interested in that are less performance sensitive - e.g. SPI, I2C and other "serial" buses. They would also benefit from having a minimal memory profile. Is it worth considering addressing a separate simpler and less performance orientated solution?
Arnd suggested something that I'm going to call a fat VirtQueues. The idea being that both data and descriptor are stored in the same VirtQueue structure. While it would necessitate copying data from guest address space to the queue and back it could be kept to the lower levels of the driver stack without the drivers themselves having to worry too much about the details. With everything contained in the VirtQueue there is only one bit of memory to co-ordinate between the primary guest and service OS which makes isolation a lot easier.
Of course this doesn't solve the problem for the more performance sensitive applications but it would be a workable demonstration of memory isolation across VMs and a useful suggestion in it's own right.
What do people think?
I think it is a good idea: everyone will agree that the first step is to implement a solution that relies on memcpys. Anything smarter is best done as a second step and probably requires new hypervisor interfaces.
From a performance perspective, whether we use a separate pre-shared buffer, or "fat VirtQueues" as Arnd suggested, the results should be very similar. So I think fat VirtQueues are a good way forward to me.
Ok, sounds good to me, too. I would also expect to see similar performance between any approach using memcpy(). Even with the current case of communication between two guests using a copy in the hypervisor, it's probably not much faster than doing a copy in the guest. Between various options for doing a copy in the guest, the modified virtqueue should be conceptually cleaner than the others.
I'm still suspicious of the doing page-flipping approaches or anything that relies on IOMMU to avoid the memcpy() for general virtqueues, as that sounds like it will be slower than less secure than the memcpy() because of the added complexity. The best approach that I imagine using for some cases where the copy is too slow would be to extend those devices to use "shared memory regions"[1] in addition to virtqueues. This is already allowed in virtio-fs, virtio-video and virtio-gpu, and may work for some but not all other device types (i.e. not virtio-net or virtio-block) that do not lean towards this model.
Arnd
[1] https://github.com/oasis-tcs/virtio-spec/blob/master/shared-mem.tex
Arnd Bergmann arnd@linaro.org writes:
On Wed, Jan 13, 2021 at 12:21 AM Stefano Stabellini via Stratos-dev stratos-dev@op-lists.linaro.org wrote:
On Tue, 12 Jan 2021, Alex Bennée wrote:
I wanted to bounce some ideas around about our aim of limited memory sharing.
At the end of last year Windriver presented their shared memory approach for hypervisor-less virto. We have also discussed QC's iotlb approach. So far there is no proposed draft the Virtio spec and there are questions about how these shared memory approaches fit within the existing Virtio memory model and how they would interact with a Linux guest driver API to minimise the amount of copying needed as data moves from a primary guest to a back-end.
Given the performance requirements for high bandwidth multimedia devices it feels like we need to get some working code published so we can compare behaviour and implementations details. I think we are still a fair way off in being able to propose any updates to the standard until we can see the changes needed across guest APIs and get some measure of performance and bottlenecks.
However there are a range of devices we are interested in that are less performance sensitive - e.g. SPI, I2C and other "serial" buses. They would also benefit from having a minimal memory profile. Is it worth considering addressing a separate simpler and less performance orientated solution?
Arnd suggested something that I'm going to call a fat VirtQueues. The idea being that both data and descriptor are stored in the same VirtQueue structure. While it would necessitate copying data from guest address space to the queue and back it could be kept to the lower levels of the driver stack without the drivers themselves having to worry too much about the details. With everything contained in the VirtQueue there is only one bit of memory to co-ordinate between the primary guest and service OS which makes isolation a lot easier.
Of course this doesn't solve the problem for the more performance sensitive applications but it would be a workable demonstration of memory isolation across VMs and a useful suggestion in it's own right.
What do people think?
I think it is a good idea: everyone will agree that the first step is to implement a solution that relies on memcpys. Anything smarter is best done as a second step and probably requires new hypervisor interfaces.
From a performance perspective, whether we use a separate pre-shared buffer, or "fat VirtQueues" as Arnd suggested, the results should be very similar. So I think fat VirtQueues are a good way forward to me.
Ok, sounds good to me, too. I would also expect to see similar performance between any approach using memcpy(). Even with the current case of communication between two guests using a copy in the hypervisor, it's probably not much faster than doing a copy in the guest. Between various options for doing a copy in the guest, the modified virtqueue should be conceptually cleaner than the others.
I'm still suspicious of the doing page-flipping approaches or anything that relies on IOMMU to avoid the memcpy() for general virtqueues, as that sounds like it will be slower than less secure than the memcpy() because of the added complexity. The best approach that I imagine using for some cases where the copy is too slow would be to extend those devices to use "shared memory regions"[1] in addition to virtqueues. This is already allowed in virtio-fs, virtio-video and virtio-gpu, and may work for some but not all other device types (i.e. not virtio-net or virtio-block) that do not lean towards this model.
Arnd,
Would you be able to draft up an EPIC to cover this work under the Stratos project hierarchy please?
Arnd[1] https://github.com/oasis-tcs/virtio-spec/blob/master/shared-mem.tex
On Fri, Jan 15, 2021 at 1:58 AM Alex Bennée alex.bennee@linaro.org wrote:
Would you be able to draft up an EPIC to cover this work under the Stratos project hierarchy please?
I have now added https://projects.linaro.org/projects/STR/issues/STR-25
ARnd
stratos-dev@op-lists.linaro.org
-
Alex Bennée -
Arnd Bergmann