* Arnd Bergmann arnd@linaro.org [2020-10-23 14:44:44]:
Also I think reserving 512kB of memory per virtqueue may not be acceptable in all cases. Some device virtqueue could be more busy than other at a given time - the busy virtqueue fills up fast and stalls for more memory to become available, while there is free memory available with other device virtqueues.
Can you give an example where 512KB is insufficient? Is this just a performance issue when you cannot submit more than 512KB in a single command and have to do it in a loop, or is there loss of functionality because all that data has to be in flight at the same time?
We are running with 1MB of shared memory (largely used for block device) and it fills up pretty fast during boot sometimes, resulting in guest delaying submitting new requests until space becomes available (which has a bearing on IO performance). We also have usecases which requires application (in guest) to read mega-bytes of data when it initializes, which we want to ensure gets done as quickly as possible.
A global (swiotlb) pool shared among multiple devices would allow for better sharing of available memory in that sense. For memory constrained configurations, a global shared pool may be desirable solution (vs dedicating per-virtqueue buffers of fixed size).
Sharing the bounce buffers across all devices would seem to add even more complexity, in particular when the devices are not all provided by the same virtual machine or hardware behind it. This means you would either have to track ownership of each page, or give all back-ends access to all the shared memory and lose isolation between them.
If only some devices share a common shmem area and swiotlb, that would seem to require additional complexity in the swiotlb implementation. My preference would be to use swiotlb as little as possible, and in particular not add features to it.
We are currently reusing swiotlb driver with just a minor change - to have it use memory from an indicated range (via device-tree) rather than allocate at runtime. For our usecase, we are fine to have all the virtio devices use the common shared memory pool.