Hi,
The following is a breakdown (as best I can figure) of the work needed
to demonstrate VirtIO backends in Rust on the Xen hypervisor. It
requires work across a number of projects but notably core rust and virtio
enabling in the Xen project (building on the work EPAM has already done)
and the start of enabling rust-vmm crate to work with Xen.
The first demo is a fairly simple toy to exercise the direct hypercall
approach for a unikernel backend. On it's own it isn't super impressive
but hopefully serves as a proof of concept for the idea of having
backends running in a single exception level where latency will be
important.
The second is a much more ambitious bridge between Xen and vhost-user to
allow for re-use of the existing vhost-user backends with the bridge
acting as a proxy for what would usually be a full VMM in the type-2
hypervisor case. With that in mind the rust-vmm work is only aimed at
doing the device emulation and doesn't address the larger question of
how type-1 hypervisors can be integrated into the rust-vmm hypervisor
model.
A quick note about the estimates. They are exceedingly rough guesses
plucked out of the air and I would be grateful for feedback from the
appropriate domain experts on if I'm being overly optimistic or
pessimistic.
The links to the Stratos JIRA should be at least read accessible to all
although they contain the same information as the attached document
(albeit with nicer PNG renderings of my ASCII art ;-). There is a
Stratos sync-up call next Thursday:
https://calendar.google.com/event?action=TEMPLATE&tmeid=MWpidm5lbzM5NjlydnA…
and I'm sure there will also be discussion in the various projects
(hence the wide CC list). The Stratos calls are open to anyone who wants
to attend and we welcome feedback from all who are interested.
So on with the work breakdown:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
STRATOS PLANNING FOR 21 TO 22
Alex Bennée
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Table of Contents
─────────────────
1. Xen Rust Bindings ([STR-51])
.. 1. Upstream an "official" rust crate for Xen ([STR-52])
.. 2. Basic Hypervisor Interactions hypercalls ([STR-53])
.. 3. [#10] Access to XenStore service ([STR-54])
.. 4. VirtIO support hypercalls ([STR-55])
2. Xen Hypervisor Support for Stratos ([STR-56])
.. 1. Stable ABI for foreignmemory mapping to non-dom0 ([STR-57])
.. 2. Tweaks to tooling to launch VirtIO guests
3. rust-vmm support for Xen VirtIO ([STR-59])
.. 1. Make vm-memory Xen aware ([STR-60])
.. 2. Xen IO notification and IRQ injections ([STR-61])
4. Stratos Demos
.. 1. Rust based stubdomain monitor ([STR-62])
.. 2. Xen aware vhost-user master ([STR-63])
1 Xen Rust Bindings ([STR-51])
══════════════════════════════
There exists a [placeholder repository] with the start of a set of
x86_64 bindings for Xen and a very basic hello world uni-kernel
example. This forms the basis of the initial Xen Rust work and will be
available as a [xen-sys crate] via cargo.
[STR-51] <https://linaro.atlassian.net/browse/STR-51>
[placeholder repository] <https://gitlab.com/cardoe/oxerun.git>
[xen-sys crate] <https://crates.io/crates/xen-sys>
1.1 Upstream an "official" rust crate for Xen ([STR-52])
────────────────────────────────────────────────────────
To start with we will want an upstream location for future work to be
based upon. The intention is the crate is independent of the version
of Xen it runs on (above the baseline version chosen). This will
entail:
• ☐ agreeing with upstream the name/location for the source
• ☐ documenting the rules for the "stable" hypercall ABI
• ☐ establish an internal interface to elide between ioctl mediated
and direct hypercalls
• ☐ ensure the crate is multi-arch and has feature parity for arm64
As such we expect the implementation to be standalone, i.e. not
wrapping the existing Xen libraries for mediation. There should be a
close (1-to-1) mapping between the interfaces in the crate and the
eventual hypercall made to the hypervisor.
Estimate: 4w (elapsed likely longer due to discussion)
[STR-52] <https://linaro.atlassian.net/browse/STR-52>
1.2 Basic Hypervisor Interactions hypercalls ([STR-53])
───────────────────────────────────────────────────────
These are the bare minimum hypercalls implemented as both ioctl and
direct calls. These allow for a very basic binary to:
• ☐ console_io - output IO via the Xen console
• ☐ domctl stub - basic stub for domain control (different API?)
• ☐ sysctl stub - basic stub for system control (different API?)
The idea would be this provides enough hypercall interface to query
the list of domains and output their status via the xen console. There
is an open question about if the domctl and sysctl hypercalls are way
to go.
Estimate: 6w
[STR-53] <https://linaro.atlassian.net/browse/STR-53>
1.3 [#10] Access to XenStore service ([STR-54])
───────────────────────────────────────────────
This is a shared configuration storage space accessed via either Unix
sockets (on dom0) or via the Xenbus. This is used to access
configuration information for the domain.
Is this needed for a backend though? Can everything just be passed
direct on the command line?
Estimate: 4w
[STR-54] <https://linaro.atlassian.net/browse/STR-54>
1.4 VirtIO support hypercalls ([STR-55])
────────────────────────────────────────
These are the hypercalls that need to be implemented to support a
VirtIO backend. This includes the ability to map another guests memory
into the current domains address space, register to receive IOREQ
events when the guest knocks at the doorbell and inject kicks into the
guest. The hypercalls we need to support would be:
• ☐ dmop - device model ops (*_ioreq_server, setirq, nr_vpus)
• ☐ foreignmemory - map and unmap guest memory
The DMOP space is larger than what we need for an IOREQ backend so
I've based it just on what arch/arm/dm.c exports which is the subset
introduced for EPAM's virtio work.
Estimate: 12w
[STR-55] <https://linaro.atlassian.net/browse/STR-55>
2 Xen Hypervisor Support for Stratos ([STR-56])
═══════════════════════════════════════════════
These tasks include tasks needed to support the various different
deployments of Stratos components in Xen.
[STR-56] <https://linaro.atlassian.net/browse/STR-56>
2.1 Stable ABI for foreignmemory mapping to non-dom0 ([STR-57])
───────────────────────────────────────────────────────────────
Currently the foreign memory mapping support only works for dom0 due
to reference counting issues. If we are to support backends running in
their own domains this will need to get fixed.
Estimate: 8w
[STR-57] <https://linaro.atlassian.net/browse/STR-57>
2.2 Tweaks to tooling to launch VirtIO guests
─────────────────────────────────────────────
There might not be too much to do here. The EPAM work already did
something similar for their PoC for virtio-block. Essentially we need
to ensure:
• ☐ DT bindings are passed to the guest for virtio-mmio device
discovery
• ☐ Our rust backend can be instantiated before the domU is launched
This currently assumes the tools and the backend are running in dom0.
Estimate: 4w
3 rust-vmm support for Xen VirtIO ([STR-59])
════════════════════════════════════════════
This encompasses the tasks required to get a vhost-user server up and
running while interfacing to the Xen hypervisor. This will require the
xen-sys.rs crate for the actual interface to the hypervisor.
We need to work out how a Xen configuration option would be passed to
the various bits of rust-vmm when something is being built.
[STR-59] <https://linaro.atlassian.net/browse/STR-59>
3.1 Make vm-memory Xen aware ([STR-60])
───────────────────────────────────────
The vm-memory crate is the root crate for abstracting access to the
guests memory. It currently has multiple configuration builds to
handle difference between mmap on Windows and Unix. Although mmap
isn't directly exposed the public interfaces support a mmap like
interface. We would need to:
• ☐ work out how to expose foreign memory via the vm-memory mechanism
I'm not sure if this just means implementing the GuestMemory trait for
a GuestMemoryXen or if we need to present a mmap like interface.
Estimate: 8w
[STR-60] <https://linaro.atlassian.net/browse/STR-60>
3.2 Xen IO notification and IRQ injections ([STR-61])
─────────────────────────────────────────────────────
The KVM world provides for ioeventfd (notifications) and irqfd
(injection) to signal asynchronously between the guest and the
backend. As far a I can tell this is currently handled inside the
various VMMs which assume a KVM backend.
While the vhost-user slave code doesn't see the
register_ioevent/register_irqfd events it does deal with EventFDs
throughout the code. Perhaps the best approach here would be to create
a IOREQ crate that can create EventFD descriptors which can then be
passed to the slaves to use for notification and injection.
Otherwise there might be an argument for a new crate that can
encapsulate this behaviour for both KVM/ioeventd and Xen/IOREQ setups?
Estimate: 8w?
[STR-61] <https://linaro.atlassian.net/browse/STR-61>
4 Stratos Demos
═══════════════
These tasks cover the creation of demos that brig together all the
previous bits of work to demonstrate a new area of capability that has
been opened up by Stratos work.
4.1 Rust based stubdomain monitor ([STR-62])
────────────────────────────────────────────
This is a basic demo that is a proof of concept for a unikernel style
backend written in pure Rust. This work would be a useful precursor
for things such as the RTOS Dom0 on a safety island ([STR-11]) or as a
carrier for the virtio-scmi backend.
The monitor program will periodically poll the state of the other
domains and echo their status to the Xen console.
Estimate: 4w
#+name: stub-domain-example
#+begin_src ditaa :cmdline -o :file stub_domain_example.png
Dom0 | DomU | DomStub
| |
: /-------------\ :
| |cPNK | |
| | | |
| | | |
/------------------------------------\ | | GuestOS | |
|cPNK | | | | |
EL0 | Dom0 Userspace (xl tools, QEMU) | | | | | /---------------\
| | | | | | |cYEL |
\------------------------------------/ | | | | | |
+------------------------------------+ | | | | | Rust Monitor |
EL1 |cA1B Dom0 Kernel | | | | | | |
+------------------------------------+ | \-------------/ | \---------------/
-------------------------------------------------------------------------------=------------------
+-------------------------------------------------------------------------------------+
EL2 |cC02 Xen Hypervisor |
+-------------------------------------------------------------------------------------+
#+end_src
[STR-62] <https://linaro.atlassian.net/browse/STR-62>
[STR-11] <https://linaro.atlassian.net/browse/STR-11>
4.2 Xen aware vhost-user master ([STR-63])
──────────────────────────────────────────
Usually the master side of a vhost-user system is embedded directly in
the VMM itself. However in a Xen deployment their is no overarching
VMM but a series of utility programs that query the hypervisor
directly. The Xen tooling is also responsible for setting up any
support processes that are responsible for emulating HW for the guest.
The task aims to bridge the gap between Xen's normal HW emulation path
(ioreq) and VirtIO's userspace device emulation (vhost-user). The
process would be started with some information on where the
virtio-mmio address space is and what the slave binary will be. It
will then:
• map the guest into Dom0 userspace and attach to a MemFD
• register the appropriate memory regions as IOREQ regions with Xen
• create EventFD channels for the virtio kick notifications (one each
way)
• spawn the vhost-user slave process and mediate the notifications and
kicks between the slave and Xen itself
#+name: xen-vhost-user-master
#+begin_src ditaa :cmdline -o :file xen_vhost_user_master.png
Dom0 DomU
|
|
|
|
|
|
+-------------------+ +-------------------+ |
| |----------->| | |
| vhost-user | vhost-user | vhost-user | : /------------------------------------\
| slave | protocol | master | | | |
| (existing) |<-----------| (rust) | | | |
+-------------------+ +-------------------+ | | |
^ ^ | ^ | | Guest Userspace |
| | | | | | |
| | | IOREQ | | | |
| | | | | | |
v v V | | \------------------------------------/
+---------------------------------------------------+ | +------------------------------------+
| ^ ^ | ioctl ^ | | | |
| | iofd/irqfd eventFD | | | | | | Guest Kernel |
| +---------------------------+ | | | | | +-------------+ |
| | | | | | | virtio-dev | |
| Host Kernel V | | | | +-------------+ |
+---------------------------------------------------+ | +------------------------------------+
| ^ | | ^
| hyper | | |
----------------------=------------- | -=--- | ----=------ | -----=- | --------=------------------
| call | Trap | | IRQ
V | V |
+-------------------------------------------------------------------------------------+
| | ^ | ^ |
| | +-------------+ | |
EL2 | Xen Hypervisor | | |
| +-------------------------------+ |
| |
+-------------------------------------------------------------------------------------+
#+end_src
[STR-63] <https://linaro.atlassian.net/browse/STR-63>
--
Alex Bennée
Hello,
This patchset adds vhost-user-gpio device's support in Qemu. The support for the
same has already been added to virtio specification and Linux Kernel.
A Rust based backend is also in progress and is tested against this patchset:
https://github.com/rust-vmm/vhost-device/pull/76
--
Viresh
Viresh Kumar (2):
hw/virtio: add boilerplate for vhost-user-gpio device
hw/virtio: add vhost-user-gpio-pci boilerplate
hw/virtio/Kconfig | 5 +
hw/virtio/meson.build | 2 +
hw/virtio/vhost-user-gpio-pci.c | 69 ++++++
hw/virtio/vhost-user-gpio.c | 343 ++++++++++++++++++++++++++++
include/hw/virtio/vhost-user-gpio.h | 35 +++
5 files changed, 454 insertions(+)
create mode 100644 hw/virtio/vhost-user-gpio-pci.c
create mode 100644 hw/virtio/vhost-user-gpio.c
create mode 100644 include/hw/virtio/vhost-user-gpio.h
--
2.31.1.272.g89b43f80a514
Hi Stefano,
Vincent gave an update on his virtio-scmi work at the last Stratos sync
call and the discussion moved onto next steps. Currently the demo setup
is intermediated by a double-ended vhost-user daemon running on the
devbox acting as a go between a number of QEMU instances representing
the front and back-ends. You can view the architecture with Vincents
diagram here:
https://docs.google.com/drawings/d/1YSuJUSjEdTi2oEUq4oG4A9pBKSEJTAp6hhcHKKh…
The key virtq handling is done over the special carve outs of shared
memory between the front end and guest. However the signalling is
currently over a virtio device on the backend. This is useful for the
PoC but obviously in a real system we don't have a hidden POSIX system
acting as a go between not to mention the additional latency it causes
with all those context switches.
I was hoping we could get some more of the Xen experts to the next
Stratos sync (17th Feb) to go over approaches for a properly hosted on
Xen approach. From my recollection (Vincent please correct me if I'm
wrong) of last week the issues that need solving are:
* How to handle configuration steps as FE guests come up
The SCMI server will be a long running persistent backend because it is
managing real HW resources. However the guests may be ephemeral (or just
restarted) so we can't just hard-code everything in a DTB. While the
virtio-negotiation in the config space covers most things we still need
information like where in the guests address space the shared memory
lives and at what offset into that the queues are created. As far as I'm
aware the canonical source of domain information is XenStore
(https://wiki.xenproject.org/wiki/XenStore) but this relies on a Dom0
type approach. Is there an alternative for dom0less systems or do we
need a dom0-light approach, for example using STR-21 (Ensure Zephyr can
run cleanly as a Dom0 guest) providing just enough services for FE's to
register metadata and BE's to read it?
* How to handle mapping of memory
AIUI the Xen model is the FE guest explicitly makes grant table requests
to expose portions of it's memory to other domains. Can the BE query the
hypervisor itself to discover the available grants or does it require
coordination with Dom0/XenStore for that information to be available to
the BE domain?
* How to handle signalling
I guess this requires a minimal implementation of the IOREQ calls for
Zephyr so we can register the handler in the backend? Does the IOREQ API
allow for a IPI style notifications using the global GIC IRQs?
Forgive the incomplete notes from the Stratos sync, I was trying to type
while participating in the discussion so hopefully this email captures
what was missed:
https://linaro.atlassian.net/wiki/spaces/STR/pages/28682518685/2022-02-03+P…
Vincent, anything to add?
--
Alex Bennée
Hi Ruchika,
I just wanted to check what was required for you to make progress using
virtio-rpmb. We've updated the spec for multi-block and I think
implementing it in the C version of the backend is pretty trivial. The
main thing I need is a decent way to test it.
The original backend was tested with the out of tree virtio-rpmb driver
which I hacked up to meet the spec. However my proposal for a new kernel
ABI fell flat:
Subject: [RFC PATCH 0/5] RPMB internal and user-space API + WIP virtio-rpmb frontend
Date: Wed, 3 Mar 2021 13:54:55 +0000
Message-Id: <20210303135500.24673-1-alex.bennee(a)linaro.org>
the main issue being not wanting to delegate crypto to inside the
kernel. So we need to return to doing the frame calculations externally
to the kernel and it simply acting as a conduit for the virtio frames. I
can just hack up my original test branch for multi-block although it
seems such a driver is unlikely to be merged anyway so it would just be
an out-of-tree testing branch.
Ilias mentioned we could test if he completes his uboot driver but I
recall there where some questions about how it fitted into u-boots
driver structure (i.e. being treated as a real device). For your work do
you still need a Linux driver for the rpmb anyway?
--
Alex Bennée
Viresh,
Apologies for the messy nature of my docker repo but there is a new
image:
https://github.com/stsquad/dockerfiles/tree/master/crossbuild/bullseye-arm64
which I have confirmed can build current Xen master:
make[4]: Leaving directory '/home/alex.bennee/lsrc/xen/xen.build.arm64-xen-master-for-bullseye/tools/pygrub'
make[3]: Leaving directory '/home/alex.bennee/lsrc/xen/xen.build.arm64-xen-master-for-bullseye/tools'
make[2]: Leaving directory '/home/alex.bennee/lsrc/xen/xen.build.arm64-xen-master-for-bullseye/tools'
make[1]: Leaving directory '/home/alex.bennee/lsrc/xen/xen.build.arm64-xen-master-for-bullseye/tools'
fakeroot sh ./tools/misc/mkdeb /home/alex.bennee/lsrc/xen/xen.build.arm64-xen-master-for-bullseye $(make -C xen xenversion --no-print-directory)
dpkg-deb: building package 'xen-upstream' in 'xen-upstream-4.17-unstable.deb'.
🕙14:00:03 alex.bennee@4372e168fe23:xen.build.arm64-xen-master-for-bullseye on xen.build.arm64-xen-master-for-bullseye [?] took 31s
➜ ls -l dist/
total 24667
-rw-r--r-- 1 alex.bennee alex.bennee 20830 Jun 7 2021 COPYING
-rw-r--r-- 1 alex.bennee alex.bennee 8632 Jun 7 2021 README
drwxr-xr-x 6 alex.bennee alex.bennee 6 Feb 11 13:59 install/
-rwxr-xr-x 1 alex.bennee alex.bennee 658 Jun 7 2021 install.sh*
-rw-r--r-- 1 alex.bennee alex.bennee 25213092 Feb 11 14:00 xen-upstream-4.17-unstable.deb
--
Alex Bennée
This series adds support for virtio-video decoder devices in Qemu
and also provides a vhost-user-video vmm implementation.
The vhost-user-video vmm currently parses virtio-vido v3 protocol
(as that is what the Linux frontend driver implements).
It then converts that to a v4l2 mem2mem stateful decoder device.
Currently this has been tested using v4l2 vicodec test driver in Linux
[1] but it is intended to be used with Arm SoCs which often implement
v4l2 stateful decoders/encoders drivers for their video accelerators.
The primary goal so far has been to allow continuing development
of virtio-video Linux frontend driver and testing with Qemu. Using
vicodec on the host allows a purely virtual dev env, and allows for
ci integration in the future by kernelci etc.
This series also adds the virtio_video.h header and adds the
FWHT format which is used by vicodec driver.
I have tested this VMM using v4l2-ctl from v4l2 utils in the guest
to do a video decode to a file. This can then be validated using ffplay
v4l2-compliance tool in the guest has also been run which stresses the
interface and issues lots of syscall level tests
See the README.md for example commands on how to configure guest kernel
and do a video decode using Qemu, vicodec using this VMM.
Linux virtio-video frontend driver code:
https://github.com/petegriffin/linux/commits/v5.10-virtio-video-latest
Qemu vmm code:
https://github.com/petegriffin/qemu/tree/vhost-virtio-video-master-v1
This is part of a wider initiative by Linaro called
"project Stratos" for which you can find information here:
https://collaborate.linaro.org/display/STR/Stratos+Home
Applies cleanly to git://git.qemu.org/qemu.git master(a3607def89).
Thanks,
Peter.
[1] https://lwn.net/Articles/760650/
Peter Griffin (8):
vhost-user-video: Add a README.md with cheat sheet of commands
MAINTAINERS: Add virtio-video section
vhost-user-video: boiler plate code for vhost-user-video device
vhost-user-video: add meson subdir build logic
standard-headers: Add virtio_video.h
virtio_video: Add Fast Walsh-Hadamard Transform format
hw/display: add vhost-user-video-pci
tools/vhost-user-video: Add initial vhost-user-video vmm
MAINTAINERS | 8 +
hw/display/Kconfig | 5 +
hw/display/meson.build | 3 +
hw/display/vhost-user-video-pci.c | 82 +
hw/display/vhost-user-video.c | 386 ++++
include/hw/virtio/vhost-user-video.h | 41 +
include/standard-headers/linux/virtio_video.h | 484 +++++
tools/meson.build | 9 +
tools/vhost-user-video/50-qemu-rpmb.json.in | 5 +
tools/vhost-user-video/README.md | 98 +
tools/vhost-user-video/main.c | 1680 ++++++++++++++++
tools/vhost-user-video/meson.build | 10 +
tools/vhost-user-video/v4l2_backend.c | 1777 +++++++++++++++++
tools/vhost-user-video/v4l2_backend.h | 99 +
tools/vhost-user-video/virtio_video_helpers.c | 462 +++++
tools/vhost-user-video/virtio_video_helpers.h | 166 ++
tools/vhost-user-video/vuvideo.h | 43 +
17 files changed, 5358 insertions(+)
create mode 100644 hw/display/vhost-user-video-pci.c
create mode 100644 hw/display/vhost-user-video.c
create mode 100644 include/hw/virtio/vhost-user-video.h
create mode 100644 include/standard-headers/linux/virtio_video.h
create mode 100644 tools/vhost-user-video/50-qemu-rpmb.json.in
create mode 100644 tools/vhost-user-video/README.md
create mode 100644 tools/vhost-user-video/main.c
create mode 100644 tools/vhost-user-video/meson.build
create mode 100644 tools/vhost-user-video/v4l2_backend.c
create mode 100644 tools/vhost-user-video/v4l2_backend.h
create mode 100644 tools/vhost-user-video/virtio_video_helpers.c
create mode 100644 tools/vhost-user-video/virtio_video_helpers.h
create mode 100644 tools/vhost-user-video/vuvideo.h
--
2.25.1