On 14/09/2021 15:44, Alex Bennée wrote:
Andrew Cooper andrew.cooper3@citrix.com writes:
On 13/09/2021 13:44, Alex Bennée wrote:
Hi,
As we consider the next cycle for Project Stratos I would like to make some more progress on hypervisor agnosticism for our virtio backends. While we have implemented a number of virtio vhost-user backends using C we've rapidly switched to using rust-vmm based ones for virtio-i2c, virtio-rng and virtio-gpio. Given the interest in Rust for implementing backends does it make sense to do some enabling work in rust-vmm to support Xen?
There are two chunks of work I can think of:
- Enough of libxl/hypervisor interface to implement an IOREQ end point.
No libxl here at all.
As of Xen 4.15, there are enough stable interfaces to implement simple IOERQ servers.
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com... was the commit where I removed the final unstable interface from varstored (terrible name) which is a dom0 backend for UEFI secure variable handling. As such, it also serves as a (not totally simple) reference of an IOERQ server.
There are a few bits and pieces of rust going on within Xen, and a whole load of plans. Also, there is a lot of interest from other downstreams in being able to write Rust backends.
We've got a placeholder xen and xen-sys crates, and placeholder work for supporting cross-compile as x86 PV and PVH stubdomains.
Are these in the rust-vmm project is elsewhere?
https://crates.io/crates/xen-sys
When I say placeholder, I really do mean placeholder.
To start this work meaningfully, we'd want to make a repo (or several) in the xen-project organisation on github or gitlab (we have both, for reasons), and set these as the upstream of the xen and xen-sys crates.
The want to have a simple IOREQ server compiled either as a dom0 backend, or as a PV or PVH stubdomains influences some of the design decisions early on, but they're all no-brainers for the longevity of the work.
Just to clarify nomenclature is a PVH stubdomain what I'm referring to as a bare metal backend, i.e: a unikernel or RTOS image that implements the backend without having to transition between some sort of userspace and it's supporting kernel?
I think so, yes, although calling it "bare metal" seems misleading for something which is a VM targetted at a specific hypervisor...
I started work on trying to reimplement varstored entirely in Rust as a hackathon project, although ran out of time trying to make hypercall buffers work (there is a bug with Box and non-global allocators causing rustc to hit an assert(). In the short term, we'll have to implement hypercall buffers in a slightly more irritating way).
Furthermore, stick to the stable hypercalls only. Xen's C libraries are disaster for cross-version compatibility, and you absolutely do not want to recompile your rust program just to run it against a different version of the hypervisor. The plan is to start with simple IOREQ servers, which are on fully stable interfaces, then stabilise further hypercalls as necessary to expand functionality.
Are the hypercalls mediated by a kernel layer or are you making direct HVC calls (on ARM) to the hypervisor from userspace?
For a dom0 backends irrespective of architecture, you need to issue ioctl()'s on the appropriate kernel device.
For a PV/PVH stubdom, you should make a call into the hypercall_page https://xenbits.xen.org/docs/latest/guest-guide/x86/hypercall-abi.html because Intel and AMD used different instructions for their equivalent of HVC.
ARM doesn't have the hypercall page ABI, so I'd expect the hypercall implementation to expand to HVC directly.
In terms of rust APIs, we'd want a crate which has target-specific implementations so the caller need not worry about the implementation details in the common case.
Where would I look in the Xen code to find the hypercalls that are considered stable and won't change between versions?
I'm afraid that's mostly in developers heads right now.
For a first pass, you can look for __XEN_TOOLS__ (This is mis-named, and ought to be called __XEN_UNSTABLE_INTERFACE__, because...) but be aware that some things currently tagged __XEN_TOOLS__ are incorrect and are in fact stable.
As a first pass, assume everything is unstable. The things contained within libxendevicemodel and libxenforeignmem are stable and were specifically made so to try and get simple IOREQ server functionality done and stable.
Almost everything else, particularly concerning the toolstack operations, is unstable. There is 15 years of organic growth and dubious decisions here, and they need unpicking carefully. We've got some hypercalls which look like they're unstable, but are actually stable (as they were exposed to guests), and therefore have ridiculous interfaces.
The "ABI v2" work is massive and complicated, and picking at some of the corners based on "what is needed to make new $FOO work" is a good way to make some inroads.
It's high time the Xen Rust working group (which has been talked about for several years now) actually forms...
Indeed part of the purpose of this email was to smoke out those who are interested in the intersection of Xen, Rust and VirtIO ;-)
The conversation has come up quite a few times in the past, but mostly by people who are also busy with other things.
~Andrew