Alex Bennée via Stratos-dev stratos-dev@op-lists.linaro.org writes:

Hi All,

We've been discussing various ideas for Stratos in and around STR-7 (common virtio library). I'd originally de-emphasised the STR-7 work because I wasn't sure if this was duplicate effort given we already had libvhost-user as well as interest in rust-vmm for portable backends in user-space. However we have seen from the Windriver hypervisor-less virtio, NXP's Zephyr/Jailhouse and the requirements for the SCMI server that there is a use-case for a small, liberally licensed C library that is suitable for embedding in lightweight backends without a full Linux stack behind it. These workloads would run in either simple command loops, RTOSes or Unikernels.

Given the multiple interested parties I'm hoping we have enough people who can devote time to collaborate on the project to make the following realistic over the next cycle and culminate in the following demo in 6 months:

Components

portable virtio backend library

• source based library (so you include directly in your project)
• liberally licensed (Apache? to facilitate above)
• tailored for non-POSIX, limited resource setups
  • e.g. avoid malloc/free, provide abstraction hooks where needed
  • not assume OS facilities (so embeddable in RTOS or Unikernel)
  • static virtio configuration supplied from outside library (location of queues etc)
• hypervisor agnostic
  • provide a hook to signal when queues need processing

Following on from a discussion with Vincent and Akashi-san last week we need to think more about how to make this hypervisor agnostic. There will always be some code that has to live outside the library but if it ends up being the same amount again what have we gained?

I suspect this should be a from scratch implementation but it's certainly worth investigating the BSD implementation as Windriver have suggested.

SCMI server

This is the work product of STR-4, embedded in an RTOS. I'm suggesting Zephyr makes the most sense here given the previous work done by Peng and Akashi-san but I guess an argument could be made for a unikernel. I would suggest demonstrating portability to a unikernel would be a stretch goal for the demo.

The server would be *build* time selectable to deploy either in a Jailhouse partition or a Xen Dom U deployment. Either way there will need to be an out-of-band communication of location of virtual queues and memory maps - I assume via DTB.

From our discussion last week Zephry's DTB support is very limited and

not designed to cope with dynamic setups. So rather than having two build time selectable configurations we should opt for a single binary with a fixed expectation of where the remote guests memory and virtqueues will exist in it's memory space. It will then be up to the VMM setting things up to ensure everything is mapped in the appropriate place in the RTOS memory map. There would also be a fixed IRQ map for signalling when things are changes to the RTOS and a fixed doorbell for signalling the other way.

I'm unfamiliar with the RTOS build process but I guess this would be a single git repository with the RTOS and glue code and git sub-projects for the virtio and scmi libraries?

Deployments

To demonstrate portability we would have:

• Xen hypervisor
  • Dom0 with Linux/xen tools
  • DomU with Linux with a virtio-scmi front-end
  • DomU with RTOS/SCMI server with virtio-scmi back-end

The Dom0 in this case is just for convenience of the demo as we don't currently have a fully working dom0-less setup. The key communication is between the two DomU guests.

However the Dom0 will need also the glue code to setup the communication and memory mapping between the two DomU guests. This could happily link with the existing Xen library for setting up the guest table mappings.

• Jailhouse
  • Linux kernel partition with virtio-scmi front-end
  • RTOS/SCMI server partition with a virtio-scmi back-end

The RTOS/SCMI server would be the same binary blob as for Xen. Again some glue setup code would be required. I'm still unsure on how this would work for Jailhouse so if we don't have any Jailhouse expertise joining the project we could do this with KVM instead.

Linux/KVM host - setup code in main host (kvmtool/QEMU launch) - KVM guest with Linux with a virtio-scmi front-end - KVM guest with RTOS/SCMI server with virtio-scmi back-end

This is closer to Windrivers' hypervisor-less virtio deployment as Jailhouse is not a "proper" hypervisor in this case just a way of partitioning up the resources. There will need to be some way for the kernel and server partitions to signal each other when queues are updated.

Platform

We know we have working Xen on Synquacer and Jailhouse on the iMX. Should we target those as well as a QEMU -M virt for those who wish to play without hardware?

Stretch Goals

• Integrate Arnd's fat virtqueues

  Hopefully this will be ready soon enough in the cycle that we can add this to the library and prototype the minimal memory cross section.

This is dependant on having something at least sketched out early in the cycle. It would allow us to simplify the shared memory mapping to just plain virtqueus.

• Port the server/library to another RTOS/unikernel

  This would demonstrate the core code hasn't grown any assumptions about what it is running in.

• Run the server blob on another hypervisor

  Running in KVM is probably boring at this point. Maybe investigate having it Hafnium? Or in a R-profile safety island setup?

So what do people think? Thoughts? Comments? Volunteers?