October 2020 - Stratos-dev - op-lists.linaro.org

Re: [Stratos-dev] My current status of Xen on QEMU TCG (-m virt)

by Stefano Stabellini

On Thu, 15 Oct 2020, Stefano Stabellini via Stratos-dev wrote: > On Thu, 15 Oct 2020, Masami Hiramatsu via Stratos-dev wrote: > > Hi, > > > > I've succeeded to make the X.org running on Dom0. > > It seems that Xorg's nouveau driver caused SIGBUS issue. Custom > > nouveau kernel driver + Xorg fbdev driver seems stable. > > (Even if it doesn't work again, I'll try to use USB-HDMI adaptor next time) > > > > So, I would like to test the virtio-video for the next step. > > Alex, how can I help you to test it? > > FYI in case it is helpful the last version of the patch series to enable > virtio (specifically virtio-mmio) in Xen by EPAM is here: > > https://marc.info/?l=xen-devel&m=159976941026226 And it is funny because while I was sending this email, they sent a new version to xen-devel! https://marc.info/?l=xen-devel&m=160278030131796

3 years, 7 months

1
0
0 0

Re: [Stratos-dev] My current status of Xen on QEMU TCG (-m virt)

by Stefano Stabellini

On Thu, 15 Oct 2020, Masami Hiramatsu via Stratos-dev wrote: > Hi, > > I've succeeded to make the X.org running on Dom0. > It seems that Xorg's nouveau driver caused SIGBUS issue. Custom > nouveau kernel driver + Xorg fbdev driver seems stable. > (Even if it doesn't work again, I'll try to use USB-HDMI adaptor next time) > > So, I would like to test the virtio-video for the next step. > Alex, how can I help you to test it? FYI in case it is helpful the last version of the patch series to enable virtio (specifically virtio-mmio) in Xen by EPAM is here: https://marc.info/?l=xen-devel&m=159976941026226

3 years, 7 months

1
0
0 0

Re: [Stratos-dev] AGL Demo Images under TCG (with and without Xen)

by Stefano Stabellini

On Wed, 14 Oct 2020, Alex Bennée via Stratos-dev wrote: > Booting with Xen as a Dom0: > > This uses the guest-loader device to setup the Dom0 kernel/initrd. I > boosted the system memory to 4G and allocated 2G to Dom0. You can find > my QEMU tree with guest-loader support at: > > https://github.com/stsquad/qemu/tree/xen/guest-loader-v1 I love it BTW > The command line is as before except more memory and a different -kernel > line to load Xen: > > ./aarch64-softmmu/qemu-system-aarch64 -cpu cortex-a57 \ > -machine type=virt,virtualization=on,gic-version=3 \ > ... > -m 4096 \ > -kernel ~/lsrc/xen.git/xen/xen \ > -append "dom0_mem=2G,max:2G loglvl=all guest_loglvl=all" \ > -device guest-loader,addr=0x42000000,kernel=$HOME/images/agl/Image,bootargs="console=ttyAMA0 root=/dev/vda" \ Can you try using console=hvc0 instead of console=ttyAMA0? If you are running on Xen as Dom0, you don't have access to the primary physical UART because it is used by Xen, and the PV console is a "hvc" not "ttyAMA". > -device guest-loader,addr=0x47000000,initrd=$HOME/images/agl/initramfs-netboot-image-qemuarm64.ext4 \ > -display gtk,show-cursor=on \ > -smp 4 > > The system boots up although I don't get any kernel messages from the > console which Xen still has control over. It throws up some complains > about un-handled writes: > > (XEN) *************************************************** > (XEN) No support for ARM_SMCCC_ARCH_WORKAROUND_1. > (XEN) Please update your firmware. > (XEN) *************************************************** > (XEN) 3... 2... 1... > (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input) > (XEN) Freed 364kB init memory. > (XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER4 > (XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER8 > (XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER12 > (XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER16 > (XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER20 > (XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER24 > (XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER28 > (XEN) d0v0: vGICR: SGI: unhandled word write 0x000000ffffffff to ICACTIVER0 > (XEN) d0v1: vGICR: SGI: unhandled word write 0x000000ffffffff to ICACTIVER0 > (XEN) d0v2: vGICR: SGI: unhandled word write 0x000000ffffffff to ICACTIVER0 > (XEN) d0v3: vGICR: SGI: unhandled word write 0x000000ffffffff to ICACTIVER0 All the ICACTIVER* messages are "benign" in the sense that ICACTIVER* are not emulated by Xen, so writes to them do nothing and Xen prints a warning. However they are very unlikely to be the cause of any issues you are seeing. > Although I couldn't access the console I was able to ssh into the > machine via my 2222 port-fwd. Very good! :-)

3 years, 7 months

1
0
0 0

[RFC PATCH 0/4] generic loader FDT support (for direct Xen boot)

by Alex Bennée

Hi, This series adds the ability to append FDT data for blobs loaded by the generic loader. My principle use-case was to be able to directly boot Xen with a kernel image which avoided having to: - get the kernel image into my system image - deal with bugs in FDT munging by -bios firmware and/or grub as such this currently a developer hack that makes *my* life easy and is thus presented as an RFC for discussion. While I've tested it with Xen I'm sure the approach would be applicable to other hypervisors or firmware which expect to consume FDT data pointing at various blobs. An example command line that launches this is (magic from -kernel): ./aarch64-softmmu/qemu-system-aarch64 -cpu cortex-a57 \ -machine type=virt,virtualization=on -display none \ -serial mon:stdio \ -netdev user,id=unet,hostfwd=tcp::2222-:22 \ -device virtio-net-pci,netdev=unet,id=virt-net,disable-legacy=on \ -device virtio-scsi-pci,id=virt-scsi,disable-legacy=on \ -blockdev driver=raw,node-name=hd,discard=unmap,file.driver=host_device,file.filename=/dev/zen-disk/debian-buster-arm64 \ -device scsi-hd,drive=hd,id=virt-scsi-hd \ -smp 4 -m 4096 \ -kernel ~/lsrc/xen.git/xen/xen \ -append "dom0_mem=1G,max:1G loglvl=all guest_loglvl=all" \ -device loader,addr=0x47000000,file=$HOME/lsrc/linux.git/builds/arm64/arch/arm64/boot/Image,len-fdt-compat=2,fdt-compat[0]='multiboot,,module',fdt-compat[1]='multiboot,,kernel',fdt-bootargs="root=/dev/sda2 ro console=hvc0 earlyprintk=xen" So a couple of choices I've made doing this: Promoting *fdt to MachineState This seemed the simplest approach to making the fdt available to the global state, especially as MachineState already has a *dtb pointer. I've converted the ARM virt machine and a RISCV machine but if this is acceptable I can convert the others. "Polluting" the generic loader arguments This was mainly out of convenience as the loader already knows the size of the blob being loaded. However you could certainly argue it makes sense to have a more generic "FDT expander" virtual device that could for example query the QOM model somehow to find the details it needs. FDT isn't the only way of passing system information up the boot chain. We could reasonably want to do a similar thing with ACPI which is what should be being used on SBSA like devices to communicate with the hypervisor. Also relying on ,, in the QemuOpt parser is ugly. It might be worth having better quoting support if I press on with this. So what do people think? Something worth developing? Alex Bennée (4): hw/board: promote fdt from ARM VirtMachineState to MachineState hw/riscv: migrate fdt field to generic MachineState device_tree: add qemu_fdt_setprop_string_array helper generic_loader: allow the insertion of /chosen/module stanzas include/hw/arm/virt.h | 1 - include/hw/boards.h | 1 + include/hw/core/generic-loader.h | 4 + include/hw/riscv/virt.h | 1 - include/sysemu/device_tree.h | 17 ++ device_tree.c | 26 +++ hw/arm/virt.c | 322 ++++++++++++++++--------------- hw/core/generic-loader.c | 42 ++++ hw/riscv/virt.c | 18 +- 9 files changed, 268 insertions(+), 164 deletions(-) -- 2.20.1

3 years, 7 months

5
12
0 0

Re: [Stratos-dev] AGL Demo Images under TCG (with and without Xen)

by Alex Bennée

Alex Bennée via Stratos-dev <stratos-dev(a)op-lists.linaro.org> writes: > Hi, > > TL;DR; > > Booted AGL Jellyfish images under QEMU TCG and got at least working > virtio block device and splash screen. It also manged to turn on my USB > camera which implies the USB pass through was working. Once the splash > screen clears and the screen re-sizes it goes blank. <snip> > -m 2048 \ The reason the compositor failed was it couldn't allocate enough memory for the larger framebuffer size. This was seen by running: cat /run/platform/display/compositor.log Upping the memory to -m 4096 seems to get further but the Weston compositor struggles under TCG with the soft rendering pipeline on a plain old framebuffer. Next stop is to investigate the virtio-gpu. -- Alex Bennée

3 years, 7 months

1
0
0 0

Re: [Stratos-dev] AGL Demo Images under TCG (with and without Xen)

by Alex Bennée

Alex Bennée via Stratos-dev <stratos-dev(a)op-lists.linaro.org> writes: > Hi, > > TL;DR; > > Booted AGL Jellyfish images under QEMU TCG and got at least working > virtio block device and splash screen. It also manged to turn on my USB > camera which implies the USB pass through was working. Once the splash > screen clears and the screen re-sizes it goes blank. > > Got the same results booting the images under Xen as Dom0 guest. > <snip> > -initrd ~/images/agl/initramfs-netboot-image-qemuarm64.ext4 \ I should note the initrd isn't required as it boots directly from the image. -- Alex Bennée

3 years, 7 months

1
0
0 0

AGL Demo Images under TCG (with and without Xen)

by Alex Bennée

Hi, TL;DR; Booted AGL Jellyfish images under QEMU TCG and got at least working virtio block device and splash screen. It also manged to turn on my USB camera which implies the USB pass through was working. Once the splash screen clears and the screen re-sizes it goes blank. Got the same results booting the images under Xen as Dom0 guest. Plain TCG boot: The command line is a curious mixture of MMIO and PCI virtio devices. I needed to decompress the initramfs and block images. I successfully booted with the following: ./aarch64-softmmu/qemu-system-aarch64 -cpu cortex-a57 \ -machine type=virt,virtualization=on,gic-version=3 \ -serial mon:stdio \ -netdev user,id=unet,hostfwd=tcp::2222-:22 \ -device virtio-net-device,netdev=unet,id=virt-net \ -drive id=disk0,file=$HOME/images/agl/agl-demo-platform-crosssdk-qemuarm64.ext4,if=none,format=raw \ -device virtio-blk-device,drive=disk0 \ -soundhw hda \ -device VGA,edid=on \ -device qemu-xhci -device usb-tablet -device usb-kbd \ -object rng-random,filename=/dev/urandom,id=rng0 \ -device virtio-rng-pci,rng=rng0 \ -device virtio-serial-device -chardev null,id=virtcon -device virtconsole,chardev=virtcon \ -m 2048 \ -kernel ~/images/agl/Image \ -append "console=ttyAMA0 root=/dev/vda" \ -initrd ~/images/agl/initramfs-netboot-image-qemuarm64.ext4 \ -display gtk,show-cursor=on My command line complains that: qemu-system-aarch64: warning: '-soundhw hda' is deprecated, please use '-device intel-hda -device hda-duplex' instead A couple of units fail on startup: [FAILED] Failed to start Driver configuration for Unicens2. See 'systemctl status unicens-config.service' for details. [FAILED] Failed to start Binding for Networking. See 'systemctl status afm-service-agl-s…etwork--1.0--main.service' for details. Adding -smp 4 makes things move a little faster. You can login on the console as root (no passwd). Booting with Xen as a Dom0: This uses the guest-loader device to setup the Dom0 kernel/initrd. I boosted the system memory to 4G and allocated 2G to Dom0. You can find my QEMU tree with guest-loader support at: https://github.com/stsquad/qemu/tree/xen/guest-loader-v1 The command line is as before except more memory and a different -kernel line to load Xen: ./aarch64-softmmu/qemu-system-aarch64 -cpu cortex-a57 \ -machine type=virt,virtualization=on,gic-version=3 \ ... -m 4096 \ -kernel ~/lsrc/xen.git/xen/xen \ -append "dom0_mem=2G,max:2G loglvl=all guest_loglvl=all" \ -device guest-loader,addr=0x42000000,kernel=$HOME/images/agl/Image,bootargs="console=ttyAMA0 root=/dev/vda" \ -device guest-loader,addr=0x47000000,initrd=$HOME/images/agl/initramfs-netboot-image-qemuarm64.ext4 \ -display gtk,show-cursor=on \ -smp 4 The system boots up although I don't get any kernel messages from the console which Xen still has control over. It throws up some complains about un-handled writes: (XEN) *************************************************** (XEN) No support for ARM_SMCCC_ARCH_WORKAROUND_1. (XEN) Please update your firmware. (XEN) *************************************************** (XEN) 3... 2... 1... (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input) (XEN) Freed 364kB init memory. (XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER4 (XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER8 (XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER12 (XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER16 (XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER20 (XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER24 (XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER28 (XEN) d0v0: vGICR: SGI: unhandled word write 0x000000ffffffff to ICACTIVER0 (XEN) d0v1: vGICR: SGI: unhandled word write 0x000000ffffffff to ICACTIVER0 (XEN) d0v2: vGICR: SGI: unhandled word write 0x000000ffffffff to ICACTIVER0 (XEN) d0v3: vGICR: SGI: unhandled word write 0x000000ffffffff to ICACTIVER0 Although I couldn't access the console I was able to ssh into the machine via my 2222 port-fwd. Note that working VirtIO under Xen Dom0 in TCG isn't super impressive as in the emulation case the VirtIO devices behave just like real HW which Dom0 has pass-through access to. The next step is to try this under KVM on a native aarch64 system. I'm still having issues with Xen which is being discussed in other threads. -- Alex Bennée

3 years, 7 months

1
0
0 0

Re: [Stratos-dev] Limited memory sharing investigation

by Alex Bennée

Arnd Bergmann via Stratos-dev <stratos-dev(a)op-lists.linaro.org> writes: > On Wed, Oct 7, 2020 at 12:37 AM Stefano Stabellini > <stefano.stabellini(a)xilinx.com> wrote: >> On Fri, 2 Oct 2020, Arnd Bergmann via Stratos-dev wrote: >> > > In a previous discussion [1], several people suggested using a vIOMMU to >> > > dynamically update the mappings rather than statically setting a memory >> > > region usable by the backend. I believe that approach is still worth >> > > considering because it satisfies the security requirement and doesn't >> > > necessarily have worse performance. There is a trade-off between bounce >> > > buffers on one hand, and map notifications on the other. >> > > >> > > The problem with static regions is that all of the traffic will require >> > > copying. Sub-page payloads will need bounce buffering anyway, for proper >> > > isolation. But for large payloads bounce buffering might be prohibitive, >> > > and using a virtual IOMMU might actually be more efficient. Instead of >> > > copying large buffers the guest would send a MAP request to the >> > > hypervisor, which would then map the pages into the backend. Despite >> > > taking plenty of cycles for context switching and setting up the maps, it >> > > might be less costly than copying. >> > >> > Agreed, I would think the iommu based approach is much more promising >> > here. >> >> The two approaches are not mutually exclusive. The first approach could be >> demoed in a couple weeks, while this approach will require months of >> work and at least one new virtio interface. >> >> My suggestion would be to hack together a pre-shared memory solution, or >> a hypervisor-mediated solution with Argo, do some benchmarks to >> understand the degradation, and figure out if the degradation is bad >> enough that we need to go down the virtio IOMMU route. > > Yes, this makes sense. I'll see if I can come up with a basic design for > virtio devices based on pre-shared memory in place of the virtqueue, > and then we can see if we can prototype the device side in qemu talking > to a modified Linux guest. If that works, the next step would be to > share the memory with another guest and have that implement > the backend instead of qemu. That seems like a reasonable set of steps. So the pre-shared region will be the source of memory for the virtqueues as well as the "direct" buffers the virtqueues reference? > >> Xen PV drivers have started with the equivalent of a virtio IOMMU in >> place, which we call "grant table". A virtual machine uses the grant >> table to share memory explicitly with another virtual machine. >> Specifically, the frontend uses the grant table to share memory with the >> backend, otherwise the backend is not allowed to map the memory. >> >> (There is a question on whether we could standardize the grant table >> interface.) >> >> Speaking from that experience, we ended up switching PV network to use >> hypervisor-based copy (without Argo; Argo came later and it is a more >> generic solution) because it was faster than the alternatives. We are >> still using the grant table for everything else (block, framebuffer, >> etc.) > > My feeling is that the grant table approach is too specific to Xen > and wouldn't lean itself to porting to most other hypervisors. The > idea of picking virtio seems to be based around the assumption that > this is already portable. > > Adding a vIOMMU requirement for regular virtio devices seems > possible and builds on existing guest drivers, but it certainly adds > complexity in all areas (front-end, hypervisor and back-end) without > being an obviously faster than a simpler approach. > > OTOH, we can probably use the existing grant table implementation > in Xen for a performance comparison, assuming that a virtio+viommu > based approach would generally be slower than that. With the existing grant tables we should be able to test a DomU vhost-user with a pre-shared chunk of memory if we can someway to pass signalling events to the DomU guest. The front-end could be in the main Dom0 initially. I'm hoping this is something that Akashi can look at once he's up to speed with Xen although I expect it might lag the KVM based PoC a bit. >> The take away is that the results might differ significantly, not just >> between a protocol and the other (net and block), but also between >> hypervisors (Xen and KVM), and between SoCs. These are difficult >> waters to navigate. Definitely - more data points required ;-) > > Agreed. > > Arnd -- Alex Bennée

3 years, 7 months

3
6
0 0

Blog post on out-of-process emulation

by Alex Bennée

Hi, Stefan from the QEMU project has written a nice blog post on out-of-process emulation (i.e. doing things outside of QEMU in this case). It's a bit of a brain dump but useful for understanding some of the issues associated with device emulation. It doesn't cover our proposed isolation ideas but is a good primer non the less. http://blog.vmsplice.net/2020/10/requirements-for-out-of-process-device.html -- Alex Bennée

3 years, 7 months

1
0
0 0

Re: [Stratos-dev] Linaro Stratos Meeting Agenda

by Arnd Bergmann

On Mon, Oct 12, 2020 at 12:07 PM Mike Holmes via Stratos-dev <stratos-dev(a)op-lists.linaro.org> wrote: > We have a small agenda for this week [1], mostly just updates on progress, > do we have any further items to raise ? > > Jean-Phillippe / Arnd do you want to take any time to talk about VirtIO-MMU > support ? It looks like there was a bit of a thread [2] Sure, I can do that. I'll try to explain there what my current understanding is where we are, and then Jean-Phillippe and others can step in and correct me or provide additional information. Arnd

3 years, 7 months

1
0
0 0

2024

2023

2022

2021

2020

Stratos-dev October 2020