Hi Viresh/Mathieu,
This is a dump of the current state of my notes for cross-building and running Xen via QEMU. It's a little bit of a stream of consciousness as I wrote things down but hopefully it proves useful in getting things started.
━━━━━━━━━━━━━━━━━━━ XEN RELATED NOTES
Alex Bennée ━━━━━━━━━━━━━━━━━━━
Table of Contents ─────────────────
1. Tasks [0/1] :tasks: .. 1. TODO Xen SMC/HVC pass through ([STR-41]) 2. Notes .. 1. Building Xen .. 2. Cross Build .. 3. Cross Packages .. 4. Configure .. 5. Build ..... 1. Just the Hypervisor .. 6. Running Dom0 .. 7. MachiatoBin Issues .. 8. Running DomU .. 9. View console of DomU guest
1 Tasks [0/1] :tasks: ═════════════
[STR-41] https://projects.linaro.org/browse/STR-41
1.1 TODO Xen SMC/HVC pass through ([STR-41]) ────────────────────────────────────────────
sstabellini_: that tracks [ 1.420679] mvebu-comphy f2120000.phy: unsupported SMC call, try updating your firmware [ 1.420720] mvebu-comphy f2120000.phy: Firmware could not configure PHY 4 with mode 15 (ret: -95), trying legacy method sstabellini_: is that a boot option? <sstabellini_> stsquad: as an example see [19:55] xen/arch/arm/platforms/xilinx-zynqmp-eemi.c:forward_to_fw <sstabellini_> stsquad: we don't have a boot option for that, you need to add a few lines of code. Actually this issue comes up often enough that a boot option would be really useful!
[STR-41] https://projects.linaro.org/browse/STR-41
2 Notes ═══════
2.1 Building Xen ────────────────
As Xen's out-of-tree support is a little sporadic I've ended up going for using git-worktree to split out the various builds.
2.2 Cross Build ───────────────
2.3 Cross Packages ──────────────────
┌──── │ apt install libpython3-dev:arm64 libfdt-dev:arm64 libncurses-dev:arm64 └────
Or as I do on hackbox do everything in a container:
┌──── │ docker run --rm -it -u (id -u) -v $HOME:$HOME -w (pwd) alex.bennee:xen-arm64 /usr/bin/fish └────
built from [my dockerfile]
[my dockerfile] https://github.com/stsquad/dockerfiles/blob/master/crossbuild/xen-arm64/Dockerfile
2.4 Configure ─────────────
┌──── │ ./configure --build=x86_64-unknown-linux-gnu --host=aarch64-linux-gnu \ │ --disable-docs --disable-golang --disable-ocamltools \ │ --with-system-qemu=/usr/bin/qemu-system-i386 └────
The location of system QEMU can be tweaked after the fact by editing /etc/default/xencommons
2.5 Build ─────────
You need to have the compilers in the command line:
┌──── │ make -j9 dist CROSS_COMPILE=aarch64-linux-gnu- XEN_TARGET_ARCH=arm64 └────
To build an Debian package which can install all the tools and binaries (replacing any distro requirements) then:
┌──── │ make -j9 debball CROSS_COMPILE=aarch64-linux-gnu- XEN_TARGET_ARCH=arm64 └────
This builds a package which you can find in ./dist/xen-upstream-4.15-unstable.deb which you can dpkg -i on your dom0 environment.
2.5.1 Just the Hypervisor ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
┌──── │ cd xen │ set -x CROSS_COMPILE aarch64-linux-gnu- │ make XEN_TARGET_ARCH=arm64 └────
2.6 Running Dom0 ────────────────
To avoid complications with broken grub and bios firmware I'm currently just using my direct boot:
┌──── │ ./qemu-system-aarch64 -machine virt,virtualization=on \ │ -cpu cortex-a57 -serial mon:stdio \ │ -nic user,model=virtio-net-pci,hostfwd=tcp::2222-:22 \ │ -device virtio-scsi-pci \ │ -drive file=/dev/zvol/hackpool-0/debian-buster-arm64,id=hd0,index=0,if=none,format=raw \ │ -device scsi-hd,drive=hd0 \ │ -display none \ │ -m 8192 \ │ -kernel ~/lsrc/xen/xen.build.arm64-xen/xen/xen \ │ -append "dom0_mem=2G,max:2G dom0_max_vcpus=4 loglvl=all guest_loglvl=all" \ │ -device guest-loader,addr=0x46000000,kernel=$HOME/lsrc/linux.git/builds/arm64/arch/arm64/boot/Image,bootargs="root=/dev/sda2 console=hvc0 earlyprintk=xen" \ │ -smp 8 └────
Care has to be taken to avoid the guest-loader address clashing with anything else and ending up corrupting the DTB. Currently QEMU doesn't do anything avoid those issues as it doesn't have visibility of where the kernel gets loaded. You can see by examining the Xen output:
┌──── │ (XEN) Loading zImage from 0000000046000000 to 0000000050000000-0000000050eb2200 │ (XEN) Loading d0 DTB to 0x0000000058000000-0x0000000058001ce8 └────
The initial xl list will check Xen and it's tools are up and running and the hypervisor and userspace ABI's are in sync. It's important that the hypercall ABI and it's userspace are in sync.
┌──── │ 13:13:43 [root@buster:~] # xl list │ Name ID Mem VCPUs State Time(s) │ Domain-0 0 4096 4 r----- 215.8 └────
2.7 MachiatoBin Issues ──────────────────────
• upgraded to Bullseye for latest Grub • built with 8250 serial (32 bit, shift = 2)
2.8 Running DomU ────────────────
The with a config like
┌──── │ # =====================================================================# Example PV Linux guest configuration │ # ===================================================================== │ # │ # This is a fairly minimal example of what is required for a │ # Paravirtualised Linux guest. For a more complete guide see xl.cfg(5) │ │ # Guest name │ name = "xenpv-initrd-guest" │ │ # 128-bit UUID for the domain as a hexadecimal number. │ # Use "uuidgen" to generate one if required. │ # The default behavior is to generate a new UUID each time the guest is started. │ #uuid = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" │ │ # Kernel image to boot │ kernel = "/home/alex/arm64-minkern" │ # kernel = "/home/alex/arm64-kernel" │ # kernel = "/home/alex/arm64-defconfig" │ │ # Ramdisk (optional) │ #ramdisk = "/boot/initrd.gz" │ │ # Kernel command line options │ extra = "console=hvc0" │ │ # Initial memory allocation (MB) │ memory = 4096 │ │ # Maximum memory (MB) │ # If this is greater than `memory' then the slack will start ballooned │ # (this assumes guest kernel support for ballooning) │ maxmem = 4096 │ │ # Number of VCPUS │ vcpus = 2 │ │ # Network devices │ # A list of 'vifspec' entries as described in │ # docs/misc/xl-network-configuration.markdown │ # vif = [ ] │ │ # Disk Devices │ # A list of `diskspec' entries as described in │ # docs/misc/xl-disk-configuration.txt │ # disk = [ '/dev/vg/guest-volume,raw,xvda,rw' ] │ # disk = [ ] └────
┌──── │ xl create simple-guest.conf │ xl list │ Name ID Mem VCPUs State Time(s) │ Domain-0 0 4096 4 r----- 221.4 │ xenpv-initrd-guest 1 4095 2 -b---- 5.3 └────
2.9 View console of DomU guest ──────────────────────────────
And to get console:
┌──── │ xl console -t pv -n 0 xenpv-initrd-guest └────
to exit you need *Ctrl-[*