Ghaf-Specific microvm Hypervisor Options

microvm is the component defining a VM’s launch services generated for systemd. It inputs a set of options mapped to the hypervisor command line call.

Nevertheless, it may happen that some hypervisor options are not supported by microvm. For example, adding specific devices. This document considers such cases.

Selecting a System VM VMM

QEMU and crosvm are VMMs (virtual machine monitors) running on top of KVM, not hypervisors themselves. Ghaf selects the system VM VMM through ghaf.virtualization.vmConfig. QEMU is the system-wide default. The x86 laptop profile keeps that default, but every generic intel-laptop target selects crosvm for all system VMs. This includes debug, release, low-memory, and store-disk variants. Machine-specific x86, aarch64, and generic VM targets keep their existing selections. Generic Intel laptop targets also select crosvm for encrypted AdminVM variants and use crosvm’s TPM passthrough path. Each system VM can override the target default independently:

vmConfig = {
  defaultSysVmVmm = "qemu";
  sysvms = {
    adminvm.vmm = "qemu"; # Optional AdminVM fallback
    netvm.vmm = "qemu"; # Optional x86 laptop rollback
  };
};

System VM keys use unhyphenated names such as adminvm, netvm, and guivm. The selection takes effect only when the VM’s evaluated configuration includes lib.ghaf.vm.applyVmConfig. Profiles and targets that construct an evaluatedConfig directly must include this helper; otherwise the default and per-VM settings are ignored. This supports a mixed fleet during migration instead of requiring every VM on a target to use the same VMM. Note that microvm.nix names its own selector microvm.hypervisor; Ghaf maps vmm onto it.

AdminVM declares its vsock CID through the VMM-independent microvm.vsock.cid option. Both the QEMU fallback and the crosvm default therefore use the VM’s configured network CID without raw QEMU arguments.

Selecting an App VM VMM

App VMs use QEMU by default. Generic intel-laptop targets select crosvm for their App VMs; machine-specific x86, aarch64, and generic VM targets retain QEMU. The target default and individual App VMs can be overridden through the same ghaf.virtualization.vmConfig interface:

vmConfig = {
  defaultAppVmVmm = "crosvm";
  appvms = {
    chrome.vmm = "qemu";
    flatpak.vmm = "crosvm";
  };
};

App VM keys use their unhyphenated base names, such as chrome, flatpak, and chromium. Encrypted App VMs remain on the crosvm default. Ghaf’s crosvm build connects its virtio TPM frontend to the same swtpm proxy socket as QEMU, and the App VM guest kernel includes ChromiumOS’s virtio TPM transport driver when that crosvm App VM has a vTPM enabled. Applying this out-of-tree driver creates a separate guest-kernel derivation until that result is available from a binary cache.

crosvm App VMs use the same mem * (balloonRatio + 1) allocation and virtio-balloon policy as QEMU App VMs. ghaf-mem-manager selects QEMU QMP or the crosvm control command from the VM’s configured VMM. It also translates between QEMU’s guest-visible memory value and crosvm’s reclaimed-memory value.

App VMs declare their vsock CID through microvm.vsock.cid, so QEMU and crosvm use the same configured CID. Runtime USB attachment through vhotplug is supported by crosvm. vhotplug records the crosvm guest port for each host USB device so identical devices can be detached independently and restored safely after daemon restarts. x86 laptop system VMs also resolve their configured PCI devices through vhotplug at startup and support runtime PCI attachment with crosvm’s removable VFIO slots.

crosvm Sandbox Boundary

Ghaf runs VMM services as the unprivileged microvm user. crosvm’s internal multiprocess minijail requires CAP_SYS_ADMIN to create PID and mount namespaces, which the service intentionally does not have. Ghaf consequently adds --disable-sandbox to crosvm system VM and App VM runners by default.

This keeps the VMM process behind the unprivileged systemd service boundary, but it disables crosvm’s additional per-device-process isolation. Treat this as an explicit migration tradeoff until Ghaf provides capability-scoped namespace setup without granting the VMM broad host privileges.

To inspect a deployed VM’s selected VMM and generated command line, run (the file name comes from microvm.nix):

cat /var/lib/microvms/admin-vm/current/share/microvm/hypervisor
admin_pid="$(systemctl show --property MainPID --value microvm@admin-vm)"
tr '\0' ' ' <"/proc/${admin_pid}/cmdline"

Options Definitions

A VM is defined under Ghaf’s subdirectory microvmConfigurations/VM_NAME/default.nix, for example:

modules/virtualization/microvm/netvm.nix
https://github.com/tiiuae/ghaf/blob/main/modules/virtualization/microvm/netvm.nix

This file contains hypervisor’s options for running the VM. For each hypervisor there is a bunch of microvm’s defined options: https://astro.github.io/microvm.nix/options.html

The way they are processed can be found in corresponding .nix files (runners) in the astro/microvm.nix repository:

The formula for setting hypervisor option is microvm.option = value;. For example:

microvm.mem = 512;
microvm.vcpu = 2;

Generated Hypervisor Start Commands

As a result of building the Ghaf tree, command lines for starting the VMs are generated. They reflect all parameters specified above—both those specified explicitly and defaults. They are located under the Ghaf’s /var/lib/microvms/ directory.

ls /var/lib/microvms/memsharevm-vm-debug/current/bin

microvm-balloon
microvm-console
microvm-run
microvm-shutdown

The command which starts the hypervisor is the microvm-run bash script. Here is a sample generated:

 #! /nix/store/96ky1zdkpq871h2dlk198fz0zvklr1dr-bash-5.1-p16/bin/bash -e
exec '/nix/store/zsf59dn5sak8pbq4l3g5kqp7adyv3fph-qemu-host-cpu-only-7.1.0/bin/qemu-system-x86_64' '-
name' 'memshare' '-M' 'microvm,accel=kvm:tcg,x-option-roms=off,isa-serial=off,pit=off,pic=off,rtc=off,
mem-merge=on' '-m' '2512' '-cpu' 'host' '-smp' '17' '-machine' 'virt,accel=kvm' '-nodefaults' '-no-
user-config' '-nographic' '-no-reboot' '-serial' 'null' '-device' 'virtio-serial-device' '-chardev'
'pty,id=con0' '-device' 'virtconsole,chardev=con0' '-chardev' 'stdio,mux=on,id=con1,signal=off' '-
device' 'virtconsole,chardev=con1' '-device' 'virtio-rng-device' '-drive' 'id=root,format=raw,
media=cdrom,file=/nix/store/xnnqb3sb1l4kbx7s0ijazph5r0c0xhx5-rootfs.squashfs,if=none,aio=io_uring' '-
device' 'virtio-blk-device,drive=root' '-kernel' '/nix/store/ds5cmyby0p4ikw91afmrzihkz351kls7-linux-
6.2/bzImage' '-append' 'console=hvc1 console=hvc0 reboot=t panic=-1 root=/dev/vda ro init=/init
devtmpfs.mount=0 stage2init=/nix/store/0mbhpna8hplbsaz1il3n99f0zincr4vs-nixos-system-memshare-
22.11.20230310.824f886/init boot.panic_on_fail loglevel=4 regInfo=/nix/store
/j8id92qsd58qjnzq4xz6v5l38rlpq6is-closure-info/registration' '-sandbox' 'on' '-qmp' 'unix:memshare.
sock,server,nowait' '-device' 'virtio-balloon' '--option 1 --option 2'

for the input parameters:

microvm.hypervisor = "qemu";

Note that microvm sets several others.

microvm.mem = 2000;
microvm.balloonMem = 512;
microvm.vcpu = 17;
microvm.qemu.extraArgs = [ "--option 1 --option 2" ];

Adding Option to Hypervisor Command Line

microvm may not supply parameters for all possible options as adding specific devices. Processing of all microvm configuration options is done in the mentioned above hypervisor’s runner .nix file.

The runners support the extraArgs parameter. It allows setting any option in QEMU command line invocation. Its value is a list of strings.

In this example the following extraArgs definition:

microvm.qemu.extraArgs = [
"-object memory-backend-file,id=mem1,mem-path=/dev/shm/virtio_pmem.img"
"-device virtio-pmem-pci,memdev=mem1,id=nv1"
];

results in the generated command line parameters:

'-object memory-backend-file,id=mem1,mem-path=/dev/shm/virtio_pmem.img' '-device v
irtio-pmem-pci,memdev=mem1,id=nv1'