go-freebsd-bhyve

Why

go-freebsd-vmm can inspect and control a virtual machine but can’t boot one, because device emulation lives in bhyve’s userspace process. This is the missing half: it describes a VM, runs bhyve(8) to execute it, supervises that process, and interprets how it ended.

          application
               |
      go-freebsd-bhyve      configuration, process lifecycle
               |
          bhyve(8)          device models, boot ROM
               |
      go-freebsd-vmm        runtime inspection and control
               |
      libvmmapi / vmm(4)

Scope

It knows about bhyve — its configuration variables, its command line, what its exit statuses mean — and nothing about the resources a VM uses. Disk paths must already exist and network backends must already be configured: nothing here creates a ZFS volume, a tap(4) interface or a bridge, or writes a firewall rule. Those belong to go-freebsd-libzfs, go-freebsd-ifc and go-freebsd-pf, and composing them belongs to whatever sits above. A VM runner that provisioned its own storage couldn’t be used with storage it didn’t provision, and couldn’t be tested on a host without it.

Three things about bhyve worth knowing

Each was established by reading bhyve’s sources and then checking against the real thing, and each shaped the API.

• A VM outlives the process that ran it. bhyve leaves the VM in the kernel when it exits, so Destroy is a required step, not a tidy-up — until it runs, the VM holds its name and its guest memory. Start refuses to run over a leftover rather than hand the previous guest’s memory and device state to the new one. Destroying goes through go-freebsd-vmm, not bhyvectl.
• Exit status 1 is ambiguous. bhyve uses it for a guest powering itself off and for most of its own startup failures — ten call sites, including a successful configuration dump. So ExitReason carries Started, established from the device node rather than the status, and 1 decodes to ExitPoweredOff or ExitStartupFailed accordingly. The rest are distinct: 0 is a reboot, 2 a halt, 3 a triple fault, 4 bhyve’s own failure, 64 a rejected command line.
• A graceful stop is not always graceful. Stop sends SIGTERM, which bhyve turns into an ACPI power button press — but only once the guest has enabled ACPI. During firmware, early boot, or for a guest that never enables it, SIGTERM kills bhyve outright. The returned reason says which happened, and the difference matters to the guest’s filesystems.

Configuration

A typed Config renders to a bhyve_config(5) file that bhyve reads with -k. PCI slots are assigned stably — host bridge at 0:0:0, LPC at 0:31:0, the rest in configuration order — so a guest’s device names don’t shift between runs, and Config.PCIAddress says where a device ended up. Variable names are compile-time constants, never built from caller input, because bhyve accepts names it doesn’t understand without a word.

Validation comes in two steps with different reach. Check decides what it can locally. Validate runs bhyve -o config.dump=true, which parses the configuration with bhyve’s own parser, reports every variable including its defaults, and exits without creating anything — no privilege needed, which is what makes the configuration layer testable in an ordinary unit test.

Neither is complete, and the limits run both ways. A CPU topology that contradicts the vCPU count passes Validate and fails at Start. And a misspelled variable can’t be caught at all: bhyve stores every name it’s given so values can refer to each other, so a typo comes back in the dump looking healthy.

Testing

Level 1 needs no privilege and no VM, only bhyve installed. It checks the configuration layer against bhyve’s own parser rather than a copy of its rules, so it catches drift when FreeBSD renames or adds a variable. Level 2 starts real bhyve processes as root; every test destroys what it created, even on failure, and leftover /dev/vmm/gobhyve-test-* objects are swept before and after the run.

Status

Done: the configuration model and renderer, validation through bhyve, PCI slot assignment, the process lifecycle with its state machine, exit interpretation and cleanup. Every wait takes a context.Context and honours it.

Next, roughly by usefulness: runtime control as a thin type over go-freebsd-vmm, replacing bhyvectl entirely; console access through nmdm(4) as an io.ReadWriteCloser; bhyveload(8) for FreeBSD guests without UEFI; save and restore through bhyve -r, which needs a BHYVE_SNAPSHOT kernel; and a guest that actually boots in CI — so far no test has booted one, because the development host has no firmware installed.

cfg := bhyve.Config{
	Name:   "web01",
	CPU:    bhyve.CPU{Count: 2},
	Memory: bhyve.Memory{Size: 2 << 30, Wired: true},
	Firmware: &bhyve.Firmware{
		ROM: "/usr/local/share/uefi-firmware/BHYVE_UEFI.fd",
	},
	Disks:      []bhyve.Disk{{Path: "/dev/zvol/zroot/vm/web01"}},
	Networks:   []bhyve.Network{{Backend: "tap0"}},
	ACPITables: true,
}

vm, err := bhyve.New(cfg)
if err != nil {
	return err
}
defer vm.Destroy()

if err := vm.Start(ctx); err != nil {
	return err
}

reason, err := vm.Wait(ctx)

$ git clone https://github.com/zombocoder/go-freebsd-bhyve