FreeBSD from Go, without shelling out

The usual way to automate FreeBSD from another language is to run jail, ifconfig, pfctl or zfs and parse what they print. That works until the output format changes, an error message turns out to be the only signal, or you need to read state back reliably to reconcile it.

So I built the layer underneath instead: a set of Go libraries that call the same kernel and library interfaces the base tools use.

Library	Replaces	Talks to
go-freebsd-ifc	ifconfig, route	SIOC* ioctls, PF_ROUTE sockets, NET_RT_DUMP
go-freebsd-netgraph	ngctl	libnetgraph, netgraph(4) control sockets
go-freebsd-jail	jail(8)	libjail jailparam_*, jail_set(2)
go-freebsd-pf	pfctl	/dev/pf ioctls
go-freebsd-libzfs	zfs, zpool	libzfs, libnvpair
go-freebsd-vmm	bhyvectl	libvmmapi, vmm(4)

On top of them sits go-freebsd-jails-exporter, a Prometheus exporter whose README can honestly say: no shell commands are executed.

Shared conventions

Each library is written independently, but they all follow the same rules, and that’s what makes them compose.

cgo stays small and captures errno. The public packages are plain Go, and the C lives in a thin internal shim. Each C wrapper returns the result and the errno and, where the C library provides one, its error string. That keeps the information intact across the cgo boundary. The libjail shim, for example, returns {jid, errnum, errmsg[1024]}.

Errors are typed. Each library has sentinels (ErrNotFound, ErrPermission, ErrExists, ErrBusy) and a structured operation error that records what was being done to which object. All of them work with errors.Is and errors.As.

Everything builds everywhere. Files guarded by build tags provide stubs on other operating systems that return ErrNotSupported. A project can depend on these libraries and still build and unit-test on a laptop.

Mutations are idempotent where that is the honest answer. Adding a bridge member that’s already there returns nil. Deleting an address that’s already gone succeeds. jail.CreateOrUpdate converges. The exceptions are deliberate: removing a VXLAN peer that doesn’t exist is an error, because it usually means the caller’s view of the table is stale.

Root-only tests are opt-in. Unit tests run anywhere. End-to-end tests need root and a FreeBSD host, and they sit behind a build tag or a make target.

Where the rule bends

go-freebsd-jail still runs jexec to execute commands inside a jail. The alternative, calling jail_attach from the Go process, would move a multithreaded runtime into the jail, and that isn’t safe. A subprocess is the right tool there.

go-freebsd-pf couldn’t use raw syscall.Ioctl either. The pfvar.h structs are full of nested unions, bitfields and layouts that change between releases, so the library uses cgo with small C accessors for union members. A side effect is that binaries must be built for the FreeBSD release they run on.

go-freebsd-bhyve runs bhyve(8) as a process on purpose. Device emulation lives in bhyve’s userspace, so there is no library call that boots a guest. What it avoids is the rest of the shell: the configuration is rendered from typed values rather than argv strings, exit statuses are decoded instead of guessed, and the leftover VM is destroyed through go-freebsd-vmm rather than bhyvectl.

How they fit together

go-freebsd-jails-exporter
├── go-freebsd-jail ── go-freebsd-ifc   (vnet: epair + bridge)
└── go-freebsd-libzfs                   (per-jail datasets)

go-freebsd-pf                           (standalone)
go-freebsd-netgraph                     (standalone)

go-freebsd-bhyve
└── go-freebsd-vmm                      (destroy, PCI addresses)

The exporter collects four things concurrently, each with its own timeout:

• jail inventory, from go-freebsd-jail;
• processes per jail, from sysctl(KERN_PROC_ALL);
• RACCT resource usage, from rctl_get_racct("jail:<name>");
• ZFS usage, by matching each jail’s path to the dataset mounted there.

Together that is a jail platform in Go, built layer by layer: network, then jails, firewall and storage, then observability.