systems · netstack · terminal software
3 min readgo-freebsd-ifc

When the Go struct is 92 bytes and the kernel wants 152

How a hand-written rt_msghdr made a whole route package silently useless, and the C-probe test that pins every ABI struct in go-freebsd-ifc now.

go-freebsd-ifc talks to the FreeBSD kernel the way ifconfig and route do: ioctls on a datagram socket for interfaces, and messages on a PF_ROUTE socket for the routing table. It worked well enough until I started building a daemon on top of it. The daemon has to reconcile routes, so it has to add them, delete them and read them back. That is where the library fell apart.

Every routing message was rejected

The kernel never got as far as parsing the routes. In rtsock.c, the first check on an incoming message is its length:

if (len < sizeof(*rtm))
        return (EINVAL);

My Go rtMsghdr was 92 bytes. On FreeBSD 14.3/amd64, sizeof(struct rt_msghdr) is 152. Every message failed that check, so the whole route package had never worked. Four separate mistakes added up to the 60-byte gap:

  • _rtm_spare1 was missing.
  • rtm_inits was a uint32, but in C it is u_long, which is 8 bytes on amd64.
  • The members of rt_metrics are all u_long, so the embedded metrics block is 112 bytes, not 56.
  • The Go field named use sat where the kernel has rtm_fmask.

Each of these is easy to get wrong when you translate a header by reading it. None of them shows up as a compile error. They show up as EINVAL, and that error doesn’t say which field is wrong.

Let C tell Go what the layout is

The fix was to stop deriving the layout by reading net/route.h. A small cgo file now exposes the real numbers from the C compiler — in spirit:

/*
#include <net/route.h>
*/
import "C"

const cSizeofRtMsghdr = C.sizeof_struct_rt_msghdr

A test then checks three things against each other: the Go struct’s size, the length of an encoded message, and the live C sizeof. If a future FreeBSD release changes the header, the test fails with the two sizes side by side. It no longer turns into a library that quietly does nothing.

There is one wrinkle. cgo is not allowed in _test.go files, so the probes live in normal source files (abi.go) and the tests import them. The same pattern now pins the VXLAN structures.

Two more bugs from the same family

The daemon turned up two more defects of the same kind: a C struct whose contents didn’t match what the Go code put in it.

Rename returned EFAULT. SIOCSIFNAME takes the new name through a pointer stored in ifr_ifru. The kernel calls copyinstr(ifr_data_get_ptr(ifr), …) on it. The Go code copied the name inline into the union, so the kernel read 16 bytes of ASCII as a userland address. The fix passes a pointer to a stable buffer.

LAGG protocol selection is still broken, and the README says so. SIOCSLAGG is _IOW('i', 144, struct lagg_reqall), but SetProto passes a struct ifreq. The kernel reads ra_proto from inside ifr_name and returns EPROTONOSUPPORT for every protocol. It’s the same class of bug as Rename, and it’s documented as known rather than hidden.

Reading the table back

A route library that can’t list routes is no use for reconciliation, so List, List4 and List6 now read the table through the NET_RT_DUMP sysctl. There was one more trap here. The address family is taken from sa_family explicitly, never inferred from the address value. net.IP treats ::ffff:0.0.0.0 and 0.0.0.0 as the same address, so an IPv4-mapped IPv6 route would otherwise show up as IPv4.

What I took from it

  • Treat every C struct you mirror in Go as unverified until a test compares it with the compiler’s sizeof.
  • When the kernel returns EINVAL or EFAULT on an ioctl that “obviously” has the right arguments, check the size and union layout before anything else.
  • Carry type information from the wire (sa_family). Don’t reconstruct it from values.
  • Write known bugs down. A documented EPROTONOSUPPORT costs the next person five minutes. An undocumented one costs them an afternoon.
← all writing