Designing a crash-safe single-file container
BFC's on-disk format puts the index at the end, opens a container from its last 56 bytes, and never rewrites what is already written. What that buys, and what it leaves visible.
BFC packs a directory tree into one file. It keeps the parts of POSIX metadata that matter (mode, mtime in nanoseconds, symlinks), gives random access to any single file, and can compress, encrypt and verify everything. The format is short enough to describe in full.
Layout
[ header 4 KiB ][ object ][ object ] … [ index ][ footer 56 B ]
All integers are little-endian.
- Header: a magic string, the block size, a features bitmask (ZSTD, AEAD), a UUID and a salt for key derivation.
- Objects: each has its own header (type, compression, encryption, mode, mtime, original and stored sizes, CRC32C), then the path, padding, and content aligned to 16 bytes.
- Index: written after the objects and sorted by path.
- Footer: tags, plus the offset, size and CRC of the index.
Write forward, never back
A writer only ever appends. When it adds a file, it writes a provisional object header, streams the content (compressing and encrypting as it goes), then goes back to fill in the header’s final sizes and CRC. That back-patch touches only the object just written. The index and footer go last.
As a result, an interrupted write never damages data that was already in the file. Readers locate everything through the footer, so a container is complete exactly when its footer is.
Open from the end
A reader opens a container by reading the last 56 bytes. The footer points at the index, the index CRC is checked, and from then on any path is a binary search away: O(log n) lookups, plus cheap prefix listing because the index is sorted.
Opening doesn’t depend on how many files there are. You never scan the data region to find a file.
Checksums in hardware
Each object carries a CRC32C. On x86-64, BFC detects SSE4.2 with cpuid and uses _mm_crc32_u64. On ARM64 it uses the ARMv8 CRC instructions. Everywhere else it uses a portable software fallback. bfc verify --deep re-checks every object’s CRC.
Compress, then encrypt, per file
- Compression: ZSTD at levels 1 to 22, default 3. The writer samples content to decide whether compression is worth it, and files under 64 bytes are stored as they are.
- Encryption: each file is sealed separately with ChaCha20-Poly1305 and its own nonce, using libsodium. The key comes from Argon2id over a password, or from a 32-byte key file.
The order is always compress first, then encrypt. The other way round, ciphertext doesn’t compress.
bfc create -c zstd -e secret -l 6 archive.bfc /data/
bfc verify --deep archive.bfc
What stays visible
Paths, sizes and metadata are not encrypted. Anyone holding the file can see its structure: which files exist, how large they are and when they were modified. Only the contents are confidential. That is a deliberate trade-off. It keeps the index plain and searchable without a key, and it lets read-only kernel drivers mount the tree.
Defensive reading
Paths are normalized on the way in: no absolute paths, no ... Extraction opens files with O_NOFOLLOW, so a crafted symlink can’t redirect a write outside the target directory. The parser has fuzz targets, because the reader’s job is to take a file someone else wrote and not trust it.