Containers in 2010

 6 min vps-lxc: templates as btrfs subvolumes, containers as writable snapshots, and the host-local image model I built around them.

I was fighting libvirt’s LXC driver—EPOLLHUP handling, cgroup plumbing—and eventually dropped the middle layer and was driving LXC directly. I posted the result to lxc-users:

well i was going to do a handful more things before releasing/sharing this, but i think it’s more than complete enough for others to start tampering with if they wish.

Daniel Lezcano, LXC’s maintainer, replied in the thread. I had hit a wall with lxc.network.type=empty while building the tool, and his answer was:

You find a bug :) It’s regression regarding the previous version. Will fix it.

The tool was vps-lxc, built around the LXC and btrfs interfaces then available.

Templates fork into domains

The core model: a template is a btrfs subvolume holding a built root filesystem; a domain—a container—is a writable snapshot of a template. Every template/domain root lifecycle operation routes through one function with three verbs:

case "${VPS_USE_FS_METHOD}" in
btrfs)
    case "${method}" in
    init)
        btrfsctl -S ${target_name} ${target_base}
    ;;
    fork)
        btrfsctl -s ${target_base}/${target_name} ${orig_base}/${orig_name}
    ;;
    del)
        btrfsctl -D ${target_name} ${target_base}
    ;;
    esac
;;

init creates a subvolume, fork snapshots one, del destroys one. A plain-cp backend sits behind the same three verbs for hosts without btrfs, selected at runtime. Forking a domain from a template is a copy-on-write metadata operation: it creates the writable root without copying the template’s bytes up front. That storage shape is one reason later image-backed containers could start quickly.

btrfsctl was the command-line tool at the time, before btrfs(8).

Exceptions, in bash, with stack traces

The tool is bash, which has no structured exception or unwind facility, so vps-lxc carries its own. vps_exception ${LINENO} fatal "msg" ${FUNCNAME} builds cmd(func):line level: msg records into a stack array; fatal runs every registered unwind handler in order, dumps the VPS_ variables under debug, prints the accumulated trace, and exits through a trap:

fatal)
    # run handlers, drop stack, exit
    local x
    VPS_STACK=("${VPS_STACK[@]}" "${exception}")
    # run the fatal exception handlers in order
    for x in "${VPS_FATAL[@]}"; do ${x}; done
    # if debug drop variables
    ${VPS_DEBUG:-false} && \
        echo "DEBUG enabled... variable list:" && \
        { set | grep ^VPS | sed -e "s/^/   /g"; }
    # drop the stack
    echo "FATAL exception encountered... stacktrace:"
    for x in "${VPS_STACK[@]}"; do echo "   ${x}"; done
    # exit and trip the exit handler
    exit 1
;;

Commands register and unregister handlers around risky sections. A template build that returns failure is routed into fatal, which deletes the half-built snapshot. The upgrade path registers an unmount handler before bind-mounting /dev, /sys, and /proc; an explicit pacman failure, SIGINT, or SIGTERM reaches that handler before the outer delete. This is a manual compensation stack, not general shell exception handling: an unchecked failing command can bypass it entirely.

Power management by inittab

LXC had no lxc-shutdown yet, so the tool built both ends of a clean-stop channel itself. On the host side, stop walks the container cgroup’s tasks file, finds the process whose comm is init, and signals it:

comm="$(ps --no-headers -o comm -p ${x})"
if [ "${comm}" = "init" ]; then
    kill -PWR ${x}

reboot does the same with SIGINT. On the guest side, the template build rewrites the container’s inittab so those signals mean what the host intends:

sed -i -e "s,agetty,\\0 -n -l /bin/autologin,g" -e "/^[^irsc]\|^c[^1]\|^$/d" rootfs/etc/inittab
cat <<EOF >> rootfs/etc/inittab
p6::ctrlaltdel:/sbin/init 6
p0::powerfail:/sbin/init 0
EOF

SIGPWR is powerfail, so the container’s init runs init 0. SIGINT is ctrlaltdel, so it runs init 6. ACPI power-button semantics, reinvented with Unix signals, with the host command and the guest inittab generated by the same tool—because LXC did not yet supply that lifecycle.

The same build-both-ends instinct shows up smaller elsewhere. The template’s bashrc reads /proc/self/cgroup, extracts the trailing path component, and prefixes the prompt with VPS[name]—“am I in a container, and which one?” answered from /proc, purely to keep humans from wrecking the wrong shell. Console accounting never touches a pidfile: enter counts CONNECTED unix sockets on the container’s LXC command socket with netstat -ax and refuses entry when the count reaches the domain’s tty limit, and bash completion derives the running-container list from the LISTENING side of the same table. The kernel’s socket table supplied the count without a separate session registry.

Shutdown raises a harder question: how does the host know the container finished going down? The answer was a per-container supervisor that watched the guest’s own utmp file with inotify and read its runlevel with runlevel(8), which accepts an alternate utmp path. Runlevel 0 meant stop the container; runlevel 6 meant stop it and start it again, because a container’s init cannot reboot a kernel. In the LXC thread I opened, the monitor combined my one-remaining-process test, Michael H. Warfield’s guest-utmp runlevel distinction, and Daniel Lezcano’s inotify wakeup. The full technique and attribution are small enough to be their own note: runlevel-utmp.

One more piece deserves its dateline: the whole package is an FHS-shaped tree meant to be cloned straight to /vps and run in place—a profile script recovers the install prefix from wherever the config file turned up, so the same tree works at /, /vps, or /opt/anything with no install step. Subcommands are sourced plugin files driven through a fixed lifecycle—the framework owns the getopts loop, the plugin owns the option semantics—with a runnable command.proto as the documented template. The filesystem discovery rhymes with Git’s external git-* commands. The lifecycle does not: vps-lxc sources each plugin into one dispatcher-owned process and option loop.

What it never had

The repository stopped after about a month of work. The mailing-list discussions continued: copy-on-write deduplication, default guests, and integrating LXC with systemd.

Some of that was timing—btrfs was experimental, LXC was rough, and the INSTALL doc carried a long list of kernel options just to get a container running. But the structural gap is plainer: a template existed only on the host that built it. There was no way to push one, pull one, or name one globally—no registry, no distribution story at all.

vps-lxc stopped at host-local templates. I then applied copy-on-write snapshots to the root filesystem itself: never boot the real root.