I ran Snap Game Services from 2022 to 2025: dedicated servers for competitive VR speedball under the banner “VR Speedball is Real Paintball.” The game server was closed. It did not expose the lifecycle API I wanted, so the public snap-gs supervisor built one from the surfaces it did expose: process exit, stdout, files, and packets.
That sounds like integration until the log format changes. The repository’s README says the real contract plainly: correctness relies entirely on the existing snapshot_server log stream, and those logs may change at whim. The system worked by making that weak boundary explicit, then arranging everything else around it.
The log was the API
Each lobby was a child process with two scanners. The stdout filter classified a line by its first bytes. A JSON object containing shooterId was a kill. One containing matchId was match state. -- BOLT -- Registered player: and its unregister twin moved the lobby through idle, occupied, and full states. “Upload complete” flushed a match early; “Finished populating pool” could mark an empty server idle.
The player IDs carried another observed convention: IDs below 1000 were bots. That let the supervisor distinguish “the last human left” from “the process is empty” even while bots remained. A changed match ID flushed the previous record, and the timestamp embedded in the game’s own match ID became both the JSON timestamp and the compressed output filename. Replaying the same final match therefore selected the same name rather than inventing a second event.
A vendor log line was state. The scanner used redundant signals where the game emitted them, but a missing line could still leave the supervisor with an incomplete view.
The filesystem was the control plane
Desired state lived under a spec directory; observed state lived under stat. The timestamp-like controls used three file bodies as three verbs:
: > spec/restart # empty, no newline: reset
echo > spec/restart # newline only: now
printf '"2022-09-01T12:00:00Z"' \
> spec/restart # JSON time: laterRemoving a control also restored its baseline. That grammar is small enough to operate with shell redirection and exact enough to distinguish reset from act-now without another daemon or command socket. Spec files records the type-specific boundaries; strings, durations, booleans, and these timestamps did not all share one parser.
Multiple lobbies formed a ring with symlinks. Each instance’s spec/peer pointed at the previous configured instance’s stat directory, with the first pointing at the last. If my neighbor became full, a systemd path unit could start me cold; if I was already running, the in-process watcher reset my idle clock. If the peer became idle, the same files could drain me. There was no fleet controller deciding capacity. The peer ring combined one neighbor’s state, systemd activation, and local timeout policy.
The watcher followed symlinks but reported events under the configured alias, so a change in another lobby’s real stat directory still arrived as peer/full. It also batched events for 200 milliseconds and composed small filters: rewrite foo.lock as a Create of foo, preserve a lastfoo event as foo, and merge duplicate names by OR-ing their operations. The writer and watcher agreed on the publication convention instead of making every consumer understand staging files.
The edge-triggered design served a practical constraint: I needed to observe and resize the service while games were running without interrupting them. The lobby watcher ignored ordinary drain and stop reasons while a match ID was present. If players remained, it also discarded any non-forced reason to stop. Capacity could wind down after the match cleared and the last player left; ordinary scaling did not kick people out to make the server graph tidier.
A pidfile that changes CPU policy
The Linux service passed three cgroup: URLs through the ordinary --pidfile option. Setup resolved the cgroup2 mount, made main, busy, and idle children, wrote their cpu.max controls, then replaced each URL with its cgroup.procs path. From the lobby’s point of view those were pidfiles: write a PID to one. From the kernel’s point of view the write moved the process into a different CPU policy.
When the lobby became idle, snap-gs wrote the game PID to the idle group’s cgroup.procs; when activity returned, it wrote the same PID to busy. The deployed unit encoded a 1 ms busy quota and a 1 ms per 10 ms idle quota. The details are in A pidfile is any file that accepts a PID.
The restart loop stayed comparably literal. Known idle and admin timeouts restarted immediately. Unknown failures accumulated against MaxFails, while any run shorter than fifteen seconds counted as unhealthy and slept out the rest of that floor before retrying. Separate sentinel errors mapped onto systemd’s named exit statuses, so “restart” could be a return value understood by the service unit rather than another side channel.
Idle was the point
The servers ran on burst-capable AWS instances, which bank CPU credit while idle and spend it while busy. That made a sawtooth the shape I wanted: flat out while a match ran, back to idle as soon as the game allowed. Most of the timeouts in the supervisor existed to get a server back to idle as fast as possible.
At its widest the service ran in about half a dozen AWS regions and local zones, three or four of them in the United States. Most of the players were there, which made it hard to support the people in Europe and elsewhere. I kept servers up for them in their own regions, and eventually I had to add scheduled downtime there: not enough activity, or too much of it at the wrong times.
Rewriting the route inside the process
The sharpest intervention was a GNU/Linux LD_PRELOAD shim. The game announced an address pair inside a length-prefixed type-7 record. The interposer wrapped send(), looked for that exact byte sequence, and replaced it with a zero-padded accelerator address of identical length. Keeping the record length fixed avoided rewriting the surrounding frame.
The shim had several memory-safety defects. It mutated the caller’s nominally const buffer, could compare past the packet tail after finding only the marker’s first byte, used fixed arrays with runtime lengths, and did not prove the replacement fit before unsigned offset arithmetic. Rewriting a closed game’s address keeps the technique beside those defects.
The match records were published as they were made, anonymized, and I still have all of them. I never built a way to find or consume them, and I never analyzed them myself. There was nowhere to put the results, and I am not sure I could have; keeping them was preemptive.
I eventually retired Snap Game Services. The supervisor remains public; the interesting part of it was everything required to operate a program that did not know it was being operated.