The snap-gs lobbies scaled up and down with no scheduler, no controller, no leader election. Each instance simply watched its neighbor’s status. With two or more configured instances, the wiring is one symlink per instance, laid down at install time:
ln -s -f -T ../../$j/stat /opt/snap-gs/SnapshotVR/$i/spec/peerInstance i’s spec/peer points at instance j’s stat directory, where j is the previous configured instance—and the first configured instance points at the last one, so the links close into a ring. A one-instance configuration removes the peer link instead. The linked status reaches two consumers: a systemd path unit can start a stopped lobby, and a running lobby’s watcher sees whether its predecessor is up, full, or idle.
One lobby had a reason to return
The ring did not elect a survivor after every lobby went idle. Installation gave that role to production lobby 1. The install.sh script created its persistent flag/up file and removed that flag from the other instances. The path unit watched three reasons to start a lobby: its own persistent flag, an explicit spec/up, or its predecessor’s full state.
Startup still had a brake. The service unit refused a requested start when the predecessor was already up but not full, or when the predecessor was idle. The failing ExecCondition slept for fifteen seconds first. That was not consensus; it was enough time for a draining predecessor to exit and remove its live status before systemd tried the persistently requested first lobby again.
Those three neighbor states drive three decisions, each a single case in the spec logic:
// stop once the peer has come up (scale down)
case !s.PeerUp.IsZero():
return true
// go down when the peer goes idle (drain)
case !s.PeerIdle.IsZero():
return trueThe third state has two paths. This line in the matching systemd .path unit starts a stopped lobby when its predecessor becomes full through the symlink:
PathExists=/opt/snap-gs/%j/%i/spec/peer/fullIf the lobby is already running, the same state resets its idle clock:
if !l.spec.Up.IsZero() || !l.spec.PeerFull.IsZero() {
lastup = now.UTC() // my neighbor being full resets my idle clock
}Put together, that is a capacity ladder that manages itself. When a lobby fills, the path unit starts its stopped neighbor; if that neighbor is already running, the watcher stops its idle countdown and keeps it warm. Scale-up pressure propagates one hop around the ring. When load drops and a lobby idles, the neighbor watching it drains and goes down—scale-down propagates the same way. A fresh lobby coming up tells its successor it may now stop. After the idle wave clears, lobby 1’s flag/up gives systemd a standing reason to bring capacity back. No component knows the global state; each knows exactly one neighbor, while installation fixes the base case.
It is fragile in exactly the way you would expect: if the symlink topology drifts—a renumbered instance, a broken link—the ring breaks and the signal stops propagating across the gap. The topology is the coordinator, so the topology has to be right. Within that constraint it is remarkably little code for elastic capacity, and it kept three regions of game servers sized to demand until the service was retired.