2 posts tagged with "self-healing"

Here is the uncomfortable truth at the heart of any platform that runs other people's code on machines it doesn't fully own: the moment after you deploy something, it starts to be wrong. A container OOMs. A host reboots. A VM's clock skews. Someone pushes a new image. The network blips for ninety seconds and a healthcheck misses. None of this is exotic — it's Tuesday. The interesting question isn't whether reality will drift from what you asked for. It's what your system does in the gap.

Fibe's answer is the oldest good idea in operations, applied uniformly: you declare what you want, and a control loop drives the world back toward it. You say "this Playground should be running." Somewhere a reconciler wakes up every sixty seconds, looks at the actual Docker truth on your Marquee, notices it disagrees, and does exactly one thing to close the gap. Then it goes back to sleep. The state you typed is the source of truth; everything the containers are actually doing is just evidence.

Kubernetes has a luxury Fibe doesn't: it owns the nodes. The kubelet lives on the box, watches the local container runtime, and the control plane can assume the daemon is there and answering. We can't assume any of that. A Player's Marquee might be a Scaleway VM we provisioned, or it might be a laptop under someone's desk reachable only over SSH. The Docker daemon can vanish mid-reconcile. The network can swallow a docker compose up halfway through. And the one rule we are never allowed to break is the one that matters most to the person using it: don't throw away my work.

Playguard is our answer. It's a control loop — the same shape as a Kubernetes reconciler — that runs once a minute per Marquee, compares declared state against real Docker state over an SSH link we don't trust, and nudges reality back toward what you asked for. The interesting parts aren't the happy paths. They're all the places where it decides to do nothing, on purpose.