18 posts tagged with "reliability"

There are two stories about what your Playground is doing. One lives in Postgres — a status that says running, error, or stopped. The other lives on a Docker host we don't own, reachable only over SSH, and you read it by running docker compose ps. Most of the time they agree; the interesting engineering is all about what happens when they don't.

This is a post about one reconciler and the small set of rules it uses to decide which story is true. The headline rule is easy to say and weirdly hard to honor: trust the daemon as the source of truth — but only when you can actually see it. A Playground on a host that clearly threw its containers away should be marked stopped; one on a host that simply didn't answer our SSH knock should be left exactly as it is. Telling those two apart is the whole job.

Every platform that rents compute eventually asks the same question in a hundred different places: is this allowed to run right now? On Fibe, "right now" means funded — a Marquee that has paid for the current service-day. We answer that question with exactly one check and one HTTP status, and we route the consequences differently depending on who's asking.

This is the story of the not-funded gate — a single 402 that sits in front of roughly a hundred call sites.

Here is the support ticket we never want to receive: "I stopped my Genie to save money over the weekend, and when I came back, everything it had learned about my codebase was gone."

For an AI coding agent, that's the worst possible failure. The agent's value compounds over time — its working state, its history, the files it wrote, the config it tuned — and all of it lives on disk, in Docker volumes, not in the container. So we drew a hard line early: stopping a Genie must never touch its data. Only one explicit, confirmed action deletes it. This post is about that contract and the small amount of code that makes "stop" and "delete" genuinely different.

Every billing system has a moment it would rather not talk about: the one where the money stops. The card expires, the wallet empties, the trial ends — and now you have an environment that costs real money to keep alive and a customer no longer paying for it. The naive answer is brutal and tempting: tear it down, move on.

We didn't want to build that. Fibe funds a Marquee — your Docker host — out of a daily wallet, and when a day can't be funded, the system enters a deliberate, recoverable sequence rather than a guillotine. This post is about that sequence: the grace incident, the three-day window, the seven-day retention, and the invariant we cared about most — starting grace never marks your billing as paid, and a suspended-but-unrepaid environment can never slip back to "active."

Here is a bug that does not look like a bug. You push a commit, a build kicks off, it goes green, your environment comes up. Everything works. Except the image inside it was built from a different commit than the one you pushed — a stale checkout, a cached layer, a race between two builds for the same branch. Nothing errored. The dashboard says success. And you spend an afternoon debugging code that was never actually deployed.

That class of failure — a build confidently wrong about what it produced — is the one we decided never to ship to a Player. So Fibe's image builds follow one stubborn rule: a build must be deterministic and honest about which commit it produced. This post is about how each build record enforces that, and the two checks that will fail a build rather than let it lie.

Every 60 seconds, Fibe's reconciler — we call it Playguard — wakes up and walks each Marquee, asking the same question: is reality what the database says it should be? It pulls git branches, refreshes credentials, pulls images, detects drift on Playgrounds, recovers stuck ones, enforces expirations, sweeps orphaned containers. All of that happens over SSH, against a real Docker daemon, on a host we don't own.

Here's the nightmare we work to avoid: two of those runs touching the same Marquee at the same time. One job tearing down a container while another rebuilds it. Two docker compose up calls fighting over the same project. A drift "fix" applied on top of a half-finished "fix." Over SSH, against a stateful daemon, concurrency isn't a performance problem — it's a correctness problem. So Playguard runs each Marquee under a lock. The interesting part isn't that we take a lock; it's that we keep proving we still hold it.

Every Playground on Fibe is born with an expiration date. Spin one up, walk away, and eight hours later it's gone — containers down, volumes reclaimed, the database row deleted. That's not a quota we enforce grudgingly; it's a default we're proud of. Idle environments are a tax everyone pays: they burn compute on a Marquee someone funds daily, pile up as ghost subdomains, and make "what's actually running here?" unanswerable. So we let them die.

But there's exactly one thing we will not do, no matter how expired an environment is: throw away your uncommitted work. If a Playground has dirty files when its clock runs out, the expiry sweep parks it rather than destroying it. This post is about that one rule — and how little code it takes to hold ephemerality and data safety at once.

Every coding agent on Fibe — a "Genie" — runs as a Docker Compose project on someone's Marquee. When a Player clicks "start," we want one thing to be unambiguously true: at most one live Genie chat per agent on that host. Not "usually one." Not "one, unless two browser tabs fire at the same millisecond." One.

The interesting part isn't the rule. It's that we enforce it in two completely independent places — the application layer reuses the existing chat, and the database has a unique index that physically cannot hold two — and that this redundancy is the point, not an oversight. Here's why, and what leaning on a real invariant buys you downstream.