18 posts tagged with "reliability"

Here is the uncomfortable truth at the heart of any platform that runs other people's code on machines it doesn't fully own: the moment after you deploy something, it starts to be wrong. A container OOMs. A host reboots. A VM's clock skews. Someone pushes a new image. The network blips for ninety seconds and a healthcheck misses. None of this is exotic — it's Tuesday. The interesting question isn't whether reality will drift from what you asked for. It's what your system does in the gap.

Fibe's answer is the oldest good idea in operations, applied uniformly: you declare what you want, and a control loop drives the world back toward it. You say "this Playground should be running." Somewhere a reconciler wakes up every sixty seconds, looks at the actual Docker truth on your Marquee, notices it disagrees, and does exactly one thing to close the gap. Then it goes back to sleep. The state you typed is the source of truth; everything the containers are actually doing is just evidence.

Fibe runs your code on a Marquee — a Docker host that might be a Scaleway VM in Paris or your own laptop behind a hotel Wi-Fi captive portal. Between your fibe command and that host sits an HTTP API, a reconciler, a websocket, Traefik, and the open internet. Every one of those hops can hiccup. So when you type fibe playgrounds create and the connection dies two seconds in, the only honest answer to "did it work?" is: We don't know, and neither do you — unless we agreed on the rules beforehand.

This post is about those rules. We call it the reliability contract: a small set of primitives — typed errors, idempotency keys, 202 polling, classified retries, a circuit breaker, and a browser layer that survives reconnects — that the Go SDK, the CLI, the MCP server, and the web UI all speak the same way. The goal is simple and a little stubborn: you should never be left staring at a half-broken state wondering what to do next.

Here's the failure we want to save you from. You ship a change, your service restarts, and for the eight seconds it takes your app to boot, every request gets a connection refused or a 502. On a dev Playground nobody notices. On something people actually use during the day, those eight seconds are a pile of errors and a Slack message asking if the site is down.

The fix is one label: fibe.gg/zerodowntime: "true". It flips a service from "stop the old container, start a new one" to a real rolling update — new replicas come up alongside the old ones, and traffic only shifts once the new ones pass a healthcheck. But the label is the easy part. The whole thing lives or dies on whether that healthcheck means anything. A healthcheck that returns 200 the instant the process boots gives you zero downtime and a wall of errors, because you cut traffic over to a replica that isn't actually ready. This guide is about getting both halves right.

Most of our nastiest bugs were never typos. They were a state machine that looked right in the PR, passed its tests, and then met a customer doing something perfectly reasonable that nobody had drawn on the whiteboard. A Playground stuck in an error state forever. A chat that vanished because a healthcheck flapped. A runtime launched on a Marquee that hadn't been funded.

So we started writing down the hard lifecycles in Alloy, a relational modeling language, and letting a solver do the part humans are worst at: trying every small, weird combination of states and asking "does your invariant actually hold here?" Often it doesn't — and the solver tells us exactly which world breaks it.

Kubernetes has a luxury Fibe doesn't: it owns the nodes. The kubelet lives on the box, watches the local container runtime, and the control plane can assume the daemon is there and answering. We can't assume any of that. A Player's Marquee might be a Scaleway VM we provisioned, or it might be a laptop under someone's desk reachable only over SSH. The Docker daemon can vanish mid-reconcile. The network can swallow a docker compose up halfway through. And the one rule we are never allowed to break is the one that matters most to the person using it: don't throw away my work.

Playguard is our answer. It's a control loop — the same shape as a Kubernetes reconciler — that runs once a minute per Marquee, compares declared state against real Docker state over an SSH link we don't trust, and nudges reality back toward what you asked for. The interesting parts aren't the happy paths. They're all the places where it decides to do nothing, on purpose.

Provisioning a Playground touches more moving parts than we'd like to admit. We SSH into a Marquee, clone source from a Prop, merge registry credentials, build an image, generate a Compose file, and bring it up behind Traefik. Each step can fail, and for wildly different reasons: a flaky network, a typo in a branch name, a server that ran out of disk, a Docker daemon that blinked.

For a long time, "what do we do when step N fails?" was answered by a human reading a Sidekiq backtrace. That doesn't scale, and worse, it's inconsistent — the same failure handled two different ways on two different days. So we turned operational judgment into data. Every failure gets a category, and the category — not a person, not an ad-hoc rescue clause — decides what's next.

Most of the time, the dangerous question in a distributed system isn't "what does this delete cascade to?" It's "what does it not cascade to — and are you sure?"

We spend a lot of energy reasoning about the edges that do fire: delete a Marquee, and a chain of consequences unfolds. But the bugs that wake you at 3am are usually the opposite shape. Someone toggles an innocuous setting — disables a Prop, edits a Secret, demotes a teammate — and a long-running environment they didn't mean to touch quietly falls over. The edge that surprised them is one nobody designed; it emerged because some well-meaning "keep things in sync on every save" hook reached one relationship too far.

There's a moment every billing engineer eventually has. A customer emails: "I was charged twice." You open the logs, and the request looks fine — one checkout, one order. Then you find the second event. Same payment, redelivered by the provider three minutes later because our 200 got lost on the wire. We processed it twice, minted the currency twice. The math was, briefly, generous to exactly the wrong person.

The lesson isn't "be more careful" — careful doesn't scale. It's that at-least-once delivery is the only honest contract a distributed system can offer, and once you accept that, exactly-once effects become something you build, on purpose, at every layer that touches money. Here's how we do it at Fibe — across wallet posts, the Mana-to-Sparks conversion, daily funding, provider webhooks, and the keys our SDK ships.

Every monitoring system hits the same fork: a check just failed — act, or wait? Act too eagerly and you turn a 400ms hiccup into a full redeploy, with a user staring at a "deploying…" spinner for nothing. Wait too long and a genuinely dead environment sits there, unreachable, getting the benefit of the doubt.

On Fibe, the watched things are AgentChats — the "Genie" coding agents that run as Docker sidecars on a Player's Marquee. They're long-lived, they sit behind Traefik with wildcard TLS, and people are actively typing into them, so a false redeploy is a visible interruption. We don't have one healthcheck; we have two, and most of the effort went not into detecting failure but into deciding which failures are worth reacting to.

Every Fibe environment costs money to keep alive. A Marquee is a real Docker host with a real VM bill behind it, and the only thing standing between "your agent is coding" and "nothing runs" is whether the wallet had enough in it this morning. That makes the wallet the most boring-looking and least forgiving piece of the platform. It is not allowed to lose a credit, double-charge a day, or — the cardinal sin of any money system — let a balance drift negative.

This is the story of how we built that wallet: two currencies, a double-entry ledger where every movement is mirrored exactly once, pessimistic locks, and idempotency keys so aggressive that the same operation can be retried a hundred times and still only happen once. None of it is clever. That is the point. In billing, clever is a liability; the goal is a system whose invariants are so tight that the failure modes simply do not exist.