Reachability and the two healthchecks: ignore the blip, redeploy the outage
· 9 min read
Every monitoring system hits the same fork: a check just failed — act, or wait? Act too eagerly and you turn a 400ms hiccup into a full redeploy, with a user staring at a "deploying…" spinner for nothing. Wait too long and a genuinely dead environment sits there, unreachable, getting the benefit of the doubt.
On Fibe, the watched things are AgentChats — the "Genie" coding agents that run as Docker sidecars on a Player's Marquee. They're long-lived, they sit behind Traefik with wildcard TLS, and people are actively typing into them, so a false redeploy is a visible interruption. We don't have one healthcheck; we have two, and most of the effort went not into detecting failure but into deciding which failures are worth reacting to.