Skip to main content

One post tagged with "redis"

View All Tags

Coordinating reconcilers with Redis token locks

· 10 min read

Every 60 seconds, Fibe's reconciler — we call it Playguard — wakes up and walks each Marquee, asking the same question: is reality what the database says it should be? It pulls git branches, refreshes credentials, pulls images, detects drift on Playgrounds, recovers stuck ones, enforces expirations, sweeps orphaned containers. All of that happens over SSH, against a real Docker daemon, on a host we don't own.

Here's the nightmare we work to avoid: two of those runs touching the same Marquee at the same time. One job tearing down a container while another rebuilds it. Two docker compose up calls fighting over the same project. A drift "fix" applied on top of a half-finished "fix." Over SSH, against a stateful daemon, concurrency isn't a performance problem — it's a correctness problem. So Playguard runs each Marquee under a lock. The interesting part isn't that we take a lock; it's that we keep proving we still hold it.