No Raft, No Control Plane: Celld's Durable Objects Rest on Conditional S3 Writes
Cloudflare's Durable Objects are hard to argue with once you have used them. A named object with its own SQLite database, exactly one instance running anywhere, no lock service, no leader election. Then you want the same thing on your own hardware, and the obvious designs all drag in a consensus cluster. Raft needs a leader, and a leader in a wide-area network is a latency tax on every write plus a temporary single point of failure every time it dies.
Deno's celld, an Apache-2.0 daemon that picked up 4,600 stars and a front-page HN thread in August, takes a different route. It runs a Cloudflare Workers application (Workers, Durable Objects, KV, Queues, D1, R2, Workflows, cron, static assets) on machines you own, with a bucket you already have as the only coordination point. No consensus service, no failure detector, no membership protocol. The architecture rests on one deceptively small primitive: a conditional write to object storage.
The bucket is the lock service
A celld fleet is a set of nodes pointed at the same S3-compatible bucket. Each Durable Object is a "cell," and each cell gets one ownership record in the bucket naming the owning node's session and carrying a fencing epoch. To acquire a cell, a node makes a conditional write: create-if-absent when no record exists, compare-and-swap against the previous record otherwise. The bucket accepts exactly one such write, so two nodes cannot own one cell at the same time. That's it. The mutual exclusion that Raft, etcd, or ZooKeeper exist to provide is reduced to one HTTP request with an If-None-Match or If-Match header.
This is the idea worth stealing even if you never deploy celld. If your storage layer already gives you atomic compare-and-swap, a lease record in that storage can replace an entire coordination cluster for single-writer-per-shard workloads. A lock service is only mandatory when your primary store cannot express "write this only if nobody changed it since I read it." S3 gained conditional-write headers in 2024, R2 and Google Cloud Storage have equivalents, and the strongest argument for running etcd in every home lab quietly evaporates for this class of problem.
Fencing, and why the epoch lives in the key
Mutual exclusion at acquisition time is not enough, because owners go bad. A node can be paused for forty seconds by a garbage collector or a VM snapshot, lose its lease, and wake up still believing it owns the cell. Celld handles this the way good fencing designs do: it doesn't trust the stale node to notice. Every activation advances the fencing epoch, and the node replicates its SQLite writes as LTX transaction data under cells/<cell>/ltx/e<epoch>/. A zombie owner can hammer the bucket all it wants. Its writes land in a superseded prefix that restore ignores.
Each node also holds a lease with an expiry, renewed after one third of its lifetime, and a node that cannot reach the bucket fences itself: it stops its cells, fails its in-flight requests, logs a SELF-FENCE: line, and exits with code 3. The docs are blunt about the consequence: run celld under a supervisor that restarts it without an attempt limit and waits at least one lease lifetime between attempts. A fenced process that stays down just costs the fleet capacity, which beats split-brain but is still an operational obligation.
The acknowledgement rule nobody else enforces
The second promise is the interesting one: celld refuses to acknowledge a write until that write provably survives a crash. A gate holds every response, including error responses and streamed chunks, until a durability proof covers it. Then, and this is the detail that separates it from most durable claims, the owner re-reads the ownership record and acknowledges only if the bucket still names it at its epoch. A partitioned node can commit locally and replicate into its own superseded prefix, but the re-read shows the new owner, so the write never gets a success response. Clients cannot act on a value that a crash can lose. The check compares a record, not a clock, so skewed clocks and paused processes cannot smuggle a false success through.
There is a latency ladder underneath this. The default posture asks one or two other fleet nodes to fsync the write, which is fast, but that needs a fleet of at least two. A lone node falls back to proving durability by uploading to the bucket, and an object-store round trip is far slower than a neighbor's fsync. Two nodes is the real minimum for production; one node is a correctness-preserving demo.
The fine print is the story
Here is where the architecture gets honest, and where the post-mortem of every failed self-hosted celld will actually live. The scheme requires the bucket to implement conditional writes correctly: create-rejects-if-exists, overwrite-fails-if-changed, read-after-write consistency, and ranged reads that return the requested bytes. No provider publishes these properties, so celld asks the store directly. Each node runs four probe writes and a ranged read before serving anything, and there is a celld diagnose command for operators. Two of the four probes must fail. A store that accepts them all cannot fence a cell.
The qualified list is Amazon S3, Cloudflare R2, Tigris, Google Cloud Storage, and Azure Blob. Backblaze B2, Hetzner Object Storage, and DigitalOcean Spaces lack the required conditional writes, and the docs say it plainly: celld is not correct on such a store. Worse, a store can accept the conditional headers and silently ignore the condition, exactly the failure celld diagnose exists to catch before two owners corrupt each other's data. Even MinIO has a named broken release that answers a conditional create with the wrong error.
Other limits follow the same pattern. A cell's V8 heap is capped at 128 MB, matching Cloudflare's own limit. Internal peer ports must sit on a private network, and celld rejects a public advertise address unless you pass a flag with --unsafe- in front of it. Anything needing GPUs or Cloudflare's network stays on Cloudflare. A takeover after a large node dies can leave a cell waiting on recovery for minutes, retrying with backoff.
What to take from it
The durable-objects shape, one writer per named entity with state attached, is genuinely useful for chat sessions, game rooms, agent state, and per-tenant workers. Celld shows the shape does not require proprietary infrastructure or a consensus library. A lease record, a fencing epoch in the object key, a durability gate before acknowledgement, and a storage probe at startup carry an astonishing amount of correctness.
I keep turning over the storage-probe habit for systems that have nothing to do with celld. Much of distributed engineering assumes the substrate does what its documentation implies. Celld's position is that documentation means nothing when a wrong answer means two writers, so you test the substrate on every boot. Cheap paranoia, well argued.
Comments