GitHub Rubicon: 7,200 Pushes per Second From One Repo
Git infrastructure was designed around a human-shaped workload. One developer, one working copy, a handful of pushes per minute, bursts around lunch and never at 3am. The busiest repositories on GitHub.com today get at most dozens of writes in a second, in short bursts, and the storage layer underneath was built to survive exactly that. Agents do not burst. They push around the clock, in parallel, by the fleet.
At GitHub Universe last week, the company announced what it built for that world, and the November 5 engineering post goes into the design. Internal testing supported 5,000 concurrent agents on a single repository sending 7,200 pushes per second, all while read latency stayed low. The project is called Rubicon, named after the point of no return the industry crossed with agentic development. If you only read one systems post this month, make it this one, because the workload driving it is coming for everyone's infrastructure, not just GitHub's.
The workload that broke the old assumptions
GitHub tracks platform scale on two axes. Horizontal scale, how many tenants fit across the fleet, is solved and carried forward. Vertical scale, how much traffic one repository can absorb, is where the requirements came from. One repo at the top of the curve grew from 44 million requests per day in March 2026 to 181 million by May, a 4x jump, and the post says growth has continued on an exponential trajectory since.
The amplification is the scary part. Every push triggers reads: CI runs, deployment systems pulling the new commit, and other agents fetching the change for context. Writes multiply into read traffic. So instead of planning for the next increment, GitHub set its target two orders of magnitude above today's busiest repo: thousands of writes per second on a single repository, multiple writer nodes because pack indexing and connectivity checks are the CPU-expensive part of a write, minimal commit latency since an agent blocks on every push, and cheap reads when nothing has changed, which is the common case.
The observation the whole design hangs on
Git objects are immutable and named by their content. Two writers can store objects at the same time without agreeing on anything. The only part of a push that needs consensus is the reference update, and a reference update is tiny: one hash swapping for another. Every decision in Rubicon follows from scoping coordination to that small, hot sliver of state.
Two prototypes that failed, and why they're the best part
The first attempt stored a write-ahead log for all of a repository's references in a single blob. Object storage gives you compare-and-swap for free through conditional writes, and the data is already durable and replicated. Reads were cheap: ask whether the blob changed, and if not, you're done. Writes were the problem. Only one conditional write to a blob can be in flight at a time, so even with aggressive batching, the repository's write ceiling was one write per round trip. That netted mid-hundreds of writes per second, nowhere near thousands.
The second attempt split references across several segment blobs by hash, so writes to different segments stopped waiting on each other. The ceiling went up, and then reads got interesting: a reader now had to check every segment to know whether anything had changed, which made read latency equal to the slowest segment. Reads outnumber writes by orders of magnitude, so GitHub had traded the wrong latency.
Both failures are worth bookmarking for your own designs. A conditional write on object storage is a CAS primitive, but a single hot key turns it into a global lock. And when your read-to-write ratio is lopsided, the prototype that raises write throughput by adding read fan-out has already lost.
Ref-CAS: a consensus group that only points
The answer is a replicated log service dedicated to references, called Ref-CAS. Each reference update is a compare-and-swap committed by a quorum: the transaction locks only the references it touches, verifies each still holds the value the writer expects, and becomes visible once a majority of the log's replicas record it. Most agents push to their own branches, so most updates never contend, and non-contending updates replicate in parallel. When two pushes hit the same reference, only one appends.
Every commit gets a log sequence number, and the repository's version is the latest LSN. Reads start with a version check: a virtual replica sends the version it holds, and if that version is still current, the reply carries no references at all. An unchanged repository costs a single comparison, which is how the system keeps the common case cheap. A replica that's behind gets the current references, the version, and pointers to pack data.
Crucially, Ref-CAS never holds or validates object data, only pointers to it. A replica uploads a push's pack to blob storage before committing the reference transaction, so a reference only ever points at objects that are already durable. That small boundary does a lot of work: the consensus group stays tiny and fast (tens of thousands of operations per second in testing) while all the heavy bytes live in cheap object storage.
Disposable compute, rebuildable cache
Spokes, GitHub's routing and placement layer, survives unchanged: partitioning, tenant isolation, fairness, load-aware routing. Underneath it sits a fleet of virtual replicas, disposable compute that serves reads, writes, or both. A replica's local disk is a cache that can be rebuilt at any time. Spokes prefers to route a repository's traffic to the same replicas to keep caches warm, but nothing about correctness depends on a request landing anywhere in particular. When a replica dies, its peers keep serving and a replacement warms up as traffic arrives. Busy repository? Add replicas.
Maintenance leaves the serving path
Every push adds a pack, and each extra pack makes lookups slightly more expensive, so the pack count can't grow without bound. In the legacy architecture, compaction ran on the same replicas that served traffic, and it had to run on every replica. Rubicon moves it out entirely. Compaction nodes claim a lease on a shard of a repository's packs, collect new packs into an open segment, seal the segment when it hits a limit on count, size, or age, and merge the packs into fewer, larger ones. Before publishing, the node verifies the new packs contain exactly the same objects as the inputs, then commits two transactions to the manifest: add the outputs, later remove the inputs. Cleanup deletes the old packs. Pack hashes map to shards, so adding shards and nodes adds throughput, and packs merge in tiers, which keeps the pack count effectively flat while pushes climb. Garbage collection follows the same lease-based, background shape.
What to steal for your own systems
You probably don't run Git infrastructure, but the pattern transfers to any stateful service moving onto object storage. Scope coordination to actual contention: agent fleets push to their own branches, and per-reference locking exploits that while a whole-repository quorum would not. Treat object storage as a cold tier, not a database with a CAS button. Make the no-change read path cost one comparison. Keep the consensus group holding pointers, never payloads. Run compaction and GC on leased background workers so serving compute never pays for maintenance. And size your target from the workload your agents are becoming, not the workload you have today.
Still on the roadmap
The post lists fast-follows: fetching a single object instead of a whole pack, grouping packs by object type, partial replicas, materialized views like prepared archives and a near-live shallow copy of the default branch, global replicas within bounded staleness, and scale-to-zero for inactive repositories. On the agent side: reading files without a clone, an atomic multi-file commit API, and event streams. The Rubicon architecture is in limited private preview now with broader rollout promised for early next year.
Two caveats before you quote the numbers. The 5,000-agent, 7,200-push-per-second figures come from GitHub's internal testing, with no published p99 breakdown and no third-party verification. And the design leans on Azure Blob Storage specifics, so porting the shape to S3 or R2 means re-checking conditional-write behavior yourself. None of that changes the main point: the constraint moved from how fast humans type to how fast agents can agree, and Rubicon is the first serious blueprint for a storage layer built for the new answer.
Comments