Back to Explore
NewsVibeFix editorialUpdated Nov 20, 2026

GitHub Rebuilds Git's Foundation: One Repository Holding 5,000 Agents Writing Code at Once

GitHub announced Project Rubicon on November 5: a new Git storage architecture that held 5,000 concurrent agents and 7,200 pushes per second on a single repository in internal testing. Packs move to Azure Blob Storage, refs to a new Ref-CAS log service, and only reference updates need global ordering. It is still in limited private preview — the numbers are a direction, not a delivered feature.

GitHub's official architecture diagram: requests pass through Spokes to virtual replicas per repository; Ref-CAS holds refs and the pack manifest, Blob Storage holds pack data.

The short version: Git hit its bottleneck first

On November 5, 2026, GitHub announced a new Git storage architecture in its official blog, codenamed Project Rubicon. In internal testing, a single repository held up under 5,000 agents working at once, taking 7,200 pushes per second, while reads stayed at consistently low latency. GitHub set its own bar even higher: at least two orders of magnitude more scale for a single repository — a hundredfold, at minimum.

This blog post is not another model launch, nor a new Copilot feature. It is about the foundation: GitHub plans to replace Git's storage layer wholesale, for a very practical reason — the people writing the code have changed. A repository used to see a handful of humans pushing at the same time; now it can have thousands of agents writing into it at once. The old architecture was designed for the rhythm of people. The new one is designed for the rhythm of an agent fleet.

The timing matters too: the architecture is currently only in limited private preview, with a broader rollout coming early next year. In other words, every number in this post is an internal test result, not something you can use today. But the direction is now explicit: infrastructure is being rebuilt from the ground up for multi-agent parallel development.

Some background first: this is the third post in a series

This post does not stand alone. It is the third entry in GitHub's series on its next-generation Git storage architecture. GitHub announced the new architecture at GitHub Universe last week; this post is the technical deep dive that followed, explaining how it is actually put together.

The first post set out the two principles: minimize coordination, and decouple storage from compute. The second covered the bridge: moving 99.99999% of repositories into Azure, scoping the write quorum down to individual references, and standing up virtual replicas backed by Azure Blob Storage. In other words, the first two posts laid the foundation; this one starts building on it.

Where this post picks up matters: virtual replicas were only the first step. The rest of Rubicon rests on one observation — very little of a write actually needs to be ordered. Git objects are immutable and named by their contents, so two writers can store them at the same time without agreeing on anything. The only part that needs global ordering is the reference update. Since that is the only part that needs coordination, that is the only part that pays for it; everything else is left open.

For readers, the full picture takes all three posts. But this one alone carries the core: the two principles, virtual replicas, Ref-CAS, and the compaction fleet, with the division of labor spelled out. The sections below take them apart in that order.

The trigger: one repository's traffic quadrupled in four months

GitHub included a chart in the post: for one high-traffic repository, daily requests grew from an average of 44 million in March 2026 to 181 million in May — roughly 4x in three months — and kept climbing on an exponential trajectory after that.

Today, the busiest repositories see at most dozens of writes per second, and only in short bursts — humans work in the daytime and go quiet at night. Agents are different: they run in parallel, around the clock. Once short bursts turn into sustained, all-day load, the old architecture's write ceiling gets hit fast. And every push drags a trail of reads behind it: CI runs, deployment systems pull the new commit, other agents fetch it for context. More writes mean disproportionately more reads — traffic amplifies.

So GitHub did not go for another incremental fix. It set the target two orders of magnitude higher. That is an admission: agent-written code will sooner or later outrun what the old storage can keep up with, and it is better to replace the foundation now than to fight fires later.

What Project Rubicon actually changes

First, the reassuring part for every Git user: for developers, Git stays Git. Push, fetch, and branching work the same way, with the same protections in place. What changes is the data layer underneath.

The new architecture splits a repository's durable state in two: Git objects (as packs) live in Azure Blob Storage; references (refs — where branches and tags point) and the pack manifest live in a newly built replicated log service called Ref-CAS, short for Reference Compare-and-Swap. Object writes need no coordination at all — anyone can store them directly. Only reference updates need global ordering, and a reference update is a tiny part of a push.

GitHub's official architecture diagram: Git, API, and Web requests flow through Spokes to each repository's virtual replicas; Ref-CAS stores refs and the pack manifest, Azure Blob Storage stores pack data, and compaction nodes work independently in the background

Requests still arrive through Spokes — GitHub's existing routing and load-balancing layer, responsible for partitioning, isolation, and tenant fairness. Under Spokes sits a group of virtual replicas: a disposable fleet of compute that serves reads, writes, or both. A virtual replica's local disk is just a cache that can be rebuilt at any time. If a replica goes away, the others keep serving, and a replacement warms its cache as traffic arrives. When a repository gets busy, more replicas are added; when it quiets down, the fleet shrinks.

Meanwhile, a separate fleet of compaction nodes works in the background: merging packs and cleaning up objects left behind by deleted branches and force-pushes. The key point is that this work never runs on the serving replicas, so it never competes with live requests for CPU and memory. Reads and writes, storage and compute, are pulled apart. The post also notes that 99.99999% of repositories have already been migrated to Azure — this foundation swap is built on a migration that is already done.

Two design principles hold the whole post together

The post opens with two principles: minimize coordination, and decouple storage from compute. All of Rubicon grows out of those two sentences.

Minimize coordination means this: in a single push, the part that actually needs everyone to agree is very small. Git objects are immutable and named by their contents, so two writers can store them at the same time without agreeing on anything. The only thing that needs global ordering is the reference update — the moment a branch pointer moves from the old commit to the new one. If only that needs coordination, only that should pay the cost of coordination. Everything else is set free.

Decouple storage from compute means: data sits in Blob Storage — cheap, durable, effectively unbounded — while compute is a stateless fleet that can grow or shrink at will. Busy repository, add replicas; quiet repository, scale back. Compaction and garbage collection are a separate crew that never competes with live requests. GitHub states the goal bluntly: every repository, from one developer's side project to the busiest monorepo, runs on the same architecture, with no special handling for the busy ones — and every improvement reaches every repository at once.

The prototypes that failed first: try, then build your own

The most revealing section of the post is where GitHub candidly describes two prototype designs that did not work.

The first was a single-blob write-ahead log: all of a repository's reference changes recorded in one blob, each update a conditional write against it. Reads were cheap, but writes were strangled — only one conditional write to a blob can be in flight at a time, so throughput landed in the mid-hundreds per second. Hundreds per second is an order of magnitude short of a target measured in thousands.

The second split references across several segment blobs, with each reference hashing to one segment. Writes to different segments no longer waited on each other, so writes got faster — but reads got slower: a read now had to check every segment, and it was only as fast as the slowest one. And reads outnumber writes by orders of magnitude. In GitHub's words, they had traded write latency for read latency, and that trade loses.

With both paths blocked, GitHub built Ref-CAS itself: a replicated log service dedicated to references. Each reference update in Ref-CAS is a compare-and-swap committed by a quorum, locking only the references it touches; agents mostly push to their own branches, so most updates never contend and replicate in parallel. When two pushes do update the same reference, only one appends to the log. Each commit gets a log sequence number (LSN), and reads start with a version check: if nothing changed, an unchanged repository costs a single comparison; if the replica is behind, it gets the current references and version, plus pointers to the pack data it needs. For reference state alone, the post says, this transacts in the tens of thousands of operations per second.

What a single push looks like in the new architecture

Split a write and a read apart, and Rubicon's division of labor becomes clear.

On a write, a virtual replica indexes the incoming pack and uploads it to Blob Storage, then commits one transaction to Ref-CAS that lists the pack in the manifest and updates the references. Only then is the push durable. Note the order: the pack is stored and durable first, and only then does a reference point at it. A reference only ever points at objects that are already durable — never at data that is still on its way. Later, a compaction node merges that pack with others to keep reads fast.

On a read, a virtual replica checks its reference version with Ref-CAS. Unchanged? A single comparison and it keeps serving from its local cache. Behind? Ref-CAS hands back the current references and version, along with pointers to the pack data, and the replica pulls the missing packs from Blob Storage into its cache. Given the read-to-write ratio, the overwhelming majority of reads hit the "nothing changed" branch, at a cost close to zero.

One detail is easy to miss: Ref-CAS never holds or validates object data. It only stores pointers. Indexing and connectivity checks happen on the virtual replica before upload. The cleaner the split, the more independently each part scales: more reads or writes, add replicas; maintenance falling behind, add compaction nodes; more contention on references, let Ref-CAS scale on its own.

The compaction problem: packs only ever grow

Every push adds a pack to Blob Storage. Git can serve a repository from many packs, but each additional pack makes lookups slightly more expensive, so the pack count cannot grow without bound. Somebody has to keep merging them — that is the compaction nodes' job.

In the legacy architecture, compaction ran on the same replicas that serve traffic, and it had to run on every replica. The new architecture moves it out entirely: compaction nodes work directly against Blob Storage and Ref-CAS, claiming ownership over a subset of a repository's data via leases — each pack's hash maps to a shard, and only one node compacts a shard at a time. A node collects new packs into an open segment, seals it once it hits a limit on count, size, or age, and combines its packs into fewer, larger ones. Before anything is published, the node checks that the new packs contain exactly the same objects as the inputs — not one more, not one fewer. Only then are the new packs uploaded and added to the manifest; a second transaction removes the inputs, and cleanup deletes them later.

Packs that are still under the size limit after compaction go back into a shard by their own hash and merge again, growing in tiers until they reach the limit. The post includes a demo: as pushes climb, the pack count stays effectively flat. That flat line is the whole point of compaction nodes.

Garbage collection is a different job: it removes objects nothing references anymore, like the ones left behind by deleted branches and force-pushes. It works much like compaction, with dedicated workers handling the compute-heavy portion — again without touching the serving replicas. Compaction is tidying up; garbage collection is throwing things away. They run separately and scale separately.

Where exactly is it faster than the old architecture?

The post does not publish a head-to-head benchmark table, but it lays out the dimensions along which the new architecture scales: reads and writes by adding virtual replicas; reference ordering by letting Ref-CAS order each reference independently instead of locking the whole repository; maintenance by adding compaction nodes; and reads further ahead through optimizations like fetching a single object instead of a whole pack, organizing packs by type, and partial replicas.

For developers, the most tangible change may not be speed at all, but the end of worrying about big repositories. GitHub's goal is for every repository to run on the same architecture, with no special handling for the busy ones. A side project that suddenly takes off and picks up a fleet of agents overnight needs no migration and no manual intervention from GitHub — the architecture keeps up on its own. It sounds like a slogan, but it is the landing point of the entire design: scaling is the architecture's job, not the on-call team's.

All of this, of course, rests on internal test data. 5,000 agents, 7,200 pushes per second, reads holding low latency — all from GitHub's own testing. The post does not disclose the test conditions: how large the repository was, how large the pushes were, or what exactly the agents were doing. Without those details, the numbers show that the design target has been demonstrated, not that your repository will behave this way tomorrow.

What comes next: working without a clone

The end of the post lists three areas worth watching.

First, making each agent operation cheaper: work without a clone — many agent tasks never need a full working tree, just a handful of files and enough context to make a change, read straight from the server; an atomic commit API that lands a multi-file change in one operation, committing the whole change or rejecting it cleanly on conflict; and event streams so agents and the systems around them can subscribe to changes and react as they happen.

Second, keep improving Git itself: lighter-weight clones, removing hidden scaling costs where Git slows down much faster than the repository grows, faster history walks, and catching the common mistakes agents make before they cause damage.

Third, the architecture itself can go further: fetching only the object that is needed, organizing packs by type, partial replicas, materialized views, global replicas, and scaling to zero.

In one sentence: GitHub is preparing for a world where thousands of agents work in a single repository — and the preparation has only just begun.

VibeFix's take

This blog post is worth reading not because it hands you a feature you can use today, but because it does the math out loud: in the agent era, the first bottleneck to break is not the model, it is infrastructure. A single repository holding 5,000 agents and 7,200 pushes per second is an internal test result; but "only order reference updates, and set everything else free" is a genuine engineering judgment.

For people doing vibe coding, the real signal is in the last section: working without a full clone, an atomic commit API, event streams. If those land next year, the way agents work will change again — from "clone the whole repository, then work" to "take only the few things you need, do the job, and leave." When that happens, this post about storage will turn out to have been the starting point.

One more angle is worth a separate mention: GitHub did not build a parallel system for this. The post opens by saying the new architecture was built underneath GitHub while it keeps running — 99.99999% of repositories moved into Azure first, then virtual replicas and Ref-CAS were connected layer by layer. For ordinary users, that means there is no "migration day" with downtime; repositories move over one by one, and the swap finishes while you keep working. That kind of gradual replacement is far steadier than a big-bang cutover, and it also explains why the post spends so much space on failed prototypes: at every step, a dead end could be walked back from without affecting production.

Source: GitHub's official blog, "Inside GitHub's new Git storage architecture," November 5, 2026. The architecture is in limited private preview; figures are subject to GitHub's later announcements.

Views 17Comments 0

Comments (0)

ME
0/1000
Loading comments...

Sources

Browse projectsPublish your project

Related articles

A developer with glasses types on a laptop at a wooden desk, an external monitor to the left showing C code on a dark background, a small plant on the desk
News
Microsoft's Coding Model Stops Playing Second Fiddle: MAI Code 2 Scores 96% of Opus 5.5 for 5% of the Cost — Read the Three Caveats First

On 17 November 2026, Microsoft launched MAI Code 2 and Code 2 Flash: the larger model has 77 billion active parameters and a one-million-token context, averaging 63.2% across four benchmarks — close to GPT-6.1 Sol and Claude Opus 5.5 — at $0.309 per task. Read the three caveats first: promotional pricing, the lowest score of the three, and no long-horizon work.

Developer WorkflowProduct LaunchModel Updates
A developer types at a dual-monitor workstation; the left screen shows a dark code editor, the right a testing dashboard with green checkmarks and bar charts
News
Did Your Agent Get Better or Worse? AWS Open-Sources UAEF to Measure Agents from Every Framework with One Ruler

On 11 November 2026, AWS open-sourced UAEF: adapters turn traces from LangGraph, LangChain, Strands and Bedrock AgentCore into one format, then 36 metrics across 7 dimensions score them, flagging regressions past 5%. The judge model is hardened too — scores must cite verbatim evidence, and never decide safety.

Developer WorkflowProduct LaunchTesting & Quality
Against a black background, a conductor seen from behind raises a baton, with swirling trails of light circling beneath him like an orchestra being conducted
News
One API for 45 Models, and It Switches Providers When One Fails: Perplexity's Orchestrator Wants to Be the Switchboard

On 17 November 2026, Perplexity launched the Orchestrator API: one interface reaching OpenAI, Anthropic, Google, xAI and open-weight models such as GLM, Kimi and DeepSeek, billed at the answering model's published rate with no platform fee. When one provider fails, the same model retries on another deployment, and preview routing assigns requests to models by task.

Developer WorkflowProduct LaunchModel Updates