要点
- Pando now powers all development in OpenAI’s monorepo; median push latency cut nearly 5× (~80%); nearly a petabyte of Git data per day.
- Design: horizontally scalable writes to many branches; reads reuse packfiles instead of minting a unique pack per client.
- Consistency model: causal consistency / per-ref serializability; a reader of several branches can see them at different times.
- 57 TB of packs in a single hour with >99% from local disk cache; commodity servers for negotiation, Express for regional pack caches.
- Built with Codex in about 10 weeks; packfile-uri and related Git client fixes contributed upstream / public OpenAI Git build.
OpenAI’s engineering post on Scaling Git at OpenAI is not a soft “we love Git” note. It is a systems claim about what agentic coding does to source control. Codex raised branch and push volume until the old monorepo path saw slow fetches, failed merges, and reliability at times below 90%. The lab’s answer was Pando, a new Git backend that kept the Git protocol and client, and changed what consistency and pack generation mean under load.
[1]What shipped
Pando aims at two goals: horizontally scalable writes optimized for concurrent pushes to many branches, and read throughput high enough to saturate client network links. It now powers all development in OpenAI’s monorepo. In production, median push latency fell by nearly 5× (about 80%). The system delivers nearly a petabyte of Git data per day. In one hour it served more than 57 TB of packs, with more than 99% of those bytes from local disk cache.
The architectural trade is explicit. Conventional distributed Git patterns the post models (three-phase commit across full replicas, or a shared write-ahead log / manifest in blob storage) still force unrelated branch pushes to contend on a repository-wide coordination point. Pando records each branch’s updates independently in a metadata store and stores incoming packs as immutable blobs. Fetching a ref replays the pack history that produced that tip. Writers only conflict when they push the same ref at once. A client reading several branches cannot assume those reads describe one repository-wide instant.
On the read path, Pando reuses packfiles instead of generating a unique pack for each client’s have/want set. Clients may download some objects they already hold. The server can serve the same bytes to many readers with little computation. Push negotiation advertises a small ref set (by default a designated trunk) so the packs clients produce depend only on what the server allowed them to omit, which makes those packs easier to reuse on later fetches. Background compaction merges L0 packs into higher-level covers (up to L4 in the monorepo), acting like an LSM skip list so fetch planning stays logarithmic in how far behind a client is.
OpenAI says a small team, with Codex, took Pando from kick-off to production in about 10 weeks. Deployment began as a read/write-through proxy beside the old system, then shifted traffic. The lab also ships a public Git distribution with packfile-uri and parallel download/index work, and tracks patches aimed at upstream.
[1]Evidence on the table
Primary claims and numbers come from OpenAI’s own production monorepo and from teaching simulations that compare three-phase commit, object-store WAL, and Pando under many concurrent writers. Hard figures include the ~5× median push cut, ~1 PB/day delivery, 57 TB/hour with >99% cache hits, median pack under 20 KiB (p99 under 3 MiB) when advertising trunk only, and clone/fetch saturating 10 Gbps links on standard cloud VMs. Independent third-party benchmarks of Pando are not in the piece.
[1]Steelman and gaps
Steelman. Once agents multiply branches and CI fetches of the same commit, repository-wide lockstep is the wrong bottleneck. Causal consistency matches how humans and agents already work: they care about the order of updates on one branch, not a global snapshot of every ref. Reusing packs is the only way to keep serving CPU off the hot path when many readers want the same history.
Gaps: readers lose a consistent multi-ref snapshot. Successive feature-branch pushes can resend objects outside trunk. The post’s simulations are synthetic teaching models, not external bake-offs. Pando is not open-sourced as a product here; OpenAI says it is exploring how to share the technology. Formal proofs and red-team agents helped OpenAI, but outsiders cannot yet reproduce the stack.
[1]Who should care, and what to watch
Platform and monorepo teams running agent-heavy CI should treat “Git protocol compatibility” as separate from “single-node Git assumptions.” Eval and infra designers should measure push-to-fetch latency under many writers, not only raw clone size. Vendors selling “AI coding” without a plan for ref contention will hit the same wall OpenAI hit late last year.
Watch over six months: whether OpenAI publishes more of Pando or a hosted form; whether upstream Git absorbs more of the packfile-uri and explicit-haves work; whether other labs report similar causal-consistency backends once agent branch volume rises.
[1]Close
The durable claim is not that Git is obsolete. It is that agent-scale writing forces the backend to drop whole-repo consistency so unrelated branches stop waiting on each other, while pack reuse keeps reads cheap. Six months on, either more monorepos ship a Pando-like split between negotiation and cached packs, or agent coding will keep breaking classic Git servers.
[1]