OpenAI Introduces Pando: A Git Backend That Cuts Median Push Latency ~80%

OpenAI has introduced Pando, a new Git backend that now serves all development in its monorepo. The OpenAI Pando Git backend keeps the standard Git client and protocol. Engineers and Codex agents still run git push and git fetch as before. Only the server changed. OpenAI team reports median push latency fell nearly 80%. Pando now delivers nearly a petabyte of Git data per day.

TL;DR

  • Scale: Nearly 1 PB of Git data per day; peak above 57 TB in 1 hour.
  • Runs on: Commodity hardware; standard cloud VMs saturate 10 Gbps links on clone and fetch.
  • Performance: Thousands of concurrent pushers without degradation, per OpenAI’s own testing.
  • Best: Median push latency down nearly 80% (about 5x).
  • Worst: Not open source; repeat branch pushes resend non-trunk history (p99 pack under 3 MiB).

What is OpenAI Pando?

Pando is a Git-compatible server backend from OpenAI that scales concurrent pushes and fetches for a monorepo. It was written in Rust and reached production in 10 weeks. One engineer designed the core and teams of Codex agents implemented it. It has served production traffic since April 2026.

Why did OpenAI need a new Git backend?

Codex multiplied parallel work. OpenAI’s commit chart shows commits landed on master up 1,378% in Q3 2026 versus Q1 2024. Its Q4 2026 estimate is +2,442%. Each push now triggers many CI fetches. By late 2025, fetches and pushes slowed, merges failed, and reliability at times fell below 90%. OpenAI concluded the bottleneck was how it served Git, not Git itself.

How does Pando work?

Pando drops 2 assumptions that most Git servers treat as requirements:

1. Repository-wide consistency: Conventional servers order every ref update repo-wide, so unrelated pushes wait. Pando uses causal consistency instead. Each branch is serialized on its own, and clients only conflict when pushing the same ref. The tradeoff: a reader viewing several branches may see them at different points in time.

2. A unique packfile per client: Standard servers build a fresh pack per fetch. Pando stores each pushed pack as an immutable blob. A “ref version” goes into a scalable metadata store. A push needs O(1) metadata reads and writes. A fetch needs O(log Δ) reads, where Δ is the ref-version gap to close.

How are pushes kept small and reusable?

With millions of branches, a full ref advertisement can exceed 100 MiB. Pando advertises only the trunk branch as the base for new feature-branch pushes. Any reader with trunk history can then fetch that branch from 1 stored pack. In OpenAI’s monorepo, the median pack was under 20 KiB.

How are fetches served at line rate?

Pando returns download links via packfile-uri, an existing Git protocol feature. It sends an empty inline pack. A delivery service called Express caches packs regionally, using consistent hashing, with hit rates above 99%.

What replaces garbage collection?

Each push creates an “L0” pack. Background workers group packs into geometrically larger covers, similar to LSM-tree compaction. The monorepo currently uses covers up to L4. Covers act as a skip list, so pack selection runs in logarithmic time.

‘)});
function draw(){names.forEach(function(n){var el=document.getElementById(‘mtp-‘+n);el.innerHTML=”;for(var i=0;i=4){cnt.L0-=4;cnt.L1++}if(cnt.L1>=4){cnt.L1-=4;cnt.L2++}if(cnt.L2>=4){cnt.L2-=4;cnt.L3=Math.min(cnt.L3+1,3)}draw();resize();if(++n>=27){clearInterval(run);run=null}},260)};
document.getElementById(‘mtp-reset’).onclick=function(){cnt={L0:0,L1:0,L2:0,L3:0};draw();resize()};
function bars(){R.querySelectorAll(‘.hb .f’).forEach(function(f){f.style.width=”0″;setTimeout(function(){f.style.width=f.getAttribute(‘data-w’)+’%’},60)})}
R.querySelectorAll(‘tr.hl’).forEach(function(tr){tr.addEventListener(‘click’,function(){document.getElementById(‘mtp-note’).textContent=tr.getAttribute(‘data-x’)})});
function resize(){try{parent.postMessage({mtpFrame:’mtp-pando’,height:R.offsetHeight+40},’*’)}catch(e){}}
window.addEventListener(‘load’,resize);window.addEventListener(‘resize’,resize);setTimeout(resize,300);
})();

Source link

Leave a Reply

Your email address will not be published. Required fields are marked *