Meet Meta’s JiTTests: LLM-Generated Tests Designed to Fail Before Risky Code Lands





Meta has moved Catching Just-in-Time Tests (JiTTests) from research prototype into its production infrastructure. Meta JiTTests is an LLM-based system that writes executable tests designed to fail on a risky code change. A failing test is concrete evidence that a diff changed behavior. Meta team reports that JiTTests caught 74 privacy SEVs at the diff stage in the first half of 2026. The method is described in the FSE 2026 research paper ‘Just-in-Time Catching Test Generation at Meta‘.

TL;DR

  • Scale: Runs on Meta backend code with hundreds of millions of lines. The research paper analyzes 22,126 generated tests.
  • Runs on: Meta’s internal infrastructure, overnight on spare capacity. Judge models in the paper: Llama3.3-70B, Gemini 3 Pro, Claude Sonnet 4. Not publicly available.
  • Performance: Engineers report a 60% true-positive rate at full scale (self-reported).
  • Best: 74 privacy SEVs caught before landing in H1 2026; privacy SEV occurrence down 20%.
  • Worst: The initial deployment confirmed only 8 of 41 reported catches (19.5%).
  • Bottom line: Best: executable proof of a bug before merge. Worst: internal only, with self-reported numbers.

What are Catching JiTTests?

Catching JiTTests are LLM-generated tests that should fail when a diff introduces a bug. Traditional hardening tests pass when written and guard against future regressions. Catching tests target the change under review, before it lands. Meta formalized the details in its 2025 paper ‘Harden and Catch’.

Most AI bug tools act like faster human reviewers. They read the diff and leave comments. JiTTests instead execute code and show a concrete behavioral difference. Meta team concedes this takes more computational steps. It argues the approach catches bug classes that LLM or agent review can miss.

How does the JiTTests pipeline work?

The production pipeline has 5 stages:

  1. Targeting: Runs overnight on spare capacity. It selects the previous day’s risky diffs, using a mechanism similar to Meta’s Diff Risk Score.
  2. Test generation: 2 diff-aware workflows produce candidate tests (explained below).
  3. LLM-as-Judge ensemble: Several LLMs score each failing test using the test code, failure trace, diff and inferred intent.
  4. RubFake: The Rule-Based False-Positive Killing Environment adds deterministic checks. Its rules flag patterns like reflection, broken mocks and not-implemented exceptions.
  5. Human-on-the-loop: The system decides whether to contact an engineer, based on the combined ranking.

Dodgy diff vs intent-aware: what is the difference?

The dodgy diff workflow assumes every diff is buggy. It treats the diff as a mutant of its parent and generates tests that tell them apart.

The intent-aware workflow first infers what the diff should do from its code, title and summary. It lists risks, builds a mutant for each, and writes tests to kill them. Tests that then fail on the real diff become ‘weak catches.’

What do the benchmarks show?

The research team analyzes 22,126 generated tests. Diff-aware methods produced 4x more candidate catches than hardening tests. They produced 20x more than tests that failed by coincidence.

The intent-aware workflow produced weak catches in 6.4% of its tests and on 7.9% of diffs. Dodgy diff scored 2.5% and 4.0%. Meta’s earlier ACH mutation-guided hardening scored 0.8% and 1.6%. The rule-based and LLM assessors cut human review load by 70%.

Source link

Leave a Reply

Your email address will not be published. Required fields are marked *