Meta's JiTTests System: How LLMs Caught 74 Privacy Bugs in H1 2026 and Cut SEVs by 20%
Meta has moved its "Catching JiTTests" system from a research prototype into full production, leveraging large language models (LLMs) to generate executable tests designed to fail on risky code changes. This system, which runs overnight on spare compute capacity, targets diffs scored as risky and uses an "LLM-as-Judge" ensemble and a rule-based filter called RubFake to rank promising failures. In H1 2026, JiTTests caught 74 serious privacy bugs at the diff stage and helped cut overall privacy SEV occurrence by 20%.
How JiTTests Leverages LLMs for Proactive Bug Detection
The core of Meta's JiTTests system lies in its ability to proactively generate targeted tests using advanced LLMs. Unlike traditional testing methods that often react to existing code, JiTTests focuses on anticipating potential issues in new or modified code. The system operates by analyzing code differences (diffs) that are flagged as potentially risky, then generating specific tests designed to expose vulnerabilities within those changes.
These LLM-generated tests run overnight, utilizing Meta's spare compute capacity. This strategic scheduling ensures that potential bugs are identified early in the development cycle, ideally before they can impact production environments. The approach is detailed in the paper, "Just-in-Time Catching Test Generation at Meta," which was presented at FSE 2026.
Filtering False Positives with "LLM-as-Judge" and RubFake
A critical challenge in automated test generation is managing false positives. JiTTests addresses this through a sophisticated filtering mechanism. The system employs an "LLM-as-Judge" ensemble, where multiple LLMs evaluate the generated test failures to determine their validity. This is complemented by a rule-based filter known as RubFake, which further refines the results by identifying and discarding non-actionable or irrelevant alerts.
This multi-layered approach ensures that engineers receive highly relevant and actionable bug reports, minimizing the time spent investigating false alarms. The effectiveness of this filtering is reflected in the feedback from Meta engineers.
Tangible Impact: Reducing Privacy SEVs and Boosting Engineer Engagement
The deployment of JiTTests has yielded measurable benefits for Meta. Beyond catching 74 serious privacy bugs in H1 2026, the system played a direct role in reducing the overall occurrence of privacy-related SEVs by 20% during the same period. This demonstrates the system's capability to significantly enhance the security and privacy posture of Meta's products.
Engineer engagement with the system has also been notable. A total of 589 Meta engineers interacted with JiTTests, representing 64% of the outreach efforts. These engineers reported a true-positive rate of approximately 60%, indicating that a substantial majority of the bugs flagged by JiTTests were indeed legitimate issues requiring attention. This high true-positive rate fosters trust and encourages continued adoption among the engineering teams.
The Broader Implications for Software Development
Meta's success with JiTTests highlights a growing trend in software development: the integration of AI tools, particularly LLMs, to automate and enhance traditionally manual processes. This proactive, AI-driven testing paradigm offers several key advantages:
- Early Bug Detection: Catching bugs at the diff stage significantly reduces the cost and effort associated with fixing them later in the development cycle.
- Improved Code Quality: By preventing risky code from landing, JiTTests contributes to a higher overall standard of code quality and stability.
- Enhanced Security and Privacy: The system's focus on privacy bugs directly strengthens user data protection.
- Developer Efficiency: Automating test generation frees up engineers to focus on more complex development tasks, while still benefiting from robust testing.
The principles behind JiTTests, as outlined in the FSE 2026 paper, suggest a future where AI-powered testing becomes a standard component of continuous integration and delivery pipelines, moving beyond just conversational AI to critical infrastructure. This approach could be particularly beneficial for large-scale software projects with rapid development cycles.
Conclusion: What's Next for AI-Powered Testing
Meta's production rollout of JiTTests represents a significant milestone in the application of LLMs for software quality assurance. The system's proven ability to catch critical bugs and reduce privacy SEVs demonstrates the practical value of AI in enhancing development workflows. As AI technology continues to evolve, we can expect to see further innovations in automated testing, making software more robust, secure, and reliable. The open-source implementation of Meta's JiTTests, available on GitHub, also paves the way for broader adoption and further research in this critical area.
Sources
- [2601.22832] Just-in-Time Catching Test Generation at Meta
- GitHub - wjddusrb03/jitcatch: AI-powered catching test generator — finds bugs before code review. First open-source implementation of Meta's JiTTests (FSE 2026). · GitHub
- Just-in-Time Catching Test Generation at Meta
- [2504.16472] Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges
- Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges
Recommended AI tools
MailerLite
Marketing Automation
Create Email Marketing Your Audience Will Love
Framer
Design
Design, animate, and publish websites in minutes with AI
HEROZ, Inc.
Data Analytics
AI Agents and SaaS Solutions Driving Digital Transformation
TRAE
Code Assistance
Collaborate with Intelligence
TestMu AI
Code Assistance
Power your software testing with AI and cloud
Vapi
Conversational AI
Build Scalable Voice AI Agents
About the Author

Albert Schaper is a co-founder of Best-AI.org. He focuses on product strategy, AI adoption, practical tool selection, and educational content that helps users compare AI products with clearer context.
More from AlbertWas this article helpful?
Found outdated info or have suggestions? Send us a note.