Key Points

  1. OpenAI developed Performance Factory, an internal Codex-powered system that automates the investigation and mitigation of software performance regressions, tracking over 1,700 investigations and identifying root causes in more than 600 cases in just over three months.
  2. The system evolved from manual debugging through specialized tools to a multi-stage agentic workflow with adversarial review, where Codex agents investigate alerts, diagnose root causes, propose fixes, and verify recovery — though only about 15 shipped fixes were exact Codex-authored pull requests.
  3. OpenAI's approach offers a replicable blueprint for enterprises seeking to balance AI-assisted development speed with human judgment, addressing bottlenecks in code review and performance monitoring as AI agents take on more coding work.

Inside OpenAI's Performance Factory: How Codex Keeps ChatGPT Fast

OpenAI has built an internal AI system called Performance Factory that automates the tedious work of finding and fixing software performance regressions — the slowdowns that creep into applications as code changes accumulate. Powered by OpenAI's Codex coding agent, the system tracked more than 1,700 investigations and identified root causes in over 600 cases in just over three months, according to an exclusive engineering report shared with VentureBeat. That averages about 17 automated investigations per day.

The system addresses a growing bottleneck in modern software development. As AI agents take on more of the burden of writing and changing code, human engineers struggle to manually review every change for performance impact. Performance Factory organizes engineering work into tasks that move through predefined workflows — starting with a performance alert, then progressing through investigation, root cause identification, code changes, review, and verification. Each task maintains its own state and findings, allowing different agents to work on the same problem at different stages.

The system evolved from OpenAI's earlier manual approach, where engineers examined logs, traces, and code changes themselves. The team first developed more than 30 specialized tools to streamline investigations, then created a reusable Codex skill called perf-forensics that captures their investigation procedures. Eventually, they connected performance alerts directly to cloud-based Codex sessions, eliminating the need for engineers to manually copy alerts and paste findings into Slack.

A key innovation is the adversarial review process. OpenAI found that agents sometimes stopped at the first plausible explanation even when evidence didn't fully support it. The team introduced a separate Codex session that receives the entire task record but starts with fresh context, explicitly instructed to find gaps, verify evidence, and test alternative explanations. Currently, the adversarial reviewer sends back around 25% of diagnoses for further investigation. Once a diagnosis passes review, Codex proposes a fix and opens a pull request for an engineer to review and merge.

At a Glance

Main CompanyOpenAI
Developer of ChatGPT and the Codex AI coding agent; built Performance Factory as an internal engineering tool.
Key ProductPerformance Factory
Internal AI system that automates performance regression investigation and mitigation using Codex agents.
Key TechnologyCodex
OpenAI's AI coding agent that powers the automated investigations and fix proposals.
Key Termperf-forensics
Reusable Codex skill capturing the performance team's investigation procedures for agent reproduction.
Key TermAdversarial review
Process using a separate Codex session to challenge original diagnoses and reduce false conclusions.
Key Financial Figure1,700+ investigations
Total automated performance investigations tracked by Performance Factory in just over three months.
Key Financial Figure600+ root causes identified
Cases where the system successfully identified the underlying cause of a performance regression.
Key Financial Figure~15 shipped fixes
Exact Codex-authored pull requests that were eventually deployed to production.
Key DateThree-month period
Duration over which the 1,700+ investigations and 600+ root causes were recorded.
IndustryAI and software development
Sector where agentic workflows are being applied to automate engineering processes.

What OpenAI's Regression Automation Reveals About Agentic Engineering

Why OpenAI Built Performance Factory Now

The timing reflects a structural shift in software engineering. As AI coding agents like Codex, GitHub Copilot, and Cursor accelerate code production, the review and verification stages become the new bottleneck. OpenAI's performance team acknowledged that ChatGPT was dealing with regressions almost daily and struggling to keep track of them. The Performance Factory is essentially an attempt to apply AI to the AI-created problem — using agents to monitor, diagnose, and propose fixes for the side effects of agent-generated code.

For context, this mirrors a broader industry pattern: as AI writes more code, the tooling to verify and maintain that code must also become AI-driven. OpenAI's approach of storing high-level findings and reproducible queries in task records — rather than transferring large volumes of raw telemetry — is a practical pattern for any organization building multi-stage agentic workflows.

The Gap Between Diagnosis and Fix

The results reveal an important limitation. Of the more than 600 investigations that reached a root cause, around 400 progressed to a proposed fix. Approximately 50 shipped fixes were influenced by or matched agent suggestions, while only around 15 shipped fixes were the exact pull requests authored by Codex. OpenAI acknowledges these results fall short of its ambitions for automated remediation. The system is far better at detecting, diagnosing, and tracking regressions than at producing reliable fixes — particularly when a new feature adds latency and requires architectural modifications.

This gap matters for enterprises evaluating similar systems. The value proposition is not full automation but rather the automation of the investigative and tracking work that consumes engineering time. OpenAI's team reports that automating regression investigations has allowed engineers to spend more time on long-term performance improvements instead of constantly responding to incidents.

Adversarial Review as a Quality Control Mechanism

The adversarial review process is perhaps the most transferable innovation. By using a separate Codex session with fresh context to challenge the original diagnosis, OpenAI addresses a fundamental weakness of AI agents: the tendency to stop at the first plausible explanation. The example of the image download diagnosis — where the reviewer noticed that the internal file-locating service had already recovered while image requests continued failing — illustrates how a fresh perspective can catch errors that a single agent misses. The fact that the reviewer still sends back around 25% of diagnoses suggests the original investigator has improved but is not infallible.

What This Means for the Agentic Engineering Market

OpenAI's Performance Factory is an internal tool, not a commercial product, but it signals the direction of enterprise AI. The lessons — avoid AI fatigue by presenting pertinent information, preserve evidence without overloading agent context, evaluate agents against historical incidents, and monitor the agents themselves — are applicable to any organization building agentic systems. Competitors like Google, Microsoft, and Anthropic are pursuing similar agentic coding capabilities, and the enterprise software market is watching closely for proven patterns.

The planned expansion into proactive optimization — comparing CPU profiles, memory data, and request traces across releases to discover slowdowns that alerts might miss — suggests OpenAI sees this as a platform for continuous performance management, not just incident response. The exploration of whether agents could eventually deploy low-risk fixes without human approval is a significant regulatory and trust question that will likely face scrutiny as autonomous code deployment becomes more common.

What Engineering Leaders Should Take From OpenAI's Performance Factory

OpenAI's Performance Factory shows that the highest-value application of coding agents today is not writing code but investigating, diagnosing, and tracking problems — work that consumes engineering time without requiring deep judgment.

For engineering leaders at companies using or evaluating AI coding agents, the key insight is that the bottleneck has shifted from code production to code verification and performance monitoring. OpenAI's system automated the investigative work while keeping humans in the loop for fix approval, achieving a practical balance between speed and reliability.

  • Start with investigation automation, not fix automation. OpenAI's results show that agents are far better at diagnosing regressions (600+ root causes identified) than at producing reliable fixes (only ~15 exact Codex-authored PRs shipped). Target the investigative and tracking stages first.
  • Implement adversarial review for any agent-generated diagnosis. OpenAI's separate Codex session with fresh context catches flawed conclusions — still sending back around 25% of diagnoses. This pattern is directly transferable to any multi-stage agentic workflow.
  • Design task records that preserve evidence without overloading context. Store high-level findings and reproducible queries rather than raw telemetry, so the next agent can verify results without transferring large data volumes.
  • Evaluate agents against historical incidents. OpenAI backtests investigations against past incidents, measuring accuracy in identifying when problems started, their impact, and potential causes. This requires care because historical evidence can expire.
  • Monitor the agents themselves. OpenAI runs a daily agent that examines recent agent logs for recurring tool failures and permission problems, creating tasks to fix issues that repeatedly block investigations.

Metrics worth watching: the percentage of agent diagnoses that pass adversarial review (currently ~75%), the ratio of proposed fixes to shipped fixes, and the time engineers spend on incident response versus strategic optimization.

Risk & Opportunity Assessment

Commercial RiskLowPerformance Factory is an internal OpenAI tool, not a commercial product, so it poses no direct revenue risk to OpenAI or its customers.
Competitive RiskMediumOpenAI's demonstrated ability to automate engineering workflows with Codex could strengthen its competitive position against Google, Microsoft, and Anthropic in the AI coding agent market.
Regulatory RiskLowThe system operates internally with human review of fixes; OpenAI's exploration of deploying low-risk fixes without human approval could attract future regulatory attention but is not yet implemented.
Reputation RiskLowThe engineering report is framed as a transparent look at internal tooling; the acknowledgment that results fall short of ambitions for automated remediation reduces the risk of overpromising.
Technology DisruptionHighThe system demonstrates a practical path for agentic workflows to automate investigative engineering work, potentially reshaping how software teams operate and reducing demand for manual performance debugging.
Commercial OpportunityMediumWhile Performance Factory is internal, the patterns and lessons could inform future OpenAI enterprise offerings or Codex features for automated code review and performance management.