PayPal Lifts Internal AI Accuracy From 43% to 95% Without Touching the Model

How PayPal Doubled Its AI Assistant's Accuracy Without Retraining

PayPal has reported that its internal data science assistant improved its question-answer accuracy from 43% to 95% — without changing the large language model underneath it. The finance platform achieved the jump by adding a proprietary layer of business knowledge that sits alongside the model rather than inside it, according to Ren Zhang, PayPal's VP and global head of AI product, speaking during VentureBeat's AI Impact Series discussion.

The problem was not data quality. The assistant already had access to vetted, high-quality information, but it still could not reliably answer business questions. What it lacked were the definitions, query patterns and analytical procedures that PayPal employees use to turn raw data into answers. A question about "new accounts," for example, might retrieve the right records but apply the wrong definition of "new" — a distinction that depends on business rules, not on what the data happens to contain.

PayPal addressed the gap with a knowledge layer built from components designed to outlast any individual model. One is a "golden query" — a predefined query specifying how to retrieve data for a particular purpose. Another is an analytical playbook that spells out how to investigate a question, including which segments, products or behaviors to examine, such as login failures or passkey use. Zhang said she deliberately designed the assistant to be model-agnostic: when a better model arrives, the team can simply plug it in without rebuilding the knowledge.

The same architecture is now being extended to agents that do more than answer questions. PayPal is building controls that determine when an agent must consult authoritative sources and when a human must approve its actions — a governance question that becomes more pressing as agents take on operational work such as incident response.

The Knowledge Layer Bet: Why PayPal Is Decoupling Expertise From Models

Why PayPal Chose Knowledge Over Model Upgrades

The headline number — 43% to 95% — is less interesting than the design decision behind it. PayPal's bet is that in enterprise AI, accumulated business context is the scarce asset, not model capability. Zhang framed the logic bluntly: "If the model is getting better, I'll just plug in and play. I should not change my knowledge." That is a direct rejection of the common enterprise pattern in which each new model release triggers a fresh round of prompt engineering, retrieval tuning and re-validation. By keeping the knowledge layer separate, PayPal converts model churn from a recurring cost into a swap-in event.

The Fine-Tuning Trade-Off PayPal Walked Back

PayPal's customer-facing commerce agent tells a complementary story. Testing four or five models, the team found larger models capable but slow — roughly 15 seconds of latency. Switching to an 8-billion-parameter model fixed latency but hurt accuracy. Fine-tuning the smaller model on commerce improved results but still fell short, and PayPal then reduced its emphasis on extensive fine-tuning. The stated concern was maintenance: fine-tuning demands specialized expertise, and every new model release creates pressure to repeat the process. Zhang's question — "How many models have come out recently? One is better than the other" — captures the treadmill. This is a notable counterpoint to the prevailing enterprise narrative that customization is the path to production-grade AI. PayPal is not abandoning fine-tuning, but it is explicitly pricing the ongoing cost against the gain.

Agents, Memory and the Human Approval Question

PayPal's agentic work, described by senior director of AI Deepika Khera, shows where the knowledge layer becomes an operational control. A seven-agent workflow investigates alerts across more than 33,000 services, pulling from Datadog, ServiceNow and internal logs to separate signal from noise. A separate security review workflow uses as many as 18 agents. The agent count is task-specific, reflecting the roles agents play, the context each needs and the tools each is permitted to access. Khera identified the central risk plainly: an incorrect diagnosis can lead to an incorrect fix. PayPal's mitigation is vetting every proposed decision against a source of truth, with memory combining certified data sources and standard operating procedures, and many decisions still requiring human verification.

Where the Evaluation Gap Bites

Zhang flagged a subtler failure mode: test conditions that do not resemble production. A system can score well on expected questions while missing the ones real users ask most often. "Your model is still the same. Your process is the same, but the distribution is completely different," she said. VentureBeat's October Agentic Reliability and Evals survey puts numbers on the pattern — 42% of organizations running pre-deployment evaluations reported an AI agent or LLM feature that passed internal tests and then failed within the prior year, and 49% of those said the most significant incident reached customers. PayPal's response is broader offline simulation, synthetic question generation and multi-turn testing, plus model-based judging alongside human assessment. For context, the survey also found 39% of respondents had no automated checks or monitoring alerts among the ways they would first discover bad AI output — a reminder that PayPal's discipline here is not yet the industry norm.

What PayPal Gains — and What Remains Unproven

The reported 1,000-plus questions and roughly 1,600 hours saved over 10 weeks give the internal assistant a concrete usage footprint, though Zhang cautioned against treating question volume as proof of productivity. That caution matters: the 95% figure is an internal accuracy metric on PayPal's own evaluation set, not an independently verified production outcome, and the article does not disclose how accuracy was measured or against what baseline of question difficulty. The strategic value of PayPal's approach is that it makes model upgrades cheap and preserves institutional knowledge — a genuine competitive advantage if it holds. The open question is whether the same knowledge-layer discipline transfers to consumer-facing systems, where users rarely explain what went wrong and failure signals are far noisier.

What Enterprise AI Teams Should Take From PayPal's Playbook

The most important insight: PayPal's 43%-to-95% jump came from encoding business definitions and procedures in a layer the model never absorbs — meaning the durable asset is your knowledge, not your model choice.

For enterprise AI and data platform leaders, the practical implications are specific:

• Audit where your accuracy failures actually originate. PayPal's assistant had clean data but applied the wrong definition of "new accounts." If your system retrieves correctly but answers incorrectly, the gap is business rules, not data or model capability — and no model upgrade will close it.
• Separate knowledge from model before your next upgrade cycle. Zhang's test is simple: when a better model ships, can you plug it in without rewriting prompts, queries and procedures? If not, every release carries a hidden re-validation cost.
• Price fine-tuning as an ongoing expense, not a one-time project. PayPal reduced fine-tuning emphasis after finding the gains did not justify the specialized expertise and repeated work each new model release demands. Weigh customization against the maintenance burden before committing.
• Vet agent decisions against a source of truth and keep human approval for high-consequence actions. PayPal's seven-agent incident workflow covers 33,000+ services, but incorrect diagnoses can trigger incorrect fixes — which is why proposed decisions are checked against certified sources and SOPs.
• Test against the question distribution your users actually produce. Strong scores on expected questions can mask failures on the queries that dominate production. PayPal is running synthetic question generation and multi-turn testing to close this gap.

Metrics worth watching: PayPal's disclosed accuracy methodology and whether the 95% figure holds in production; the latency-versus-accuracy outcome for the customer-facing commerce agent; and whether the knowledge-layer pattern spreads across the enterprise AI market as model release cadence continues.