Aitomaton Blog · Benchmarks

AI on the Support Frontline: What the 5,000 Agent Study Showed

The AI assistant suggests answers to agents rather than replying autonomously. We review published numbers, tenure disparities, and limits of generalizability with primary citations.

Evening rush. Chat queue grows, junior reps struggle with recurring questions, while senior leads spend hours answering identical inquiries. The owner asks: what happens if every rep gets an AI copilot?

SummaryIn the journal edition of the experiment (QJE, 2025), a GPT-3 based assistant assisted 5,172 customer support agents: resolutions per hour increased by 15% on average. Novice staff improved speed and resolution quality, while top veterans saw speed rise slightly and quality dip. This represents team augmentation rather than replacement: most agents worked in the Philippines, so generalizing requires careful analysis.

What was actually measured in the 5,000 support agent experiment?

Researchers connected a GPT-3 based assistant to chat support operators: the model observed the conversation and suggested responses, while the human representative decided whether to approve, modify, or ignore the recommendation. The primary performance metric was the number of successfully resolved customer inquiries per hour. In total, the journal cohort tracked 5,172 operators at a Fortune 500 enterprise software provider across a phased deployment.

The experimental design was straightforward: operators handled the same live ticket queues from the same customer base, with the only difference being access to real-time AI suggestions. Human representatives remained fully accountable for the resolution. Consequently, the findings demonstrate the power of workforce augmentation rather than automated replacement.

Crucially for business leaders seeking to adopt these insights: this was a single software vendor with English-speaking support, where the assistant suggested responses rather than conversing directly with customers. Below, we examine who gained the most and why metrics cannot simply be copied.

Job · owner of an expanding company managing high-volume chat support

When inbound ticket queues surge, junior staff struggle with repetitive questions, and senior leads spend hours on routine replies, I want provide every operator with an AI copilot to discover tangible throughput and quality improvements, so that scale operational capacity without inflating payroll while preserving human judgment on critical decisions.

+15%
resolved tickets per hour: average effect of assistant access across 5,172 operators in QJE journal editionQJE 140(2) / NBER WP 31161 · 2025-05

How it relates to your taskThis represents a copilot assisting an operator, not autonomous bots talking directly to clients. Measure your own pilot.

g = −0.23
human + AI pairing underperforming the best single performer; yet outperforming solo humans (g = +0.64)Nature Human Behaviour 2024 · 2024-10-28

How it relates to your taskHybrid collaboration wins only when processes are structured: routine to models, judgment to humans.

81.7%
tau-Voice track leader: approximately one in five real-world telephone tasks still failstaubench.com · 2026-10-12

How it relates to your taskVoice AI is remarkably capable, but requires human failover safety nets, matching our architectural pattern.

Who benefited more: novices or experienced professionals?

The published journal analysis reveals significant heterogeneity: less experienced operators demonstrated marked gains in both turnaround speed and customer satisfaction ratings. For top-tier veterans, velocity increased marginally while resolution quality dipped slightly, because generic suggestions diluted their specialized problem-solving intuition.

Staff Cohort Observed Shift Practical Operational Meaning
Less experienced Both speed and resolution quality improve The copilot accelerates junior onboarding to team benchmarks
Top-tier veterans Speed rises marginally, quality dips slightly Standardized formulations occasionally underperform custom expertise
Combined sample +15% resolved inquiries per hour The average reflects widely differing departmental outcomes

The +15% headline figure represents an aggregate mean. Novices equipped with suggestions ramped up to baseline velocity in weeks rather than months. Conversely, veteran operators who had already mastered system edge cases found pre-generated templates occasionally disruptive.

In our engineering experience, this outcome is logical: suggestions distribute accumulated team knowledge to newer members who lack it, while offering negligible uplift to experts whose workflows are already refined. For hiring and training, copilots deliver maximum leverage during onboarding and provide administrative relief for senior staff.

Why these figures cannot be transferred directly one-to-one?

The sample was drawn from a single enterprise software vendor, the GPT-3 assistant operated strictly in an advisory capacity, and rollout followed a sequential adoption curve rather than a classic double-blind randomized control trial. Most agents operated from offshore facilities in the Philippines, as highlighted in policy research by CERRE. Different linguistic and cultural environments naturally exhibit distinct performance dynamics.

Secondary peer-reviewed sources corroborate this balanced view. IMF Staff Discussion Note SDN/2024/001 analyzes this study within broader macroeconomic labor impacts, while the CERRE report explicitly notes the concentration of offshore talent pools.

What pushes you away from the old way

Inbound queues expand, junior staff take months to achieve target proficiency, and hiring additional personnel strains unit economics.

What pulls you to the new one

AI copilots at the operator desk: faster routine resolutions, rapid onboarding, and +15% hourly throughput demonstrated empirically.

What worries you

Our business operates in different languages and regional markets; will it work here? Pilot testing on internal conversational data is essential.

What keeps you with the habit

Operators are managing currently. However, high workloads cause veteran fatigue and elevated response latency during peak hours.

A realistic framing is clear: external benchmarks indicate operational direction rather than unconditional guarantees. The durable takeaway centers on architecture (an advisory copilot assisting a human operator who retains authority), while the exact productivity multiple must be verified on company-specific customer interactions.

When does human + AI collaboration work, and when does it fail?

A meta-analysis of 106 controlled experiments published in Nature Human Behaviour (2024) synthesized 370 effect sizes from 2020 through 2023. The cross-study average for human-AI pairings versus the best individual agent was g = −0.23 (95% CI: −0.39 to −0.07), indicating that uncoordinated collaboration often trails an optimal autonomous model. In decision-making tasks, pairings lost (g = −0.27), whereas in content creation and synthesis, collaboration delivered clear advantages.

The conclusion for enterprise design: tasks must be partitioned systematically. Models provide instantaneous speed and broad retrieval, while human specialists supply contextual judgment, exception handling, and ethical accountability.

Why even a strong voice agent still operates with a human in the loop?

Independent industry benchmarks show conversational AI advancing rapidly, but enterprise-grade reliability requires structured human supervision across real-world telephone and omnichannel environments.

Text tau-bench

Evaluates multi-turn customer service conversations where language models interact with dynamic APIs and simulated user personas. Success requires the final system state to match ground truth exactly across complex retail, airline, and telecom scenarios.

Voice tau-Voice

Evaluates speech interactions across four demanding service domains. The current state-of-the-art model scores 81.7%, failing roughly one in five tasks due to acoustic noise, overlapping interruptions, and non-standard syntax.

Regional Speech KSC2

The KSC2 open corpus encompasses over 1,128 hours and 520,000 phrases featuring code-switching. Transcription error rates jump from 6.3% on clean read speech to 36.8% on spontaneous conversation, demonstrating why automated systems require human validation.

In multilingual commercial environments, spontaneous conversation and mixed vocabulary are routine operating realities. The AI agent maintains active dialogue and prepares structured drafts, while ambiguous utterances or sensitive discussions escalate seamlessly to human specialists.

How to empower your frontline support without replacing your team?

The formula derived from empirical literature is consistent: generative models handle high-frequency routine interactions, while humans retain authority over complex scenarios, exceptions, and financial decisions. At aitomaton, this is implemented through AI chat agents on first contact, voice agents managing inbound telephony, and the unified AI CRM platform ensuring full attribution and auditability.

In production practice, routine inquiries, appointment scheduling, and ticket classification flow to the conversational agent, which responds within seconds and links history to customer records. Escalations transfer directly to human staff with complete summaries.

  1. Delegate repetitive inquiriesRoute frequent questions, status inquiries, booking requests, and initial triage to conversational agents.
  2. Define strict escalation boundariesEnsure financial transactions, complaints, and policy exceptions route immediately to human representatives.
  3. Maintain human supervisionStaff verify complex tickets, while comprehensive audit trails record whether messages originated from agents or humans.
  4. Benchmark internal metricsTrack resolution velocity and customer satisfaction before and after deployment on live company channels.

End-to-end customer interaction flow with an augmented frontline

  1. InboundCustomer initiates contact

    Inbound chat or call enters the unified inbox; the AI agent responds instantly to clarify details.

  2. RecordComplete CRM recording

    The interaction links to the customer profile, preserving complete history and responsible owner.

  3. Follow-upContextual re-engagement

    Follow-ups and reminders trigger solely within the existing dialogue initiated by the user.

  4. OutcomeSeamless human resolution

    Complex cases escalate to staff with concise briefing notes, recorded in channel analytics.

What this gives your business

  1. Number+15% hourly throughput in 5,172 agent study
    What it means for youAdvisory AI accelerates resolution; juniors achieve competency faster.
    What aitomaton does about itChat and voice agents on first contact, complex tickets routed to staff.
    ValueTeams resolve higher volumes without headcount inflation.
  2. Numberg = −0.23 structural boundary risk
    What it means for youPairings fail when roles and responsibilities lack clear definition.
    What aitomaton does about itExplicit agent parameters and instant context handover to representatives.
    ValueAI delivers velocity while humans preserve judgment.
  3. NumberTransparent supervisory audit
    What it means for youDiscrepancies are caught prior to customer impact through audit logs.
    What aitomaton does about itHuman in the loop with transparent labeling of bot vs human responses.
    ValueSystemic accountability and verified enterprise reliability.

In summary, the 5,000-agent experiment provides clear evidence for workforce augmentation over automated replacement. Partitioning tasks by operational risk, establishing verified boundaries, and measuring internal outcomes delivers sustainable customer service excellence.

Frequently Asked Questions

What was actually measured in the 5,000 support agent experiment?

The study by Brynjolfsson, Li, and Raymond connected an AI assistant based on GPT-3 to customer chat agents: it suggested replies while humans decided whether to send, edit, or ignore. The journal version (QJE, 2025) tracked 5,172 agents and documented a +15% average increase in resolutions per hour.

Who benefited more from the assistant: novices or experienced veterans?

According to the journal version, less experienced agents improved both speed and quality. For top veterans, speed rose slightly while quality dipped marginally, as standardized suggestions averaged out their nuanced strengths.

Why might experimental results not replicate directly in another company?

The study analyzed one English-language software company, the assistant only provided suggestions, rollout was staggered, and most agents operated in the Philippines. Different operating environments yield distinct results.

When does human + AI collaboration underperform isolated systems?

According to the Nature Human Behaviour (2024) meta-analysis, human-AI pairings averaged worse than the best single performer (g = −0.23), losing ground in decision-making (g = −0.27), while outperforming solo humans (g = +0.64).

Can an autonomous voice AI agent operate completely without human staff?

Not yet. The leading model on the tau-Voice benchmark achieves 81.7%, failing roughly one out of every five tasks. Spontaneous speech error rates remain high, making human supervision essential.

What does this evidence mean for customer support teams?

Augmentation rather than replacement: routine speed goes to models, high-stakes decisions and accountability stay with humans. Deploy frontline agents with human oversight and evaluate your own metrics.

See it on your own data

Three days free trial or a live demo tailored to your workflows.

Updated:

Book a platform demo

We will show AI agents working on your business processes, review your funnel and answer all your questions.