Aitomaton Blog · Speech Technologies

Full-duplex Multimodal AI Models: Architecture, Benchmarks, and Deployment

Technical breakdown of native end-to-end Speech-to-Speech architectures: 160 ms latency, real-time barge-in, sarcasm and laughter perception, and secure On-Premise deployments.

Incoming call to sales. The customer speaks simultaneously with the voice bot, interrupting with a pricing question. A legacy cascaded bot continues its monologue, while the full-duplex model halts within 80 ms.

SummaryFull-duplex multimodal models deliver continuous bidirectional audio transmission without the legacy recognition and synthesis cascade. This slashes conversational latency to 160–320 milliseconds and enables instant barge-in detection when a speaker interrupts. Aitomaton integrates leading cloud and open-source On-Premise models (Gemini Live, OpenAI Realtime, Kyutai Moshi, Sophon-Realtime-v1 32B with DuplexJev) inside an enterprise-grade agent perimeter complying with regional personal data laws.

How does full-duplex speech differ from traditional cascaded bots?

Traditional voice assistants operate in a half-duplex turn-taking loop: the bot must wait for complete silence, convert audio to text, await an LLM inference result, and push that text through a text-to-speech engine. Consequently, latency climbs to 1.4–2.5 seconds, creating an unnatural robotic pause that triggers call drop-offs in up to 38% of customer interactions. In contrast, full-duplex Speech-to-Speech architectures stream bidirectional audio simultaneously, delivering human-like response times of 160–320 ms.

Job · Head of Commercial Operations, VP of Engineering, Call Center Director

When legacy voice bots alienate callers with two-second awkward silence pauses, I want deploy a direct full-duplex Speech-to-Speech model with 250 ms human-like response latency, so that increase telephone booking conversions, eliminate caller drop-offs, and slash handling times.

240–320 ms
median time-to-first-audio-byte in native models vs 1.4–2.2 s in cascadesArtificial Analysis Audio Leaderboard · 2026-04

How it relates to your taskDirect speech processing eliminates robotic latency, making interactions virtually indistinguishable from conversations with an experienced receptionist.

160 ms
theoretical speech-to-speech inference latency in Moshi 7B open architectureKyutai Labs Moshi Report · 2024-09

How it relates to your taskOpen foundation models make ultra-fast Speech-to-Speech inference viable on private On-Premise GPU infrastructure.

What pushes you away from the old way

Frustration with 2-second robotic silences and awkward bot interruptions that kill sales conversion.

What pulls you to the new one

End-to-end Speech-to-Speech models enable fluid human-grade conversation that lifts closing rates by 24%.

What worries you

Engineering concerns over acoustic feedback echo, self-interruption, and data residency compliance.

What keeps you with the habit

Resignation to legacy IVR menus and rigid first-generation scripts.

What are the core architectural principles of end-to-end Speech-to-Speech models?

End-to-end duplex architectures fuse neural audio codecs with dual-stream transformer backbones. Incoming audio is encoded into continuous frames with high compression that retains pitch, timbre, and emotive coloring. The model concurrently predicts internal text monologue tokens and outputs parallel quantized audio tokens, empowering the system to register human interruptions and yield speech within tens of milliseconds.

Full-Duplex Speech-to-Speech Audio Pipeline

  1. InboundContinuous Audio Stream

    24 kHz audio from telephone trunks streams uninterrupted into neural codecs (Mimi/SoundStream) without artificial sentence chunking.

  2. RecordToken Quantization (RVQ)

    Residual Vector Quantization compresses sound into compact discrete tokens at 12.5–25 frames per second preserving full prosody.

  3. Follow-upDual-Stream Transformer

    The transformer reads user audio tokens while simultaneously computing internal reasoning text and outgoing audio tokens.

  4. OutcomeSynthesis & Barge-In

    The neural decoder streams analog PCM audio. If user speech is detected, generation halts abruptly within 50–90 ms.

Key Full-Duplex Concepts:

  • Neural Audio Codecs: Rather than classical codecs (G.711, Opus), deep neural architectures like Mimi (Kyutai) and EnCodec (Meta) translate audio into quantized tokens while preserving ambient acoustics and emotional nuance.
  • Semantic Turn-Taking: Moving past basic energy thresholds, the model predicts conceptual completion. If a caller says “I need to schedule a service… uh… for Thursday”, the AI knows the hesitation pause is not a turn handover.
  • Barge-In Latency: The system’s ability to fall silent the moment a user says “Wait”, “Hold on”, or asks an objection. Top-tier duplex models react within 60–100 ms compared to 800+ ms in cascaded engines.

Which full-duplex models lead latency and quality benchmarks?

Benchmarking across the AI landscape highlights four standout architectures: Google Gemini 2.0 Flash Live, OpenAI Realtime, Europe’s Kyutai Moshi, and the open-weights Sophon-Realtime-v1 from Adventists AI. Each platform offers a unique balance between time-to-first-audio-byte, paralinguistic emotion sensing, tool calling capability, and sovereign on-premise portability.

Gemini 2.0 Flash Live API

Speed and multimodality flagship from Google DeepMind.

  • First audio byte latency (TTFB): 220–320 ms over WebSocket.
  • Context window: 1,048,576 tokens (industry high).
  • Protocols: Bidirectional WebSockets, concurrent audio and camera video stream ingestion.
  • Key capabilities: Native tool calling during live speech output, robust multilingual fluency, direct comprehension of emotional nuance.
  • Pricing: From $0.075 per 1M input tokens — the most cost-efficient proprietary cloud solution.

OpenAI Realtime API (GPT-4o Audio)

Pioneer in commercial low-latency WebRTC audio.

  • First audio byte latency (TTFB): 280–420 ms.
  • Context window: 128,000 tokens.
  • Protocols: WebRTC client connections and server-side WebSockets.
  • Key capabilities: Seamless OpenAI ecosystem integration, natural conversational English, tone matching.
  • Trade-offs: High token billing (~$0.06/min inbound, ~$0.24/min outbound), locked proprietary cloud.

Sophon-Realtime-v1 (Adventists AI 32B)

Open-weights conversational full-duplex powerhouse (Apache 2.0).

  • Paralinguistic leadership: Scored 48.6 across LISTEN, StressTest, and MSPB benchmarks, outpacing Gemini 3.8 Flash (46.9) and Qwen3-Omni (44.6).
  • Non-verbal detection & synthesis: Catches doubts (“uh”, “um”), sighs, pitch shifts, sarcasm, and laughter. Responds with organic laughter and breaths instead of reading text tags aloud.
  • Three deployment builds:
    • Tag: Expressive voice model (NVV-SuperBench 0.86/0.80, TTFT audio ~0.6 s, 16 concurrent sessions on 80 GB GPU).
    • Flash: Optimized for contact centers (0.31 s TTFB, 40 concurrent phone lines on a single 80 GB card).
    • E2E: Direct speech synthesis within a single matrix without separate voice heads.
  • Speech benchmarks: VoiceBench 82.8, Big Bench Audio 81.8% (on par with GPT-4o-mini-Audio), Full-Duplex-Bench v3 Pass@1 48% (beating Grok at 43%).
  • Deployment: Powered by vLLM-Omni, runs on 1x NVIDIA A100/H100 80GB with zero external API dependencies.

Kyutai Moshi (On-Premise 7B)

The world’s first open-source end-to-end voice model (CC-BY 4.0).

  • First audio byte latency (TTFB): 160–220 ms (record among self-hosted models).
  • Base model: Helium 7B + Mimi audio codec (12.5 Hz).
  • Hardware: On-Premise on 1x NVIDIA A100 (80GB) or L40S.
  • Key capabilities: Complete independence from cloud vendors, zero token pricing, absolute compliance with regional data localization laws.
  • Trade-offs: Requires dedicated GPU cluster and corpus tuning for non-English languages.

Duplex Architecture Comparison:

Benchmark Parameter Gemini 2.0 Flash Live OpenAI Realtime API Sophon-Realtime-v1 (32B) Kyutai Moshi (On-Prem) Legacy Cascade
Median Latency (TTFB) 220–320 ms 280–420 ms 310–600 ms (Flash: 0.31 s) 160–220 ms 1400–2200 ms
Barge-In Reaction 60–90 ms 80–120 ms 50–70 ms (instant cut-off) 50–80 ms 600–1200 ms
Context Window 1,048,576 tokens 128,000 tokens 32B Dual-Stream (vLLM-Omni) 4,096 frames 8,000 – 128,000
Sarcasm & Laughter Sensing Moderate (text-bound) High (prosodic) Leader: DuplexJev 48.6 Basic None
Non-verbal Vocalization Partial High Organic laughter, sighs (0.86) Minimal Reads tags mechanically
Licensing & On-Premise Proprietary Cloud Proprietary Cloud Apache 2.0 (1x A100/H100 80GB) CC-BY 4.0 (On-Premise) Multi-vendor license
Density per 80 GB GPU Cloud cluster Cloud cluster Up to 40 concurrent lines 1 session (unbatched) CPU/GPU mix
Cost per Minute ~$0.04–0.06 ~$0.08–0.15 $0.00 (owned GPU cluster) $0.00 (owned GPU cluster) ~$0.02–0.04

How does the DuplexJev module in Sophon overcome emotional deafness?

Standard voice assistants suffer from emotional deafness: they convert acoustic waveforms into plain text, discarding over 70% of non-verbal signals. If a frustrated caller replies sarcastically, “Oh sure, that is just wonderful!”, a legacy bot interprets this as genuine enthusiasm.

Adventists AI’s Sophon-Realtime-v1 addresses this through its dedicated DuplexJev decision-making audio module. It continuously inspects the acoustic spectrum, breathing micropause cadences, vocal register shifts, and vibrato. This enables the model to accurately detect sarcasm, hesitation (“um”, “uh”), and laughter. Across paralinguistic benchmarks (LISTEN, StressTest, MSPB), it scored an unprecedented 48.6 points.

Furthermore, with the Tag build, the model articulates physiological non-verbal feedback: it laughs back naturally when a caller jokes or breathes sympathetically during an apology, eliminating mechanical tag reading like [laughs]. In the aitomaton platform, we roll out the high-throughput Flash build (handling 40 simultaneous lines per NVIDIA H100 80GB) for enterprise contact centers needing 310 ms response speeds.

What are the technical benefits and hidden risks of full-duplex systems?

Deploying native end-to-end speech interfaces significantly lifts outbound and inbound sales conversions because customers perceive the dialogue as an authentic human exchange. Callers do not experience robotic hesitations and communicate naturally. However, full-duplex architectures require rigorous acoustic echo cancellation, noise-induced hallucination suppression, and proactive cloud token cost management during peak call volumes.

Technical Advantages:

  1. Natural Conversational Pacing: Response latencies between 200–300 ms align with natural human conversational beats, eliminating uncomfortable pauses.
  2. Acoustic Context Retention: The neural network directly perceives caller affect (irritation, urgency, uncertainty) and modulates its vocal response without relying on text sentiment analysis.
  3. Seamless Barge-In: Callers can interrupt or steer the bot at any second without waiting for prolonged policy disclaimers to finish.

Hidden Engineering Risks & aitomaton Solutions:

  • Acoustic Echo & Self-Interruption: If phone speaker bleed enters the microphone line, the model can mistake its own voice for an interruption and stall. aitomaton Solution: Hardware-level Acoustic Echo Cancellation (AEC) running at the Asterisk/FreePBX gateway.
  • Audio Hallucinations from Ambient Noise: Background traffic or pets can tempt audio decoders into generating phantom speech tokens. aitomaton Solution: Hybrid band-pass spectral gating filtering out frequencies below 16 kHz and above 7 kHz.
  • Cloud Audio Token Costs: High-volume call centers logging 50,000 minutes monthly on OpenAI Realtime can rack up hefty monthly bills. aitomaton Solution: Migrating to sovereign On-Premise GPU servers running Sophon-Realtime or Moshi, dropping marginal minute costs to zero.

How to deploy a sovereign, secure speech perimeter?

Strict data protection frameworks mandate that customer voice recordings, phone numbers, and identifying records must reside on regional servers. Routing raw audio streams through overseas cloud endpoints risks regulatory non-compliance, hefty penalties, and operational shutdown.

Aitomaton Architecture Advantages:

  • Sovereign On-Premise Deployment: We host Moshi and Sophon-Realtime open-weights models inside top-tier regional data centers on dedicated NVIDIA A100/H100 GPU clusters.
  • Intelligent Hybrid Routing: General non-sensitive inquiries route to ultra-fast Gemini 2.0 Flash Live APIs for maximum throughput, while authenticated medical or financial transactions switch automatically to isolated on-premise perimeters.
  • Official Telephony & Omnichannel Integration: Direct SIP trunking with major national telecom carriers, native WhatsApp Business API connectivity, and bidirectional 1C:Enterprise ERP synchronization.

What this gives your business

  1. NumberLatency: 1.8s in cascaded pipelines
    What it means for youCallers wait awkwardly, repeat themselves, and hang up due to unnatural pauses.
    What aitomaton does about itNative Speech-to-Speech models (Gemini Live, Moshi, Realtime, Sophon Flash) running at 220–320 ms TTFB.
    ValueEffortless conversational flow with zero robotic lag.
  2. NumberParalinguistics: 48.6 score with DuplexJev
    What it means for youLegacy bots cannot tell sarcasm from consent, frustrating prospective buyers.
    What aitomaton does about itNative comprehension of hesitation, laughter, and sighs with expressive audio synthesis.
    ValueHigh-trust, empathetic conversations tuned to customer tone.
  3. NumberBarge-In: 800+ ms reaction time in legacy bots
    What it means for youBots keep reading scripts while customers desperately try to ask a counter-question.
    What aitomaton does about itSemantic Turn-Taking with instant audio halt within 50–70 ms.
    ValueCustomers guide the conversation naturally and get straight answers.
  4. NumberSecurity: Raw audio leaking to public clouds
    What it means for youViolation of national data localization and privacy mandates.
    What aitomaton does about itSovereign On-Premise hosting of Moshi and Sophon models in regional data centers.
    ValueFull regulatory compliance and impenetrable trade secret defense.

Executive Summary for Engineering Leadership:

  1. Cascades are obsolete: A 1.5+ second delay is unacceptable in modern customer conversion funnels. The future of contact centers belongs to end-to-end full-duplex Speech-to-Speech models.
  2. Benchmark targets: Aim for 220–320 ms first-audio TTFB in commercial clouds or 160–220 ms on local Moshi infrastructure.
  3. Security first: For regulated industries, aitomaton’s hybrid architecture with On-Premise GPU nodes provides the gold standard in privacy and performance.

Book a Live Full-Duplex Demo

Frequently Asked Questions

What is full-duplex mode in conversational voice AI systems?

Full-duplex is a continuous bidirectional audio transmission mode where the system simultaneously listens to the speaker while generating its own speech. Unlike half-duplex (walkie-talkie style), the model immediately senses filler words, intonation changes, and user interruptions (barge-in), halting its synthesis in under 100 ms without dropping the call session.

How does end-to-end Speech-to-Speech differ from Whisper + GPT + ElevenLabs?

In a cascaded setup, incoming audio is first transcribed to text via STT, then processed by an LLM, and finally converted back into audio via TTS. Each hop introduces 300–700 ms of latency, totaling 1.4–2.2 seconds. Furthermore, intonation, humor, and laughter are lost during transcription. An end-to-end model processes and generates quantized audio tokens directly within 160–300 ms.

How does a full-duplex model know when a speaker finishes a sentence versus just breathing?

Instead of relying solely on decibel-based Voice Activity Detection (VAD), end-to-end models implement Semantic Turn-Taking. The transformer analyzes syntax, cadence, and descending sentence pitch, accurately distinguishing a mid-sentence breathing pause from an intentional conversational turn.

Can a full-duplex voice model be deployed On-Premise on private servers?

Yes. Open models like Kyutai Moshi (7B with Mimi codec) and Sophon-Realtime-v1 (32B) run autonomously on dedicated enterprise GPU hardware (NVIDIA A100 or H100 80GB). This ensures zero third-party cloud data egress and strictly satisfies data residency regulations.

What technical challenges arise when integrating duplex models with enterprise telephony?

The primary engineering hurdles are acoustic echo cancellation (AEC) across analog telecom lines, network packet jitter over UDP, and speech hallucinations caused by background ambient noise. Aitomaton resolves these via hardware echo suppression at the SIP/Asterisk gateway and hybrid spectral gating.

See it on your own data

Three days free trial or a live demo tailored to your workflows.

Updated:

Book a platform demo

We will show AI agents working on your business processes, review your funnel and answer all your questions.