Building Real-Time Voice AI Agents: Choosing Models, Frameworks, Protocols, and Hosting
A practical guide to building fast, natural, production-ready voice AI.Organizations are racing to bring natural, real-time voice AI to their customers. The need spans two primary surfaces: modernizing clunky IVR menus in legacy contact centers, or embedding voice-first interfaces directly into web…
A practical guide to building fast, natural, production-ready voice AI.Organizations are racing to bring natural, real-time voice AI to their customers. The need spans two primary surfaces: modernizing clunky IVR menus in legacy contact centers, or embedding voice-first interfaces directly into web apps, mobile apps, and smart devices where typing isn’t ideal.The goal is always the same: a conversational assistant that listens, reasons, acts, and speaks without lag or awkward pauses. Achieving this takes a full end-to-end architecture, combining foundation models (like OpenAI Realtime, Amazon Nova Sonic), orchestration frameworks (like Strands Agents), and scalable hosting targets (like Bedrock AgentCore or managed container environments). This post breaks down the voice agent stack layer by layer so you can select the right tools for your platform.The Voice Agent StackVoice engineering just had its “LAMP stack” moment. The space has matured, patterns have hardened, and developers are finally building with a standardized multi-layer architecture. Telling someone “I’m running Nova Sonic + Strands BidiAgent on AgentCore with WebRTC” now explains your full system architecture just as fast as “MEAN stack” did for web devs a decade ago.Whether you’re killing off legacy contact center IVRs or dropping voice interfaces straight into mobile apps and smart devices, the goal is identical: zero-lag, natural conversations that actually get stuff done.Here’s the voice agent stack:This blog post will walk you through each layer: what the choices are, how they compare, and the option trade-offs.1. Model: What Hears, Thinks, and SpeaksThe model layer is the biggest fork in the road for your voice architecture. It sets your latency floor, dictates how much client-side glue code you’ll need to write, and shapes the overall feel of the user experience.You’ve got two main paradigms here, plus an operational pattern that levels up how you stream the audio.Cascaded Pipeline (ASR > LLM > TTS)This is the classic “daisy-chain” approach: separate services chained together in sequence.The flow looks like this: Microphone > VAD > ASR > LLM > TTS > Speaker.On top of running those three core models (AST, LLM, TTS), your orchestration layer has to handle voice activity detection (VAD), sample rate conversions, and barge-in logic on its own.The Ecosystem:Cloud Giants: AWS (Transcribe + Bedrock LLM + Polly), Azure (Speech + Text Chat), Google Cloud (Speech-to-Text + Vertex AI).Niche Speed Demons: Deepgram or AssemblyAI for ASR; ElevenLabs or Cartesia for high-fidelity TTS.The Tradeoff: You get ultimate flex. You can swap out any piece, plug in a hyper-specialized text LLM, or use custom voice clones. The downside? Additive latency (you’re usually looking at 1.2 to 2 seconds round-trip) and a ton of plumbing code to keep state in sync across three different APIs.Bidirectional Streaming (Speech-to-Speech)Instead of chaining models, a single neural network handles speech input, reasoning, and speech output inside one continuous session. Audio goes in, audio comes out. No chopping things up into separate ASR or TTS steps.Frontier Players:Amazon Nova Sonic: Built specifically for low-latency conversational agents, async tool execution, and handling noisy real-world audio.OpenAI Realtime API: Powering the GPT-4o realtime family, featuring asynchronous function calling and seamless context retention.Google Gemini Live: Multimodal streaming that handles real-time audio (and video) with native emotion, pitch dynamics, and smart turn-taking.The Tradeoff: Speed and smoothness. Because speech recognition, tone modulation, and turn management all happen inside the model pass, response times drop to ~300ms–600ms. The setup is vastly simpler since the model handles its own VAD and interruptions natively. The main catch? You’re locked into the provider’s native voice catalog and connection model.The Extension: Going Full-DuplexWhile S2S describes what the model is doing, full-duplex describes how you stream it over persistent connections like WebSockets or WebRTC. Think of full-duplex as an operational superpower built on top of S2S:True Simultaneous I/O: The user can talk while the model is speaking without locking up the channel. Audio flows both ways at the same time.Instant Barge-In: The second the user interrupts mid-sentence, the server fires a truncation event (like an input_audio_buffer.speech_started signal). The client dumps its local playback queue instantly, and the model drops unplayed tokens, making interruptions feel totally natural instead of clunky.Paradigm Comparison2. Framework: What Orchestrates the Voice SessionJust as web apps need Express or Django to handle the HTTP lifecycle, voice agents need frameworks to handle the audio streaming lifecycle: turn-taking, interruption, codec conversion, tool dispatch during speech, and session state. This layer is the voice equivalent of what LangChain, CrewAI, or Strands Agents are for text/multimodal agents, but tuned for real-time audio constraints.Strands Agents is a model-driven SDK designed for building production-ready AI agents across multiple providers. Serving as an intuitive, high-level wrapper over raw S2S model SDKs, it BidiAgent abstracts away low-level WebSockets, event loop complexities, and tool subscription wiring. Developers can build rich voice agent business logic and multi-agent integrations using a single, unified event interface.BidiAgent supports long-running conversations with real-time audio streaming, seamless mid-sentence interruptions, and concurrent processing, delivering a consistent developer experience across the top 3 S2S model providers: Amazon Nova 2 Sonic, OpenAI Realtime, and Gemini Multimodal Live.LiveKit Agents is a room-based real-time media framework with provider-agnostic plugins that supports both bidirectional streaming and cascaded pipelines, with native Nova 2 Sonic integration for speech-to-speech mode. For cascaded pipelines, you can swap STT, LLM, and TTS providers without rewriting orchestration. It includes built-in WebRTC transport, multi-party support, and integrated VAD and turn detection.Pipecat is a pipeline-oriented framework with composable frame processors that also supports both bidirectional streaming and cascaded pipelines, with native Nova 2 Sonic integration available alongside traditional STT/LLM/TTS chains. It provides fine-grained control over audio buffering and timing, an event-driven architecture with backpressure, and pluggable processors at every stage.Many teams also build custom orchestration when integrating with proprietary contact center infrastructure or specific media pipelines.3. Integrations: What the Agent Can DoWhat makes a voice agent “agentic” rather than purely conversational is its ability to take action. The integration layer connects the agent to business systems through tools, knowledge retrieval, and other agents, enabling it to look up accounts, verify identity, schedule appointments, process payments, or escalate to humans mid-conversation.Tools & Service Endpoints: Backend tools rely on serverless compute or standard microservices, exposed through generic API Gateways via RESTful or gRPC endpoints. These represent callable functions that the model invokes during a conversation to handle database queries, CRM updates, payment processing, appointment scheduling, or any custom business action.RAG & Vector Datastores: To deliver fluid voice interactions, RAG must operate with ultra-low latency to avoid disrupting conversational flow. Instead of embedding complex ingestion logic in the streaming loop, the agent queries external vector databases (such as Pinecone, Qdrant, or pgvector) via fast RESTful APIs or MCP. For typical use cases like querying product FAQs, policy documentation, or knowledge base articles, the agent extracts semantic intent mid-conversation, fetches matching chunks from the vector index, and injects them into the model’s context window, enabling immediate, accurate spoken answers without delaying the audio stream.Tool Discovery & Protocol Standardization: Tool integration leverages open standards like the Model Context Protocol (MCP) to present backend services as discoverable, self-describing tool catalogs. By registering RESTful or HTTP endpoints to an MCP server, agents auto-discover available tools at session initialization using standard token-based or API key authentication. This makes tools reusable across different agents and cleanly decouples tool authoring from agent orchestrations.Sub-Agents (Agent-to-Agent Delegation): Sub-agent architectures support Agent-to-Agent (A2A) invocation, enabling a primary agent to delegate specialized tasks to downstream sub-agents with dedicated permissions, isolated runtime environments, or domain-specific models. For example, an authentication agent can verify user identity within a secure boundary, a payment agent can operate in a strict PCI-compliant scope, and a domain specialist agent can process complex niche datasets.Skills & Composable Modules: Skills act as reusable, modular packages that bundle system prompts, tool schemas, and execution logic into single composable units. For instance, a “call transfer” skill packages the required API tool definition, conversational prompting instructions, and escalation logic into a single module that any agent can adopt without reimplementation.Architectural Flexibility: All of these capabilities can be implemented using managed cloud services or built via self-managed open-source components. The managed path reduces operational overhead, while self-managed microservices give full infrastructure control. Most production architectures leverage a hybrid approach tailored to their security and scaling requirements.4. Channel: How Users Reach the AgentThe channel layer determines how audio flows between the user and the agent. There are four common protocols used for voice agents, and each maps to different types of voice agent architectures and solutions.WebSocket and WebRTC (Web, Mobile, IoT)Digital-native channels. These are the two primary protocols for connecting clients directly to a voice agent over the internet, and they serve different purposes.WebSocket provides a persistent, bidirectional TCP connection for streaming audio between a client and the agent runtime. The client is responsible for capturing audio (microphone access, encoding) and playing back received audio. WebSocket is simple to implement, works behind firewalls and proxies, and gives you full control over the audio pipeline. It fits well when you’re building a custom client experience and want to manage audio capture, encoding, and playback yourself.WebRTC is a real-time communication protocol originally designed for peer-to-peer voice and video calls. It operates over UDP for lower latency, includes built-in echo cancellation, noise suppression, automatic gain control, and adaptive bitrate. WebRTC handles the entire audio capture and playback lifecycle natively in the browser or mobile SDK, so the client-side integration is simpler for voice use cases. It’s the better fit when you want production-quality audio handling out of the box, especially in browser and mobile environments where acoustic conditions vary.The key differences:PSTN and SIP (Telephony / Contact Center Intelligence)Telephony channels connect voice agents to the phone network. PSTN provides direct phone number connectivity for any business that wants callers to reach an AI agent, while SIP enables integration with enterprise contact center platforms.PSTN is how end users make phone calls. CPaaS providers bridge the phone network and your agent. Twilio offers Media Streams and Conversation Relay. Vonage provides WebSocket audio streaming. Bandwidth and Telnyx offer direct PSTN with WebSocket delivery. The provider handles number provisioning, PSTN interconnect, and codec negotiation.Your agent receives audio over WebSocket. This is the fastest path to “dial a number, talk to an AI agent,” well suited for SMBs that want an AI agent answering their phone (a restaurant taking reservations, a dental office booking appointments, an auto repair shop scheduling service), as well as for development, prototyping, and production deployments that don’t require enterprise contact center platform integration.SIP is how enterprise contact center platforms route calls. Direct trunk integration connects your agent to platforms like Genesys Cloud CX, Five9, NICE CXone, or Cisco Webex Contact Center. SIP requires signaling (UDP/TCP 5060), RTP media handling (UDP port ranges), and codec conversion (G.711 mulaw 8kHz to PCM 16kHz). This is the production path for enterprise CCI deployments where the voice agent sits alongside or replaces human agents in an existing contact center workflow.Different Channels for Different SolutionsThe channel you choose reflects the type of voice agent you’re building. Digital products use WebSocket or WebRTC. Phone-based self-service uses PSTN via a CPaaS provider. Enterprise CCI uses SIP into existing contact center platforms.5. Hosting: Where It RunsVoice agents have unique infrastructure requirements that standard HTTP setups struggle to handle: persistent WebSocket connections, bidirectional streaming, low latency, and continuous session lifecycle management.Whether you are deploying managed cloud services, orchestrating containers, or managing bare metal, here is how the hosting options break down across environments.Managed Voice Agent RuntimesServices like Amazon Bedrock AgentCore BidiRuntime (or equivalent managed agent runtimes) give you the lowest operational burden.How it works: Native support for persistent WebSockets and WebRTC without relying on connection-upgrading hacks. You supply the containerized application code or a Python package entry point, and the runtime auto-scales based on active streaming connections.Best for: Teams who want to jump straight from local development to production without configuring network proxies, load balancers, or custom scaling hooks.Managed Containers (AWS ECS/Fargate, Azure Container Apps, GCP Cloud Run)Running Docker containers on managed serverless or elastic container platforms balances control with low infrastructure management.How it works: You package your voice agent, context bridge, or proxy into lightweight containers. Load balancers (like AWS ALB, Azure Application Gateway, or GCP HTTPS Load Balancing) handle TLS termination and WebSocket connection routing.Best for: Standard WebSocket/WebRTC voice agents, HTTP fallback routes, and telephony proxy adapters that sit between CPaaS providers (Twilio, Vonage) and your agent logic.Kubernetes Orchestration (AWS EKS, Azure AKS, GCP GKE, On-Prem OpenShift / Vanilla K8s)For workloads that demand low-level networking, Kubernetes provides full control over how containers communicate with client devices and telephony networks.How it works: Kubernetes handles custom container pods that can bind directly to host network interfaces (hostNetwork: true) or utilize Network Load Balancers (NLBs) with raw UDP support.Best for: Direct SIP/RTP media streaming, custom audio codecs (G.711, Opus), or multi-cloud environments requiring identical container configurations across public clouds and on-premises data centers.Virtual Machines & On-Premises Bare Metal (AWS EC2, Azure VMs, GCP Compute Engine, Private Cloud)Deploying self-managed containers (via Docker Compose, Podman) or raw processes on VMs/bare-metal servers gives total architectural freedom.How it works: Auto Scaling Groups or custom orchestrators scale instances based on active connection counts rather than just standard CPU/RAM usage. You retain full control over the underlying OS, kernel-level network tunings, and local hardware access.Best for: On-premises enterprise telephony integrations, strict compliance requirements preventing cloud deployments, or custom low-level C++/Rust audio processing pipelines.Example Voice Agent StacksJust like web developers describe their stack in a single line, voice agent builders can do the same:What’s NextGetting Started: To begin building and deploying real-time voice applications across your target platforms, reference these core technical guides and developer resources:Developer DocumentationAmazon Nova 2 Sonic Developer Guide: API specs for bidirectional speech-to-speech streaming, voice activity detection (VAD), and turn-taking logic.Amazon Bedrock AgentCore Runtime & Gateway Developer Guide: Configuration manuals for persistent WebSocket/WebRTC endpoints, session microVM isolation, and MCP server tools.Strands Agents BidiAgent SDK Documentation: Framework documentation for real-time audio orchestration, async tool dispatch, and streaming state management.Hands-on Code & Blueprintsaws-samples/sample-voice-agent-on-aws: Production-ready reference code covering WebSocket, WebRTC, and telephony proxy architectures.aws-samples/sample-nova-sonic-websocket-agentcore: End-to-end boilerplate featuring a React client, Strands BidiAgent runtime, and automated CloudFormation deployment.Deep Dives & TutorialsIntroducing Amazon Nova 2 Sonic: Architectural breakdown of real-time speech-to-speech models and low-latency audio processing.Building Multi-Agent Voice Systems: Designing scalable, multi-agent voice assistants with Nova Sonic and AgentCore.Building Intelligent AI Voice Agents with Pipecat and Amazon Bedrock: Step-by-step tutorial on integrating open-source audio streaming frameworks with managed Bedrock models.This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!Building Real-Time Voice AI Agents: Choosing Models, Frameworks, Protocols, and Hosting was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI