How AI Voice Agents Work: Inside the Production Pipeline

Written by: Matt Beucler, CEO, Plura AI

Updated August 23, 2026

Key Takeaways

• AI voice agents run on a four-layer stack of STT, LLM reasoning with tool calls, TTS, and carrier to support sub-second, autonomous phone conversations.
• Production performance depends on low-latency streaming components, accurate Voice Activity Detection, and carrier-level SHAKEN/STIR authentication to reduce spam labeling.
• Tool calling lets agents work directly with CRMs, calendars, and payment systems mid-conversation, while zero-trust security and compliance controls keep enterprise deployments safe.
• Stateful cross-channel memory and branded caller ID ensure callers avoid repeating themselves and see verified company identity on every call.
• Plura AI delivers this full stack on 100% U.S. infrastructure; start a conversation with Plura AI today to see production voice agents in action.

The Four-Layer Stack Behind Production Voice Agents

Every production AI voice agent runs on four core layers. These layers separate a polished demo from a system that handles 500+ daily interactions without breaking.

1. STT (Speech-to-Text): Converts incoming caller audio into a text transcript in real time. Streaming STT models like Deepgram Nova-3 achieve P50 first-token latency of 106 ms inside a co-located cloud environment.⁴
2. LLM reasoning + tools: The transcript enters a Large Language Model that decides what to say and whether to call an external tool such as a CRM lookup, calendar booking, or order-status check. This layer drives the agent’s reasoning.
3. TTS (Text-to-Speech): The LLM’s text response is synthesized into audio and streamed back to the caller. Streaming TTS starts playing audio from the first sentence while the LLM continues generating the rest.
4. Carrier: The telephony layer originates and terminates the call over PSTN or VoIP. Many AI voice tools rent this layer from a CPaaS provider like Twilio.⁴ Plura owns its FCC-licensed audio bridging carrier, so branded caller ID, SHAKEN/STIR authentication, and compliance enforcement run at origination, not as bolt-ons.

The 5-Step Production Pipeline in Live Conversations

Those four layers form the architecture. The following five-step sequence shows how a real conversation moves through this stack from the moment a caller speaks to the moment they hear a reply.

1. Voice Activity Detection (VAD): VAD runs continuously on the incoming audio stream and classifies each 20 ms frame as speech or silence. It fires an end-of-utterance signal when the caller stops speaking, which triggers STT finalization. Weak VAD causes agents to cut callers off or wait too long to respond.
2. Streaming STT: The audio frames pass to a streaming STT model that emits partial transcripts as the caller speaks and a final transcript when VAD signals end-of-turn. Streaming STT models like Deepgram Nova-3 can produce final transcripts at low latency in cascaded pipeline tests, so downstream stages start work before the caller finishes speaking.
3. LLM reasoning + tool calls: The final transcript plus conversation history enters the LLM. When the LLM determines a tool call is needed, such as a CRM lookup, calendar check, or order query, it executes that call and appends the result before generating a response. This recursive loop supports multi-step tool chains inside a single conversational turn.
4. Streaming TTS: A sentence buffer accumulates LLM tokens until a complete sentence forms, then forwards that sentence to TTS synthesis. Streaming TTS can reach competitive time-to-first-byte latency in production tests, so the caller hears the first sentence while the LLM is still generating the second.
5. Carrier origination with SHAKEN/STIR: The synthesized audio travels over the carrier layer to the caller’s phone. SHAKEN/STIR (Secure Telephone Identity Revisited / Signature-based Handling of Asserted information using toKENs) authenticates the call’s origin at the carrier level and reduces spam-label risk. Plura issues branded caller ID directly through its FCC-licensed carrier rather than inheriting a third-party reseller’s reputation.

Run your numbers through Plura’s ROI calculator to see how this pipeline compares with your current cost per contact.

Handling Barge-In and Interruptions on Live Calls

Barge-in, where a caller speaks while the agent is mid-sentence, is one of the hardest problems in production voice AI. The VAD layer runs a state machine: IDLE transitions to LISTENING when speech appears, LISTENING transitions to PROCESSING after a silence threshold of roughly 300 to 700 ms, and PROCESSING transitions to SPEAKING when TTS begins. When the caller speaks during SPEAKING, VAD fires an interruption signal.

That signal must move through the entire pipeline in under 200 ms to feel natural. A Pipecat-based pipeline propagates a StartInterruptionFrame that clears TTS playback, LLM generation, and buffers within one 20 ms audio frame. The system then reconciles conversation state so only the audio the caller actually heard is recorded as spoken. Logging the full LLM output would corrupt later responses with information the caller never received.

The harder problem is separating a real interruption from a pause, a backchannel such as “uh-huh,” or background noise. Semantic turn detection, which evaluates partial transcripts with a lightweight model, can reduce false interruptions compared with VAD alone while adding minimal latency.

Plura’s production deployments combine acoustic VAD with semantic endpointing so the agent stops when a caller truly interrupts and continues through natural pauses. This approach delivers barge-in recovery above 90%, which industry benchmarks identify as the standard for natural-feeling voice agents.

Tool Calling: How Voice Agents Work with Your Systems

Tool calling separates an AI voice agent from a basic voice bot. A voice bot matches speech to a script. A voice agent calls external systems such as CRM, calendar, order management, or payment processors mid-conversation and acts on the result.

The mechanism uses a recursive loop. The LLM receives the conversation history plus a set of tool definitions formatted as JSON schemas. If the model returns a tool_calls object, the orchestration layer executes those calls, appends the results to the conversation history, and re-enters the LLM. If the model returns a text response instead, that text streams through the sentence buffer to TTS. No open-source speech-to-speech model currently supports function calling, so cascaded STT-LLM-TTS pipelines remain the preferred architecture for enterprise deployments that require tool use.

Security at the tool boundary must hold at scale. Each tool defines a clear purpose and input schema, uses least-privilege credentials, and includes parameter validation, timeouts, retries, idempotency controls, and a complete audit record. Consequential operations such as writes, payments, or account changes use explicit approval gates separate from the model’s output.

Plura enforces zero-trust tool execution. Tool-call payloads are constrained to only the data required for the current turn, write actions are separated from read actions, and every tool invocation is logged with a tool_invocation_id for cross-layer observability. This architecture supports Plura’s SOC 2 and HIPAA compliance posture while reducing the security work operators need to build themselves.¹

Voice Activity Detection Settings for Enterprise Calls

VAD acts as the gatekeeper of the entire pipeline. Weak settings cause the agent to talk over callers or pause long enough that callers assume the line dropped. WebRTC VAD can miss a substantial portion of speech frames in real-world conditions, while Silero VAD achieves higher true positive rates.

Silero VAD, a 2 MB model, operates on 20 ms audio chunks and evaluates each frame on a single CPU core before any buffering delay.⁴ Production settings for Silero VAD are tuned to the environment, often with a higher activation threshold in noisy conditions plus suitable minimum silence duration and speech padding.

Silero alone does not cover enterprise requirements. Hybrid systems combine acoustic VAD with a small language model under 10 billion parameters that evaluates partial transcripts for syntactic completeness. An incomplete phrase like “I was walking down the…” raises the silence threshold to 1,500 ms. A complete question like “Can you book the flight for Tuesday?” triggers the LLM at 500 ms. This dynamic endpointing approach can reduce the rate at which agents talk over users compared with threshold-based VAD alone.

Plura tunes production VAD settings per deployment vertical, since contact center ambient noise differs from home-services calls, and iterates those settings continuously against real call data rather than setting them once at launch.

Streaming TTS and Perceived Response Time

Non-streaming TTS waits for the LLM to finish generating the entire response, then synthesizes it as a single audio block. Naive sequential execution creates long delays before first audio, which often feel like dropped calls.

Streaming TTS uses a sentence buffer. The buffer accumulates LLM tokens until it detects sentence-ending punctuation, excluding abbreviations and decimals, and enforces a minimum character threshold of roughly 10 characters before forwarding the sentence to TTS. The TTS engine begins emitting audio chunks from that first sentence while the LLM continues generating the second and third. This overlapping execution converts a sum-of-stages latency into roughly the maximum of individual stage latencies and sharply cuts perceived response time.

The latency budget for TTS in a production system stays tight. Plura’s TTS layer is co-located with LLM inference and the carrier layer, which removes the multi-hop network overhead that inflates TTS latency in API-reseller architectures.

Latency Targets for Production Voice Agents

Production voice AI systems enforce an 800 ms end-to-end latency budget from end-of-speech detection to first audio output. Exceeding 1 second makes the conversation feel robotic, and above 1.5 seconds callers often hang up or repeat themselves. The 800 ms budget breaks down as follows:

• VAD end-of-speech detection: approximately 100 ms
• STT finalization: 100–200 ms
• LLM response generation (time-to-first-token): 200–400 ms
• TTS time-to-first-byte: 150–200 ms
• Telephony round-trip: 50–150 ms

By April 2026, median end-to-end voice AI latency in production environments reached roughly 680 ms, down from approximately 1,200 ms in 2024.³ That median now sits comfortably inside the 800 ms target that defines natural-feeling conversations. P95 latency tells the real story, because noise, accents, and barge-in conditions typical of enterprise calls can push P95 into the zone where deals fall apart.

Co-location provides the single most effective architectural fix. Placing ASR, LLM, TTS, and telephony in the same data center reduces network latency and can deliver large reductions in total system latency for already-optimized voice agents. Plura’s carrier-owned stack places STT, LLM inference, TTS, and call origination on the same U.S. infrastructure, which removes the multi-hop penalty that API-reseller architectures cannot avoid.

Run your numbers through Plura’s ROI calculator to model what latency reduction means for your contact rates and cost per conversation.

AI Voice Bot vs. AI Voice Agent in Production

AI voice bots rely on rule-based decision trees with no tool calling or cross-turn memory. AI voice agents use LLM reasoning per turn with API calls inside a turn and session-level context. Plura AI Voice Agent adds LLM reasoning plus stateful memory across channels, recursive tool loops with zero-trust execution boundaries, and Plura’s FCC-licensed audio bridging carrier with SHAKEN/STIR.

Live Call Example: Tool Calls Inside a Conversation

The following transcript shows how tool calls surface inside a live conversation on Plura’s pipeline. Each bracketed note shows what the system executes between turns.

Caller: “Hi, I need to reschedule my appointment for Thursday.”

[VAD fires end-of-utterance. STT finalizes transcript. LLM detects scheduling intent and calls calendar API with the caller’s phone-keyed record from Stateful Conversation Database.]

Agent: “I can see your Thursday 2 p.m. appointment with Dr. Reyes. What day works better for you?”

Caller: “Monday morning if possible.”

[LLM calls calendar API to check Monday availability and returns an open slot at 9 a.m.]

Agent: “Monday at 9 a.m. is open. Should I move it there and send you a confirmation text?”

Caller: “Yes, perfect.”

[LLM calls calendar write API to reschedule, calls SMS API to send confirmation, and logs the outcome to Stateful Conversation Database.]

Agent: “Done. You’re confirmed for Monday at 9 a.m. You’ll get a text in a moment. Is there anything else I can help with?”

This exchange, which includes two tool calls, one write, and one SMS trigger, completes inside the conversational turn without a human stepping in.

Stateful Memory Across Channels and Carrier-Level Controls

Most AI voice tools treat each call as a fresh session, so a caller who texted at 9 a.m. must re-explain when the call arrives at noon. Plura’s Stateful Conversation Database keys every interaction, including voice, AI SMS, RCS, and AI webchat, to the same customer token such as phone number, email, or ID. Every channel inherits the full memory of prior touchpoints, including pricing offers, objections, qualification status, and sensitive-data redactions.

Carrier-level compliance controls address eight production problems that API-reseller stacks cannot solve at origination:

• Real-time DNC (Do Not Call) scrubbing against federal and state registries before every dial²
• TCPA (Telephone Consumer Protection Act, 47 U.S.C. § 227) consent records that are timestamped and immutable²
• Quiet-hours enforcement via automatic time-zone detection on every outbound contact
• SHAKEN/STIR caller ID authentication on every outbound call
• Branded caller ID issued directly through Plura’s FCC-licensed carrier
• HIPAA-aligned encryption, access controls, and audit logging for protected health information¹
• SOC 2 Type II controls with continuous monitoring and third-party audits¹
• ISO certification and GDPR coverage for operations with European exposure¹

These controls operate as first-class layers of the platform, not third-party add-ons. Operators using Plura’s managed workflows and conversation intelligence tools can export audit-ready records in one click. Customers remain responsible for their own regulatory obligations and certifications, and Plura provides infrastructure that supports that posture.

Run your numbers through Plura’s ROI calculator and see the cost comparison against your current contact-center model.

Frequently Asked Questions

How long does it take to go live with an AI voice agent on Plura?

Deployment timelines depend on conversation complexity. A straightforward inbound qualification flow typically goes live within days. A multi-step intake, such as a 25-question health-history survey with conditional routing, runs closer to one to two months because the workflow logic requires design, testing, and validation against real call scenarios.

Plura’s onboarding sequence covers a discovery audit, intake of sample calls and existing scripts, an overnight build of a dynamic conversation mockup, a review meeting, engineering build of the production workflow, a pilot on a subset of real calls, and full go-live. Every annual contract includes a 90-day opt-out window if the deployment is not delivering.

What compliance frameworks does Plura support?

Plura supports compliance with SOC 2, HIPAA, ISO certification, GDPR, SHAKEN/STIR caller ID verification, TCPA, and DNC requirements.¹ Every outbound contact is checked against federal and state DNC registries before dial. Consent records are timestamped and immutable, and quiet-hours rules enforce automatically through time-zone detection.

HIPAA-aligned encryption and audit logging cover protected health information across all channels. Customers are responsible for their own regulatory obligations and certifications, and Plura provides infrastructure that supports that posture. Operators with specific compliance questions should consult qualified legal counsel.

Which systems can Plura connect to?

Plura connects to 50+ tools across CRM, calendar, attribution, document signing, payment processing, data enrichment, and collaboration categories. CRM integrations include HubSpot, Salesforce, and Zoho.⁴ Calendar integrations include Cal.com, Calendly, and Google Calendar. Automation integrations include Go High Level, Make, and Zapier.

The AI Lead Intelligence layer enriches every lead in real time from 30+ data sources during the conversation across voice, SMS, RCS, and webchat, so qualification happens at the moment of contact rather than in a downstream batch job. The full integration directory is at plura.ai/integrations.

How does Plura handle calls flagged as spam?

Spam labels sit at the carrier level and require a carrier-level solution. Plura issues branded caller ID directly through its FCC-licensed carrier, so calls present with the company’s name rather than “Spam Likely” or an unfamiliar number. SHAKEN/STIR authentication runs on every outbound call, and the destination carrier uses that signal to evaluate origination.

Most AI voice tools cannot address this because they rent the carrier layer from a CPaaS provider and inherit that provider’s caller ID reputation. Plura’s spam-label remediation operates inside the platform at origination.

Plura’s ROI calculator at plura.ai/calculator models the cost comparison against your current contact-center economics using your own inputs, including number of agents, hourly rate, talk utilization, and volume. The illustrative default scenario, a 15-agent operation at $20 per hour with standard overhead and 40% talk utilization, costs $60,000 per month. Replacing that team with Plura at $15 per hour and 100% talk utilization drops the monthly cost to $14,400, with 30-day savings of $45,600 and 12-month savings of $547,200.³

Across the platform’s customer base, Plura reports 3x average ROI in 90 days, 47% average pipeline growth, and 90% faster lead-response time.³ Individual results depend on deployment configuration, call volume, and vertical.

What happens when the AI voice agent does not know the answer or a caller goes off-script?

Plura’s workflows include explicit guardrails at every conversation node. When a caller’s response falls outside the workflow’s defined paths, such as an unfamiliar request, a sensitive disclosure, or a high-stakes objection, the agent escalates. The system can warm-transfer the call to a U.S. agent, flag the conversation in the Unified Inbox, or route to a designated escalation queue.

Negotiation nodes carry BATNA (Best Alternative to a Negotiated Agreement) floors and ceilings, which define the boundaries inside which the AI can negotiate, so the agent does not improvise on outcomes that matter. Sensitive data such as protected health information, payment data, and personally identifiable information is redacted at the field level and routed through HIPAA-aligned channels. The AI does not freestyle on consequential decisions.

Updated August 23, 2026

---