Call a business today and there is a reasonable chance the thing that answers is software. Not a phone tree asking you to press one. Something that greets you, listens while you explain a problem in your own words, and books an appointment before you hang up.
That is an AI voice agent. The concept takes one sentence to explain. The engineering takes the rest of this article, because everything interesting about voice agents comes from one constraint: the whole thing has to happen in about a second.
The short answer
An AI voice agent is software that holds a spoken conversation over the phone or another voice channel. It converts your speech to text and works out what you want. Then it decides what to do and speaks a response back, in a continuous loop.
It differs from the phone menus that came before it in one important way. A menu makes you conform to its structure by pressing numbers or saying a keyword it was waiting for. A voice agent handles you saying "yeah hi, I need to move my Thursday appointment, actually make it next week if you have anything in the afternoon" and gets it right.
The pipeline, and why every stage is a compromise
Most voice agents chain three technologies in real time, sitting on top of a fourth that connects the call.
Speech to text converts the caller's audio into text as they speak. Not after they finish. As they speak, streaming partial results, because waiting for a complete utterance costs time the budget does not have.
A language model reads that text, works out intent, pulls the details that matter such as dates and account numbers, checks whatever systems it needs, then decides what to say or do.
Text to speech turns the reply back into spoken audio.
Telephony carries the call. This is the part people forget until it breaks. Twilio's ConversationRelay, one common option, handles the speech conversions and session management, plus low-latency communication with your application, and connects to your logic over a WebSocket: your application receives transcribed caller speech as structured messages and sends back text, which the service speaks to the caller.
Newer speech-to-speech models compress the middle. OpenAI's Realtime API supports building agents that listen, reason, and speak while calling tools, without the separate transcription and synthesis steps, and OpenAI positions Realtime sessions as best for live audio that needs low latency. Fewer conversions means less delay. The tradeoff is that you lose the explicit text boundary between stages. That boundary is convenient, because it is where a chained pipeline naturally puts logging and redaction. Speech-to-speech platforms do provide transcripts and controls, so this is a question of where you place them rather than whether you can.
The one second rule explains everything else
Here is the number that governs the whole design. AssemblyAI puts the target for the full round trip at around one second, which is about where a conversation stops feeling like a walkie-talkie.
One second, for audio to reach the server, transcribe, reach the model, generate a response, then synthesize speech and travel back. Every architectural decision in a voice agent is someone spending or saving milliseconds inside that budget.
This is why voice agents are meaningfully harder than chat agents doing identical work. A chatbot that takes four seconds to answer feels thoughtful. A voice agent that takes four seconds feels broken, and callers start saying "hello? hello?" into the gap.
It also explains why the demo is never the hard part. Getting an agent to answer one clean question in a quiet room is a weekend. Getting it to hold a conversation with a caller in a parking lot who changes their mind halfway through is the actual work.
Turn-taking is the part nobody demos
The hardest problem in voice is not understanding words. It is knowing when the person has finished saying them.
Early systems waited for silence. Go quiet for a fixed interval and the system assumes your turn is over. Anyone who has paused to think mid-sentence knows how that goes: you get interrupted, you start again, the agent interrupts again, then you ask for a human.
Better systems use neural end-of-turn detection, which reads tonality and pacing, the human signals that mean "I am still going" versus "your turn". AssemblyAI reports roughly 300 milliseconds to reach that decision with its own realtime model, where turn detection ships inside the transcription model rather than as a separate stage.
The other half is barge-in, letting a caller interrupt the agent mid-sentence the way people interrupt each other. Twilio exposes this as configuration rather than a single switch: whether caller speech or keypad input stops playback at all, and how sensitive that trigger is, where high is most responsive and low requires more confident speech before cutting off. It can also filter backchannel when that option is switched on, so an "uh-huh" or "okay" while the agent is talking does not count as an interruption. It is off by default.
Those settings are where a voice agent feels polished or infuriating, and they are almost never mentioned in a sales demo.
What people actually use them for
Three patterns cover most deployments.
Reception and scheduling. The front desk case. This is the most common starting point because call volume is high and the conversations are structured. We do not re-cover it here: our guide to the AI receptionist works through the use case and its economics in depth.
Outbound follow-up. Appointment reminders, confirmations, payment follow-ups, lapsed-customer outreach. Easier than inbound in one respect, because the agent opens with a known purpose rather than an unknown problem. Harder in another: regulation around automated outbound calling is real and varies by jurisdiction, and that is a question for your counsel before it is a question for your engineers.
Support triage. Answering the common questions, gathering the details a human would otherwise spend the first three minutes collecting, and handing off with context attached. The related pattern in text is covered in our guide to the AI customer service agent.
Across all three, the boring detail that decides success is the handoff. An agent that cannot recognise it is out of its depth, and cannot pass a caller to a person with everything already gathered, produces worse outcomes than no agent at all.
Build or buy
There are good off-the-shelf voice products. Most businesses should evaluate them first, and plenty should stop there.
Buy when the workflow is standard. Booking, answering common questions, taking a message, routing by department. If your process resembles other businesses in your category, a configured product is the faster and cheaper path, and paying to construct the same thing from parts is waste. Get quotes and timelines from two vendors before assuming otherwise.
Build when the conversation has to reach into your systems. The line is not conversation quality, because vendors are good at that. It is what happens after the agent understands. If completing the call means checking eligibility in one system and writing to a second, while applying rules that only exist in your team's heads, you need something that can be programmed rather than configured.
Build when the data path is the constraint. Regulated work changes the question. Where audio is stored, how long transcripts live, which subprocessors touch protected information, whether you can produce an audit trail on demand. Some vendors handle this well. Many cannot, and finding out during a compliance review is expensive.
Build when the agent is one part of a larger system. A voice agent that answers well but drops work at the boundary just relocates the manual step. If the call should trigger a workflow that spans several systems, the voice layer is a component, not the product. That is the same reasoning we lay out in what is an AI agent.
We build the custom side of this, and our first question on a call is whether an existing product already covers your workflow. If it does, that is the advice. The AI voice agents page covers what a build with us involves if it does not.
How to tell a good one from a demo
Vendor demos are built on the happy path. Three questions cut through them.
Ask what the end-to-end latency is on a real call rather than in a lab. Then ask whether it holds when the agent has to look something up mid-conversation. A lookup that takes two seconds blows the budget described above, so ask how the call flow covers that delay.
Ask to hear a call where the caller interrupts or changes their mind, or gives a date in a format nobody planned for. That is where turn-taking and intent handling either hold up or fall apart.
Ask what happens on failure. Specifically, how many attempts before it gives up, and what it passes to the human. Then ask whether the caller has to repeat themselves. A demo that cannot show you the failure path is showing you the wrong thing.
Frequently asked questions
How much does an AI voice agent cost?
Pricing splits by model: products bill per minute or per call, custom builds are priced as projects with an ongoing operating cost. The comparison that matters is total cost against the labor and missed calls you carry today. Our AI receptionist pricing breakdown runs the numbers.
Can callers tell they are talking to AI?
Often yes, and the ones who cannot usually work it out within a turn or two. Chasing indistinguishability is the wrong goal, and disclosure rules for automated calling vary by jurisdiction, so check what applies to you before deciding how the agent introduces itself. Our own preference is a competent agent that says what it is.
What happens when it does not understand?
That is a design decision you make, not something the technology settles. A well-built agent recognises repeated failure, stops trying, then transfers to a person with the conversation so far attached. A badly built one loops. Ask any vendor to demonstrate the failure path, not just the happy path.
Is an AI voice agent the same as an AI receptionist?
An AI receptionist is one application of a voice agent, specifically the front-desk job of answering and qualifying and booking. Voice agent is the underlying technology, which also covers outbound calling and support triage, along with internal use.
Where to start
If you are early and want to know whether this fits your operation, the free 30-minute AI Strategy Call is the fast version. If you already know the workflow and want it mapped and scoped, the Current State Assessment produces a workflow map and a costed plan.
Sources
- AssemblyAI, "AI voice agents: what they are and how they work". Documents the speech-to-text and text-to-speech pipeline around a language model, puts the natural-conversation round trip at around one second, and describes neural end-of-turn detection reading tonality and pacing to reach a decision in roughly 300 milliseconds.
- Twilio, "ConversationRelay". Documents the telephony layer handling speech conversions and session management, with low-latency communication to your application over a WebSocket, plus the interruptible and interruptSensitivity controls and backchannel filtering described above.
- OpenAI, "Realtime API". Documents speech-to-speech agents that listen, reason, and speak while calling tools, and states that Realtime sessions are best for live audio that needs low latency.