Voice is the interface humans were built for. We learn it before we can read, we use it without thinking, and it carries far more than words — timing, hesitation, tone, the willingness to be interrupted. So when people imagine talking to an AI, they imagine talking. It feels obvious. That obviousness is a trap, because the gap between "feels natural" and "is natural" is enormous, and it is a systems-engineering gap, not a model gap.
At Althea we build voice-first agents for healthcare — patient access, care coordination, adherence, the unglamorous but essential work of helping people get the care they need. Healthcare is a high-stakes proving ground: the tolerance for errors is low, the conversations are emotionally loaded, and the person on the other end is often anxious or unwell. If voice works there, the engineering behind it is real.
Latency is a feeling, not a metric
The first thing you learn building voice agents is that latency is not a number on a dashboard — it is an emotional experience. In a chat interface, a one-second delay is invisible. In a conversation, it is a held breath. Humans are exquisitely tuned to conversational timing; a pause that is a few hundred milliseconds too long reads as hesitation, confusion, or rudeness. We notice it before we can name it.
This reframes the entire engineering problem. You are not building a pipeline that produces a correct response; you are building one that produces a well-timed response, where timing is part of correctness. And the pipeline is deep: capture audio, detect speech, transcribe, understand, reason, decide, generate language, synthesize speech, and play it back — all while the human might start talking again at any moment.
The model's answer can be perfect and the interaction can still fail, because the answer arrived half a second too late or stepped on the person mid-sentence. In voice, when you respond is part of what you respond.
Every stage adds delay, and the delays don't just add — they interact. You end up doing things that look strange from a pure-accuracy standpoint: streaming partial results, predicting turn boundaries before the person has finished, beginning to formulate a response before you've heard the whole sentence, and being ready to gracefully abandon it. Optimizing the average is not enough; the tail latencies are what people actually feel.
This leads to a counterintuitive design principle: it is often better to be fast and recoverable than slow and certain. A system that commits early to a response and is prepared to abandon it when the person keeps talking feels more natural than one that waits for complete certainty before saying anything. Humans don't wait for certainty either — we start sentences before we know how they'll end, and we repair on the fly. Building a voice agent that feels human means accepting that some of your work will be thrown away, and engineering for that gracefully rather than trying to eliminate it.
Turn-taking is the real problem
Most of the genuinely hard problems in voice are not about understanding language. They are about the choreography of a conversation:
- Interruptions. A person cuts in. The agent has to stop talking immediately, figure out what was said over its own voice, and recover — without losing the thread.
- Backchannels. "Mm-hm," "right," "okay" are not turns. Treating every utterance as a request to respond makes an agent feel deaf to the rhythm of the conversation.
- Silence. Sometimes silence means "I'm thinking," sometimes it means "your turn," sometimes it means the line dropped. The agent has to disambiguate in real time.
- Repair. When something goes wrong — a misheard name, a wrong assumption — humans repair conversations fluidly. An agent that can't repair gracefully feels broken even when its language is flawless.
None of these are language-modeling problems. They are real-time control problems, and they live in the orchestration layer that sits around the models, not inside them. This is where I spend most of my attention, because this is where voice agents actually succeed or fail.
The hardest of these is the interruption, because it forces the system to do several things at once under hard time pressure. The moment a person starts speaking over the agent, the agent has to stop producing audio almost instantly — a delay of even a few hundred milliseconds here feels like being talked over by someone who isn't listening. But stopping is the easy half. The system also has to figure out what the person said while the agent's own voice was still playing, decide whether what they said was a real interruption or just a backchannel, and then recover the conversational thread from wherever it was abandoned. Get any part of that wrong and the interaction feels broken, no matter how good the underlying language understanding is.
Emotion is signal, not decoration
In healthcare especially, the how of what someone says carries as much information as the words. A patient who is frightened, confused, or in pain communicates that through pace, pitch, and hesitation long before — and sometimes instead of — saying it explicitly. An agent that flattens all of this into a transcript has thrown away exactly the signal a good human operator would attend to most.
This cuts in two directions, and both matter. On the input side, prosody and timing are evidence about the person's state that should inform how the agent behaves — when to slow down, when to reassure, when to stop pushing through a script and escalate to a person. On the output side, the agent's own voice has to carry appropriate warmth and pacing, because a clinically-correct sentence delivered in a flat, hurried tone can land as cold or dismissive to someone who is already anxious. Voice is a higher-bandwidth channel than text precisely because it carries this, and a system that treats it as "text with extra steps" leaves most of the bandwidth on the floor.
A voice agent is a closed-loop system
It will not surprise anyone who has read my other writing that I see a voice agent as a closed loop. It senses (audio), represents (what was said, what's meant, what state the conversation is in), reasons (what to do), and acts (speaks or takes an action) — and then the human responds, and the loop closes. The person's next utterance is shaped by what the agent just did, which means errors compound and recoveries matter more than any single perfect turn.
Building these systems well means designing for that loop explicitly:
- Longitudinal memory so the agent carries context across a conversation and across calls, rather than starting cold each time.
- Grounding and constraints so that in a clinical setting the agent acts within safe, well-defined bounds and doesn't improvise where it shouldn't.
- Human-in-the-loop control so that when confidence drops or stakes rise, a person can step in — and the handoff is smooth rather than jarring.
- Feedback that closes the loop, so the system improves from real interactions instead of staying frozen at deployment.
The closed-loop framing also explains why a single bad turn is rarely fatal but a bad recovery is. In a one-shot system, an error is an error. In a conversation, an error is just the start of a repair sequence — and humans are remarkably forgiving of a system that misunderstands once but recovers gracefully, and remarkably unforgiving of one that misunderstands and then plows ahead as if nothing happened. So we spend disproportionate effort on what the agent does after something goes wrong: noticing the misunderstanding, acknowledging it, and getting back on track the way a competent human would.
Why a phone call, and not an app
It's worth being explicit about why we lean so hard into voice when a chat interface would be so much easier to build. The reason is who healthcare actually has to reach. The populations that consume the most care and have the worst access are disproportionately older, sicker, less comfortable with apps, and more likely to be navigating a moment of stress or fear when they reach out. For a great many of these people, a phone call is not a fallback channel — it is the channel. It's the thing they'll actually use.
An app puts the burden of adaptation on the person: download this, create an account, learn this interface, navigate these menus. A phone call puts the burden on the system, which is where it belongs. The person does the most natural thing in the world — they talk — and the engineering absorbs the difficulty of meeting them there. That inversion is the entire ethical and practical case for voice in this domain. The hard interface for the builder is the easy interface for the human, and in healthcare the human in question is often the one with the least slack to spare.
This is also why I don't think of voice as a transitional technology we'll graduate out of once some better modality arrives. For human-to-system interaction in moments that are stressful, hands-busy, or simply human, speaking will keep being the lowest-friction thing a person can do. The interface that asks the least of the person tends to win, and for an enormous range of situations that interface is the voice we've all had since before we could read.
What we actually measure
Because latency is a feeling and conversation quality is hard to reduce to a single number, the metrics that matter for voice are not the ones you'd reach for first. Word error rate on the transcription, for instance, is necessary but badly incomplete — an agent can transcribe every word correctly and still feel terrible to talk to because its timing is off, or wonderful to talk to despite mishearing a few words it recovered from gracefully.
So we pay attention to things that are closer to the lived experience of the conversation. How long, in practice, between the person finishing and the agent starting — not on average, but at the tail, because the slow turns are the ones that get remembered. How often the agent talks over someone, and how fast it stops when it does. How often a conversation requires a repair, and how often those repairs succeed. How often the agent correctly recognizes that it's out of its depth and hands off to a human cleanly, versus pushing ahead and digging a hole. And, underneath all of it, whether the person actually accomplished what they called to do — because a delightful conversation that fails to get someone their appointment is still a failure.
These are harder to measure than a single accuracy score, and that difficulty is itself the point. Optimizing the easy metric while the hard ones quietly degrade is one of the most common ways a voice system can look good on paper and feel broken in practice. The discipline is in refusing to let the measurable stand in for the real.
Why I think voice is worth the difficulty
It would be easier to build for text. Text is forgiving, asynchronous, and tolerant of latency. But text is also a narrow channel, and for the populations healthcare most needs to reach — older patients, people in distress, people who simply find a phone call easier than an app — voice is not a nicety. It is access.
That is why voice, for all its difficulty, is the interface I keep coming back to. It is the most natural way for a human to interact with an AI system, and "most natural for the human" is exactly the right thing to optimize for, even when — especially when — it is the hardest thing to build. The difficulty isn't a reason to avoid voice. It's the moat. The teams willing to do the unglamorous real-time-systems work that voice demands will build interfaces that feel effortless to the people using them — and "feels effortless" is, in the end, the entire point of an interface.