.gif){kind=spacer}
Every voice AI vendor claims to be fast. To actually evaluate that claim, it helps to understand what latency is, what causes it, where it tends to fall apart, and what actually fixes it - rather than taking a single headline number at face value.
What latency actually is
In a voice AI context, latency is the time between when a caller finishes speaking and when they start hearing a response. It's usually measured as time to first token - how long it takes the system to produce the beginning of a response, not the entire sentence.
This matters because voice conversation has a natural rhythm. In human conversation, people respond within a beat - typically well under a second. Anything past that starts to feel like a gap, not a pause for thought.
What actually affects latency
A voice call runs through three separate steps, back to back:
- Speech-to-text - converts what the caller says into text
- The model responds - reads the text and generates a reply
- Text-to-speech - converts that reply into audio
Delay can build up at any one of these steps, and a few specific factors determine how much:
- Where the model runs. If the model isn't hosted on the vendor's own infrastructure, every response has to leave that infrastructure, reach a separate model provider, and come back - a round trip that adds real time before the caller hears anything.
- How the pipeline is wired together. Even with a fast model, poor handoffs between speech-to-text, the model, and text-to-speech can add delay independently of any single component.
- Whether text-to-speech streams. If the system waits for a full response before starting to speak, the caller waits for the whole sentence to be generated. If it streams, the caller starts hearing the answer as it's produced.
- Load. A system that performs well in a quiet demo can behave very differently once it's handling real call volume. This is the factor benchmark numbers hide most often.
The problem: benchmarks and real calls don't match
Most published latency numbers are measured under ideal conditions - a single call, no concurrent load, a clean test environment. That's a legitimate number, but it's not the number that determines what a customer actually experiences.
The gap shows up specifically under real call volume. A system that performs well at a median latency of 600-700 milliseconds in a single-call test can degrade substantially once dozens of calls are running at once - sometimes exceeding 1.5 to 2 seconds at the 95th percentile, the range where a real customer notices something is wrong. That's the number that rarely makes it onto a vendor's homepage.
This is also why raw model benchmarks don't tell you much about how a voice agent will feel in production. A model can score well on general-purpose speed tests and still produce a noticeably laggy call, because the benchmark isn't measuring the network hop, the streaming behavior, the pipeline handoffs, or what happens under real concurrent load.
Why this matters more for voice than for chat
A slightly delayed chat response is barely noticeable - people are used to typing and waiting. Voice conversation has different rules. Human conversation has a rhythm, and a pause past about a second reads as "is this broken," not "it's thinking." Callers don't rationalize the gap. They feel it, and either wait it out, get annoyed, or hang up - and this is exactly the moment where a laggy system under load loses the call it was supposed to handle.
How to actually optimize it - at the model level
Prompt tweaks and pipeline tuning help at the margins, but the structural fix happens at the model-server level:
- Remove the third-party round trip. Running the model on infrastructure you control eliminates the single biggest source of delay in most voice AI stacks.
- Optimize for time to first token specifically, not overall model quality benchmarks that weren't designed with voice conversation in mind.
- Stream text-to-speech properly, so the caller starts hearing a response before the full sentence is generated.
- Test and tune under real load, not just single-call conditions, since that's where most platforms actually fall apart.
Where Astra stands
Astra is NuPlay AI's own LLM, run on our own infrastructure. There's no third-party model call in the loop - a response never leaves to reach a separate provider and come back. In production, not a demo, Astra delivers sub-1-second, end-to-end voice latency, measured in real production traffic rather than a single-call test environment.
What to ask before you believe a latency claim
Before taking any vendor's latency number at face value, ask a few direct questions:
- Is this measured end-to-end, or just one part of the pipeline?
- Was it measured under real call volume, or a single, quiet test call?
- Does the model run on your own infrastructure, or does every response leave it?
A vendor who can answer all three clearly is worth taking seriously. A vendor with only a headline number is worth a second look.
