The 800-millisecond budget behind a voice agent that sounds normal

Every voice-agent demo sounds fine. The demo runs on a laptop microphone, wideband audio, one speaker, no queue, no CRM. Production runs on an 8kHz phone line into a system that has to look something up before it can answer.

The gap between those two experiences is not intelligence. It is 800 milliseconds, and it is already spent by the time anyone argues about which model to use.

800ms is a perception threshold, not a performance goal

Human conversational turn gaps average around 500ms. Past roughly 800ms a listener stops hearing a pause and starts hearing a problem — a bad line, a person who did not understand, a machine.

So the working production target is a median voice-to-voice latency of 800ms, measured from the caller’s last syllable to the first audible syllable back, with p95 under 1,500ms. That distinction matters more than the median. Above about 1.2s at p50, callers begin talking over the agent, which triggers the interruption logic, which makes the agent stop and restart — and the call degrades faster than the latency number alone suggests.

The published Daily/Pipecat breakdown splits the budget roughly like this:

• mic capture and network in: 50–100ms
• voice activity detection and turn commit: 50–100ms
• speech-to-text finalisation: 100–200ms
• LLM time to first token: 80–100ms
• text-to-speech time to first byte: 80–100ms
• playback buffering out: 50–100ms

Add the midpoints and you are at the ceiling. There is no slack in that budget for anything you have not listed.

The codec spends part of it before your stack sees the audio

Telephony audio is G.711 µ-law, 8kHz, mono. Twilio Media Streams delivers it as base64-encoded raw µ-law over a WebSocket with no file headers, and allows exactly one bidirectional stream per call — so if you were planning a second stream for a parallel transcription path, you cannot have one. DTMF is inbound-only on that stream.

Narrowband is not just a latency question, it is an accuracy question that becomes a latency question. Fricatives collapse, digit strings blur, and the ASR returns lower-confidence hypotheses that your logic then has to confirm — which costs a whole extra turn, roughly 3–4 seconds, not 40 milliseconds. The cheapest way to blow a latency budget is to save 50ms in the pipeline and spend it on “sorry, could you repeat the last four digits?”

Endpointing is the largest discretionary line, and its default is wrong

Deciding the caller has finished is the single most-tuned parameter in voice AI, and it is tuned against a genuine tradeoff rather than optimised.

Deepgram’s endpointing parameter defaults to 10ms of silence. Nobody ships that: at 10ms the agent interrupts anyone who breathes. Production stacks override it and pair it with utterance_end_ms and vad_events. Deepgram’s Flux model, built specifically for voice-agent turn detection, exposes the decision directly as eot_threshold (0.5–1.0, default 0.7), eager_eot_threshold (0.3–0.9) and eot_timeout_ms (default 5000).

Typical end-of-utterance delay lands at 200–500ms. Below about 250ms the agent starts cutting people off mid-sentence, particularly anyone who pauses to read a number off a bill. Above 500ms the whole conversation feels slow no matter how fast the model is.

This is where most of the tuning effort goes, and it is worth saying plainly: you are choosing between an agent that interrupts and an agent that lags. There is no setting that avoids both.

One sequential CRM lookup costs more than the model

Time to first token is 80–150ms without a tool call and 200–500ms with one. Each additional sequential lookup into a CRM or billing system adds 300–800ms.

Two chained lookups — identify the customer, then fetch their invoice — will overrun the entire voice-to-voice budget on their own, and no amount of model selection recovers it. The fixes are architectural, not model-level: prefetch on the calling number at connect so the account is already in context before the first turn, run independent lookups in parallel rather than in sequence, and let the agent speak a real acknowledgement while the call is in flight instead of buffering silence.

That last one is the difference between a system that feels fast and a system that is fast. They are not the same thing, and only one of them is what the caller is grading.

Bulgarian pays for a stage the English stack does not have

Here is the specific constraint that changes the architecture rather than the tuning.

Deepgram Flux — the turn-detection-native model — supports English plus nine languages: Spanish, French, German, Hindi, Russian, Portuguese, Japanese, Italian and Dutch. Bulgarian is not among them. Bulgarian is supported by Nova-3 and Nova-2, and by ElevenLabs Multilingual v2, Flash v2.5 and v3 on the synthesis side.

The consequence is structural. An English agent can let the ASR own endpointing. A Bulgarian agent cannot, so it runs Silero VAD plus a separate semantic turn model alongside Nova-3 — an extra component in the hot path, with its own inference time, on the tightest stage of the budget. Plan the Bulgarian turn-detection stage at the top of the 50–100ms band, not the bottom, and give back the difference somewhere else.

Word error rate compounds it. Published ranges run 5–8% on clean English wideband, 8–15% on English telephony at 8kHz, and 12–25% on Bulgarian telephony once you include surnames and alphanumerics. A 20% WER on a ten-digit ЕГН is not a transcription problem, it is a business-process problem.

Which is why identification should run on the keypad, with ASR as fallback and a read-back confirmation loop — never on ASR alone. Load keyterm boosting with the actual product and tariff names and a Bulgarian surname list. And baseline WER on the client’s own recordings before anyone quotes a containment number, because the containment figure is downstream of the WER figure and quoting it first is guessing.

Barge-in is a buffer-flushing problem, and “да” makes it worse

When a caller interrupts, the agent has usually already streamed several seconds of audio into a buffer downstream. Stopping generation does not stop playback. You have to explicitly drop what is queued — Twilio’s clear message, LiveKit’s interrupt() — or the agent talks over the person who just interrupted it.

Then there is echo. On speakerphone without acoustic echo cancellation, the agent hears its own voice, treats it as speech, and interrupts itself. Talk-over rate above about 3% of call duration is the signal that one of these two things is broken.

The Bulgarian wrinkle is real and, as far as we can tell, unsolved by the standard controls. English backchannel is lexically distinct: “mhm” and “uh-huh” are never answers to a question, so ignoreBackchannel and word-count heuristics like LiveKit’s min_words can filter them safely. In Bulgarian, „да“ is both the most common backchannel and the affirmative answer to every confirmation prompt the agent asks. A filter that ignores it drops real answers; a filter that honours it lets every acknowledgement cut the agent off. The workable compromise is contextual — suppress single-token acknowledgements everywhere except immediately after a confirmation prompt — and it needs measuring on your own calls, not assuming.

Where this budget stops being the interesting problem

Latency is the constraint that decides whether the agent is usable. It is rarely the constraint that decides whether it is worth building.

Twilio’s outbound rate to Bulgarian mobile numbers is roughly $0.2109 per minute against $0.0331 to landline. Termination, not inference, dominates the unit cost of a Bulgarian voice agent. Forty milliseconds shaved off TTS is engineering satisfaction; a minute of average handle time removed is the business case.

And there is a rule worth writing into the runbook before launch: when p95 voice-to-voice crosses threshold under load, the agent routes to the queue rather than delivering three-second pauses. A hold time is a known experience. An agent that has become slow is a broken one.

We should be clear about what this article is. We have not yet shipped a Bulgarian voice agent — our delivered work is a European car marketplace, a platform for an AI automation agency, the Ministry of Education and Science dictionary at beron.mon.bg, and email-marketing automation for a beauty brand. This is the budget we would hold a build to, and the first thing we would measure on your recordings. It is not a case study, and we would rather say so than let you find out later.