Voice agents in noisy environments (Drive-Thrus, Contact Centers, Field)

If you are deploying a voice agent into a drive-thru, a contact center floor, a service van, or a self-service kiosk, three settings decide whether it works. voice_focus picks the noise model — near-field for headsets and phones, far-field for rooms and open spaces. voice_focus_threshold sets how hard that model works, from 0.0 to 1.0, defaulting to 0.7 on Universal-3.6 Pro Realtime. And mode decides how much response time you are willing to spend on transcript quality: min_latency, balanced, or max_accuracy.

Those are three separate dials, not one. The first two are fixed when you open the connection; the third you can change mid-call. The rest of this post is about which combination fits which deployment, why that connect-time constraint matters more than it sounds, and what to do about the part that actually breaks in noise: entity accuracy.

Noise does not break words, it breaks entities

The intuitive model of noise is that it degrades everything a little. That is not what the data shows, and it is why teams misdiagnose the problem.

Common words survive background noise reasonably well, because the model has enormous linguistic context to fall back on. If half of “can I get a large” is masked by an engine, the sentence still resolves. Entities have no such safety net. A name, a street address, an order number, a phone number, a confirmation code — these are, by construction, sequences the model cannot predict from context. When the acoustics get hard, that is where the errors concentrate.

The Pipecat open STT benchmark, which uses real agent conversations rather than read speech, shows the gap clearly. Universal-3.5 Pro Realtime posts a 6.99% word error rate and a 15.31% entity error rate — entity errors run more than double the word error rate on the same audio. Break that down further and the pattern holds: 16.92% on names, 6.28% on places, 3.55% on phone numbers. For comparison on the same benchmark, Deepgram Flux posts 15.58% WER and 50.50% entity error, ElevenLabs Scribe v2 posts 9.76% and 39.70%, and Google Chirp3 posts 9.04% and 21.51%. The full table is on the benchmarks page.

The operational point: if you are monitoring your deployment on word error rate alone, a noisy site can look acceptable while it is failing on every order number. Instrument entity accuracy separately, on the entity types your product actually depends on.

The three dials, and which one you can still change mid-call

Before the scenario table, the constraint that shapes how you use it.

voice_focus selects the noise model. near-field suits audio captured close to the speaker — headsets, handsets, a phone held or clipped near the mouth — where the dominant problem is other people talking nearby. far-field suits audio captured at a distance, where reverberation and distance attenuation dominate: rooms, kiosks, drive-thru lanes, conference tables.

voice_focus_threshold is the strength dial, from 0.0 to 1.0, defaulting to 0.7. Higher values suppress more aggressively. It requires voice_focus to be set. This is the parameter that resolves the tradeoff every noisy deployment runs into — suppress too little and background speech leaks into the transcript, suppress too much and you clip the quiet or distant speaker you actually wanted. Most deployments should start at the default and only move once they have real audio to judge against.

Both are set at connect. Neither is accepted in an UpdateConfiguration message — so in practice you are choosing the noise model when you open the connection, not adjusting it turn by turn. That is the design constraint worth planning around. If your kiosk sits in a lobby that is quiet at 7am and loud at noon, you either pick a setting that holds across the day or cycle the session when conditions change. For a drive-thru, where conditions vary by vehicle rather than by hour, it means picking a setting that holds for the worst realistic case rather than the average one.

mode is the latency/accuracy control — min_latency, balanced (the default), or max_accuracy — and unlike the two above, it is freely mutable mid-session with an UpdateConfiguration message. That asymmetry is genuinely useful. You can run a call on balanced for the conversational parts and switch to max_accuracy for the stretch where the customer reads out a confirmation number or an address, then drop back. You cannot do the equivalent with the noise model. Adapt accuracy on the fly; pick the noise model once, correctly.

near-field vs far-field vs max_accuracy: which setting for which environment

The mapping below is our recommendation rather than documented canon — the parameters and their ranges are documented, but which value suits which room is engineering judgment. Treat the thresholds as starting points to tune against your own recordings.

Two notes. First, balanced is the default mode and voice_focus is off unless you set it, so an unconfigured session is running no noise model at all in balanced mode — which is exactly wrong for a drive-thru. Second, Voice Focus is a $0.10/hr add-on on top of the Universal-3.6 Pro Realtime streaming rate, with no concurrency fees or per-agent subscriptions.

When to move off 0.7

Do not tune the threshold from intuition. Record twenty real interactions at the site, transcribe them at the default, and read the failures. If background speech is appearing in transcripts as though the customer said it, raise the threshold — higher values suppress more aggressively. If the customer’s own words are dropping out — especially quiet speakers, children, or anyone standing further back than the design assumed — lower it. Those two failure modes look completely different in a transcript, which is what makes this dial tunable at all rather than a matter of taste.

The latency budget, by layer

“How fast is it” is two questions, and blending them into one range is how teams end up with a budget that does not add up.

Streaming speech-to-text returns transcripts continuously as the audio arrives, finalizing at the end of each turn. That is the transcription layer alone: audio in, finished text out. End-of-turn detection does not run on a silence timer — when a speaker pauses, the model evaluates what has been said so far to judge whether the turn is complete. That matters more in noise than it sounds, because a silence-only detector in a loud room either fires early on a background gap or never fires at all. Turn detection exposes a minimum and maximum turn silence and a voice-activity threshold, all tunable — check the streaming turn-detection documentation for the current defaults rather than hard-coding values from a blog post, ours included, because they are set by the mode preset and they change again when speaker labels are on.

The full voice agent loop — Universal-3.6 Pro Realtime for speech-to-text, the language model, and AssemblyAI’s Text-to-Speech API for voice output, orchestrated with LiveKit, Pipecat, or your own loop — is the number the customer actually experiences. It stacks the model and the speech synthesis on top of the transcription layer.

Use the right number for the right conversation. If you are building the stack yourself and comparing transcription vendors, measure the transcription layer on your own audio and compare like with like. If you are budgeting the user-perceived pause before the agent starts speaking, the end-to-end figure measured on your own stack is the one that matters. Moving to max_accuracy spends against whichever budget you are tracking, which is why the table above ties the mode to the interaction type — and why the ability to change it mid-session is worth using rather than picking one value for the whole call.

Barge-in is semantic, not volume-triggered

This is the behaviour that separates a voice agent that survives a noisy room from one that does not, and it is easy to miss because it is invisible when it works.

A naive interruption model listens for energy on the input and stops the agent talking whenever it hears some. In a drive-thru or on a contact center floor, that model fails constantly: a passenger laughs, a neighbouring agent finishes a sentence, a door closes, and your agent stops mid-word for no reason the customer can see.

Interruption on Universal-3.6 Pro Realtime is transcript-driven, not energy-driven. The barge-in signal is the SpeechStarted message, and it is only emitted once the model has produced an actual transcript — a passenger’s laugh, a slammed door, or an engine revving does not fire it; a customer saying “wait, stop” does. That distinction is doing a lot of work in exactly the environments this post is about, where the microphone is picking up far more sound than the one person you care about.

If brief back-channels are still cutting the agent off, the documented lever is interruption_delay — how long after speech begins before the first partial is emitted, from 0 to 1000 ms. It follows the mode by default (0 for min_latency, 500 for balanced and max_accuracy). Raise it rather than reaching for vad_threshold.

When a genuine interruption does land, the server tells you clearly: SpeechStarted arrives before the first Turn of the new turn, which is your cue to stop text-to-speech playback and listen. Record where playback stopped if you are logging what the agent actually delivered versus what it intended to say — in a noisy deployment those diverge more often, and the difference is worth having in your analytics rather than discovering it in a complaint.

Fixing the entity problem: context and keyterms

Once the noise model and the mode are set correctly, the remaining headroom in a noisy deployment is almost entirely in entity accuracy. Two levers move it, and they do different jobs.

Transcription context passes the agent’s own spoken reply into the transcription request. When the agent has just asked “what’s the confirmation number on your order?”, the model is no longer transcribing a mumbled string in isolation; it knows what shape of answer is coming. This matters most in exactly the conditions this post is about: short replies, bad acoustics, no surrounding sentence to disambiguate against.

On Universal-3.6 Pro Realtime the field is agent_context: send the agent’s most recent spoken reply in an UpdateConfiguration message, capped at 1750 characters, each time the agent finishes speaking. Each update replaces the previous value, so it always describes the question the customer is answering right now. The static prompt connection parameter does the complementary job of describing the domain for the whole session.

The published measurement comes from the streaming API, across a benchmark of 10,000+ voice agent audio files: passing the agent’s reply as context cut word error rate by 8.9%, rising to 16.4% when a context prompt is added on top. The per-class breakdown is measured on that combined configuration: place-name entities down 30.7%, fabrications down 27.0%, medical entities down 26.2%, name entities down 21.8%, short-utterance errors down 20.5%, and entity errors overall down 11.1%. The short-utterance and place-name numbers are the ones to watch for a noisy deployment — those are the failure modes a drive-thru or a kiosk produces all day, and they are the two categories context helps most.

Context Carryover, a rolling memory across turns, is on by default, so the model retains what has already been established in the conversation rather than treating each turn as cold.

Keyterms — keyterms_prompt, up to 100 strings — is the other lever, and the one people under-use. It biases the model toward an explicit list: your menu items, your product SKUs, your store names, the street names in your service area, the medication names in your formulary. It is included at no extra cost on Universal-3.5 Pro Realtime. If your drive-thru sells something with a proper noun in the name, it belongs in the keyterms list before you do anything else.

The two compose. Context tells the model what this turn is about; keyterms tell it what vocabulary exists in your world. Full parameter documentation is in prompting and keyterms.

Language coverage in noisy environments

Noise and code-switching compound, and multilingual deployments in loud settings are where stacks fail hardest. A bilingual customer at a drive-thru window switching languages mid-order is a genuinely difficult input, and a pipeline that detects language per session cannot handle it.

Universal-3.5 Pro Realtime supports 18 languages — English, Spanish, French, German, Italian, Portuguese, Arabic, Danish, Dutch, Finnish, Hebrew, Hindi, Japanese, Mandarin, Norwegian, Swedish, Turkish, and Vietnamese — with native code-switching inside a single utterance rather than routing between per-language models. On the async code-switching benchmark across five language pairs, Universal-3.5 Pro posts a 7.69 average normalized WER against 8.77 for ElevenLabs Scribe v2 and 12.22 for Deepgram Nova-3 Multilingual.

Streaming caps at those 18 languages. If you have seen a 99-language figure associated with AssemblyAI, that belongs to universal-2, a previous-generation async model at $0.15/hr kept for legacy pre-recorded integrations. It is not a voice agent model and there is no 99-language streaming path.

Configuration is a single decision: pass language_codes as a list to bias toward a known set, or omit it for auto-detection and free code-switching. On mixed-language traffic, omitting it is usually the better choice.

Connecting to Universal-3.6 Pro Realtime

One WebSocket, one endpoint. The handshake has an order to it: open the connection with the noise model and vocabulary settings in the query string, wait for the Begin message, and only then start streaming raw audio frames. Check that the model echoed in Begin matches the speech_model you asked for — unrecognized query parameters are ignored rather than rejected, and that is the most common way a first integration silently misconfigures itself.

import asyncio
import json
import os
from urllib.parse import urlencode

import websockets

# Streaming speech-to-text on Universal-3.6 Pro Realtime.
ENDPOINT = "wss://streaming.assemblyai.com/v3/ws"

# Tuned for a noisy, fixed-location deployment (drive-thru, kiosk).
# voice_focus and voice_focus_threshold are connection parameters;
# they cannot be changed with UpdateConfiguration. mode, prompt,
# keyterms_prompt, interruption_delay and agent_context CAN be changed
# mid-session with an UpdateConfiguration message.
CONNECTION_PARAMS = {
    "speech_model": "universal-3-6-pro",
    "sample_rate": 16000,                  # PCM16, mono
    "encoding": "pcm_s16le",
    "voice_focus": "far-field",            # near-field | far-field (unset = off)
    "voice_focus_threshold": 0.9,          # 0.0-1.0, default 0.7
    "keyterms_prompt": json.dumps([        # up to 100 strings, JSON-encoded
        "Baja Crunchwrap",
        "Willowbrook Plaza",
        "curbside",
    ]),
    # Describe the domain for the whole session.
    "prompt": "Drive-thru ordering at a fast-casual restaurant.",
}

# Sent after Begin. min_turn_silence / max_turn_silence are deliberately
# NOT set; the mode preset picks them, and max_accuracy holds the turn
# open longer while a customer reads out an order number.
INITIAL_CONFIG = {
    "type": "UpdateConfiguration",
    "mode": "max_accuracy",                # min_latency | balanced | max_accuracy
    "interruption_delay": 500,             # ms, 0-1000; raise if back-channels cut in
}


async def run_agent(audio_source, synthesize):
    url = f"{ENDPOINT}?{urlencode(CONNECTION_PARAMS)}"
    headers = {"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}

    async with websockets.connect(url, additional_headers=headers) as ws:
        # 1. Wait for Begin before sending a single audio frame, and check
        #    the model: unknown query params are ignored, not rejected.
        begin = json.loads(await ws.recv())
        assert begin["type"] == "Begin"
        assert begin["configuration"]["model"] == "universal-3-6-pro"

        # 2. Apply the mutable settings for the start of the call.
        await ws.send(json.dumps(INITIAL_CONFIG))

        async def send_audio():
            # Raw PCM16 frames, 50-1000 ms each, paced in real time.
            async for chunk in audio_source:
                await ws.send(chunk)
            await ws.send(json.dumps({"type": "Terminate"}))

        async def receive_events():
            async for message in ws:
                event = json.loads(message)

                if event["type"] == "SpeechStarted":
                    # Only fires once a transcript exists, so noise alone
                    # does not trigger it: stop TTS playback and listen.
                    print("customer started speaking")

                elif event["type"] == "Turn" and event["end_of_turn"]:
                    print("user:", event["transcript"])
                    reply = "Got it. What name is the order under?"
                    synthesize(reply)  # swap in your TTS provider
                    # Tell the model what answer to expect next.
                    await ws.send(json.dumps({
                        "type": "UpdateConfiguration",
                        "agent_context": reply,  # max 1750 chars
                    }))

                elif event["type"] == "Termination":
                    print("audio seconds:", event["audio_duration_seconds"])

        await asyncio.gather(send_audio(), receive_events())


asyncio.run(run_agent(audio_source, synthesize))

The event surface

The client sends raw binary audio plus four JSON messages — UpdateConfiguration, ForceEndpoint, KeepAlive, and Terminate — and the server replies with Begin, SpeechStarted, Turn, Heartbeat, and Termination, plus SpeakerRevision when speaker labels are on.

For a noisy deployment, two of those earn special attention. SpeechStarted tells you when the model believes a new turn has begun, which is the fastest way to diagnose a threshold that is set wrong — if it fires on an empty lane, your suppression is too weak. Turn with end_of_turn: true, and its end_of_turn_confidence, tell you when and how confidently the endpointer closed the turn, which is the signal you want trending on a dashboard per site.

One migration note: the speech model parameter is singular on the streaming path (speech_model) while the async API takes the plural speech_models as a list, and the legacy realtime endpoint is retired and should not appear in your code. Full details in the streaming WebSocket reference.

A fuller walkthrough of the build is in the Python voice agent tutorial, and the Universal-3.6 Pro Realtime launch post covers the model. If you need per-speaker attribution in a room, streaming speaker diarization handles up to 10 speakers with revision, correcting within about half a second of the stream ending.

What good looks like in production

Siro, one of the voice AI teams building on AssemblyAI, runs speech capture in one of the least forgiving acoustic environments there is — real field sales conversations, recorded on a phone in someone’s pocket, outdoors, in cars, in customers’ homes. Their published results: a 90% reduction in customer complaints and support tickets and a 36% improvement in close rate. That is what near-field capture plus disciplined entity handling looks like once it is tuned, and it is a useful counterweight to the assumption that difficult audio is simply a cost of doing business in the field.

The pattern across deployments that work is unremarkable. Pick voice_focus to match the physical microphone placement and get it right at connect, because it cannot be changed mid-session. Start the threshold at the 0.7 default and tune it against recordings from the actual site. Use mode dynamically — balanced for conversation, max_accuracy for the entity-dense stretches. Leave the silence windows to the mode preset rather than hard-coding them. Pass a prompt on every session and agent_context on every turn, keep keyterms current with your real vocabulary, and measure entity accuracy rather than word accuracy. None of that is exotic. It is mostly a matter of not leaving the defaults in place because the demo sounded fine — and knowing which defaults are worth leaving alone.

Frequently asked questions

What is the best setting for a voice agent in a noisy environment?

Set voice_focus to far-field with voice_focus_threshold toward the high end of its range, and mode to max_accuracy, for fixed-location noise like drive-thrus and kiosks where the microphone sits at a distance from the speaker. Use near-field at the default threshold with balanced for headsets, handsets, and phones, which covers most contact center and field traffic. These are three independent dials and all three are worth setting deliberately.

What is the difference between near-field and far-field?

near-field is for audio captured close to the speaker — headsets, handsets, phones held or clipped near the mouth — where the main problem is background speech from nearby people. far-field is for audio captured at a distance — rooms, kiosks, drive-thru lanes, conference tables — where reverberation and distance attenuation dominate. Both are values of voice_focus, which is a $0.10/hr add-on on streaming.

What does voice_focus_threshold do?

It sets how aggressively the selected noise model separates the target speaker from everything else, on a scale of 0.0 to 1.0 — higher values are more aggressive. The default is 0.7. It requires voice_focus to be set. Raise it when background speech is leaking into transcripts as though the customer said it; lower it when the customer’s own words are being clipped, which typically shows up first with quiet speakers or anyone standing further back than the deployment assumed.

Can I change noise settings mid-call?

Not the noise settings. voice_focus and voice_focus_threshold are connection parameters and are not accepted in an UpdateConfiguration message, so you cannot adjust them turn by turn. mode is different: it is freely mutable mid-session with UpdateConfiguration, so you can run a call on balanced and switch to max_accuracy for the stretch where a customer reads out an address or a confirmation number, then switch back.

Does AssemblyAI have a noise suppression endpoint?

No. There is no separate noise suppression endpoint and no standard/aggressive intensity setting. Noise handling is voice_focus on the existing Universal-3.6 Pro Realtime streaming connection, set to near-field or far-field, with strength controlled by voice_focus_threshold. The min_latency / balanced / max_accuracy values belong to mode, a separate control governing the latency-accuracy tradeoff rather than noise.

Will background noise make the agent interrupt itself?

No, because the barge-in signal is transcript-driven rather than volume-triggered. SpeechStarted is only emitted once the model has produced an actual transcript, so ambient noise that carries no speech does not stop the agent. If brief back-channels are still cutting in, raise interruption_delay (0–1000 ms) rather than vad_threshold. When a real interruption happens, the server emits SpeechStarted before the first Turn of the new turn — stop playback and listen.

Why do voice agents get names and order numbers wrong in noisy rooms?

Because entities carry no linguistic context to recover from. Common words survive background noise because the surrounding sentence disambiguates them; a confirmation code or a surname does not. On the Pipecat benchmark, Universal-3.5 Pro Realtime’s entity error rate of 15.31% runs more than double its 6.99% word error rate on the same audio. The levers are transcription context — agent_context, which cut word error rate by 8.9% across 10,000+ files and 16.4% with a context prompt on top, with short-utterance errors down 20.5% and place-name errors down 30.7% on that combined configuration — and keyterms, which bias the model toward your own vocabulary at no extra cost.