Building voice AI that works in a noisy environment

When a voice AI system fails in a noisy environment, the instinct is to look for a better speech-recognition model. That is usually the wrong first move. In our experience the order of leverage is: microphone and placement first, then echo cancellation and endpointing, then the recognizer, then the dialogue design — and underneath all of it, a test set recorded in the real environment, because a system tuned on quiet audio tells you nothing about how it behaves next to a conveyor belt.

This article walks the pipeline stage by stage, explains how noise breaks each one, and gives a test methodology you can actually run.

The pipeline, and where noise gets in

A conversational voice system is a chain:

microphone -> echo cancellation -> noise suppression -> VAD / endpointing
           -> streaming ASR -> language model / dialogue logic -> TTS -> speaker

Noise does different damage at each link:

Errors compound down the chain, which is why fixes at the top are worth more than fixes at the bottom.

Capture: the cheapest decibels you will ever buy

Signal-to-noise ratio is set at the microphone. No model recovers information that was never captured.

Distance dominates. Speech level falls off quickly with distance while ambient noise stays roughly constant. A headset microphone a few centimeters from the mouth will beat a far-field device across the counter by a margin no software closes. If the use case allows a headset or a handheld push-to-talk device, take it.

Directionality helps. A cardioid or boom microphone pointed at the speaker rejects off-axis noise. For far-field devices, a microphone array with beamforming does the same electronically. Most conferencing-grade array hardware ships with beamforming on board; use it rather than reinventing it.

Watch the capture path. Common problems that masquerade as "the ASR is bad":

• Clipping from input gain set too high. Clipped audio is permanently distorted.
• Automatic gain control pumping up the noise floor during pauses.
• A telephony leg that narrows audio to 8 kHz. If any leg is narrowband, test with narrowband audio.
• Bluetooth headsets dropping to a low-bitrate hands-free profile when the microphone is active.
• An operating system or browser applying its own processing on top of yours. In WebRTC, echoCancellation, noiseSuppression, and autoGainControl are getUserMedia constraints; decide deliberately which are on.

Push-to-talk is underrated. In industrial settings, a physical button removes the hardest problem in the pipeline — deciding when someone is speaking to the system — entirely.

Echo cancellation and barge-in

If the agent speaks through a loudspeaker and users can interrupt it (barge-in), the microphone hears the agent's own voice. Without acoustic echo cancellation (AEC), the system detects its own speech as a user interruption, or transcribes itself.

AEC works by taking the audio being played as a reference signal and subtracting its estimated echo from the microphone input. Practical points:

• Use a proven implementation. The WebRTC audio processing module (AEC3) is the common choice and is built into browsers and most WebRTC SDKs; Speex DSP is a lighter alternative for embedded use.
• AEC needs the reference signal aligned in time with the capture. Variable output latency, such as Bluetooth speakers, degrades it badly.
• Headsets avoid the problem by removing the acoustic path.
• Add a semantic check on barge-in: require a minimum duration of detected speech before stopping playback, so a cough or a door slam does not halt the agent mid-sentence. Some teams also ignore very short transcripts such as a lone "okay" during playback.

Noise suppression: measure before you trust it

Open-source suppressors such as RNNoise (a small recurrent network from Xiph, light enough for real-time use on modest CPUs) and DeepFilterNet (a heavier, full-band deep-filtering model), plus commercial SDKs such as Krisp, NVIDIA Maxine, and Picovoice Koala, can make noisy audio sound dramatically cleaner to a human listener.

That is not the same as making it easier for a recognizer.

The only reliable approach is an A/B on your own test set: run identical recordings through the recognizer with and without suppression, and compare word error rate per noise condition. Typical findings: suppression helps at very low signal-to-noise ratios and with non-stationary noise like background speech; it helps little or hurts at moderate noise levels. A reasonable design applies it conditionally, or uses suppressed audio for VAD while sending the original audio to the recognizer.

One case where suppression is clearly valuable: when audio also goes to a human, such as a call that may be transferred to an agent.

VAD and endpointing: where conversations actually break

Voice activity detection (VAD) decides which audio frames contain speech. Endpointing decides when the user has finished their turn. In noise, these are the most user-visible failures:

• False starts: noise classified as speech. The agent "hears" something and responds to nothing, or barge-in fires spuriously.
• Never-ending turns: steady background noise keeps the detector above threshold, so the end of turn is never declared and the agent sits silent.
• Clipped turns: the user pauses to think or read a number, and the agent jumps in.

Silero VAD is a widely used open model; its get_speech_timestamps utility exposes the parameters that matter:

from silero_vad import load_silero_vad, read_audio, get_speech_timestamps

model = load_silero_vad()
wav = read_audio("sample.wav", sampling_rate=16000)

segments = get_speech_timestamps(
    wav,
    model,
    sampling_rate=16000,
    threshold=0.6,                # speech probability cutoff; default is 0.5
    min_speech_duration_ms=250,   # ignore blips shorter than this
    min_silence_duration_ms=700,  # "hangover": silence needed to end a segment
    speech_pad_ms=100,            # padding so word onsets are not clipped
)

Tuning guidance:

• Raise threshold in noisy environments to cut false starts, and confirm on your test set that quiet talkers are still detected.
• The silence duration is a direct trade between responsiveness and cutting people off. A short value suits quick command-style exchanges; dictating an address or a part number needs longer. Make it context-dependent: extend it when the agent has just asked for a number.
• Keep generous padding. Clipped first syllables are a disproportionate source of recognition errors.
• Energy-based VADs fail in exactly the conditions this article is about. Use a model-based one.

Several streaming ASR providers and voice-agent frameworks now offer endpointing that considers the transcript as well as the silence, so that "my number is, um" is not treated as complete. It is worth evaluating, with the same test set.

ASR: choose on your audio, and guard against invented text

Choose by testing on your recordings. Published leaderboards are measured on audio that does not resemble your site. Run the candidates — a hosted streaming service or two, and an open model such as Whisper if self-hosting matters — on your own test set, and compare word error rate per noise band. Rankings frequently reorder under noise and accents. Check streaming support as well: Whisper is a batch model, and streaming it requires chunking strategies that add latency and complexity.

Whisper invents text on non-speech audio. This is a well-documented behavior: given silence, music, or noise, Whisper can emit fluent sentences that were never spoken, often phrases resembling video captions. In a voice agent, that is a phantom user turn. Mitigations:

from faster_whisper import WhisperModel

model = WhisperModel("large-v3", device="cuda", compute_type="float16")

segments, info = model.transcribe(
    "turn.wav",
    vad_filter=True,                  # Silero VAD gate: non-speech never reaches the model
    vad_parameters=dict(min_silence_duration_ms=500),
    condition_on_previous_text=False, # stops one bad segment from seeding the next
    no_speech_threshold=0.6,
    log_prob_threshold=-1.0,
    compression_ratio_threshold=2.4,  # catches repetitive, looping output
)

for seg in segments:
    if seg.no_speech_prob > 0.6 and seg.avg_logprob < -1.0:
        continue  # drop likely non-speech
    print(seg.text)

VAD gating is the most effective single measure: do not send the recognizer audio with no speech in it. The thresholds shown are the library defaults; tune them on your data rather than copying them.

Bias toward your vocabulary. Product names, part numbers, and jargon are where noisy recognition fails first. Most hosted services support some form of vocabulary hinting — phrase lists, custom vocabularies, keyword boosting, or speech adaptation, depending on the vendor. With Whisper, the initial_prompt parameter nudges spelling and vocabulary, though it is a soft hint, not a constraint. A post-recognition step that snaps near-misses to a known catalog (fuzzy or phonetic matching against valid SKUs, for instance) is often more dependable than any of these.

Dialogue design: assume the transcript is wrong sometimes

No recognizer is perfect in noise, so the dialogue layer must tolerate errors instead of amplifying them.

• Confirm critical slots explicitly. Quantities, amounts, account numbers, names, and addresses get read back: "That was four-seven-one-one, correct?" Read digits individually. Casual slots do not need confirmation; do not make the agent tedious.
• Use confidence. If the recognizer exposes word-level confidence, a low-confidence digit triggers confirmation while a high-confidence one proceeds.
• Constrain when you can. Validate against check digits, known order numbers, or a customer's actual open items. A recognized number that matches nothing valid is a recognition error, not a customer error.
• Re-prompt usefully. "Sorry, I didn't catch the last four digits" beats "I didn't understand." After two failures, change modality: send a text link, use the keypad, or hand off to a person.
• Tell the language model the input is noisy. A system prompt instruction that transcripts may contain recognition errors, and that it should ask rather than guess when a request is ambiguous or implausible, is a cheap guard against confident wrong actions. Verify its effect on your task-success metric like any other change.

Testing under real operating conditions

This is the section that separates systems that work from systems that demo.

1. Record the real noise. Go to the site with the production microphone and record ambient noise across conditions: peak shift, machinery cycling, vehicles, public-address announcements, nearby conversations. Thirty minutes per condition is a good start.

2. Record clean speech. A scripted set of representative utterances, including the hard ones — numbers, names, domain terms — from speakers covering the accents, ages, and genders of your real users. Transcribe them carefully; this is your ground truth.

3. Mix at controlled signal-to-noise ratios. Combine speech and noise at, say, 20, 10, 5, and 0 dB. This yields a large, labeled, repeatable test set from modest recording effort.

import numpy as np

def mix_at_snr(speech: np.ndarray, noise: np.ndarray, snr_db: float) -> np.ndarray:
    noise = np.resize(noise, speech.shape)
    speech_power = np.mean(speech ** 2)
    noise_power = np.mean(noise ** 2) + 1e-12
    gain = np.sqrt(speech_power / (noise_power * 10 ** (snr_db / 10)))
    mixed = speech + gain * noise
    return mixed / max(1.0, np.max(np.abs(mixed)))  # avoid clipping

4. Also record in place. Synthetic mixing misses two real effects: reverberation, and the fact that people raise and alter their voices in noise (the Lombard effect). Keep a smaller set of utterances actually spoken on site as a reality check on the mixed set.

5. Measure the right things, in bands.

• Word error rate per SNR band and per speaker group. A single aggregate number hides the cliff. The jiwer Python library computes it.
• Slot accuracy on critical fields. A 10% word error rate is tolerable if the errors are in filler words and intolerable if they are in digits.
• Task success rate end to end, per band.
• False-start and cut-off rates for endpointing.
• Invented-text rate on noise-only clips.

6. Find the cliff and decide what to do beyond it. Every system has an SNR below which it stops being usable. Know where that is, compare it with measured site conditions, and design the fallback — push-to-talk, a headset, a touch screen — for when conditions exceed it.

Rerun the whole set on every change to any stage. A test set like this is what makes the noise-suppression A/B and the VAD tuning described above possible at all.

The latency budget

Every noise mitigation costs time, and conversational turn-taking is unforgiving: pauses much beyond a second feel broken. Account for each stage:

The endpointing silence window is often the biggest contributor and it is pure waiting, which is why context-dependent windows and transcript-aware endpointing pay off. Streaming everything — partial ASR results, streamed model output, streamed TTS — matters more than shaving any single stage.

Summary

Work from the top of the chain: get the microphone close, cancel the echo, tune endpointing for the environment, gate the recognizer with VAD, and confirm what matters. Test on recorded site noise at controlled levels, and treat every component claim — including ours — as a hypothesis until your own word error rates agree. If you would like help building that test harness or the pipeline around it, get in touch.