Voice Interaction in Vibe Projects: ASR, TTS, Latency Budgets, and Degradation

Two boundaries first. One: the site published three voice-related news pieces this year — Sesame's voice agents, Microsoft's MAI voice agent stack, Simon Willison building a blog feature with voice. Those three are news events; this article is about engineering execution: news tells you what happened, this guide tells you which pitfalls to dodge and how to write the code. Two: the site's "Streaming Output UX" guide (ai-streaming-output-ux-guide) covers the interface experience of "streaming text output"; this one covers the "voice channel" itself — microphone in, sound out, and how to build the pipeline in between.

Voice interaction is the quintessential "demo dazzles, launch crashes" feature in vibe projects. Ten lines of code in a browser can make a page "understand" your speech, which makes it feel done. After launch comes reality: no sound on Safari, a grayed-out button on Firefox, garbled transcription on the subway, a cloud bill that explodes in month one. This guide follows the real engineering order: first decide whether your product needs voice at all, then pick ASR/TTS options, then build the real-time dialogue pipeline, do the cost math, and design degradation — ending with a runnable browser-side voice input demo.

First: does your product actually need voice?

Three questions before writing code. A "no" to any of them means skip voice for now:

• Is the scenario "hands are busy" or "novelty"? Driving, cooking, walking, working with gloves — voice is essential when hands are occupied. Adding a voice button to a vibe project used at a desk is mostly novelty. And novelty retention is brutal: 30% of users try it in week one, 3% remain in week two.
• Is the input "hard to say" or "faster to type"? Voice suits open-ended, fuzzy expression ("find me a weekend-friendly café with cats"); structured input (dates, amounts, account numbers) is faster and more accurate typed — with no risk of one misheard digit.
• Does the environment allow speaking? Offices, commutes, subways — more than half of real usage contexts don't suit speaking aloud. Voice can only ever be an optional channel, never the only one.

Pass all three, then read on. The three sturdiest landing spots for voice in vibe projects: voice dialogue in AI companion/assistant products, voice input for tools (search boxes, notes), and TTS narration for content (article read-aloud, navigation announcements).

ASR voice input: Web Speech API vs cloud ASR

Two routes, judged on four dimensions:

Route 1: Web Speech API (browser-native)

webkitSpeechRecognition / SpeechRecognition is the browser's built-in speech recognition — ten lines and it runs. Its true shape:

• Cost: free. Audio goes to the browser vendor's servers; nothing on your bill.
• Latency: middling. Audio travels over the public internet to vendor servers and back; first-word results typically around a second.
• Accuracy: fine for everyday speech; proper nouns, names, places, accents, and noisy environments degrade it noticeably. Chrome and Safari run different engines underneath — accuracy differs and must be tested separately.
• Offline: not supported. Chrome's implementation sends audio to Google's servers; no network, no recognition.
• Compatibility (2026 status, per MDN): Chrome/Edge work (webkit prefix); Safari 14.1+ (macOS) / 14.5+ (iOS) works (same webkit prefix); Firefox doesn't support it by default (hidden behind a flag); in-app browsers (e.g., WeChat webviews) frequently disable the microphone outright. Electron is the nastiest trap: the API may exist, but the recognition service is the browser vendor's proprietary service, not part of the open-source Chromium build — interface present, zero results.

Verdict: Web Speech API is the fastest route to a working demo — right for MVP validation, low-frequency scenarios, and zero-budget solo projects. Wrong for: production features with accuracy requirements, products with heavy Firefox share, and offline scenarios.

Route 2: cloud ASR (capture audio yourself, call an API)

The browser only captures audio (MediaRecorder or AudioWorklet); recognition goes through a cloud vendor's streaming ASR. The advantage is consistency: accuracy doesn't vary by browser, language and domain models are selectable, and you get control (custom hotwords, punctuation, speaker diarization). The price: you build audio capture and resampling yourself (browsers disagree on sample rates — normalizing to 16kHz/16-bit PCM inside an AudioWorklet is the standard move), manage WebSocket connections, and pay per use.

Selection advice: while daily voice requests stay under ~10k, use Web Speech API to get the flow working; migrate to cloud ASR only when the bottleneck is genuinely accuracy or compatibility. Adopting cloud ASR too early means debugging "audio capture" and "recognition quality" simultaneously — doubling your problem-localization surface.

TTS narration: speechSynthesis vs cloud TTS

The TTS (text-to-speech) comparison is subtler:

• Browser speechSynthesis: the best-supported half of the Web Speech API — Chrome, Edge, Firefox, and Safari all expose window.speechSynthesis. Zero cost, works offline (uses the OS speech engine). Downsides: "system" voices with poor emotion and naturalness; voice lists differ completely per platform, and Chinese voices simply don't exist on some Android devices; Chrome's famous gotcha — getVoices() often returns an empty array on first call because voices load asynchronously; you must listen for the voiceschanged event first.
• Cloud TTS: a generation ahead in naturalness, with SSML (Speech Synthesis Markup Language: controlling pauses, emphasis, rate, pronunciation — whether "123" reads as "one hundred twenty-three" or "one two three"), selectable voices, even voice cloning. The cost is per-character billing — price long-form narration carefully (see the cost section).

Field advice: notifications, reminders, short announcements (<100 chars) are fine with speechSynthesis; content read-aloud, AI assistant responses, and brand-voice requirements deserve cloud TTS. A common combo: ship v1 on speechSynthesis to validate "do users actually tap narration," migrate to cloud TTS once the data exists — don't prepay a cloud bill for an unvalidated need.

SSML deserves its own paragraph

Cloud TTS's SSML isn't a "premium feature" — it's the basics of "saying it right": <break time="500ms"/> controls pauses (essential for addresses and verification codes), <say-as interpret-as="telephone"> reads phone numbers digit by digit instead of as one giant number, <prosody rate="slow"> slows down for elderly users. TTS narration without SSML sounds like a newscast with all the punctuation eaten.

The real-time voice dialogue pipeline: VAD → ASR → LLM → TTS

A "voice assistant" isn't "ASR + TTS glued together" — there's a four-stage pipeline in the middle. Per-stage latency budgets (empirical targets; measure your own chain):

Stage	Latency budget	What it does
VAD (voice activity detection)	200–500ms	Decides "the person started / finished speaking." End-of-speech detection (endpointing) is the hardest parameter to tune: too eager and it interrupts thinking; too lazy and the conversation feels trans-Pacific.
Streaming ASR	300–800ms (first word)	Words appear while speaking. Streaming ASR rewrites interim results ("I want an ap… an Apple phone") — the UI must handle that snap-back.
LLM inference	500–1500ms (first token)	Streaming output is mandatory: waiting for the full sentence before narrating doubles dialogue latency. Feed the LLM's output stream into TTS as it arrives.
TTS first audio	300–800ms	First audio chunk out. With streaming synthesis, synthesis starts the moment the LLM's first clause arrives — no waiting for the full text.

How many milliseconds count as usable? From "user finished speaking" to "assistant starts making sound": <1200ms is excellent (near human-conversation rhythm), 1200–2000ms usable (the pause is felt but acceptable), >2000ms and users start saying "hello? still there?" Optimize in this order: LLM first-token first (streaming + short system prompt + smaller model), then TTS first-audio (streaming synthesis), then VAD parameters last. The root cause of most "the voice assistant is so slow" complaints is "waiting for the LLM's full sentence before calling TTS" — turning serial into a streaming pipeline can halve a sentence's latency.

VAD implementation: the simplest workable browser-side approach is energy detection (compute the volume envelope in an AudioWorklet; above threshold counts as speech); for higher demands, lightweight models like Silero VAD run in-browser/on-device. Many cloud voice-agent APIs ship built-in turn detection, saving you the tuning.

Runnable example: Web Speech API voice input demo

A production-usable voice input needs: feature detection, interim results, errors, auto-stop. Below is a complete, runnable single-file demo (works directly in Chrome/Edge/Safari; requires HTTPS or localhost):

<!doctype html><html><head><meta charset="utf-8"><title>Voice input demo</title></head>
<body style="font-family:sans-serif;max-width:560px;margin:40px auto">
<h2>Voice input demo</h2>
<textarea id="box" rows="4" style="width:100%" placeholder="Tap the mic and speak…"></textarea><br><br>
<button id="mic">🎤 Start speaking</button> <span id="hint" style="color:#666"></span>
<script>
const SR = window.SpeechRecognition || window.webkitSpeechRecognition;
const mic = document.getElementById('mic'), box = document.getElementById('box'), hint = document.getElementById('hint');

// 1) Feature detection: no support means keyboard directly (degradation, not an error)
if (!SR) {
  mic.disabled = true;
  hint.textContent = 'This browser has no speech recognition; please type (works in Chrome/Edge/Safari)';
} else {
  const rec = new SR();
  rec.lang = 'zh-CN';          // Mandarin; zh-HK for Cantonese, en-US for English
  rec.interimResults = true;   // interim results: words appear while speaking
  rec.continuous = false;      // single-utterance mode: auto-stops; fits input boxes
  let finalText = '';

  rec.onresult = (e) => {
    let interim = '';
    for (const r of e.results) {
      if (r.isFinal) finalText += r[0].transcript;
      else interim += r[0].transcript;
    }
    box.value = finalText + interim; // interim rewrites are normal
  };
  rec.onerror = (e) => {
    // common errors, handled honestly: not-allowed (no mic permission),
    // no-speech (heard nothing), network (offline)
    hint.textContent = 'Recognition error: ' + e.error + '; you can type instead';
    mic.textContent = '🎤 Start speaking'; listening = false;
  };
  rec.onend = () => { mic.textContent = '🎤 Start speaking'; listening = false; hint.textContent = ''; };

  let listening = false;
  mic.onclick = () => {
    if (listening) { rec.stop(); return; }
    finalText = box.value ? box.value + ' ' : ''; // append mode, never clobbers input
    try { rec.start(); } catch (err) { return; }   // repeat start() throws; swallow it
    listening = true;
    mic.textContent = '⏹ Stop';
    hint.textContent = 'Listening… it stops automatically when you finish';
  };
}

// 2) TTS narration: mind the voiceschanged async gotcha
function speak(text) {
  if (!window.speechSynthesis) return;
  const u = new SpeechSynthesisUtterance(text);
  u.lang = 'zh-CN'; u.rate = 1.0;
  const pickVoice = () => {
    const vs = speechSynthesis.getVoices().filter(v => v.lang.startsWith('zh'));
    if (vs.length) u.voice = vs[0];
    speechSynthesis.speak(u);
  };
  if (speechSynthesis.getVoices().length) pickVoice();
  else speechSynthesis.onvoiceschanged = pickVoice; // Chrome often returns [] on first call
}
</script></body></html>

Five production details hide in this demo: interimResults=true puts words on screen while speaking (without it, users stare at a silent box doubting their life); continuous=false single-utterance mode fits input boxes — only continuous dictation wants true; error codes told honestly (not-allowed / no-speech / network are the big three); repeat rec.start() throws and must be wrapped; TTS voice pickup must go through voiceschanged. Each maps to a real user complaint.

Mobile permissions and UX: when to ask + barge-in

Permission timing

Microphone permission is a one-shot psychological gate: if the user taps "Don't allow," many browsers silently deny that origin forever — re-applying means digging through settings. So the iron rule: never request microphone permission on page load; only at the exact moment the user taps the microphone button. Tap → system prompt → the user understands "why" (they just tapped the mic) — the highest grant rate. Requesting early or bundling it into onboarding both convert measurably worse.

iOS Safari adds another wrinkle: mic permission is tightly coupled to "is the page in the foreground" — background the page and the audio stream may die on return; listen for visibility changes and rebuild. And in-app browsers like WeChat's often disable the microphone outright — in such environments the voice button should be hidden, not shown as an untappable button (feature-detect microphone availability, not just API existence).

Barge-in (interruption handling)

In human conversation, interruption is normal: "wait wait, that's not what I meant." A voice assistant without barge-in forces users to sit through the entire narration — miserable. How it works: while TTS plays, VAD keeps listening; once user speech crosses the threshold, immediately speechSynthesis.cancel() (or stop the cloud TTS stream), discard text past the truncation point, and re-run the pipeline from the new input.

Two details: the interruption threshold sits one notch above normal VAD, or playback picked up by the mic "interrupts itself" (guaranteed on devices with poor echo cancellation); after interruption, mark in the LLM context that "the previous utterance was cut off," or it will keep elaborating on words never spoken.

The cost math: per-character vs per-minute billing

Cloud voice billing comes in exactly two models — don't mix them up when doing the math:

• ASR billed per audio minute: however many minutes of audio you send (usually rounded up to 15s), that's what you pay. Note "silence costs money too" — 30 seconds of a user holding the mic while daydreaming bills in full. So VAD first, sending only speech-bearing segments to ASR, saves real money.
• TTS billed per character: by input-text character count (punctuation included). SSML tags are usually excluded (vendor wording varies). Estimate before long-form narration: a 10,000-character article read aloud ≈ 10,000 characters; 1,000 narrations a day = 10M characters/day.

Unit prices vary widely across vendors and change often — treat each vendor's current pricing page as authoritative; this guide deliberately prints no prices (printed prices become expired misinformation). The formula: monthly cost ≈ daily active voice users × avg voice minutes per user × ASR unit price + daily narrated characters × TTS unit price. Run it once at 10x your expected volume — if you can't afford that number, the architecture needs quotas and degradation (next section) before launch, not after.

Three money-savers: VAD-trimmed silence (discounts the ASR bill directly); cached TTS results (replay the recording for repeated lines — assistant greetings and error prompts are high-frequency repeats); daily voice quotas for free users, falling back to text when exceeded.

Degradation: ASR failure always falls back to keyboard

Voice degradation isn't "show an error toast" — it's a full fallback chain, every level a usable product:

1. Browser unsupported (Firefox, no mic permission, in-app webview): hide the voice button entirely; the keyboard input is everything. A grayed-out untappable button isn't degradation — it's decoration.
2. Recognition failed (no-speech, network, low confidence): keep whatever was already transcribed (never clear it!), leave it editable in the input, and offer both "speak again" and "just edit the text."
3. Cloud service down: Web Speech API and cloud ASR back each other up. Cloud-primary setups should auto-switch to browser-native recognition within 30 seconds — at least never go dark.
4. Cost circuit breaker: daily voice quota exhausted → automatic pure-text mode, with an honest UI note ("today's voice quota is used up") instead of silently getting slower.

Write the degradation chain in v1, not "after user complaints." Voice is the least reliable input channel — network, noise, accent, permissions; if any of the four breaks with no fallback, that's one complete user churned.

Launch checklist

• Three questions passed: hands busy, speaking beats typing, environment allows it.
• ASR choice: MVP on Web Speech API (remember feature-detect + hide the button on Firefox); migrate to cloud when volume or accuracy demands.
• TTS choice: speechSynthesis for short announcements (handle voiceschanged); cloud TTS + SSML for long content / brand voices.
• Latency: user-finished to first sound <2000ms usable, <1200ms excellent; turn "wait for the full LLM sentence" into a streaming pipeline first.
• Mobile: permission only on button tap; barge-in with echo protection; tested on a real iOS device.
• Cost: bill computed at 10x expected volume; VAD-trimmed silence; cached high-frequency TTS; quotas for free users.
• Degradation: four-level fallback chain written; quota exhaustion messaged honestly, never silently slower.

Voice interaction is, at its core, adding the "most natural but least reliable" channel to a product. All the engineering is scaffolding between "natural" and "reliable": VAD so it never waits blindly, streaming so it's never blindly slow, degradation so it never blindly crashes. With the scaffolding up, voice is a plus; without it, the more dazzling the demo, the harder the launch-day fall.