How to build push-to-talk dictation with the Sync API

Hold a key, talk, let go, watch the words appear. That's the entire interaction, and it's deceptively hard to get right.

The hard part isn't the transcription. It's that dictation is one of the few interfaces where the user is staring at the spot where the text will appear, doing nothing else, for the entire duration of the wait. There's no loading state that makes 900ms feel acceptable. Either the words show up fast enough to feel like typing, or your feature feels broken.

Most speech-to-text integrations don't have to care about this, because they're transcribing something nobody is watching. Dictation does.

The Sync API exists for exactly this shape of request: you POST a short audio clip and get the finished transcript back in the same HTTP response. No job ID, no polling loop, no WebSocket to manage. It runs on Universal-3.5 Pro, and on a 2-second clip it returns in roughly 134ms at p50.

So let's build it. By the end of this post you'll have a working push-to-talk dictation feature: capture on key-down, transcribe on key-up, render the text inline, and get proper nouns spelled correctly.

What you'll build

A single-key dictation control that:

1. Starts recording when the user presses and holds a key (or taps a mic button)
2. Stops on release and packages the audio as WAV
3. Sends it to POST https://sync.assemblyai.com/transcribe
4. Renders the returned text into the focused input
5. Handles the failure cases that will otherwise bite you in production

Prerequisites: an AssemblyAI API key, Python 3.8+ or Node.js 18+, and a browser for the capture layer. The Sync STT quickstart covers the base call if you want to run it once before building the UI around it.

Set your key once:

export ASSEMBLYAI_API_KEY=<your-key>

Step 1: Capture audio on tap-to-talk

The Sync API accepts WAV or raw PCM. Browsers give you WebM from MediaRecorder by default, so the cleanest path is to capture raw PCM through an AudioWorklet and write a WAV header yourself, or capture at 16kHz mono and convert.

Here's the capture half — a recorder that starts on key-down and resolves to a WAV Blob on key-up:

class PushToTalkRecorder {
  async start() {
    this.stream = await navigator.mediaDevices.getUserMedia({
      audio: { channelCount: 1, sampleRate: 16000, echoCancellation: true },
    });

    this.context = new AudioContext({ sampleRate: 16000 });
    const source = this.context.createMediaStreamSource(this.stream);
    this.chunks = [];

    // ScriptProcessor is deprecated but universally supported;
    // swap in an AudioWorklet for production.
    this.processor = this.context.createScriptProcessor(4096, 1, 1);
    this.processor.onaudioprocess = (e) => {
      this.chunks.push(new Float32Array(e.inputBuffer.getChannelData(0)));
    };

    source.connect(this.processor);
    this.processor.connect(this.context.destination);
  }

  stop() {
    this.processor.disconnect();
    this.stream.getTracks().forEach((t) => t.stop());
    this.context.close();

    const samples = this.chunks.flatMap((c) => Array.from(c));
    return encodeWav(Int16Array.from(samples, (s) => s * 0x7fff), 16000);
  }
}

‍

encodeWav is a 40-line function that prepends a standard 44-byte RIFF header to your PCM samples. Nothing AssemblyAI-specific happens there.

Two constraints to design around: audio must be at least 80 milliseconds and at most 2 minutes. The floor matters more than you'd think — a user who taps the key instead of holding it will generate a 30ms clip, and that request comes back as audio_too_short. Check the duration client-side before you spend a round trip on it.

Step 2: Send it and get the transcript back

This is the whole API surface. One POST, one response.

import os
from assemblyai.sync.v1 import SyncTranscriber

transcriber = SyncTranscriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])
result = transcriber.transcribe("./dictation.wav")
print(result.text)

import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const result = await client.sync.transcribe("./dictation.wav");
console.log(result.text);

If you're wiring this up as a server route behind your browser client, the raw HTTP call is just as small. The X-AAI-Model header is required on every request:

import requests

with open("dictation.wav", "rb") as f:
    audio = f.read()

response = requests.post(
    "https://sync.assemblyai.com/transcribe",
    headers={
        "Authorization": "<YOUR_API_KEY>",
        "X-AAI-Model": "universal-3-5-pro",
    },
    files={"audio": ("dictation.wav", audio, "audio/wav")},
    timeout=60,
)
response.raise_for_status()
print(response.json()["text"])

You get back the transcript plus per-word confidence, an overall confidence score, the audio duration, a session_id, and request_time_ms — the server-side processing time for that request:

{
  "text": "Ship the migration on Thursday and loop in Priya.",
  "words": [
    { "text": "Ship", "confidence": 0.97 },
    { "text": "the", "confidence": 0.99 }
  ],
  "confidence": 0.96,
  "audio_duration_ms": 2840,
  "session_id": "eb92c4ff-4bbb-429f-9b99-7279d7fe738f",
  "request_time_ms": 138.2
}

Log session_id on every request, not just the failures. It's the first thing support asks for.

Step 3: Make it feel instant with connection pre-warming

Here's the part most dictation builds miss.

Sync STT is a single request/response, which means connection setup is inside your latency budget. If the HTTPS connection isn't already open when the user releases the key, the first thing your request does is negotiate one: DNS resolution, then a TCP handshake, then a TLS handshake. That's a round trip each, before a single byte of audio moves. For a client near the serving region it's a few tens of milliseconds. Intercontinental, it can be well over 100ms.

None of that work depends on your audio. So don't pay for it while the user is waiting — pay for it while they're still talking.

GET /warm is an unauthenticated no-op that exists purely to force your HTTP client through that handshake and drop the resulting connection into its pool. Call it the instant recording starts:

import assemblyai as aai
from assemblyai.sync.v1 import SyncTranscriber

# Hold the idle connection long enough to cover the whole recording.
aai.settings.keepalive_expiry = 120

with SyncTranscriber(api_key="<YOUR_API_KEY>") as transcriber:
    transcriber.warm()          # key-down: handshake runs while the user talks

    # ... recording finishes ...

    result = transcriber.transcribe("dictation.wav")   # no handshake here
    print(result.text)

import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

// warm() resolves to a boolean and never throws.
await client.sync.warm();       // key-down

// ... recording finishes ...

const result = await client.sync.transcribe("dictation.wav");  // key-up

Three caveats decide whether this actually helps:

• Same client object. The warm and the transcribe have to share a connection pool — one requests.Session, one AssemblyAI client, one process. Two separate curl invocations each open their own connection, so warming does nothing for them.
• Same base URL. A connection warmed against the global endpoint doesn't help a request sent to the EU or US residency endpoint.
• Timing. Pooled connections expire. httpx drops idle connections after 5 seconds by default, and the server closes idle connections after a few minutes. Warm too early and you've bought nothing. "The moment recording starts" is the right trigger precisely because it's the last moment you know audio is coming but don't have it yet.

/warm is idempotent and cheap, so calling it again to refresh an aging connection is fine. The connection pre-warming docs go deeper on pool behavior per HTTP client.

Step 4: Get names and jargon right

Out of the box, Universal-3.5 Pro handles general dictation well. Where it needs help is the vocabulary it can't know: your users' colleagues, your product names, internal shorthand.

Two config fields solve this, and they do different jobs.

keyterms_prompt takes an explicit list of terms and biases the decoder toward them. This is the right tool when you have a vocabulary list — a contacts table, a product catalog, a repo's service names:

from assemblyai.sync.v1 import SyncTranscriber, SyncTranscriptionConfig

config = SyncTranscriptionConfig(
    keyterms_prompt=["Priya Raghunathan", "Kubernetes", "AssemblyAI", "Datadog"],
)

transcriber = SyncTranscriber(api_key="<YOUR_API_KEY>", config=config)
result = transcriber.transcribe("dictation.wav")

prompt takes a natural-language description of the audio — not instructions, a description. It primes the model for a whole domain rather than specific tokens:

const result = await client.sync.transcribe("dictation.wav", {
  prompt: "Engineering standup notes about a database migration.",
  keyterms_prompt: ["Priya Raghunathan", "Kubernetes", "Datadog"],
});

A word of warning on both: start with neither. Universal-3.5 Pro is tuned to work without them, and stuffing in a large list of common words causes overcorrection — the model starts hearing your keyterms where they weren't spoken. Add terms only for the specific words you keep watching it get wrong. Keep the total under 2048 characters across all terms, and spell each one exactly as you want it to appear in the output. The prompting and keyterms guide has the full specificity ladder.

If you're building dictation into a multilingual product, language_codes takes one code or several, across the 18 supported languages.

"The speed difference is immediately noticeable — our users see their conversations transcribed almost instantaneously. It feels so much more responsive than what we were using before." — Jonathan Kim, Software Engineer, Granola

Try it yourself: get a free API key and run the code above against a 3-second recording of your own voice, or drop a clip into the Sync Playground first.

Step 5: Handle the cases that break in production

Dictation fails in a small number of specific ways, and they're all recoverable if you plan for them.

from assemblyai.sync.v1 import SyncTranscriber, SyncTranscriptError

try:
    result = transcriber.transcribe("dictation.wav")
except SyncTranscriptError as error:
    if error.error_code == "audio_too_short":
        pass                                   # tap, not a hold — ignore silently
    elif error.status_code in (429, 503):
        retry_after(error.retry_after)          # back off and retry once
    else:
        show_dictation_error(error.error_code)  # surface the error code for support

The one that matters most for perceived quality is the first one. Users tap keys by accident constantly, and a dictation feature that throws a red toast every time is worse than one that stays quiet.

Where dictation goes next

The build above is a complete dictation feature, and for most products it's enough. But notice what it doesn't do: it gives you exactly what the user said, disfluencies and all. "Um, so, ship the — ship the migration Thursday, uh, and loop in Priya."

That's correct transcription. It's often not what you want on screen.

The gap between verbatim transcript and the text a user actually wanted typed is where dictation gets interesting, and it's a different problem from speech recognition — it's a cleanup pass. Some teams handle it with a small model after the fact. Some accept the disfluencies as honest. The right answer depends on whether your users are drafting prose or issuing commands, and it's worth deciding deliberately rather than by default.

Start with the transcript being right. Everything downstream depends on it.

Frequently asked questions

What's the difference between dictation and real-time transcription?

Dictation is a completed utterance sent in one piece after the user stops speaking — the Sync API's shape. Real-time transcription streams audio continuously over a WebSocket and returns partial results as the person is still talking. If your UI shows text only after the user releases a key, you want Sync. If it shows words appearing mid-sentence, you want Real-time STT.

How long can a single dictation be?

Between 80 milliseconds and 2 minutes, up to 40MB. Longer recordings should go to Pre-recorded STT, which has no practical duration ceiling.

Do I need to pre-warm the connection?

‍ No — everything works without it. Pre-warming removes the DNS, TCP, and TLS handshake from the critical path, which matters most when your users are geographically distant from the serving region. It's a few lines of code for a meaningful cut in perceived latency.

Why is my dictation returning audio_too_short?

The clip is under the 80ms floor, almost always because the user tapped a key rather than holding it. Check duration client-side and skip the request entirely rather than showing an error.

Can I dictate in languages other than English?

Yes. Pass language_codes with one code for monolingual audio or several for multilingual audio, across the 18 supported languages.

How do I make it spell my users' names correctly?

‍ Pass those names in keyterms_prompt, spelled exactly as you want them to appear. Keep the list to terms the model actually gets wrong — a long list of common words causes overcorrection.