How to add voice-note transcription to your app

Someone sends a 40-second voice note. The recipient is on a train, in a meeting, or just doesn't want to listen to 40 seconds of anything. So the note sits there, unplayed, for six hours.

Every app that ships voice messages eventually ships transcription, and for the same reason: a voice note is faster to send and slower to receive, and that asymmetry is where the feature dies. WhatsApp added it. Telegram added it. Apple added it. If your product has a mic button, this is on your roadmap whether it's written down or not.

The good news is that it's a small build. Voice notes are the canonical short-clip job — a few seconds to a couple of minutes, one speaker, already recorded by the time you see them. That's precisely the shape the Sync API is built for: POST the audio, get the finished transcript back in the same response. No job IDs, no polling, no queue.

Let's build the feature end to end, then handle the backlog of notes your users have already sent.

What you'll build

1. Receive an uploaded voice note on your server
2. Transcribe it in a single call
3. Render the text inline, under the waveform
4. Handle multilingual inboxes
5. Backfill the notes already sitting in your database

Prerequisites: an AssemblyAI API key, Python 3.8+ or Node.js 18+.

Step 1: Receive the note and check it before you send it

The Sync API accepts WAV or raw PCM (S16LE), between 80 milliseconds and 2 minutes, up to 40MB. Most mobile clients record something else — AAC in an m4a container, Opus in WebM — so you'll be converting.

import subprocess

def to_wav(source_path: str, target_path: str) -> None:
    """Normalize any incoming voice note to 16kHz mono WAV."""
    subprocess.run(
        ["ffmpeg", "-i", source_path, "-ar", "16000", "-ac", "1",
         "-c:a", "pcm_s16le", "-y", target_path],
        check=True,
        capture_output=True,
    )

Two guards worth adding before the request, because both failure modes are common and both are cheap to catch locally:

‍

• Duration under 80ms. Users fat-finger the mic button constantly. These produce audio_too_short, and there's no reason to spend a round trip discovering that.
• Duration over 2 minutes. Some people record five-minute voice notes. That's a real thing your product has to handle — see the routing section below.

Step 2: Transcribe it

This is the entire integration.

import os
from assemblyai.sync.v1 import SyncTranscriber

transcriber = SyncTranscriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])
result = transcriber.transcribe("./voice_note.wav")

print(result.text)

import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const result = await client.sync.transcribe("./voice_note.wav");
console.log(result.text);

Or over plain HTTP, if you'd rather not add an SDK. The X-AAI-Model header is required:

import requests

with open("voice_note.wav", "rb") as f:
    audio = f.read()

response = requests.post(
    "https://sync.assemblyai.com/transcribe",
    headers={
        "Authorization": "<YOUR_API_KEY>",
        "X-AAI-Model": "universal-3-5-pro",
    },
    files={"audio": ("voice_note.wav", audio, "audio/wav")},
    timeout=60,
)
response.raise_for_status()

result = response.json()
print(result["text"])

You get back the transcript, per-word confidence scores, an overall confidence value, the audio duration, a session_id, and the server-side processing time:

{
  "text": "Hey, running about ten minutes late — start without me.",
  "words": [
    { "text": "Hey", "confidence": 0.98 },
    { "text": "running", "confidence": 0.96 }
  ],
  "confidence": 0.97,
  "audio_duration_ms": 3120,
  "session_id": "eb92c4ff-4bbb-429f-9b99-7279d7fe738f",
  "request_time_ms": 141.6
}

The model here is Universal-3.5 Pro — the same flagship model behind our pre-recorded transcription, not a stripped-down fast variant. On a 2-second clip the whole round trip lands around 134ms at p50. If you want the base call in isolation before wiring it into your upload handler, the Sync STT quickstart is four lines.

Step 3: Render it in the right place

This is a product decision more than an engineering one, and it's where most implementations go wrong.

The transcript is not a replacement for the voice note. It's a preview of it. The pattern that works: waveform stays, transcript appears beneath it as collapsed text with a "show more" affordance, and playback still works. The user decides whether reading is enough.

Where it gets interesting is when you transcribe. Two options:

On send. The sender's client uploads, you transcribe immediately, and the note arrives with text already attached. Costs you a transcription for every note including the ones nobody opens, but the recipient never waits.

On open. You transcribe the first time a recipient views the thread. Cheaper if a meaningful share of notes go unread — and since the transcript comes back in a single call in roughly the time it takes to render the message row, the user experience is close to indistinguishable.

For most messaging products, on-open wins on cost and loses nothing perceptible. For products where notes are searchable — a productivity app, a CRM, anything with a search bar over message history — transcribe on send, because you need the text whether or not anyone opened the message.

Want to see it work? Get a free API key or drop a voice note into the Sync Playground.

Step 4: Handle multilingual inboxes

Voice notes are where multilingual products stop being able to pretend everyone speaks English. A user's inbox is whatever languages their contacts speak, often several within one thread.

language_codes takes a single code for monolingual audio or a list for multilingual audio, across 18 supported languages:

from assemblyai.sync.v1 import SyncTranscriptionConfig

# You know this user's threads are Spanish and English.
config = SyncTranscriptionConfig(language_codes=["es", "en"])
result = transcriber.transcribe("./voice_note.wav", config=config)

const result = await client.sync.transcribe("./voice_note.wav", {
  language_codes: ["es", "en"],
});

If your app already knows something about the conversation — the participants' locales, the language of the last twenty text messages in the thread — feed that in rather than guessing globally. You almost always have better signal than a language-agnostic default.

"We were searching for the best realtime ASR model for our voice agent pipeline in Fireflies. The new Universal 3.5 Pro speech model from Assembly is best so far in terms of accuracy, latency and language switching." — Foysal Osmany, Software Engineer, Fireflies

Step 5: Backfill the notes you already have

Shipping this feature to an existing product means every user immediately asks the same question: what about all my old voice notes?

Backfilling is a bounded, one-time job, and it's mostly about being polite to the API. Run notes concurrently, cap the concurrency, and honor the backpressure signals:

import asyncio
from assemblyai.sync.v1 import SyncTranscriber, SyncTranscriptError

async def transcribe_one(transcriber, note, semaphore):
    async with semaphore:
        for attempt in range(3):
            try:
                result = await asyncio.to_thread(transcriber.transcribe, note.path)
                return note.id, result.text
            except SyncTranscriptError as error:
                if error.status_code in (429, 503) and attempt < 2:
                    await asyncio.sleep(error.retry_after or 2 ** attempt)
                    continue
                return note.id, None       # log session_id, move on

async def backfill(notes):
    semaphore = asyncio.Semaphore(10)      # tune to your account
    with SyncTranscriber(api_key="<YOUR_API_KEY>") as transcriber:
        return await asyncio.gather(
            *(transcribe_one(transcriber, n, semaphore) for n in notes)
        )

Three things that make a backfill go smoothly:

• Respect Retry-After on 429 and 503. The error carries it. Exponential backoff without honoring the header just makes you rate-limited for longer.
• Never fail the whole batch on one bad note. Some of your archived audio is corrupt, zero-length, or in a format nobody remembers supporting. Log the session_id, mark the row, move on.
• Backfill newest-first. Users open recent threads. A backfill that starts at the beginning of time finishes the part anyone cares about last.

What about notes longer than two minutes

Some users record five-minute voice notes. This is not a bug in your product, it's a category of user, and you need a path for them.

Anything over 2 minutes or 40MB should route to Pre-recorded STT, which handles long-form audio and gives you speaker diarization and speech understanding features that don't apply to a 20-second note anyway. The split is simple:

if duration_ms <= 120_000:
    text = sync_transcribe(path)        # instant, inline
else:
    job_id = submit_async(path)         # transcript lands shortly after

Your UI can be honest about the difference — short notes get text immediately, long notes get it in a moment. Nobody who records a five-minute voice note expects instant anything.

The thing nobody tells you about voice notes

Once the text exists, your product changes in a way that's easy to miss.

Voice notes have historically been a black hole in every product that supports them: unsearchable, unquotable, unlinkable, invisible to every feature you've built on top of text. Transcribe them and they join the rest of your product. They show up in search. They can be replied to with a quote. They can be summarized, translated, turned into a task — the same downstream moves any other speech-to-text output unlocks.

The transcription is a two-day build. What it unlocks is the part worth planning for — and it's a good argument for transcribing on send rather than on open, even where on-open is cheaper.

Frequently asked questions

What audio formats does the Sync API accept for voice notes?

‍ WAV or raw PCM (S16LE). Mobile clients typically record AAC or Opus, so convert server-side — a one-line ffmpeg call to 16kHz mono WAV covers it.

How long can a voice note be?

Between 80 milliseconds and 2 minutes, up to 40MB. Longer recordings should route to Pre-recorded STT, which handles long-form audio.

Should I transcribe voice notes when they're sent or when they're opened?

Transcribe on open if you're optimizing cost and notes often go unread. Transcribe on send if voice notes need to be searchable, since you need the text whether or not anyone opened the message.

Can I transcribe voice notes in multiple languages?

Yes. language_codes accepts one code for monolingual audio or a list for multilingual audio, across 18 supported languages. Feed in whatever locale signal your app already has rather than guessing.

How do I transcribe a backlog of existing voice notes?

Run them concurrently with a capped semaphore, honor Retry-After on 429 and 503 responses, and let individual failures fall through to a log instead of failing the batch. Process newest-first so recent threads become useful immediately.

What happens if a user taps the mic button by accident?

Clips under 80 milliseconds return audio_too_short. Check duration client-side and skip the request rather than surfacing an error for a note nobody meant to send.

‍