Text to speech vs speech to text: what's the difference and how they work together

Speech to text (STT) turns spoken audio into written text. Text to speech (TTS), also called speech synthesis, does the reverse: it turns written text into spoken audio. They’re mirror-image technologies, and the most useful thing they do today is work together. In a voice agent, STT is how the agent listens and TTS is how it talks back, with turn detection and a large language model (LLM) sitting between them.

Most explanations stop at “one goes in, the other goes out.” That’s true, and it’s also the least interesting part. The hard problems live at the handoffs: knowing when the caller has finished talking, getting the reply out loud fast, and stopping mid-sentence when the caller cuts in. This guide covers both halves, the key differences between them, and how they connect into one real-time voice loop.

Text to speech vs speech to text: the short answer

Here’s the side-by-side view.

The direction of the data is the obvious difference. The less obvious one is what “good” means. STT has a right answer: the words that were spoken. TTS has many acceptable answers, and the job is to pick one that sounds like a person who understood what they were saying.

What is speech to text?

Speech to text is AI that converts spoken audio into written text. You’ll also see it called automatic speech recognition (ASR). The two terms describe the same capability: ASR names the technology, speech to text names the output.

A modern STT model does three jobs. It turns raw audio into acoustic features, predicts the most likely sequence of words from those features, and formats the result with punctuation, casing, and entities rendered properly (so “five five five one two one two” becomes a phone number). Accuracy is usually reported as word error rate: the share of words the model substituted, dropped, or inserted compared with a human reference transcript.

STT runs in two main modes:

• Pre-recorded (async): you send a finished file and get a full transcript back. AssemblyAI’s pre-recorded speech-to-text leads with Universal-3.5 Pro, which supports 18 languages with native code switching and contextual prompting.
• Streaming (real-time): you send audio as it’s spoken over a WebSocket and get partial transcripts while the person talks, then a final, formatted transcript when their turn ends. AssemblyAI’s streaming speech-to-text leads with Universal-3.6 Pro Realtime, which covers 32 languages with native code switching.

For voice agents, streaming is the mode that matters. A transcript that shows up after the call ends can’t power a conversation.

What is speech synthesis (text to speech)?

Speech synthesis is the machine generation of human-sounding speech, and text to speech is its most common form: you give the system a string of text and it returns audio of a voice saying it. Our guide to how modern conversational TTS works goes deep on the pipeline. The short version has four steps:

1. Text normalization: expanding things nobody says as written. “Dr.” becomes “Doctor,” “3:30pm” becomes “three thirty p m.”
2. Pronunciation: working out how each word sounds, including words spelled the same but said differently.
3. Prosody: deciding pitch, timing, stress, and pauses. This is what separates a human-sounding voice from a flat one.
4. Waveform generation: a neural model turns all of that into audio samples, streamed out in small chunks so playback can start before the sentence is finished.

TTS used to mean screen readers and GPS directions. Today the fastest-growing job is conversation, and conversational TTS has a different set of requirements: short replies, text that arrives a token at a time from an LLM, and a voice that has to stop the instant the caller interrupts.

Five differences that matter when you build with them

Once you’re building, the “one goes in, one goes out” framing hides the differences that shape your design.

1. What counts as an error

An STT error is measurable against ground truth: a wrong digit in an account number, a dropped name. A TTS error is often a judgment call (flat intonation, odd emphasis), with one big exception. Numbers, codes, and names have to be read out exactly, and that’s where both sides fail most visibly.

2. Where the latency hides

For STT, the clock that matters starts when the speaker stops talking and ends when your app has a final transcript. For TTS, it starts when the first text arrives and ends when the listener hears sound: time to first audio. That’s a different number from time to first byte, and our breakdown of TTS latency metrics explains why the gap matters.

3. What streams in

Streaming STT takes a continuous flow of audio and has to decide on its own where one utterance ends. Streaming TTS takes a flow of text tokens and has to decide when it holds enough of a sentence to start speaking naturally.

4. What context helps

STT gets better when you tell it what to expect, through keyterms or a prompt describing the domain. TTS gets better when the text is written to be spoken: short sentences, and numbers written the way you want them said.

5. How you evaluate them

STT evaluation is mostly quantitative: run your own audio, compute WER and entity error rate. TTS evaluation needs ears: listening tests, side-by-side comparisons, and a hard look at tricky strings. Our guide to evaluating TTS for voice agents lays out a practical method.

How speech to text and text to speech work together: the voice loop

Put STT and TTS on either side of an LLM and you have the basic loop behind every cascaded voice agent: listen, decide the turn is over, think, speak. There are other ways to build one, and our comparison of voice agent architectures walks through the trade-offs. But the chained loop is the one most production teams run, because each stage can be swapped, tuned, and debugged on its own. Here’s what happens on every turn.

Step 1: Streaming speech to text listens

The caller’s audio streams to the STT model over a WebSocket. Universal-3.6 Pro Realtime returns partial transcripts while the caller is still talking, and each partial re-transcribes the whole turn so far, so the text gets more accurate as the turn goes on. Partials are useful for live captions and for warming up the LLM early. The final transcript is the one to act on.

Step 2: Turn detection decides the caller is done

This is the step most people skip when they explain voice agents, and it’s the one callers feel most. End a turn too early and the agent talks over someone reading out a phone number. End it too late and the agent feels slow.

Universal-3.6 Pro Realtime judges the end of a turn from what was said as well as how long the silence lasts. When the caller pauses, the model checks whether the sentence reads as complete. If the turn ends partway through something like a phone number or an email address, entity-aware endpointing holds the turn open so the whole entity lands in one transcript. On AssemblyAI’s voice-agent benchmark, that cut the phone-number entity error rate from 7.0% to 2.2%. Our piece on turn detection vs forced endpoints covers why silence-only rules break down.

Step 3: The LLM decides what to say

The final transcript goes to an LLM along with the conversation history, any tool results, and your instructions. The LLM streams its reply back token by token. You can bring your own model or route requests through LLM Gateway, which runs open and frontier LLMs behind one API key.

Step 4: Text to speech talks back

Here’s the handoff that makes or breaks perceived speed. Waiting for the LLM to finish its whole reply before starting TTS wastes time the caller can hear. Instead, stream the tokens into a TTS session as they arrive. The AssemblyAI Text-to-Speech API takes text over one WebSocket session per conversation, starts speaking once it holds enough complete sentences, and streams raw audio back so playback starts on the first bytes.

This Python fragment, adapted from the TTS quickstart, shows the shape. You supply llm_tokens(), user_started_talking(), and play():

import os

import assemblyai as aai

config = aai.TTSConfig(voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", language="english")  # Audrey

with aai.TTSSession(config, api_key=os.environ["ASSEMBLYAI_API_KEY"]) as session:
    for token in llm_tokens():
        session.generate(token)  # joined exactly as sent: include the spaces
    flush_id = session.flush()

    for chunk in session.audio(flush_id):  # raw 24 kHz 16-bit mono PCM
        if user_started_talking():
            session.cancel()  # barge-in: the server drops the rest of the reply
            break
        play(chunk)

Step 5: Barge-in closes the loop

Real callers interrupt. When they do, STT and TTS have to cooperate. The streaming STT session sends a SpeechStarted message as soon as it has a real transcript for new speech, so background noise alone doesn’t trigger it. That’s your signal to stop playback and cancel the TTS request in flight. Then the loop starts again at step 1.

If you want to build this end to end, our walkthrough of a chained STT-LLM-TTS voice agent is a good next step. AssemblyAI TTS also ships plugins for LiveKit and Pipecat, so you can drop it into an existing agent framework.

Where the handoffs break

When a voice agent feels wrong, the cause usually sits between two stages. A few patterns show up again and again:

• Split entities. A silence-only endpointer ends the turn halfway through a card number. The LLM gets half the digits and asks the caller to repeat themselves. Fix it on the STT side with entity-aware turn detection.
• The format round trip. STT writes “$4.50” and “Dr. Lee.” The LLM passes them along. TTS has to turn them back into words. Every trip through written form is a chance to misread, so check how your TTS reads the exact strings your agent says most.
• Waiting on whole replies. Buffering the full LLM response before calling TTS adds the LLM’s entire generation time to every turn. Stream tokens through, and flush at sentence ends.
• A new connection per sentence. Opening a fresh TTS connection for every sentence pays the handshake each time and runs into limits on new sessions. Keep one session open per conversation.
• Mismatched audio formats. Phone calls typically run on 8 kHz μ-law audio. Ask TTS for the telephony encoding your carrier expects (pcm_mulaw or pcm_alaw) so the audio is ready for the call as it arrives.

Choosing speech to text and text to speech for a voice agent

You can mix vendors, and plenty of teams do. Either way, judge each half on what the loop needs from it.

For speech to text, look for:

• Streaming with partials and a formatted final transcript
• Turn detection that understands content, especially entities
• Strong entity accuracy on names, numbers, and IDs
• The languages your callers speak, including mid-call switching

For text to speech, look for:

• A model built for conversation, with natural delivery on short replies
• Streaming text input, so speech starts before the LLM finishes
• A cancel that stops the request in flight for barge-in
• Telephony encodings and per-word timings for syncing transcripts to audio
• Data zones that match where your users are

On the AssemblyAI side, Universal-3.6 Pro Realtime covers the listening half. The AssemblyAI Text-to-Speech API covers the speaking half with AssemblyAI-TTS-1, a model built for conversational voice agents and trained on millions of hours of audio, including real conversational data. It delivers frontier-level quality, runs faster and costs less than other production voice agent TTS models, and has unlimited concurrency. It speaks English, Spanish, French, German, Italian, and Portuguese, offers a library of thousands of voices plus instant voice cloning in the dashboard, and runs in US and EU data zones. One API key covers STT, LLM Gateway, and TTS.

The loop is the product

“Text to speech vs speech to text” sounds like a choice between two things. For anyone building with voice, it’s a pairing. You need both, and the quality your users notice comes from how cleanly the two halves hand off to each other.

So measure the loop end to end. Time how long a caller waits from the moment they stop talking to the moment they hear a reply. Count how often an interruption stops the agent cleanly. Listen to how it reads back the numbers your callers give it. Those three checks will tell you more about your voice agent than any single-component benchmark.

Frequently asked questions

What is the difference between text to speech and speech to text?

Speech to text converts spoken audio into written text, while text to speech converts written text into spoken audio. STT is judged mainly on word error rate, and TTS on how natural and intelligible it sounds and how fast the first audio arrives. In a voice agent you use both: STT to listen and TTS to reply.

What is speech synthesis?

Speech synthesis is the machine generation of human-sounding speech, and text to speech is its most common form. A modern system normalizes the text, works out pronunciation and prosody, and then uses a neural model to generate the audio waveform. The terms speech synthesis and TTS are often used interchangeably.

What are STT and TTS models?

STT models are AI models that transcribe speech into text, such as AssemblyAI’s Universal-3.6 Pro Realtime for streaming audio and Universal-3.5 Pro for recorded files. TTS models generate speech from text, such as AssemblyAI-TTS-1. Voice agents pair one of each with an LLM in between.

Do voice agents need both speech to text and text to speech?

A cascaded voice agent needs both: streaming STT turns the caller’s speech into text, turn detection decides when they’ve finished, an LLM writes the reply, and TTS speaks it. Keeping the stages separate lets you choose, tune, and debug each one independently.

Is speech recognition the same as speech to text?

For practical purposes, yes. Automatic speech recognition (ASR) is the technology, and speech to text describes its output, a written transcript. Voice recognition is a separate task: identifying who is speaking.

Can I use AssemblyAI for both speech to text and text to speech?

Yes. AssemblyAI offers streaming and pre-recorded speech-to-text, LLM Gateway for the reasoning step, and the AssemblyAI Text-to-Speech API for the reply, all under one API key. The TTS API works over HTTP or a WebSocket session, has Python and JavaScript SDKs, and has plugins for LiveKit and Pipecat.