How to build an AI voice translator in Python

There's a specific moment in this project where it stops being a coding exercise. You record ten seconds of yourself talking in English, hit submit, wait about half a minute, and then hear your own voice say something in Japanese. Not a synthetic narrator reading a translation. You. Speaking a language you can't speak.

Our own Misra Turp built exactly that app and described the feeling as "honestly kind of eerie to hear yourself speak languages that you cannot actually speak." She's right. It's also about 100 lines of Python.

The trick is that a voice translator isn't one hard problem. It's three easy ones chained together:

1. Speech-to-text — turn the recording into English text.
2. Translation — turn that English text into Spanish, Turkish, and Japanese text.
3. Text-to-speech — read each translation back in a clone of your voice.

Wrap those three calls in a Gradio interface and you have a web app with a microphone button and three audio players. That's the whole architecture.

This guide walks through the working code, then covers what's changed in the AssemblyAI API since the original tutorial shipped in 2024 — because a couple of things have, and one of them lets you delete a dependency entirely.

What you'll need

• Python 3.8+
• An AssemblyAI API key — grab one from your dashboard. Transcription is billed per second with no minimums, so a handful of ten-second clips costs fractions of a cent.
• An ElevenLabs API key plus a cloned voice. Instant voice cloning needs about a minute of audio; professional cloning is a paid feature that wants at least 30 minutes. Misra uploaded roughly 33 minutes of her existing videos and noted the optimal upper limit is around three hours.
• The packages: gradio, assemblyai, translate, and elevenlabs.

All the code below comes from the Voice-to-Voice-translator repo, which ships two versions: simple_vtv.py (a minimal gr.Interface, three languages) and voice_translator.py (a gr.Blocks layout with six languages and the translated text displayed alongside each clip). We'll build the simple one and then look at what the fancier one adds.

How the pipeline fits together

Start with the imports and the shape of the app. Four functions: one orchestrator and one per stage.

import gradio as gr
import assemblyai as aai
from translate import Translator
from elevenlabs import VoiceSettings
from elevenlabs.client import ElevenLabs
import uuid
from pathlib import Path

Gradio gives you two ways to build. gr.Interface connects an input to an output for you — you specify a function, its inputs, and its outputs, and it renders the layout. gr.Blocks hands you manual control over rows, columns, and groups. The simple version uses Interface, which is why it fits in one screen:

audio_input = gr.Audio(
    sources=["microphone"],
    type="filepath"
)

demo = gr.Interface(
    fn=voice_to_voice,
    inputs=audio_input,
    outputs=[gr.Audio(label="Spanish"), gr.Audio(label="Turkish"), gr.Audio(label="Japanese")]
)

if __name__ == "__main__":
    demo.launch()

Two details in there matter more than they look. sources=["microphone"] restricts the input to recording — drop "upload" in there too if you also want file uploads. And type="filepath" is the one people get wrong: it makes Gradio hand your function the path to the recorded file, which is exactly what a transcription API wants. Misra typed file_name first, which isn't a valid value.

One workflow tip while you're iterating. Running python simple_vtv.py starts the app once, and you won't see changes on refresh. Running gradio simple_vtv.py instead watches the file and reloads on save.

Step 1: transcribe the recording with AssemblyAI

Here's the transcription function as the repo ships it:

def audio_transcription(audio_file):

    aai.settings.api_key = "<your-assemblyai-api-key>"

    transcriber = aai.Transcriber()
    transcription = transcriber.transcribe(audio_file)

    return transcription

Three lines of real work. You pass a local file path and the SDK handles uploading, submitting, and polling for you — the call blocks until the transcript is done, so there's no polling loop to write. If you've never used the API before, converting speech to text in Python covers the same call in isolation.

Notice what this function returns: the whole response object, not transcription.text. That's deliberate, and it's the detail worth copying. The orchestrator needs to know whether the job failed before it starts spending money on translation and speech generation:

def voice_to_voice(audio_file):

    #transcribe audio
    transcription_response = audio_transcription(audio_file)

    if transcription_response.status == aai.TranscriptStatus.error:
        raise gr.Error(transcription_response.error)
    else:
        text = transcription_response.text

A completed transcription job is either completed or error — there's no third state to handle once the blocking call returns. So checking for error and falling through to .text covers everything. Raising gr.Error surfaces the API's own error message in the Gradio UI instead of dumping a stack trace to your terminal.

Step 2: translate the text

The original tutorial uses Python's translate module, which defaults to the free MyMemory provider. Misra's honest assessment: "it can be a bit more context aware at times, but overall for just to kind of use it personally I think it's good enough."

def text_translation(text):

    translator_es = Translator(from_lang="en", to_lang="es")
    es_text = translator_es.translate(text)

    translator_tr = Translator(from_lang="en", to_lang="tr")
    tr_text = translator_tr.translate(text)

    translator_ja = Translator(from_lang="en", to_lang="ja")
    ja_text = translator_ja.translate(text)

    return es_text, tr_text, ja_text

Three near-identical blocks, which the author cheerfully calls out as "kind of hacking a solution together." The six-language version in voice_translator.py does the obvious cleanup:

def translate_text(text: str) -> str:

    languages = ["ru", "tr", "sv", "de", "es", "ja"]
    list_translations = []

    for lan in languages:
        translator = Translator(from_lang="en", to_lang=lan)
        translation = translator.translate(text)
        list_translations.append(translation)

    return list_translations

All you need is the two-letter language code, and from_lang is a parameter like any other — so an English-to-anything translator becomes a Spanish-to-anything translator by changing one string. Expose it as a dropdown and users pick their own source language.

Worth knowing: you can skip this dependency entirely now. See the 2026 update below.

Step 3: generate speech in your own voice

This is the stage that makes the demo land, and it's mostly ElevenLabs' own sample code with three values changed. Before you write any of it, clone your voice in the ElevenLabs dashboard under Voices, then copy the voice ID (clicking it copies to your clipboard).

def text_to_speech(text):

    client = ElevenLabs(
        api_key= "<your-elevenlabs-api-key>",
    )

    # Calling the text_to_speech conversion API with detailed parameters
    response = client.text_to_speech.convert(
        voice_id="<your-voice-id>", #clone your voice on elevenlabs dashboard and copy the id
        optimize_streaming_latency="0",
        output_format="mp3_22050_32",
        text=text,
        model_id="eleven_multilingual_v2", # use the turbo model for low latency, for other languages use the `eleven_multilingual_v2`
        voice_settings=VoiceSettings(
            stability=0.5,
            similarity_boost=0.8,
            style=0.5,
            use_speaker_boost=True,
        ),
    )

    # Generating a unique file name for the output MP3 file
    save_file_path = f"{uuid.uuid4()}.mp3"

    # Writing the audio to a file
    with open(save_file_path, "wb") as f:
        for chunk in response:
            if chunk:
                f.write(chunk)

    print(f"{save_file_path}: A new audio file was saved successfully!")

    # Return the path of the saved audio file
    return save_file_path

The three edits that matter:

• voice_id — the sample defaults to a stock voice. Swap in your clone's ID or the whole point evaporates.
• model_id="eleven_multilingual_v2" — the sample ships with a turbo English model. The turbo model is faster; the multilingual one is the one that can actually speak Japanese.
• voice_settings — stability 0.5, similarity boost 0.8, style 0.5, speaker boost on. These are the values Misra landed on after experimenting in the ElevenLabs speech playground. Speaker boost trades a little generation speed for closer similarity to the source voice.

The uuid.uuid4() filename is doing real work here, by the way — you're calling this function three times in a row and each call needs its own file.

Step 4: wire it together (and the pathlib gotcha)

Here's the full orchestrator:

def voice_to_voice(audio_file):

    #transcribe audio
    transcription_response = audio_transcription(audio_file)

    if transcription_response.status == aai.TranscriptStatus.error:
        raise gr.Error(transcription_response.error)
    else:
        text = transcription_response.text

    es_translation, tr_translation, ja_translation = text_translation(text)

    es_audi_path = text_to_speech(es_translation)
    tr_audi_path = text_to_speech(tr_translation)
    ja_audi_path = text_to_speech(ja_translation)

    es_path = Path(es_audi_path)
    tr_path = Path(tr_audi_path)
    ja_path = Path(ja_audi_path)

    return es_path, tr_path, ja_path

Those three Path() conversions are the non-obvious part. text_to_speech returns a plain string, and if you hand that string straight to a Gradio audio component it won't play. Wrapping it in a pathlib.Path fixes it. It's a small thing that costs an hour if you don't know it.

Everything else is Gradio doing its job: the three paths returned from voice_to_voice map, in order, onto the three gr.Audio components declared in the outputs list. No event wiring required.

Run it, allow microphone access, record something, and submit. In the video, "hello it is a beautiful day today but I'm a little bit cold because the AC in this room is blowing really hard" comes back in Spanish, Turkish, and Japanese in roughly 20 to 30 seconds. Most of that is the three sequential text-to-speech calls, not the transcription.

The nicer version: gr.Blocks, six languages, visible text

The gr.Interface version works but looks like a form. voice_translator.py uses gr.Blocks to lay out two rows of three language cards, each pairing an audio player with the translated text, plus explicit Submit and Clear buttons and a styled waveform:

input_audio = gr.Audio(
    sources=["microphone"],
    type="filepath",
    show_download_button=True,
    waveform_options=gr.WaveformOptions(
        waveform_color="#01C6FF",
        waveform_progress_color="#0066B4",
        skip_length=2,
        show_controls=False,
    ),
)

And the wiring, which is now explicit rather than inferred:

    output_components = [ru_output, tr_output, sv_output, de_output, es_output, 
jp_output, ru_text, tr_text, sv_text, de_text, es_text, jp_text]
    submit.click(fn=voice_to_voice, inputs=audio_input, outputs=output_components, 
show_progress=True)

Twelve outputs — six audio players and six markdown blocks — which is why the main function ends with a return statement that unpacks twelve values by index. It's not elegant, but it shows the pattern: Gradio maps returns to outputs positionally, so the order of that output_components list has to match the order of your return exactly.

What's changed since 2024: the 2026 update

This tutorial shipped in June 2024. The architecture has aged well — three stages, chained — but three specific things in the AssemblyAI half of the code have moved. If you're building this today, use these instead.

The Python SDK import path and client construction changed

The repo uses import assemblyai as aai, sets a module-level aai.settings.api_key, and constructs aai.Transcriber(). The current pre-recorded quickstart imports from a versioned namespace and passes the key to the client directly:

import os

from assemblyai.prerecorded.v2 import Transcriber

transcriber = Transcriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])

transcript = transcriber.transcribe("https://assembly.ai/wildfires.mp3")
print(transcript.text)

Reading the key from an environment variable is the other upgrade here. The repo hard-codes placeholder strings, which is fine for a placeholder and a bad habit in anything you push.

The error check survives the migration almost unchanged — TranscriptStatus just moves to a top-level import:

from assemblyai import TranscriptStatus
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

config = TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    language_detection=True
)
transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

if transcript.status == TranscriptStatus.error:
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)

Note speech_models — plural, and a list. The singular speech_model parameter is deprecated for async transcription, and the old best and nano model identifiers are gone. Omit speech_models entirely and you get the default, ["universal-3-5-pro", "universal-2"]: Universal-3.5 Pro handles its 18 supported languages and anything else falls back to Universal-2's 99-language coverage. For a voice translator whose input could be any language, that default is exactly what you want.

Short clips have their own endpoint now

A voice translator records ten- to twenty-second clips. That's a perfect fit for the Sync API, which was added after this video and returns a transcript in a single call with no polling at all:

import os

from assemblyai.sync.v1 import SyncTranscriber

transcriber = SyncTranscriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])
result = transcriber.transcribe("./sample.wav")
print(result.tex

The constraints are worth reading before you swap it in: it accepts WAV or raw PCM audio between 80 milliseconds and 120 seconds, takes a local file path or raw bytes but not a URL, and runs on Universal-3.5 Pro across 19 languages. Failures raise SyncTranscriptError rather than returning an error status, so the error-handling branch looks different. Anything over two minutes still belongs on the pre-recorded API.

You can delete the translation dependency

This is the big one. Translation is now a built-in Speech Understanding feature — 86 target languages, requested as part of the transcription call. The entire text_translation function and the translate package can go:

data = {
  "audio_url": audio_url,
  "language_detection": True,
  "speaker_labels": True,  # Enable speaker labels
  "speech_understanding": {
    "request": {
      "translation": {
        "target_languages": ["es", "de"],  # Translate to Spanish and German
        "formal": True  # Use formal language style
      }
    }
  }
}

The response comes back with a translated_texts object keyed by language code, which you feed straight into your text-to-speech calls:

print("--- Translations ---")
for language_code, translated_text in transcript['translated_texts'].items():
  print(f"{language_code.upper()}:")
  print(translated_text[:200] + "...\n")

One request instead of six, no third-party translation provider, and a formal flag that controls whether you get formal pronouns and grammatical forms — which matters more than you'd think when your cloned voice is the one saying it. There's also force_translation for the case where the detected source language matches a target and translation would otherwise be skipped. Full parameter list is in the Translation docs.

If you'd rather keep the two stages separate — say you already have transcripts and only need translations — the same config can be posted against an existing transcript ID. And if your use case is live rather than batch, our real-time translation service in JavaScript shows the streaming variant of this pipeline.

What this pattern is actually for

Misra's list of ideas from the video is a good starting point: recording a WhatsApp voice note in your friend's language, or using your own cloned voice as a pronunciation model when you're learning a language, because imitating yourself is easier than imitating a stranger. Samsung shipped live call translation around the same time, though as she points out, that generates a synthetic voice rather than yours.

The commercial version of this is media localization, and it lives or dies on the accuracy of stage one. Ollang, which localizes video content at scale, put it plainly:

"The 30-40% reduction in speech-to-text errors has significantly improved our production efficiency and client satisfaction. We've achieved industry-leading word error rates for non-English audio, which is critical for serving our enterprise clients."

— Ebru Yildirim, Founder & CEO, Ollang

That's the thing this toy project teaches that a diagram can't: errors compound in one direction. A misheard word in stage one becomes a mistranslated phrase in stage two and then gets spoken confidently, in your voice, in stage three. Nothing downstream can recover it. Which is why the boring stage — the transcription — is the one worth spending your accuracy budget on, and why picking the right Python speech recognition approach matters more than picking the right voice model.

From here, the natural next builds are a live version using multilingual streaming transcription instead of batch, or a conversational one — our real-time AI voice bot in Python is the same three stages with an LLM wedged in the middle. And if you're weighing which voice model to clone with, the top text-to-speech APIs in 2026 compares the field.

Frequently asked questions

How do I convert voice to text using Python?

Install the AssemblyAI Python SDK with pip install assemblyai, create a Transcriber with your API key, and call transcribe() with a local file path or a URL. The SDK handles uploading, submitting, and polling in that single call, so a working script is about five lines. Files can run up to 10 hours and 5 GB; clips under two minutes can use the Sync API for a millisecond-latency response instead.

What is the best way to build an AI voice translator in Python?

Chain three stages: speech-to-text, translation, then text-to-speech. Transcribe the recording with AssemblyAI, request translations in the same API call using the built-in Translation feature, then send each translated string to a text-to-speech provider — using a cloned voice if you want the output to sound like the original speaker. Gradio gives you a browser UI with a microphone input and audio outputs in about 15 lines.

Can I translate speech without a separate translation API?

Yes. AssemblyAI's Translation feature translates transcripts into 86 languages as part of the transcription request, so you don't need a separate machine translation provider. Add a speech_understanding.request.translation object with your target_languages, and the response includes a translated_texts field keyed by language code. You can also translate an existing transcript by ID.

Is there a live voice translator, or does this only work on recordings?

Both are possible, but they use different APIs. The project in this guide is batch: you finish recording, then the pipeline runs. For live translation you'd stream audio over the WebSocket API with Universal-3.5 Pro Realtime, which returns partial and final transcripts in a few hundred milliseconds and supports mid-sentence code switching across 18 languages, then translate and synthesize each completed turn.

How much audio do I need to clone my voice?

ElevenLabs' instant voice cloning needs roughly one minute of clean audio, and its professional cloning tier requires at least 30 minutes. In the original tutorial, about 33 minutes produced the clips in the demo, and the recommended optimum is closer to three hours for a near-flawless clone. Professional cloning is a paid ElevenLabs feature; AssemblyAI handles the transcription and translation stages, not voice generation.

What does it cost to run an AI voice translator?

AssemblyAI transcription is billed per second with no minimums or upfront commitments — Universal-3.5 Pro is $0.21/hour of audio and Universal-2 is $0.15/hour, so ten-second test clips cost fractions of a cent. Translation is priced as a Speech Understanding feature; text-to-speech is billed separately by your voice provider. Current rates are listed on the AssemblyAI pricing page.

‍