Voice cloning

​

Create a voice

• One person, speaking the voice’s language. Pick the language in the dialog and have the speaker read in it: a speaker reading English into a voice set to French is cloned on the wrong sounds.
• Ten to thirty seconds of speech. The dashboard uses up to thirty seconds of a recording under two minutes long, and shows when a recording is long enough. Read the script it offers, calm or excited, or your own text at a natural pace.
• Consent. You attest that the speaker consented to the clone, and your account must have accepted the biometric data terms on the dashboard’s data controls page.

​

Speak with it

export ASSEMBLYAI_API_KEY=<your-key>
export ASSEMBLYAI_VOICE_ID=<your-voice-id>

import os

import assemblyai as aai

config = aai.TTSConfig(voice_id=os.environ["ASSEMBLYAI_VOICE_ID"])

with aai.TTSSession(config, api_key=os.environ["ASSEMBLYAI_API_KEY"]) as session:
    c = session.begin.configuration
    print(f"voice_id {c['voice_id']}, resolved {c['resolved_voice']['voice_id']}")
    print(f"language {c['language']}, model {c['model']}")
    print(f"voice expires at {c['voice_expires_at']}")

    with open("cloned.pcm", "wb") as out:
        for chunk in session.speak("Hi, thanks for calling. How can I help you today?"):
            out.write(chunk)  # raw 24 kHz 16-bit mono PCM

import { createWriteStream } from "node:fs";
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });
const session = client.tts.session({ voiceId: process.env.ASSEMBLYAI_VOICE_ID });

const { configuration: c } = await session.connect();
console.log(`voice_id ${c.voice_id}, resolved ${c.resolved_voice.voice_id}`);
console.log(`language ${c.language}, model ${c.model}`);
console.log(`voice expires at ${c.voice_expires_at}`);

const out = createWriteStream("cloned.pcm");
for await (const chunk of session.speak("Hi, thanks for calling. How can I help you today?")) {
  out.write(chunk); // raw 24 kHz 16-bit mono PCM
}
out.end();
await session.close();

curl -sS --fail-with-body -D headers.txt https://tts.us.assemblyai.com/v1/speech \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -d '{"text": "Hi, thanks for calling. How can I help you today?", "voice_id": "'"$ASSEMBLYAI_VOICE_ID"'", "container": "wav"}' \
  -o cloned.wav

voice_id 01a139bf-78a0-7d18-9a4e-6c0f2b8d1e37, resolved 01a139bf-78a0-7d18-9a4e-6c0f2b8d1e37
language english, model assemblyai-tts-1
voice expires at 1797154212

​

Language and model

​

Data zones

​

Consent period

​

Renew consent

renew = aai.TTSConfig(voice_id=os.environ["ASSEMBLYAI_VOICE_ID"], extend_biometric_consent=True)

with aai.TTSSession(renew) as session:  # a session
    for chunk in session.speak("Consent renewed."):
        play(chunk)
print(session.termination.voice_expires_at)  # the expiry after the renewal

response = aai.TextToSpeech().synthesize("Consent renewed.", renew)  # or an HTTP request
print(response.voice_expires_at)

const session = client.tts.session({ voiceId: process.env.ASSEMBLYAI_VOICE_ID, extendBiometricConsent: true }); // a session
await session.connect();
for await (const chunk of session.speak("Consent renewed.")) play(chunk);
const summary = await session.close();
console.log(summary?.voice_expires_at); // the expiry after the renewal

const response = await client.tts.synthesize({ // or an HTTP request
  text: "Consent renewed.",
  voice_id: process.env.ASSEMBLYAI_VOICE_ID,
  extend_biometric_consent: true,
});
console.log(response.voiceExpiresAt);

curl -sS --fail-with-body -D headers.txt https://tts.us.assemblyai.com/v1/speech \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -d '{"text": "Consent renewed.", "voice_id": "'"$ASSEMBLYAI_VOICE_ID"'", "extend_biometric_consent": true, "stream": false}' \
  -o renewed.pcm
grep -i x-tts-voice-expires-at headers.txt

​

Errors

• voice_not_found (close 3012): no voice with this ID or slug that your account can use on this host. It does not exist, was deleted, belongs to another account, or belongs to the other data zone. Check the ID and the host.
• voice_expired (close 3012): the consent period ended. Create the voice again.
• voice_blocked (close 3012): the voice is blocked. Contact support.
• voice_not_paired (close 3014): the voice is not ready on the session’s model, usually because it was just created. Try again shortly; if it keeps failing, check the voice in the dashboard. The texts of this refusal are in Error handling.
• not_owner (close 3006): extend_biometric_consent was set on a voice another account owns. Leave it out.
• biometric_terms_not_accepted (close 3006, 403 over HTTP): your account has not accepted the biometric data terms. Accept them on the dashboard’s data controls page.
• invalid_parameter on language (close 3006): the language is not the voice’s own. Leave it out.
• model_not_found, model_retired (close 3006, 3014) and feature_not_enabled (close 3006, 400 over HTTP): a model_id the region does not know, a retired model, or a host where cloned voices are not available. Leave model_id out, and connect to tts.us.assemblyai.com or tts.eu.assemblyai.com.

​

Related

• Voices: the voice catalog and the AssemblyAI voices to start with.
• WebSocket API basics: how voice_id and extend_biometric_consent are checked at connect.
• Error handling: every close code and what to do about it.
• OpenAI compatibility: a cloned voice through the OpenAI SDKs.
• Cloud endpoints and data residency: the US and EU hosts.