Node.js voice agent with AssemblyAI Universal-3.6 Pro Realtime

Build a real-time voice agent in Node.js using the AssemblyAI Universal-3.6 Pro Realtime model (universal-3-6-pro) for speech-to-text and AssemblyAI Text-to-Speech for the agent’s voice — no Python required, no heavy framework dependencies.

Two modes in one repo:

1. Terminal agent (src/agent.js) — mic input via mic, plays TTS audio in your terminal
2. Browser server (src/server.js) — Node.js WebSocket server with a browser UI using getUserMedia

Why AssemblyAI Universal-3.6 Pro Realtime for Node.js?

*WER figures from AssemblyAI's English voice-agent benchmark of 12,460 scripted voice-agent scenarios; latency from Pipecat's open STT benchmark (time from end of speech to final transcript).*

Universal-3.6 Pro Realtime's turn detection eliminates the need for a separate VAD library — the model combines semantic context with voice activity to decide when a speaker has actually finished, not just when they've gone quiet.

Heads up on model IDs: this tutorial uses universal-3-6-pro, the current streaming default. universal-3-5-pro stays available if you need to pin the previous model; if you're still on the legacy u3-rt-pro ID, switch to universal-3-6-pro.

Quick start

git clone https://github.com/kelsey-aai/voice-agent-nodejs-assemblyai
cd voice-agent-nodejs-assemblyai

npm install
cp .env.example .env
# Edit .env with your API keys

Terminal agent

npm start
# Speak into your mic — Ctrl+C to quit

Browser agent

npm run server
# Open http://localhost:3000

AssemblyAI WebSocket URL

const AAI_WS_URL =
  `wss://streaming.assemblyai.com/v3/ws` +
  `?speech_model=universal-3-6-pro` +
  `&encoding=pcm_s16le` +
  `&sample_rate=16000` +
  `&min_turn_silence=300` +   // ms of silence before the speculative end-of-turn check
  `&max_turn_silence=1500` +  // hard ceiling: turn ends after this much silence regardless
  `&token=${ASSEMBLYAI_API_KEY}`;

Turn detection

AssemblyAI v3 uses three event types. Handle them like this:

ws.on("message", async (data) => {
  const msg = JSON.parse(data.toString());

  if (msg.type === "Begin") {
    console.log(`Session: ${msg.id}`);
  }

  if (msg.type === "Turn" && !msg.end_of_turn) {
    process.stdout.write(`\r${msg.transcript}`);
  }

  if (msg.type === "Turn" && msg.end_of_turn) {
    const reply = await generateResponse(msg.transcript);
    await speak(reply);
  }
});

Sending audio

Browser (getUserMedia + ScriptProcessor):

processor.onaudioprocess = (e) => {
  const float32 = e.inputBuffer.getChannelData(0);
  const int16 = new Int16Array(float32.length);
  for (let i = 0; i < float32.length; i++) {
    int16[i] = Math.max(-32768, Math.min(32767, Math.round(float32[i] * 32767)));
  }
  ws.send(int16.buffer);
};

Terminal (mic package):

const micStream = micInstance.getAudioStream();
micStream.on("data", (chunk) => {
  aaiWs.send(chunk); // raw PCM s16le bytes
});

Speak replies with AssemblyAI Text-to-Speech

The turn handler above ends with speak(reply). Implement it with the AssemblyAI Text-to-Speech API, so the whole agent runs on one AssemblyAI key apart from your LLM. Its assemblyai-tts-1 model is built for conversational voice agents and trained on millions of hours of audio, including real conversational data. It delivers frontier-level quality, runs faster and costs less than other production voice agent TTS models, and has unlimited concurrency. Install the JavaScript SDK and SoX:

npm install assemblyai   # 4.42.0 or later
# SoX plays the raw audio stream: brew install sox (macOS) or sudo apt install sox (Linux)

Open one text-to-speech session when the agent starts and reuse it for every reply. speak() streams the reply’s audio as raw 24 kHz, 16-bit mono PCM while it’s synthesized, so the terminal agent pipes each chunk straight into SoX and playback starts with the first chunk. Set the voice explicitly; browse others in the voices guide.

// Text-to-speech for the terminal agent: one AssemblyAI session for the whole conversation
import { spawn } from "node:child_process";
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const tts = client.tts.session({
  voiceId: "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", // Audrey
  language: "english",
});
await tts.connect(); // waits for the Begin frame

// The session closes after two idle minutes; keep it open between turns
setInterval(() => tts.keepAlive(), 60_000);

let player = null; // the SoX process playing the current reply

async function speak(text) {
  // SoX reads raw 24 kHz, 16-bit, mono PCM from stdin and plays it
  const proc = spawn("play", ["-q", "-t", "raw", "-r", "24000", "-e", "signed", "-b", "16", "-c", "1", "-"], {
    stdio: ["pipe", "ignore", "ignore"],
  });
  player = proc;
  proc.stdin.on("error", () => {}); // the player was stopped by a barge-in
  const finished = new Promise((resolve) => proc.on("close", resolve));

  // speak() sends the text, flushes it and yields the audio as it is synthesized.
  // The loop ends at the reply's FlushDone, or at once after cancel().
  for await (const chunk of tts.speak(text)) {
    proc.stdin.write(chunk);
  }
  proc.stdin.end();
  await finished; // wait until the reply has finished playing
  if (player === proc) player = null;
}

Barge-in. To let the user interrupt, keep the mic streaming while the agent talks (use headphones, or the mic will hear the speakers) and cancel the reply when a real turn starts. cancel() stops synthesis and ends the speak() loop at once; killing the player stops the audio it already holds. Ignoring one-word turns keeps a “mhm” from cutting the agent off.

// Barge-in: the user talks over the agent
function bargeIn() {
  tts.cancel();   // stops synthesis and ends the speak() loop at once
  player?.kill(); // stops the audio already handed to the player
  player = null;
}

// In the AssemblyAI "message" handler, before the end_of_turn branch:
if (msg.type === "Turn" && player) {
  const words = msg.transcript.split(/\s+/).filter(Boolean).length;
  if (words >= 2) bargeIn(); // ignore one-word backchannels like "mhm"
}

When the agent shuts down, close the session so it ends cleanly:

// Ctrl+C: end the text-to-speech session cleanly
process.on("SIGINT", async () => {
  await tts.close(); // sends Terminate and waits for the session summary
  process.exit(0);
});

Browser agent. The browser server can speak replies the same way. Open one session per browser connection on the server, where the API key already lives, and forward each chunk to the page as a binary WebSocket message:

// src/server.js: speak each reply to the browser that asked for it.
// One text-to-speech session per browser connection; the server keeps the API key.
const tts = client.tts.session({
  voiceId: "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", // Audrey
  language: "english",
});
await tts.connect();

for await (const chunk of tts.speak(reply)) {
  browserWs.send(chunk); // binary message: raw 24 kHz 16-bit PCM
}

The audio is raw PCM with no header, so an <audio> element can’t play it. Decode it on the page with the Web Audio API and schedule each chunk after the previous one:

// Browser page: set ws.binaryType = "arraybuffer", then call playPcm() for each binary message
const playCtx = new AudioContext(); // resume it inside a click handler
let playAt = 0;

function playPcm(arrayBuffer) {
  const pcm = new Int16Array(arrayBuffer);
  const buffer = playCtx.createBuffer(1, pcm.length, 24000);
  buffer.copyToChannel(Float32Array.from(pcm, (s) => s / 32768), 0);

  const src = playCtx.createBufferSource();
  src.buffer = buffer;
  src.connect(playCtx.destination);
  playAt = Math.max(playAt, playCtx.currentTime); // schedule after the previous chunk
  src.start(playAt);
  playAt += buffer.duration;
}

For streaming LLM tokens into the session with generate() and flush(), see the Text-to-Speech quickstart. A session closes after two idle minutes and lasts at most an hour; the limits page has the details.

Tuning turn detection

Universal-3.6 Pro Realtime uses end-of-turn detection that combines semantic context with voice activity. Steer it with a high-level mode preset, then fine-tune the two silence windows:

Note: end_of_turn_confidence_threshold does not apply to Universal-3.6 Pro Realtime — it belongs to the older universal-streaming models. Turn detection here is handled by the model, controlled by the silence windows above.

Keyterm prompting (mid-session)

Inject domain-specific vocabulary after the session starts without restarting:

ws.send(JSON.stringify({
  type: "UpdateConfiguration",
  keyterms: ["AssemblyAI", "Universal-3.6 Pro Realtime", "your-product-name"],
}));

Conversation context (mid-session)

Universal-3.6 Pro Realtime can transcribe each user turn in the context of what your agent just said — after your agent asks a question, the model is primed for the answer, which sharpens short replies and spelled-out entities. Push your agent's last reply with the same UpdateConfiguration message after each agent turn:

ws.send(JSON.stringify({
  type: "UpdateConfiguration",
  agent_context: "Thanks! What's the email address on your account?",
}));

Requirements

• Node.js 18+
• npm install installs: ws, openai, assemblyai, mic, dotenv
• SoX for audio playback: brew install sox (macOS) or sudo apt install sox (Linux)

Deploy to Railway, Render, or Fly.io

# Set environment variables in the platform dashboard, then:
npm run server

The browser server is stateless per-connection — each WebSocket session has its own AssemblyAI connection and conversation history.

Resources

• Universal-3.6 Pro Realtime streaming API reference
• AssemblyAI Node.js SDK
• AssemblyAI Text-to-Speech quickstart
• AssemblyAI Playground

Frequently asked questions

How do I build a voice agent in Node.js?

Open a WebSocket from Node.js to AssemblyAI's streaming endpoint (wss://streaming.assemblyai.com/v3/ws?speech_model=universal-3-6-pro), stream microphone or browser audio as PCM, and act on the Turn messages the model sends back — when end_of_turn is true, pass the transcript to your LLM and speak the reply with AssemblyAI Text-to-Speech through the JavaScript SDK’s client.tts.session(). You don't need a framework or a separate VAD library: Universal-3.6 Pro Realtime handles turn detection itself.

What's the best streaming speech-to-text API for a JavaScript/Node.js voice agent?

AssemblyAI's Universal-3.6 Pro Realtime is a strong fit for Node.js voice agents: it's a plain WebSocket (no SDK required), posts a 5.19% word error rate on AssemblyAI's English voice-agent benchmark, delivers final transcripts a median 307 ms after the speaker stops on Pipecat's open benchmark, and includes built-in turn detection, keyterm prompting, and Context Carryover. You connect with the ws package and handle Begin / Turn messages directly.

How do I connect to the AssemblyAI streaming WebSocket from Node.js?

Use the ws package to open wss://streaming.assemblyai.com/v3/ws with speech_model=universal-3-6-pro, encoding=pcm_s16le, and sample_rate=16000, authenticating with your API key. Stream 16 kHz PCM audio as binary messages, and parse incoming JSON messages by type: Begin (session start), Turn (partial and final transcripts), and end-of-turn signals via the end_of_turn field.

How does turn detection work without a VAD library in Node.js?

Universal-3.6 Pro Realtime does turn detection server-side, combining semantic context with voice activity to decide whether the speaker is finished, so you don't need to run a client-side VAD. You control the timing with min_turn_silence and max_turn_silence in the WebSocket URL, or shift the overall balance with the mode preset (min_latency / balanced / max_accuracy).

How do I update keyterms or context mid-conversation in Node.js?

Send an UpdateConfiguration message over the open WebSocket — no reconnect needed. Pass keyterms to boost domain vocabulary, or agent_context to feed the model your agent's most recent reply so it transcribes the next user turn in context. Both take effect on the following turn.