Node.js voice agent with AssemblyAI Universal-3.6 Pro Realtime

Build a real-time voice agent in Node.js using the AssemblyAI Universal-3.6 Pro Realtime model (universal-3-6-pro) for speech-to-text — no Python required, no heavy framework dependencies.

Two modes in one repo:

1. Terminal agent (src/agent.js) — mic input via mic, plays TTS audio in your terminal
2. Browser server (src/server.js) — Node.js WebSocket server with a browser UI using getUserMedia

Why AssemblyAI Universal-3.6 Pro Realtime for Node.js?

*WER figures from AssemblyAI's English voice-agent benchmark of 12,460 scripted voice-agent scenarios; latency from Pipecat's open STT benchmark (time from end of speech to final transcript).*

Universal-3.6 Pro Realtime's turn detection eliminates the need for a separate VAD library — the model combines semantic context with voice activity to decide when a speaker has actually finished, not just when they've gone quiet.

Heads up on model IDs: this tutorial uses universal-3-6-pro, the current streaming default. universal-3-5-pro stays available if you need to pin the previous model; if you're still on the legacy u3-rt-pro ID, switch to universal-3-6-pro.

Quick start

git clone https://github.com/kelsey-aai/voice-agent-nodejs-assemblyai
cd voice-agent-nodejs-assemblyai

npm install
cp .env.example .env
# Edit .env with your API keys

Terminal agent

npm start
# Speak into your mic — Ctrl+C to quit

Browser agent

npm run server
# Open http://localhost:3000

AssemblyAI WebSocket URL

const AAI_WS_URL =
  `wss://streaming.assemblyai.com/v3/ws` +
  `?speech_model=universal-3-6-pro` +
  `&encoding=pcm_s16le` +
  `&sample_rate=16000` +
  `&min_turn_silence=300` +   // ms of silence before the speculative end-of-turn
check
  `&max_turn_silence=1500` +  // hard ceiling: turn ends after this much silence 
regardless
  `&token=${ASSEMBLYAI_API_KEY}`;

Turn detection

AssemblyAI v3 uses three event types. Handle them like this:

ws.on("message", async (data) => {
  const msg = JSON.parse(data.toString());

  if (msg.type === "Begin") {
    console.log(`Session: ${msg.id}`);
  }

  if (msg.type === "Turn" && !msg.end_of_turn) {
    process.stdout.write(`\r${msg.transcript}`);
  }

  if (msg.type === "Turn" && msg.end_of_turn) {
    const reply = await generateResponse(msg.transcript);
    await speak(reply);
  }
});

Sending audio

Browser (getUserMedia + ScriptProcessor):

processor.onaudioprocess = (e) => {
  const float32 = e.inputBuffer.getChannelData(0);
  const int16 = new Int16Array(float32.length);
  for (let i = 0; i < float32.length; i++) {
    int16[i] = Math.max(-32768, Math.min(32767, Math.round(float32[i] * 32767)));
  }
  ws.send(int16.buffer);
};

Terminal (mic package):

const micStream = micInstance.getAudioStream();
micStream.on("data", (chunk) => {
  aaiWs.send(chunk); // raw PCM s16le bytes
});

Tuning turn detection

Universal-3.6 Pro Realtime uses end-of-turn detection that combines semantic context with voice activity. Steer it with a high-level mode preset, then fine-tune the two silence windows:

Note: end_of_turn_confidence_threshold does not apply to Universal-3.6 Pro Realtime — it belongs to the older universal-streaming models. Turn detection here is handled by the model, controlled by the silence windows above.

Keyterm prompting (mid-session)

Inject domain-specific vocabulary after the session starts without restarting:

ws.send(JSON.stringify({
  type: "UpdateConfiguration",
  keyterms: ["AssemblyAI", "Universal-3.6 Pro Realtime", "your-product-name"],
}));

Conversation context (mid-session)

Universal-3.6 Pro Realtime can transcribe each user turn in the context of what your agent just said — after your agent asks a question, the model is primed for the answer, which sharpens short replies and spelled-out entities. Push your agent's last reply with the same UpdateConfiguration message after each agent turn:

ws.send(JSON.stringify({
  type: "UpdateConfiguration",
  agent_context: "Thanks! What's the email address on your account?",
}));

Requirements

• Node.js 18+
• npm install installs: ws, openai, elevenlabs, mic, dotenv
• macOS: afplay (built-in) for audio playback
• Linux: aplay or mpg123 for audio playback

Deploy to Railway, Render, or Fly.io

# Set environment variables in the platform dashboard, then:
npm run server

The browser server is stateless per-connection — each WebSocket session has its own AssemblyAI connection and conversation history.

Resources

• Universal-3.6 Pro Realtime streaming API reference
• AssemblyAI Node.js SDK
• AssemblyAI Playground

Frequently asked questions

How do I build a voice agent in Node.js?

Open a WebSocket from Node.js to AssemblyAI's streaming endpoint (wss://streaming.assemblyai.com/v3/ws?speech_model=universal-3-6-pro), stream microphone or browser audio as PCM, and act on the Turn messages the model sends back — when end_of_turn is true, pass the transcript to your LLM and send the reply to a TTS provider. You don't need a framework or a separate VAD library: Universal-3.6 Pro Realtime handles turn detection itself.

What's the best streaming speech-to-text API for a JavaScript/Node.js voice agent?

AssemblyAI's Universal-3.6 Pro Realtime is a strong fit for Node.js voice agents: it's a plain WebSocket (no SDK required), posts a 5.19% word error rate on AssemblyAI's English voice-agent benchmark, delivers final transcripts a median 307 ms after the speaker stops on Pipecat's open benchmark, and includes built-in turn detection, keyterm prompting, and Context Carryover. You connect with the ws package and handle Begin / Turn messages directly.

How do I connect to the AssemblyAI streaming WebSocket from Node.js?

Use the ws package to open wss://streaming.assemblyai.com/v3/ws with speech_model=universal-3-6-pro, encoding=pcm_s16le, and sample_rate=16000, authenticating with your API key. Stream 16 kHz PCM audio as binary messages, and parse incoming JSON messages by type: Begin (session start), Turn (partial and final transcripts), and end-of-turn signals via the end_of_turn field.

How does turn detection work without a VAD library in Node.js?

Universal-3.6 Pro Realtime does turn detection server-side, combining semantic context with voice activity to decide whether the speaker is finished, so you don't need to run a client-side VAD. You control the timing with min_turn_silence and max_turn_silence in the WebSocket URL, or shift the overall balance with the mode preset (min_latency / balanced / max_accuracy).

How do I update keyterms or context mid-conversation in Node.js?

Send an UpdateConfiguration message over the open WebSocket — no reconnect needed. Pass keyterms to boost domain vocabulary, or agent_context to feed the model your agent's most recent reply so it transcribes the next user turn in context. Both take effect on the following turn.