Build a voice research agent with Render Workflows and AssemblyAI

Building voice interfaces for complex tasks is hard. Speech input and output need to feel instant, but the work happening in the background — LLM calls, multi-stage search, synthesis — can take real time. If you block the voice channel waiting on tool execution, you end up with awkward pauses and brittle sessions.

This tutorial walks through a reference architecture that solves this problem by separating the voice channel from background orchestration. You'll use AssemblyAI's Universal-3.6 Pro Realtime speech-to-text, LLM Gateway and Text-to-Speech, plus Render Workflows, Mastra agents, and You.com search to build a voice research experience that reliably completes in under 60 seconds.

All the code for this project is available on GitHub. The voice-channel code in this post is newer than the repo’s src/assemblyai/voice-agent.ts, so swap it in when you clone.

Why not just let the LLM handle everything?

The typical approach to voice-first apps looks like this:

Stream audio from the browser to a voice API
The voice API calls an LLM
The LLM runs tool calls (search, extract, synthesize) directly

This works fine for simple commands like "what time is it?" But for anything heavier, it falls apart quickly. A single LLM turn blocked by multiple search calls stalls the voice channel. One failed search can drop the whole session. And you get no durable execution record, no retry logic, no way to replay a failed stage.

The fix is to separate the voice channel from the orchestration. The voice agent stays in its own lightweight task. Everything else — classification, planning, search, synthesis, verification — runs as a chain of discrete, retry-able workflow tasks with their own compute, timeouts, and logs.

Architecture overview

The demo runs two services on Render:

Web service — manages sessions, connects browser WebSockets to workflow tasks via a reverse-WS broker, and streams real-time progress via Server-Sent Events (SSE).

Workflow service — executes a chain of isolated tasks: voiceSession, research, classify_ask, plan_queries, search_branch, synthesize, and verify.

When a user opens the mic, the Web service dispatches a voiceSession task. That task opens streaming sessions to AssemblyAI’s Universal-3.6 Pro Realtime and Text-to-Speech, tunnels audio to and from the browser, and — when the user asks a research question — triggers the rest of the research pipeline.

Render workflow illustration as described in the architecture above

Orchestrating with Render workflows

The Web service uses the Render SDK to kick off the root task:

// src/render/workflow-dispatcher.ts
import { Render } from "@renderinc/sdk";

export function createWorkflowDispatcher(
  config: WorkflowDispatcherConfig
): WorkflowDispatcher {
  process.env.RENDER_API_KEY = config.apiKey;
  const render = new Render();

  return {
    async startVoiceSession(sessionId, taskToken, publicWebUrl) {
      const started = await render.workflows.startTask(
        `${config.workflowSlug}/voiceSession`,
        [sessionId, taskToken, publicWebUrl]
      );
      const runId = started.taskRunId;
      if (!runId) throw new AppError("UPSTREAM_WORKFLOW", "missing taskRunId");
      return runId;
    },
    // ... cancelTaskRun(), used by the session-cleanup daemon
  };
}

Inside voiceSession, each subsequent stage is defined as its own workflow task with a dedicated compute plan, timeout, and retry policy:

// src/render/tasks/mastra/classify-ask.ts
import { task } from "@renderinc/sdk/workflows";

export const classify_ask = task(
  {
    name: "classify_ask",
    plan: "starter",
    timeoutSeconds: 30,
    retry: { maxRetries: 2, waitDurationMs: 500, backoffScaling: 1.5 },
  },
  async function classify_ask(
    sessionId: string,
    topic: string
  ): Promise<{ shape: AskShape }> {
    // ...
  }
);

// src/render/tasks/mastra/synthesize.ts
export const synthesize = task(
  {
    name: "synthesize",
    plan: "starter",
    timeoutSeconds: 120,
    retry: { maxRetries: 1, waitDurationMs: 1_000, backoffScaling: 1.5 },
  },
  async function synthesize(
    sessionId: string,
    briefingId: string,
    topic: string,
    branches: BranchResult[],
    shape: AskShape = "narrative"
  ): Promise<SynthesizeResult> {
    // ...
  }
);

The classifier gets a tight 30-second timeout and the synthesizer gets 120 seconds. Every stage in the repo runs on the starter plan, so moving one stage to a bigger instance is a one-line change to its plan. Each stage fails independently — with its own logs, latency metrics, and replay options in the Render dashboard. No custom queuing infrastructure required.

The reverse WebSocket tunnel

The voiceSession task opens WebSocket connections in two directions:

Out to AssemblyAI: Universal-3.6 Pro Realtime for speech in, Text-to-Speech for speech out
One back to the Web service broker

The broker pairs the browser's WebSocket with the task's WebSocket using the session ID and token, creating a two-way audio tunnel:

// src/server.ts: route each WebSocket upgrade by path
server.on("upgrade", (req, socket, head) => {
  if (!req.url) {
    socket.destroy();
    return;
  }
  const url = new URL(req.url, "http://x");
  const sessionId = url.searchParams.get("sessionId");
  if (!sessionId) {
    socket.destroy();
    return;
  }

  if (url.pathname === "/ws/client") {
    wss.handleUpgrade(req, socket, head, (ws) => {
      broker.registerClient(sessionId, ws);
    });
    return;
  }

  if (url.pathname === "/ws/task") {
    const token = url.searchParams.get("token") ?? "";
    wss.handleUpgrade(req, socket, head, (ws) => {
      const ok = broker.registerTask(sessionId, token, ws);
      if (!ok) ws.close();
    });
    return;
  }

  socket.destroy();
});

// src/render/session-broker.ts: only the task holding the session's token can pair
registerTask(sessionId, token, ws) {
  const slot = slots.get(sessionId);
  if (!slot || slot.token !== token) {
    try { ws.close(); } catch { /* noop */ }
    return false;
  }
  slot.taskWS = ws;

  ws.on("message", (raw) => {
    const s = slots.get(sessionId);
    if (!s) return;
    const text = toText(raw);
    if (s.clientWS?.readyState === OPEN) {
      s.clientWS.send(text);
    } else {
      s.taskToClientBuffer.push(text);
    }
  });
  ws.on("close", () => cleanup(sessionId));
  ws.on("error", () => cleanup(sessionId));

  flushBuffers(sessionId);
  return true;
},

On each finished turn from Universal-3.6 Pro Realtime, the task sends the transcript to LLM Gateway with a single research tool. When the model calls it, the task handles the call directly — no extra round-trip to the browser. It launches the research chain, waits for a briefing, then speaks it through an AssemblyAI Text-to-Speech session. After each reply it passes what the agent said back to the transcriber as agent_context, so the next user turn is transcribed with both sides of the conversation in context:

// src/assemblyai/voice-channel.ts
import { AssemblyAI, type ChatCompletionMessage } from "assemblyai";

const client = new AssemblyAI({ apiKey: opts.apiKey });

// Speech in: Universal-3.6 Pro Realtime, at the browser's 24 kHz PCM16.
const stt = client.streaming.transcriber({
  speechModel: "universal-3-6-pro",
  sampleRate: 24_000,
  agentContext: opts.greeting,
});
// Speech out: AssemblyAI TTS, which also streams 24 kHz PCM16 by default.
const tts = client.tts.session({ voiceId: opts.voiceId, sampleRate: 24_000 });
await Promise.all([stt.connect(), tts.connect()]);

const history: ChatCompletionMessage[] = [
  { role: "system", content: opts.systemPrompt },
];

async function say(text: string): Promise<void> {
  opts.onBrowserMessage({ type: "transcript", role: "assistant", text, final: true });
  for (const sentence of text.match(/[^.!?]+[.!?]*\s*/g) ?? [text]) {
    for await (const chunk of tts.speak(sentence)) {
      opts.onBrowserMessage({
        type: "audio",
        audio: Buffer.from(chunk).toString("base64"),
      });
    }
  }
  // Tell the STT model what the agent just said, so the next turn is
  // transcribed in context.
  stt.updateConfiguration({ agent_context: text.slice(-1_750) });
}

async function respond(userText: string): Promise<void> {
  history.push({ role: "user", content: userText });
  const res = await client.llmGateway.chatCompletions({
    model: "claude-haiku-4-5-20251001",
    messages: history,
    tools: TOOLS, // one "research" function that takes a topic
    max_tokens: 300,
  });
  const message = res.choices[0]?.message;
  const call = (message?.tool_calls as ToolCall[] | undefined)?.find(
    (c) => c.function.name === "research"
  );

  if (call) {
    const { topic } = JSON.parse(call.function.arguments) as { topic?: string };
    await say("On it. Give me about a minute.");
    const briefing = await opts.onResearch(topic?.trim() || userText);
    history.push({ role: "assistant", content: briefing });
    await say(briefing);
    return;
  }

  const reply = typeof message?.content === "string" ? message.content : "";
  if (reply) {
    history.push({ role: "assistant", content: reply });
    await say(reply);
  }
}

let queue = Promise.resolve();
stt.on("turn", (turn) => {
  if (!turn.transcript) return;
  opts.onBrowserMessage({
    type: "transcript",
    role: "user",
    text: turn.transcript,
    final: turn.end_of_turn,
  });
  if (!turn.end_of_turn) return;
  queue = queue
    .then(() => respond(turn.transcript))
    .catch((err: unknown) =>
      opts.onBrowserMessage({ type: "error", message: String(err) })
    );
});

Shape-aware research with Mastra

Not all questions are the same. "Tell me about bonobos" needs prose with citations. "List every tribe in the Bible" needs a complete enumeration. Running a fixed-strategy pipeline for both is wasteful.

The first stage uses a lightweight Mastra agent to classify the user's question into one of five shapes: narrative, enumeration, comparison, specific, or recent.

// src/render/tasks/mastra/classify-ask.ts
const shapeSchema = z.enum([
  "narrative",
  "enumeration",
  "comparison",
  "specific",
  "recent",
]);

const agent = classifierAgent(config.ANTHROPIC_MODEL);
const result = await agent.generate(`User ask: "${topic}"\n\nClassify. JSON only.`);
const text = (result as { text?: string }).text ?? "";
const shape = parseShape(text);

function parseShape(text: string): AskShape {
  const fenced = text.match(/```(?:json)?\s*([\s\S]*?)```/);
  const raw = fenced ? fenced[1]!.trim() : text.trim();
  try {
    const parsed = JSON.parse(raw) as { shape?: unknown };
    const result = shapeSchema.safeParse(parsed.shape);
    if (result.success) return result.data;
    throw new Error("shape not in enum");
  } catch (err) {
    logger.warn(
      { err, raw: text.slice(0, 200) },
      "classify_ask: unparseable, defaulting to narrative"
    );
    return "narrative";
  }
}

The planner uses the shape to determine how many search queries to run — up to 40 for enumeration, fewer for narrative. The synthesizer then uses a shape-specific system prompt: enumeration gets "return a complete list with one item per line", narrative gets "weave sources into prose with inline citations".

Enforcing a hard 60-second deadline

Long wait times kill demos. The research chain uses a racePartial helper that collects search results until an AbortSignal.timeout fires, then proceeds with whatever it has:

// src/render/tasks/research.ts
const OVERALL_BUDGET_MS = 55_000; // 55s leaves ~5s for broker + browser render
const SYNTH_RESERVE_MS = 12_000;   // minimum runway we need to actually write the briefing

const deadline = Date.now() + OVERALL_BUDGET_MS;
const remaining = () => Math.max(0, deadline - Date.now());

const searchBudget = Math.max(5_000, remaining() - SYNTH_RESERVE_MS);
const branches = await racePartial(
  plan.queries.map((q) =>
    Promise.resolve(search_branch(sessionId, q.angle, q.query, q.tier))
  ),
  searchBudget
);

async function racePartial<T>(
  promises: Promise<T>[],
  budgetMs: number
): Promise<T[]> {
  const settled: Array<T | typeof PENDING> = promises.map(() => PENDING);
  const allDone = Promise.all(
    promises.map((p, i) =>
      p.then((r) => {
        settled[i] = r;
      }).catch(() => {
        /* failed branch contributes nothing */
      })
    )
  );
  await Promise.race([allDone, timer(budgetMs)]);
  return settled.filter((r): r is T => r !== PENDING);
}

const PENDING = Symbol("pending");

function timer(ms: number): Promise<void> {
  return new Promise<void>((resolve) => {
    const signal = AbortSignal.timeout(ms);
    if (signal.aborted) resolve();
    else signal.addEventListener("abort", () => resolve(), { once: true });
  });
}

The 12_000 ms reserve keeps runway for the synthesize pass that runs after search completes. Verify only runs if at least 6 seconds are left after that.

If 7 out of 12 search branches complete before the deadline (12 is illustrative — the planner can schedule up to 40 for enumeration-type queries), the synthesizer proceeds with those 7 sources.

Verification and retry

Before the briefing reaches text-to-speech, a verifier agent checks it against the original request:

// src/render/tasks/mastra/verify.ts
const agent = verifierAgent(config.ANTHROPIC_MODEL, shape);
const prompt = [
  `User request: "${topic}"`,
  ``,
  `Briefing:`,
  briefingText.slice(0, 8_000),
  ``,
  `Evaluate. JSON only.`,
].join("\n");

const result = await agent.generate(prompt);
const text = (result as { text?: string }).text ?? "";
const parsed = parseVerdict(text); // { passes, reason, feedback }

// src/render/tasks/research.ts: inside the plan -> search -> synthesize loop
const verdict = await verify(sessionId, topic, result.content, shape);

if (verdict.passes || attempt >= 1) break;

// Retry only if a full second pass could fit in what's left.
if (remaining() < RETRY_RESERVE_MS) {
  logger.warn(
    { sessionId, remainingMs: remaining() },
    "verify failed but no budget for retry — shipping current briefing"
  );
  break;
}

attempt += 1;
feedback = verdict.feedback || verdict.reason || "";

The system allows one retry. If the verifier still isn't satisfied, the briefing ships as a degraded answer. A less-than-perfect response in 60 seconds is better than silence.

Real-time progress streaming

Each task publishes phase events as it runs, and the Web service streams them to the browser over SSE:

// Each task publishes phase events, e.g. src/render/tasks/mastra/classify-ask.ts
await events.publish({
  sessionId,
  at: Date.now(),
  kind: "ask.classified",
  shape,
});

// src/routes.ts: the Web service streams them to the browser
app.get("/api/sessions/:id/events", (c) => {
  const sessionId = c.req.param("id");
  return streamSSE(c, async (stream) => {
    const unsubscribe = deps.events.subscribe(sessionId, (event) => {
      stream
        .writeSSE({
          event: "phase",
          data: JSON.stringify(event),
          id: String(event.at),
        })
        .catch(() => {});
    });
    stream.onAbort(() => unsubscribe());
    await new Promise<void>(() => {});
  });
});

The frontend subscribes to /api/sessions/:id/events and updates a live activity feed — no polling required.

Handling concurrent sessions

To keep the demo stable at scale, a few guardrails are in place:

Concurrency cap — POST /api/start returns 503 AT_CAPACITY when 100 active sessions are running
Session TTL — each session expires 15 minutes after creation
Cleanup daemon — a loop runs every 60 seconds, cancels expired workflow tasks via render.workflows.cancelTaskRun(), and marks sessions closed in Postgres
Token-based routing — / redirects to /s/{token}, so a browser reload resumes the existing session instead of consuming new capacity

Deploying on Render

The repo includes a render.yaml Blueprint for the Web service and Postgres database. For the Workflow service, you'll create it separately:

Use the Deploy to Render button in the GitHub README to provision the Web service and database
Create a new Workflow service in the dashboard, pointing to the same repo with start command node dist/render/tasks/index.js
Set the required environment variables on both services: ASSEMBLYAI_API_KEY (LLM Gateway needs a paid AssemblyAI account), ANTHROPIC_API_KEY, YOU_API_KEY, and RENDER_API_KEY

The included preDeployCommand will run npm run migrate on each deploy—adjust this to fit your own migration setup.

What you can take from this

This architecture is worth adapting for any voice application that needs to do real work in the background. A few patterns that transfer well:

Separate voice from orchestration. Keep the voice channel in its own task. Don't block it with tool execution.
Use workflow tasks for each pipeline stage. Independent timeouts, retries, and logs make debugging much easier than a monolithic pipeline.
Shape-aware planning improves output quality. Classifying the request before searching means your synthesizer gets the right prompting for the right kind of answer.
Hard deadlines with partial results beat open-ended waiting. Users will forgive an incomplete answer. They won't forgive silence.

👉 Check out the full source on GitHub and the Universal-3.6 Pro Realtime, LLM Gateway tool calling and Text-to-Speech docs to get started.