Voice shopping has crossed an inflection point. Search by typing is being replaced by search by saying, and the products that win are the ones that can hear "do you have those hiking boots in a ten, the waterproof ones, under one fifty" and come back with an answer instead of a search results page.
That sentence is also a decent stress test. It has a size, a price ceiling, a product attribute, and an implied reference to something said earlier. Get any one of them wrong and you've either shown the customer the wrong boot or, worse, put the wrong boot in their cart.
So that's what we're building. A Python voice agent that handles product search, cart management, order tracking, and checkout, on three AssemblyAI pieces that share one API key: Universal-3.6 Pro Realtime for streaming speech-to-text, LLM Gateway for the model that decides which tool to call, and the Text-to-Speech API for Riley's voice. No orchestration framework. One Python file you can read top to bottom.
Why voice e-commerce is different from voice support
Voice support agents have a narrow job. Look up an account, answer a question, escalate. The vocabulary is bounded and the failure mode is usually "the customer repeats themselves."
Commerce isn't like that.
Entity accuracy is the product. Sizes, SKUs, colors, prices, order numbers, ZIP codes, the last four of a card. These are the tokens a shopping agent gets wrong most often, and they're the tokens where being wrong actually costs money. A support agent that mishears a name asks again. A shopping agent that mishears "size ten" as "size nine" ships the wrong box.
This is the reason the model underneath matters more here than almost anywhere else. Universal-3.6 Pro Realtime posts a 14.4% entity error rate on AssemblyAI's English voice-agent benchmark, which runs on 12,460 scripted voice-agent scenarios rather than clean read speech. Here's the full picture:
| Metric | Universal-3.6 Pro Realtime | Deepgram Flux EN | ElevenLabs Scribe v2 | Deepgram Nova-3 |
|---|---|---|---|---|
| Word error rate | 5.19% | 13.50% | 7.78% | 8.64% |
| Entity error rate | 14.4% | 30.1% | 18.5% | 26.1% |
| Names | 10.9% | 29.0% | 14.8% | 24.3% |
| Codes / IDs | 10.0% | 46.1% | 12.0% | 27.6% |
| Phone numbers | 2.4% | 11.5% | 3.4% | 4.5% |
Codes and IDs at 10.0% are your SKUs, order numbers, and confirmation codes. Phone numbers at 2.4% is your callback number and the last four of the card. Those two rows are most of what a checkout flow actually has to hear correctly.
Shopping conversations are exploratory. Nobody opens with a fully-formed request. They start with "I need something for a wet trail" and narrow down over four turns. The agent has to hold the accumulated constraints without making the customer restate them.
The stakes shift mid-conversation. Browsing is low stakes. "Place the order" is not. The same conversation has to be loose at the start and rigorous at the end, and the transition is invisible to the model unless you build it in. We'll handle that with a confirmation gate in the tool schema, not a prompt suggestion.
Your customers don't all speak one language cleanly. They switch mid-sentence, especially around product names and numbers. Universal-3.6 Pro Realtime covers 32 languages with mid-sentence code-switching built in rather than bolted on — Hinglish included — so "mujhe woh size ten black wale chahiye" transcribes as what was actually said instead of collapsing into whichever language you configured. If you know the language up front, commit to it with language_codes and take the accuracy gain.
Get accurate entity recognition on sizes, SKUs, order numbers, and addresses — the tokens that cost you money when they're wrong. Start with a free API key and clear docs.
What you'll build
Four workflows, one agent:
- Product search — natural language in, two or three spoken results out.
- Cart management — add, view, remove, with variant confirmation.
- Order tracking — look up a status and read a tracking number aloud without mangling it.
- Checkout assistance — a confirmation gate that won't fire on an ambiguous "sure, I guess."
The stack:
- Universal-3.6 Pro Realtime (universal-3-6-pro) — the streaming speech-to-text model, over a raw WebSocket at wss://streaming.assemblyai.com/v3/ws. Turn detection is built in: each finished utterance arrives as a Turn event with end_of_turn set. $0.45/hr; pricing has the details.
- LLM Gateway — an OpenAI-compatible chat completions API in front of Claude, GPT, Gemini, and more, on the same AssemblyAI API key. The seven shopping tools ride along on every request, and the model answers with tool calls your code runs.
- AssemblyAI Text-to-Speech — built for conversational voice agents and trained on millions of hours of audio, including real conversational data. It delivers frontier-level quality, runs faster and costs less than other production voice agent TTS models, and has unlimited concurrency. One session stays open for the whole conversation, and cancel() stops a reply the moment the customer cuts in.
- Python 3.9+, the assemblyai SDK (1.8.2 or later), websockets, pyaudio.
- A mocked catalog and order database — swap in Shopify, Commerce Cloud, or whatever you actually run.
If you're new to the pieces, the streaming quickstart and the text-to-speech quickstart cover the connection basics that this post moves quickly through.
Setup
pip install -U "assemblyai>=1.8.2" "websockets>=14" pyaudio python-dotenvCreate a .env file:
ASSEMBLYAI_API_KEY=your_api_key_hereGrab a key from the dashboard. That one key covers all three connections: the streaming speech-to-text WebSocket, LLM Gateway, and the text-to-speech session.
Everything below goes in one file, shop_assistant.py, in the order shown.
Step 1: Define the shopping tools
The agent's capabilities are just JSON schemas. Seven of them, in the OpenAI function format that LLM Gateway accepts, so the same list works with any tool-capable model you point it at.
import json
def tool(name, description, properties=None, required=None):
"""One OpenAI-style function tool, the format LLM Gateway accepts."""
return {
"type": "function",
"function": {
"name": name,
"description": description,
"parameters": {
"type": "object",
"properties": properties or {},
"required": required or [],
},
},
}
TOOLS = [
tool(
"search_products",
"Search the product catalog. Use whenever the customer describes "
"what they're looking for, even vaguely.",
{
"query": {
"type": "string",
"description": "What the customer is looking for, in their own words",
},
"max_price": {
"type": "number",
"description": "Price ceiling in dollars, if the customer gave one",
},
"size": {"type": "string", "description": "Requested size, if mentioned"},
"color": {"type": "string", "description": "Requested color, if mentioned"},
"limit": {"type": "integer", "description": "Max results to return. Default 3."},
},
["query"],
),
tool(
"get_product_details",
"Full details for one product: variants, stock, return policy.",
{"product_id": {"type": "string"}},
["product_id"],
),
tool(
"add_to_cart",
"Add a specific variant to the cart. Only call this after the customer "
"has confirmed size, color, and quantity.",
{
"product_id": {"type": "string"},
"variant_id": {"type": "string", "description": "Specific size/color variant"},
"quantity": {"type": "integer", "description": "Default 1"},
},
["product_id", "variant_id"],
),
tool("view_cart", "Read back the current cart contents and total."),
tool(
"remove_from_cart",
"Remove one line item from the cart.",
{"line_item_id": {"type": "string"}},
["line_item_id"],
),
tool(
"checkout",
"Place the order. ONLY call this when the customer has given an explicit, "
"unambiguous verbal confirmation. Pass their exact words.",
{
"confirmation_phrase": {
"type": "string",
"description": "The customer's exact confirming words, verbatim",
}
},
["confirmation_phrase"],
),
tool(
"track_order",
"Look up status and tracking for an existing order.",
{"order_id": {"type": "string"}},
["order_id"],
),
]Look at checkout for a second. confirmation_phrase is required, and its description says "verbatim." That's deliberate.
Most voice checkout bugs come from the model deciding that "yeah okay sure whatever" means yes. By forcing the agent to hand you the actual words it heard, you get to make that decision in Python, where you can reason about it, log it, and change it without touching a prompt. A prompt instruction is a request. A required parameter your function validates is a gate.
Step 2: Implement the backend (mocked)
Stub catalog, stub cart, stub orders. Replace all three with real calls when you wire this to a live store.
CATALOG = [
{
"product_id": "P-1042",
"name": "Ridgeline Waterproof Hiking Boot",
"price": 139.00,
"description": "Full-grain leather, seam-sealed, Vibram outsole.",
"return_policy": "60 days, unworn",
"variants": [
{"variant_id": "P-1042-10-BLK", "size": "10", "color": "black", "stock": 8},
{"variant_id": "P-1042-10-BRN", "size": "10", "color": "brown", "stock": 0},
{"variant_id": "P-1042-11-BLK", "size": "11", "color": "black", "stock": 3},
],
},
{
"product_id": "P-2213",
"name": "Trailhead Mid Waterproof",
"price": 118.00,
"description": "Lightweight synthetic upper, gusseted tongue.",
"return_policy": "60 days, unworn",
"variants": [
{"variant_id": "P-2213-10-GRY", "size": "10", "color": "grey", "stock": 14},
{"variant_id": "P-2213-11-GRY", "size": "11", "color": "grey", "stock": 6},
],
},
]
CART = []
ORDERS = {
"ORD-9981": {
"status": "in transit",
"carrier": "UPS",
"tracking": "1Z999AA10123456784",
"eta": "Thursday",
"items": ["Ridgeline Waterproof Hiking Boot, size 10, black"],
}
}
ACCEPTED_CONFIRMATIONS = [
"yes",
"yes please",
"place the order",
"go ahead",
"confirm",
"buy it",
]
def _find_product(product_id):
return next((p for p in CATALOG if p["product_id"] == product_id), None)
def _find_variant(product, variant_id):
return next((v for v in product["variants"] if v["variant_id"] == variant_id), None)
def run_tool(name: str, args: dict) -> dict:
"""Dispatch a tool call from the agent and return a JSON-serializable result."""
if name == "search_products":
query = args.get("query", "").lower()
results = []
for product in CATALOG:
haystack = f"{product['name']} {product['description']}".lower()
if not any(word in haystack for word in query.split() if len(word) > 3):
continue
if args.get("max_price") and product["price"] > args["max_price"]:
continue
available = product["variants"]
if args.get("size"):
available = [v for v in available if v["size"] == str(args["size"])]
if args.get("color"):
available = [v for v in available if v["color"] == args["color"].lower()]
if not any(v["stock"] > 0 for v in available):
continue
results.append(
{
"product_id": product["product_id"],
"name": product["name"],
"price": product["price"],
"description": product["description"],
"available_variants": [v for v in available if v["stock"] > 0],
}
)
return {"results": results[: args.get("limit", 3)], "count": len(results)}
if name == "get_product_details":
product = _find_product(args["product_id"])
return product or {"error": "not_found"}
if name == "add_to_cart":
product = _find_product(args["product_id"])
if not product:
return {"error": "product_not_found"}
variant = _find_variant(product, args["variant_id"])
if not variant:
return {"error": "variant_not_found"}
quantity = args.get("quantity", 1)
if variant["stock"] < quantity:
return {
"error": "insufficient_stock",
"available": variant["stock"],
"alternatives": [
v for v in product["variants"] if v["stock"] >= quantity
],
}
line_item = {
"line_item_id": f"LI-{len(CART) + 1}",
"product_id": product["product_id"],
"variant_id": variant["variant_id"],
"name": product["name"],
"size": variant["size"],
"color": variant["color"],
"quantity": quantity,
"unit_price": product["price"],
}
CART.append(line_item)
return {"added": line_item, "cart_size": len(CART)}
if name == "view_cart":
total = sum(item["unit_price"] * item["quantity"] for item in CART)
return {"items": CART, "total": round(total, 2)}
if name == "remove_from_cart":
before = len(CART)
CART[:] = [i for i in CART if i["line_item_id"] != args["line_item_id"]]
return {"removed": before != len(CART), "cart_size": len(CART)}
if name == "checkout":
if not CART:
return {"error": "empty_cart"}
phrase = args.get("confirmation_phrase", "").lower().strip(" .!")
if phrase not in ACCEPTED_CONFIRMATIONS:
return {
"error": "ambiguous_confirmation",
"heard": args.get("confirmation_phrase"),
"instruction": "Do not place the order. Ask the customer to confirm clearly.",
}
total = sum(item["unit_price"] * item["quantity"] for item in CART)
order_id = f"ORD-{9982 + len(ORDERS)}"
ORDERS[order_id] = {
"status": "confirmed",
"carrier": "UPS",
"tracking": None,
"eta": "Thursday",
"items": [f"{i['name']}, size {i['size']}, {i['color']}" for i in CART],
}
CART.clear()
return {"order_id": order_id, "total": round(total, 2), "status": "confirmed"}
if name == "track_order":
order = ORDERS.get(args["order_id"].upper())
return order or {"error": "order_not_found"}
return {"error": f"unknown_tool:{name}"}The interesting return value is ambiguous_confirmation. It doesn't just fail — it hands the agent an instruction. The model gets told, in the tool result, what to do next. That's a much more reliable steering mechanism than hoping a system prompt line survives twelve turns of conversation.
Step 3: Write a shopping-aware system prompt
SYSTEM_PROMPT = """
You are Riley, a voice shopping assistant for an outdoor gear retailer.
You are speaking out loud. Everything you say will be heard, not read.
SEARCH
- When a customer describes what they want, call search_products immediately.
Don't ask clarifying questions first — search, then refine.
- Read back the top 2-3 results conversationally. Name, one detail, price.
Never read a list of six things aloud.
- Say prices as words: "one hundred thirty-nine dollars", not "139.00".
VARIANTS
- Before calling add_to_cart, confirm size, color, and quantity. Never assume.
- If a variant is out of stock, say so and offer what is in stock.
CHECKOUT
- ONLY call checkout if the customer responds with a clear yes.
- If the response is ambiguous, ask again. Do not interpret "sure I think so"
or "yeah maybe" as confirmation.
- Before asking, state the total and the number of items.
ORDER NUMBERS AND TRACKING
- Read tracking numbers in pairs with pauses: "1Z, 99, 9A, A1..."
- Read order numbers letter by letter for the prefix, then the digits.
ALWAYS
- Keep replies short and conversational. Two sentences is usually enough.
- Never invent products, prices, stock, or delivery dates. If a tool didn't
return it, you don't know it.
- If a tool result contains an "instruction" field, follow it.
- Plain sentences only: no markdown, bullet points, or emoji. Your words go
straight to text-to-speech.
"""The "instruction" line matters. It's what makes the ambiguous_confirmation return value actually work. The last rule is there because every word the model writes gets spoken: a stray asterisk or bullet becomes noise in the customer's ear.
The price formatting rule matters more than it looks. Text-to-speech engines read "139.00" in ways that range from fine to "one three nine point zero zero," and the customer hears a number they don't trust.
Step 4: Put the tools in the LLM request
Tool calling lives in the LLM call. Each customer turn goes to LLM Gateway with the conversation so far and the TOOLS list. When the model wants data, it answers with tool calls instead of words; your code runs them with run_tool and appends the results, and the loop goes around until the model has something to say.
LLM_MODEL = "claude-haiku-4-5-20251001" # any tool-capable model on LLM Gateway
MAX_TOOL_ROUNDS = 5
def _arguments(tool_call) -> dict:
"""Tool arguments arrive as a JSON string; parse them defensively."""
args = tool_call.function.arguments
if isinstance(args, str):
args = json.loads(args or "{}")
return args or {}
async def respond(gateway, messages: list) -> str:
"""Run one assistant turn: call tools until the model answers in words.
`messages` is the running conversation. It's updated in place, so the
next turn sees every tool call and result from this one.
"""
for _ in range(MAX_TOOL_ROUNDS):
completion = await gateway.chat.completions.create(
model=LLM_MODEL,
messages=messages,
tools=TOOLS,
max_tokens=300,
)
# Collect text and tool calls across every choice: some providers
# return the text and the tool calls as separate choices.
text = " ".join(
c.message.content for c in completion.choices if c.message.content
).strip()
calls = [
tc for c in completion.choices for tc in (c.message.tool_calls or [])
]
if not calls:
messages.append({"role": "assistant", "content": text})
return text
messages.append(
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": tc.id,
"type": "function",
"function": {
"name": tc.function.name,
"arguments": json.dumps(_arguments(tc)),
},
}
for tc in calls
],
}
)
for tc in calls:
args = _arguments(tc)
print(f" -> {tc.function.name}({args})")
result = run_tool(tc.function.name, args)
messages.append(
{
"role": "tool",
"tool_call_id": tc.id,
"content": json.dumps(result),
}
)
return "Sorry, I lost track of that. Could you say it one more time?"Two details are worth a second look. Some providers return the model's text and its tool calls as separate entries in choices, so respond() gathers both across every choice instead of reading choices[0] alone. And messages is the single source of truth for the conversation: every tool call and tool result stays in it, which is how Riley still knows the cart total three turns after view_cart ran.
LLM Gateway uses the same API key as everything else here. Swap LLM_MODEL for any tool-capable model on the available models list. If you'd rather call OpenAI or another provider directly, any async client with an OpenAI-style chat.completions.create() drops into respond() unchanged.
Step 5: Wire up the audio
Two WebSockets and a speaker. Microphone audio streams to Universal-3.6 Pro Realtime; each finished turn goes to respond(); Riley's reply streams back from one text-to-speech session that stays open for the whole conversation.
import asyncio
import json
import os
import string
from urllib.parse import urlencode
import assemblyai as aai
import pyaudio
import websockets
from dotenv import load_dotenv
load_dotenv()
API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
MIC_RATE = 16000 # what we send to speech-to-text
SPEAKER_RATE = 24000 # text-to-speech's native output: 16-bit mono PCM
FRAMES_PER_BUFFER = 800 # 50 ms of microphone audio per message
STT_URL = "wss://streaming.assemblyai.com/v3/ws?" + urlencode(
{
"speech_model": "universal-3-6-pro",
"sample_rate": MIC_RATE,
"encoding": "pcm_s16le",
# Catalog names the recognizer should expect to hear.
"keyterms_prompt": json.dumps(["Ridgeline", "Trailhead Mid", "Vibram"]),
}
)
TTS_CONFIG = aai.TTSConfig(
voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", # Audrey
language="english",
)
GREETING = "Hey, I'm Riley. What are you shopping for today?"
# Short acknowledgements that shouldn't cut Riley off mid-sentence. "yes" and
# "no" are left out on purpose: at checkout they are real answers.
BACKCHANNELS = frozenset(
{"mhm", "mm", "uh", "um", "hmm", "ah", "oh", "okay", "ok", "right", "yeah", "yep"}
)
_STRIP = str.maketrans("", "", string.punctuation)
def is_interruption(text: str) -> bool:
words = text.lower().translate(_STRIP).split()
return len(words) >= 2 and not all(w in BACKCHANNELS for w in words)
class Riley:
def __init__(self, stt_ws, tts, gateway, speaker):
self.stt_ws = stt_ws
self.tts = tts
self.gateway = gateway
self.speaker = speaker
self.messages = [
{
"role": "system",
"content": f'{SYSTEM_PROMPT}\nYou opened the call by saying: "{GREETING}"',
}
]
self.reply_task = None
self.speaking = False
async def say(self, text: str):
print(f"Riley: {text}")
# Tell the speech model what Riley just asked, so a one-word answer
# like "ten" or "yes" is heard in context.
await self.stt_ws.send(
json.dumps({"type": "UpdateConfiguration", "agent_context": text[-1750:]})
)
self.speaking = True
try:
async for chunk in self.tts.speak(text):
await asyncio.to_thread(self.speaker.write, chunk)
finally:
self.speaking = False
async def handle_turn(self, text: str):
if self.messages[-1]["role"] == "user":
# The customer kept talking before Riley answered: one turn.
self.messages[-1]["content"] += " " + text
else:
self.messages.append({"role": "user", "content": text})
try:
reply = await respond(self.gateway, self.messages)
except Exception as err: # an LLM or network failure: recover out loud
print(f"LLM error: {err}")
reply = "Sorry, I hit a snag on my end. Could you say that again?"
if reply:
await self.say(reply)
async def barge_in(self):
if self.reply_task and not self.reply_task.done():
if self.speaking:
self.speaking = False
await self.tts.cancel() # the server drops the rest of the reply
self.reply_task.cancel()
def start_reply(self, coro):
self.reply_task = asyncio.create_task(coro)
async def on_turn(self, event: dict):
text = event.get("transcript", "").strip()
if not text:
return
if self.speaking and not is_interruption(text):
return # "mhm", "okay": the customer is listening, not interrupting
await self.barge_in()
if event.get("end_of_turn"):
print(f"Customer: {text}")
self.start_reply(self.handle_turn(text))
async def run_assistant():
audio = pyaudio.PyAudio()
mic = audio.open(
format=pyaudio.paInt16,
channels=1,
rate=MIC_RATE,
input=True,
frames_per_buffer=FRAMES_PER_BUFFER,
)
speaker = audio.open(
format=pyaudio.paInt16, channels=1, rate=SPEAKER_RATE, output=True
)
async with websockets.connect(
STT_URL, additional_headers={"Authorization": API_KEY}
) as stt_ws, aai.AsyncTTSSession(
TTS_CONFIG, api_key=API_KEY
) as tts, aai.AsyncLLMGateway(api_key=API_KEY) as gateway:
riley = Riley(stt_ws, tts, gateway, speaker)
print("Connected. Start talking (headphones keep Riley from hearing herself).")
riley.start_reply(riley.say(GREETING))
async def send_audio():
while True:
chunk = await asyncio.to_thread(
mic.read, FRAMES_PER_BUFFER, exception_on_overflow=False
)
await stt_ws.send(chunk)
async def read_transcripts():
async for raw in stt_ws:
event = json.loads(raw)
if event.get("type") == "Turn":
await riley.on_turn(event)
async def keep_tts_alive():
# A TTS session closes after two idle minutes.
while True:
await asyncio.sleep(60)
if not riley.speaking:
await tts.keep_alive()
try:
await asyncio.gather(send_audio(), read_transcripts(), keep_tts_alive())
finally:
await stt_ws.send(json.dumps({"type": "Terminate"}))
mic.close()
speaker.close()
audio.terminate()
if __name__ == "__main__":
try:
asyncio.run(run_assistant())
except KeyboardInterrupt:
print("\nSession ended.")That's the whole client. A few things are doing more work than their line count suggests.
Turn detection comes from the speech model. You don't run voice activity detection or decide when the customer has finished speaking. Universal-3.6 Pro Realtime judges from what's been said whether the turn is complete, then sends a Turn event with end_of_turn set, with the final transcript landing a median 307 ms after the speaker stops on Pipecat's open benchmark. The defaults are tuned for conversation, so leave the turn-silence settings alone unless testing shows a reason to change them. If you want the reasoning behind assembling the stack this way, the production ceiling post walks through what breaks and where.
Barge-in is three lines. When the customer starts talking over Riley, barge_in() calls cancel() on the text-to-speech session, which stops the reply in flight and tells the server to drop the rest, then cancels the reply task so nothing stale gets spoken. The is_interruption() check keeps a listening "mhm" or "okay" from cutting Riley off mid-sentence, following the backchannel guidance in the docs. "Yes" is deliberately missing from that list, because at checkout it's the most important word in the conversation.
One voice session per conversation. AsyncTTSSession opens once, and each reply is one speak() call on it, yielding raw 24 kHz 16-bit mono PCM as it's synthesized, so playback starts before the whole sentence is ready. A session closes after two idle minutes, which is what keep_tts_alive() is for. The voice comes from TTS_CONFIG; browse the voice library for another voice_id, or clone your brand's own voice from the dashboard.
Give the model the agent's own question
Here's a detail that's easy to skip and shouldn't be.
The hardest audio in this entire application is one word long. It's the "yes" after "Should I place the order?" Short utterances carry almost no acoustic context, and a bare "yeah" at the end of a noisy session is exactly where recognition falls over.
Universal-3.6 Pro Realtime takes agent_context — you pass in what your agent just asked, and the model uses it to resolve the reply. That's the UpdateConfiguration message at the top of say(): before every reply plays, Riley tells the speech model what she's about to ask, so a one-word "ten" or "yes" is heard as an answer to that question. It has the biggest effect on exactly the replies a checkout depends on: short confirmations, sizes, and spelled-out entities. The voice agent docs cover the field, including its 1,750-character limit.
Context Carryover — a short rolling memory of the conversation — is on by default, so the model already has some sense of what's been said. agent_context sharpens it to the specific question on the table.
Both are included in the price. There's no add-on for either.
Step 6: Test the four workflows
Run python shop_assistant.py, put on headphones so Riley doesn't hear herself through your speakers, and talk. Here's what each path should sound like.
Search:
You: "Show me waterproof hiking boots under one fifty." Riley: "I've got two. The Ridgeline Waterproof — full-grain leather, one hundred thirty-nine dollars. And the Trailhead Mid, a bit lighter, one hundred eighteen. Want details on either?"
Add to cart:
You: "The Ridgeline, size ten, in black." Riley: "Size ten, black — that's in stock. Just the one pair?" You: "Yep." Riley: "Added. Your cart's at one hundred thirty-nine dollars."
Note that Riley confirmed the variant before calling add_to_cart, and asked about quantity rather than assuming.
Checkout:
You: "Okay, let's do it." Riley: "One item, one hundred thirty-nine dollars. Should I place the order?" You: "Yeah, sure, I guess." Riley: "I want to make sure — should I place it? A clear yes or no." You: "Yes." Riley: "Done. Order number O-R-D nine nine eight two."
The agent called checkout with "yeah sure I guess", the function rejected it, and the tool result told the agent to re-ask. That whole exchange is three lines of Python, not a prompt engineering problem.
Order tracking:
You: "Where's order O-R-D nine nine eight one?" Riley: "In transit with UPS, arriving Thursday. Tracking is 1Z, 99, 9A, A1, 01, 23, 45, 67, 84."
Grouping the tracking number in pairs is the difference between a customer who can write it down and one who asks you to repeat it four times.
Test how accurately streaming transcription handles sizes, SKUs, order numbers, and spoken addresses — on your own audio, in the browser.
Where this gets harder in production
The demo works. Production is where the interesting problems live.
Personalization behind auth. Add a get_customer_profile tool, gate it on an authenticated session, and add what comes back to the system message in messages, so every later LLM request carries it without touching the audio connections. Order history, saved sizes, and default shipping address turn "what size?" into "your usual ten?" — which is a materially better conversation.
Refinement across turns. "Something for a wet trail" → "under one fifty" → "actually, in black" is one search, refined three times. Keep a plain dict of accumulated constraints in your application code, merge each new search_products call into it, and pass the merged filters to your catalog. Don't ask the model to remember state you can hold yourself.
Noise. In-store kiosks, drive-thrus, and phone calls are not desk microphones. Universal-3.6 Pro Realtime exposes voice_focus to isolate the primary speaker and suppress background speech — near-field for headsets and handsets, far-field for rooms and kiosks. And instead of tuning low-level thresholds, you pick a mode: min_latency, balanced (the default), or max_accuracy for the loud environments. A kiosk in a mall entrance is a far-field plus max_accuracy situation, and it's two more entries in the STT_URL parameters.
Multiple speakers. A couple shopping together is two voices on one microphone. Streaming diarization labels speakers live and then sends a single re-clustering correction within about half a second of the stream ending, up to 10 speakers.
Everything you can't test. Accents, cross-talk, a toddler in the background, someone reading a card number off a phone screen. This is why the entity numbers at the top of the post are on a voice-agent benchmark and not on read speech. How you evaluate a speech model determines whether the number you're quoting predicts anything about your traffic.
Where to take it from here
Change the microphone and this same code is a different product. Twilio Media Streams in front of it, with speech-to-text and text-to-speech both switched to 8 kHz pcm_mulaw, and it's a phone ordering line. A press-and-hold button in a React Native app, and it's in-app voice. A far-field mic array, and it's an in-store kiosk. A smart speaker skill, and it's ambient reordering.
But here's the thing worth noticing, and it isn't the audio plumbing.
The portable asset is the tool registry. search_products, add_to_cart, checkout, track_order — that's your commerce API expressed as verbs a conversation can reach. Once it exists, every new channel is a transport problem, not a product problem. The web team, the phone team, and the kiosk team stop shipping four subtly different definitions of "add to cart" and start calling the same seven functions.
Most teams building voice commerce start with the transcript and work outward. Start with the verbs instead. The transcript is a detail; the verbs are the product.
If you'd rather buy than build, Ringly offers an ecommerce AI receptionist that handles the phone side for online stores without you running the stack yourself.
Speech-to-text, LLM Gateway, and text-to-speech on one AssemblyAI API key. Get a key and have a working agent today.
Frequently asked questions
How do I build a voice-powered shopping assistant for e-commerce?
Define your core shopping actions — search, add to cart, checkout, track order — as OpenAI-style function tools, then connect three pieces: Universal-3.6 Pro Realtime streaming speech-to-text, which transcribes the customer and detects the end of each turn; an LLM through LLM Gateway, which receives the tools with every request and decides when to call your functions; and the AssemblyAI Text-to-Speech API, which speaks the answer back. You supply a system prompt, the tool registry, and the functions those tools map to. All three run on one AssemblyAI API key.
Can a voice shopping assistant handle product variants like size, color, and quantity?
Yes, and it comes down to two things: defining variant fields as explicit add_to_cart parameters, and instructing the agent in the system prompt to confirm them before calling the tool. On the recognition side, Universal-3.6 Pro Realtime posts a 14.4% entity error rate on AssemblyAI's English voice-agent benchmark, including 2.4% on phone numbers and 10.0% on codes and IDs — the alphanumeric strings that sizes, SKUs, and order codes most resemble. Passing agent_context with the agent's own question helps most on one-word answers like "ten" or "black."
How do I prevent accidental orders in a voice checkout flow?
Use two layers. First, the system prompt forbids calling checkout without an explicit yes. Second — and this is the one that actually holds — the checkout tool requires a confirmation_phrase parameter containing the customer's verbatim words, and your Python function validates it against an accepted list before doing anything. If the phrase is ambiguous, return an error with an instruction field telling the agent to re-ask. Validation in code survives long conversations in a way prompt instructions don't.
What channels can I deploy a voice shopping assistant on?
In-app voice, in-store kiosks, phone lines via Twilio Media Streams, and smart speakers all run on the same speech-to-text, LLM, and text-to-speech calls and the same tool registry. What changes between them is audio transport and environment: phone lines use 8 kHz pcm_mulaw in both directions, voice_focus takes near-field for handsets and headsets and far-field for kiosks and rooms, and the max_accuracy mode suits places where background noise is constant. Your tool definitions and backend functions don't change at all.
Should I build on AssemblyAI directly or use Vapi or Retell for e-commerce?
They solve different parts of the problem, and it's worth being precise about which one you need. Vapi and Retell are voice agent platforms — you configure an agent, and they handle orchestration, telephony, and a lot of operational surface for you, which is genuinely faster if your commerce logic is simple or your team is small. Building directly on AssemblyAI's speech-to-text, LLM Gateway, and text-to-speech APIs is infrastructure: your own code, with full control over conversation design and tool integrations. If you're wiring into a real catalog with inventory rules, pricing logic, and a checkout you're accountable for, that control tends to be the deciding factor. Worth noting that this isn't strictly either/or — Vapi has a native AssemblyAI integration, so you can run Universal-3.6 Pro Realtime as the transcriber inside a Vapi agent (here's the setup), and this comparison covers the orchestration tradeoffs in more depth.
How do I personalize a voice shopping assistant for authenticated customers?
Add a get_customer_profile tool that's gated on an authenticated session, then add what it returns to the system message in your conversation history. Every later LLM request carries it, so the agent can reference order history, saved sizes, and a default shipping address without reconnecting or restarting the conversation. You only need to pass the profile once.
How much does it cost to run a voice shopping assistant on AssemblyAI?
You pay for each piece separately, on one AssemblyAI account. Universal-3.6 Pro Realtime streaming speech-to-text is $0.45/hr, LLM Gateway bills by tokens at the rate of the model you choose, and text-to-speech is $15 per 1M characters, billed on the characters you send it. The pricing page has current rates for all three.
What speech model should I use for a voice shopping assistant?
Universal-3.6 Pro Realtime (universal-3-6-pro) is the streaming flagship and AssemblyAI's most accurate model for voice agents. It posts a 5.19% word error rate and 14.4% entity error rate on AssemblyAI's English voice-agent benchmark, covers 32 languages with mid-sentence code-switching, and includes keyterm prompting — which is worth using for brand names and product lines your catalog depends on. For recorded audio like post-purchase call reviews, Universal-3.5 Pro is the async equivalent at $0.21/hr.
