Building a voice-powered e-commerce shopping assistant

Voice shopping has crossed an inflection point. Search by typing is being replaced by search by saying, and the products that win are the ones that can hear "do you have those hiking boots in a ten, the waterproof ones, under one fifty" and come back with an answer instead of a search results page.

That sentence is also a decent stress test. It has a size, a price ceiling, a product attribute, and an implied reference to something said earlier. Get any one of them wrong and you've either shown the customer the wrong boot or, worse, put the wrong boot in their cart.

So that's what we're building. A Python voice agent that handles product search, cart management, order tracking, and checkout, over a single WebSocket, using the Voice Agent API. No orchestration framework. No SDK. About 200 lines.

Why voice e-commerce is different from voice support

Voice support agents have a narrow job. Look up an account, answer a question, escalate. The vocabulary is bounded and the failure mode is usually "the customer repeats themselves."

Commerce isn't like that.

Entity accuracy is the product. Sizes, SKUs, colors, prices, order numbers, ZIP codes, the last four of a card. These are the tokens a shopping agent gets wrong most often, and they're the tokens where being wrong actually costs money. A support agent that mishears a name asks again. A shopping agent that mishears "size ten" as "size nine" ships the wrong box.

This is the reason the model underneath matters more here than almost anywhere else. Universal-3.5 Pro Realtime posts a 15.31% entity error rate on Pipecat's open STT benchmark, which runs on real agent conversations rather than clean read speech. Here's the full picture:

Places at 6.28% is your shipping address. Phone numbers at 3.55% is your order number, your confirmation code, and the last four of the card. Those two rows are most of what a checkout flow actually has to hear correctly.

Shopping conversations are exploratory. Nobody opens with a fully-formed request. They start with "I need something for a wet trail" and narrow down over four turns. The agent has to hold the accumulated constraints without making the customer restate them.

The stakes shift mid-conversation. Browsing is low stakes. "Place the order" is not. The same conversation has to be loose at the start and rigorous at the end, and the transition is invisible to the model unless you build it in. We'll handle that with a confirmation gate in the tool schema, not a prompt suggestion.

Your customers don't all speak one language cleanly. They switch mid-sentence, especially around product names and numbers. Universal-3.5 Pro Realtime covers 18 languages with mid-sentence code-switching built in rather than bolted on — Hinglish included — so "mujhe woh size ten black wale chahiye" transcribes as what was actually said instead of collapsing into whichever language you configured. If you know the language up front, commit to it with language_codes and take the accuracy gain.

What you'll build

Four workflows, one agent:

1. Product search — natural language in, two or three spoken results out.
2. Cart management — add, view, remove, with variant confirmation.
3. Order tracking — look up a status and read a tracking number aloud without mangling it.
4. Checkout assistance — a confirmation gate that won't fire on an ambiguous "sure, I guess."

The stack:

• Voice Agent API — one WebSocket that bundles speech-to-text, the LLM, text-to-speech, turn detection, and tool calling. Flat $4.50/hr, roughly 1 second end-to-end.
• Universal-3.5 Pro Realtime (universal-3-5-pro) — the streaming model underneath it. On its own it's $0.45/hr base; through the Voice Agent API it's already in the flat rate.
• Python 3.9+, websockets, pyaudio.
• A mocked catalog and order database — swap in Shopify, Commerce Cloud, or whatever you actually run.

If you've never built on the Voice Agent API before, the getting-started guide covers the connection basics that this post moves quickly through.

Setup

pip install websockets pyaudio python-dotenv

Create a .env file:

ASSEMBLYAI_API_KEY=your_api_key_here

Grab a key from the dashboard. One endpoint for everything:

wss://agents.assemblyai.com/v1/ws

That's the whole integration surface. No separate speech-to-text connection, no LLM client, no TTS provider.

Step 1: Define the shopping tools

The agent's capabilities are just JSON schemas. Seven of them.

import json

TOOLS = [
    {
        "type": "function",
        "name": "search_products",
        "description": (
            "Search the product catalog. Use whenever the customer describes "
            "what they're looking for, even vaguely."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": "What the customer is looking for, in their own words",
                },
                "max_price": {
                    "type": "number",
                    "description": "Price ceiling in dollars, if the customer gave one",
                },
                "size": {"type": "string", "description": "Requested size, if mentioned"},
                "color": {"type": "string", "description": "Requested color, if mentioned"},
                "limit": {"type": "integer", "description": "Max results to return. Default 3."},
            },
            "required": ["query"],
        },
    },
    {
        "type": "function",
        "name": "get_product_details",
        "description": "Full details for one product: variants, stock, return policy.",
        "parameters": {
            "type": "object",
            "properties": {"product_id": {"type": "string"}},
            "required": ["product_id"],
        },
    },
    {
        "type": "function",
        "name": "add_to_cart",
        "description": (
            "Add a specific variant to the cart. Only call this after the customer "
            "has confirmed size, color, and quantity."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "product_id": {"type": "string"},
                "variant_id": {"type": "string", "description": "Specific size/color variant"},
                "quantity": {"type": "integer", "description": "Default 1"},
            },
            "required": ["product_id", "variant_id"],
        },
    },
    {
        "type": "function",
        "name": "view_cart",
        "description": "Read back the current cart contents and total.",
        "parameters": {"type": "object", "properties": {}},
    },
    {
        "type": "function",
        "name": "remove_from_cart",
        "description": "Remove one line item from the cart.",
        "parameters": {
            "type": "object",
            "properties": {"line_item_id": {"type": "string"}},
            "required": ["line_item_id"],
        },
    },
    {
        "type": "function",
        "name": "checkout",
        "description": (
            "Place the order. ONLY call this when the customer has given an explicit, "
            "unambiguous verbal confirmation. Pass their exact words."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "confirmation_phrase": {
                    "type": "string",
                    "description": "The customer's exact confirming words, verbatim",
                }
            },
            "required": ["confirmation_phrase"],
        },
    },
    {
        "type": "function",
        "name": "track_order",
        "description": "Look up status and tracking for an existing order.",
        "parameters": {
            "type": "object",
            "properties": {"order_id": {"type": "string"}},
            "required": ["order_id"],
        },
    },
]

Look at checkout for a second. confirmation_phrase is required, and its description says "verbatim." That's deliberate.

Most voice checkout bugs come from the model deciding that "yeah okay sure whatever" means yes. By forcing the agent to hand you the actual words it heard, you get to make that decision in Python, where you can reason about it, log it, and change it without touching a prompt. A prompt instruction is a request. A required parameter your function validates is a gate.

Step 2: Implement the backend (mocked)

Stub catalog, stub cart, stub orders. Replace all three with real calls when you wire this to a live store.

CATALOG = [
    {
        "product_id": "P-1042",
        "name": "Ridgeline Waterproof Hiking Boot",
        "price": 139.00,
        "description": "Full-grain leather, seam-sealed, Vibram outsole.",
        "return_policy": "60 days, unworn",
        "variants": [
            {"variant_id": "P-1042-10-BLK", "size": "10", "color": "black", "stock": 8},
            {"variant_id": "P-1042-10-BRN", "size": "10", "color": "brown", "stock": 0},
            {"variant_id": "P-1042-11-BLK", "size": "11", "color": "black", "stock": 3},
        ],
    },
    {
        "product_id": "P-2213",
        "name": "Trailhead Mid Waterproof",
        "price": 118.00,
        "description": "Lightweight synthetic upper, gusseted tongue.",
        "return_policy": "60 days, unworn",
        "variants": [
            {"variant_id": "P-2213-10-GRY", "size": "10", "color": "grey", "stock": 14},
            {"variant_id": "P-2213-11-GRY", "size": "11", "color": "grey", "stock": 6},
        ],
    },
]

CART = []

ORDERS = {
    "ORD-9981": {
        "status": "in transit",
        "carrier": "UPS",
        "tracking": "1Z999AA10123456784",
        "eta": "Thursday",
        "items": ["Ridgeline Waterproof Hiking Boot, size 10, black"],
    }
}

ACCEPTED_CONFIRMATIONS = [
    "yes",
    "yes please",
    "place the order",
    "go ahead",
    "confirm",
    "buy it",
]


def _find_product(product_id):
    return next((p for p in CATALOG if p["product_id"] == product_id), None)


def _find_variant(product, variant_id):
    return next((v for v in product["variants"] if v["variant_id"] == variant_id), None)


def run_tool(name: str, args: dict) -> dict:
    """Dispatch a tool call from the agent and return a JSON-serializable result."""

    if name == "search_products":
        query = args.get("query", "").lower()
        results = []
        for product in CATALOG:
            haystack = f"{product['name']} {product['description']}".lower()
            if not any(word in haystack for word in query.split() if len(word) > 3):
                continue
            if args.get("max_price") and product["price"] > args["max_price"]:
                continue
            available = product["variants"]
            if args.get("size"):
                available = [v for v in available if v["size"] == str(args["size"])]
            if args.get("color"):
                available = [v for v in available if v["color"] == args["color"].lower()]
            if not any(v["stock"] > 0 for v in available):
                continue
            results.append(
                {
                    "product_id": product["product_id"],
                    "name": product["name"],
                    "price": product["price"],
                    "description": product["description"],
                    "available_variants": [v for v in available if v["stock"] > 0],
                }
            )
        return {"results": results[: args.get("limit", 3)], "count": len(results)}

    if name == "get_product_details":
        product = _find_product(args["product_id"])
        return product or {"error": "not_found"}

    if name == "add_to_cart":
        product = _find_product(args["product_id"])
        if not product:
            return {"error": "product_not_found"}
        variant = _find_variant(product, args["variant_id"])
        if not variant:
            return {"error": "variant_not_found"}
        quantity = args.get("quantity", 1)
        if variant["stock"] < quantity:
            return {
                "error": "insufficient_stock",
                "available": variant["stock"],
                "alternatives": [
                    v for v in product["variants"] if v["stock"] >= quantity
                ],
            }
        line_item = {
            "line_item_id": f"LI-{len(CART) + 1}",
            "product_id": product["product_id"],
            "variant_id": variant["variant_id"],
            "name": product["name"],
            "size": variant["size"],
            "color": variant["color"],
            "quantity": quantity,
            "unit_price": product["price"],
        }
        CART.append(line_item)
        return {"added": line_item, "cart_size": len(CART)}

    if name == "view_cart":
        total = sum(item["unit_price"] * item["quantity"] for item in CART)
        return {"items": CART, "total": round(total, 2)}

    if name == "remove_from_cart":
        before = len(CART)
        CART[:] = [i for i in CART if i["line_item_id"] != args["line_item_id"]]
        return {"removed": before != len(CART), "cart_size": len(CART)}

    if name == "checkout":
        if not CART:
            return {"error": "empty_cart"}
        phrase = args.get("confirmation_phrase", "").lower().strip(" .!")
        if phrase not in ACCEPTED_CONFIRMATIONS:
            return {
                "error": "ambiguous_confirmation",
                "heard": args.get("confirmation_phrase"),
                "instruction": "Do not place the order. Ask the customer to confirm clearly.",
            }
        total = sum(item["unit_price"] * item["quantity"] for item in CART)
        order_id = f"ORD-{9982 + len(ORDERS)}"
        ORDERS[order_id] = {
            "status": "confirmed",
            "carrier": "UPS",
            "tracking": None,
            "eta": "Thursday",
            "items": [f"{i['name']}, size {i['size']}, {i['color']}" for i in CART],
        }
        CART.clear()
        return {"order_id": order_id, "total": round(total, 2), "status": "confirmed"}

    if name == "track_order":
        order = ORDERS.get(args["order_id"].upper())
        return order or {"error": "order_not_found"}

    return {"error": f"unknown_tool:{name}"}

The interesting return value is ambiguous_confirmation. It doesn't just fail — it hands the agent an instruction. The model gets told, in the tool result, what to do next. That's a much more reliable steering mechanism than hoping a system prompt line survives twelve turns of conversation.

Step 3: Write a shopping-aware system prompt

SYSTEM_PROMPT = """
You are Riley, a voice shopping assistant for an outdoor gear retailer.
You are speaking out loud. Everything you say will be heard, not read.

SEARCH
- When a customer describes what they want, call search_products immediately.
  Don't ask clarifying questions first — search, then refine.
- Read back the top 2-3 results conversationally. Name, one detail, price.
  Never read a list of six things aloud.
- Say prices as words: "one hundred thirty-nine dollars", not "139.00".

VARIANTS
- Before calling add_to_cart, confirm size, color, and quantity. Never assume.
- If a variant is out of stock, say so and offer what is in stock.

CHECKOUT
- ONLY call checkout if the customer responds with a clear yes.
- If the response is ambiguous, ask again. Do not interpret "sure I think so"
  or "yeah maybe" as confirmation.
- Before asking, state the total and the number of items.

ORDER NUMBERS AND TRACKING
- Read tracking numbers in pairs with pauses: "1Z, 99, 9A, A1..."
- Read order numbers letter by letter for the prefix, then the digits.

ALWAYS
- Keep replies short and conversational. Two sentences is usually enough.
- Never invent products, prices, stock, or delivery dates. If a tool didn't
  return it, you don't know it.
- If a tool result contains an "instruction" field, follow it.
"""

That last line matters. It's what makes the ambiguous_confirmation return value actually work.

The price formatting rule matters more than it looks. Text-to-speech engines read "139.00" in ways that range from fine to "one three nine point zero zero," and the customer hears a number they don't trust.

Step 4: Wire the WebSocket

One connection. Audio out, events in.

import asyncio
import base64
import json
import os

import pyaudio
import websockets
from dotenv import load_dotenv

load_dotenv()

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
WS_URL = "wss://agents.assemblyai.com/v1/ws"

SAMPLE_RATE = 24000
CHANNELS = 1
FORMAT = pyaudio.paInt16
FRAMES_PER_BUFFER = 1024


async def run_assistant():
    audio = pyaudio.PyAudio()
    mic = audio.open(
        format=FORMAT,
        channels=CHANNELS,
        rate=SAMPLE_RATE,
        input=True,
        frames_per_buffer=FRAMES_PER_BUFFER,
    )
    speaker = audio.open(
        format=FORMAT,
        channels=CHANNELS,
        rate=SAMPLE_RATE,
        output=True,
    )

    headers = {"Authorization": f"Bearer {API_KEY}"}

    # Tool results are queued here and sent once reply.done arrives.
    pending_results = []

    async with websockets.connect(WS_URL, additional_headers=headers) as ws:
        await ws.send(
            json.dumps(
                {
                    "type": "session.update",
                    "session": {
                        "system_prompt": SYSTEM_PROMPT,
                        "greeting": "Hey, I'm Riley. What are you shopping for today?",
                        "output": {"voice": "jane"},
                        "tools": TOOLS,
                    },
                }
            )
        )

        async def send_audio():
            while True:
                chunk = await asyncio.to_thread(
                    mic.read, FRAMES_PER_BUFFER, exception_on_overflow=False
                )
                await ws.send(
                    json.dumps(
                        {
                            "type": "input.audio",
                            "audio": base64.b64encode(chunk).decode("utf-8"),
                        }
                    )
                )

        async def handle_messages():
            async for raw in ws:
                event = json.loads(raw)
                event_type = event.get("type")

                if event_type == "session.ready":
                    print("Session ready. Start talking.")

                elif event_type == "transcript.user":
                    print(f"Customer: {event['text']}")

                elif event_type == "transcript.agent":
                    print(f"Riley: {event['text']}")

                elif event_type == "reply.audio":
                    speaker.write(base64.b64decode(event["data"]))

                elif event_type == "tool.call":
                    name = event["name"]
                    args = event["arguments"]
                    print(f"  -> {name}({args})")
                    # Run the tool now, but queue the result — tool results
                    # are only accepted once reply.done has arrived.
                    pending_results.append(
                        {
                            "type": "tool.result",
                            "call_id": event["call_id"],
                            "result": json.dumps(run_tool(name, args)),
                        }
                    )

                elif event_type == "reply.done":
                    if event.get("status") == "interrupted":
                        # Caller barged in: drop any audio still queued.
                        speaker.stop_stream()
                        speaker.start_stream()
                    while pending_results:
                        await ws.send(json.dumps(pending_results.pop(0)))
                    print("---")

        await asyncio.gather(send_audio(), handle_messages())


if __name__ == "__main__":
    try:
        asyncio.run(run_assistant())
    except KeyboardInterrupt:
        print("\nSession ended.")

That's the whole client. Six event types to handle, and the last of them carries the bookkeeping — draining queued tool results and dropping stale audio when the caller interrupts.

Notice what isn't in there. No voice activity detection. No barge-in detection — the server tells you it happened; all you do is drop the audio you had queued. No LLM client, no streaming token buffer, no text-to-speech queue, no code deciding whether the customer has finished speaking. Turn detection happens server-side, reading the punctuation it predicts rather than counting silence, at roughly 300ms. If you want the reasoning behind that design, the production ceiling post walks through what breaks when you assemble those pieces yourself.

Give the model the agent's own question

Here's a detail that's easy to skip and shouldn't be.

The hardest audio in this entire application is one word long. It's the "yes" after "Should I place the order?" Short utterances carry almost no acoustic context, and a bare "yeah" at the end of a noisy session is exactly where recognition falls over.

Universal-3.5 Pro Realtime takes agent_context — you pass in what your agent just asked, and the model uses it to resolve the reply. Across a benchmark of 20,000 voice agent audio files, passing agent context cut word error rate by 10.2%, with short-utterance errors down 13.7% and place-name entities down 15.5%. For a checkout confirmation and a spoken shipping address, those are the two numbers that matter.

Context Carryover — a short rolling memory of the conversation — is on by default, so the model already has some sense of what's been said. agent_context sharpens it to the specific question on the table.

Both are included in the price. There's no add-on for either.

Step 5: Test the four workflows

Run it and talk. Here's what each path should sound like.

Search:

You: "Show me waterproof hiking boots under one fifty." Riley: "I've got two. The Ridgeline Waterproof — full-grain leather, one hundred thirty-nine dollars. And the Trailhead Mid, a bit lighter, one hundred eighteen. Want details on either?"

Add to cart:

You: "The Ridgeline, size ten, in black." Riley: "Size ten, black — that's in stock. Just the one pair?" You: "Yep." Riley: "Added. Your cart's at one hundred thirty-nine dollars."

Note that Riley confirmed the variant before calling add_to_cart, and asked about quantity rather than assuming.

Checkout:

You: "Okay, let's do it." Riley: "One item, one hundred thirty-nine dollars. Should I place the order?" You: "Yeah, sure, I guess." Riley: "I want to make sure — should I place it? A clear yes or no." You: "Yes." Riley: "Done. Order number O-R-D nine nine eight two."

The agent called checkout with "yeah sure I guess", the function rejected it, and the tool result told the agent to re-ask. That whole exchange is three lines of Python, not a prompt engineering problem.

Order tracking:

You: "Where's order O-R-D nine nine eight one?" Riley: "In transit with UPS, arriving Thursday. Tracking is 1Z, 99, 9A, A1, 01, 23, 45, 67, 84."

Grouping the tracking number in pairs is the difference between a customer who can write it down and one who asks you to repeat it four times.

Where this gets harder in production

The demo works. Production is where the interesting problems live.

Personalization behind auth. Add a get_customer_profile tool, gate it on an authenticated session, and pass what comes back into the conversation with a mid-conversation session.update rather than opening a new connection. Order history, saved sizes, and default shipping address turn "what size?" into "your usual ten?" — which is a materially better conversation.

Refinement across turns. "Something for a wet trail" → "under one fifty" → "actually, in black" is one search, refined three times. Keep a plain dict of accumulated constraints in your application code, merge each new search_products call into it, and pass the merged filters to your catalog. Don't ask the model to remember state you can hold yourself.

Noise. In-store kiosks, drive-thrus, and phone calls are not desk microphones. Universal-3.5 Pro Realtime exposes voice_focus to isolate the primary speaker and suppress background speech — near-field for headsets and handsets, far-field for rooms and kiosks. And instead of tuning low-level thresholds, you pick a mode: min_latency, balanced (the default), or max_accuracy for the loud environments. A kiosk in a mall entrance is a far-field plus max_accuracy situation and it's two config values, not a tuning project.

Multiple speakers. A couple shopping together is two voices on one microphone. Streaming diarization labels speakers live and then sends a single re-clustering correction within about half a second of the stream ending, up to 10 speakers.

Everything you can't test. Accents, cross-talk, a toddler in the background, someone reading a card number off a phone screen. This is why the entity numbers at the top of the post are on a benchmark of real agent conversations and not on read speech. How you evaluate a speech model determines whether the number you're quoting predicts anything about your traffic.

Where to take it from here

Change the microphone and this same code is a different product. Twilio Media Streams in front of it, and it's a phone ordering line. A press-and-hold button in a React Native app, and it's in-app voice. A far-field mic array, and it's an in-store kiosk. A smart speaker skill, and it's ambient reordering.

But here's the thing worth noticing, and it isn't the WebSocket.

The portable asset is the tool registry. search_products, add_to_cart, checkout, track_order — that's your commerce API expressed as verbs a conversation can reach. Once it exists, every new channel is a transport problem, not a product problem. The web team, the phone team, and the kiosk team stop shipping four subtly different definitions of "add to cart" and start calling the same seven functions.

Most teams building voice commerce start with the transcript and work outward. Start with the verbs instead. The transcript is a detail; the verbs are the product.

Frequently asked questions

How do I build a voice-powered shopping assistant for e-commerce?

Register your core shopping actions — search, add to cart, checkout, track order — as tools on the Voice Agent API, then open a single WebSocket to wss://agents.assemblyai.com/v1/ws. The API transcribes the customer, decides when to call your functions, and speaks the results back over the same connection. You supply a system prompt, a tool registry, and the functions those tools map to; you don't build a speech-to-text pipeline, turn detection, or a text-to-speech queue.

Can a voice shopping assistant handle product variants like size, color, and quantity?

Yes, and it comes down to two things: defining variant fields as explicit add_to_cart parameters, and instructing the agent in the system prompt to confirm them before calling the tool. On the recognition side, Universal-3.5 Pro Realtime posts a 15.31% entity error rate on Pipecat's open STT benchmark, including 3.55% on phone numbers — the alphanumeric strings that sizes, SKUs, and order codes most resemble. Passing agent_context with the agent's own question cuts word error rate a further 10.2%, which matters most on one-word answers like "ten" or "black."

How do I prevent accidental orders in a voice checkout flow?

Use two layers. First, the system prompt forbids calling checkout without an explicit yes. Second — and this is the one that actually holds — the checkout tool requires a confirmation_phrase parameter containing the customer's verbatim words, and your Python function validates it against an accepted list before doing anything. If the phrase is ambiguous, return an error with an instruction field telling the agent to re-ask. Validation in code survives long conversations in a way prompt instructions don't.

What channels can I deploy a voice shopping assistant on?

In-app voice, in-store kiosks, phone lines via Twilio Media Streams, and smart speakers all run on the same API connection and the same tool registry. What changes between them is audio transport and environment: use voice_focus: near-field for handsets and headsets, far-field for kiosks and rooms, and the max_accuracy mode where background noise is constant. Your tool definitions and backend functions don't change at all.

How does the AssemblyAI Voice Agent API compare to Vapi or Retell for e-commerce?

They solve different parts of the problem, and it's worth being precise about which one you need. Vapi and Retell are voice agent platforms — you configure an agent, and they handle orchestration, telephony, and a lot of operational surface for you, which is genuinely faster if your commerce logic is simple or your team is small. The Voice Agent API is infrastructure: one WebSocket, your own code, full control over conversation design and tool integrations, at a flat $4.50/hr. If you're wiring into a real catalog with inventory rules, pricing logic, and a checkout you're accountable for, that control tends to be the deciding factor. Worth noting that this isn't strictly either/or — Vapi has a native AssemblyAI integration, so you can run Universal-3.5 Pro Realtime as the transcriber inside a Vapi agent (here's the setup), and this comparison covers the orchestration tradeoffs in more depth.

How do I personalize a voice shopping assistant for authenticated customers?

Add a get_customer_profile tool that's gated on an authenticated session, then inject what it returns using a mid-conversation session.update rather than reconnecting. That lets the agent reference order history, saved sizes, and a default shipping address without dropping the connection or restarting the conversation. Context Carryover keeps a rolling memory of the conversation itself, so you only need to pass the profile once.

How much does it cost to run a voice shopping assistant on AssemblyAI?

The Voice Agent API is a flat $4.50/hr that covers speech-to-text, the LLM, voice generation, turn detection, and tool calling in one line item — no per-token surcharges and no separate vendor bills. New accounts start with free credits, and billing is per second with no minimums. If you'd rather run just the streaming model and bring your own LLM and text-to-speech, Universal-3.5 Pro Realtime is $0.45/hr base; full pricing is here.

What speech model should I use for a voice shopping assistant?

Universal-3.5 Pro Realtime (universal-3-5-pro) is the streaming flagship and the speech foundation under the Voice Agent API. It posts 6.99% pooled word error rate and 15.31% entity error rate on Pipecat's open STT benchmark, covers 18 languages with mid-sentence code-switching, and includes keyterm prompting — which is worth using for brand names and product lines your catalog depends on. For recorded audio like post-purchase call reviews, Universal-3.5 Pro is the async equivalent at $0.21/hr.