Voice AI for call tracking platforms: What to build and what to buy

If you build call tracking software, you already ship transcription. Every platform in the category does. That stopped being a differentiator somewhere around 2023, right about when it stopped being hard.

What's still hard is everything downstream of the transcript — and the decisions your team makes about the speech layer determine how much of it you get to build.

This post is written for the product manager or engineer inside a call tracking company, not for the marketer shopping for one. The question isn't whether to add AI. It's which parts of the voice stack are your product and which parts are infrastructure you should be renting.

"The capabilities AssemblyAI enables us to build help businesses market confidently while saving time and money. It's powerful, almost magical to see it work."

— Ryan Johnson, Chief Product Officer, CallRail

CallRail is a useful reference point precisely because they made this decision early. They bought the speech layer and spent their engineering on conversation intelligence, and they went on to double their conversation intelligence customer base. The interesting part isn't that they added transcription. It's what they didn't build.

What a call tracking platform actually has to build

Strip a call tracking product down and there are four layers.

The telephony layer. Number pools, dynamic number insertion, call routing, recording. This is your product. It's where attribution actually happens — a lead calls the number that was rendered on the PPC landing page, and that number is why you can tell the agency which campaign produced the revenue. Nobody should be renting this.

The attribution and reporting layer. Tying calls back to sessions, campaigns, keywords and forms; scoring lead quality; feeding the CRM. Also your product, also where your moat is.

The intelligence layer. Summaries, sentiment, keyword spotting, lead qualification, coaching signals. This is contested territory. The models are commodity; the schema and the workflow are not. Build the schema, rent the models.

The speech layer. Turning audio into accurate, structured, redactable text. This is infrastructure. It's the layer where the gap between "works in a demo" and "works on a 4-minute call from a car on a 2010-era mobile network" is enormous, and where staying current means retraining continuously.

Almost every call tracking team that gets into trouble does so by misclassifying layer four as layer three.

The accuracy that attribution actually depends on

Here's the failure mode nobody catches in evaluation.

You benchmark two speech APIs on a sample of recorded calls, look at word error rate, and pick the lower number. Six months later your customers are complaining that the intelligence features are unreliable, and when you dig in, the transcripts are fine — except the business names are wrong, the phone numbers have transposed digits, and the product SKUs the caller read out are garbage.

That's not a WER problem. Those words are a rounding error in WER. It's an entity accuracy problem, and for a call tracking platform entities are the product. A lead-qualification model that can't reliably read back a callback number is worse than no lead-qualification model, because it produces confident wrong answers. We've made the broader version of this argument in why word error rate is broken.

The numbers worth asking a vendor for are entity-level. On our real-conversation streaming benchmark, Universal-3.5 Pro Realtime posts a 15.31% entity error rate overall, 16.92% on names, 6.28% on places and 3.55% on phone numbers — that last one being the number a call tracking product lives or dies on. Pooled WER on the same benchmark is 6.99%, against 15.58% for Deepgram Flux, 9.76% for ElevenLabs Scribe v2 and 9.04% for Google Chirp3. On pre-recorded audio, English mean WER across 26 datasets and 80,000+ files is 5.6%, median 4.9%. Everything is published at assemblyai.com/benchmarks with the methodology attached.

You can also move entity accuracy yourself, which is the part most teams miss. A call tracking platform knows the account's business name, its product vocabulary, its competitor names and its service area before the call is transcribed. Feed those in as keyterms — up to 1,000 terms of six words or fewer on pre-recorded audio, for $0.05/hr — and the model stops guessing at the words that matter most to that specific customer. Detail in the docs on prompting and keyterms.

What you build on top of the transcript

Once the transcript is trustworthy, the intelligence features are mostly a question of which pieces you buy à la carte and which you write yourself.

Speech Understanding covers the deterministic analysis, priced per hour and billed only when you turn it on:

Summaries, lead scoring and call classification go somewhere different. Those are prompt problems, and they belong in LLM Gateway — one API across 33 models from Anthropic, OpenAI, Google and Qwen, with cross-provider fallback, structured JSON output and prompt caching. This is the piece that changed most since the last version of this post. You no longer pick a summary shape from a fixed menu. You define the JSON schema your product needs — lead_quality, service_requested, callback_committed, objection — and you get it back filled in, and you can change models without rewriting the integration.

For a call tracking platform that's a meaningful unlock, because lead qualification criteria differ per vertical. A dentist's definition of a qualified lead is not a personal injury firm's. A fixed summarization endpoint can't express that. A prompt template per vertical can.

Calabrio built on the same foundation for the enterprise contact center side of this market:

"The transcription accuracy, reliability, and speed of AssemblyAI's API have greatly enhanced our operations, reinforcing our trust in their technology and solidifying our partnership."

— Raj Shankar, SVP Product, Calabrio

Siro, on the field sales side, reports a 36% improvement in close rate and a 90% reduction in customer complaints and support tickets from the same class of analysis. WhatConverts, which does lead tracking for marketing agencies, runs transcription plus PII redaction so agency staff can review and value leads without handling raw sensitive data. Nextiva and Gorgias sit on adjacent parts of the same problem.

Recording calls responsibly: consent, redaction, and the compliance stack

This is the section the previous version of this post skipped, and it shouldn't have. If your platform records phone calls, compliance isn't a feature — it's a precondition, and your customers will ask you about it in the sales cycle.

To be clear about what follows: this is a description of the landscape, not legal advice. Your counsel owns your policy and your customers' counsel owns theirs. What you own is the technical surface that makes their policy implementable.

Consent to record. In the US, call recording is governed by a patchwork. Federal law and most states permit recording with one party's consent. Roughly a dozen states require all parties to consent — California, Florida, Illinois, Maryland, Massachusetts, Pennsylvania and Washington among them — and the details vary by state and by whether the call crosses state lines. Practically, this means a call tracking platform needs per-account and often per-number recording controls, a configurable disclosure at call start, and an audit trail showing which policy applied to which call. That's product work, not vendor work.

TCPA. The Telephone Consumer Protection Act governs outbound calling and texting — consent for automated dialing, do-not-call obligations, and disclosure requirements. It matters here because call tracking platforms increasingly ship outbound and callback features, and because AI-generated or prerecorded voice on an outbound call sits squarely inside its scope. If you're adding outbound automation, this is the regime to build against.

Payment data. If a caller reads a card number aloud — and on service and booking calls, they will — that recording is now in scope for PCI DSS. The mitigation is redaction at both layers. PII Redaction on the audio ($0.05/hr) strips the values from the waveform so the stored recording never contains them; PII Redaction on the text ($0.08/hr) strips them from the transcript and everything downstream — your search index, your CRM sync, your LLM prompts. Doing only the text layer is a common and expensive mistake, because the recording is usually the thing that gets subpoenaed. AssemblyAI is certified to PCI DSS v4.0, and the Voice Agent API is PCI-certified end to end.

GDPR and CCPA. A call recording containing a person's voice, name and phone number is personal data under both. The obligations that reach your architecture are data residency, deletion on request, and processor agreements. EU data residency is available at api.eu.assemblyai.com and streaming.eu.assemblyai.com at the same price as the US endpoints, so serving EU accounts doesn't require a separate commercial negotiation.

Health data. Medical and dental practices are a large share of the call tracking market, and their inbound calls routinely contain protected health information. If your customers process PHI:

AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI.

The BAA can be signed in minutes without a sales call. The broader stack is SOC 2 Type 2, ISO 27001:2022 and PCI DSS v4.0.

The part nobody had to think about in 2024: voice agents

Here's the change that should be reshaping call tracking roadmaps right now.

For twenty years the value proposition of call tracking was that a human answered the phone and you needed to know which campaign sent them. That premise is being renegotiated. A growing share of inbound calls are now answered by a voice agent that qualifies the lead, books the appointment and hands off only the calls that need a person. Home services, dental, legal intake, property management — the exact verticals that buy call tracking — are where this is moving fastest.

That's either an existential problem for your category or the biggest expansion opportunity in it, depending on whether you ship the agent.

The strategic case for shipping it is strong. You already own the number. You already own the routing. You already own the attribution data that tells the agent which campaign this caller came from, which is context no standalone voice agent vendor has. An agent that opens with the right greeting for the right campaign is a better agent, and only you can build it.

The Voice Agent API is a flat $4.50/hr, billed per second of connected conversation time. One WebSocket bundles Universal-3.5 Pro Realtime for speech recognition, a Voice Agent LLM tuned for spoken conversation rather than text chat, and Voice Agent TTS, at roughly one second end to end. Every feature is included in the $4.50/hr rate. There are no per-layer add-ons, concurrency fees, or per-agent subscriptions. Prompting, Voice Focus, advanced turn detection, interruption detection that ignores backchannels like "mhm" and "right," recordings and transcripts, infrastructure hosting, and bring-your-own-Twilio SIP trunking with no per-minute markup all sit inside that number.

That last item matters more than it sounds for a call tracking company. You already have carrier relationships and number inventory. Being able to point your existing SIP trunk at an agent, without a markup on minutes you're already paying for, is the difference between a viable product line and a margin problem. Worth reading alongside the hidden costs in a voice agent stack and how turn detection works, because turn detection is what separates an agent people tolerate from one they hang up on.

If you'd rather assemble the pieces yourself, the streaming model is available on its own at $0.45/hr with agent_context, which passes the agent's own question to the recognizer so it hears the caller's reply through that lens — worth 10.2% off WER across a 20,000-file voice agent benchmark. Time to final transcript is a 285 ms median, 374 ms at p95. Choosing a speech-to-text API for voice agents covers the tradeoffs.

Where the line actually falls

Every call tracking team eventually asks whether to bring transcription in-house. At high call volume the per-hour pricing starts to look like a tax, and someone runs the numbers on self-hosting an open model.

Run those numbers properly and they usually settle the question in the other direction, but not for the reason people expect. The cost of a GPU fleet is knowable. What isn't knowable is the cost of falling behind: multilingual coverage is a per-language investment, diarization is a separate research problem from transcription, redaction accuracy has legal consequences when it's wrong, and the model that's competitive the day you deploy it is a generation behind within two quarters unless someone owns retraining full time. Meanwhile transcription with diarization is $0.23/hr, billed per second, with unlimited concurrency and no minimums — a number that doesn't change when Monday morning call volume triples.

The better framing isn't build versus buy. It's this: what would you build with the six engineers you'd otherwise assign to speech? For a call tracking platform, the answer is almost always attribution, verticalized lead scoring, or the voice agent — all of which are worth more to your customers than a marginally cheaper transcript.

CallRail figured that out early, which is why they're a useful example and not just a logo. They didn't win by having better speech recognition than their competitors. They won by treating speech recognition as settled and spending the next three years on what sits above it. Call center analytics and conversation intelligence go deeper on what "above it" looks like.

Frequently asked questions

How does AssemblyAI compare to AWS Transcribe for contact center analytics?

The difference that matters for contact center work is entity accuracy and the surface built around the transcript, not headline word error rate. On our real-conversation streaming benchmark, Universal-3.5 Pro Realtime posts a 3.55% error rate on phone numbers and 15.31% on entities overall, and pooled WER of 6.99%; all methodology is published at assemblyai.com/benchmarks so you can reproduce it. Beyond accuracy, the practical differences are keyterm prompting for account-specific vocabulary, PII redaction on both audio and text, speaker diarization with live revision, and LLM Gateway for custom call classification through one API. Pricing is $0.21/hr for pre-recorded transcription with per-second billing and no minimums.

What speech-to-text API do call tracking platforms use?

CallRail builds its conversation intelligence on AssemblyAI, as do Calabrio in the enterprise contact center space, Siro in field sales, and WhatConverts for agency lead tracking. The common requirement is entity-level accuracy on business names and callback numbers rather than general transcription quality, plus PII redaction on the stored recording. Pre-recorded transcription runs on Universal-3.5 Pro at $0.21/hr with diarization at $0.02/hr.

How do I redact PII from recorded phone calls?

Redact at both layers, not one. PII Redaction on audio ($0.05/hr) removes the values from the recording itself so the stored file never contains them; PII Redaction on text ($0.08/hr) removes them from the transcript and everything downstream — search indexes, CRM syncs, and any prompts you send to an LLM. Text-only redaction is the common mistake, because the audio file is usually what ends up in a discovery request. AssemblyAI is certified to PCI DSS v4.0 and SOC 2 Type 2, and offers a standard BAA for customers processing PHI.

Can a call tracking platform build a voice agent for inbound calls?

Yes, and call tracking platforms are unusually well positioned to, because they already own the number, the routing and the campaign attribution that tells the agent where the caller came from. The Voice Agent API is a flat $4.50/hr billed per second, bundling streaming speech recognition, a conversation-tuned LLM and text-to-speech in one WebSocket at roughly one second end-to-end latency. Bring-your-own-Twilio SIP trunking is included with no per-minute markup, which matters if you already hold carrier relationships.

Is recording and transcribing calls with AI compliant with regulations?

Compliance depends on your policy and your jurisdiction, not on the transcription vendor — but the vendor determines what's technically possible. In the US, about a dozen states require all-party consent to record, and the TCPA governs outbound automated calling, so platforms typically need per-account recording controls and a configurable disclosure at call start. On the infrastructure side, AssemblyAI holds SOC 2 Type 2, ISO 27001:2022 and PCI DSS v4.0, offers PII redaction on audio and text, provides EU data residency at the same price as US endpoints, and signs a standard BAA for PHI. None of this is legal advice — talk to counsel about your policy.

What does it cost to add transcription and analysis to a call tracking platform?

Pre-recorded transcription is $0.21/hr on Universal-3.5 Pro. A realistic call tracking configuration — transcription, diarization, entity detection, sentiment and dual-layer PII redaction — lands at $0.46/hr, billed per second with no minimums, no concurrency fees and no upfront commitment. Real-time transcription for live agent assist is $0.45/hr, and a fully bundled voice agent is $4.50/hr. The free tier includes 185 hours of pre-recorded and 333 hours of streaming transcription to size it before committing; full detail on the pricing page.

‍

‍