What is a vector database? Embeddings, indexes, and RAG over audio transcripts

Three years ago we published a four-minute explainer on vector databases. It has since passed 675,000 views, which tells you something: a lot of developers still want a straight answer to "what is a vector database, and do I actually need one?"

The answer has gotten more interesting since 2023. Back then, vector databases were the hot new primitive and companies were raising hundreds of millions of dollars to build them. Today the index itself is close to a commodity — Postgres has pgvector, Elasticsearch and OpenSearch ship vector search, Redis has it built in, and Cloudflare will run one at the edge. What hasn't commoditized is the part nobody talks about: the quality of the data you put in.

That gap matters most for spoken data. If you're building retrieval over calls, meetings, podcasts, or support conversations, your retrieval quality is capped by your transcript quality long before it's capped by your choice of index. So this post does two things: it explains vector databases from first principles — embeddings, indexes, similarity search — and then shows you the actual code for running RAG over audio data, where most of the interesting problems live.

What is a vector database?

A vector database indexes and stores vector embeddings for fast retrieval and similarity search. That's the whole definition, and it has two load-bearing halves.

Embeddings are the representation: a list of numbers that captures the meaning of a piece of data. Indexes are the access path: a data structure that makes finding nearby vectors fast enough to query in production. Most explanations cover the first and skip the second, which is why people come away thinking a vector database is just an array of floats. It isn't — the index is the database part.

Everything else you'll read about vector databases (metadata filtering, hybrid search, sharding, CRUD, backups) is infrastructure built around those two ideas.

Why relational databases struggle with unstructured data

The usual estimate is that the large majority of data in the world — somewhere north of 80% — is unstructured: social media posts, images, video, and audio. None of that fits neatly into rows and columns.

Take an image. If you want to store it in a relational database and later search for similar images, what typically happens is that somebody manually assigns keywords or tags to it. From the raw pixel values alone, you can't meaningfully search for "images like this one." The same is true for a wall of unstructured text, and it's especially true for audio and video.

So you have two options. Tag everything by hand, which doesn't scale and encodes whatever vocabulary the tagger happened to use that day. Or find a different representation of the data — one that a computer can compare directly.

That second option is what embeddings give you.

Vector embeddings: turning speech, text, and images into numbers

An embedding is a list of numbers that represents your data in a different way. AI models produce them. You can embed a single word, a whole sentence, an image, or — relevant here — a chunk of a transcript.

Once your data is numerical, one very useful thing falls out for free: you can find similar items by measuring the distance between vectors and doing a nearest-neighbor search. Items that mean similar things land near each other. Items that don't, don't.

Diagrams always draw this in two dimensions because two dimensions fit on a slide. Real embeddings have hundreds or thousands of dimensions. The all-mpnet-base-v2 model used later in this post produces 768-dimensional vectors; plenty of production systems run 1,536 or higher.

The distance metric you pick matters more than people expect:

• Cosine similarity compares direction and ignores magnitude. It's the usual default for text and transcript search.
• Euclidean (L2) distance compares absolute position. Fine when magnitude carries meaning.
• Dot product is fastest and works well when your vectors are already normalized.

Worth noting: embeddings show up all over Voice AI, not just in retrieval. Speaker embeddings are how diarization figures out who spoke when — a voice gets mapped to a vector, and similar vectors get clustered into the same speaker. Same math, different job.

Indexes: why storing embeddings isn't enough

Here's where it gets interesting. Storing your data as embeddings gets you nothing on its own.

Running a query across thousands of vectors by computing an exact distance to every single one is brute force, and it's extremely slow. Scale that to millions of vectors and it's simply not a thing you can do inside a request. That's why the vectors also need to be indexed.

An index is a data structure that facilitates the search process. The indexing step maps your vectors into a new structure that enables much faster searching. This is a whole research field on its own — HNSW graphs, IVF, product quantization, DiskANN — and the details vary by database.

The one thing you do need to internalize: these are approximate nearest neighbor (ANN) algorithms. They trade a small amount of recall for an enormous gain in speed. You're not getting the mathematically perfect top-10 anymore; you're getting a very good top-10, thousands of times faster. For semantic search over transcripts that tradeoff is almost always correct. For something like a financial reconciliation, it might not be.

What vector databases are actually used for

Four patterns cover the overwhelming majority of real deployments.

Long-term memory for large language models. This is RAG — retrieval-augmented generation. You embed your own documents, retrieve the relevant chunks at query time, and pass them into the model's context. It's the most common reason developers reach for a vector database, and it's the reason the category exploded alongside LLM applications.

Semantic search. Searching by meaning instead of exact string match. A user asks "how do we handle refunds after 30 days" and you surface the passage that says "returns outside the one-month window require manager approval" — no shared keywords at all.

Similarity search over images, audio, and video. "Find me something like this one," without describing it in words first.

Ranking and recommendations. For an online retailer, suggesting items similar to a customer's past purchases is just finding the nearest neighbors of an item in your catalog.

Do you actually need a vector database?

Probably not on day one, and this hasn't changed since 2023.

For many projects a dedicated vector database is overkill. A traditional database with a vector extension works fine. So does an in-process library. So, honestly, does a NumPy array — if you have a few thousand vectors and you're computing cosine similarity in memory, that's a legitimate architecture, not a hack. The code later in this post uses scikit-learn's NearestNeighbors for exactly that reason.

Reach for a dedicated vector database when you hit one of these:

• Your corpus is large enough that exact search blows your latency budget (roughly: hundreds of thousands of vectors and up).
• You need metadata filtering alongside similarity — "only search transcripts from this account, in the last 90 days."
• You need real-time upserts, not a nightly rebuild.
• You need durability, replication, and someone else's on-call rotation.

If none of those apply yet, start with pgvector in the Postgres you already run. Migrating later is easy; the embeddings are portable, and the index is the part you're throwing away anyway.

Your vector database options in 2026

The 2023 list — Pinecone, Weaviate, Chroma, Redis, Qdrant, Milvus, Vespa — is all still here, which is more than you can say for most 2023 AI infrastructure. What changed is that the "add vectors to the database you already have" camp got real.

Pick on operational fit, not benchmarks. Every one of these will do approximate nearest neighbor search well enough that your retrieval quality is determined by your embeddings and your chunking — not by the index.

Vector databases for audio: RAG over transcripts

Here's the case the generic tutorials skip. RAG needs text documents, and a huge amount of the most valuable data in a company isn't text — it's recorded calls, meetings, webinars, interviews, and support conversations. You can't embed an .mp3 and expect semantic search over what was said in it.

So the pipeline gets one extra step at the front: transcribe first, then chunk, embed, store, and retrieve. The code below comes from AssemblyAI's rag-langchain-audio-data example repo, and pairs with our guide to using audio data in LangChain with Python.

Install the dependencies:

pip install assemblyai langchain openai python-dotenv chromadb 
sentence-transformers

Step 1: transcribe the audio into documents

The AssemblyAI document loader for LangChain handles transcription and hands you back LangChain documents, so audio enters your pipeline looking exactly like any other source:

from langchain.document_loaders import AssemblyAIAudioTranscriptLoader

URLs = [
    "https://storage.googleapis.com/aai-web-samples/langchain_agents_webinar.opus",
    "https://storage.googleapis.com/aai-web-samples/langchain_document_qna_webinar.opus",
    "https://storage.googleapis.com/aai-web-samples/langchain_retrieval_webinar.opus"
]

def create_docs(urls_list):
    l = []
    for url in urls_list:
        print(f'Transcribing {url}')
        l.append(AssemblyAIAudioTranscriptLoader(file_path=url).load()[0])
    return l

docs = create_docs(URLs)

The transcript text lands in page_content and the full API response — language, audio URL, formatting settings, and any Speech Understanding results you enabled — lands in metadata.

You can also pass a config object to turn on features that make retrieval better downstream. Speaker labels are the big one, because "who said this" is usually part of the answer:

import assemblyai as aai
from langchain.document_loaders import AssemblyAIAudioTranscriptLoader

config = aai.TranscriptionConfig(
    speaker_labels=True, auto_chapters=True, entity_detection=True
)

loader = AssemblyAIAudioTranscriptLoader(file_path="./your_file.mp3", config=config)

Step 2: chunk the transcript

The example repo splits on character count, which is the standard approach:

from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(docs)

# modify metadata because some AssemblyAI returned metadata is not in a compatible form for the Chroma db
for text in texts:
    text.metadata = {"audio_url": text.metadata["audio_url"]}

But speech has natural boundaries that a character counter can't see, and the loader can split on them for you. Setting transcript_format returns multiple documents split by sentence or paragraph instead of one blob:

from langchain.document_loaders import AssemblyAIAudioTranscriptLoader
from langchain.document_loaders.assemblyai import TranscriptFormat

loader = AssemblyAIAudioTranscriptLoader(
    file_path="./your_file.mp3",
    transcript_format=TranscriptFormat.SENTENCES,
)

docs = loader.load()

The available options are TEXT, SENTENCES, PARAGRAPHS, SUBTITLES_SRT, and SUBTITLES_VTT. Chunking on paragraphs tends to beat chunking on 1,000 characters for conversational audio, because it stops cutting sentences in half mid-thought.

Step 3: embed the chunks and store them

from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import Chroma

def make_embedder():
    model_name = "sentence-transformers/all-mpnet-base-v2"
    model_kwargs = {'device': 'cpu'}
    encode_kwargs = {'normalize_embeddings': False}
    return HuggingFaceEmbeddings(
        model_name=model_name,
        model_kwargs=model_kwargs,
        encode_kwargs=encode_kwargs
    )

hf = make_embedder()
db = Chroma.from_documents(texts, hf)

That's the vector database, created in one line. Chroma runs in-process here, which is the right call for a prototype.

Step 4: retrieve and generate

from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

def make_qa_chain():
    llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)
    return RetrievalQA.from_chain_type(
        llm,
        retriever=db.as_retriever(search_type="mmr", search_kwargs={'fetch_k': 3}),
        return_source_documents=True
    )

Note search_type="mmr" — maximal marginal relevance. Straight similarity search over a transcript tends to return three chunks that all say roughly the same thing, because speakers repeat themselves. MMR penalizes redundancy in the result set, which matters much more for spoken data than for documentation.

And return_source_documents=True gives you citations. In this example each chunk carries its audio_url, so every answer can point back to the recording it came from.

Retrieving with timestamps, not just text

One thing transcripts give you that documents don't: every word has a time. If you keep those timestamps in your vector metadata, retrieval stops returning quotes and starts returning moments you can jump to.

Our transcript citations guide does this without a vector database at all — sliding-window groups of sentences, embedded with a search-tuned model, then a nearest-neighbor search in memory:

def sliding_window(elements, distance, stride):
    """Create sliding windows of elements"""
    idx = 0
    results = []
    while idx + distance < len(elements):
        results.append(elements[idx:idx + distance])
        idx += (distance - stride)
    return results

# Initialize embedder
embedder = SentenceTransformer("multi-qa-mpnet-base-dot-v1")
embeddings = {}

# Create sliding window of sentences and generate embeddings
sentence_groups = sliding_window(sentences, 5, 2)

for sentence_group in sentence_groups:
    combined_text = " ".join([sentence["text"] for sentence in sentence_group])
    start = sentence_group[0]["start"]
    end = sentence_group[-1]["end"]

    embeddings[(start, end, transcript_id, combined_text)] =
embedder.encode(combined_text)

Five sentences per window with two overlapping keeps each chunk semantically whole while capping quote length at roughly 30 seconds. The start and end times ride along in the key, so a match comes back with a timestamp attached.

Then the search itself, using cosine distance rather than the default Euclidean:

from sklearn.neighbors import NearestNeighbors

np_embeddings = np.array(list(embeddings.values()))
metadata = list(embeddings.keys())

knn = NearestNeighbors(n_neighbors=3, metric="cosine")
knn.fit(np_embeddings)
distances, indices = knn.kneighbors([llm_gateway_embedding])

That's a complete semantic search over audio in about 20 lines, with no vector database in the stack. Add one when the corpus outgrows memory — not before.

The part that actually determines your retrieval quality

You can tune chunk_size and swap HNSW for IVF all week. If the transcript renders an account number as 4-0-8 when the caller actually said 4-8-0, no index recovers that. The embedding faithfully encodes the wrong words, retrieval faithfully returns them, and your model faithfully cites them.

This is why transcript accuracy is a retrieval problem, not just a transcription problem. Three failure modes hit RAG over audio specifically:

• Entity errors. Names, account numbers, addresses, and product SKUs are exactly what people search for, and exactly what generic models get wrong. Universal-3.5 Pro was built around this, with contextual prompting so you can prime the model with the domain vocabulary and prior context it's about to hear.
• Missing speaker attribution. "We'll waive the fee" means something different depending on who said it. Universal-3.5 Pro produces the transcript and the speaker diarization jointly, and it's optimized for cpWER rather than DER — 30.17 average cpWER versus 37.92 for Deepgram Nova-3 English and 35.26 for ElevenLabs Scribe v2.
• Language switching mid-sentence. A chunk that garbles half its words because the speaker switched languages is a chunk that will never be retrieved. Universal-3.5 Pro handles native code-switching across 18 languages with no configuration.

Metaview, which builds an AI notetaker for recruiting conversations, put the point well:

"Since moving to AssemblyAI, we've seen a meaningful improvement in the confidence tail of our production transcripts....What stands out is not just the model quality, but the way [they] let us bring real meeting context into transcription, from calendar titles to organizations, domains, and participant names, so recruiting conversations come through with the nuance our customers depend on."

— Shahriar Tajbakhsh, Co-founder and CTO, Metaview

The confidence tail is the right thing to watch. Average accuracy tells you how the transcript reads; the tail tells you which chunks are quietly unretrievable.

Practical version: transcription runs at $0.21/hr with Universal-3.5 Pro, billed per second. It is almost certainly the cheapest line item in your RAG pipeline and the one with the largest effect on output quality. Spend there before you spend on a managed index.

Where this goes next

The interesting shift since 2023 isn't that vector databases got better. It's that they got boring — in the good way, the way Postgres is boring. ANN search is now a feature you switch on in infrastructure you already run, and the differentiation moved one layer up.

Which means the competitive question for anyone building retrieval over conversations is no longer "which vector database." It's whether your pipeline preserves what was actually said, who said it, and when. Get the transcript right and a NumPy array will beat a competitor running a managed index on bad text. That's an uncomfortable conclusion if you've been shopping for databases, and a useful one if you haven't started yet.

Frequently asked questions

What is a vector database and how does it work?

A vector database indexes and stores vector embeddings for fast retrieval and similarity search. It works in two stages: an AI model converts your data into embeddings — lists of numbers that capture meaning — and an index maps those vectors into a structure that supports fast approximate nearest neighbor search. At query time your query is embedded too, and the database returns the vectors closest to it by cosine, Euclidean, or dot-product distance.

Is a vector database SQL or NoSQL?

Most dedicated vector databases are closer to NoSQL — they're purpose-built document or key-value stores with an ANN index rather than relational engines. But the distinction is blurring fast. pgvector adds vector columns and similarity operators to Postgres, so you can run vector search in plain SQL alongside joins and transactions, and Redis, Elasticsearch, and OpenSearch have all added vector search to existing engines.

What are the top vector databases to consider?

The dedicated options most teams evaluate are Pinecone (fully managed), Chroma (embedded, good for prototyping), Qdrant (strong metadata filtering), Weaviate (hybrid keyword-plus-vector search), Milvus (distributed, billion-scale), and Vespa (retrieval and ranking engine). The vector-enabled alternatives are pgvector for Postgres, Redis, and Elasticsearch or OpenSearch. Pick based on operational fit — all of them do ANN search well enough that your embeddings and chunking matter more than your index choice.

Is there a free vector database?

Yes. Chroma, Qdrant, Weaviate, Milvus, and Vespa are all open source and free to self-host, and pgvector is a free extension for a Postgres instance you may already be running. Most managed services also offer a free tier for development. For small corpora you can skip the database entirely and run cosine similarity over a NumPy array or scikit-learn's NearestNeighbors.

Can you use a vector database for audio and video data?

Yes, and there are two distinct approaches. You can embed the audio signal directly for acoustic similarity search — "find clips that sound like this" — or you can transcribe the audio to text first and embed the transcript, which is what you want for semantic search and RAG over what was said. The second approach is far more common for calls, meetings, and podcasts, because it lets speech reuse the same retrieval stack as the rest of your documents.

How do I build RAG over audio files or meeting transcripts?

Transcribe the audio into text documents, split the transcript into chunks, embed each chunk, store the embeddings in a vector store, then retrieve the relevant chunks at query time and pass them to an LLM. The AssemblyAI document loader for LangChain collapses the first step into a few lines and returns transcripts as LangChain documents, with optional speaker labels and entity detection carried in the metadata. Keep the timestamps in that metadata so every retrieved answer can link back to the exact moment in the recording.

‍