What are word embeddings? From Word2Vec to modern embedding models

Ask Google what word embeddings are today and the answer you get back leads with king − man + woman = queen. That analogy comes from a 2013 paper. It's a great teaching tool and it's still the first thing almost every explainer reaches for.

Meanwhile, you probably called an embedding API this week without thinking about it. Something chunked a document, sent the chunks to a model, got back a few thousand floats per chunk, and stuffed them in a vector index so a chatbot could find them again. Nobody involved was doing word arithmetic.

Both of those things are "embeddings," and the gap between them is where most confusion lives. So this post does two jobs. First, the concepts — dense vectors, embedding space, cosine similarity, and the classic models (Word2Vec, GloVe, fastText, ELMo) that made the idea work. Those are the foundations, and they're genuinely worth understanding because everything since inherits their vocabulary. Second, what actually changed: how the field moved from one vector per word to one vector per passage, what the current embedding models look like, and the dimension and cost tradeoffs that decide what you ship.

Where something is historical context, it's labeled as such. Where it's current practice, that's labeled too.

Why text has to become numbers

The problem embeddings solve is blunt: AI models can't do anything with text. They do arithmetic. Matrix multiplications, gradients, dot products. Feed a model the string "the coffee was excellent" and there's nothing to multiply.

So every NLP pipeline starts with the same step: turn text into numbers. Embeddings are one way to do that. They're not the only way, and they weren't the first — which is the fastest route to understanding what makes them different.

Historical context: what came before embeddings

One-hot encoding

You've probably met one-hot encoding in other contexts. Applied to text, it works like this: build one very long vector, as long as the number of words in your vocabulary. To represent a single word, fill that vector with zeros except for the one cell that corresponds to your word.

A 50,000-word vocabulary means every single word is a 50,000-dimensional vector containing one 1 and 49,999 zeros. It's a valid numerical representation. It's also a spectacularly inefficient use of space, and — more damning — it tells you nothing. "Coffee" and "espresso" are exactly as far apart as "coffee" and "bulldozer." Every word is equidistant from every other word.

Count-based representations

Count-based methods squeeze a whole sentence into a single vector by counting things. Three you'll still see in production:

• Bag of words. Ignore word order entirely. Just count how many times each vocabulary word appears and make that the vector. "The dog bit the man" and "the man bit the dog" come out identical.
• N-grams. Same idea, but instead of counting single words you count groups of n consecutive words. This recovers a little word order at the cost of a much larger feature space.
• TF-IDF. Term frequency–inverse document frequency tracks how often a word appears in one document against how often it appears across the whole corpus. The point is to down-weight words like "the," "of," and "and" that appear everywhere, and up-weight the words that make a specific document distinctive.

These approaches carried NLP for years, and TF-IDF in particular is still a perfectly reasonable baseline — modern hybrid search often runs BM25, a TF-IDF descendant, alongside a vector index. But they share three limits:

1. No context. A word's representation doesn't change based on the words around it.
2. No handling of unseen words. A word that wasn't in the training vocabulary has no representation at all.
3. Sparsity. The vectors are mostly zeros, which wastes both memory and modeling capacity.

What makes an embedding an embedding

The goal of a word embedding is to represent a word as a dense vector, in a space where similar words end up close together. Both halves of that sentence are doing work.

Dense means the vector isn't mostly zeros. Every dimension carries signal. And the embedding typically has far fewer dimensions than your vocabulary has words — 300 dimensions standing in for 50,000 words, rather than one dimension per word.

Similar has a specific technical meaning here, and it's not "means the same thing." It's the distributional hypothesis: similar words are words that get used in similar contexts. "Tea" and "coffee" are similar because they both show up near "breakfast," "drink," and "cup." "Tea" and "pea" are spelled almost identically and are not similar at all, because they appear in completely different company. A good embedding puts tea near coffee and nowhere near pea — which is exactly the thing one-hot encoding can't do.

Embedding space and cosine similarity

Embedding space is just where the vectors live. Embed each word into a single number and you get points on a line, where the distance between two points is their dissimilarity. Embed into two numbers and you get arrows on a plane, with both direction and magnitude. Past three dimensions you lose the ability to picture it, but the math doesn't care.

Embed a word into a vector of length 384 and you have a point in 384-dimensional space. You can't visualize it, but you can still measure how far apart two vectors are. The standard measure is cosine similarity — the cosine of the angle between two vectors, which ranges from −1 (opposite) through 0 (unrelated) to 1 (identical direction). Cosine similarity is preferred over plain Euclidean distance in most text work because it cares about direction rather than magnitude, so a long document and a short query can still score as a close match.

The king − man + woman thing

Here's the demo that made word embeddings famous. In a well-trained embedding space, the relative positions of words encode relationships, not just similarity. The offset from "man" to "woman" turns out to be roughly the same offset as "king" to "queen." So you can do arithmetic on meaning:

>>> result = word_vectors.most_similar(positive=['woman', 'king'], negative=['man'])
>>> most_similar_key, similarity = result[0]
>>> print(f"{most_similar_key}: {similarity:.4f}")
queen: 0.7699

That's the documented example from Gensim's own KeyedVectors reference, and it really does return "queen."

Two caveats that the demo usually skips. First, nobody programmed that relationship in — the model extracted it from raw text, which is the genuinely remarkable part. Second, it's cherry-picked. Try your own analogies and the results get shakier fast. Ask a Word2Vec model for restaurant − dinner + cocktail, hoping for "bar," and you get "eatery," "bartender," "bartenders" — close, not right. GloVe returns "parasol," "espresso," "brewery." fastText does better with "bar restaurant," "restaurant bar," "cocktail making," "wine bar," "nightclub." Analogy arithmetic is a party trick that happens to reveal real structure. It is not a benchmark.

Historical context: how the classic embeddings were trained

All of these learn from large corpora — a lot of text. What differs is what they're trained to predict.

A trainable embedding layer

The simplest option: put an embedding layer in front of your model, initialize it with random weights, and let it learn during training alongside everything else. You end up with an embedding specialized to your data and your task, which is the appeal. The cost is that you need a lot of data and a lot of time to get a good one.

This isn't a historical footnote — it's what the transformer architecture does. Before the encoder and decoder, a transformer has an embedding layer whose whole job is turning tokens into vectors. Every large language model you've used has one.

Word2Vec (2013)

Word2Vec takes one-hot encoded words and learns embeddings by using the surrounding sentence context. Two variants:

• Continuous bag of words (CBOW). Slide a window over the text. Take the words surrounding a middle word, feed them to a neural network, and try to predict the missing middle word.
• Skip-gram. The exact opposite. Take the middle word and try to predict the words that should surround it.

The network is deliberately shallow — one hidden layer, where the number of neurons in that hidden layer is the embedding size. Once the network performs well, you throw away the prediction task and keep the weights: the output weights for CBOW, the incoming weights for skip-gram. The prediction task was never the point. It was scaffolding for learning good vectors.

GloVe (2014)

GloVe stands for Global Vectors, and it extends Word2Vec by looking beyond the local window. Word2Vec only ever sees a few words at a time; GloVe also uses global corpus statistics via a word co-occurrence matrix. The training objective is to learn vectors whose dot product equals the logarithm of the words' probability of co-occurrence. Local context plus global counts.

fastText (2016)

fastText is also an extension of Word2Vec, in a different direction. Instead of training on whole words, it splits words into character n-grams — subwords — and trains on those. A word's embedding is composed from its pieces.

That single change fixes the out-of-vocabulary problem. A word fastText has never seen still has subwords it recognizes, so it still gets a sensible vector. Misspellings degrade gracefully instead of falling off a cliff. And it works much better on morphologically rich languages like German or Turkish, where a single root generates dozens of inflected forms that a whole-word model treats as unrelated tokens. Every modern tokenizer uses some version of this idea.

ELMo (2018)

ELMo is where the story turns. Until ELMo, a word had one vector, permanently. ELMo made the embedding depend on the sentence it appears in. From the paper: "our representations differ from traditional word embeddings in that each token is assigned a representation that is a function of the entire input sentence."

ELMo's representations come from a bidirectional LSTM trained on a language modeling task — predicting the next and previous words. Training on language modeling forces the model to absorb context, which means it can finally distinguish homonyms. "He was known to be fair" and "the fair was so much fun" get different vectors for "fair." And because ELMo's first layer operates on characters rather than whole words, it handles typos well.

ELMo's specific architecture didn't last — BERT arrived months later and did contextual representation better with transformers. But the conceptual shift did last, completely. Contextual is now the default, and "static word embedding" is the special case.

Trying the classics yourself

The classic models are still a few lines away, and poking at them is the fastest way to build intuition. Gensim ships a downloader for pretrained vectors:

import gensim.downloader as api

api.info()  # return dict with info about available models/datasets

model = api.load("glove-twitter-25")  # load glove vectors
model.most_similar("cat")  # show words that similar to word 'cat'

api.info() lists what's available, and the catalog is a nice tour of the era: word2vec-google-news-300 (3,000,000 vectors, 1,662 MB), glove-wiki-gigaword-100 (400,000 vectors, 128 MB), glove-twitter-25 (1,193,514 vectors, 104 MB), fasttext-wiki-news-subwords-300 (999,999 vectors, 958 MB). Same algorithms, different training corpora — Google News, Wikipedia, Twitter — and the corpus shows in the results.

Ask each model for the nearest neighbors of "tea" and you get a useful lesson in training data. GloVe returns coffee, milk, wine, cream, ice, juice — beverages, tightly clustered. fastText returns tea, coffee, teas, tea bags — heavily influenced by its subword decomposition, so it surfaces morphological relatives alongside semantic ones. Neither is wrong. They're answering slightly different questions.

You can also check distances directly, which is where the tea/pea point becomes concrete rather than rhetorical:

>>> similarity = word_vectors.similarity('woman', 'man')
>>> similarity > 0.8
True

In a Word2Vec model, the cosine distance between "tea" and "coffee" comes out around 0.43, while "tea" and "pea" sit around 0.7. Farther apart, despite one letter of difference. That's the whole promise of embeddings in one number.

Current practice: from word embeddings to text embeddings

Here's the shift that matters most, and it's the one 2022-era explainers miss. In practice, almost nobody embeds individual words anymore.

What you embed now is a span of text — a sentence, a paragraph, a chunk of a document, a whole support ticket — and you get back one vector representing the whole thing. The term of art moved from "word embeddings" to "text embeddings" or "sentence embeddings" for exactly that reason.

Why the change? Because contextual transformers made per-word vectors an implementation detail rather than a product. Inside any transformer, tokens still get vectors and those vectors still get contextualized layer by layer. But the thing you want for search, clustering, classification, or retrieval is a single fixed-size representation of a passage, not 400 per-token vectors you have to figure out how to combine.

And combining them is harder than it sounds. Mean-pooling raw BERT token vectors produces a sentence representation that's famously mediocre at similarity tasks — the model was never trained to make its pooled output meaningful. What fixed it was training encoders explicitly on the similarity objective, with contrastive learning that pulls related pairs together and pushes unrelated pairs apart. That's the Sentence Transformers lineage, and it's the ancestor of every embedding API you can call today.

Three other things changed alongside it:

• Task-aware embeddings. Modern models want to know what you're embedding for. Cohere's API takes an input_type of search_query, search_document, classification, clustering, or image, and embeds differently for each. Qwen3-Embedding supports instruction customization for task-specific optimization. A query and the document it should match are asymmetric, and the good models now model that asymmetry.
• Long context. Word2Vec had a window of a few words. Voyage's current models take 32,000 tokens; Cohere's embed-v4.0 takes 128,000. You can embed an entire contract.
• Multimodal. Cohere's embed-v4.0 handles text, images, and mixed content like PDFs in a shared space, so a text query can retrieve a chart.

What a modern embedding call looks like

Three examples, all from official documentation. The OpenAI API:

from openai import OpenAI

client = OpenAI()

response = client.embeddings.create(
    input="Your text string goes here",
    model="text-embedding-3-small"
)

print(response.data[0].embedding)

Google's Gemini API, with the output dimension set explicitly:

from google import genai
from google.genai import types

client = genai.Client()

result = client.models.embed_content(
    model="gemini-embedding-2",
    contents="What is the meaning of life?",
    config=types.EmbedContentConfig(output_dimensionality=768)
)

[embedding_obj] = result.embeddings
embedding_length = len(embedding_obj.values)

print(f"Length of embedding: {embedding_length}")

And running a model yourself with Sentence Transformers, which is still the shortest path to embeddings that never leave your machine:

from sentence_transformers import SentenceTransformer

# 1. Load a pretrained Sentence Transformer model
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# The sentences to encode
sentences = [
    "The weather is lovely today.",
    "It's so sunny outside!",
    "He drove to the stadium.",
]

# 2. Calculate embeddings by calling model.encode()
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# 3. Calculate the embedding similarities
similarities = model.similarity(embeddings, embeddings)
print(similarities)

Notice how little the interface has changed since model.most_similar("cat"). Text in, vector out, compare with cosine. The models got radically better. The shape of the API didn't.

The 2026 embedding model landscape

Here's where the field sits now, split between hosted APIs and open-weight models you can run yourself.

Two patterns are worth pulling out of that table. Every one of these models has flexible dimensions, which was not true three years ago. And the open-weight models are no longer the compromise option — Qwen3-Embedding-8B held the top spot on MTEB's multilingual leaderboard, and EmbeddingGemma runs a genuinely useful embedding model on a phone.

On benchmarks: MTEB, the Massive Text Embedding Benchmark, is the standard reference, spanning 56 datasets across 8 tasks and up to 112 languages. Read it with the caveat its own authors give — "model performance varies a lot depending on the task and dataset," so check the tab that matches your task rather than the headline average. There is no single best embedding model. There's a best model for retrieval on your domain, in your languages, at your latency budget.

Dimensions, storage, and cost

This is the part the tutorials skip, and it's usually what decides your architecture.

An embedding is a list of floats. At float32, each dimension costs 4 bytes. So a 3072-dimensional vector is about 12 KB. Embed a million chunks and you're storing roughly 12 GB of vectors — before indexes, before metadata, before replicas. Drop to 768 dimensions and the same corpus is about 3 GB. Every query also gets faster, because there's less arithmetic per comparison.

Historically you'd have to switch models to change that. Now you don't, thanks to Matryoshka Representation Learning — training that front-loads the most important information into the earliest dimensions, so you can truncate a vector and it still works. OpenAI describes it plainly: developers can shorten embeddings, removing numbers from the end of the sequence, "without the embedding losing its concept-representing properties." Google notes that gemini-embedding-2 at 768 dimensions delivers performance comparable to its higher-dimension counterparts. One model, one API call, four different storage budgets.

The second lever is quantization. Instead of shrinking the vector, shrink each number. Cohere and Voyage both return int8, uint8, binary, and ubinary alongside float. int8 is a 4x reduction against float32. Binary is dramatic: as Cohere's docs put it, "if you have a vector of 1024 binary embeddings, it will become 1024/8 => 128 bytes." That's 32x smaller than the float32 equivalent, and binary vectors compare with Hamming distance, which is very fast.

The practical recipe most teams land on: retrieve a wide candidate set with cheap binary or int8 vectors, then rescore the top few hundred with full-precision vectors or a reranker. You keep most of the quality and a fraction of the cost.

Per-token API pricing is the smallest lever of the three, and it moves often — Voyage lists $0.12 per million tokens for voyage-4-large, $0.06 for voyage-4, and $0.02 for voyage-4-lite, with 200 million free tokens per account, but check each provider's current pricing page before you model costs. Embedding is usually cheap relative to generation. Storing and searching the vectors is what adds up.

Where embeddings sit in RAG

Retrieval-augmented generation is why embeddings went from an NLP subtopic to infrastructure. The pipeline is short:

1. Chunk. Split your corpus into passages. This is the step people underinvest in and then blame the model for — chunk boundaries that split a thought in half produce vectors that represent neither half. Semantic chunking, informed by work on text segmentation, beats fixed-size splitting almost every time.
2. Embed and index. Run each chunk through an embedding model with input_type=search_document or the equivalent, and store the vectors in a vector database.
3. Embed the query. Same model, but as a query rather than a document.
4. Retrieve. Cosine similarity against the index, top-k results.
5. Rerank. Optionally rescore those candidates with a cross-encoder, which reads query and passage together and is far more accurate than comparing two independent vectors.
6. Generate. Hand the surviving passages to an LLM as context.

Notice that the embedding model only touches steps 2 through 4. It's one component in a system where chunking strategy, hybrid keyword-plus-vector retrieval, and reranking often matter more than which model produced the vectors. Swapping text-embedding-3-large for voyage-4-large rarely fixes a retrieval system. Fixing the chunks usually does. That's the single most useful thing to know if you're building LLM applications right now.

Embeddings in Voice AI: voices and transcripts

Embeddings aren't a text-only idea. The same trick — map something messy into a vector space where distance means similarity — is how modern speech systems handle voices.

Speaker embeddings

A speaker embedding is a vector representing someone's voice rather than their words — pitch, timbre, cadence, resonance, the shape of their vowel formants. Two clips of the same person land close together; two different people land far apart. That's the mechanism behind speaker diarization and speaker identification, and conceptually it's identical to word embeddings. Different input, same geometry.

The numbers are smaller than you'd expect. AssemblyAI's speaker identification guide builds a voice-matching system on NVIDIA's TitaNet model, which produces 192-dimensional speaker embeddings — a fraction of a text embedding's size. You generate a fingerprint per known speaker, store it in a vector database, and match new utterances by cosine similarity:

from nemo.collections.asr.models import EncDecSpeakerLabelModel

speaker_model = EncDecSpeakerLabelModel.from_pretrained("nvidia/speakerverification_en_titanet_large")
# Upload the known speakers.
for speaker, audio_file in known_speakers.items():
    embedding = speaker_model.get_embedding(audio_file)
    add_speaker_embedding_to_pinecone(speaker, embedding)

Then identification is a vector search with a confidence threshold:

from sklearn.metrics.pairwise import cosine_similarity

def find_closest_speaker(utterance_embedding, local_embeddings=None, local_only=False, threshold=0.5):
    def cosine_sim(a, b):
        return cosine_similarity(a.reshape(1, -1), b.reshape(1, -1))[0][0]

    best_match = {"speaker_name": "No match found", "score": 0}
    ...
    # Check if the best match meets the threshold.
    if best_match["score"] < threshold:
        return "No match found", 0

    return best_match["speaker_name"], best_match["score"]

Tea and coffee, 0.43 apart. Two recordings of the same voice, similarly close. Same idea, different modality.

Embedding transcripts for semantic search

The other half is more directly useful: audio is a retrieval corpus that most teams never index. Calls, meetings, interviews, and podcasts contain the answers people search for, and they're invisible to a vector index until they're text.

The speech-to-text step is what makes them searchable, and the pattern is the standard RAG pipeline with one twist: transcript chunks carry timestamps, so a retrieval hit points at a moment in the audio, not just a passage. AssemblyAI's docs walk through exactly this — transcribe with Universal-3.5 Pro, pull sentences with timestamps, then embed a sliding window over them:

# Initialize embedder
embedder = SentenceTransformer("multi-qa-mpnet-base-dot-v1")
embeddings = {}

# Create sliding window of sentences and generate embeddings
sentence_groups = sliding_window(sentences, 5, 2)

for sentence_group in sentence_groups:
    combined_text = " ".join([sentence["text"] for sentence in sentence_group])
    start = sentence_group[0]["start"]
    end = sentence_group[-1]["end"]

    embeddings[(start, end, transcript_id, combined_text)] = embedder.encode(combined_text)

Five sentences per window with two overlapping — roughly 30-second chunks, with overlap so an answer that straddles a boundary still appears intact in at least one chunk. That's semantic chunking applied to speech. The retrieval side is plain cosine nearest-neighbors:

knn = NearestNeighbors(n_neighbors=3, metric="cosine")
knn.fit(np_embeddings)
distances, indices = knn.kneighbors([llm_gateway_embedding])

Each match comes back with a start timestamp, an end timestamp, and a confidence — so you can jump straight to the audio. Combine that with the LLM Gateway for the generation step, or the Speech Understanding API for structured metadata to filter on before you ever run a vector search, and you have conversational RAG.

One warning, and it's the whole reason accuracy matters upstream of embeddings: a transcription error becomes an embedding error becomes a retrieval miss. If a product name is transcribed wrong, no amount of embedding quality will find it. As Dr. Shane Lynn, CEO of EdgeTier, puts it:

The transcript quality is critical, both for user perception and our AI models. Once you lose trust in transcript accuracy, you erode trust in the product. For text classification, phrase detection, and agent evaluation, the language has to be correct — otherwise, the whole system falls apart.

Embeddings are exactly the kind of downstream system he's describing. Garbage in, confidently-scored garbage out.

What actually changed, and what didn't

Read the classic papers and then read a current embedding API reference, and the surprising thing is how little the interface moved. Text in, dense vector out, cosine similarity to compare. That contract has held for over a decade.

What changed is where the difficulty lives. In 2013, the hard part was producing a good vector — training Word2Vec on a large corpus was real work, and a well-behaved embedding space was an achievement. In 2026, producing the vector is one HTTP request from a dozen vendors, at fractions of a cent, with the model choice mattering less than the marketing suggests.

The hard part moved downstream. It's now chunking, query-document asymmetry, hybrid retrieval, reranking, dimension and quantization budgets, and — for anyone working with conversations — whether the text being embedded is a faithful record of what was said. Those are engineering decisions, not model decisions.

Which is a good reason to learn Word2Vec anyway. Not because you'll deploy it, but because once you've watched a shallow network learn that tea sits near coffee purely from co-occurrence, every strange retrieval result you ever debug will make more sense. The geometry hasn't changed. Only the scale has.

Frequently asked questions

What are the different types of word embeddings?

They fall into two groups. Static embeddings give each word one fixed vector regardless of context — Word2Vec, GloVe, and fastText are the canonical examples. Contextual embeddings give a word a different vector depending on the sentence it appears in, starting with ELMo and now standard in every transformer-based model. Modern practice adds a third category: sentence or text embeddings, which represent a whole passage as a single vector.

What is Word2Vec vs BERT?

Word2Vec produces one static vector per word, learned from local context windows with a shallow single-hidden-layer network. BERT is a deep bidirectional transformer that produces a different vector for a word every time, based on the full sentence around it — so "bank" in "river bank" and "bank account" get different representations. BERT is far more capable, but Word2Vec is dramatically cheaper and its vectors can be looked up from a table with no inference at all.

Do LLMs use embeddings?

Yes, in two distinct places. Every LLM has an internal embedding layer that converts input tokens into vectors before the first transformer block — that's unavoidable architecture. Separately, LLM applications use standalone embedding models to power retrieval, so the model can be handed relevant context it wasn't trained on. Those are different models doing different jobs, which is a common source of confusion.

Can you give me an example of a word embedding?

In a 300-dimensional GloVe model, the word "tea" is a list of 300 floating-point numbers. On its own that list is meaningless, but its position relative to other words is not: the cosine distance from "tea" to "coffee" is roughly 0.43, while "tea" to "pea" is roughly 0.7 — farther apart despite nearly identical spelling, because the two words appear in completely different contexts.

How do I choose an embedding model for RAG?

Start with the MTEB leaderboard tab that matches your actual task and languages rather than the overall average, since performance varies a lot by task and dataset. Then filter by practical constraints: maximum input length against your chunk size, output dimensions against your storage budget, whether quantized output is supported, and whether you need to run it on your own hardware. Test two or three candidates on your own queries and documents — a domain-specific evaluation of 50 real queries tells you more than any public benchmark.

How many dimensions should my embeddings have?

Fewer than the default, usually. Most current models are trained with Matryoshka Representation Learning, so you can truncate the vector and keep most of the quality — Google notes that gemini-embedding-2 at 768 dimensions performs comparably to its higher-dimension configurations. Since a 3072-dimensional float32 vector costs about 12 KB versus about 3 KB at 768, start low, measure retrieval quality on your own data, and only add dimensions if the metrics ask for them.

Can you embed audio the same way you embed text?

Yes, and it's standard practice. Speaker embeddings map a voice to a vector — NVIDIA's TitaNet produces 192-dimensional vectors — so two recordings of the same person land close together under cosine similarity, which is how speaker diarization and speaker identification work. For semantic search over spoken content, the usual approach is to transcribe first, then embed the transcript in timestamped chunks so a retrieval hit points to a specific moment in the recording.

‍