Word Embeddings

Before attention-based contextual embeddings (Attention & Transformers), NLP's biggest representational leap was learning that words could be represented as dense vectors capturing meaning — not just an arbitrary ID or a one-hot vector with no relationship to any other word.

Before Embeddings: Count Vectorization and TF-IDF

What is it? The classical, pre-neural way to turn text into numbers a model can use: represent every document as a vector of word counts over the vocabulary — no learning, no notion of meaning, just arithmetic on which words appear and how often.

How does it work? A Count Vectorizer builds a fixed vocabulary from a corpus and represents each document as a vector where position $i$ holds how many times vocabulary word $i$ appears in that document — a vector as long as the entire vocabulary, almost entirely zeros for any real document. TF-IDF (Term Frequency–Inverse Document Frequency) refines raw counts with the same two corrections BM25 is itself built on: term frequency (how often a word appears in this document) weighted down by how common that word is across the whole corpus — "the" appearing 10 times contributes almost nothing, a rare technical term appearing twice contributes a lot.

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "This is the first document.",
    "This document is the second document.",
    "And this is the third one.",
]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)  # sparse matrix, shape (3, vocab_size)
print(vectorizer.get_feature_names_out())  # the learned vocabulary

Real, from scratch — the same corpus, no library, so you can see exactly what "term frequency weighted by inverse document frequency" actually computes:

Why is it useful? It's simple, fast, needs no training, and works surprisingly well as a baseline for search and document classification — a real production keyword-search system (and BM25 specifically) is a direct descendant of this exact idea.

Limitation: Every vector is sparse (vocabulary-sized, almost all zeros) and carries no semantic meaning — "dog" and "puppy" get completely unrelated vectors despite being near-synonyms, because TF-IDF only knows which exact words appeared, never what they mean. Word order is thrown away entirely too (a "bag of words").

TF-IDF's core limitation — no notion of meaning, only exact word overlap — is exactly the gap dense word embeddings were built to close.

The Core Idea: The Distributional Hypothesis

"You shall know a word by the company it keeps" — words that appear in similar contexts tend to have similar meanings. Every method on this page operationalizes that single linguistic insight differently, but it's the shared foundation: learn a word's vector representation from the other words that tend to surround it across a large text corpus, with no explicit definition of "meaning" ever provided.

word2vec

Introduced two closely related, shallow neural network training schemes for learning word vectors directly from raw text with no labels:

• CBOW (Continuous Bag of Words): predict a target word from its surrounding context words — given "the ___ sat on the mat," predict "cat." Faster to train, tends to work better on frequent words.
• Skip-gram: the reverse — given a target word, predict its surrounding context words. Slower to train, tends to work better on rare words (it gets more training signal per rare-word occurrence, since it's predicting multiple context words from that single occurrence rather than needing many occurrences to average over as CBOW's context does).
• Why it works despite the shallow architecture: neither is really "about" the prediction task itself — the prediction task is just a means to force the model to compress word co-occurrence statistics into a dense vector's weights via backpropagation. The learned embedding matrix (mapping each vocabulary word to a dense vector) is the actual product; the CBOW/skip-gram prediction head is discarded after training.
• The famous linear-structure property: word2vec embeddings famously support vector arithmetic that reflects semantic relationships — $\text{vec("king")} - \text{vec("man")} + \text{vec("woman")} \approx \text{vec("queen")}$ — direct empirical evidence that the learned vector space captures real relational structure, not just similarity clustering.

Try that exact analogy (and others) below — real cosine similarity, real nearest-neighbor lookup, and a real PCA projection down to 2D, computed live over a small hand-built demo vocabulary (not a trained model):

• Negative sampling: computing a full softmax over the entire vocabulary at every training step is expensive at real vocabulary sizes (see Loss Functions) — negative sampling reframes training as binary classification (is this context word real, or one of a handful of randomly sampled "negative" fake ones), a large practical speedup that made training word2vec at scale feasible. Real per-step operation counts, not a vague "it's faster":

GloVe (Global Vectors)

Where word2vec learns from local context windows one training example at a time, GloVe works directly from the corpus-wide co-occurrence matrix (how often every word appears near every other word, counted across the entire corpus), factorizing it — in spirit, the same low-rank matrix factorization idea as Recommender Systems and SVD, applied to word co-occurrence counts instead of ratings. GloVe's explicit use of global corpus statistics (rather than word2vec's local-window-at-a-time training) was its original selling point; in practice, the two methods produce embeddings of broadly comparable quality, and the choice between them mattered less than the shared idea both represent.

fastText and Subword Embeddings

fastText extends word2vec by representing each word as a bag of character n-grams rather than a single atomic unit — "apple" is represented via pieces like "app", "ppl", "ple" in addition to the whole word. Two direct practical payoffs: it handles out-of-vocabulary words at inference time (a novel word can still get a reasonable embedding, built from its recognizable character pieces, where word2vec/GloVe simply have no vector for a word never seen in training), and it captures morphological similarity for free (related word forms like "run"/"running"/"runner" share n-grams and end up with similar embeddings, without needing lemmatization as a preprocessing step). This character-piece idea is a direct conceptual ancestor of the subword tokenization (BPE/WordPiece/SentencePiece) every modern LLM uses.

Why Contextual Embeddings Superseded Static Ones

Every method above produces exactly one fixed vector per word, regardless of context — "bank" gets the same embedding whether it means a riverbank or a financial institution, which is a real, structural limitation, not a training-data problem more data would fix. Contextual embeddings (ELMo, then BERT-style Transformer encoders — see Attention & Transformers — The BERT Lineage and the dedicated BERT page for the real input representation and pretraining objectives) fix this by computing a different representation for each word occurrence, conditioned on its actual surrounding sentence via self-attention — "bank" in "river bank" and "bank" in "savings bank" get genuinely different vectors, computed on the fly from context, rather than a static lookup. This is the single biggest reason static word embeddings (word2vec/GloVe/fastText) were largely superseded for state-of-the-art NLP — not because the distributional hypothesis stopped being true, but because a fixed-per-word vector was always a structural compromise contextual models no longer need to make.

Where static embeddings are still used: extremely resource-constrained settings (a static embedding lookup is far cheaper than a Transformer forward pass), as a fast baseline, and as an interpretable/lightweight feature in classical ML pipelines (see Machine Learning) where a full contextual model is unnecessary overhead for the task's actual difficulty.

BERT gives one contextual vector per token — the next real question is how that becomes the single vector per sentence that dense retrieval and bi-encoders actually need.

From Contextual Tokens to a Single Sentence Vector: Sentence-BERT

What is it? Sentence-BERT (SBERT) (Reimers & Gurevych, 2019) — the fine-tuning recipe that turns a pretrained BERT-family encoder's per-token output into a single sentence vector that's actually good under cosine similarity. This is, concretely, the answer to "what embedding model is a bi-encoder actually using": every sentence-transformers model referenced on this site (all-MiniLM-L6-v2 and similar, used in the retrieval & reranking and rag pages' real code) is an SBERT-style bi-encoder.

How does it work? Take a pretrained encoder like BERT, add a pooling step over its per-token outputs — commonly mean pooling (averaging every token's vector into one) — to collapse the sequence down to a single fixed-size vector. Then fine-tune that whole setup with a siamese/triplet network structure: feed sentence pairs (or triplets) through the same encoder+pooling, and train on labeled sentence-pair data (natural language inference, semantic textual similarity, and large-scale question-answer/search-query pairs) so that genuinely similar sentences land close together under cosine similarity and dissimilar ones land far apart.

Why is it useful? A pretrained BERT was never trained to make [CLS] (or an average of its token vectors) behave well under cosine similarity — that objective (masked language modeling) has nothing to do with sentence-level similarity, so raw BERT embeddings make poor sentence vectors (see the BERT page's GoDeeper on exactly this pitfall). SBERT's fine-tuning step fixes that specifically, and the payoff is dramatic: the original paper reports finding the most similar pair among 10,000 sentences drops from ~65 hours with raw BERT (requiring a full cross-encoder-style forward pass per pair) to ~5 seconds with SBERT (embed once, compare with cosine similarity) — while matching BERT's accuracy on the underlying task. That's the entire reason bi-encoder retrieval is fast enough to search millions of documents in the first place.

Limitation: SBERT-style embeddings are only as good as their fine-tuning data — a model fine-tuned on general NLI/STS data can underperform on a narrow domain (legal, medical) with its own vocabulary and notion of similarity, the same domain-fit tradeoff RAG's embedding-model section already flags.

Next: NLP Task Taxonomy — the tasks these representations (static, contextual, or sentence-level) actually get used for.