Classical NLP: Tokenization, POS Tagging, NER & Parsing

Before "tokenization" meant BPE splitting text for a Transformer (see Foundation Model Internals), it meant something more basic: turning raw text into linguistically meaningful units at all. This page is that older, still-relevant layer.

Intuition: Structure Before Meaning

Every technique on this page answers a question that has to be settled before a model can reason about meaning at all: where does one unit end and the next begin (tokenization), what grammatical role does each unit play (POS tagging), which spans refer to real-world entities (NER), and how do the units relate to each other (parsing). None of these require understanding what a sentence means — only its structure — which is exactly why they predate (and still coexist alongside) modern semantic, embedding-based NLP.

Text Normalization

Tokenization (classical sense): splitting text into words/sentences — genuinely harder than it sounds. Watch a real rule-based tokenizer handle exactly the cases that break naive whitespace splitting:

This is a different, coarser-grained operation than an LLM's subword tokenizer, which operates on the output of (or in place of) this step.

Stemming (crude, rule-based suffix stripping) and lemmatization (dictionary-backed, linguistically correct base-form lookup) both aim at the same goal — reducing a word to a canonical form — through genuinely different mechanisms. Real rules applied to real words, side by side:

Why this still matters: search and information retrieval systems (matching "running shoes" against a document containing "run shoe") still commonly use stemming/lemmatization for query expansion and indexing, even in systems that also use modern embeddings — see RAG — Dense vs. Sparse Retrieval's BM25/sparse retrieval, which benefits directly from this normalization.

Part-of-Speech (POS) Tagging

Assigning a grammatical category (noun, verb, adjective, ...) to every word in a sentence. Classical approaches — Hidden Markov Models and Conditional Random Fields — frame this as sequence labeling: predict the most likely tag sequence given the word sequence, modeling the dependency between adjacent tags as well as each word's own likely tags. Step through a real Viterbi decode over a tiny HMM — the exact dynamic-programming algorithm, not a diagram of it:

Modern approaches use a small Transformer or BiLSTM (see Sequence Models — Bidirectional RNN/LSTM) trained end-to-end on labeled data, but the underlying task framing — sequence labeling — is unchanged and reused directly for NER below.

Why it still matters: POS tags are a useful feature for downstream tasks (parsing below depends on them), and understanding sequence labeling here is the direct conceptual bridge to NER and to general sequence-labeling problems in NLP Task Taxonomy.

Named Entity Recognition (NER)

Identifying and classifying spans of text into predefined categories — person names, organizations, locations, dates, monetary amounts. Framed identically to POS tagging as a sequence-labeling problem, using the standard BIO tagging scheme. Real decoding from per-token tags into grouped entity spans:

Why it still matters in an LLM-heavy stack: NER remains a common production pipeline stage even alongside LLMs — extracting structured entities (customer names, order IDs, dates) from unstructured text for downstream systems is often cheaper, faster, and more reliably schema-conformant with a dedicated NER model than with an LLM prompted to "extract the entities," especially at high request volume where a small fine-tuned NER model's latency and cost advantage over an LLM call matters.

Syntactic Parsing

Recovering a sentence's grammatical structure — not just tagging individual words, but the relationships between them. Toggle between the two dominant representations of the same sentence:

• Constituency parsing: breaks a sentence into nested phrases (noun phrases, verb phrases) forming a tree, following a formal grammar — "the structure a sentence diagram in grade school represents," made rigorous.
• Dependency parsing: instead of nested phrases, represents each word as depending on exactly one "head" word (a verb's subject and object both depend on the verb) — a directed graph (a tree, rooted at the sentence's main verb) rather than a nested-phrase tree. Dependency parsing has become the more commonly used representation in modern NLP pipelines because it more directly exposes "who did what to whom" relationships useful for downstream tasks like information extraction.
• Why this still matters: even though end-to-end neural models (and LLMs) rarely require an explicit parse tree as an input feature anymore, parsing-derived relationships remain useful for structured information extraction, grammar/style checking tools, and as an interpretability lens — asking "does this model's attention pattern correspond to real syntactic dependencies" is a genuine, still-active interpretability research question, directly reusing this classical structure as a ground truth to compare against.

Code: A Real Viterbi Decoder

The exact algorithm the POS-tagging diagram above steps through, in full:

import numpy as np

def viterbi(words, tags, transition, emission, start_probs):
    n, T = len(words), len(tags)
    prob = np.zeros((n, T))
    backpointer = np.zeros((n, T), dtype=int)

    for t, tag in enumerate(tags):
        prob[0, t] = start_probs[tag] * emission[tag].get(words[0], emission[tag]["<UNK>"])

    for i in range(1, n):
        for t, tag in enumerate(tags):
            scores = [prob[i-1, tp] * transition[tags[tp]][tag] for tp in range(T)]
            backpointer[i, t] = np.argmax(scores)
            prob[i, t] = max(scores) * emission[tag].get(words[i], emission[tag]["<UNK>"])

    # Backtrace from the best final state -- exactly what the diagram's
    # bold path traces, column by column, in reverse.
    path = [int(np.argmax(prob[-1]))]
    for i in range(n - 1, 0, -1):
        path.insert(0, backpointer[i, path[0]])
    return [tags[i] for i in path]

Next: Word Embeddings — moving from discrete linguistic categories to continuous, learned representations of meaning.