Training Pipeline

Going from a randomly-initialized Transformer to something like ChatGPT is a multi-stage pipeline, not a single training run.

Pretraining

The model is trained on massive amounts of text with a simple self-supervised objective: predict the next token, given everything before it. Loss is cross-entropy between the predicted token distribution and the actual next token (see Probability & Statistics). This single objective, at sufficient scale, produces a model with broad world knowledge, grammar, reasoning patterns, and style — but one that isn't yet reliably helpful or instructable; it just continues text plausibly.

The Data Pipeline

Pretraining data is not "download the internet and go" — it's a multi-stage pipeline, and the quality of each stage directly shows up in the final model:

Crawl → Clean → Deduplicate → Filter → Tokenize → Pack → Pretraining

• Crawl: raw sources — Common Crawl (a public web crawl), curated sources (Wikipedia, books, code repositories), and increasingly licensed/proprietary data.
• Clean: strip HTML/boilerplate, fix encoding issues, remove obviously broken or non-natural-language text — raw web crawl is dominated by navigation menus, ads, and garbage that provides no useful training signal.
• Deduplicate: remove exact and near-duplicate documents — web crawl data is heavily duplicated (the same article mirrored/scraped repeatedly), and training on duplicates wastes compute and can cause the model to memorize specific duplicated passages verbatim rather than generalizing, a real, measured effect in the literature.
• Filter: remove low-quality, toxic, or otherwise undesirable content — via heuristic rules (document length, symbol-to-word ratio) and learned quality classifiers (a smaller model trained to distinguish "high-quality" text, e.g. Wikipedia-like, from low-quality web text).
• Contamination checking: specifically verify that benchmark evaluation sets (the ones the model will later be scored on) haven't leaked into the training data — a real, easy-to-introduce-by-accident problem, since web-scale crawls can and do contain copies of published benchmark questions and answers, and training on them invalidates the resulting evaluation scores the same way data leakage invalidates a classical ML evaluation.
• Tokenize and pack: convert cleaned text into token IDs (see Foundation Model Internals — Tokenization), then pack many documents back-to-back into fixed-length training sequences (with a separator token between documents) so every training batch is fully utilized — leaving sequences padded to a fixed length instead would waste significant compute on padding tokens that carry no signal.

Data Mixtures and Curriculum

Not all data sources are weighted equally — pretraining corpora are deliberately mixed from multiple sources (web text, code, books, academic papers, multilingual text) in specific proportions, tuned because the mixture measurably affects downstream capability (e.g. including code data measurably improves a model's general reasoning ability, not just its coding ability). Some pretraining runs also use a data curriculum — deliberately changing the mixture over the course of training (e.g. upweighting higher-quality data in later stages) rather than sampling from a single fixed distribution throughout.

Scaling Laws and the Chinchilla Result

Scaling laws are empirical relationships describing how model loss improves as you scale up model size, dataset size, and compute — power-law relationships fit from many training runs at different scales, precise enough to extrapolate and plan a training run's expected final loss before running it at full scale.

The Chinchilla paper's specific, influential finding: for a fixed compute budget, earlier practice (exemplified by GPT-3-era models) trained models that were too large relative to the amount of data they saw — compute-optimal training instead scales model size and training tokens together, roughly in a fixed ratio (Chinchilla's finding was approximately 20 training tokens per parameter). A smaller model trained compute-optimally on more data reliably beats a larger model undertrained on less data, for the same total compute cost — a result that directly reshaped how frontier labs allocate compute between "bigger model" and "more data," and part of why data pipeline quality (above) matters as much as architecture.

Compute, Batching, and Precision

• Compute budget: total training compute is often approximated as $C \approx 6ND$ (a standard rule of thumb) where $N$ is parameter count and $D$ is training tokens — the "6" absorbing the forward and backward pass FLOPs per token per parameter — the quantity scaling laws are fit against.

• Batch size and gradient accumulation, mixed precision, checkpointing: covered in full in GPU/AI Infrastructure & Distributed Training — pretraining is precisely the setting that infrastructure page's parallelism strategies (data/tensor/pipeline parallelism, FSDP, mixed precision) exists for, run continuously for weeks across thousands of GPUs.
• Pretraining evaluation: tracked primarily via the training loss curve itself (see the animated training-loss visual) and perplexity on a held-out set (literally $e^{\text{cross-entropy loss}}$ — how "surprised" the model is by held-out text, lower is better), plus periodic evaluation against standard benchmarks to track emerging capabilities over the course of the run, not just at the end.

Supervised Fine-Tuning (SFT)

The pretrained model is fine-tuned on a smaller, curated dataset of (instruction, ideal response) pairs, written or curated by humans. This teaches the model the format and behavior of being a helpful assistant — following instructions, answering directly instead of just continuing text, refusing certain requests.

RLHF — Reinforcement Learning from Human Feedback

SFT alone isn't enough to capture nuanced preferences ("this response is better, not just acceptable"). RLHF adds:

1. Collect human rankings of multiple model outputs for the same prompt.
2. Train a reward model to predict which output humans would prefer.
3. Fine-tune the LLM with reinforcement learning (typically PPO — Proximal Policy Optimization) to maximize the reward model's score, while a KL-divergence penalty (see Probability & Statistics) keeps the fine-tuned model from drifting too far from the original SFT model — preventing it from degenerating into responses that game the reward model without actually being good.

Reward hacking: when the model finds a way to score highly on the reward model without genuinely satisfying human intent (e.g. being unnecessarily verbose because the reward model correlates length with thoroughness). A core failure mode RLHF practitioners actively guard against.

Direct Preference Optimization (DPO)

A simpler alternative to RLHF: instead of training a separate reward model and running full RL, DPO reformulates the preference-learning objective so it can be optimized directly on preference pairs (chosen vs. rejected response) with a supervised-learning-style loss. Removes the complexity and instability of RL while achieving comparable results — widely adopted because it's simpler to implement and tune.

Group Relative Policy Optimization (GRPO)

A more recent RL approach (notably used in DeepSeek's models) that removes the need for a separate value/critic network by instead comparing a group of sampled outputs for the same prompt against each other's average reward — reducing training cost and complexity, particularly effective for reasoning-focused fine-tuning.

Other Preference-Optimization Objectives

DPO and GRPO are the two most widely deployed, but the same "learn directly from preference/reward signal without a full separate RL loop" family has several other named variants worth recognizing:

• IPO (Identity Preference Optimization): addresses a specific theoretical weakness in DPO's loss — DPO can overfit when preference pairs are close to deterministic (nearly always preferring one response), pushing the model toward extreme, overconfident probability ratios. IPO adds a regularization term that keeps the objective well-behaved even in that regime, without needing DPO's KL-penalty coefficient to be tuned as carefully.
• KTO (Kahneman-Tversky Optimization): named for the behavioral-economics loss-aversion research it's inspired by — unlike DPO, KTO doesn't need paired preference data (chosen vs. rejected for the same prompt) at all, just independent binary labels ("this output was good" / "this output was bad") — a real practical advantage, since unpaired binary feedback is often far cheaper to collect at scale than carefully constructed preference pairs.
• ORPO (Odds Ratio Preference Optimization): folds preference optimization directly into the SFT stage itself — a single combined loss (standard SFT cross-entropy plus an odds-ratio-based preference term) trains instruction-following and preference alignment simultaneously, removing the separate SFT-then-DPO two-stage pipeline entirely.
• RLOO (REINFORCE Leave-One-Out): a simplified policy-gradient RL approach (using the classic REINFORCE algorithm rather than PPO) that, like GRPO, avoids training a separate critic/value network — instead using other samples in the same batch as a variance-reducing baseline ("leave one out" of the batch to estimate the baseline for each sample) — computationally cheaper than PPO-based RLHF while still being a genuine RL method, not a supervised reformulation like DPO/IPO/KTO/ORPO.

The practical pattern across all of these: the field has been steadily moving away from full RLHF's separate-reward-model-plus-PPO complexity toward objectives that get similar alignment quality more cheaply and stably — DPO-family methods differ mainly in what data format they need (paired vs. unpaired) and what theoretical failure mode of plain DPO they patch, while GRPO/RLOO stay genuinely RL-based but drop PPO's critic network specifically.

Parameter-Efficient Fine-Tuning (PEFT)

Full fine-tuning updates every parameter — expensive and requires storing a full copy of the model per fine-tuned variant. PEFT methods update far fewer parameters:

• LoRA (Low-Rank Adaptation): freezes the original weights and injects small trainable low-rank matrices alongside them (see Linear Algebra — this works because weight updates during fine-tuning empirically have low effective rank). Dramatically cuts trainable parameters and memory.

• QLoRA: combines LoRA with quantizing the frozen base model to 4-bit precision, enabling fine-tuning of very large models on a single consumer GPU.
• DoRA (Weight-Decomposed Low-Rank Adaptation): decomposes each frozen weight matrix into a magnitude component and a direction component, applying LoRA's low-rank update only to the direction while letting magnitude adapt separately and freely — closes a measurable quality gap between LoRA and full fine-tuning by giving the model an extra, cheap degree of freedom (magnitude) that plain LoRA's low-rank constraint doesn't allow, at a small additional parameter cost over standard LoRA.
• Adapters: small trainable modules (typically a down-projection, a nonlinearity, and an up-projection) inserted between frozen Transformer layers, rather than alongside a weight matrix like LoRA — one of the original PEFT approaches, predating LoRA.
• IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations): instead of adding new weight matrices at all, learns a small set of per-channel rescaling vectors that multiply existing activations (in attention and the MLP) elementwise — dramatically fewer trainable parameters than even LoRA, at some capacity cost, and notably doesn't add any extra inference latency once merged since it's a pure elementwise rescale.
• Prefix tuning: prepends a small number of trainable virtual tokens' worth of key/value vectors to every attention layer (not actual text tokens, learned continuous vectors) — the frozen model attends to these learned prefixes alongside the real input, steering behavior without touching any of the model's own weights at all.
• Prompt tuning: a simplification of prefix tuning — instead of learned KV vectors injected at every layer, learns a small number of continuous "soft prompt" embeddings prepended only at the input layer. Fewer trainable parameters than prefix tuning, generally somewhat less expressive, but simpler to implement and still effective at larger model scales.
• Quantization-aware fine-tuning: distinct from QLoRA's "quantize the frozen base, train full-precision LoRA adapters on top" — this instead simulates quantization effects during the fine-tuning forward/backward pass itself (see LLM Inference Optimization — QAT), producing a model that's fine-tuned to already be robust to the precision loss it'll actually run at in production, rather than being fine-tuned in full precision and only quantized afterward.

PEFT frameworks like Unsloth make this training step itself faster and more memory-efficient — but training and serving are different jobs. Once fine-tuning produces adapter weights, they still need to be handed off to a real inference engine (vLLM, llama.cpp, SGLang) to actually serve traffic — see LLM Hosting & Serving Patterns for that handoff, and why Unsloth itself isn't what you deploy.

Knowledge Distillation

Train a smaller "student" model to mimic a larger "teacher" model's output distribution (not just the hard labels, but the full soft probability distribution, which carries more information about the teacher's "reasoning"). Produces smaller, faster models that retain much of the teacher's capability — used heavily to create deployable small models from expensive frontier models.

Quantization

Reduces the numerical precision used to store model weights — from 32-bit or 16-bit floats down to 8-bit or 4-bit integers (int8, int4; GPTQ and AWQ are popular quantization algorithms). Cuts memory footprint and can speed up inference substantially, at a small, often negligible, cost to output quality — essential for running large models on limited hardware.

Next: Prompt Engineering — getting the most out of a model without touching its weights at all.