Advanced RL

RL Fundamentals covered tabular methods — a $Q$-value stored explicitly for every state-action pair, which only works for small, discrete state spaces. Everything on this page addresses what happens once state spaces get too large for a table, action spaces become continuous, or the clean "agent interacts live with an environment" assumption itself breaks down.

Intuition: Each Method Fixes One Specific Assumption Break

Every topic below exists because one specific assumption from RL Fundamentals stopped holding, and each fix is narrowly scoped to that one break: SAC and TD3 fix specific statistical failure modes of actor-critic methods once a neural network stands in for the Q-table. Model-based RL and world models break the assumption that a policy can only be updated from a real environment transition. Offline RL breaks the "agent can freely explore" assumption. Imitation and inverse RL break the "a reward function is given" assumption. Keep that mapping in mind — it turns this page from a list of algorithm names into "here's the assumption, here's exactly what breaks, here's the fix."

Deep RL: DQN, Policy Gradients, Actor-Critic, PPO

Covered as architectures in Advanced Architectures — Reinforcement Learning Networks: DQN replaces the Q-table with a neural network (necessary once the state space is too large to enumerate — an image-based state space, for instance, has no finite table to fill in), policy gradient methods directly parameterize and optimize the policy for continuous action spaces where "take the argmax" isn't well-defined, actor-critic combines both for better sample efficiency, and PPO constrains policy updates to a trust region for training stability. Revisit that page for the architectural detail — this page assumes it and covers what sits beyond it.

SAC and TD3: Refining Continuous Control

Two widely-used refinements specifically for continuous action spaces (robotics, physical control), addressing specific weaknesses in vanilla actor-critic/policy-gradient methods.

TD3: Fixing Overestimation Bias

Overestimation bias is a specific, well-documented statistical failure: once a neural network approximates $Q$, its estimates carry noise — and taking a $\max$ over noisy estimates is a biased estimator of the true max, even when the noise itself averages to zero, because the max operation systematically favors whichever estimate got lucky. Run the real simulation below — 400 trials, one Monte Carlo run each:

TD3 (Twin Delayed DDPG) fixes this with three specific tricks: training two independent value networks and using the more conservative (minimum) of the two as the target (directly counteracting the upward-bias direction, exactly as shown above), delaying policy updates relative to value updates (letting the value estimate stabilize first), and adding noise to the target action (smoothing the value estimate against small errors in the exact action taken).

SAC: Trading Off Return Against Exploration

SAC (Soft Actor-Critic) adds an entropy bonus to the objective — the agent is rewarded not just for expected return, but for maintaining some randomness in its policy: $J = \mathbb{E}[Q] + \alpha H(\pi)$. This explicitly trades off exploration against exploitation within the objective itself, rather than requiring a separately-tuned exploration schedule. Drag $\alpha$ and watch which of two real policies — one peaked and greedy, one spread and exploratory — the objective actually prefers:

This tends to produce more robust policies that don't collapse prematurely onto a single, possibly-suboptimal deterministic behavior — exactly the failure mode a purely greedy objective (α=0) is prone to.

Model-Based RL and World Models

Every method above this point is model-free: it never actually knows how the environment works, only ever updating a policy or value function from real transitions the agent actually experienced. Model-based RL breaks that assumption directly — learn a model of the environment's own dynamics ($\hat{s}_{t+1} = f(s_t, a_t)$, usually paired with a learned reward model too), then train or plan using that learned model instead of, or alongside, real interaction.

A world model is the specific, now-standard instance of this idea for deep RL: a learned, compressed simulator of the environment the policy can train inside, needing far fewer real environment steps than a model-free method. Ha and Schmidhuber's original "World Models" paper (2018) made this concrete: a variational autoencoder compresses raw observations into a small latent space, a recurrent dynamics model predicts how that latent state evolves under an action, and the policy trains entirely inside the resulting learned simulator — real environment data refreshes the model only occasionally, it doesn't train the policy directly. In one of the paper's results, a policy trained entirely inside its own hallucinated version of a VizDoom level, never seeing a single real frame during training, still scored well above the game's own "solved" threshold once dropped into the real environment.

DreamerV3 (Hafner et al.) is the widely-used current successor: a single configuration that works across 150+ diverse tasks with no per-task tuning, and performance that keeps improving as the model itself is scaled up — the same scaling behavior this site's LLM pages describe, here applied to a learned environment model instead of a learned language model. DeepMind's Genie line takes the idea a step further: instead of using the world model as scaffolding to train a policy, the world model itself is the product — an interactive, generative simulator a human or another agent can act inside directly.

The real tradeoff, not a free upgrade: a model-based agent is only as good as its learned model. In any state the model never saw real data for, its predictions are wrong in exactly the "confidently wrong on unseen states" way Offline RL's extrapolation problem and imitation learning's compounding error already describe elsewhere on this page — model-based RL doesn't remove that failure mode, it just moves it from a value function or a cloned policy to a learned environment model instead.

Offline RL

Standard RL assumes the agent can freely interact with the environment, collecting new experience as it learns. Offline RL removes that assumption entirely — the agent learns from a fixed, pre-collected dataset of past experience, with no ability to try new actions and see what happens. This matters enormously wherever live experimentation is expensive, slow, or unsafe — a physical robot, a medical treatment policy, a recommender system where "trying a bad action" means showing a real user a bad recommendation.

The core added difficulty beyond standard RL: without the ability to explore, the algorithm has no way to correct a value-function error for actions poorly represented in the fixed dataset. Watch a real regression fit exploit exactly that gap:

Naively extrapolating can produce a policy that looks good on paper (high estimated value) but fails badly in practice specifically because it exploits the value function's errors on data it never saw — a distribution-shift problem closely related to training-serving skew in the classical ML sense, just for RL's action distribution instead of a feature distribution.

Imitation Learning

Instead of learning from a reward signal at all, learn a policy directly from demonstrations of the desired behavior — a human (or existing expert policy) performing the task, with the agent trained (often via straightforward supervised learning) to reproduce the demonstrated actions given the observed states. The most direct form, behavioral cloning, is simple but has a well-known compounding-error problem: small mistakes push the agent into states slightly outside the demonstrated distribution, where it's never been trained and makes further mistakes. Same-sized per-step errors, fed two different ways:

Behavioral cloning's error accumulates because the agent's own (already-drifted) state feeds into its next decision — a real feedback loop, not a metaphor, and exactly what the red trace above is doing: a genuine running sum of per-step error. Techniques that actively query the expert during training (rather than training purely offline on a fixed demonstration set) break that feedback loop by periodically resetting to the true state, exactly like the bounded green trace.

Inverse RL

The reverse problem from imitation learning's direct behavior-cloning: instead of directly copying demonstrated actions, infer the reward function that would explain why an expert behaved the way they did, then solve the resulting RL problem to derive a policy. This is more information-rich than plain imitation learning when it works — a recovered reward function can generalize to states/situations the original demonstrations never covered, where a directly-cloned policy has no principled way to generalize — at the cost of being a fundamentally harder, underdetermined problem (many different reward functions can rationalize the same observed behavior equally well).

RLHF, RLAIF, and GRPO in the RL Framework

Training Pipeline — RLHF and GRPO are genuine instances of this page's framework. Click through the mapping below to see exactly how each piece of RL vocabulary lands on the LLM side:

Seeing RLHF as "just RL, with a learned reward model standing in for the environment's reward function" is what makes the KL-divergence penalty, the reward-hacking failure mode (see Alignment & RLHF), and GRPO's critic-free design all make sense as specific engineering responses to specific, generic RL problems, not LLM-specific magic.

Code: TD3's Twin-Critic Target, For Real

import torch

# TD3's core fix -- take the MIN of two critics as the target, exactly
# what closed the overestimation gap in the diagram above.
def td3_target(critic1, critic2, target_actor, reward, next_state, gamma, noise_std=0.2):
    next_action = target_actor(next_state)
    next_action += torch.clamp(torch.randn_like(next_action) * noise_std, -0.5, 0.5)  # target policy smoothing
    q1 = critic1(next_state, next_action)
    q2 = critic2(next_state, next_action)
    target_q = reward + gamma * torch.min(q1, q2)  # the twin-critic min, not a single critic's estimate
    return target_q

Next: RL Environment & Reward Engineering — the algorithms above only matter as much as the environment and reward function they're trained against, which is where most of the real engineering work in a modern post-training pipeline actually lives.