Reinforcement Learning from Human Feedback (RLHF) is the training technique that turned large language models from interesting research artifacts into commercial products. A pretrained LLM, before any RLHF, is a competent next-token predictor that will happily continue an offensive joke, generate confidently wrong information, or refuse none of the requests a sensible product would refuse. RLHF — and its 2026 successors like Direct Preference Optimization (DPO), Constitutional AI, and reward-model distillation — is what aligns those raw capabilities with human values, instruction-following, and the specific behaviors that make ChatGPT, Claude, and Gemini usable.
Without RLHF (or a comparable preference-tuning step), modern AI products do not exist. Understanding the technique is therefore one of the highest-leverage investments any AI builder, evaluator, or policymaker can make.
The training pipeline RLHF sits in
Frontier transformer-based language models go through three training stages. First, pretraining: the model learns from a vast text corpus by predicting the next token. The output is a base model that knows a lot about language and the world but does not specifically follow instructions or behave the way an assistant should. Second, supervised fine-tuning (SFT): the model is fine-tuned on instruction-response pairs written by trained labelers, teaching it to produce helpful responses to instructions rather than continuing arbitrary text. Third, preference tuning: the model is further refined based on human (or AI) judgments about which of two model outputs is better. RLHF is the canonical preference-tuning method; DPO and its descendants are simpler alternatives that achieve similar results without explicit reinforcement learning.
The order matters. SFT teaches the model the basic shape of being an assistant; preference tuning sharpens that shape into the specific behaviors users expect. Skip preference tuning and your model is helpful but rough. Skip SFT and preference tuning has nothing to refine. Skip pretraining and there’s no general-purpose capability to align in the first place.
How RLHF actually works
The classic RLHF pipeline as described in the InstructGPT paper has three steps:
Step 1: Collect human preference data. For a stream of prompts, generate two or more responses from the SFT model. Show the responses to a human labeler and ask which is better. Repeat thousands of times. The output is a dataset of (prompt, response_A, response_B, preferred) triples that encodes human judgments about response quality.
Step 2: Train a reward model. Use the preference dataset to train a separate neural network — usually initialized from the SFT model — that outputs a scalar “reward” given a prompt and a response. The reward model learns to predict which response a human labeler would prefer. This is the load-bearing piece: a good reward model captures human preference; a bad reward model produces a misaligned final policy.
Step 3: Optimize the policy with reinforcement learning. The SFT model becomes the policy. For each training prompt, the policy generates a response, the reward model scores it, and a reinforcement learning algorithm — historically Proximal Policy Optimization (PPO) — updates the policy weights to produce higher-reward responses on future prompts. A KL-divergence penalty against the original SFT model prevents the policy from drifting too far from sensible language behavior just to maximize reward.
The output of this loop is a model whose outputs reliably match what human labelers preferred — helpful, harmless, honest, and aligned with whatever specific behaviors the labeling guidelines specified.
Why RLHF is hard to do well
RLHF is conceptually simple and operationally fragile. Several failure modes are notorious.
Reward hacking. The policy finds outputs that score high on the reward model but are bad by any sensible human criterion — verbose, sycophantic, hedged, refusing reasonable requests. The reward model is a proxy for human preference; the policy optimizes the proxy, not the underlying goal.
Mode collapse. The policy converges on a narrow distribution of responses that hit the reward model’s sweet spot. Outputs become formulaic, lose creativity, and feel “AI-flavored” in the specific way ChatGPT outputs did in 2023.
Annotator disagreement. Human labelers disagree about which response is better, and their disagreements often correlate with demographic, cultural, or training-instruction differences. A reward model trained on disagreement data captures the average preference of the labeling pool, which may not represent any actual user.
Distribution shift. Reward models trained on the SFT model’s outputs become unreliable on the trained policy’s later outputs — the policy generates things the reward model wasn’t trained on. Iterative re-collection of preference data on the current policy’s outputs is one mitigation; constitutional methods that rely less on human labels are another.
Compute cost. PPO-based RLHF requires running the policy, the reward model, and a reference model in parallel during training, with rollouts and gradient updates that are 5-10x more expensive per token than supervised fine-tuning. Frontier-scale RLHF runs are major engineering and infrastructure undertakings.
The 2026 alternatives: DPO, IPO, KTO, and Constitutional AI
The complexity of full RLHF prompted a wave of simpler alternatives. Direct Preference Optimization (DPO), introduced in 2023, reformulates the preference-tuning objective so it can be optimized directly on the preference dataset without ever training a separate reward model or running PPO. The math is more elegant; the operational profile is dramatically simpler. Many open-weights models in 2026 — and quietly some closed-frontier models — use DPO or its descendants in place of full RLHF.
Identity Preference Optimization (IPO) and Kahneman-Tversky Optimization (KTO) further refine the DPO family with different theoretical assumptions about how human preferences map to optimization targets. Each has trade-offs; the field has not converged on a single winner.
Constitutional AI (CAI), originated by Anthropic, replaces or supplements human preferences with a set of written principles (the “constitution”) that an AI evaluator uses to critique and revise model outputs. This converts preference labeling from a human-bandwidth-limited process to an AI-bandwidth-limited process, and produces models whose alignment properties are tied to an explicit, inspectable set of principles rather than the inscrutable averaged preferences of a labeling pool.
Most frontier models in 2026 use some hybrid: a mixture of human preference data, AI-generated preference data, constitutional methods, and DPO-style optimization. The exact recipes are closely held by frontier labs because the recipe — not just the parameter count — is the moat.
RLHF beyond chat
Preference tuning is no longer just for chatbots. Coding-tuned models are RLHF’d on developer preference data — often “did the code pass tests” or “did the code match the intended fix” — to make them useful as AI coding agents. Image models are preference-tuned on aesthetic ratings to produce more visually appealing outputs. Multi-agent systems are increasingly trained with reinforcement learning over agent trajectories, where the reward is task completion rather than per-response human preference.
The general pattern — capture preferences as data, train a reward signal, optimize a policy against that signal, watch carefully for reward hacking — generalizes well beyond language. Nearly every commercial AI deployment now has some preference-tuning component in its training pipeline.
What RLHF means for users and organizations
For users, RLHF is invisible: it’s the reason the assistant you talk to doesn’t refuse normal requests, doesn’t generate offensive output, and follows your instructions reasonably well. For organizations deploying AI, RLHF has practical implications. The personality, refusal patterns, hedging tendencies, and tone of any frontier model are downstream of its preference-tuning process — which means they vary noticeably between providers and shift with each new model version. Prompt engineering partly works around these defaults; for serious deployments, fine-tuning or the use of system prompts that explicitly counteract over-cautious tuning is sometimes necessary.
For policymakers and AI safety practitioners, RLHF is the most concrete intervention point for behavioral alignment in deployed models. The constitution that a Constitutional AI system uses, the labeling guidelines a preference-data pipeline follows, and the reward model architecture used in the optimization loop are all places where values get encoded into AI behavior. Auditing those decisions matters more than auditing the model architecture.
Where to learn more
The InstructGPT paper from OpenAI remains the cleanest tutorial introduction to classic RLHF. Anthropic’s “Constitutional AI” paper is the canonical reference for the CAI approach. The DPO paper from Rafailov et al. introduces the simpler optimization formulation. For broader context on how RLHF fits into AI training, the large language model primer covers the full pipeline; the fine-tuning guide covers the lighter-weight customization techniques most teams will use directly.
RLHF and its descendants are the layer where AI systems are shaped to behave the way humans want. Understanding that layer is understanding why your assistant acts the way it does — and how to build assistants that act differently when you need them to.