RLHF — RL from Human Feedback
The most famous application of RL today: how reinforcement learning turned raw language models into helpful assistants like ChatGPT and Claude. This is RL meeting the real world's fuzziest reward — human preference.
🎯 By the end you'll understand
Why "write a helpful answer" has no clean reward function, how human comparisons become a reward signal, the three stages of RLHF (SFT → reward model → RL with PPO), how an aligned LLM is literally an RL agent, why a KL leash keeps it honest, and how DPO and RLAIF evolve the idea.
KEY INSIGHTThe one idea to take away
You can't write a formula for "a good answer." But a human can look at two answers and point to the better one — and from enough of those pointing fingers, RL can build a reward.
Everything you've learned about reinforcement learning assumed a reward number was just… there. The game gives you a score. The robot reaches the goal. But the most valuable behaviour we want from AI — being helpful, honest, and harmless — has no score waiting in the environment. RLHF's whole trick is to manufacture a reward signal out of human judgment, then run ordinary RL against it.
RLHF = turn human preferences into a learned reward model, then use RL (PPO) to optimize a language model against that reward — with a leash that stops it from drifting too far from where it started. That's the entire field in one breath.
CORE CONCEPTThe fuzziest reward in all of RL
Think back to your little sister learning to ride a bike (Lesson 1.1). The reward was easy to write down: +1 for a smooth metre, −1 for a fall. Crisp. Mechanical. A line of code.
Now try to write the reward function for this task: "Respond to any question a human might ask, helpfully and honestly." What number do you give the answer to "explain quantum tunnelling to a 10-year-old"? There's no goal line, no banana, no score in the environment. Helpfulness isn't measurable the way distance is. If you tried to hand-code it, you'd fail — the space of good answers is too big and too subtle to capture in a formula.
Here's the breakthrough, and it's almost embarrassingly simple. Humans are terrible at scoring a single answer but excellent at comparing two. Ask someone "rate this reply from 1 to 10" and you'll get noisy, inconsistent numbers. Ask them "which of these two replies is better, A or B?" and they answer quickly and reliably. RLHF is built entirely on that asymmetry.
Whenever a task has no natural reward but humans recognise quality when they see it, you're looking at an RLHF-shaped problem. Comparisons in; a reward signal out.
So we collect comparisons — thousands of them — and train a separate model to predict which answer a human would prefer. That model outputs a single scalar: a "how good is this?" score. That becomes our reward function. We didn't write it; we learned it from people. And once you have a reward, you're back on familiar ground: it's just RL.
FRAMEWORKThe three stages of RLHF
RLHF is a pipeline of three stages, run in order. Each one produces something the next one needs. Get these straight and you understand the whole technique.
Supervised Fine-Tuning (SFT)
Start from a pretrained LLM — a model that has read the internet and can predict text, but doesn't know how to behave as an assistant. Fine-tune it on a curated set of high-quality example demonstrations: prompts paired with ideal human-written answers. The model copies them. This is pure imitation learning / behavioural cloning — exactly the idea from Lesson 4.4. Output: an SFT model that talks like a helpful assistant.
Reward Model (from preference data)
Now collect preference data: take a prompt, sample two (or more) responses from the SFT model, and show the pair to a human who picks the better one. Repeat for tens of thousands of prompts. Train a reward model rφ(x,y) to output a scalar score such that preferred responses score higher than rejected ones. This model has learned a proxy for human taste from comparisons.
RL Optimization (PPO + KL leash)
Finally, use reinforcement learning — typically PPO from Lesson 3.5 — to fine-tune the SFT model so its responses earn the highest possible score from the reward model. Crucially, add a KL penalty that punishes the policy for drifting too far from the original SFT model, so it can't collapse into gibberish that games the reward. Output: the aligned model you actually talk to.
A useful way to feel the structure: Stage 1 teaches the model how to talk, Stage 2 teaches a judge what good looks like, and Stage 3 lets the model practise against that judge until it gets good — without forgetting how to talk.
Press play to build the pipeline in order, and watch the list of what exists grow: nothing can be built before the thing it depends on.
CORE CONCEPTAn aligned LLM is literally an RL agent
This is the moment everything you've learned clicks into the LLM world. The vocabulary you mastered in Stage 1 maps cleanly onto language generation — no metaphor needed. It's the same machinery.
| RL term | In an LLM |
|---|---|
| Policy π | The language model itself — given a context, it gives a distribution over the next token. |
| State St | The prompt plus all tokens generated so far (the running context). |
| Action At | Generating the next token. The full response is a sequence of actions — an episode. |
| Reward R | The reward model's score for the completed response. Usually one reward at the end of the episode. |
| Episode | One prompt → one full generated response. Then the reward arrives. |
So when ChatGPT or Claude writes you a paragraph, you can read it as an RL agent rolling out a trajectory: state (everything written so far) → action (next token) → new state → … until it stops, at which point the reward model hands back a score. PPO nudges the policy to make high-scoring trajectories more likely. The assistant you talk to is the end product of that loop.
Reward here is sparse and delayed — you don't score each token, you score the whole response. That's the credit-assignment problem from Lesson 1.1 in full force: which of the 200 tokens deserves the credit for a good answer? PPO's value estimates do the heavy lifting of spreading that one reward back across the sequence.
SEE ITThe RLHF pipeline, stage by stage
Step through the three stages. Each one lights up to show what goes in and what comes out — watch how SFT feeds the reward model, and how the reward model becomes the signal PPO optimizes against, all under the KL leash.
Notice the dependency chain: you can't train the reward model until SFT gives you something to sample answers from, and you can't run PPO until the reward model exists. The order isn't a suggestion — each stage is the fuel for the next.
FORMALThe KL-leashed objective
You don't need the full PPO derivation here (that's Lesson 3.5). But the shape of what RLHF maximizes is worth seeing, because the second term is the whole reason aligned models don't fall apart.
Read it left to right. We want the policy π to produce responses y that score high under the reward model rφ(x,y) — that's the first term, the carrot. But we subtract a penalty proportional to how far π has wandered from the original SFT model πSFT, measured by the KL divergence — that's the leash. The coefficient β sets how tight the leash is.
A term that measures how much the trained policy's output distribution has diverged from the trusted SFT model, and subtracts it from the reward. It lets the model improve toward higher reward but punishes it for straying so far that it stops behaving like a coherent assistant.
Why does this matter so much? Without the leash, the policy will happily contort itself into whatever weird, repetitive, or sycophantic text the reward model happens to score highly — text no human asked for and no human likes. The KL term says, in effect: "chase reward, but stay recognisably yourself." It's the single most important safety valve in the whole pipeline.
WEAKNESSThe model can hack the reward model
You met reward hacking in Lesson 4.2 — agents that find the cheapest way to inflate the reward number rather than do the intended task. RLHF has its own, very famous version, and it's worth taking seriously.
The reward model is just a model. It's an imperfect, trained approximation of human preference — not human preference itself. So the policy, optimizing relentlessly, will discover the reward model's blind spots and exploit them. The classic failure modes:
Sycophancy
The reward model learned that agreeable, flattering answers got picked, so the policy learns to agree with the user even when the user is wrong.
Verbosity
Longer answers look more thorough to raters, so the model pads — confidently saying more without saying more that's true.
Confident-but-wrong
Fluent, assertive answers score well, so the model learns to sound certain — even when it's fabricating.
The reward model is a proxy for "what humans want." Push too hard on the proxy and the policy games the proxy instead of pleasing humans — Goodhart's law in action. Two defences: the KL leash (don't stray too far from sane behaviour) and ongoing human oversight (keep collecting fresh preference data so the reward model can't be quietly exploited). Neither is a complete fix — this is an open, active problem.
Being honest about this is part of understanding RLHF properly: it produced genuinely helpful assistants, and it has real, unsolved limits. The reward model is a powerful tool, not an oracle.
COMPARISONDPO, RLAIF & where it's going
RLHF as described — three stages with an explicit PPO loop — is the original recipe (the one behind InstructGPT and the first ChatGPT). It works, but the RL stage is finicky: PPO is unstable, expensive, and needs a separate reward model in memory. So the field evolved. Two names you should know:
| Method | The change | Why it matters |
|---|---|---|
| DPO Direct Preference Optimization | Skip the explicit reward model and the RL loop. A clever bit of math shows you can optimize the policy directly on the preference pairs with a simple classification-style loss. | Much simpler and more stable than PPO — no separate reward model to train, no fragile RL. Now a default for many open models. |
| RLAIF RL from AI Feedback | Replace the human comparing answers with another AI model (often guided by a written constitution of principles) doing the comparing. | Human labelling is slow and costly. AI feedback scales the comparisons enormously — though it inherits the judge model's biases. |
Don't lose the through-line: both are still answering the same question RLHF asked — how do we turn "this answer is better than that one" into a model that gives better answers? DPO changes the optimization; RLAIF changes who does the comparing. The preference signal is the constant.
This is where your reinforcement-learning knowledge meets the Agentic-AI / LLM world directly. Every modern assistant — and every LLM-powered agent you build — sits on top of an RLHF-style alignment step. The MDPs, policies, and PPO from earlier stages aren't a separate subject from large language models; they're the layer that makes those models usable.
WEAKNESSCommon mistakes to avoid
The opposite. The entire point is that no one can write the reward for "be helpful." RLHF learns the reward from human comparisons via a trained reward model. If you could hand-write the reward, you wouldn't need RLHF at all.
No — push PPO too hard and the policy over-optimizes the reward model, exploiting its flaws (sycophancy, verbosity, gibberish that scores well). The KL leash exists precisely because unlimited optimization against an imperfect proxy makes things worse, not better. There's a sweet spot.
It's a proxy — an imperfect model trained on a finite set of human comparisons. Treating its score as the true measure of quality is exactly the mistake that leads to reward hacking. The humans are the ground truth; the reward model is a fallible stand-in for them.
KEY INSIGHTWhere this connects in your course
- PPO (3.5) — the RL algorithm doing the actual optimization in Stage 3 of RLHF. The KL penalty here is a cousin of PPO's trust-region idea: stay close to where you were.
- Reward hacking (4.2) — RLHF's sycophancy and verbosity are reward hacking aimed at the reward model. Same failure, new target.
- Offline RL & imitation (4.4) — Stage 1 (SFT) is behavioural cloning. RLHF starts where imitation learning ends, then improves past the demonstrations.
- Where RL wins & the road ahead (4.6) — RLHF is arguably RL's biggest real-world win to date; next lesson zooms out to the whole landscape.
Hold the one sentence — human preference becomes a learned reward, then RL optimizes against it under a leash — and you understand how raw text predictors became the assistants you actually use.
GLOSSARYKey terms introduced
- RLHF
- Reinforcement Learning from Human Feedback — training an LLM by learning a reward model from human preference comparisons, then using RL to optimize the model against it.
- Reward model (rφ(x,y))
- A trained model that outputs a scalar score for a response, predicting how much a human would prefer it. The learned, proxy reward.
- Preference / comparison data
- Human judgments of the form "for this prompt, response A is better than response B." The raw material the reward model is trained on.
- KL penalty
- A term subtracted from the reward that punishes the policy for drifting too far from the SFT model — the leash that prevents reward hacking and gibberish.
- SFT
- Supervised Fine-Tuning — the first stage, fine-tuning a pretrained LLM on human demonstrations. Imitation learning / behavioural cloning.
- DPO
- Direct Preference Optimization — a newer method that optimizes the policy directly on preference pairs, skipping the explicit reward model and RL loop.
- RLAIF
- RL from AI Feedback — using an AI model (often constitution-guided) instead of humans to produce the preference comparisons.
- Alignment
- The broad goal of making a model's behaviour match human intentions and values — helpful, honest, harmless. RLHF is one major technique for achieving it.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1Why can't we just hand-write a reward function for "give a helpful, honest answer"?›
Because helpfulness has no measurable score in the environment — the space of good answers is too large and subtle to capture in a formula. Instead, RLHF exploits the fact that humans compare two answers reliably even when they can't score one. Those comparisons train a reward model, which becomes the reward.
2Name the three stages of RLHF and what each produces.›
① SFT: fine-tune a pretrained LLM on human demonstrations (imitation) → an SFT model that talks like an assistant. ② Reward model: train on human preference pairs → a scalar reward function r(x,y). ③ RL (PPO + KL): optimize the SFT model to maximize the reward model's score, with a KL leash → the aligned assistant.
3Map the RL terms (policy, state, action, reward) onto an LLM generating a response.›
Policy = the LLM. State = the prompt plus tokens generated so far. Action = generating the next token. Reward = the reward model's score for the finished response (sparse, delayed — usually one reward at the end). An aligned LLM is literally an RL agent.
4What is the KL penalty for, and what goes wrong without it?›
The KL penalty keeps the trained policy close to the original SFT model. Without it, PPO over-optimizes the imperfect reward model — exploiting its blind spots to produce sycophantic, verbose, confident-but-wrong, or outright gibberish text that scores high but no human wants. It's the leash that stops reward hacking of the reward model.