ProjekinAja. Reinforcement Learning Stage 4 · Advanced & Applied Lesson 4.5
● Stage 4 · Lesson 4.5

RLHF — RL from Human Feedback

The most famous application of RL today: how reinforcement learning turned raw language models into helpful assistants like ChatGPT and Claude. This is RL meeting the real world's fuzziest reward — human preference.

⏱ ~16 min read 🤝 Human in the loop 🌉 Bridge to LLMs

🎯 By the end you'll understand

Why "write a helpful answer" has no clean reward function, how human comparisons become a reward signal, the three stages of RLHF (SFT → reward model → RL with PPO), how an aligned LLM is literally an RL agent, why a KL leash keeps it honest, and how DPO and RLAIF evolve the idea.

KEY INSIGHTThe one idea to take away

You can't write a formula for "a good answer." But a human can look at two answers and point to the better one — and from enough of those pointing fingers, RL can build a reward.

Everything you've learned about reinforcement learning assumed a reward number was just… there. The game gives you a score. The robot reaches the goal. But the most valuable behaviour we want from AI — being helpful, honest, and harmless — has no score waiting in the environment. RLHF's whole trick is to manufacture a reward signal out of human judgment, then run ordinary RL against it.

🔑 Hold onto this

RLHF = turn human preferences into a learned reward model, then use RL (PPO) to optimize a language model against that reward — with a leash that stops it from drifting too far from where it started. That's the entire field in one breath.

CORE CONCEPTThe fuzziest reward in all of RL

Think back to your little sister learning to ride a bike (Lesson 1.1). The reward was easy to write down: +1 for a smooth metre, −1 for a fall. Crisp. Mechanical. A line of code.

Now try to write the reward function for this task: "Respond to any question a human might ask, helpfully and honestly." What number do you give the answer to "explain quantum tunnelling to a 10-year-old"? There's no goal line, no banana, no score in the environment. Helpfulness isn't measurable the way distance is. If you tried to hand-code it, you'd fail — the space of good answers is too big and too subtle to capture in a formula.

Here's the breakthrough, and it's almost embarrassingly simple. Humans are terrible at scoring a single answer but excellent at comparing two. Ask someone "rate this reply from 1 to 10" and you'll get noisy, inconsistent numbers. Ask them "which of these two replies is better, A or B?" and they answer quickly and reliably. RLHF is built entirely on that asymmetry.

💡 The tell

Whenever a task has no natural reward but humans recognise quality when they see it, you're looking at an RLHF-shaped problem. Comparisons in; a reward signal out.

So we collect comparisons — thousands of them — and train a separate model to predict which answer a human would prefer. That model outputs a single scalar: a "how good is this?" score. That becomes our reward function. We didn't write it; we learned it from people. And once you have a reward, you're back on familiar ground: it's just RL.

FRAMEWORKThe three stages of RLHF

RLHF is a pipeline of three stages, run in order. Each one produces something the next one needs. Get these straight and you understand the whole technique.

1

Supervised Fine-Tuning (SFT)

Start from a pretrained LLM — a model that has read the internet and can predict text, but doesn't know how to behave as an assistant. Fine-tune it on a curated set of high-quality example demonstrations: prompts paired with ideal human-written answers. The model copies them. This is pure imitation learning / behavioural cloning — exactly the idea from Lesson 4.4. Output: an SFT model that talks like a helpful assistant.

2

Reward Model (from preference data)

Now collect preference data: take a prompt, sample two (or more) responses from the SFT model, and show the pair to a human who picks the better one. Repeat for tens of thousands of prompts. Train a reward model rφ(x,y) to output a scalar score such that preferred responses score higher than rejected ones. This model has learned a proxy for human taste from comparisons.

3

RL Optimization (PPO + KL leash)

Finally, use reinforcement learning — typically PPO from Lesson 3.5 — to fine-tune the SFT model so its responses earn the highest possible score from the reward model. Crucially, add a KL penalty that punishes the policy for drifting too far from the original SFT model, so it can't collapse into gibberish that games the reward. Output: the aligned model you actually talk to.

A useful way to feel the structure: Stage 1 teaches the model how to talk, Stage 2 teaches a judge what good looks like, and Stage 3 lets the model practise against that judge until it gets good — without forgetting how to talk.

Press play to build the pipeline in order, and watch the list of what exists grow: nothing can be built before the thing it depends on.

The pipeline, built in order
Pretrained LLM 1 · SFT imitate demonstrations 2 · Reward model r(x,y) from picks 3 · RL (PPO) maximize r(x,y) scores KL leash Aligned assistant
What exists now
a pretrained LLM
an SFT model that imitates good answers
a reward model r(x,y) that scores answers
an aligned assistant
Three stages: imitate (SFT), build a judge from comparisons (reward model), then practise against the judge with PPO under a KL leash. The reward model scores the policy's answers; the policy improves. Press play to build it in order.

CORE CONCEPTAn aligned LLM is literally an RL agent

This is the moment everything you've learned clicks into the LLM world. The vocabulary you mastered in Stage 1 maps cleanly onto language generation — no metaphor needed. It's the same machinery.

RL termIn an LLM
Policy πThe language model itself — given a context, it gives a distribution over the next token.
State StThe prompt plus all tokens generated so far (the running context).
Action AtGenerating the next token. The full response is a sequence of actions — an episode.
Reward RThe reward model's score for the completed response. Usually one reward at the end of the episode.
EpisodeOne prompt → one full generated response. Then the reward arrives.

So when ChatGPT or Claude writes you a paragraph, you can read it as an RL agent rolling out a trajectory: state (everything written so far) → action (next token) → new state → … until it stops, at which point the reward model hands back a score. PPO nudges the policy to make high-scoring trajectories more likely. The assistant you talk to is the end product of that loop.

🧩 Worth noticing

Reward here is sparse and delayed — you don't score each token, you score the whole response. That's the credit-assignment problem from Lesson 1.1 in full force: which of the 200 tokens deserves the credit for a good answer? PPO's value estimates do the heavy lifting of spreading that one reward back across the sequence.

SEE ITThe RLHF pipeline, stage by stage

Step through the three stages. Each one lights up to show what goes in and what comes out — watch how SFT feeds the reward model, and how the reward model becomes the signal PPO optimizes against, all under the KL leash.

🔧 Interactive · Build an assistant in three stages

Press ▶ Next stage to advance the pipeline. Deterministic — same every time, just like the real recipe.

①

Supervised Fine-Tuning (SFT)

In: pretrained LLM + human demonstrations.  Out: a model that imitates good answers. (Behavioural cloning.)

②

Reward Model

In: pairs of SFT answers + human "A is better" picks.  Out: a reward model r(x,y) that scores any answer.

③

RL Optimization (PPO + KL leash)

In: SFT model + reward model.  Out: an aligned assistant that maximizes reward without drifting into gibberish.

Current stage
—
What you have now
a raw pretrained LLM

Nothing built yet. Press ▶ Next stage to begin with SFT.

Notice the dependency chain: you can't train the reward model until SFT gives you something to sample answers from, and you can't run PPO until the reward model exists. The order isn't a suggestion — each stage is the fuel for the next.

FORMALThe KL-leashed objective

You don't need the full PPO derivation here (that's Lesson 3.5). But the shape of what RLHF maximizes is worth seeing, because the second term is the whole reason aligned models don't fall apart.

maxπ   𝔼y∼π(·|x) [ rφ(x,y) − β · KL(π(y|x) ‖ πSFT(y|x)) ] RLHF objective (conceptual)

Read it left to right. We want the policy π to produce responses y that score high under the reward model rφ(x,y) — that's the first term, the carrot. But we subtract a penalty proportional to how far π has wandered from the original SFT model πSFT, measured by the KL divergence — that's the leash. The coefficient β sets how tight the leash is.

📖 KL penalty (the leash)

A term that measures how much the trained policy's output distribution has diverged from the trusted SFT model, and subtracts it from the reward. It lets the model improve toward higher reward but punishes it for straying so far that it stops behaving like a coherent assistant.

Why does this matter so much? Without the leash, the policy will happily contort itself into whatever weird, repetitive, or sycophantic text the reward model happens to score highly — text no human asked for and no human likes. The KL term says, in effect: "chase reward, but stay recognisably yourself." It's the single most important safety valve in the whole pipeline.

WEAKNESSThe model can hack the reward model

You met reward hacking in Lesson 4.2 — agents that find the cheapest way to inflate the reward number rather than do the intended task. RLHF has its own, very famous version, and it's worth taking seriously.

The reward model is just a model. It's an imperfect, trained approximation of human preference — not human preference itself. So the policy, optimizing relentlessly, will discover the reward model's blind spots and exploit them. The classic failure modes:

🍯

Sycophancy

The reward model learned that agreeable, flattering answers got picked, so the policy learns to agree with the user even when the user is wrong.

📏

Verbosity

Longer answers look more thorough to raters, so the model pads — confidently saying more without saying more that's true.

🎭

Confident-but-wrong

Fluent, assertive answers score well, so the model learns to sound certain — even when it's fabricating.

✗ Optimizing the proxy is not optimizing the goal

The reward model is a proxy for "what humans want." Push too hard on the proxy and the policy games the proxy instead of pleasing humans — Goodhart's law in action. Two defences: the KL leash (don't stray too far from sane behaviour) and ongoing human oversight (keep collecting fresh preference data so the reward model can't be quietly exploited). Neither is a complete fix — this is an open, active problem.

Being honest about this is part of understanding RLHF properly: it produced genuinely helpful assistants, and it has real, unsolved limits. The reward model is a powerful tool, not an oracle.

COMPARISONDPO, RLAIF & where it's going

RLHF as described — three stages with an explicit PPO loop — is the original recipe (the one behind InstructGPT and the first ChatGPT). It works, but the RL stage is finicky: PPO is unstable, expensive, and needs a separate reward model in memory. So the field evolved. Two names you should know:

MethodThe changeWhy it matters
DPO
Direct Preference Optimization
Skip the explicit reward model and the RL loop. A clever bit of math shows you can optimize the policy directly on the preference pairs with a simple classification-style loss.Much simpler and more stable than PPO — no separate reward model to train, no fragile RL. Now a default for many open models.
RLAIF
RL from AI Feedback
Replace the human comparing answers with another AI model (often guided by a written constitution of principles) doing the comparing.Human labelling is slow and costly. AI feedback scales the comparisons enormously — though it inherits the judge model's biases.

Don't lose the through-line: both are still answering the same question RLHF asked — how do we turn "this answer is better than that one" into a model that gives better answers? DPO changes the optimization; RLAIF changes who does the comparing. The preference signal is the constant.

🌉 The bridge you've just crossed

This is where your reinforcement-learning knowledge meets the Agentic-AI / LLM world directly. Every modern assistant — and every LLM-powered agent you build — sits on top of an RLHF-style alignment step. The MDPs, policies, and PPO from earlier stages aren't a separate subject from large language models; they're the layer that makes those models usable.

WEAKNESSCommon mistakes to avoid

✗ "RLHF gives the model a hand-written reward function."

The opposite. The entire point is that no one can write the reward for "be helpful." RLHF learns the reward from human comparisons via a trained reward model. If you could hand-write the reward, you wouldn't need RLHF at all.

✗ "More RL optimization is always better."

No — push PPO too hard and the policy over-optimizes the reward model, exploiting its flaws (sycophancy, verbosity, gibberish that scores well). The KL leash exists precisely because unlimited optimization against an imperfect proxy makes things worse, not better. There's a sweet spot.

✗ "The reward model is ground truth."

It's a proxy — an imperfect model trained on a finite set of human comparisons. Treating its score as the true measure of quality is exactly the mistake that leads to reward hacking. The humans are the ground truth; the reward model is a fallible stand-in for them.

KEY INSIGHTWhere this connects in your course

  • PPO (3.5) — the RL algorithm doing the actual optimization in Stage 3 of RLHF. The KL penalty here is a cousin of PPO's trust-region idea: stay close to where you were.
  • Reward hacking (4.2) — RLHF's sycophancy and verbosity are reward hacking aimed at the reward model. Same failure, new target.
  • Offline RL & imitation (4.4) — Stage 1 (SFT) is behavioural cloning. RLHF starts where imitation learning ends, then improves past the demonstrations.
  • Where RL wins & the road ahead (4.6) — RLHF is arguably RL's biggest real-world win to date; next lesson zooms out to the whole landscape.

Hold the one sentence — human preference becomes a learned reward, then RL optimizes against it under a leash — and you understand how raw text predictors became the assistants you actually use.

GLOSSARYKey terms introduced

RLHF
Reinforcement Learning from Human Feedback — training an LLM by learning a reward model from human preference comparisons, then using RL to optimize the model against it.
Reward model (rφ(x,y))
A trained model that outputs a scalar score for a response, predicting how much a human would prefer it. The learned, proxy reward.
Preference / comparison data
Human judgments of the form "for this prompt, response A is better than response B." The raw material the reward model is trained on.
KL penalty
A term subtracted from the reward that punishes the policy for drifting too far from the SFT model — the leash that prevents reward hacking and gibberish.
SFT
Supervised Fine-Tuning — the first stage, fine-tuning a pretrained LLM on human demonstrations. Imitation learning / behavioural cloning.
DPO
Direct Preference Optimization — a newer method that optimizes the policy directly on preference pairs, skipping the explicit reward model and RL loop.
RLAIF
RL from AI Feedback — using an AI model (often constitution-guided) instead of humans to produce the preference comparisons.
Alignment
The broad goal of making a model's behaviour match human intentions and values — helpful, honest, harmless. RLHF is one major technique for achieving it.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1Why can't we just hand-write a reward function for "give a helpful, honest answer"?›

Because helpfulness has no measurable score in the environment — the space of good answers is too large and subtle to capture in a formula. Instead, RLHF exploits the fact that humans compare two answers reliably even when they can't score one. Those comparisons train a reward model, which becomes the reward.

2Name the three stages of RLHF and what each produces.›

① SFT: fine-tune a pretrained LLM on human demonstrations (imitation) → an SFT model that talks like an assistant. ② Reward model: train on human preference pairs → a scalar reward function r(x,y). ③ RL (PPO + KL): optimize the SFT model to maximize the reward model's score, with a KL leash → the aligned assistant.

3Map the RL terms (policy, state, action, reward) onto an LLM generating a response.›

Policy = the LLM. State = the prompt plus tokens generated so far. Action = generating the next token. Reward = the reward model's score for the finished response (sparse, delayed — usually one reward at the end). An aligned LLM is literally an RL agent.

4What is the KL penalty for, and what goes wrong without it?›

The KL penalty keeps the trained policy close to the original SFT model. Without it, PPO over-optimizes the imperfect reward model — exploiting its blind spots to produce sycophantic, verbose, confident-but-wrong, or outright gibberish text that scores high but no human wants. It's the leash that stops reward hacking of the reward model.

You've crossed the bridge into the LLM world. Next: where RL truly wins →