Proximal Policy Optimization (PPO)
The workhorse of modern RL: a policy-gradient method that takes steps big enough to learn fast, but not so big they destroy the policy. The algorithm behind RLHF and much of applied RL today.
🎯 By the end you'll understand
Why a single policy-gradient step can collapse a good policy, the idea of a trust region that keeps each update close to home, and PPO's famous clipped objective — the cheap trick that caps how far one update can move the policy, giving you stable, sample-efficient learning across a huge range of tasks.
KEY INSIGHTThe one idea to take away
Take the biggest improvement you safely can — and not one step further.
You've already met policy gradients (3.3) and actor-critic (3.4). Both nudge the policy toward better actions. PPO answers a question those methods leave dangerously open: how big should the nudge be? Too small and learning crawls. Too big and you can shove a perfectly good policy off a cliff. PPO's whole job is to take the largest safe step on each batch of data.
The data you train on was collected by the old policy. If a single update moves the policy too far, that data no longer describes what the new policy would actually do — so the update is now based on a lie. PPO clips each update so the policy can never wander too far from the one that gathered the data.
CORE CONCEPTWhy a single step can be dangerous
Imagine you're adjusting the seat and handlebars on your little sister's bike between rides. After watching one ride, you make a tweak. A small tweak — seat up a centimetre — and your observation from the last ride still roughly applies; she'll behave about the same, just a little better. But if you crank everything to the extreme based on that single ride, you've changed the bike so much that everything you learned from watching her no longer holds. She might fall over before she even starts.
Policy gradients have exactly this problem. Here's the mechanism, precisely. In Stage 3 you collect a batch of experience by running the current policy, then estimate the gradient that points toward higher reward and step the parameters θ in that direction. The trouble: the gradient is only a local estimate, accurate near the policy that produced the data. The advantages A(s,a) you computed (from 3.4) describe how good each action was under the old policy's behaviour. Step a little and they still roughly hold. Step a lot and you've created a policy that visits different states, takes different actions, and faces a world your batch never measured.
And policy-gradient steps can be surprisingly large. A high advantage multiplied by a steep gradient can swing some action probabilities from rare to near-certain in one update. The policy collapses onto whatever looked best in that one noisy batch, exploration dies, and the next batch — collected by this broken policy — is garbage. From there, returns can crater and never recover. This isn't hypothetical; uncontrolled policy-gradient updates are notoriously unstable.
When you see a training curve that climbs nicely and then suddenly nose-dives and stays down, you're usually watching a policy that took one step too far. The fix is not a smaller learning rate everywhere — it's a principled limit on how far the policy itself is allowed to move per update.
INTUITIONTrust regions: from TRPO to PPO
The clean idea is a trust region: only trust your gradient estimate within a small neighbourhood of the current policy, and refuse to update beyond that neighbourhood. Improve as much as you like — but stay close to the policy that gathered the data, where your advantages are still meaningful.
The first algorithm to do this rigorously was TRPO (Trust Region Policy Optimization). It maximises the expected improvement subject to a hard constraint: the new policy can't differ from the old one by more than a tiny amount (measured by KL divergence, a distance between probability distributions). TRPO works beautifully and rarely collapses — but enforcing that constraint requires second-order math (it approximates the curvature of the objective and solves a constrained optimization each step). It's complex to implement and heavy to run.
A bounded neighbourhood around the current policy within which the gradient estimate is trusted. Updates are restricted to stay inside it, so the policy improves without straying so far that the collected data — and the advantages computed from it — stop being valid.
PPO (Proximal Policy Optimization, Schulman et al., 2017) keeps TRPO's spirit — stay close to the old policy — but throws away the expensive machinery. Instead of a hard constraint solved with second-order optimization, PPO bakes the "stay close" pressure directly into a simple objective you can optimize with ordinary gradient ascent (Adam, the works). "Proximal" literally means "near": each new policy stays near the previous one. The result is almost as stable as TRPO, a fraction of the code, and it became the default RL algorithm almost overnight.
FORMALThe probability ratio r(θ)
To talk about "how far the policy moved," PPO needs a number that measures change for a specific action. That number is the probability ratio: how much more — or less — likely the new policy makes an action compared to the old policy that collected the data.
Read it as a multiplier on the action's probability:
- r(θ) = 1 — the new policy is identical to the old one for this action. No change.
- r(θ) = 1.3 — the new policy makes this action 30% more likely.
- r(θ) = 0.7 — the new policy makes it 30% less likely.
The plain policy-gradient objective rewards pushing r(θ) up for actions with positive advantage A (good actions) and down for negative A (bad actions): you maximise r(θ) · A. That's correct in spirit — but notice there's nothing stopping r(θ) from rocketing to 5 or 10 in a single update. Make a good action ten times more likely on the basis of one noisy batch and you've blown straight past the trust region. The ratio gives PPO exactly the handle it needs: cap the ratio, and you cap how far the policy can move.
FORMALThe PPO clipped objective
Here's the whole algorithm in one expression. PPO maximises:
It looks busy, but it's two ideas working together. Take them one at a time, with ε = 0.2 (a common value), so the allowed band for the ratio is [0.8, 1.2].
The clip. The function clip(r, 1−ε, 1+ε) simply refuses to let the ratio leave the band [1−ε, 1+ε]. If the new policy tries to make an action more than 1+ε times as likely, the clipped term pins it at 1+ε. Beyond the band, the objective flattens — its gradient goes to zero — so there's simply no incentive to push the policy any further. The reward for overshooting is removed.
The min. Why also take the minimum of the clipped and unclipped terms? Because clipping alone would be too forgiving in one direction. The min makes the objective a pessimistic lower bound on the improvement. Walk through the two cases:
| Case | What it means | Effect on the update |
|---|---|---|
| A > 0 (good action) | We want to make it more likely, so r rises. | Once r > 1+ε, the term is clipped — no more reward for going further. Gain is capped. |
| A < 0 (bad action) | We want to make it less likely, so r falls. | Once r < 1−ε, the term is clipped — no more reward for suppressing it further. |
The min ensures that whenever the unclipped objective would be more optimistic than the clipped one, PPO uses the clipped (smaller) value — so the update never gets rewarded for moving the policy too far in the helpful direction. Crucially, if a step accidentally made a bad action more likely (the ratio moved the wrong way), the min lets the full unclipped penalty through, so PPO can still correct big mistakes. It clips the upside of overshooting, but keeps the downside of errors.
Clipping caps the probability ratio to the band [1−ε, 1+ε] so a single update can't move the policy too far. The clip range ε (the hyperparameter, e.g. 0.2) sets how wide that band is — your trust-region size. This ε is the clip range, not the exploration ε from ε-greedy (2.8). Same Greek letter, completely different knob.
That's it. No KL constraint to solve, no second-order curvature — just a clipped multiplication you can differentiate and feed to any optimizer. PPO is TRPO's safety, delivered by one well-chosen min.
SEE ITThe clip in action
Two things to feel here. First, drag the probability ratio and watch the objective: without clipping it grows forever, but PPO's clipped version flattens the moment the ratio leaves the band [1−ε, 1+ε] — that flat region is what removes the incentive to overshoot. Second, hit Run training to compare two learners on the same task: one taking raw, unclipped steps (it lurches, then collapses) versus PPO (capped steps, steady climb).
The picture says everything: outside the dashed band the green clipped objective is flat, so gradient ascent gets nothing for pushing the ratio further — the policy is gently held inside its trust region. In the training curves, the unclipped learner spikes on a lucky batch, oversteps, and collapses; PPO takes capped steps and climbs steadily. Clipping prevents overshoot, and you can see it.
FRAMEWORKHow PPO actually runs
PPO is an actor-critic method (3.4): an actor (the policy πθ) and a critic (a value network V(s)) that supplies the advantages A. One more trick gives PPO its sample efficiency: because the clip keeps updates safe, PPO can squeeze multiple epochs of minibatch gradient updates out of each batch of collected data — re-using the same rollouts several times instead of throwing them away after one step. Here's the loop.
Collect rollouts
Run the current policy πθold in the environment for a batch of steps, recording states, actions, rewards, and the action probabilities πθold(a|s).
Estimate advantages
Use the critic V(s) to compute each step's advantage A — "how much better than expected was this action?" (Usually via GAE; the advantage idea is straight from 3.4.)
Optimize the clipped objective — for several epochs
Loop over the batch in minibatches, several passes (epochs), each time computing r(θ) against the frozen old probabilities and taking a gradient-ascent step on LCLIP. The clip keeps every step safe even on re-used data.
Update the critic, then repeat
Fit V(s) to the observed returns, set θold ← θ, throw the old batch away, and collect fresh rollouts with the improved policy.
for iteration in range(num_iterations): # 1. collect a batch with the CURRENT (old) policy batch = collect_rollouts(policy) old_logp = batch.log_probs # frozen: π_old(a|s) # 2. advantages from the critic (e.g. GAE) adv = compute_advantages(batch, value_net) # 3. MULTIPLE epochs of minibatch clipped updates for epoch in range(K_epochs): for mb in minibatches(batch): ratio = exp(policy.log_prob(mb) - old_logp[mb]) # r(θ) unclipped = ratio * adv[mb] clipped = clip(ratio, 1-eps, 1+eps) * adv[mb] loss = -mean(min(unclipped, clipped)) # maximize L_CLIP loss.backward(); optimizer.step() fit_value_net(value_net, batch) # 4. update the critic
Notice old_logp is captured once and held fixed through all the epochs — that's the "old policy" anchor the ratio measures against. The clip is the only thing letting you safely run those inner epochs without the policy running away.
Press play to watch that anchor at work on one good action, epoch by epoch. Choose ε first: it decides where PPO cuts the update short.
COMPARISONWhy PPO became the default
PPO didn't win on a single benchmark record. It won on practicality — the combination of properties practitioners actually need.
| Property | What PPO gives you |
|---|---|
| Simple | A few lines on top of actor-critic. No KL solver, no second-order math (unlike TRPO). |
| Stable | The clip prevents the catastrophic collapses that plague raw policy gradients. |
| Sample-efficient enough | Multiple epochs per batch reuse data — far better than one-step REINFORCE. |
| Robust | Works across discrete and continuous control, locomotion, games, and LLM tuning — often with little re-tuning. |
That robustness is why PPO is the core optimizer in RLHF (Reinforcement Learning from Human Feedback) — the technique used to align large language models. There, the policy is the LLM, the reward comes from a learned reward model trained on human preferences, and PPO's clip is exactly what keeps the model from drifting too far from its starting point and breaking. You'll see that whole pipeline in Lesson 4.5.
PPO is not magic. It still needs tuning — ε, learning rate, number of epochs, batch size, and the advantage estimator all matter. And it's on-policy: each batch must be collected by a recent version of the policy and is mostly used once, so for some tasks it's less sample-efficient than off-policy methods like SAC (next lesson) that learn from a big replay buffer. PPO trades peak data-efficiency for reliability — and very often, reliability is what wins.
WEAKNESSCommon mistakes to avoid
Up to a point. Past the trust region, a bigger step trains on data that no longer describes the new policy, and a good policy can collapse in a single update. PPO deliberately caps the step because the largest safe step beats the largest possible step.
No. In PPO, ε is the clip range — the width of the trust band [1−ε, 1+ε] on the probability ratio (e.g. 0.2). The ε in ε-greedy (2.8) is the probability of taking a random exploratory action. Same letter, unrelated knobs.
PPO is policy-based (actor-critic). It directly optimizes the policy πθ via the clipped objective; the value network is only a critic supplying advantages. DQN (3.2) learns a Q-value and acts greedily from it — a different family entirely.
KEY INSIGHTWhere this connects in your course
- Policy gradients (3.3) — PPO is a policy-gradient method; r(θ) and the advantage A come straight from there.
- Actor-critic & A2C (3.4) — PPO is actor-critic: the critic's advantages weight every clipped update.
- Continuous control — DDPG, TD3, SAC (3.6) — the off-policy alternatives PPO trades sample-efficiency against; next lesson.
- RLHF for LLMs (4.5) — PPO is the optimizer that aligns large language models to human preferences.
Hold the one sentence — take the biggest safe improvement, then stop — and PPO's clip, its trust region, and its place at the heart of modern RL all follow from it.
GLOSSARYKey terms introduced
- PPO (Proximal Policy Optimization)
- An actor-critic policy-gradient algorithm that keeps each update close to the previous policy via a clipped objective. Stable, simple, the modern default.
- TRPO (Trust Region Policy Optimization)
- PPO's predecessor: enforces a hard KL trust-region constraint with second-order optimization. Very stable but complex and heavy.
- Trust region
- A bounded neighbourhood around the current policy within which the gradient is trusted; updates stay inside it so collected data remains valid.
- Probability ratio (r(θ))
- πθ(a|s) / πθold(a|s) — how much more or less likely the new policy makes an action versus the old one.
- Clipping
- Capping the probability ratio to [1−ε, 1+ε] so a single update can't move the policy too far; flattens the objective outside the band.
- Clip range (ε)
- The hyperparameter setting the width of the trust band (e.g. 0.2). Distinct from the exploration ε of ε-greedy.
- On-policy
- Learning only from data collected by the current (or very recent) policy, used roughly once. Reliable but less data-efficient than off-policy methods.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1Why is a too-large policy-gradient step dangerous?›
The batch was collected by the old policy, so its advantages are only valid near that policy. A big step creates a policy that visits different states and takes different actions than the data describes — the gradient was based on stale information, and a good policy can collapse in one update.
2What is the probability ratio r(θ), and what does the clip do to it?›
r(θ) = πθ(a|s) / πθold(a|s) — how much more/less likely the new policy makes an action than the old one. PPO clips it to [1−ε, 1+ε]: outside that band the objective flattens (zero gradient), removing any incentive to push the policy further and so capping the step size.
3How does PPO get its trust-region behaviour without TRPO's heavy machinery?›
Instead of solving a hard KL constraint with second-order optimization, PPO bakes "stay close" into a single clipped objective — min(r·A, clip(r,1−ε,1+ε)·A) — which you can maximise with ordinary gradient ascent. Nearly TRPO's stability, a fraction of the code.
4Why can PPO run several epochs of updates on the same batch, and why is it still "on-policy"?›
The clip keeps every update small, so the policy stays close to the one that gathered the data — making it safe to re-use that data for a few epochs (good sample efficiency). But once those epochs move the policy, the batch is stale and discarded; new data must come from the updated policy. Learning only from recent-policy data is what makes PPO on-policy.