ProjekinAja. Reinforcement Learning Stage 3 · Deep RL Lesson 3.5
● Stage 3 · Lesson 3.5

Proximal Policy Optimization (PPO)

The workhorse of modern RL: a policy-gradient method that takes steps big enough to learn fast, but not so big they destroy the policy. The algorithm behind RLHF and much of applied RL today.

⏱ ~15 min read 🛡️ Safe, stable updates ⭐ The default choice

🎯 By the end you'll understand

Why a single policy-gradient step can collapse a good policy, the idea of a trust region that keeps each update close to home, and PPO's famous clipped objective — the cheap trick that caps how far one update can move the policy, giving you stable, sample-efficient learning across a huge range of tasks.

KEY INSIGHTThe one idea to take away

Take the biggest improvement you safely can — and not one step further.

You've already met policy gradients (3.3) and actor-critic (3.4). Both nudge the policy toward better actions. PPO answers a question those methods leave dangerously open: how big should the nudge be? Too small and learning crawls. Too big and you can shove a perfectly good policy off a cliff. PPO's whole job is to take the largest safe step on each batch of data.

🔑 Hold onto this

The data you train on was collected by the old policy. If a single update moves the policy too far, that data no longer describes what the new policy would actually do — so the update is now based on a lie. PPO clips each update so the policy can never wander too far from the one that gathered the data.

CORE CONCEPTWhy a single step can be dangerous

Imagine you're adjusting the seat and handlebars on your little sister's bike between rides. After watching one ride, you make a tweak. A small tweak — seat up a centimetre — and your observation from the last ride still roughly applies; she'll behave about the same, just a little better. But if you crank everything to the extreme based on that single ride, you've changed the bike so much that everything you learned from watching her no longer holds. She might fall over before she even starts.

Policy gradients have exactly this problem. Here's the mechanism, precisely. In Stage 3 you collect a batch of experience by running the current policy, then estimate the gradient that points toward higher reward and step the parameters θ in that direction. The trouble: the gradient is only a local estimate, accurate near the policy that produced the data. The advantages A(s,a) you computed (from 3.4) describe how good each action was under the old policy's behaviour. Step a little and they still roughly hold. Step a lot and you've created a policy that visits different states, takes different actions, and faces a world your batch never measured.

And policy-gradient steps can be surprisingly large. A high advantage multiplied by a steep gradient can swing some action probabilities from rare to near-certain in one update. The policy collapses onto whatever looked best in that one noisy batch, exploration dies, and the next batch — collected by this broken policy — is garbage. From there, returns can crater and never recover. This isn't hypothetical; uncontrolled policy-gradient updates are notoriously unstable.

💡 The tell

When you see a training curve that climbs nicely and then suddenly nose-dives and stays down, you're usually watching a policy that took one step too far. The fix is not a smaller learning rate everywhere — it's a principled limit on how far the policy itself is allowed to move per update.

INTUITIONTrust regions: from TRPO to PPO

The clean idea is a trust region: only trust your gradient estimate within a small neighbourhood of the current policy, and refuse to update beyond that neighbourhood. Improve as much as you like — but stay close to the policy that gathered the data, where your advantages are still meaningful.

The first algorithm to do this rigorously was TRPO (Trust Region Policy Optimization). It maximises the expected improvement subject to a hard constraint: the new policy can't differ from the old one by more than a tiny amount (measured by KL divergence, a distance between probability distributions). TRPO works beautifully and rarely collapses — but enforcing that constraint requires second-order math (it approximates the curvature of the objective and solves a constrained optimization each step). It's complex to implement and heavy to run.

📖 Trust region

A bounded neighbourhood around the current policy within which the gradient estimate is trusted. Updates are restricted to stay inside it, so the policy improves without straying so far that the collected data — and the advantages computed from it — stop being valid.

PPO (Proximal Policy Optimization, Schulman et al., 2017) keeps TRPO's spirit — stay close to the old policy — but throws away the expensive machinery. Instead of a hard constraint solved with second-order optimization, PPO bakes the "stay close" pressure directly into a simple objective you can optimize with ordinary gradient ascent (Adam, the works). "Proximal" literally means "near": each new policy stays near the previous one. The result is almost as stable as TRPO, a fraction of the code, and it became the default RL algorithm almost overnight.

FORMALThe probability ratio r(θ)

To talk about "how far the policy moved," PPO needs a number that measures change for a specific action. That number is the probability ratio: how much more — or less — likely the new policy makes an action compared to the old policy that collected the data.

r(θ) = πθ(a|s) / πθold(a|s) probability ratio

Read it as a multiplier on the action's probability:

The plain policy-gradient objective rewards pushing r(θ) up for actions with positive advantage A (good actions) and down for negative A (bad actions): you maximise r(θ) · A. That's correct in spirit — but notice there's nothing stopping r(θ) from rocketing to 5 or 10 in a single update. Make a good action ten times more likely on the basis of one noisy batch and you've blown straight past the trust region. The ratio gives PPO exactly the handle it needs: cap the ratio, and you cap how far the policy can move.

FORMALThe PPO clipped objective

Here's the whole algorithm in one expression. PPO maximises:

LCLIP(θ) = 𝔼[ min( r(θ)·A , clip(r(θ), 1−ε, 1+ε)·A ) ] PPO clipped surrogate objective

It looks busy, but it's two ideas working together. Take them one at a time, with ε = 0.2 (a common value), so the allowed band for the ratio is [0.8, 1.2].

The clip. The function clip(r, 1−ε, 1+ε) simply refuses to let the ratio leave the band [1−ε, 1+ε]. If the new policy tries to make an action more than 1+ε times as likely, the clipped term pins it at 1+ε. Beyond the band, the objective flattens — its gradient goes to zero — so there's simply no incentive to push the policy any further. The reward for overshooting is removed.

The min. Why also take the minimum of the clipped and unclipped terms? Because clipping alone would be too forgiving in one direction. The min makes the objective a pessimistic lower bound on the improvement. Walk through the two cases:

CaseWhat it meansEffect on the update
A > 0 (good action)We want to make it more likely, so r rises.Once r > 1+ε, the term is clipped — no more reward for going further. Gain is capped.
A < 0 (bad action)We want to make it less likely, so r falls.Once r < 1−ε, the term is clipped — no more reward for suppressing it further.

The min ensures that whenever the unclipped objective would be more optimistic than the clipped one, PPO uses the clipped (smaller) value — so the update never gets rewarded for moving the policy too far in the helpful direction. Crucially, if a step accidentally made a bad action more likely (the ratio moved the wrong way), the min lets the full unclipped penalty through, so PPO can still correct big mistakes. It clips the upside of overshooting, but keeps the downside of errors.

📖 Clipping & the clip range ε

Clipping caps the probability ratio to the band [1−ε, 1+ε] so a single update can't move the policy too far. The clip range ε (the hyperparameter, e.g. 0.2) sets how wide that band is — your trust-region size. This ε is the clip range, not the exploration ε from ε-greedy (2.8). Same Greek letter, completely different knob.

That's it. No KL constraint to solve, no second-order curvature — just a clipped multiplication you can differentiate and feed to any optimizer. PPO is TRPO's safety, delivered by one well-chosen min.

SEE ITThe clip in action

Two things to feel here. First, drag the probability ratio and watch the objective: without clipping it grows forever, but PPO's clipped version flattens the moment the ratio leaves the band [1−ε, 1+ε] — that flat region is what removes the incentive to overshoot. Second, hit Run training to compare two learners on the same task: one taking raw, unclipped steps (it lurches, then collapses) versus PPO (capped steps, steady climb).

🛡️ Interactive · How clipping caps the step — and saves the policy

Move the ratio across the dashed band; the green (clipped) curve goes flat outside it. Then run training and watch the unclipped policy crash while PPO keeps improving; the playback bar pauses, steps back and scrubs to the update where it goes wrong. Everything here is deterministic — same result every time.

1.00
0.20
Unclipped objective r·A
—
PPO clipped objective
—
Trust band [1−ε, 1+ε]
[0.80, 1.20]

For a good action (A > 0): inside the band both curves agree. Push r past 1+ε and the clipped curve flattens — the step stops paying off.

Episode return over training

Press ▶ Run training. Red is the learner taking raw, unclipped steps; purple is PPO.

The picture says everything: outside the dashed band the green clipped objective is flat, so gradient ascent gets nothing for pushing the ratio further — the policy is gently held inside its trust region. In the training curves, the unclipped learner spikes on a lucky batch, oversteps, and collapses; PPO takes capped steps and climbs steadily. Clipping prevents overshoot, and you can see it.

FRAMEWORKHow PPO actually runs

PPO is an actor-critic method (3.4): an actor (the policy πθ) and a critic (a value network V(s)) that supplies the advantages A. One more trick gives PPO its sample efficiency: because the clip keeps updates safe, PPO can squeeze multiple epochs of minibatch gradient updates out of each batch of collected data — re-using the same rollouts several times instead of throwing them away after one step. Here's the loop.

1

Collect rollouts

Run the current policy πθold in the environment for a batch of steps, recording states, actions, rewards, and the action probabilities πθold(a|s).

2

Estimate advantages

Use the critic V(s) to compute each step's advantage A — "how much better than expected was this action?" (Usually via GAE; the advantage idea is straight from 3.4.)

3

Optimize the clipped objective — for several epochs

Loop over the batch in minibatches, several passes (epochs), each time computing r(θ) against the frozen old probabilities and taking a gradient-ascent step on LCLIP. The clip keeps every step safe even on re-used data.

4

Update the critic, then repeat

Fit V(s) to the observed returns, set θold ← θ, throw the old batch away, and collect fresh rollouts with the improved policy.

ppo.pyclipped policy update
for iteration in range(num_iterations):
    # 1. collect a batch with the CURRENT (old) policy
    batch = collect_rollouts(policy)
    old_logp = batch.log_probs           # frozen: π_old(a|s)

    # 2. advantages from the critic (e.g. GAE)
    adv = compute_advantages(batch, value_net)

    # 3. MULTIPLE epochs of minibatch clipped updates
    for epoch in range(K_epochs):
        for mb in minibatches(batch):
            ratio = exp(policy.log_prob(mb) - old_logp[mb])   # r(θ)
            unclipped = ratio * adv[mb]
            clipped   = clip(ratio, 1-eps, 1+eps) * adv[mb]
            loss = -mean(min(unclipped, clipped))    # maximize L_CLIP
            loss.backward(); optimizer.step()

    fit_value_net(value_net, batch)        # 4. update the critic

Notice old_logp is captured once and held fixed through all the epochs — that's the "old policy" anchor the ratio measures against. The clip is the only thing letting you safely run those inner epochs without the policy running away.

Press play to watch that anchor at work on one good action, epoch by epoch. Choose ε first: it decides where PPO cuts the update short.

Six epochs on one batch
Clip range
1−ε 1+ε No clip PPO slope 0 0.60.81.0 1.21.41.6 probability ratio r(θ): new policy ÷ old policy
Epoch
6
r, no clip
1.48
r, PPO
1.24
maximise r·A (no clip) maximise the clipped objective (PPO) trust band [1−ε, 1+ε]
One good action (A = 1) after six epochs on the same batch, with each epoch moving r by 0.08 times the slope of the objective it climbs. Without the clip r reached 1.48. With ε = 0.2, PPO stopped at 1.24, just past the band. Press play to see where it stopped, or change ε.

COMPARISONWhy PPO became the default

PPO didn't win on a single benchmark record. It won on practicality — the combination of properties practitioners actually need.

PropertyWhat PPO gives you
SimpleA few lines on top of actor-critic. No KL solver, no second-order math (unlike TRPO).
StableThe clip prevents the catastrophic collapses that plague raw policy gradients.
Sample-efficient enoughMultiple epochs per batch reuse data — far better than one-step REINFORCE.
RobustWorks across discrete and continuous control, locomotion, games, and LLM tuning — often with little re-tuning.

That robustness is why PPO is the core optimizer in RLHF (Reinforcement Learning from Human Feedback) — the technique used to align large language models. There, the policy is the LLM, the reward comes from a learned reward model trained on human preferences, and PPO's clip is exactly what keeps the model from drifting too far from its starting point and breaking. You'll see that whole pipeline in Lesson 4.5.

🔑 Honest note

PPO is not magic. It still needs tuning — ε, learning rate, number of epochs, batch size, and the advantage estimator all matter. And it's on-policy: each batch must be collected by a recent version of the policy and is mostly used once, so for some tasks it's less sample-efficient than off-policy methods like SAC (next lesson) that learn from a big replay buffer. PPO trades peak data-efficiency for reliability — and very often, reliability is what wins.

WEAKNESSCommon mistakes to avoid

✗ "Bigger policy steps are always better — they learn faster."

Up to a point. Past the trust region, a bigger step trains on data that no longer describes the new policy, and a good policy can collapse in a single update. PPO deliberately caps the step because the largest safe step beats the largest possible step.

✗ "PPO's ε is the same ε as exploration."

No. In PPO, ε is the clip range — the width of the trust band [1−ε, 1+ε] on the probability ratio (e.g. 0.2). The ε in ε-greedy (2.8) is the probability of taking a random exploratory action. Same letter, unrelated knobs.

✗ "PPO is value-based, like DQN."

PPO is policy-based (actor-critic). It directly optimizes the policy πθ via the clipped objective; the value network is only a critic supplying advantages. DQN (3.2) learns a Q-value and acts greedily from it — a different family entirely.

KEY INSIGHTWhere this connects in your course

  • Policy gradients (3.3) — PPO is a policy-gradient method; r(θ) and the advantage A come straight from there.
  • Actor-critic & A2C (3.4) — PPO is actor-critic: the critic's advantages weight every clipped update.
  • Continuous control — DDPG, TD3, SAC (3.6) — the off-policy alternatives PPO trades sample-efficiency against; next lesson.
  • RLHF for LLMs (4.5) — PPO is the optimizer that aligns large language models to human preferences.

Hold the one sentence — take the biggest safe improvement, then stop — and PPO's clip, its trust region, and its place at the heart of modern RL all follow from it.

GLOSSARYKey terms introduced

PPO (Proximal Policy Optimization)
An actor-critic policy-gradient algorithm that keeps each update close to the previous policy via a clipped objective. Stable, simple, the modern default.
TRPO (Trust Region Policy Optimization)
PPO's predecessor: enforces a hard KL trust-region constraint with second-order optimization. Very stable but complex and heavy.
Trust region
A bounded neighbourhood around the current policy within which the gradient is trusted; updates stay inside it so collected data remains valid.
Probability ratio (r(θ))
πθ(a|s) / πθold(a|s) — how much more or less likely the new policy makes an action versus the old one.
Clipping
Capping the probability ratio to [1−ε, 1+ε] so a single update can't move the policy too far; flattens the objective outside the band.
Clip range (ε)
The hyperparameter setting the width of the trust band (e.g. 0.2). Distinct from the exploration ε of ε-greedy.
On-policy
Learning only from data collected by the current (or very recent) policy, used roughly once. Reliable but less data-efficient than off-policy methods.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1Why is a too-large policy-gradient step dangerous?›

The batch was collected by the old policy, so its advantages are only valid near that policy. A big step creates a policy that visits different states and takes different actions than the data describes — the gradient was based on stale information, and a good policy can collapse in one update.

2What is the probability ratio r(θ), and what does the clip do to it?›

r(θ) = πθ(a|s) / πθold(a|s) — how much more/less likely the new policy makes an action than the old one. PPO clips it to [1−ε, 1+ε]: outside that band the objective flattens (zero gradient), removing any incentive to push the policy further and so capping the step size.

3How does PPO get its trust-region behaviour without TRPO's heavy machinery?›

Instead of solving a hard KL constraint with second-order optimization, PPO bakes "stay close" into a single clipped objective — min(r·A, clip(r,1−ε,1+ε)·A) — which you can maximise with ordinary gradient ascent. Nearly TRPO's stability, a fraction of the code.

4Why can PPO run several epochs of updates on the same batch, and why is it still "on-policy"?›

The clip keeps every update small, so the policy stays close to the one that gathered the data — making it safe to re-use that data for a few epochs (good sample efficiency). But once those epochs move the policy, the batch is stale and discarded; new data must come from the updated policy. Learning only from recent-policy data is what makes PPO on-policy.

The default RL algorithm, in your toolkit. Next: continuous control →