Policy Gradients & REINFORCE
A completely different idea: instead of learning values and acting greedily, learn the policy itself — and nudge it directly toward the actions that earned more reward.
🎯 By the end you'll understand
The split between value-based and policy-based RL, why we'd ever skip values and optimise the policy directly, the policy gradient theorem (∇θJ = 𝔼[∇θ log π(a|s) · Gt]), the REINFORCE algorithm, and how a baseline tames its biggest weakness — variance.
KEY INSIGHTThe one idea to take away
Stop computing the value of every action and picking the best. Just adjust the policy itself: push up the probability of actions that led to high reward, push down the ones that didn't.
Everything in Stage 2 and Lesson 3.2 followed one recipe: learn a value function, then act greedily with respect to it. Policy gradients throw that recipe out. There is no argmax over Q. Instead the agent is a knob-covered probability machine, and learning means turning the knobs so good actions become more likely. It's still trial and error — but now with a steering wheel bolted directly onto the action probabilities.
A policy network π(a|s;θ) outputs a probability for each action. We define how much total reward that policy earns, call it J(θ), and then do gradient ascent: θ ← θ + α ∇θJ(θ). We climb the reward hill by reshaping the policy itself.
CONTEXTTwo families of RL
Step back and look at the whole field. Almost every algorithm you've met falls into one of two families, and the difference is simply what the agent actually learns.
Value-based methods learn a value function — usually Q(s,a), the expected return of taking action a in state s. The policy is never stored directly; it's implied: "look at all the Q-values for this state and take the biggest." Q-learning (Lesson 2.6) and Deep Q-Networks (3.2) are the headline acts here. The agent is a critic that scores moves, and acting just means picking the highest score.
Policy-based methods skip the scoring step entirely. They learn the policy π(a|s;θ) directly — a function (here a neural network with parameters θ) that takes a state and outputs the probability of each action. There's no value table to consult; the network simply tells you what to do, as a probability distribution you sample from. This lesson lives entirely in this family.
Think of it like two ways of becoming a good chess player. The value-based player memorises a number for every position-and-move ("this capture is worth +3") and always plays the highest number. The policy-based player instead builds an instinct — a gut feeling that, in this kind of position, you usually push the pawn — and refines that instinct directly every time a game goes well or badly. Same goal, very different machinery.
A family of methods that directly parameterise and learn the policy π(a|s;θ), optimising its parameters θ to maximise expected return — without first learning a value function and acting greedily on it.
CORE CONCEPTWhy learn the policy directly?
If value-based methods already work — and DQN famously cracked Atari — why bother with a whole second family? Three concrete reasons.
Continuous & high-dimensional actions
DQN must compute maxa Q(s,a) — fine when there are 4 buttons, hopeless when the action is "set each of 7 robot joint torques to a real number." A policy network can just output those numbers (or the parameters of a distribution over them). No max over an infinite set.
Stochastic policies
Sometimes the best strategy is genuinely random — think rock-paper-scissors, or bluffing in poker, or any game with hidden information. A greedy value policy is deterministic; a policy network can learn to be 30% rock, 30% paper, 40% scissors. Randomness also gives built-in exploration for free.
It optimises what we actually care about
We want high return. Policy gradients optimise return directly. Value methods optimise a proxy (accurate Q-values) and hope good actions follow — they can wobble or diverge when the value estimate is off.
The trade-off, as we'll see, is that policy gradients are high-variance and sample-hungry. But for continuous control — robotics, locomotion, anything with a steering wheel or a throttle — they're often the natural choice.
If your action space is a handful of discrete buttons, value-based (DQN) is a strong default. If actions are continuous, high-dimensional, or the optimal play is genuinely random, reach for policy-based methods.
FRAMEWORKThe objective: what does "good policy" even mean?
To optimise the policy we first need a single number that says how good it is. That number is the objective, written J(θ): the expected return when the agent follows policy πθ.
Read it plainly: run the policy, collect a full trajectory, add up the discounted rewards (the return Gt from Lesson 1.3), and average over all the trajectories the policy could produce. A better policy makes high-return trajectories more likely, so J(θ) goes up.
Now the move is mechanical. We have a number we want to maximise, and that number is a function of the parameters θ. So we do what all of deep learning does: gradient ascent. Compute the gradient of J with respect to θ — the direction in parameter space that increases expected return the fastest — and take a step that way.
Notice the plus sign. Supervised learning does gradient descent to minimise a loss; here we ascend to maximise reward. Same calculus, opposite direction. The entire challenge of this lesson is computing that gradient ∇θJ(θ) when J is an expectation over trajectories we can only sample, never write down in closed form.
FORMALThe policy gradient theorem
Here's the beautiful result that makes all of this possible. We can't differentiate "expected return" directly — the environment's dynamics are buried inside the expectation and we don't have their formula. But the policy gradient theorem shows the gradient takes a shape we can sample:
We won't derive it — but let's read its shape, because the shape is the whole intuition. The gradient is an expectation (so we can estimate it by averaging over sampled experience) of two pieces multiplied together:
∇θ log π(at|st;θ) — the direction
This points in the direction of θ-space that makes action at more probable in state st. It's the "make this action more likely" arrow.
Gt — the weight
The return that actually followed. It scales the arrow: a big positive return means "take a big step toward this action"; a negative return flips the arrow into "make this action less likely."
Put them together and the rule is almost embarrassingly intuitive:
Increase the log-probability of every action you took, in proportion to how much reward followed it. Good actions get pushed up; bad actions get pushed down.
If an episode went great (Gt large and positive), every action along the way gets reinforced. If it went badly (Gt negative), those actions get suppressed. The policy literally reshapes itself toward whatever earned reward. Why the log? It's a convenient identity (∇ log π = ∇π / π) that turns the awkward gradient-of-an-expectation into a clean expectation we can sample — and it nicely normalises by how probable the action already was. You don't need the algebra; just trust that log π is the right thing to push on.
The gradient of expected return with respect to the policy parameters, ∇θJ(θ) = 𝔼[∇θ log π(a|s;θ) · Gt]. Following it (gradient ascent) increases the probability of high-return actions.
SEE ITNudging the policy
Here's the policy as the agent sees it: a probability over four actions, summing to 1. We start uniform — the agent has no idea which is best. Each episode, the agent samples an action; the environment hands back a return; and we apply the policy-gradient rule. Watch the bars move: high returns grow an action's probability, low returns shrink it. Over time the policy concentrates on the good action.
Action C is the good one here — it keeps returning a high reward, so its bar climbs while the others fade. That climb is the policy gradient at work: no value table, no argmax — just probabilities being nudged by the reward that followed. Flip on the baseline and the updates get gentler and steadier, because we subtract off a "typical" return and react only to what's better or worse than expected.
CORE CONCEPTREINFORCE, step by step
The simplest algorithm that implements the policy gradient is called REINFORCE (also "Monte Carlo policy gradient"). The name nods to its mechanism — it reinforces good actions — and "Monte Carlo" tells you it learns from complete episodes, exactly the family you met in Lesson 2.9.
The loop is short:
Run a full episode
Follow the current policy πθ from start to finish, recording every (st, at, rt). Because the policy is stochastic, you naturally explore.
Compute the returns Gt
Working backward from the end, accumulate the discounted reward that followed each step. Now every action has a number saying how things turned out.
Nudge the parameters
For each step, take a gradient-ascent step in the direction ∇θ log π(at|st) · Gt. Good-return actions become more likely; bad ones less.
Repeat
New episode, new experience, another nudge. Slowly the policy climbs the reward hill.
for episode in range(num_episodes): # 1. roll out a full episode with the current policy states, actions, rewards = run_episode(policy) # 2. compute the return G_t at each step (discounted, backward) G = 0 returns = [] for r in reversed(rewards): G = r + gamma * G returns.insert(0, G) # 3. one gradient-ASCENT step per (state, action, return) for s, a, Gt in zip(states, actions, returns): grad = grad_log_prob(policy, s, a) # ∇θ log π(a|s) theta += alpha * grad * Gt # push up π(a|s) ∝ Gt
The Monte Carlo policy gradient algorithm: roll out a complete episode, compute the return Gt for each step, then update θ ← θ + α ∇θ log π(at|st) · Gt. Because it uses real, full-episode returns, it's unbiased but high-variance.
Why "unbiased but high variance"? Same coin you flipped in Lesson 2.9. Gt is a real, sampled return — on average it's exactly right, so the gradient estimate points the correct way in expectation (unbiased). But a single episode's return swings wildly from luck: the same good action might be followed by a string of unlucky rewards one episode and lucky ones the next. Those swings make each individual update noisy, so REINFORCE learns slowly and needs a lot of episodes. That's the price of pure Monte Carlo.
CORE CONCEPTCutting variance with a baseline
Here's the elegant fix. The problem with raw Gt is that it's measured against zero. If every action in a state tends to return around +100, then every action gets a big positive push — the policy can barely tell them apart, and the updates jitter on the absolute scale of the rewards.
So subtract a baseline b(s) — a reference level of "how good do we usually do from this state?" — and weight the gradient by (Gt − b(s)) instead:
Now an action is pushed up only if it did better than typical (Gt > b) and pushed down if it did worse (Gt < b). The natural choice for the baseline is the state value V(s) from Stage 2 — the expected return from that state. Then Gt − V(st) answers a sharper question: "was this action better than what I expected from here?" That quantity has a name you'll meet next lesson — the advantage.
The magic property: subtracting a baseline that doesn't depend on the action does not change the gradient on average — it only reduces its variance. Mathematically, the extra term 𝔼[∇θ log π · b(s)] sums to zero across actions, so the update still points the same way in expectation. You get steadier learning for free, with no bias. That's why the baseline is one of the highest-value tricks in all of policy-gradient RL.
A baseline is usually a learned value estimate V(s). So now you have two networks: a policy (the actor) and a value (the critic). That pairing is exactly actor-critic (Lesson 3.4) — the baseline idea, taken all the way.
COMPARISONValue-based vs policy-based
The two families side by side. Neither is universally better — they trade different strengths.
| Value-based (DQN, Q-learning) | Policy-based (REINFORCE) | |
|---|---|---|
| What it learns | Q(s,a) — a value for each action | π(a|s;θ) — the policy directly |
| How it acts | greedily: argmaxa Q(s,a) | sample from the output probabilities |
| Action spaces | discrete only (needs a max) | discrete or continuous / high-dim |
| Policy type | deterministic (after greedy step) | naturally stochastic |
| Optimises | a proxy (value accuracy) | expected return directly |
| Sample efficiency | better (reuses data via replay) | worse — on-policy, sample-hungry |
| Stability | can diverge, but low-variance targets | unbiased but high variance |
This is exactly why the field eventually combined them: take the policy's directness and the value function's low-variance signal, and you get actor-critic — best of both worlds. That's the road we're on.
WEAKNESSCommon mistakes to avoid
No — that's the value-based family. Policy gradients learn the policy π(a|s;θ) directly; there's no Q(s,a) and no argmax. A value function may appear later only as a baseline, never as the thing you act on.
The opposite. Because it uses full Monte Carlo returns, REINFORCE is unbiased but high-variance and sample-hungry — it can need many episodes and is sensitive to the step size. The variance is precisely the problem a baseline (and later, a critic) is meant to fix.
It does not. As long as the baseline doesn't depend on the action, subtracting it leaves the gradient unchanged in expectation — it only reduces variance. That's the whole reason it's safe to use.
KEY INSIGHTWhere this connects in your course
- Policy (2.2) — we defined π(a|s) as the agent's strategy; here we make it a trainable network and optimise it head-on.
- Monte Carlo (2.9) — REINFORCE is the policy-gradient cousin of MC value estimation: full-episode returns, unbiased, high variance.
- Deep Q-Networks (3.2) — the value-based contrast. PG sidesteps DQN's max-over-actions bottleneck for continuous control.
- Actor-Critic & A2C (3.4) — turn the baseline into a learned critic and you get the next, more powerful method.
- PPO & modern policy methods (3.5) — today's workhorse algorithms are policy gradients with clever variance and step-size control bolted on.
Hold the one sentence — push up the probability of actions that earned more reward — and the rest of Stage 3 is just making that push less noisy and more stable.
GLOSSARYKey terms introduced
- Policy-based RL
- Methods that learn the policy π(a|s;θ) directly, rather than learning values and acting greedily.
- Policy gradient
- The gradient of expected return w.r.t. policy parameters: ∇θJ = 𝔼[∇θ log π · Gt].
- REINFORCE
- The Monte Carlo policy-gradient algorithm: full-episode returns, unbiased, high variance.
- Objective J(θ)
- The expected discounted return under the current policy — the quantity we maximise.
- Gradient ascent
- Stepping parameters up the objective: θ ← θ + α∇θJ (the mirror of gradient descent).
- Baseline b(s)
- A state-dependent reference subtracted from the return to reduce variance without adding bias (often V(s)).
- Variance
- How much a gradient estimate swings from sample to sample; high variance means noisy, slow learning.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1In one sentence, how does a policy-based method differ from a value-based one like DQN?›
A value-based method learns Q(s,a) and acts greedily (argmax); a policy-based method learns the policy π(a|s;θ) directly and samples actions from it — no value table, no max.
2Read the policy gradient ∇θJ = 𝔼[∇θ log π(a|s) · Gt] in plain English.›
"Increase the probability of each action you took, in proportion to the return that followed it." ∇θ log π is the direction that makes the action more likely; Gt scales it — positive returns push the action up, negative returns push it down.
3Why is REINFORCE high-variance, and what does subtracting a baseline do about it?›
It uses full Monte Carlo returns, which swing a lot from luck — so each update is noisy. Subtracting a baseline b(s) (e.g. V(s)) reweights by (Gt − b), reacting only to how much better or worse than expected the action did. This cuts variance without biasing the gradient.
4Give one situation where policy-based RL clearly beats value-based, and why.›
Continuous control (e.g. setting robot joint torques). DQN would need maxa Q(s,a) over an infinite action set, which is intractable; a policy network can simply output the continuous action (or a distribution over it). Stochastic-optimal games (poker, rock-paper-scissors) are another: a policy can be genuinely random, a greedy value policy can't.