ProjekinAja. Reinforcement Learning Stage 3 · Deep RL Lesson 3.3
● Stage 3 · Lesson 3.3

Policy Gradients & REINFORCE

A completely different idea: instead of learning values and acting greedily, learn the policy itself — and nudge it directly toward the actions that earned more reward.

⏱ ~15 min read 🎯 Policy-based RL 🧮 The policy gradient

🎯 By the end you'll understand

The split between value-based and policy-based RL, why we'd ever skip values and optimise the policy directly, the policy gradient theorem (∇θJ = 𝔼[∇θ log π(a|s) · Gt]), the REINFORCE algorithm, and how a baseline tames its biggest weakness — variance.

KEY INSIGHTThe one idea to take away

Stop computing the value of every action and picking the best. Just adjust the policy itself: push up the probability of actions that led to high reward, push down the ones that didn't.

Everything in Stage 2 and Lesson 3.2 followed one recipe: learn a value function, then act greedily with respect to it. Policy gradients throw that recipe out. There is no argmax over Q. Instead the agent is a knob-covered probability machine, and learning means turning the knobs so good actions become more likely. It's still trial and error — but now with a steering wheel bolted directly onto the action probabilities.

🔑 Hold onto this

A policy network π(a|s;θ) outputs a probability for each action. We define how much total reward that policy earns, call it J(θ), and then do gradient ascent: θ ← θ + α ∇θJ(θ). We climb the reward hill by reshaping the policy itself.

CONTEXTTwo families of RL

Step back and look at the whole field. Almost every algorithm you've met falls into one of two families, and the difference is simply what the agent actually learns.

Value-based methods learn a value function — usually Q(s,a), the expected return of taking action a in state s. The policy is never stored directly; it's implied: "look at all the Q-values for this state and take the biggest." Q-learning (Lesson 2.6) and Deep Q-Networks (3.2) are the headline acts here. The agent is a critic that scores moves, and acting just means picking the highest score.

Policy-based methods skip the scoring step entirely. They learn the policy π(a|s;θ) directly — a function (here a neural network with parameters θ) that takes a state and outputs the probability of each action. There's no value table to consult; the network simply tells you what to do, as a probability distribution you sample from. This lesson lives entirely in this family.

Think of it like two ways of becoming a good chess player. The value-based player memorises a number for every position-and-move ("this capture is worth +3") and always plays the highest number. The policy-based player instead builds an instinct — a gut feeling that, in this kind of position, you usually push the pawn — and refines that instinct directly every time a game goes well or badly. Same goal, very different machinery.

📖 Policy-based RL

A family of methods that directly parameterise and learn the policy π(a|s;θ), optimising its parameters θ to maximise expected return — without first learning a value function and acting greedily on it.

CORE CONCEPTWhy learn the policy directly?

If value-based methods already work — and DQN famously cracked Atari — why bother with a whole second family? Three concrete reasons.

🎚️

Continuous & high-dimensional actions

DQN must compute maxa Q(s,a) — fine when there are 4 buttons, hopeless when the action is "set each of 7 robot joint torques to a real number." A policy network can just output those numbers (or the parameters of a distribution over them). No max over an infinite set.

🎲

Stochastic policies

Sometimes the best strategy is genuinely random — think rock-paper-scissors, or bluffing in poker, or any game with hidden information. A greedy value policy is deterministic; a policy network can learn to be 30% rock, 30% paper, 40% scissors. Randomness also gives built-in exploration for free.

🎯

It optimises what we actually care about

We want high return. Policy gradients optimise return directly. Value methods optimise a proxy (accurate Q-values) and hope good actions follow — they can wobble or diverge when the value estimate is off.

The trade-off, as we'll see, is that policy gradients are high-variance and sample-hungry. But for continuous control — robotics, locomotion, anything with a steering wheel or a throttle — they're often the natural choice.

💡 The tell

If your action space is a handful of discrete buttons, value-based (DQN) is a strong default. If actions are continuous, high-dimensional, or the optimal play is genuinely random, reach for policy-based methods.

FRAMEWORKThe objective: what does "good policy" even mean?

To optimise the policy we first need a single number that says how good it is. That number is the objective, written J(θ): the expected return when the agent follows policy πθ.

J(θ) = 𝔼πθ[ Gt ] = 𝔼πθ[ R1 + γR2 + γ2R3 + … ] the objective — expected discounted return under the policy

Read it plainly: run the policy, collect a full trajectory, add up the discounted rewards (the return Gt from Lesson 1.3), and average over all the trajectories the policy could produce. A better policy makes high-return trajectories more likely, so J(θ) goes up.

Now the move is mechanical. We have a number we want to maximise, and that number is a function of the parameters θ. So we do what all of deep learning does: gradient ascent. Compute the gradient of J with respect to θ — the direction in parameter space that increases expected return the fastest — and take a step that way.

θ ← θ + α ∇θJ(θ) gradient ascent on the policy parameters (α = step size / learning rate)

Notice the plus sign. Supervised learning does gradient descent to minimise a loss; here we ascend to maximise reward. Same calculus, opposite direction. The entire challenge of this lesson is computing that gradient ∇θJ(θ) when J is an expectation over trajectories we can only sample, never write down in closed form.

FORMALThe policy gradient theorem

Here's the beautiful result that makes all of this possible. We can't differentiate "expected return" directly — the environment's dynamics are buried inside the expectation and we don't have their formula. But the policy gradient theorem shows the gradient takes a shape we can sample:

∇θJ(θ) = 𝔼πθ[ ∇θ log π(at|st;θ) · Gt ] the policy gradient theorem

We won't derive it — but let's read its shape, because the shape is the whole intuition. The gradient is an expectation (so we can estimate it by averaging over sampled experience) of two pieces multiplied together:

↗

∇θ log π(at|st;θ) — the direction

This points in the direction of θ-space that makes action at more probable in state st. It's the "make this action more likely" arrow.

⚖️

Gt — the weight

The return that actually followed. It scales the arrow: a big positive return means "take a big step toward this action"; a negative return flips the arrow into "make this action less likely."

Put them together and the rule is almost embarrassingly intuitive:

Increase the log-probability of every action you took, in proportion to how much reward followed it. Good actions get pushed up; bad actions get pushed down.

If an episode went great (Gt large and positive), every action along the way gets reinforced. If it went badly (Gt negative), those actions get suppressed. The policy literally reshapes itself toward whatever earned reward. Why the log? It's a convenient identity (∇ log π = ∇π / π) that turns the awkward gradient-of-an-expectation into a clean expectation we can sample — and it nicely normalises by how probable the action already was. You don't need the algebra; just trust that log π is the right thing to push on.

📖 Policy gradient

The gradient of expected return with respect to the policy parameters, ∇θJ(θ) = 𝔼[∇θ log π(a|s;θ) · Gt]. Following it (gradient ascent) increases the probability of high-return actions.

SEE ITNudging the policy

Here's the policy as the agent sees it: a probability over four actions, summing to 1. We start uniform — the agent has no idea which is best. Each episode, the agent samples an action; the environment hands back a return; and we apply the policy-gradient rule. Watch the bars move: high returns grow an action's probability, low returns shrink it. Over time the policy concentrates on the good action.

📊 Interactive · Watch the policy concentrate on the winning action

Press ▶ Step episode to run one episode and nudge the policy. The returns are hand-authored so every run is identical. Toggle the baseline to feel how it smooths the updates. The playback bar under the numbers pauses, steps back and scrubs to any episode.

0 1
Episode
0
Action sampled
—
Return this episode
—

Uniform policy — every action equally likely. Press ▶ Step episode.

Action C is the good one here — it keeps returning a high reward, so its bar climbs while the others fade. That climb is the policy gradient at work: no value table, no argmax — just probabilities being nudged by the reward that followed. Flip on the baseline and the updates get gentler and steadier, because we subtract off a "typical" return and react only to what's better or worse than expected.

CORE CONCEPTREINFORCE, step by step

The simplest algorithm that implements the policy gradient is called REINFORCE (also "Monte Carlo policy gradient"). The name nods to its mechanism — it reinforces good actions — and "Monte Carlo" tells you it learns from complete episodes, exactly the family you met in Lesson 2.9.

The loop is short:

1

Run a full episode

Follow the current policy πθ from start to finish, recording every (st, at, rt). Because the policy is stochastic, you naturally explore.

2

Compute the returns Gt

Working backward from the end, accumulate the discounted reward that followed each step. Now every action has a number saying how things turned out.

3

Nudge the parameters

For each step, take a gradient-ascent step in the direction ∇θ log π(at|st) · Gt. Good-return actions become more likely; bad ones less.

4

Repeat

New episode, new experience, another nudge. Slowly the policy climbs the reward hill.

reinforce.pyMonte Carlo policy gradient
for episode in range(num_episodes):
    # 1. roll out a full episode with the current policy
    states, actions, rewards = run_episode(policy)

    # 2. compute the return G_t at each step (discounted, backward)
    G = 0
    returns = []
    for r in reversed(rewards):
        G = r + gamma * G
        returns.insert(0, G)

    # 3. one gradient-ASCENT step per (state, action, return)
    for s, a, Gt in zip(states, actions, returns):
        grad = grad_log_prob(policy, s, a)   # ∇θ log π(a|s)
        theta += alpha * grad * Gt        # push up π(a|s) ∝ Gt
📖 REINFORCE

The Monte Carlo policy gradient algorithm: roll out a complete episode, compute the return Gt for each step, then update θ ← θ + α ∇θ log π(at|st) · Gt. Because it uses real, full-episode returns, it's unbiased but high-variance.

Why "unbiased but high variance"? Same coin you flipped in Lesson 2.9. Gt is a real, sampled return — on average it's exactly right, so the gradient estimate points the correct way in expectation (unbiased). But a single episode's return swings wildly from luck: the same good action might be followed by a string of unlucky rewards one episode and lucky ones the next. Those swings make each individual update noisy, so REINFORCE learns slowly and needs a lot of episodes. That's the price of pure Monte Carlo.

CORE CONCEPTCutting variance with a baseline

Here's the elegant fix. The problem with raw Gt is that it's measured against zero. If every action in a state tends to return around +100, then every action gets a big positive push — the policy can barely tell them apart, and the updates jitter on the absolute scale of the rewards.

So subtract a baseline b(s) — a reference level of "how good do we usually do from this state?" — and weight the gradient by (Gt − b(s)) instead:

∇θJ(θ) = 𝔼[ ∇θ log π(at|st;θ) · ( Gt − b(st) ) ] policy gradient with a baseline

Now an action is pushed up only if it did better than typical (Gt > b) and pushed down if it did worse (Gt < b). The natural choice for the baseline is the state value V(s) from Stage 2 — the expected return from that state. Then Gt − V(st) answers a sharper question: "was this action better than what I expected from here?" That quantity has a name you'll meet next lesson — the advantage.

The magic property: subtracting a baseline that doesn't depend on the action does not change the gradient on average — it only reduces its variance. Mathematically, the extra term 𝔼[∇θ log π · b(s)] sums to zero across actions, so the update still points the same way in expectation. You get steadier learning for free, with no bias. That's why the baseline is one of the highest-value tricks in all of policy-gradient RL.

🔑 The bridge to next lesson

A baseline is usually a learned value estimate V(s). So now you have two networks: a policy (the actor) and a value (the critic). That pairing is exactly actor-critic (Lesson 3.4) — the baseline idea, taken all the way.

COMPARISONValue-based vs policy-based

The two families side by side. Neither is universally better — they trade different strengths.

Value-based (DQN, Q-learning)Policy-based (REINFORCE)
What it learnsQ(s,a) — a value for each actionπ(a|s;θ) — the policy directly
How it actsgreedily: argmaxa Q(s,a)sample from the output probabilities
Action spacesdiscrete only (needs a max)discrete or continuous / high-dim
Policy typedeterministic (after greedy step)naturally stochastic
Optimisesa proxy (value accuracy)expected return directly
Sample efficiencybetter (reuses data via replay)worse — on-policy, sample-hungry
Stabilitycan diverge, but low-variance targetsunbiased but high variance

This is exactly why the field eventually combined them: take the policy's directness and the value function's low-variance signal, and you get actor-critic — best of both worlds. That's the road we're on.

WEAKNESSCommon mistakes to avoid

✗ "Policy gradients learn Q-values."

No — that's the value-based family. Policy gradients learn the policy π(a|s;θ) directly; there's no Q(s,a) and no argmax. A value function may appear later only as a baseline, never as the thing you act on.

✗ "REINFORCE is low-variance and efficient."

The opposite. Because it uses full Monte Carlo returns, REINFORCE is unbiased but high-variance and sample-hungry — it can need many episodes and is sensitive to the step size. The variance is precisely the problem a baseline (and later, a critic) is meant to fix.

✗ "A baseline biases the gradient."

It does not. As long as the baseline doesn't depend on the action, subtracting it leaves the gradient unchanged in expectation — it only reduces variance. That's the whole reason it's safe to use.

KEY INSIGHTWhere this connects in your course

  • Policy (2.2) — we defined π(a|s) as the agent's strategy; here we make it a trainable network and optimise it head-on.
  • Monte Carlo (2.9) — REINFORCE is the policy-gradient cousin of MC value estimation: full-episode returns, unbiased, high variance.
  • Deep Q-Networks (3.2) — the value-based contrast. PG sidesteps DQN's max-over-actions bottleneck for continuous control.
  • Actor-Critic & A2C (3.4) — turn the baseline into a learned critic and you get the next, more powerful method.
  • PPO & modern policy methods (3.5) — today's workhorse algorithms are policy gradients with clever variance and step-size control bolted on.

Hold the one sentence — push up the probability of actions that earned more reward — and the rest of Stage 3 is just making that push less noisy and more stable.

GLOSSARYKey terms introduced

Policy-based RL
Methods that learn the policy π(a|s;θ) directly, rather than learning values and acting greedily.
Policy gradient
The gradient of expected return w.r.t. policy parameters: ∇θJ = 𝔼[∇θ log π · Gt].
REINFORCE
The Monte Carlo policy-gradient algorithm: full-episode returns, unbiased, high variance.
Objective J(θ)
The expected discounted return under the current policy — the quantity we maximise.
Gradient ascent
Stepping parameters up the objective: θ ← θ + α∇θJ (the mirror of gradient descent).
Baseline b(s)
A state-dependent reference subtracted from the return to reduce variance without adding bias (often V(s)).
Variance
How much a gradient estimate swings from sample to sample; high variance means noisy, slow learning.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1In one sentence, how does a policy-based method differ from a value-based one like DQN?›

A value-based method learns Q(s,a) and acts greedily (argmax); a policy-based method learns the policy π(a|s;θ) directly and samples actions from it — no value table, no max.

2Read the policy gradient ∇θJ = 𝔼[∇θ log π(a|s) · Gt] in plain English.›

"Increase the probability of each action you took, in proportion to the return that followed it." ∇θ log π is the direction that makes the action more likely; Gt scales it — positive returns push the action up, negative returns push it down.

3Why is REINFORCE high-variance, and what does subtracting a baseline do about it?›

It uses full Monte Carlo returns, which swing a lot from luck — so each update is noisy. Subtracting a baseline b(s) (e.g. V(s)) reweights by (Gt − b), reacting only to how much better or worse than expected the action did. This cuts variance without biasing the gradient.

4Give one situation where policy-based RL clearly beats value-based, and why.›

Continuous control (e.g. setting robot joint torques). DQN would need maxa Q(s,a) over an infinite action set, which is intractable; a policy network can simply output the continuous action (or a distribution over it). Stochastic-optimal games (poker, rock-paper-scissors) are another: a policy can be genuinely random, a greedy value policy can't.

Next: add a critic and turn the baseline into a second network →