ProjekinAja. Reinforcement Learning Stage 3 · Deep RL Lesson 3.4
● Stage 3 · Lesson 3.4

Actor-Critic & A2C

The best of both worlds — pair a policy (the actor) with a value function (the critic) so the agent learns faster and steadier than pure policy gradients.

⏱ ~16 min read 🎭 Actor + critic ⚡ A2C / A3C

🎯 By the end you'll understand

How an actor (the policy π(a|s;θ)) and a critic (a value function V(s;w)) work together, how the critic's advantage A(s,a) = Q(s,a) − V(s) ≈ R + γV(s′) − V(s) — the TD error from Lesson 2.5 — replaces REINFORCE's noisy Monte Carlo return, and what A2C (and its ancestor A3C) actually do.

KEY INSIGHTThe one idea to take away

Keep the policy that chooses actions — but bolt a coach onto it. The coach gives quick "that was better or worse than expected" feedback after every move, instead of making the policy wait for the final score.

In Lesson 3.3 you learned to nudge a policy up toward actions that earned reward — and you saw its flaw: REINFORCE waits for the whole episode to finish, then weights every action by a wild, lucky-or-unlucky Monte Carlo return. Actor-critic fixes both problems at once. It hires a second network, the critic, whose only job is to estimate how good things are — and that estimate replaces the noisy return with a clean, low-variance signal you can apply step by step.

🔑 Hold onto this

Two networks learning together: the actor π(a|s;θ) chooses actions; the critic V(s;w) evaluates them. The critic's verdict — the advantage A(s,a) — tells the actor "this action was better/worse than average from here," and the actor adjusts. This is the baseline idea from 3.3 turned into a living, learning second network.

CORE CONCEPTThe actor and the critic

Stage 2 gave us value functions: how good a situation is. Lesson 3.3 gave us policies you can train directly. Actor-critic is the moment those two ideas shake hands. The whole method is two components with two different jobs:

🎭

The actor — the policy π(a|s;θ)

Exactly the policy network from 3.3. Given a state, it outputs a probability over actions and samples one. The actor is the part that acts — it's what we ultimately deploy.

🧑‍🏫

The critic — a value function V(s;w)

A second network with its own parameters w. It never picks an action. It only evaluates: "from this state, how much return do I expect?" That estimate is the feedback the actor learns from.

The crucial shift is where the learning signal comes from. In REINFORCE, the signal was Gt — the actual, sampled, full-episode return. Honest, but noisy, and only available once the episode ends. In actor-critic, the critic's estimate stands in for that return. Because a learned value function averages over many experiences, it's far smoother than any single episode's luck — that's the variance cut. And because we can ask the critic for an estimate at any moment, the actor no longer has to wait for the episode to end. It can learn from a single step.

That second point is the same leap you made in Lesson 2.5: from Monte Carlo (wait for the end) to temporal-difference learning (update from the very next step using a bootstrapped estimate). Actor-critic is, at heart, policy gradients made TD-style.

📖 Actor

The policy component π(a|s;θ) that selects actions. It is updated by policy gradient, weighted by the critic's advantage signal.

📖 Critic

The value function component V(s;w) (or Q) that estimates expected return. It does not choose actions; it evaluates them, providing the actor's learning signal.

INTUITIONThe coach and the student

Picture a student learning a sport — say, your little sister moving on from the bike to learning tennis. The student is the actor: she's the one swinging the racket, trying shots, building instinct for what to do in each situation. Left alone, she'd only find out how she did at the very end — did she win the match or not? That's REINFORCE: one big delayed verdict, and she has to guess which of her hundred shots actually mattered.

Now put a coach courtside — the critic. The coach doesn't grab the racket. The coach watches and, after every single shot, gives a quick read: "ooh, that cross-court was better than I'd have expected from that position" or "that drop shot was worse than usual, don't lean on it." The student doesn't wait for the final score; she adjusts shot by shot, steered by feedback that's specific, immediate, and calibrated to what's normal from each situation.

The actor tries moves; the critic gives quick "better-or-worse-than-expected" feedback after every move — instead of only at the end of the game.

Two things make this work. First, the coach's feedback is relative — "better than expected from here" — not absolute. A great shot from a hopeless position still deserves praise; a sloppy shot from an easy setup deserves a frown. Second, the coach is also learning: early on the coach's reads are rough, but every rally sharpens them. Student and coach improve together. That dual learning is the engine of actor-critic — and, as we'll see at the end, also its main hazard.

FORMALThe advantage function

Let's turn "better or worse than expected" into math. The critic estimates V(s), the expected return from a state — the average over all the actions the policy might take there. A specific action, meanwhile, has its own expected return: Q(s,a). The gap between them is exactly the coach's verdict, and it has a name — the advantage:

A(s,a) = Q(s,a) − V(s) the advantage — how much better than average action a is in state s

Read it plainly: V(s) is "how I typically do from here," Q(s,a) is "how I do if I take this action," so A(s,a) is the difference — positive if the action beat the average, negative if it fell short. This is precisely the baseline idea from Lesson 3.3, where we subtracted V(s) from the return. The advantage is the baselined signal, given a proper name.

But we don't have Q(s,a) lying around — the critic only learns V. Here's the trick: estimate Q(s,a) with one real step of experience plus the critic's estimate of where you land. After taking action a you observe reward R and next state s′, so Q(s,a) ≈ R + γV(s′). Substitute that in:

A(s,a) ≈ R + γV(s′) − V(s) the advantage, estimated from one step — this is the TD error δ from Lesson 2.5!

Look closely at what's inside that highlight. R + γV(s′) − V(s) is exactly the TD error δ you met in Lesson 2.5 — "what actually happened, minus what I predicted." That's the quiet bombshell of this lesson: the advantage estimate and the TD error are the same quantity. The number the critic computes to correct its own value estimate is the very same number it hands the actor as feedback. One signal, two jobs.

🔑 One signal, two jobs

The TD error δ = R + γV(s′) − V(s) tells the critic how wrong its prediction was (so it can fix V) and tells the actor how much better than expected the action was (so it can nudge π). Compute it once, use it twice.

So the actor's update is the policy gradient from 3.3, with the advantage in place of the raw return Gt:

θ ← θ + αθ · ∇θ log π(a|s;θ) · A(s,a) the actor update — policy gradient weighted by the advantage

When the advantage is positive (the action beat the average), the step pushes π(a|s) up. When it's negative, the step pushes it down. Same "make good actions more likely" rule as REINFORCE — but the weight is now a calibrated, low-variance "better-than-expected" score instead of a wildly swinging full-episode return.

📖 Advantage function

A(s,a) = Q(s,a) − V(s): how much better (or worse) action a is than the policy's average behaviour in state s. Estimated in practice as the TD error R + γV(s′) − V(s).

FRAMEWORKTwo update streams

Every step of actor-critic does two updates, driven by that single TD error δ = R + γV(s′) − V(s). Keep them straight — they're easy to confuse but do opposite-feeling jobs.

Critic updateActor update
What learnsvalue params wpolicy params θ
HowTD learning (Lesson 2.5)policy gradient (Lesson 3.3)
Goalmake V(s) a better predictormake good actions more likely
Uses δ asprediction error to minimiseadvantage weight on the gradient
Rulew ← w + αw δ ∇wV(s)θ ← θ + αθ δ ∇θ log π(a|s)

The critic uses δ to chase its own prediction toward reality: if the outcome was better than it guessed (δ > 0), it raises V(s); if worse, it lowers it. The actor uses the same δ to reshape behaviour: a positive δ means "that action over-delivered, do it more." Notice the actor literally reuses the critic's error term as its advantage — that reuse is the elegance of the whole method.

💡 The tell

If a method keeps a policy and a value function, and updates the policy using the value function's estimate (not the raw return), it's actor-critic. If there's no value function at all, it's plain REINFORCE. If there's no explicit policy and you act by argmax, it's value-based (DQN).

CORE CONCEPTThe architecture

Concretely, what do these two networks look like? You have a choice. You can keep them separate — two independent networks, one outputting action probabilities, one outputting a single value number. Or you can let them share a body — common layers that read the state, then split into two "heads": a policy head and a value head. Sharing is common in deep RL (especially from pixels) because both jobs need the same features — "understand the situation" — and it's cheaper to learn that understanding once.

📸 State s Shared layers (read & understand s) 🎭 Actor head policy π(a|s;θ) → action probabilities 🧑‍🏫 Critic head value V(s;w) → one number: expected return advantage A(s,a) steers the actor ↑
One state in; two heads out. The actor proposes an action, the critic scores the situation, and the critic's advantage feeds back to steer the actor. (Separate networks would simply duplicate the shared body.)

Either way, the data flow is the same loop: the state goes in, the actor picks an action, the environment returns a reward and next state, the critic computes δ, and both heads update. Round and round — just like Lesson 1.1's loop, now with a coach embedded inside the agent.

SEE ITActor + Critic, side by side

Here are both halves at once. On the left, the actor: a probability over four actions, starting uniform. On the right, the critic: its single value estimate V(s) for this state. Each step the agent samples an action and earns a reward; the critic compares the outcome to its expectation to produce an advantage — green if the action beat expectations, red if it fell short — and that signal drives the actor's bars up or down. Toggle between the noisy pure-policy method and steady actor-critic to feel the difference.

🎭 Interactive · The critic's advantage steers the actor

Press ▶ Step to take one action. The rewards are hand-authored so every run is identical — Action C is the good one. Watch the critic's V(s) climb as it learns, and the advantage shrink as predictions catch up to reality. The playback bar under the numbers pauses, steps back and scrubs to any step.

🎭 ACTOR — π(a|s) 0 1 🧑‍🏫 CRITIC — V(s) 0 10 V=0.0
Step
0
Action · reward
—
Advantage A(s,a)
—

Uniform actor, critic V(s)=0. Press ▶ Step.

Notice the rhythm in Actor-Critic mode: at first the critic guesses low, so even mediocre actions look "better than expected" (big advantages, jumpy bars). But the critic learns fast — V(s) climbs toward the true value — and soon the advantage for the good action stays positive while the others go negative. The bars settle cleanly. Flip to REINFORCE and the same action's weight swings on raw, uncalibrated returns: the bars lurch around far more before converging. That jitter is the variance the critic removes.

CORE CONCEPTA2C, A3C, and the loop

"Actor-critic" names the idea; A2C — Advantage Actor-Critic — names the practical, modern recipe built on it. The "advantage" in the name is exactly the A(s,a) you just met. The "2" distinguishes it from its older sibling, A3C (Asynchronous Advantage Actor-Critic), which came first.

A3C's big idea was parallelism: run many actor-critic workers at once, each on its own copy of the environment, each computing gradients and pushing them asynchronously to a shared set of parameters. The diversity of all those independent experiences decorrelates the data and stabilises learning — a clever alternative to the experience replay buffer that DQN used in Lesson 3.2. A2C is the synchronous simplification: still many parallel workers, but they all step together and their gradients are averaged in one clean batch before a single update. It turned out the synchronisation made A2C simpler, easier to tune, and often just as good — so A2C became the everyday default.

Here's the core loop, stripped to its essence (one worker shown; A2C just runs several and averages):

a2c.pyAdvantage Actor-Critic
for update in range(num_updates):
    # 1. collect a short rollout with the current actor
    batch = []
    s = env.reset()
    for t in range(n_steps):
        a = actor.sample(s)              # π(a|s;θ)
        s2, r, done = env.step(a)
        batch.append((s, a, r, s2, done))
        s = s2

    # 2. critic estimates values; compute the advantage = TD error
    for s, a, r, s2, done in batch:
        target = r + (0 if done else gamma * critic.V(s2))
        delta  = target - critic.V(s)      # δ = advantage A(s,a)

        # 3a. critic learns: shrink its prediction error (TD)
        w += alpha_w * delta * grad_V(critic, s)

        # 3b. actor learns: push π(a|s) by the advantage
        theta += alpha_t * delta * grad_log_prob(actor, s, a)

That's the whole game. Step 2's delta is computed once and feeds both updates — the critic shrinks it (3a), the actor rides it (3b). Modern algorithms (PPO, SAC, A3C, even AlphaGo's training) are all variations on this skeleton. Actor-critic isn't one algorithm so much as the shape of nearly all of today's deep RL.

📖 A2C / A3C

A2C (Advantage Actor-Critic): the synchronous, practical actor-critic — parallel workers whose advantage-weighted gradients are averaged into one update. A3C is its asynchronous ancestor, where workers push gradients independently to shared parameters.

WEAKNESSWhen two learners collide

An honest warning, because actor-critic isn't free magic. You now have two networks learning at once, and each depends on the other. The actor leans on the critic's value estimates to decide which actions to favour — but the critic is itself a moving target, learning from data the actor's choices generate. Two coupled learners chasing each other can wobble, oscillate, or spiral off.

Concretely: if the critic is wrong early on (and it always is), it can push the actor toward bad actions; the actor then visits states that mislead the critic further; and the loop reinforces the mistake. We also traded something for the variance reduction — bootstrapping off the critic's estimate introduces bias (if V is off, the advantage is off). That's the classic bias-variance deal from Lesson 2.5: REINFORCE was unbiased but high-variance; actor-critic is lower-variance but biased while the critic is still learning.

In practice this means actor-critic is sensitive to learning rates. The critic usually needs to learn a bit faster than the actor (so its feedback is trustworthy), and too-large actor steps can destabilise everything. Taming exactly this fragility — keeping the policy from changing too much per update — is the entire motivation for the next lesson.

🔑 The bridge to next lesson

Actor-critic is powerful but twitchy. PPO (Lesson 3.5) is actor-critic with a safety leash: it clips how far the policy can move in a single update, keeping the two learners from destabilising each other. PPO is, essentially, "actor-critic that won't fall over."

WEAKNESSCommon mistakes to avoid

✗ "The critic chooses the actions."

No — the actor chooses actions; the critic never acts. The critic only outputs a value estimate V(s) that's used to evaluate and guide. Acting by reading the value (an argmax) would make it value-based DQN, not actor-critic.

✗ "The advantage is just the raw return."

The advantage is the return (or one-step target) minus the critic's baseline: A = R + γV(s′) − V(s). It measures better-or-worse-than-expected, not raw outcome — that subtraction is exactly what cuts the variance, and it equals the TD error δ.

✗ "Actor-critic is unrelated to policy gradients."

It is a policy-gradient method. The actor update is the same ∇θ log π · (weight) rule from REINFORCE — actor-critic simply replaces the noisy return weight with the critic's advantage. It's REINFORCE-with-a-baseline, where the baseline became a learned network.

KEY INSIGHTWhere this connects in your course

  • Value functions (2.3) — the critic is a value function V(s); everything you learned about what values mean applies directly to the critic's job.
  • TD learning (2.5) — the advantage estimate R + γV(s′) − V(s) is literally the TD error δ; the critic learns by the same temporal-difference update.
  • Policy gradients & REINFORCE (3.3) — the actor update is the policy gradient, with the critic's advantage replacing the raw return; the baseline trick from 3.3 became a learned critic here.
  • PPO (3.5) — actor-critic with a clipped, stabilised update so the two coupled learners don't destabilise each other.
  • Modern & applied methods (3.6) — SAC, A3C, and most state-of-the-art algorithms are actor-critic at heart; this is the template the field runs on.

Hold the one sentence — let a critic's "better-than-expected" signal steer the actor, step by step — and the rest of deep RL is mostly about making that critic accurate and that steering safe.

GLOSSARYKey terms introduced

Actor
The policy component π(a|s;θ) that selects actions; updated by policy gradient weighted by the advantage.
Critic
The value-function component V(s;w) that evaluates states; provides the actor's learning signal, never picks actions.
Advantage function (A(s,a))
Q(s,a) − V(s): how much better than average an action is. Estimated as R + γV(s′) − V(s).
TD error as advantage
The one-step advantage estimate δ = R + γV(s′) − V(s) is the same number as the TD error: it corrects the critic and weights the actor.
A2C / A3C
Advantage Actor-Critic (synchronous, averaged gradients) and its Asynchronous ancestor (parallel workers pushing gradients independently).
Variance reduction
Replacing the noisy Monte Carlo return with the critic's calibrated advantage, giving steadier, faster learning (at the cost of some bias).

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1What are the two components of an actor-critic method, and what does each do?›

The actor is the policy π(a|s;θ) — it chooses actions. The critic is a value function V(s;w) — it evaluates states and supplies the actor's learning signal (the advantage). The critic never acts; the actor never evaluates.

2Write the advantage and explain why it equals the TD error.›

A(s,a) = Q(s,a) − V(s). Estimating Q(s,a) with one real step gives Q(s,a) ≈ R + γV(s′), so A ≈ R + γV(s′) − V(s) — which is exactly the TD error δ from Lesson 2.5 ("what happened minus what I predicted"). The same number corrects the critic and weights the actor's update.

3How does actor-critic improve on REINFORCE?›

It replaces REINFORCE's noisy full-episode return Gt with the critic's advantage, which is smoother (lower variance) and available every step via bootstrapping — so the agent can learn step-by-step (TD-style) instead of waiting for the episode to end. The trade-off is a little bias while the critic is still learning.

4What's the difference between A2C and A3C, and why can two learners be unstable?›

A3C runs parallel workers that push gradients asynchronously to shared parameters; A2C is the synchronous version that steps workers together and averages their gradients into one update (simpler, often just as good). Instability arises because the actor and critic are coupled — each learns from the other, so a wrong critic can mislead the actor and vice versa. It motivates careful learning rates and, in 3.5, PPO's clipping.

Next: put a safety leash on the actor-critic update with PPO →