Continuous Control — DDPG, TD3 & SAC
A robot doesn't pick from a menu of buttons. It outputs torques, angles, throttle, steering — continuous actions. These are the algorithms built for that world, and how each one fixes the one before it.
🎯 By the end you'll understand
Why argmax-over-actions (the heart of DQN) collapses when actions are continuous, how DDPG trains an actor to output the best action, how TD3 repairs DDPG's overestimation with three small tricks, how SAC adds entropy to get exploration and stability for free — and a clean mental map of when to reach for which algorithm.
KEY INSIGHTThe one idea to take away
When you can't list the actions, you can't pick the best one by checking each. So stop searching for the best action — train a network to output it.
DQN (Lesson 3.2) works by scoring every action and grabbing the highest: argmaxa Q(s,a). That's a finite shopping list — fine for "left, right, fire." But a robot arm's elbow can take any torque between −2 and +2 newton-metres. There is no list. The whole family of continuous-control algorithms is one answer to that problem, refined three times over.
DDPG → TD3 → SAC is a ladder, not three rivals. DDPG is the first off-policy actor-critic for continuous actions. TD3 fixes DDPG's habit of overestimating value. SAC adds a reward for staying exploratory, which buys stability and sample efficiency. Each rung keeps the previous good ideas and patches a specific flaw.
CORE CONCEPTWhy DQN breaks on a robot arm
Picture the cleaning robot from Stage 2, but now give it real motors. In the discrete world it chose from {go north, go south, go east, go west} — four options. DQN scored all four with Q(s,a) and took the winner. Easy.
Now make it a robot arm reaching for a cup. The action isn't "north." It's a vector of joint torques — say six numbers, each a real value in some range. The number of possible actions is no longer four. It's ∞. You cannot loop over every torque combination to find the argmax, and you cannot train a network with one output neuron per action when there are infinitely many actions.
An RL problem where actions are real-valued — torques, angles, velocities, voltages — rather than a finite set of choices. The action space is a continuous range (often many dimensions at once), so you cannot enumerate actions or take a simple argmax over them. This is the natural setting for robotics, locomotion, steering, and most physical control.
One workaround is to chop each torque into, say, 20 bins and pretend it's discrete. We'll return to why that's a trap in the mistakes section. The better answer is the actor-critic idea from Lesson 3.4: keep a critic that scores actions, and add an actor whose job is to produce a good action so you never have to search.
FRAMEWORKDDPG — let a network output the action
Imagine teaching your little sister to ride a bike again — but this time, instead of "pedal" or "don't pedal," the only thing that matters is exactly how hard she squeezes the brake. Squeeze too little, she rolls into the fence; too much, she flips. The right answer is a precise number on a dial. You can't hand her a list of choices. You need her to learn to turn the dial to the right spot. That's what a deterministic actor does.
DDPG (Deep Deterministic Policy Gradient) is an off-policy actor-critic for continuous actions. It runs two networks:
The actor μ(s; θ)
A deterministic policy. Feed it a state, it outputs one specific action — the exact torque/angle to use. No probabilities, just "do this."
The critic Q(s,a; w)
Scores how good the actor's chosen action is — same Q-function idea as DQN, but it takes the action as an input instead of one output per action.
A policy that maps each state to a single action, written a = μ(s), with no randomness in the output. Contrast with a stochastic policy π(a|s) that gives a probability distribution over actions. Because the action is a definite number, you can take the gradient of the critic with respect to that action and push the actor toward higher value — the core trick of the deterministic policy gradient.
Here's the clever part. How do you improve the actor? You don't have labels for "the right torque." But you do have a critic that can tell you whether nudging the action up or down would raise its Q-value. So you train the actor to output the action that maximises the critic, by following the gradient straight through the critic into the actor:
Read it plainly: adjust the actor's weights θ so the action it outputs gets the highest score the critic can give. The critic, meanwhile, learns the usual way — from temporal-difference targets on transitions sampled from a buffer.
And where have you seen a replay buffer and target networks before? DQN, Lesson 3.2. DDPG borrows both tricks wholesale:
- Replay buffer — store transitions, sample mini-batches at random, so the critic trains on decorrelated data.
- Target networks — slow-moving copies of actor and critic that provide a stable TD target (DDPG nudges them gently every step, called a soft update, rather than copying outright).
- Exploration noise — a deterministic actor would always output the same action and never explore, so DDPG adds noise to the action while acting.
Before the honest warning, watch that climb for a single state: press play, then give the critic one wrongly optimistic spot and play it again.
DDPG is powerful but brittle. It's famously sensitive to hyperparameters and random seeds — the same code can soar or collapse depending on luck. Worse, it tends to overestimate Q: the actor learns to exploit spots where the critic is wrongly optimistic, and the two feed each other's errors. That single weakness is exactly what the next algorithm targets.
CORE CONCEPTTD3 — three small fixes for DDPG
TD3 (Twin Delayed DDPG) is DDPG with three patches bolted on. Nothing exotic — just three disciplined habits that stop the overestimation spiral. The name itself packs in two of them: Twin and Delayed.
Twin critics — take the minimum
Train two independent Q-networks. For the TD target, use the smaller of their two estimates. If a critic is wrongly optimistic about some action, the other usually isn't — taking the min refuses to trust the rosy one. This directly attacks overestimation.
Delayed policy updates
Update the actor (and the targets) less often than the critics — typically once every two critic updates. Let the critics settle on a reasonable value estimate before the actor chases it. Chasing a noisy, half-trained critic is how DDPG goes off the rails.
Target policy smoothing
When computing the target, add a little clipped noise to the target action. This says "the value of an action shouldn't change wildly if you wiggle it slightly," smoothing the critic and stopping the actor from exploiting sharp, brittle spikes in the Q-landscape.
The twin-critic min is the headline, so here it is in symbols. For a target action ã, the TD3 target value uses:
Taking the min of two estimates gives a slightly pessimistic target — and a touch of pessimism is exactly the cure for chronic optimism. The result: TD3 is markedly more stable than DDPG, and it became a standard strong baseline for continuous control almost overnight.
If someone says "DDPG but with two critics, slower actor updates, and noisy targets," they're describing TD3. The relationship is parent and improved child, not two separate algorithms.
CORE CONCEPTSAC — get paid to stay curious
DDPG and TD3 both chase one thing: maximum return. SAC (Soft Actor-Critic) changes the goal itself. It rewards the agent for return and for keeping its policy as random — as exploratory — as it can get away with while still winning. That extra "stay random" term is called entropy.
Think of learning to drive. A nervous learner who only ever knows one exact way to take a corner is fragile — one surprise and they're lost. A confident driver keeps options open, stays a little flexible, and so handles the unexpected. SAC bakes that "keep your options open" instinct right into the objective:
Decode it term by term. Σt Rt is the usual return — the rewards we've cared about all course. H(π(·|s)) is the entropy of the policy: a measure of how spread-out / uncertain its action choices are. High entropy means "still trying lots of things"; low entropy means "always doing the same thing." The knob α (the temperature) sets how much we value that randomness versus raw reward. SAC can even tune α automatically.
A formulation where the objective is to maximise reward plus the policy's entropy — i.e. succeed while staying as random as possible. The agent is explicitly paid to keep exploring rather than collapsing onto a single rigid behaviour. SAC is the best-known algorithm in this family.
What does that buy you? Quite a lot, which is why SAC is often the default choice for continuous control today:
Strong exploration, for free
The entropy bonus pushes the agent to keep trying varied actions — no need to hand-tune separate exploration noise the way DDPG does.
Stability & robustness
Like TD3, SAC uses twin critics with the min trick. Combined with the entropy term, it's far less seed-sensitive than DDPG.
Sample efficiency
It's off-policy with a replay buffer, so it reuses old experience heavily — important when each real-robot trial is slow or expensive.
A stochastic policy
Unlike DDPG/TD3's deterministic actor, SAC's actor outputs a distribution — naturally exploratory, and graceful when the best action is genuinely uncertain.
SEE ITPick the right algorithm
Two questions decide most of it: what kind of actions does your environment take, and what do you care about most — raw stability, squeezing the most out of every sample, or aggressive exploration? Set the toggles and the recommender names a method and tells you, in one line, why.
Notice the pattern the recommender encodes: discrete problems route to DQN or PPO; continuous problems route up the DDPG → TD3 → SAC ladder, and your priority picks the rung. There's no single "best" — there's a best for your situation.
COMPARISONThe full lineup
Here's the whole Stage-3 cast in one table. Read it top to bottom: discrete methods first, then the continuous ladder. The last column — "best for" — is the one to memorise.
| Algorithm | Action space | On / off-policy | Key idea | Best for |
|---|---|---|---|---|
| DQN | discrete | off-policy | Q-network + argmax; replay + target net | discrete actions (Atari, grids) |
| PPO | both | on-policy | clipped policy-gradient updates, very robust | a reliable default; large-scale tasks |
| DDPG | continuous | off-policy | deterministic actor μ(s) + critic; DQN tricks | continuous control (but brittle) |
| TD3 | continuous | off-policy | twin critics (min) + delayed actor + target smoothing | stable continuous control |
| SAC | continuous | off-policy | max-entropy: return + α·entropy; stochastic actor | the go-to for continuous control |
One more axis ties it together: on-policy vs off-policy. PPO (Lesson 3.5) is on-policy — it learns only from data its current policy just collected, then throws it away. That's stable and simple but sample-hungry. DDPG, TD3 and SAC are off-policy — they keep a replay buffer and reuse old experience many times, which is far more sample-efficient. The classic trade-off:
On-policy (PPO): simpler and very stable, but needs lots of fresh data. Off-policy (DDPG/TD3/SAC): sample-efficient by reusing experience, but historically trickier to stabilise — which is exactly why TD3 and SAC's stabilising tricks matter so much. On a slow physical robot where every trial is precious, that sample efficiency is often decisive.
WEAKNESSCommon mistakes to avoid
Discretising can work for one or two coarse dimensions, but it explodes fast: 6 joints chopped into 20 bins each is 206 ≈ 64 million actions — un-maxable. And fine control needs fine bins, making it worse. Worst of all, discretisation throws away the smoothness of the action space (close torques behave similarly), which is precisely what actor-critic methods exploit. Continuous-control algorithms exist because this shortcut breaks down.
They share a skeleton, but TD3 adds three deliberate fixes — twin critics (take the min), delayed actor updates, and target policy smoothing — aimed squarely at DDPG's overestimation and instability. In practice TD3 is meaningfully more reliable. Treating them as interchangeable hides the whole point of the upgrade.
Entropy is a means, not the goal. Crank α too high and the agent values randomness over reward — it stays scattered and never commits to good behaviour. Too low and you lose the exploration and stability benefits, collapsing toward brittle TD3-like behaviour. SAC works because it balances entropy against return (and often auto-tunes α), not because entropy is free reward.
KEY INSIGHTWhere this connects in your course
- DQN (3.2) — the replay buffer and target networks that DDPG, TD3 and SAC all inherit started here, along with the argmax trick that continuous control had to abandon.
- Actor-critic (3.4) — the actor-emits-action / critic-scores-it structure is the backbone of every algorithm in this lesson.
- PPO (3.5) — the on-policy counterpart; the stability-vs-sample-efficiency trade-off is the through-line between that lesson and this one.
- Model-Based RL (4.1) — next, we ask whether the agent can learn a model of the world to plan ahead, instead of learning purely from trial and error.
- Applications & robotics (Stage 4) — continuous control is what makes legged robots walk, arms grasp, and vehicles steer. SAC and TD3 are the workhorses you'll see again when we get applied.
That's a wrap on Stage 3. You can now read the whole deep-RL landscape — discrete vs continuous, on-policy vs off-policy, value-based vs actor-critic — and say which tool fits a problem and why. Next, Stage 4 turns from "how the algorithms work" to "how to actually build things with them."
GLOSSARYKey terms introduced
- Continuous control
- An RL problem with real-valued, often multi-dimensional actions (torques, angles, speeds) — no finite action list to argmax over.
- Deterministic policy gradient
- Improving a deterministic actor μ(s) by pushing its weights along the gradient of the critic Q(s, μ(s)) — climbing the critic to find higher-value actions.
- DDPG
- Deep Deterministic Policy Gradient — the first off-policy actor-critic for continuous actions; uses a deterministic actor, a critic, a replay buffer, target networks, and added exploration noise. Powerful but brittle.
- TD3
- Twin Delayed DDPG — DDPG plus twin critics (min), delayed policy updates, and target policy smoothing; fixes overestimation and improves stability.
- Twin critics
- Two independent Q-networks whose minimum is used for the TD target, producing a slightly pessimistic estimate that counters value overestimation.
- SAC
- Soft Actor-Critic — an off-policy, maximum-entropy actor-critic with a stochastic policy; strong exploration, stability, and sample efficiency. A common default for continuous control.
- Maximum-entropy RL
- RL that maximises reward plus policy entropy — the agent is rewarded for staying random/exploratory while still succeeding.
- Entropy (H)
- A measure of how spread-out / uncertain a policy's action choices are. High entropy = still exploring many actions; low entropy = committed to one.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1Why can't DQN be applied directly to a robot arm's continuous torques?›
DQN picks the action by argmaxa Q(s,a) over a finite list. Continuous torques form an infinite range (and many dimensions at once), so there's no list to max over and no way to have one output neuron per action. Continuous-control methods replace the max with an actor that directly outputs the action.
2What three things does TD3 add to DDPG, and what problem do they fix?›
(1) Twin critics — train two Q-nets and use the min for the target; (2) delayed policy updates — update the actor less often than the critics; (3) target policy smoothing — add clipped noise to the target action. Together they fight DDPG's overestimation of Q and its instability.
3What does SAC's entropy term actually do, and why is more entropy not always better?›
SAC maximises return + α·H(π), paying the agent to keep its policy random/exploratory while still winning — giving strong exploration and stability. But entropy is a means, not the goal: too much α makes the agent value randomness over reward and never commit to good behaviour. SAC works by balancing entropy against return.
4You have a slow physical robot where each trial is expensive. On-policy PPO or an off-policy method — and why?›
Favour an off-policy method (SAC or TD3). They keep a replay buffer and reuse old experience many times, so they're far more sample-efficient — crucial when data is costly. PPO is on-policy and discards data after each update, needing lots of fresh trials. SAC is often the go-to here for its sample efficiency plus built-in exploration and stability.