Temporal-Difference Learning
How an agent learns values from raw experience — one step at a time, without waiting for the final outcome. This is the bridge from Bellman's theory to learning that actually works.
🎯 By the end you'll understand
Why the Bellman equation alone can't teach an agent that doesn't know its environment, the core idea of temporal-difference (TD) learning — nudging a guess toward a better guess — the TD(0) update for state values, what the TD error δ means, and why TD is the engine running underneath Q-learning and SARSA.
KEY INSIGHTThe one idea to take away
After every single step, the agent corrects its estimate of a state's value using the reward it just saw plus its current guess of what comes next. It learns a guess from a guess — and that's enough.
In Lesson 2.4 the Bellman equation gave us a beautiful relationship: a state's value equals the immediate reward plus the discounted value of where you land next. But it assumed you knew the environment — the rewards, the transition probabilities. A real agent dropped into the world knows none of that. Temporal-difference learning is how it figures values out anyway: not by solving equations, but by living through experience and adjusting after each step.
TD learning turns the Bellman equation (a fact about true values) into a Bellman update rule (a way to learn values from samples). You don't wait for the episode to finish — you learn online, step by step, using your own current estimates as a stand-in for the future.
CORE CONCEPTBellman tells us the relationship — but the agent is in the dark
Recall the Bellman equation for the value of a state under a policy:
This is a statement about true values: it says the value of being in state s is the average, over everything that could happen, of the next reward plus the discounted value of the next state. It's exactly right. The problem is the word "average" — that expectation 𝔼 hides the environment's dynamics: which rewards appear, with what probability, and where each action lands you. Our agent doesn't have that. It can't take the expectation because it doesn't know the distribution it would average over.
So what can the agent do? It can act, and watch what happens. Each step gives it a single sample: it was in state St, it got reward Rt+1, and it landed in St+1. The art of value learning is squeezing good value estimates out of a stream of these noisy samples. There are two opposite philosophies for doing it.
| Approach | When does it update? | What does it learn from? |
|---|---|---|
| Monte Carlo (Lesson 2.9) | only after the whole episode ends | the actual, complete return Gt it observed |
| Temporal-Difference (this lesson) | after every single step | the one reward seen, plus its own estimate of the next state |
Monte Carlo is patient and honest: it waits for the truth — the real return — then learns from it. TD is impatient and clever: it refuses to wait, and substitutes its own current guess of the future for the part it hasn't lived yet. That substitution is the whole trick, and it has a name: bootstrapping.
INTUITIONRevising your arrival time as you drive
Imagine you're driving across town to meet a friend, and as you set off you estimate you'll arrive at 6:00. That estimate is your "value" of the current state — your prediction of the outcome from here.
Now, the Monte Carlo way to learn would be to just drive, note that you actually arrived at 6:18, and only then update your sense of how long this trip takes. You learn one lesson per trip, and only at the very end.
But that's not how anyone drives. You hit your first intersection and the light's red, you crawl through a jam you didn't expect, and right there — without having arrived yet — you revise: "Hmm, this is slower than I thought, I'm probably looking at 6:10 now." You didn't wait for the truth. You took the small piece of new information (the reward of this step: a few lost minutes) and combined it with your updated guess of the time remaining, and nudged your overall estimate.
Every time you revise your ETA before arriving, using fresh information plus your best guess of what's left, you are doing temporal-difference learning. You're correcting one prediction (now) using a slightly better-informed prediction (a moment later) — a guess learning from a guess.
This is why it's called temporal-difference: you learn from the difference between two predictions made at two nearby moments in time — your estimate before the step, and your better-informed estimate after it. That difference is the signal.
FORMALThe TD(0) update for state values
Here is the rule that makes all of this precise. The agent keeps a table (or function) of value estimates V(s). After it takes a step from St, sees reward Rt+1, and lands in St+1, it updates:
Let's name every piece, because each one carries meaning:
V(St) — the old estimate
What the agent currently believes the value of the state it just left is. This is the thing being corrected.
α — the learning rate (step size)
A small number, usually between 0 and 1, controlling how big a nudge each step makes. Small α = cautious, slow, stable. Large α = bold, fast, jittery.
Rt+1 + γ V(St+1) — the TD target
The new, better-informed estimate: the real reward just observed, plus the discounted value of where the agent actually landed. This is the right-hand side of Bellman, but sampled from one real step.
The bracket — the TD error δ
δ = Rt+1 + γ V(St+1) − V(St). The gap between the better estimate (target) and the old estimate. The surprise.
Read the rule in plain words: move the old estimate a fraction α of the way toward the better estimate. If the target says the state is worth more than you thought, you raise V(St) a little. If less, you lower it a little. You never jump all the way — you ease toward the truth, one sampled step at a time.
Notice the update uses V(St+1) — the agent's own current estimate of the next state's value — as part of the target. It's pulling itself up by its own bootstraps: improving one guess by leaning on another guess. The next-state estimate is wrong too, at first. But as every state's estimate slowly improves, the targets get better, and the whole table converges toward the true values. That self-referential improvement is bootstrapping, and it's what separates TD from Monte Carlo (which never bootstraps — it always uses the real, final return).
Why does leaning on a wrong guess work at all? Because the reward Rt+1 in the target is real — a genuine drop of truth injected every step. Each update mixes a little real signal into the estimates, and that truth slowly washes the error out of the whole system. Bellman is the fixed point this process is crawling toward.
CORE CONCEPTThe TD error δ: a measure of surprise
The bracket deserves its own spotlight, because it's the most important quantity in this whole family of algorithms:
The TD error is literally what actually happened minus what you expected. It is a prediction error:
The difference between the TD target (reward seen + discounted estimate of the next state) and the current estimate of the state you left. δ > 0 means things went better than expected — raise the estimate. δ < 0 means worse than expected — lower it. δ = 0 means no surprise, so no change. The update is simply V(St) ← V(St) + α δ.
Think back to the drive. You estimated 6:00. After crawling through the jam, your better-informed estimate is 6:10. The TD error is +10 minutes of "worse than I thought" — a negative surprise about your speed — and it pushes your prediction in that direction. The size of the surprise sets the size of the lesson.
Here's a genuinely beautiful connection. In the late 1990s, neuroscientists (Schultz, Dayan, Montague) found that dopamine neurons in the brain don't fire for rewards themselves — they fire for unexpected rewards, and go quiet when an expected reward fails to appear. That pattern looks strikingly like a TD error: a signal of reward prediction error, not reward. It's one of the most cited bridges between machine learning and biology. The honest caveat: the brain is vastly more complicated than TD(0), and "dopamine = TD error" is a useful model, not a settled fact. But it's a real, serious hypothesis — and a lovely hint that this little update rule taps into something deep about how learning works.
SEE ITWatch a value estimate correct itself
Here's a single state whose value you're trying to learn. The agent keeps visiting it and stepping forward, each time seeing a reward Rt+1 and landing in a next state whose estimated value V(St+1) we'll show you. Step through the experience and watch the TD update pull V(St) toward its true value. With this stream — reward +1 and a next state valued at 10 under γ = 0.9 — the true value is the TD target 1 + 0.9·10 = 10 — see how close TD gets, and how the surprises shrink as it learns.
Two things to feel here. First, the estimate converges — it climbs from 0 toward 10 and the TD error (the surprise) shrinks toward zero as the guess gets good. Second, drag α around and re-run: a tiny α creeps up slowly and smoothly; a large α covers most of the gap in a single step. It never passes the target: each update moves the estimate a fraction α of the way there, and α is at most 1. So here a large α only looks fast, because every step brings the same sample. Its cost shows when the samples differ from step to step: the estimate then jumps after each new sample instead of averaging them. There's a real trade-off, and "faster" is not automatically "better."
EXAMPLEA TD update by hand
Let's do one on paper so the arithmetic is unmistakable. Suppose:
- The agent's current estimate of the state it's in is V(St) = 6.
- It takes a step, receives reward Rt+1 = 2, and lands in a next state it currently estimates at V(St+1) = 10.
- Discount γ = 0.9, learning rate α = 0.1.
Step 1 — form the TD target (the better-informed estimate):
Step 2 — compute the TD error (target minus old estimate):
A positive surprise: the state turned out worth more than the agent thought, so its estimate should go up.
Step 3 — apply the update (nudge a fraction α of the way):
Notice the estimate moved from 6 to 6.5 — only 10% of the way toward the target of 11, exactly as α = 0.1 dictates. It didn't leap to 11, because any single step is a noisy sample; we trust it only a little. Do this thousands of times across many steps and the estimates settle into the true values that satisfy Bellman.
Redo it with α = 0.5 instead. Target is still 11, error still +5, so the new value is 6 + 0.5·5 = 8.5 — a bigger leap. Bigger α, bigger step. Now you can feel exactly what the slider in the widget is doing.
FRAMEWORKWhy TD is so powerful
TD learning isn't just a neat trick — it's the workhorse underneath modern reinforcement learning, for several concrete reasons.
It learns online, step by step
No waiting for an episode to end. The agent improves during the task, from the very first step — exactly what you need for long or never-ending tasks (a robot that runs all day, a trading agent).
It works on incomplete episodes
Some tasks have no natural "end." Monte Carlo is stuck — there's no final return to wait for. TD doesn't care; it only ever needs the next single step.
Lower variance than Monte Carlo
A full return Gt sums up many random rewards, so it's noisy (high variance). The TD target uses just one reward plus a stable estimate, so each update is far less noisy — often it learns faster in practice. (The trade-off: TD's target is slightly biased while its estimates are still wrong. Lesson 2.9 unpacks this bias–variance trade.)
It's the engine under Q-learning & SARSA
Everything you just learned generalises from state values V(s) to action values Q(s,a). Swap V for Q in the update and you get the algorithms that actually control behaviour — starting with Q-learning next.
That's the payoff for the bootstrapping gamble: an algorithm that learns from raw, ongoing experience, cheaply and continuously. Hold the picture of the stepper widget — a guess being nudged toward a better guess, the surprises shrinking — and you understand the core of value-based RL.
WEAKNESSCommon mistakes to avoid
That's Monte Carlo, not TD. The whole point of temporal-difference learning is that it updates after every step, using its own estimate of the future in place of the not-yet-seen return. It can learn from a single transition, mid-episode, even in tasks that never end.
Bigger α means bigger nudges, which can learn faster early on, but each update then trusts the latest sample more. In noisy environments, where the samples differ from step to step, the estimate keeps jumping after each one and never settles. Too small and it crawls. Good practice often starts α moderate and decays it over time so estimates eventually converge. Faster ≠ better. The widget cannot show this cost, because its sample never changes.
No. δ = Rt+1 + γ V(St+1) − V(St). The reward is only one of three terms. The TD error is a prediction error — how much the outcome (reward plus the discounted value of where you landed) differed from what you expected. A big reward that you fully anticipated produces a TD error of zero.
KEY INSIGHTWhere this connects in your course
- Value functions (2.3) — TD is how the agent actually estimates the V(s) you met there, from experience instead of an oracle.
- The Bellman equation (2.4) — TD turns Bellman's equality into a sampled update rule; Bellman's true values are the fixed point TD converges to.
- Q-Learning (2.6) — the very next step: apply this same TD idea to action values Q(s,a) to learn an optimal policy.
- Monte Carlo methods (2.9) — the patient alternative that waits for the full return; we'll compare bias vs variance head-to-head.
Hold the one sentence — nudge a guess toward a better guess after every step — and the algorithms ahead are just this idea, applied to action values and wrapped in a strategy for exploring.
GLOSSARYKey terms introduced
- Temporal-difference (TD) learning
- Learning value estimates from experience by updating each step toward a target built from the reward seen plus a current estimate of the next state.
- TD error (δ)
- The prediction error Rt+1 + γV(St+1) − V(St): how much better (+) or worse (−) the step turned out than expected.
- TD target
- The better-informed estimate Rt+1 + γV(St+1) that the old estimate is nudged toward.
- Bootstrapping
- Updating an estimate using another estimate (here, V(St+1)) rather than waiting for the actual final return.
- Learning rate (α)
- The step size, between 0 and 1, controlling how far each update moves the estimate toward the target.
- Online learning
- Updating during the task, after each step, rather than only at the end of an episode.
- TD(0)
- The simplest temporal-difference method, which looks just one step ahead to form its target.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1Why can't an agent just use the Bellman equation directly to compute values?›
Bellman involves an expectation over the environment's rewards and transitions — the dynamics. A real agent doesn't know those, so it can't take the average. TD lets it learn the values from sampled experience instead, one step at a time.
2Write the TD(0) update and explain what "bootstrapping" means in it.›
V(St) ← V(St) + α[Rt+1 + γV(St+1) − V(St)]. Bootstrapping is the fact that the target uses V(St+1) — the agent's own current estimate of the next state — rather than a real, fully-observed return. It improves a guess by leaning on another guess.
3You're in a state you value at 4. You step, get reward +1, land in a state valued at 5, with γ = 0.8 and α = 0.5. Compute the TD error and the new value.›
Target = 1 + 0.8·5 = 1 + 4 = 5. TD error δ = 5 − 4 = +1. New value = 4 + 0.5·1 = 4.5. A positive surprise nudged the estimate up halfway toward the target.
4Name two advantages TD has over Monte Carlo, and the catch.›
It learns online / from incomplete episodes (no need to wait for an ending), and it has lower variance per update (one reward plus an estimate, not a noisy full return). The catch: while its estimates are still wrong, the bootstrapped target is biased — it leans on imperfect guesses — so it can be slow to shake off early errors. (Bias–variance trade-off, Lesson 2.9.)