ProjekinAja. Reinforcement Learning Stage 2 · How the Agent Learns Lesson 2.9
● Stage 2 · Lesson 2.9

Monte Carlo vs TD

Two ways to learn values from experience — wait for the whole story, or learn as you go. Understanding the trade-off ties all of Stage 2 together.

⏱ ~14 min read ⚖️ Bias vs variance 🏁 Closes Stage 2

🎯 By the end you'll understand

How Monte Carlo learns values from complete episodes using the real return, how temporal-difference (TD) learning revises its estimate every single step by bootstrapping, and the bias–variance trade-off that decides which one to reach for. You'll also see why n-step methods and TD(λ) live on a spectrum right between them.

KEY INSIGHTThe one idea to take away

Monte Carlo waits for the whole episode and learns from what actually happened. TD doesn't wait — it learns at every step by trusting its own current guess about what comes next.

Both methods are trying to answer the same question you met back in Lesson 2.3: how good is this state? The value V(s) is the expected return from s. Neither method is handed that number — they both estimate it from experience. The only difference is when they update and what target they aim at. That single difference ripples out into bias, variance, speed, and which kinds of tasks each one can even handle.

🔑 Hold onto this

Imagine grading a driving route. Monte Carlo waits until you arrive, then grades every intersection using the actual total trip time. TD grades each intersection on the spot, using the time so far plus your best guess of the time still left. One is honest but jumpy; the other is smooth but trusts itself.

CORE CONCEPTMonte Carlo: wait for the whole story

Picture driving the same commute every morning, trying to learn how long the trip really takes from each landmark — your driveway, the highway on-ramp, the bridge, the office garage. A Monte Carlo learner does the obvious, patient thing: it drives all the way to work, checks the clock, and only then looks back and says, "Okay — from the on-ramp it took me 22 minutes today. Let me nudge my estimate for the on-ramp toward 22."

That's the whole idea. Monte Carlo learns from complete episodes. It plays an episode all the way to a terminal state, computes the actual return Gt from each state it visited — the real, discounted sum of every reward that followed — and then moves each state's value estimate a little toward that observed return.

Recall the return from Lesson 1.3:

Gt = Rt+1 + γRt+2 + γ2Rt+3 + … + γT−t−1RTthe actual return from time t to episode end T

The key word is actual. Monte Carlo doesn't guess what happens after the on-ramp — it has the receipts. The episode already finished, so every reward in that sum is a real number that genuinely occurred. This is what we mean when we say Monte Carlo does no bootstrapping: it never builds its estimate on top of another estimate. The target is reality, full stop.

📖 Monte Carlo (MC) value update

After an episode finishes, for each visited state St, move its value toward the actual return that followed:

V(St) ← V(St) + α[ Gt − V(St) ]MC update — target is the real return Gt

Read that bracket as an error: "how far off was my estimate from what actually happened?" If the on-ramp paid off better than expected, Gt − V(St) is positive and we nudge the value up. The learning rate α controls how big each nudge is, exactly as it did in Lesson 2.5. Average enough episodes and V(s) converges to the true expected return — because the average of the real returns is the expected return, by definition.

💡 Why it's "Monte Carlo"

The name comes from the casino — it's the same idea as estimating a probability by sampling lots of random trials and averaging. Here each finished episode is one sample of the return, and we average our way to the truth.

CORE CONCEPTTD: learn as you go

Now meet the impatient driver. Temporal-difference learning, from Lesson 2.5, refuses to wait until the office to learn anything. At every intersection it makes a small correction on the spot, using one real reward plus its current guess about the value of wherever it just landed.

So when TD pulls onto the highway, it doesn't need to know the final arrival time. It says: "That ramp cost me 3 minutes of real reward, and I already estimate the highway is worth, say, 18 more. So my target for the driveway is 3 + 18 = 21 — let me nudge the driveway toward 21." Then it drives on and does the same thing at the next step.

📖 TD(0) value update

At every step, after observing reward Rt+1 and next state St+1:

V(St) ← V(St) + α[ Rt+1 + γV(St+1) − V(St) ]TD update — target is reward + discounted estimate of the next state

That highlighted target, Rt+1 + γV(St+1), is the TD target, and the bracket is the famous TD error δt. Notice the part that makes TD what it is: the target contains V(St+1) — TD's own estimate of the next state. It is literally pulling itself up by its own bootstraps. That's bootstrapping: updating a guess using another guess.

This is exactly why Q-learning (2.6) and SARSA (2.7) are TD methods — they swap V for Q and bootstrap off Q(St+1, ·), but the skeleton is identical: one real reward, plus a discounted estimate of the future, minus the old estimate.

💡 The crucial superpower

Because TD only ever needs the next reward and the next state, it can learn from incomplete episodes — and even from episodes that never end at all. Monte Carlo simply cannot do this; it has nothing to compute until the story finishes.

FORMALThe two updates side by side

Put them next to each other and the entire lesson is visible in two lines. Both have the same shape — V(St) ← V(St) + α[ target − V(St) ] — they differ only in the target.

V(St) ← V(St) + α[ Gt − V(St) ]Monte Carlo · target = the actual full return (real, no estimate)
V(St) ← V(St) + α[ Rt+1 + γV(St+1) − V(St) ]TD(0) · target = one real reward + an estimate of the rest

Stare at the two highlighted targets — that contrast is the heart of everything that follows:

SEE ITMC vs TD on the same episode

Here's one short morning commute: four landmarks — Driveway → On-ramp → Bridge → Office (terminal) — with a real reward earned between each. Every value starts at 0. Set γ = 1 and α = 0.5 so the arithmetic is clean. Pick a method, then step through and watch when and how the value estimates change.

🚗 Interactive · One episode, two ways to learn

Same rewards every time (deterministic). MC: nothing moves until you arrive, then all states jump toward the real return. TD: each state updates the moment you leave it.

🏠 Driveway
0
🛣️ On-ramp
0
🌉 Bridge
0
🏢 Office
★
terminal
Method
Monte Carlo
Rewards collected
— → — → —
Updates so far
0

Pick a method, then press ▶ Step to begin the drive.

Run TD first and watch each box light up and change as you pass it — learning happens online, mid-trip. Then switch to MC: notice every box stays frozen at 0 until you reach the Office, at which point all three snap toward the real return at once. Same episode, same final rewards — utterly different rhythm of learning. That is the difference made visceral.

FRAMEWORKThe bias–variance trade-off

Why prefer one over the other? It comes down to a classic tension you'll see everywhere in machine learning: bias versus variance. Think of it as a trade-off between being honest and being steady.

Monte Carlo is unbiased but high-variance. Its target Gt is the genuine return, so on average it aims at exactly the right number — zero bias. But a single episode is the sum of many random events: every reward, every action choice, every chance outcome along the whole trajectory. One unlucky episode (a freak traffic jam) can hand the driveway a wildly high or low return, yanking the estimate around. Honest, but jumpy.

TD is low-variance but biased. Its target depends on just one random reward plus a stable estimate V(St+1), so far less randomness leaks in — the updates are smooth and steady. The catch: that estimate is currently wrong (especially early in training), so TD is aiming at a slightly-off target. It's biased. As the estimates improve, the bias shrinks, but it's there in a way it never is for MC.

🔑 The one-liner

MC trades steadiness for honesty (unbiased, but noisy). TD trades honesty for steadiness (biased, but calm). In practice TD's lower variance usually wins — it tends to learn faster and is far more sample-efficient — which is why nearly all the workhorse algorithms (Q-learning, SARSA, and their deep cousins) are TD methods.

One more practical edge for TD: because it bootstraps off the structure of the problem (the Bellman relationship between a state and its successor), it can exploit that structure to converge quickly. Monte Carlo throws that structure away and just averages whole returns — simpler to reason about, but slower to learn.

COMPARISONWhen to use which

Lay the properties out and the decision usually makes itself:

PropertyMonte CarloTD(0)
Targetactual return GtRt+1 + γV(St+1)
Biasunbiasedbiased (bootstraps)
Variancehighlow
Bootstrapping?noyes
When it updatesonly at episode endevery step (online)
Learns online from incomplete episodes?noyes
Needs episodes to terminate?yes — requiredno
Works in continuing (never-ending) tasks?noyes
Typical speed / sample efficiencyslowerusually faster
🎲

Reach for Monte Carlo when…

the task is clearly episodic (it ends — a game, a maze, a hand of cards), you can afford to wait for episodes to finish, and you want unbiased estimates with dead-simple logic and no bootstrapping to worry about.

⏱️

Reach for TD when…

you need to learn online from a flowing stream of experience, the task is continuing or very long, or you simply want faster, more sample-efficient learning. This covers most real problems — and it's why Q-learning and SARSA are TD methods at heart.

CORE CONCEPTThe spectrum between them

MC and TD aren't a binary choice — they're the two ends of a dial. TD(0) looks one real step ahead, then bootstraps. Monte Carlo looks all the way to the end and never bootstraps. What lives in between?

n-step methods turn the dial: take n real rewards, then bootstrap off the estimate at step t+n. With n = 1 you have plain TD; let n → ∞ (the whole episode) and you recover Monte Carlo. Larger n means more real reward in the target (less bias) but more accumulated randomness (more variance) — you're literally sliding along the bias–variance trade-off.

Gt(n) = Rt+1 + γRt+2 + … + γn−1Rt+n + γnV(St+n)n-step return — n real rewards, then bootstrap

TD(λ) is the elegant unifier: instead of committing to one value of n, it blends all the n-step returns at once, weighting them with a decay factor λ ∈ [0, 1]. Set λ = 0 and you get TD(0); set λ = 1 and you get (essentially) Monte Carlo. The λ knob lets you tune exactly how far along the spectrum you want to sit — often a middle value learns faster than either extreme.

💡 The mental picture

One dial, two ends. Crank it toward TD for steady, online, biased learning; crank it toward MC for honest, unbiased, episode-end learning. n-step and TD(λ) let you stop anywhere in between.

WEAKNESSCommon mistakes to avoid

✗ "MC and TD give the same updates — just different timing."

No. They aim at different targets. MC's target is the real return Gt; TD's is Rt+1 + γV(St+1), which leans on an estimate. They generally produce different value estimates along the way and have different bias and variance — not merely a scheduling difference.

✗ "TD is unbiased because it uses a real reward."

TD is biased. Yes, it uses one real reward — but its target also contains γV(St+1), a bootstrapped guess that's wrong while learning. Monte Carlo is the unbiased one, because its target is built entirely from real returns.

✗ "Monte Carlo works fine for never-ending tasks."

It can't. MC needs an actual return, which requires the episode to terminate so the reward sum is finite and computable. For continuing (non-terminating) tasks, you need TD (or another bootstrapping method) that learns from each step without waiting for an end that never comes.

KEY INSIGHTWhere this connects in your course

  • TD learning (2.5) — this lesson recaps the TD(0) update and places it opposite Monte Carlo on the bias–variance map.
  • Q-learning (2.6) & SARSA (2.7) — both are TD methods: they bootstrap off Q(St+1, ·), inheriting TD's low variance, bias, and online learning.
  • The return & discounting (1.3) — Monte Carlo's target Gt is exactly the discounted return you first computed there.
  • Exploration vs exploitation (2.8) — whichever value-learning method you pick, you still need ε-style exploration to gather honest experience to learn from.

This closes Stage 2: you now have the full toolkit for learning values and policies from experience with tables. But what happens when there are too many states to fit in a table — millions of pixels, continuous controls? In Stage 3, Lesson 3.1 we hit that wall head-on — When Tables Break — and bring in neural networks to approximate V and Q. The TD ideas you just learned carry straight over: deep RL is, at its core, TD learning with a function approximator.

GLOSSARYKey terms introduced

Monte Carlo (MC)
A value-learning method that waits for an episode to finish, then updates each visited state toward the actual return observed from it. No bootstrapping.
Return (Gt)
The actual discounted sum of all rewards from time t to the episode's end — Monte Carlo's learning target.
Bootstrapping
Updating a value estimate using another value estimate (e.g. TD's γV(St+1)) rather than a real, complete return. The defining feature of TD.
Bias–variance trade-off
The tension between aiming at the right number on average (low bias) and producing steady, low-noise updates (low variance). MC is unbiased/high-variance; TD is biased/low-variance.
Episodic vs continuing
Episodic tasks terminate (games, mazes); continuing tasks run forever. MC needs episodic tasks; TD handles both.
n-step methods / TD(λ)
A spectrum between TD and MC: n-step uses n real rewards then bootstraps; TD(λ) blends all n-step returns with a decay λ (λ=0 ⇒ TD, λ=1 ⇒ MC).

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1In one sentence each, what target does Monte Carlo aim at, and what target does TD(0) aim at?›

Monte Carlo aims at the actual return Gt — the real discounted sum of all rewards to the episode's end. TD(0) aims at Rt+1 + γV(St+1) — one real reward plus its own current estimate of the next state's value (bootstrapping).

2Which method is unbiased, which has lower variance, and why?›

MC is unbiased — its target is the genuine return, so on average it hits the truth — but it's high variance because a full episode accumulates the randomness of every reward and choice along the way. TD has lower variance — its target depends on just one random reward plus a stable estimate — but it's biased because that estimate is wrong while learning.

3Why can't you use Monte Carlo for a continuing (never-ending) task, while TD has no problem?›

MC needs the actual return, which requires the episode to terminate so the reward sum is finite and computable — in a never-ending task there's nothing to compute. TD only ever needs the next reward and the next state, so it learns from each step online without waiting for an end.

4Where do n-step methods and TD(λ) sit relative to MC and TD, and what knob moves you between the extremes?›

They sit on the spectrum between them. For n-step methods, the knob is n: n = 1 is TD(0), n → ∞ is Monte Carlo; larger n means less bias but more variance. For TD(λ), the knob is λ: λ = 0 gives TD(0) and λ = 1 gives (essentially) MC, blending all n-step returns in between.

That closes Stage 2. Next: why tables break, and the leap to deep RL →