The Bellman Equation
The single most important equation in reinforcement learning — a beautifully simple idea that says the value of now is the reward you get next plus the discounted value of what comes after.
🎯 By the end you'll understand
Why a state's value is defined recursively instead of summed over the whole future, the Bellman equations for V and Q, the Bellman optimality equations behind every value-based algorithm, and how to compute a value by hand with a one-step backup.
KEY INSIGHTThe one idea to take away
The value of where I am equals the reward I get next, plus the discounted value of where I land next.
That one sentence is the Bellman equation. It looks almost too simple to matter — but it quietly turns an impossible problem (sum up every possible future) into an easy one (look just one step ahead). Almost every algorithm in the rest of this course is built on it.
A value isn't computed from scratch over an endless future. It's defined in terms of itself: value(now) = reward(next) + γ · value(next). Value points to value points to value — a chain you can solve one link at a time.
INTUITIONValue is recursive, not a giant sum
Back in Lesson 2.3 we defined the value of a state, V(s), as the expected return from that state — the total discounted reward you can expect if you start there and follow your policy. Written out, the return looks frightening:
To get V(s) the naive way, you'd have to imagine every path the future could take, discount each reward, and average over all of them. For any real problem that's a sum over a tree of futures so vast you could never finish it. So how does RL ever compute a value? With a trick that's almost embarrassingly clever once you see it.
Look hard at that return. Peel off the very first reward and stare at what's left:
The thing inside the brackets is another return — the return starting from the very next step, Gt+1. So:
That is the whole secret. The future is just "the next reward, then the same kind of future shifted forward by one." This is a recursive definition: the return refers back to the return. Take the expectation of both sides and you get a relationship between the value of this state and the value of the next one — the Bellman equation.
You never evaluate a chess position by mentally playing out every game to checkmate. You think: "this move wins a pawn now (the immediate reward), and the position it leaves me in looks good (the value of the next state)." Position-value = immediate gain + how good the resulting position is. That's Bellman thinking, and it's how grandmasters and engines both work.
FORMALThe state-value Bellman equation
Take the expectation of Gt = Rt+1 + γGt+1 under a policy π, conditioned on starting in state s. Because the expected return from the next state is its value, 𝔼[Gt+1] = V(St+1), you land on the central formula:
Read it left to right, in plain words:
The value of being in state s
How good this spot is if you act according to policy π from here on.
Expected — averaged over what could happen
Your policy may pick different actions, and the environment may land you in different next states. The 𝔼 averages over all of it, weighted by how likely each is.
The reward you get next
The immediate payoff for the step you're about to take. The "now" part.
The discounted value of where you land
The recursive term: the value of the next state, shrunk by the discount factor γ from Lesson 1.3. The "everything after" part — packed into a single number.
A relationship stating that the value of a state equals the expected immediate reward plus the discounted expected value of the next state. It expresses value recursively — each value defined in terms of the values that follow it — so the whole future collapses into a single one-step lookahead.
The win is enormous. The naive formula needs the entire infinite future. The Bellman equation needs only one reward and one neighbouring value. The rest of the future is already folded inside V(St+1) — you never unfold it.
FORMALThe action-value Bellman equation
The exact same logic applies to the action-value Q(s,a) from Lesson 2.3 — "how good is it to take action a in state s, then follow π?" Take action a now, collect Rt+1, land in St+1, then let the policy pick the next action At+1:
Same shape, same heartbeat: reward now + discounted value of what comes next. The only difference is bookkeeping. V asks "how good is this state?"; Q asks "how good is this state-action pair?" and so its recursive term reaches forward to the value of the next state-action pair. Hold onto Q — it's the form Q-learning will attack in Lesson 2.6.
V(s) is the average of Q(s,a) over the actions your policy takes; Q(s,a) is the immediate reward plus the discounted V of the next state. They're two camera angles on the same recursive truth.
SEE ITWatch values flow backward — the Bellman backup
Here's a tiny chain of four states. You start on the left and move right toward the Goal, which pays +10. The other transitions pay nothing (0). Every value starts at 0 because the agent knows nothing yet. Each time you press ▶ Backup, every state updates itself using the Bellman rule:
Notice the rhythm. After one backup only S3 (the Goal's neighbour) lights up — it can finally "see" the +10 one step away, so it becomes 0 + 0.9 · 10 = 9. After the next backup that value flows one more step left, and so on. Reward propagates backward, one link per pass, until nothing changes — exactly how value iteration works, and a preview of how Temporal-Difference learning in Lesson 2.5 nudges values one step at a time. Crank γ down and the wave dies out faster; that's the discount throttling how far the future reaches back.
CORE CONCEPTThe Bellman optimality equation
So far we've evaluated a given policy π. But the real prize in RL isn't grading a policy — it's finding the best one. For that we need the value under optimal behaviour, written V* and Q*.
The change is tiny but profound. Instead of averaging over whatever actions a policy happens to take, the optimal agent always picks the best next action. So the expectation over the policy becomes a max over actions:
Read the Q* version as: "the best you can do from (s,a) is the reward you get next, plus the discounted value of doing the best possible thing from wherever you land." The inner maxa′ is the optimistic assumption that, in the future, you'll act perfectly.
The recursive relationship satisfied by the optimal value functions V* and Q*. It replaces the policy-average with a maximum over actions — "always take the best next action" — and so describes the value of behaving optimally. Solve it and you have the best possible play in every state.
That inner maxa′ Q*(St+1, a′) is not a detail to skim past — it is literally the term Q-learning will estimate from experience. Once you have Q*, the optimal policy is trivial: in any state, just pick the action with the largest Q* value, argmaxa Q*(s,a). The Bellman optimality equation is the bridge from "knowing values" to "acting perfectly."
EXAMPLEA worked backup by hand
Let's compute a value with nothing but a pen, using the same chain from the widget. Four states, γ = 0.9. Moving into the Goal pays +10; every other move pays 0. We'll fill in the known piece, then back up.
Step 0 — what we already know. The Goal is terminal, so its value is just its own reward: V(Goal) = 10. Everything else starts unknown (call it 0).
Backup the Goal's neighbour, S3. From S3 the only move lands in the Goal, paying 0 on the way:
Now S2, using the value we just computed. This is the recursive magic — we never touch the Goal again; S3's value already contains it:
And finally S1:
| State | Reward to move on | Bellman backup | Value |
|---|---|---|---|
| S1 | 0 | 0 + 0.9 × 8.1 | 7.29 |
| S2 | 0 | 0 + 0.9 × 9.0 | 8.10 |
| S3 | 0 | 0 + 0.9 × 10 | 9.00 |
| Goal | — | terminal | 10.00 |
Two things to feel here. First, every state is worth slightly less than the one ahead of it — that's γ = 0.9 shaving 10% off the promise of the future at each step, the discounting from Lesson 1.3 made concrete. Second, we computed each value from only its neighbour, never from the full future. That's the Bellman equation doing exactly what it promised: turning an infinite sum into a chain of one-step backups. (And because there's a single path here, these values are already optimal — the max has nothing to choose between.)
FRAMEWORKWhy this is the keystone of RL
It's hard to overstate how much rides on this one equation. The naive definition of value asks for the impossible: average over every branch of the future. The Bellman equation replaces it with a local, one-step relationship you can apply over and over until the values settle. That single move makes value computation tractable.
Value iteration
Keep applying the Bellman optimality backup to every state until values stop changing — exactly the widget above. That's the classic planning method.
Temporal-Difference learning (2.5)
Do a Bellman-style backup from real experience, one step at a time, without ever knowing the model. The next lesson.
Q-learning (2.6)
Learn Q* directly using the maxa′ from the optimality equation. Off-policy, model-free.
Deep RL / DQN (Stage 3)
A neural network approximates Q*, trained to make the two sides of the Bellman equation agree. Same backup, scaled up.
Look down that list: TD, Q-learning, DQN — every one is a different way of making value(now) = reward(next) + γ · value(next) hold true. Get this equation in your bones and the rest of the course becomes variations on a single theme.
WEAKNESSCommon mistakes to avoid
That's the definition, not how you compute it. The Bellman equation's entire point is that you never sum the future — you fold it into the value of the next state and look just one step ahead. Reaching for the infinite sum means you've missed the trick.
The equation is a relationship that's always true about values — it holds whether or not you know the transition probabilities. Knowing the model lets you solve it directly (value iteration). When you don't know it, you instead learn the values by sampling experience — that's TD (2.5) and Q-learning (2.6). The equation is the target; the lessons ahead are how to hit it without a map.
They are not. The 𝔼 in Vπ averages over the actions your policy takes — it grades a given behaviour. The max in V* picks the single best action — it describes optimal behaviour. Swapping one for the other quietly changes what value you're computing.
KEY INSIGHTWhere this connects in your course
- Discounting, γ (1.3) — the γ that shrinks the recursive term is the same discount factor that defined the return; here it controls how far reward reaches back.
- Value functions V and Q (2.3) — Bellman is just those definitions written recursively; this lesson turns them from descriptions into something computable.
- Temporal-Difference learning (2.5) — does a Bellman backup from real, sampled experience, one step at a time, with no model.
- Q-learning (2.6) — learns Q* by directly using the maxa′ from the Bellman optimality equation.
Hold the one sentence — value of now = reward next + discounted value of next — and almost every algorithm ahead is a way of making that equation come true.
GLOSSARYKey terms introduced
- Bellman equation
- The recursive relationship V(s) = 𝔼[Rt+1 + γV(St+1)]: value of a state = expected reward + discounted value of the next state.
- Recursive / bootstrapping
- Defining (or estimating) a value in terms of other values rather than from the raw future. Updating a guess using another guess is "bootstrapping."
- Backup
- One application of the Bellman rule that updates a value from the reward and the value of successor state(s) — value "backed up" from the next step.
- Bellman optimality equation
- The Bellman relationship for optimal behaviour, using max over actions instead of the policy average.
- V* (optimal state-value)
- The value of each state when the agent always acts optimally.
- Q* (optimal action-value)
- The value of taking action a in state s, then acting optimally; argmaxa Q*(s,a) gives the best action.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1State the Bellman equation for Vπ in one plain-English sentence.›
The value of a state equals the expected immediate reward plus the discounted value of the next state: Vπ(s) = 𝔼[Rt+1 + γVπ(St+1)]. It defines value recursively — in terms of the value that follows it.
2Why is the recursive form such a big deal compared to the raw definition of return?›
The raw return is an infinite sum over every possible future — impossible to compute directly. The Bellman equation folds that entire future into the single number V(St+1), so you only ever need one reward and one neighbouring value. It turns an impossible sum into a local, iterable one-step backup.
3What changes between the Bellman equation for Vπ and the Bellman optimality equation for V*?›
The expectation over the policy's actions becomes a max over actions. Instead of averaging over whatever a given policy does, the optimal version picks the best next action. That maxa′ Q* is exactly what Q-learning uses.
4On the 4-state chain (γ = 0.9, Goal = +10), compute V(S2) given V(S3) = 9.›
V(S2) = r + γ · V(S3) = 0 + 0.9 × 9 = 8.1. Notice you never touched the Goal — its +10 is already baked into V(S3). That's bootstrapping in action.