ProjekinAja. Reinforcement Learning Stage 2 · How the Agent Learns Lesson 2.3
● Stage 2 · Lesson 2.3

Value Functions — V and Q

How does the agent know a situation is "good" before the rewards actually arrive? It learns a sense of long-term worth — and that sense has two names: V(s) and Q(s,a).

⏱ ~14 min read 🧭 The agent's compass 🤖 Cleaning robot

🎯 By the end you'll understand

What a value function is and why it captures long-term worth rather than the next reward; the difference between the state value V(s) and the action value Q(s,a); why an agent reads Q to choose and V to judge; and why value always quietly depends on the policy being followed.

KEY INSIGHTThe one idea to take away

A value function is the agent's sense of long-term worth: it tells the agent how good a situation is, not by the reward in hand, but by all the reward it can expect to collect from here on.

Rewards arrive one at a time, often late. If the agent only ever looked at the reward right in front of it, it would be hopelessly short-sighted. The value function fixes that — it lets the agent look ahead and feel, in advance, how promising a state or an action is. It's the bridge between "what just happened" and "what's likely to happen."

🔑 Hold onto this

There are two value functions, and the difference is one word. V(s) answers "how good is this state?" Q(s,a) answers "how good is this action in this state?" The agent reads Q to decide what to do, and V to judge where it is.

CORE CONCEPTKnowing "good" before the reward arrives

Think about sitting down at a piano. You don't know the "right" notes, so you experiment. After a while you start to feel that certain combinations of keys sound like they're heading somewhere good — long before the song finishes. If I play a good combination of keys, there's a good chance I'll earn the satisfaction (the reward) of a nice melody. That expected goodness — that hunch about where things are leading — is exactly what a value function captures.

In Lesson 1.3 we said the agent's true goal is the return: the total reward collected over time, not the single reward of one step. The trouble is, the agent has to act now, with the full return still in the future. It can't wait until the end of the song to find out whether the opening chord was a good idea.

So the agent learns a shortcut. For any situation, it estimates: "if things go the way I expect from here, roughly how much total reward am I looking at?" That estimate is the value. It's the agent's way of pricing the future into the present — a forecast of long-term worth attached to the here and now.

💡 The tell

Reward is what you get; value is what you expect. Reward is a single number that lands after an action. Value is a forecast of all the reward still to come. Confusing the two is the single most common mistake in RL — keep them apart.

DEFINITIONV(s) — the worth of a state

The first value function measures how good it is to simply be in a state. We write it V(s), read "vee of ess," and call it the state value.

📖 Value function · V(s)

The value function captures how good a situation (state) is in terms of long-term reward — not just the immediate reward. The value of a state s is written V(s), the state value: roughly, the total reward the agent can expect to collect starting from s and behaving well from there on.

A high V(s) means "this is a promising place to be — good things are likely from here." A low V(s) means "not much reward left to gather here." Crucially, both judgements look past the immediate moment. A state can hand you nothing right now yet still have a high value, because it sets up big rewards a few steps later. V(s) is the agent's appraisal of a whole position, the way a chess player feels a board is "winning" long before checkmate.

INTUITIONThe cleaning robot's house

Let's make this concrete with a friendly little robot. Picture a house with three rooms: a kitchen, a living room, and a bedroom. Each room can be in one of two conditions — clean or dirty. Our robot earns a reward every time it cleans a dirty room, and its goal, as always, is to maximise its total reward over time.

A state here is "which room the robot is in, and what's dirty." For any such state, V(s) tells the robot the overall goodness of being there — assuming it goes on to behave optimally. Two examples bring it to life:

🤖🍳 s₁ · robot in dirty kitchen lots of reward still to earn V(s₁) = 12.5 🤖🛋️ s₂ · robot in clean living room little reward left here V(s₂) = 4
Same robot, same reward function — but the dirty kitchen is the more valuable place to stand, because more reward can be earned from it.

Say state s1 is "robot standing in a dirty kitchen." There's a lot of cleaning to be done, which means a lot of reward still available — so V(s1) = 12.5, a high value. Now take s2, "robot standing in a clean living room." Only small touch-ups are left right where it is, so the prospects are slimmer and the value is lower: V(s2) = 4.

Here is where those two numbers come from. Each job pays its reward once: in the kitchen, picking up the trash pays +10 and sweeping the floor +5; in the living room, dusting the shelves pays +3 and tidying the cushions +2. The robot does one job per step, and a reward one step away counts half, a discount of γ = 0.5 (Lesson 1.3). In the kitchen the best plan is the trash first, then the floor: 10 + 0.5·5 = 12.5. In the living room it is the shelves, then the cushions: 3 + 0.5·2 = 4.

Notice the robot hasn't earned anything yet in either state — these numbers describe potential. That's the whole point of value: it scores a situation by its future, not by what just landed in the robot's pocket.

DEFINITIONQ(s,a) — the worth of an action

V(s) tells the robot how good a place is — but it doesn't tell it what to do there. Standing in the dirty kitchen, the robot still has choices: pick up the trash, sweep the floor, or wander off. To choose well, it needs to score each action, not just the room. That's the job of the second value function.

📖 Action-value function · Q(s,a)

The action-value function Q(s,a) — read "queue of ess, ay" — measures how good it is to take a specific action a while in a specific state s, again counting long-term reward. It's the value of a state paired with a particular move.

Back in the kitchen, the robot sees some trash on the floor and weighs its options. Each option has a Q-value — a forecast of the total reward that follows if it commits to that move and then keeps behaving well:

StateAction aQ(s,a)What it means
kitchen (trash on floor)pick up trash+12.5clears the mess → biggest payoff
kitchen (trash on floor)sweep floor+10helps, but the trash is still there
kitchen (trash on floor)leave the room0does nothing useful here

Each Q-value is the reward for that first move plus the discounted reward that follows it. Pick up the trash first and the robot earns 10 now and the 5 for sweeping one step later: 10 + 0.5·5 = 12.5. Sweep first and it is the big reward that gets halved: 5 + 0.5·10 = 10. Leaving ends the robot's work in the room, so it earns 0.

The robot doesn't agonise. It scans the three Q-values, finds the largest — Q(kitchen, pick up trash) = +12.5 — and takes that action. That largest Q-value is also the kitchen's V: the value of a state, for a robot that behaves well, is the Q-value of the best action there. Picking the action with the highest Q-value is so central it has its own name, the argmax (literally "the argument that maximises"): the action a for which Q(s,a) is greatest.

a* = argmaxa Q(s,a)choose the highest-value action

This single move — "look up the Q-values, take the biggest" — is the engine behind Q-learning, the algorithm we'll build in Lesson 2.6. If you can learn good Q-values, you barely need anything else to act well: the best policy falls right out of the argmax.

💡 V judges, Q chooses

V(s) rates a whole state; Q(s,a) rates each move within it. The agent uses Q to pick its next action (which option is best?) and uses V to judge how good its current situation is (am I in a strong position?). Same idea — long-term worth — measured at two different resolutions.

SEE ITThe cleaning robot's values

Here's the three-room house. Use the toggle to place the robot in a room — its current state. You'll see the state value V(s) for where it's standing, and the Q(s,a) of every action available there, drawn as bars. The robot picks the highest bar — the argmax. Watch which action lights up as you move it around. Every room follows the rules above (each job pays once, γ = 0.5, leaving earns 0); the bedroom's jobs are making the bed (+8) and vacuuming the rug (+6).

🤖 Interactive · Where is the robot, and what will it do?

Pick the robot's room. V tells you how good that situation is; Q tells the robot which move to make — the brightest bar wins.

🍳 🛋️ 🛏️ Kitchen Living room Bedroom 🤖
Click a room or use the buttons below to move the robot.
Current state
Kitchen
State value V(s)
12.5
Robot will choose
pick up trash

The robot reads Q to choose, V to judge the state.

Try the clean living room: its V(s) is low — there's little left to gain — and every action's Q-value is modest. The dirty kitchen and bedroom are far more valuable states, and in each the robot zeroes in on the action that clears the most mess. That's value functions doing their two jobs at once: rating the situation and steering the choice.

FORMALValue is just expected return

Now the precise version, and it's beautifully short. Remember the return Gt from Lesson 1.3 — the discounted total of all future rewards from time t onward. The value of a state is simply the expected return if you start there:

V(s) = 𝔼[ Gt | St = s ]state value = expected return from state s

Read it plainly: "V(s) is the average total reward I'd collect, over all the ways the future could unfold, given that I'm in state s right now." The 𝔼 (the expectation, just a fancy "average") is there because the future is uncertain — different episodes play out differently — so value is the typical outcome, not a guarantee.

The action value is the same idea, with one extra condition pinned down — that you take action a first:

Q(s,a) = 𝔼[ Gt | St = s, At = a ]action value = expected return from s after taking a

So both V and Q are expected returns. The only difference is what's fixed: V fixes the state, while Q fixes the state and the first action. That's why the robot's Q(kitchen, pick up trash) = +12.5 is a forecast — "+12.5 of discounted reward in total if I start by grabbing that trash, then play on well." Value, all the way down, is the return you expect.

CORE CONCEPTValue depends on the policy

One subtlety to tuck away. We keep saying "assuming the robot behaves well" — and that assumption is doing real work. Value is never an absolute property of a state; it depends on how the agent will act from there. That behaviour is the policy π from Lesson 2.2.

A brilliant policy squeezes lots of reward out of a state, so its value is high. A lazy policy that ignores the trash earns little, so the same state is worth less under it. To make this explicit, we tag value with the policy it assumes — Vπ(s) and Qπ(s,a) — meaning "the value of s when following policy π." When we quote the value under the best possible policy, that's the optimal value, written V*(s) and Q*(s,a) — and those are the numbers the robot really wants to learn.

🔑 Tie it together

Value is a forecast of return, and a forecast only makes sense given a plan. Change the plan (the policy) and the value changes. So whenever you see a value, ask "under which policy?" — usually the answer is "the best one we can find," and that quest for the optimal values is exactly what the rest of Stage 2 is about.

That raises an obvious question: where do these numbers actually come from? We worked them out by hand for the robot, because its house is tiny and we wrote the rewards ourselves, but a real agent has to compute them. The key realisation is that the value of a state is tied to the values of the states it leads into — a self-referential relationship. Pinning that down gives us the most important equation in all of RL, the Bellman equation, which is next.

WEAKNESSCommon mistakes to avoid

✗ "Value = the immediate reward."

No — value is the expected return, the discounted total of all future reward, not the single reward of the next step. A state can pay nothing right now yet have a high value because it sets up big rewards later. Reward is what you get; value is what you expect.

✗ "Value is fixed, regardless of policy."

Value always assumes a way of behaving. The same state is worth more under a good policy than a bad one — that's why we write Vπ and Qπ. When people say "the value of s," they almost always mean "under the optimal policy," V*(s).

✗ "You always need both Q and V — they're the same thing."

They're related but not interchangeable. V(s) scores a state; Q(s,a) scores an action in a state. For choosing, Q is what you need (take the argmax). Many algorithms — Q-learning included — learn only Q and never store V at all.

KEY INSIGHTWhere this connects in your course

  • The return (1.3) — value is the expected return; everything here rests on Gt and the discount γ.
  • Policy (2.2) — value is always measured under a policy π; change the policy and the values change.
  • The Bellman equation (2.4) — how a state's value links to the values of the next states, turning value into something we can actually compute.
  • Q-learning (2.6) — learning Q(s,a) from experience, then acting greedily by argmax — the algorithm this whole lesson sets up.

Hold the pair in your head — V judges the state, Q chooses the action, and both are just expected return — and the rest of Stage 2 is the story of how an agent learns those numbers for itself.

GLOSSARYKey terms introduced

Value function (V(s))
A measure of how good a state is in terms of long-term reward — the expected return from that state.
Action-value function (Q(s,a))
How good it is to take action a in state s — the expected return after that action.
State value
Another name for V(s): the worth of being in a state, assuming the agent behaves well from there.
Expected return
The average, over all possible futures, of the discounted total reward — what every value function estimates.
argmax
"The argument that maximises" — the action with the highest Q-value; how the agent turns values into a choice.
Optimal value (V*, Q*)
The value of a state or action under the best possible policy — the numbers the agent ultimately wants to learn.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1In one sentence, what's the difference between V(s) and Q(s,a)?›

V(s) says how good a state is; Q(s,a) says how good a specific action is within that state. The agent reads Q to choose what to do and V to judge where it is.

2Why can a state with zero immediate reward still have a high value?›

Because value is the expected return — the total of all future reward, not the next step. A state that pays nothing now but leads to big rewards soon (like the robot's dirty kitchen) has a high value because of what comes next.

3The robot in the kitchen sees Q-values of +12.5 (pick up trash), +10 (sweep), 0 (leave). Which does it take, and what's that rule called?›

It takes pick up trash — the highest Q-value (+12.5). Choosing the action with the largest Q(s,a) is the argmax, and it's the core of how Q-learning acts.

4Why do we write Vπ(s) instead of just V(s)?›

Because value depends on the policy π the agent follows. The same state is worth more under a good policy than a bad one, so the superscript names which behaviour the value assumes. Under the best policy it becomes the optimal value V*(s).

Next: how values link to each other — the Bellman equation →