SARSA & On-/Off-Policy
Q-learning's close cousin — and the key distinction it reveals: does the agent learn about the policy it's actually following, or the best one it could?
🎯 By the end you'll understand
The SARSA update and how a single symbol separates it from Q-learning, the difference between on-policy and off-policy learning, why SARSA learns a safer path along a cliff than Q-learning does, and how to pick the right one for your problem.
KEY INSIGHTThe one idea to take away
SARSA learns how good the policy you're actually following is — mistakes and all. Q-learning learns how good the best policy could be, no matter how recklessly you explore.
In Lesson 2.6 you met Q-learning, which quietly assumes that next step you'll do the smartest thing possible. SARSA drops that assumption. It updates using the action you really take next — wobbles, exploratory detours and all. That tiny change splits all of value-based RL into two camps, and gives us the single most important vocabulary word in the field: on-policy versus off-policy.
Q-learning's update looks at maxa Q(St+1,a) — the best next action. SARSA's update looks at Q(St+1,At+1) — the action it's about to actually take. That swap is the whole lesson.
CORE CONCEPTMeet SARSA
Imagine you're learning to drive, and a cautious instructor is sitting beside you. Q-learning's instructor is an optimist: every time you make a move, they grade it as if your next move will be flawless. SARSA's instructor is a realist: they grade your current move based on what you're genuinely going to do next — including the nervous over-correction you're already starting to make. Same lesson, a more honest grader.
The name SARSA is just a roll-call of the five things one update needs, in order: the State you're in, the Action you took, the Reward you got, the next State you landed in, and the next Action you're going to take. State, Action, Reward, State, Action — S, A, R, S′, A′. Every letter is a real quantity the algorithm waits to observe before it learns.
That last letter is the kicker. Q-learning never needs to know which action you'll actually pick next — it just imagines the best one. SARSA insists on watching your real next choice before it updates. Here is its rule:
It's the same machinery as every temporal-difference update you've already seen: a learning rate α nudges the old estimate toward a fresh TD target, Rt+1 + γ Q(St+1,At+1). The bracket is the TD error — the gap between what we expected and what the next step actually suggests. Nothing exotic. The only special thing is which Q-value sits inside the target.
Be in St → pick At (using your usual ε-greedy policy) → get reward Rt+1 and next state St+1 → pick the next action At+1 first → only then update Q(St,At) using that real At+1. You commit to your next move before you grade your last one.
FORMALOne symbol that changes everything
Let's put the two updates shoulder to shoulder. They are almost identical — read them as a "spot the difference" puzzle, because there is exactly one difference, and the entire chapter hangs on it.
Everything outside the highlight is the same. The difference is the target's last term:
- SARSA plugs in Q(St+1,At+1) — the value of the action it's genuinely about to take, which sometimes is a random exploratory move.
- Q-learning plugs in maxa Q(St+1,a) — the value of the best action available next, whether or not the agent ends up taking it.
Notice what this means. If your policy is purely greedy — always picking the max — then At+1 is the argmax, and the two updates collapse into one. The difference only ever shows up because real agents explore (the ε-greedy idea from Lesson 2.8). It's exploration that drives a wedge between "the action I take" and "the best action," and SARSA is the algorithm that takes that wedge seriously.
DEFINITIONOn-policy vs off-policy
That wedge has a name. An algorithm is on-policy if it learns about the very policy it is using to choose actions. It is off-policy if it learns about one policy while behaving according to a different one.
To say this cleanly we name two policies:
Behaviour policy
The policy the agent actually uses to act right now — usually exploratory (ε-greedy), so it tries random things sometimes.
Target policy
The policy the agent is trying to learn the value of — the one its Q-values describe.
SARSA is on-policy. Its behaviour policy and its target policy are the same ε-greedy policy. By using the real At+1, it folds its own exploration into the values it learns. It answers the question: "How good is what I'm really doing, exploratory wobbles included?"
Q-learning is off-policy. It behaves with an exploratory ε-greedy policy, but the max in its target means it learns the values of the greedy optimal policy regardless of how it explores. Behaviour and target come apart. It answers: "How good is the best I could possibly do?"
On-policy: learn the value of the policy you're following (behaviour = target). Off-policy: learn the value of a different, usually optimal, policy while following an exploratory one (behaviour ≠ target). Off-policy methods can learn the best plan from messy, even random, experience — at the cost of ignoring the danger their own exploration creates.
EXAMPLEThe cliff-walking story
Here's the classic problem that makes the distinction vivid — and it's the canonical example for a reason. Picture a grid. You start in the bottom-left corner and need to reach a goal in the bottom-right corner. Every step costs you −1 (so you're nudged to be efficient). But the entire bottom edge between start and goal is a cliff: step onto it and you get a brutal −100 and are flung back to the start.
What's the optimal route? Hug the cliff edge — walk straight along the row just above the drop. It's the shortest path, so it costs the least −1-per-step penalty. Q-learning, ever the optimist, learns exactly this. Its max assumes that next step it will always make the perfect move, so it sees no danger in walking the very edge.
But during training, the agent still explores — every so often ε-greedy throws in a random step. On the cliff edge, one random step downward means −100 and a long fall. So Q-learning, while it learns the optimal path, racks up nasty falls along the way. It learns the best plan but suffers for it.
SARSA tells a different story. Because its update uses the action it actually takes next — including those occasional random steps — it feels the risk of exploring near the cliff. The −100 that exploration occasionally triggers leaks back into the Q-values of the cliff-edge cells, making them look genuinely worse. So SARSA learns to walk a safer path one row higher, away from the edge. It's a slightly longer route, but it almost never falls while learning.
SARSA's target includes Q(St+1,At+1), and At+1 is sometimes a random exploratory step. Near a cliff, that random step is dangerous, so the danger is baked into the value. Q-learning's max ignores exploration entirely, so it never accounts for its own missteps — it learns the optimal path and explores recklessly.
This is on-policy versus off-policy made physical. SARSA values the policy it's truly running (an explorer near a cliff = risky), so it steers clear. Q-learning values the ideal greedy policy (a perfect walker fears no edge), so it takes the edge.
SEE ITSARSA vs Q-Learning on the cliff
Same grid, same rewards, two algorithms. Flip the toggle to see the route each one settles on — and watch the update equation change to match. The start is bottom-left (▶), the goal is bottom-right (★), and the red strip is the cliff. Then press play to follow one training episode on that route, and watch what a single exploratory step costs each learner.
Same environment, same exploration, two answers. Q-learning's max makes it fearless; SARSA's real At+1 makes it cautious. Neither is "wrong" — they're optimising different things.
COMPARISONSide by side
The whole contrast in one view:
| SARSA | Q-learning | |
|---|---|---|
| Next-state term | Q(S′,A′) (action taken) | maxa Q(S′,a) (best action) |
| Policy type | On-policy | Off-policy |
| What it learns | value of the policy it follows (with exploration) | value of the optimal greedy policy |
| Behaviour vs target | same policy | different policies |
| Behaviour near risk | cautious — avoids the cliff | bold — walks the edge |
| While learning | fewer costly mistakes | more falls during exploration |
| Converges to | optimal as exploration → 0 | optimal regardless of exploration |
| Best when | safety matters during learning | you want the optimal policy, mistakes are cheap |
One subtlety worth a footnote: SARSA also converges to the optimal policy — but only if you gradually shrink exploration (ε → 0) as it learns. As the wobbles fade, "the policy it follows" and "the optimal policy" become the same thing, and SARSA's cautious detour straightens out into the cliff-edge optimum. Q-learning gets there even while still exploring full-tilt.
FRAMEWORKWhen to use which
The choice comes down to one question: do mistakes during learning actually hurt?
Exploration is costly or dangerous and the agent learns in the real world — a physical robot near stairs, a self-driving car, a process where a bad move breaks something expensive. You want behaviour that respects the risk of its own exploration while still learning. SARSA's caution is a feature, not a bug.
You want the optimal policy and exploration mistakes are cheap — a simulator, a game, anywhere you can fall off a thousand cliffs for free and only the final learned policy matters. Off-policy learning is also more flexible: it can learn the optimal policy from any source of experience, even logged data or another agent's actions, because it doesn't care how the data was generated. That flexibility makes off-policy the foundation for the deep RL methods in Stage 3.
A good mental shortcut: SARSA = "learn safely, behave the way you train"; Q-learning = "learn the best, however you stumble there."
WEAKNESSCommon mistakes to avoid
They share the same skeleton, but the one differing term — Q(S′,A′) versus maxa Q(S′,a) — produces genuinely different behaviour, like the safe-vs-edge cliff paths. They only coincide when there's no exploration. The difference is the whole point.
Finding the optimal policy isn't free. Off-policy methods can be less stable, and they ignore the cost of their own exploration — fine in a simulator, dangerous on real hardware. "Optimal in the end" is worth little if the agent destroys itself getting there. On-policy SARSA is often the right choice.
The opposite. SARSA must explore — its behaviour policy is ε-greedy, just like Q-learning's. "On-policy" doesn't mean greedy; it means it learns about the exploratory policy it's running, rather than pretending it will always act greedily next.
KEY INSIGHTWhere this connects in your course
- Temporal-difference learning (2.5) — both updates are TD updates: an old estimate nudged toward Rt+1 + γ·(next value). SARSA and Q-learning differ only in that "next value."
- Q-learning (2.6) — the off-policy method this lesson sits beside; the max in its target is exactly what makes it off-policy.
- Exploration vs exploitation (2.8) — exploration is what splits behaviour from target. Without it SARSA and Q-learning would be the same; with it, the on/off-policy distinction comes alive.
You now hold the master key to value-based RL: every method either learns about the policy it follows (on-policy) or a different one (off-policy). Hold the cliff picture — safe detour vs daring edge — and you'll never confuse the two again.
GLOSSARYKey terms introduced
- SARSA
- An on-policy TD control algorithm whose update uses Q(St+1,At+1) — the action actually taken next. Named for State, Action, Reward, State, Action.
- On-policy
- Learning the value of the same policy the agent uses to act (behaviour policy = target policy).
- Off-policy
- Learning the value of one policy (usually the optimal greedy one) while acting under a different, exploratory policy.
- Behaviour policy
- The policy the agent actually follows to choose actions — typically exploratory (ε-greedy).
- Target policy
- The policy whose value the agent is trying to learn — what its Q-values describe.
- Cliff walking
- A benchmark gridworld with a high-penalty cliff that exposes the difference between on- and off-policy learning: Q-learning hugs the edge, SARSA detours.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What is the single difference between the SARSA and Q-learning updates, and what does it spell out?›
The next-state term. SARSA uses Q(St+1,At+1) — the value of the action it actually takes next — while Q-learning uses maxa Q(St+1,a) — the value of the best next action. SARSA learns the value of the policy it's truly following (on-policy); Q-learning learns the optimal policy's value (off-policy).
2Define on-policy and off-policy using "behaviour policy" and "target policy."›
On-policy: behaviour policy = target policy — you learn the value of the policy you're using to act. Off-policy: behaviour ≠ target — you act with an exploratory policy but learn the value of a different (usually optimal greedy) one.
3In cliff walking, why does SARSA learn a path further from the cliff than Q-learning?›
Because SARSA's update includes the action it actually takes next, which is sometimes a random exploratory step. Near the cliff that step risks the −100, so the danger leaks into the cliff-edge Q-values and SARSA learns to avoid them. Q-learning's max assumes a perfect next move, so it never feels the risk of its own exploration and walks the edge.
4You're training a real robot that operates near a flight of stairs. SARSA or Q-learning, and why?›
SARSA. Exploration mistakes are physically costly — a fall could damage the robot. SARSA is on-policy, so it accounts for the risk of its own exploratory actions and learns cautious behaviour while training. Q-learning would learn the optimal route but explore recklessly, risking real falls along the way.