Policy — The Agent's Brain
The policy is the agent's strategy — its rulebook for what to do in every situation. It starts out random, then experience refines it into something that looks a lot like skill.
🎯 By the end you'll understand
What a policy is — the agent's internal strategy for choosing actions — and how to read it as a mapping π(a|s) from states to actions. You'll see how a policy starts random and gets refined through feedback, and you'll know the difference between a deterministic and a stochastic policy.
KEY INSIGHTThe one idea to take away
A policy is the agent's brain: a rule that says, "when you're in this situation, do that." Everything the agent has learned lives inside it.
In Lesson 2.1 you saw the agent and the environment trading actions and rewards back and forth. But what actually decides which action to take? That decision-maker is the policy. It's the part of the agent that turns "here's my situation" into "here's what I'll do" — and it's the thing that gets better as the agent learns.
A policy is a mapping from states to actions. Written π(a|s), it answers one question over and over: given the state I'm in, which action should I take? Learning, in RL, is the slow improvement of that mapping.
CORE CONCEPTThe agent's brain
Think of the policy as the agent's internal decision-making guide — honestly, it's a lot like our brain. When you face a situation, your brain doesn't roll dice; it draws on everything you've experienced and picks what to do. The policy plays exactly that role for an RL agent.
Formally, a policy is usually written π (the Greek letter pi), and it's a mapping from states to actions. That's the whole idea in five words. You give it a state — a snapshot of the current situation — and it hands you back an action. "When you're in this state, take this action." Do that at every step, and you've described the agent's entire behaviour.
The agent's strategy for selecting actions: a function that maps each state s to an action a (or to a set of probabilities over actions). It encodes everything the agent has learned about how to behave.
Here's the part that makes it interesting: the policy isn't handed to the agent fully formed. It's grown. It starts off clumsy and arbitrary, and through interaction and feedback it gets shaped into a real strategy. The rest of this lesson is about that journey — from a random rulebook to a good one.
INTUITIONBuilding a policy at the chessboard
Picture the very first game of chess you ever played. You barely knew how the pieces moved, so your choices were essentially random moves — push a pawn here, swing the queen out there, no real plan behind any of it. That scattershot set of choices was your first policy. A bad one, but a policy all the same.
Now play a hundred more games. Slowly, you start noticing patterns. Controlling the centre tends to go well. Developing your pieces early leaves you better placed. Out of those patterns you build little mental guidelines: don't bring the queen out too early; develop your knights and bishops before your rooks.
That growing collection of guidelines is your policy. Nobody handed you a textbook answer for every position — you distilled the rules yourself from the rewards and punishments of winning and losing. The chess master's policy and the beginner's policy are the same kind of object: a state-to-action mapping. The master's has just been refined by far more experience.
Whenever you can phrase your behaviour as "in situation X, I do Y," you're describing a policy. Skill isn't a different thing from a policy — skill is just a better policy.
EXAMPLECrossing the street
Let's make the state → action mapping crisp with the simplest example there is. You're standing at a crossing, and the pedestrian signal is red.
- State — the pedestrian signal is red.
- Policy — cross only when the light is green.
- Action — so right now, you wait; when the light turns green, you walk.
Read it left to right and you can see the policy doing its one job: it takes the state (red light) and produces the action (wait). Change the state to "green light" and the same policy produces a different action (walk). The policy is the rule sitting in the middle, translating where-you-are into what-you-do.
INTUITIONLearning a policy on the bike
Back to your little sister and her bike. Remember, no one tells her how to ride — she has to build her own policy out of trial and error. And a policy, for her, is just a set of answers to "what do I do when I feel like this?"
Her states are the situations she keeps running into, and for each one she needs an action:
- Feeling unsteady / wobbly → pedal a little faster to gain balance.
- Approaching a turn → slow down and lean gently into it.
- Riding over a bumpy patch → grip the handlebars tight and hold steady.
At the start she has no good policy. When she feels unsteady she pedals too slowly and tips over; when she meets a turn she steers too sharply and over-corrects. Each fall is a −1, each smooth metre a +1 🍌 — and that feedback is what reshapes the mapping. With practice, "unsteady → pedal slowly" gets replaced by "unsteady → pedal faster," and "turn → yank the bars" becomes "turn → slow and lean." Her policy evolves.
| State s | Early policy (poor) | Refined policy (good) |
|---|---|---|
| feels unsteady | pedal slowly & freeze | pedal faster to balance |
| approaching a turn | turn sharply, hit the brakes | slow down, lean gently in |
| bumpy path ahead | loosen grip, tense up | grip tight, hold steady |
Read each row as one entry in her rulebook: state on the left, the action π chooses on the right. Learning to ride a bike is rewriting the right-hand column until it earns the most bananas.
SEE ITWatch a policy evolve
Here's the whole arc in one place: an agent learning the street-crossing-style task across several episodes. It begins with a random policy — mostly wrong actions, low return — and with each episode the feedback rewrites the rulebook toward the right actions and a rising return. Step through and watch the random → refined journey happen.
Nothing in there is magic. Each episode, one more entry in the mapping flips from a wrong action to the right one, the return climbs, and what started as a random policy turns into a refined one. That climb — arbitrary moves giving way to a strategy that maximises cumulative reward — is the entire story of learning in RL. (How the agent knows which action is better is the job of value functions, next lesson.)
COMPARISONDeterministic vs stochastic policies
Policies come in two flavours, and the difference is just how the mapping answers.
A deterministic policy maps each state to exactly one action — no hesitation. We write it a = π(s): same state in, same action out, every time. "Red light → wait" is deterministic. There's a single arrow leaving each state.
A stochastic policy maps each state to a probability distribution over actions — that's exactly what the notation π(a|s) means: "the probability of choosing action a given state s." In a state it might be 70% pedal faster, 20% coast, 10% brake. Roll the (metaphorical) dice and act accordingly.
Why ever be random on purpose? Because a stochastic policy lets the agent explore — try actions it isn't sure about so it can discover something better — and it's the natural choice when the environment is uncertain or partly hidden. A deterministic policy commits; a stochastic one keeps its options open. A deterministic policy is really just the special case where one action gets probability 1.
| Deterministic | Stochastic | |
|---|---|---|
| Maps a state to | one action | a distribution over actions |
| Notation | a = π(s) | π(a|s) |
| Behaviour | same state → same action | same state → maybe different actions |
| Good when | environment is clear & settled | you need to explore / it's uncertain |
FORMALA formal definition
You now have the picture, so here's the precise version, all in one breath.
A policy π is a mapping from states to actions (deterministic) — or to a probability distribution over actions (stochastic) — that defines the agent's behaviour. A deterministic policy gives a = π(s); a stochastic policy gives π(a|s) = P[At=a | St=s]. The aim of learning is to find a policy that maximises expected cumulative reward (the return).
The policy starts as a random policy — arbitrary choices — and through interaction and feedback shifts toward a refined strategy that earns the most reward over time. That shift has a name we'll use constantly in this stage: policy improvement. Every algorithm ahead — Q-learning, SARSA, policy gradients — is ultimately a recipe for nudging π from random toward optimal.
A policy is a decision-making guide, shaped by experience, that helps an RL agent choose actions that maximise rewards and minimise penalties.
WEAKNESSCommon mistakes to avoid
It isn't a script you run start to finish. A policy reacts to the current state — give it a different situation and it gives a different action. It's a rule (state → action), not a recording. The agent never plays back a sequence; it consults the rulebook anew at every step.
The whole point is that the agent discovers the mapping from rewards — like you distilling chess guidelines from games you played, not from a list you were handed. A useful policy also has to generalise to states it hasn't seen exactly before, which memorising can't do.
Not so. Randomness is often essential: it lets the agent explore instead of locking onto the first decent action, and in uncertain or partly-hidden environments a stochastic policy can genuinely outperform any deterministic one. The best policy depends on the problem.
KEY INSIGHTWhere this connects in your course
- What RL is (1.1) — the policy is the agent's strategy inside the trial-and-error loop you first met there.
- Interaction & feedback (2.1) — the reward signal from each interaction is exactly what reshapes the policy episode after episode.
- Value functions: V and Q (2.3) — next we ask how the agent judges which action is better, so it knows which way to nudge π.
- Q-learning (2.6) — a concrete algorithm that learns action-values and reads off a better policy from them, automating the random → refined journey.
Hold the one sentence — a policy maps states to actions, and learning is improving that map — and the rest of this stage is just making it precise and making it work.
GLOSSARYKey terms introduced
- Policy (π)
- The agent's strategy — its decision-making guide for choosing an action in each state.
- State → action mapping
- The core form of a policy: feed it a state, it returns an action (or a distribution over actions).
- Deterministic policy (a = π(s))
- Maps each state to exactly one action; same state always yields the same action.
- Stochastic policy (π(a|s))
- Maps each state to a probability distribution over actions; same state may yield different actions.
- Random policy
- The untrained starting point — actions chosen arbitrarily, before any learning.
- Policy evolution / improvement
- The process by which feedback reshapes a policy from random toward one that maximises cumulative reward.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1In one sentence, what is a policy?›
A policy is the agent's strategy — a mapping from states to actions, π(a|s), that says which action to take in each situation. It's the agent's brain, shaped by experience.
2Map the street-crossing example onto state, policy, and action.›
State = the pedestrian signal is red; policy = cross only when it's green; action = wait now, walk when it turns green. The policy is the rule turning the state into the action.
3What's the difference between a deterministic and a stochastic policy?›
A deterministic policy maps each state to exactly one action (a = π(s)) — same state, same action. A stochastic policy maps a state to a probability distribution over actions (π(a|s)) — same state may give different actions, which helps the agent explore.
4How does a policy go from random to good?›
Through interaction and feedback. It starts as a random policy; rewards and penalties from each episode reshape the state → action mapping, replacing wrong actions with better ones until the policy maximises cumulative reward. That's policy improvement — exactly like distilling chess guidelines from games you played.