ProjekinAja. Reinforcement Learning Stage 2 · How the Agent Learns Lesson 2.2
● Stage 2 · Lesson 2.2

Policy — The Agent's Brain

The policy is the agent's strategy — its rulebook for what to do in every situation. It starts out random, then experience refines it into something that looks a lot like skill.

⏱ ~13 min read 🧠 The decision-maker 🎯 Core concept

🎯 By the end you'll understand

What a policy is — the agent's internal strategy for choosing actions — and how to read it as a mapping π(a|s) from states to actions. You'll see how a policy starts random and gets refined through feedback, and you'll know the difference between a deterministic and a stochastic policy.

KEY INSIGHTThe one idea to take away

A policy is the agent's brain: a rule that says, "when you're in this situation, do that." Everything the agent has learned lives inside it.

In Lesson 2.1 you saw the agent and the environment trading actions and rewards back and forth. But what actually decides which action to take? That decision-maker is the policy. It's the part of the agent that turns "here's my situation" into "here's what I'll do" — and it's the thing that gets better as the agent learns.

🔑 Hold onto this

A policy is a mapping from states to actions. Written π(a|s), it answers one question over and over: given the state I'm in, which action should I take? Learning, in RL, is the slow improvement of that mapping.

CORE CONCEPTThe agent's brain

Think of the policy as the agent's internal decision-making guide — honestly, it's a lot like our brain. When you face a situation, your brain doesn't roll dice; it draws on everything you've experienced and picks what to do. The policy plays exactly that role for an RL agent.

Formally, a policy is usually written π (the Greek letter pi), and it's a mapping from states to actions. That's the whole idea in five words. You give it a state — a snapshot of the current situation — and it hands you back an action. "When you're in this state, take this action." Do that at every step, and you've described the agent's entire behaviour.

📖 Policy π(a|s)

The agent's strategy for selecting actions: a function that maps each state s to an action a (or to a set of probabilities over actions). It encodes everything the agent has learned about how to behave.

Here's the part that makes it interesting: the policy isn't handed to the agent fully formed. It's grown. It starts off clumsy and arbitrary, and through interaction and feedback it gets shaped into a real strategy. The rest of this lesson is about that journey — from a random rulebook to a good one.

INTUITIONBuilding a policy at the chessboard

Picture the very first game of chess you ever played. You barely knew how the pieces moved, so your choices were essentially random moves — push a pawn here, swing the queen out there, no real plan behind any of it. That scattershot set of choices was your first policy. A bad one, but a policy all the same.

Now play a hundred more games. Slowly, you start noticing patterns. Controlling the centre tends to go well. Developing your pieces early leaves you better placed. Out of those patterns you build little mental guidelines: don't bring the queen out too early; develop your knights and bishops before your rooks.

That growing collection of guidelines is your policy. Nobody handed you a textbook answer for every position — you distilled the rules yourself from the rewards and punishments of winning and losing. The chess master's policy and the beginner's policy are the same kind of object: a state-to-action mapping. The master's has just been refined by far more experience.

💡 The tell

Whenever you can phrase your behaviour as "in situation X, I do Y," you're describing a policy. Skill isn't a different thing from a policy — skill is just a better policy.

EXAMPLECrossing the street

Let's make the state → action mapping crisp with the simplest example there is. You're standing at a crossing, and the pedestrian signal is red.

Read it left to right and you can see the policy doing its one job: it takes the state (red light) and produces the action (wait). Change the state to "green light" and the same policy produces a different action (walk). The policy is the rule sitting in the middle, translating where-you-are into what-you-do.

State: 🔴 red signal is red State: 🟢 green signal is green π the policy Action: ⏸ wait Action: 🚶 walk
A policy sits between states and actions: feed it a state, it returns an action. Same rule, different state, different action.

INTUITIONLearning a policy on the bike

Back to your little sister and her bike. Remember, no one tells her how to ride — she has to build her own policy out of trial and error. And a policy, for her, is just a set of answers to "what do I do when I feel like this?"

Her states are the situations she keeps running into, and for each one she needs an action:

At the start she has no good policy. When she feels unsteady she pedals too slowly and tips over; when she meets a turn she steers too sharply and over-corrects. Each fall is a −1, each smooth metre a +1 🍌 — and that feedback is what reshapes the mapping. With practice, "unsteady → pedal slowly" gets replaced by "unsteady → pedal faster," and "turn → yank the bars" becomes "turn → slow and lean." Her policy evolves.

State sEarly policy (poor)Refined policy (good)
feels unsteadypedal slowly & freezepedal faster to balance
approaching a turnturn sharply, hit the brakesslow down, lean gently in
bumpy path aheadloosen grip, tense upgrip tight, hold steady

Read each row as one entry in her rulebook: state on the left, the action π chooses on the right. Learning to ride a bike is rewriting the right-hand column until it earns the most bananas.

SEE ITWatch a policy evolve

Here's the whole arc in one place: an agent learning the street-crossing-style task across several episodes. It begins with a random policy — mostly wrong actions, low return — and with each episode the feedback rewrites the rulebook toward the right actions and a rising return. Step through and watch the random → refined journey happen.

🧠 Interactive · A policy refining itself, episode by episode

Press ▶ Next episode. Each episode the agent re-reads its rulebook for four states and tallies the return. Watch wrong actions turn right. The bar under the totals lets you pause, step back or drag between episodes.

Episode 0 · the policy hasn't started learning yet.

State sPolicy π(s) says…Outcome
🔴 red light——
🟢 green light——
😬 unsteady——
↩ sharp turn——
Right actions this episode
0 / 4
Episode return
—

A brand-new agent guesses. Press ▶ Next episode to start refining.

Nothing in there is magic. Each episode, one more entry in the mapping flips from a wrong action to the right one, the return climbs, and what started as a random policy turns into a refined one. That climb — arbitrary moves giving way to a strategy that maximises cumulative reward — is the entire story of learning in RL. (How the agent knows which action is better is the job of value functions, next lesson.)

COMPARISONDeterministic vs stochastic policies

Policies come in two flavours, and the difference is just how the mapping answers.

A deterministic policy maps each state to exactly one action — no hesitation. We write it a = π(s): same state in, same action out, every time. "Red light → wait" is deterministic. There's a single arrow leaving each state.

A stochastic policy maps each state to a probability distribution over actions — that's exactly what the notation π(a|s) means: "the probability of choosing action a given state s." In a state it might be 70% pedal faster, 20% coast, 10% brake. Roll the (metaphorical) dice and act accordingly.

π(a|s) = P[ At = a | St = s ]a stochastic policy: a distribution over actions

Why ever be random on purpose? Because a stochastic policy lets the agent explore — try actions it isn't sure about so it can discover something better — and it's the natural choice when the environment is uncertain or partly hidden. A deterministic policy commits; a stochastic one keeps its options open. A deterministic policy is really just the special case where one action gets probability 1.

DeterministicStochastic
Maps a state toone actiona distribution over actions
Notationa = π(s)π(a|s)
Behavioursame state → same actionsame state → maybe different actions
Good whenenvironment is clear & settledyou need to explore / it's uncertain

FORMALA formal definition

You now have the picture, so here's the precise version, all in one breath.

📖 Policy, formally

A policy π is a mapping from states to actions (deterministic) — or to a probability distribution over actions (stochastic) — that defines the agent's behaviour. A deterministic policy gives a = π(s); a stochastic policy gives π(a|s) = P[At=a | St=s]. The aim of learning is to find a policy that maximises expected cumulative reward (the return).

The policy starts as a random policy — arbitrary choices — and through interaction and feedback shifts toward a refined strategy that earns the most reward over time. That shift has a name we'll use constantly in this stage: policy improvement. Every algorithm ahead — Q-learning, SARSA, policy gradients — is ultimately a recipe for nudging π from random toward optimal.

🔑 Key Takeaway

A policy is a decision-making guide, shaped by experience, that helps an RL agent choose actions that maximise rewards and minimise penalties.

WEAKNESSCommon mistakes to avoid

✗ "A policy is a fixed list of moves."

It isn't a script you run start to finish. A policy reacts to the current state — give it a different situation and it gives a different action. It's a rule (state → action), not a recording. The agent never plays back a sequence; it consults the rulebook anew at every step.

✗ "A good policy is memorised, not learned."

The whole point is that the agent discovers the mapping from rewards — like you distilling chess guidelines from games you played, not from a list you were handed. A useful policy also has to generalise to states it hasn't seen exactly before, which memorising can't do.

✗ "Deterministic is always better than stochastic."

Not so. Randomness is often essential: it lets the agent explore instead of locking onto the first decent action, and in uncertain or partly-hidden environments a stochastic policy can genuinely outperform any deterministic one. The best policy depends on the problem.

KEY INSIGHTWhere this connects in your course

  • What RL is (1.1) — the policy is the agent's strategy inside the trial-and-error loop you first met there.
  • Interaction & feedback (2.1) — the reward signal from each interaction is exactly what reshapes the policy episode after episode.
  • Value functions: V and Q (2.3) — next we ask how the agent judges which action is better, so it knows which way to nudge π.
  • Q-learning (2.6) — a concrete algorithm that learns action-values and reads off a better policy from them, automating the random → refined journey.

Hold the one sentence — a policy maps states to actions, and learning is improving that map — and the rest of this stage is just making it precise and making it work.

GLOSSARYKey terms introduced

Policy (π)
The agent's strategy — its decision-making guide for choosing an action in each state.
State → action mapping
The core form of a policy: feed it a state, it returns an action (or a distribution over actions).
Deterministic policy (a = π(s))
Maps each state to exactly one action; same state always yields the same action.
Stochastic policy (π(a|s))
Maps each state to a probability distribution over actions; same state may yield different actions.
Random policy
The untrained starting point — actions chosen arbitrarily, before any learning.
Policy evolution / improvement
The process by which feedback reshapes a policy from random toward one that maximises cumulative reward.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1In one sentence, what is a policy?›

A policy is the agent's strategy — a mapping from states to actions, π(a|s), that says which action to take in each situation. It's the agent's brain, shaped by experience.

2Map the street-crossing example onto state, policy, and action.›

State = the pedestrian signal is red; policy = cross only when it's green; action = wait now, walk when it turns green. The policy is the rule turning the state into the action.

3What's the difference between a deterministic and a stochastic policy?›

A deterministic policy maps each state to exactly one action (a = π(s)) — same state, same action. A stochastic policy maps a state to a probability distribution over actions (π(a|s)) — same state may give different actions, which helps the agent explore.

4How does a policy go from random to good?›

Through interaction and feedback. It starts as a random policy; rewards and penalties from each episode reshape the state → action mapping, replacing wrong actions with better ones until the policy maximises cumulative reward. That's policy improvement — exactly like distilling chess guidelines from games you played.

You've met the agent's brain. Next: how it judges which action is best →