ProjekinAja. Reinforcement Learning Stage 2 · How the Agent Learns Lesson 2.8
● Stage 2 · Lesson 2.8

Exploration vs Exploitation

The fundamental dilemma of all learning: do you take the best thing you know, or try something new that might be better? Get the balance wrong and the agent either never learns — or never wins.

⏱ ~14 min read ⚖️ The core trade-off 🎰 Bandits & ε-greedy

🎯 By the end you'll understand

Why every learning agent faces a tug-of-war between exploiting what it already knows and exploring to learn more, why neither extreme works, and how the simple ε-greedy rule — and decaying ε — strikes a workable balance. You'll see exactly why a pure-greedy agent can lock onto a worse option forever.

KEY INSIGHTThe one idea to take away

To find the best, you sometimes have to skip the best you know — spend a little of today's reward buying information that pays off tomorrow.

An agent learning by trial and error is stuck with a genuine paradox. The only way to know which action is best is to try all of them enough times. But every time it tries a so-so action just to learn, it gives up the reward it could have collected from its current favourite. Acting on what you know and acting to learn more pull in opposite directions — and you can only do one per step.

🔑 Hold onto this

Exploit = cash in: take the best action you currently believe in. Explore = invest: try a different action to gather information that might reveal something better. You can't do both at once, yet you need both — pure exploiting goes stale, pure exploring never pays off.

INTUITIONThe restaurant you go to every Friday

Forget agents for a moment. It's Friday night and you're choosing where to eat. There's the place you know is good — you've been ten times, the noodles are reliably great. And there's the new spot that opened down the street. You have no idea if it's amazing or terrible.

Going to your reliable favourite is exploitation: you take the best option you currently know about and bank a good meal. Trying the new place is exploration: you gamble tonight's dinner on the chance of discovering something even better — a new favourite you'd never have found otherwise.

Here's the catch that makes this the whole lesson. If you always go to your old favourite, you'll never discover that the new place is twice as good. You'll be perfectly happy and quietly wrong for years. But if you always try somewhere new, you'll eat a lot of mediocre dinners and never settle into the good thing you found. The smart move is obvious in spirit: mostly enjoy your favourite, but every so often try something new — and lean toward exploring when you're new in town and know almost nothing.

💡 The tell

Any time the value of your options is unknown and must be discovered by trying them, you face the explore–exploit trade-off. It shows up in dating, investing, A/B testing, picking a route home — and in every RL agent that learns from experience.

CORE CONCEPTThe multi-armed bandit

RL researchers strip the restaurant problem down to its bare bones and call it the multi-armed bandit. Picture a row of slot machines (a "one-armed bandit" is old slang for a slot machine — so several of them is a multi-armed bandit). Each machine has its own hidden average payout. You don't know which one pays best. You get a fixed number of pulls. Which arm do you play, and how often?

To find the best machine you must sample each one enough times to estimate its payout — that's exploration. But every pull you spend testing a bad machine is a pull you didn't spend on the good one — so you also want to exploit the best machine you've found so far. The bandit is the explore–exploit dilemma in its purest form, with all the messy state and sequencing of full RL stripped away.

🎰 Arm A true avg = ? "the okay one" 🎰 Arm B true avg = ? "the best one" 🎰 Arm C true avg = ? "the worst one"
Three arms, three hidden average payouts. The labels are revealed here for the story — the agent doesn't know them. It only sees the reward it gets each time it pulls.

This is exactly the situation a full RL agent faces inside every state: standing at a state, it has a set of possible actions, each with an unknown long-term value Q(s,a), and it must decide whether to take its current-best action or try another to sharpen its estimates. Bandits are the cleanest possible sandbox for studying that choice.

WEAKNESSWhy pure exploiting — and pure exploring — both fail

It's tempting to think the agent should just always pick the action it currently believes is best. That's called the greedy strategy, and it has a fatal flaw.

Suppose Arm A pays out a modest +3 on your very first try, while you haven't touched Arm B yet. A pure-greedy agent now believes A is the best action it knows — because it's the only action it knows anything good about. So it pulls A again. And again. Its estimate of A keeps getting confirmed, and it never tries B — even though B's true average is a far better +8. The agent has locked onto a worse option and convinced itself it's optimal. In RL we say it got stuck in a local optimum: the best choice among the few it bothered to explore, but not the best overall.

✗ Pure exploitation

Locks onto the first "okay" option it stumbles into and stops learning. It can never discover a better action it never tried. Confident, efficient, and quietly wrong.

So maybe the fix is the opposite — keep trying everything, forever? That fails too. An agent that always picks a random action is a brilliant scientist who refuses to ever use what it learned. It builds perfect estimates of every arm and then keeps pulling the bad ones anyway, throwing away reward on every step. It learns what's best and never cashes in.

✗ Pure exploration

Never commits to the good action it discovered, so it leaves enormous reward on the table. Knowing the answer is worthless if you never act on it.

The answer can't be either extreme. It has to be a mixture: exploit most of the time, but explore just enough to keep discovering. The question is how to mix them — and the simplest good answer has a name.

FORMALThe ε-greedy rule

The standard, beautifully simple solution is ε-greedy (epsilon-greedy). It introduces one knob: a small probability ε of doing something random.

📖 ε-greedy action selection

On each step, with probability ε pick a random action (explore); otherwise — with probability 1 − ε — pick the greedy action, the one with the highest current estimated value (exploit).

At = a random action, with probability ε   ·   argmaxa Q(s,a), with probability 1 − ε ε-greedy policy

Here ε (the Greek letter epsilon) is just a small number between 0 and 1 — the exploration rate. A typical value is ε = 0.1, meaning the agent explores on roughly 1 step in 10 and exploits on the other 9. Turn the knob and you slide between the two failures we just met:

That tiny 10% of randomness is what guarantees the agent keeps testing arms it has written off — so it can catch a great option it underestimated early. It's the restaurant rule made precise: usually go to your favourite, but roll the dice now and then.

🔑 Why such a small ε?

Once estimates are roughly right, you want to collect reward, not keep testing. A small ε keeps a thin trickle of exploration alive — enough to never fully stop learning, cheap enough that exploiting still dominates.

SEE ITExplore or exploit?

Here's a 3-armed bandit with fixed, hidden average payouts: Arm A pays 3, Arm B pays 8 (the best!), Arm C pays 2. The agent doesn't know these — it only learns by pulling. Set the exploration rate ε, then step through pulls and watch the estimates fill in, which arm the agent currently believes is best, and the cumulative reward.

🎰 Interactive · A 3-armed bandit, stepped by hand

Set ε, then press ▶ Pull. At ε = 0 the agent greedily exploits whatever looked good first — and can get stuck on the wrong arm. Nudge ε up and watch it discover Arm B.

0.00
Arm A
—
0 pulls
Arm B
—
0 pulls
Arm C
—
0 pulls
Believed-best arm
—
Cumulative reward
0
Pulls taken
0

No pulls yet. Set ε and press ▶ Pull.

Try it twice. With ε = 0, the agent commits early — often to Arm A, whose first lucky-looking pull made it the greedy favourite — and never finds out Arm B is far better, so its cumulative reward grows slowly. Now reset, push ε to 0.20, and run again: those occasional random pulls eventually land on Arm B, its estimate climbs above the rest, and from then on the agent exploits the real best arm — earning noticeably more over the same number of pulls.

💡 Note on this demo

To stay fully reproducible, this widget is deterministic — both the per-arm payouts and the "is this step an explore step?" decision come from fixed schedules, not a random number generator. Real ε-greedy uses actual randomness, but the lesson it teaches is identical: a sprinkle of exploration finds the best arm; none of it leaves you stuck.

FRAMEWORKDecaying ε — explore early, exploit late

A fixed ε works, but it's a little dumb in one way: it explores just as much on step 10,000 as on step 1. Early on, the agent knows almost nothing, so exploration is incredibly valuable. Late on, its estimates are sharp and that same exploration is just wasted reward on arms it has already ruled out.

The fix is decaying ε (also called annealing ε): start with a high exploration rate and shrink it over time. Explore aggressively while you're ignorant; exploit more and more as your estimates firm up.

εt = max( εmin,   ε0 · dt ) a simple decay schedule (d < 1)

Read it plainly: ε starts at ε0 (say 1.0 — explore everything at first) and is multiplied by a decay factor d (say 0.99) every step, sliding toward — but never below — a small floor εmin (say 0.05). The floor matters: it keeps a thin stream of exploration alive forever, so the agent can still notice if the world changes under it.

Why this schedule works is intuitive once you see it as a budget. Information is most valuable when you have least of it. Spending your exploration budget early — when one good discovery reshapes everything — and conserving it later — when you mostly just want to collect — gets more total reward than spreading it evenly. It's the new-in-town rule again: explore the city in your first month, settle into favourites by your second year.

🔑 The shape to remember

High ε early (you know little → exploring pays) → low ε late (estimates are good → exploiting pays), with a small permanent floor so learning never fully stops.

CONTEXTOther ways to balance the trade-off

ε-greedy is the workhorse because it's dead simple and works well. But it's a little blunt — when it explores, it picks uniformly at random, treating a clearly-terrible arm the same as a promising-but-uncertain one. Smarter strategies exist. This course does not go deeper into them, but it's worth knowing their names now:

✨

Optimistic initialisation

Start every action's estimate too high. The agent is then "disappointed" into trying each one (reality pulls the inflated estimate down), guaranteeing early exploration with no randomness at all.

🌡️

Softmax / Boltzmann

Instead of "best or random," pick each action with a probability proportional to its estimated value. Better-looking actions are explored more often than clearly-bad ones — exploration that's graded, not blind.

🎯

UCB (Upper Confidence Bound)

Be "optimistic in the face of uncertainty": prefer actions that are either high-value or rarely-tried (so their estimate is uncertain). It explicitly explores what it's most unsure about.

All three chase the same goal as ε-greedy — explore enough to find the best, exploit enough to profit from it — just with more finesse about which non-greedy actions to try. For most of this course, plain ε-greedy is all you need.

CORE CONCEPTTying it back to learning

This isn't a side-topic — it's the missing ingredient from the algorithms you just learned. When Q-learning (2.6) and SARSA (2.7) run, the agent has to act in the environment to generate experience, and the policy it uses to act is almost always ε-greedy. Without exploration, those updates only ever see the actions the agent already favours, and learning stalls in a local optimum — the value tables get refined for a strategy that may not be the best one.

There's a subtler link too. Recall the on-policy / off-policy distinction from 2.7. SARSA is on-policy: it learns the value of the policy it actually follows — exploration and all — so it accounts for the cost of those occasional random moves (it learns a "safe" path that respects its own slip-ups). Q-learning is off-policy: it explores with ε-greedy but learns the value of the greedy optimal policy regardless. Same exploration mechanism, different thing learned from it. How you explore and what you learn are deeply intertwined.

💡 In one line

Q-learning and SARSA tell the agent how to update its values; ε-greedy tells it how to act while collecting the experience those updates feed on. You need both halves to actually learn.

WEAKNESSCommon mistakes to avoid

✗ "Just always exploit the best known action."

That's pure greedy (ε = 0). It locks onto the first decent action it finds and can never discover a better one it didn't try — a textbook local optimum. The agent ends up confidently sub-optimal.

✗ "More exploration is always better."

No — exploration costs reward. Past the point where your estimates are good, extra exploring just throws away payoff on actions you've already ruled out. The goal is the right balance, not maximum curiosity.

✗ "Set ε once and never change it."

A fixed ε wastes exploration late in training when you barely need it. Decaying ε — high early, low late, with a small floor — almost always earns more, because it spends its exploration budget when information is most valuable.

KEY INSIGHTHow it connects in your course

  • Q-learning (2.6) — uses an ε-greedy behaviour policy to gather experience, then learns the optimal greedy values off-policy. Exploration is what feeds its updates.
  • SARSA & on-/off-policy (2.7) — SARSA learns the value of its actual ε-greedy policy (exploration included); Q-learning learns the greedy policy despite exploring. The split hinges on how exploration is treated.
  • Reward shaping (4.2) — where the exploration problem comes back: when reward is too rare for random exploration to find, that lesson answers with more frequent hints, and names smarter exploration (curiosity, intrinsic motivation) as the other response.

Hold the one sentence — spend a little reward to buy information, then cash in — and ε-greedy, decaying ε, and every fancier scheme are just different ways of striking that balance.

GLOSSARYKey terms introduced

Exploration
Trying a non-best action to gather information that might reveal a better option.
Exploitation
Taking the action you currently believe is best, to maximise reward right now.
Greedy
The strategy of always picking the highest-estimated action (argmaxa Q(s,a)) — pure exploitation.
ε-greedy
Pick a random action with probability ε, otherwise the greedy action — a simple explore/exploit mix.
ε (exploration rate)
A small number in [0,1] setting how often the agent explores; e.g. 0.1 = explore ~10% of steps.
Epsilon decay (annealing)
Shrinking ε over time — explore a lot early, exploit more later, often with a small floor.
Multi-armed bandit
The minimal explore/exploit problem: several actions with unknown payouts; sample to find the best.
Local optimum
A choice that's best among the few options explored, but not the best overall — what pure greedy gets stuck in.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1In one sentence, what's the difference between exploration and exploitation?›

Exploitation means taking the action you currently believe is best to grab reward now; exploration means trying a different action to gain information that might lead to higher reward later. You can do only one per step, but you need both.

2Why can a pure-greedy (ε = 0) agent end up earning less than one that explores a bit?›

It commits to the first action that looked good and never tries the others — so if a better action exists that it underestimated (or never sampled), it never finds out. It gets stuck in a local optimum, confidently picking a worse option forever. A little exploration eventually discovers the truly best action.

3State the ε-greedy rule, and say what a typical ε value is and why it's small.›

With probability ε pick a random action (explore); otherwise pick the greedy highest-value action (exploit). A typical value is ε = 0.1 — small because once estimates are decent you mostly want to collect reward, keeping only a thin trickle of exploration so learning never fully stops.

4Why decay ε over time instead of keeping it fixed?›

Exploration is most valuable early, when the agent knows little and one discovery reshapes everything; it's mostly wasted later, when estimates are already good. Decaying ε spends the exploration budget when information is worth most and shifts toward exploiting as the agent learns — usually earning more total reward. A small floor keeps a little exploration alive in case the environment changes.

You now know how agents decide what to try. Next: how they learn from full episodes vs. single steps →