ProjekinAja. Reinforcement Learning Stage 2 · How the Agent Learns Lesson 2.1
● Stage 2 · Lesson 2.1

Interaction & Feedback

Zoom into the heartbeat of learning: the agent acts, the world answers with a reward, and that exchange — repeated over and over — is everything.

⏱ ~13 min read 🔁 The learning loop 🚦 Stage 2 begins

🎯 By the end you'll understand

What it really means for an agent to interact with an environment — observing a state and choosing an action — and how the environment's feedback (reward or penalty) shapes behaviour over time. You'll see why the agent chases the biggest cumulative reward, balancing the tempting payoff right now against the bigger one later.

KEY INSIGHTThe heartbeat of learning

Learning in RL isn't fed from a textbook of right answers — it's a loop: act, see what the world does back, adjust. That exchange between agent and environment is the whole engine.

In Stage 1 you met the cast — agent, environment, state, action, reward — and the idea that an agent learns by trial and error. Now we slow the film right down and watch the single frame that repeats forever: the agent acts, the environment responds, and a little number called the reward says "good" or "bad". No labelled data. No teacher. Just interaction and feedback, the same way you learned almost everything important in your life.

🔑 Hold onto this

Every RL system runs on one repeating exchange: the agent observes a state and takes an action; the environment returns a reward and a new state. Learning is what happens when that exchange repeats thousands of times and the agent slowly steers toward more reward.

CORE CONCEPTInteraction: observing, then acting

At every moment, the agent does two things in sequence. First it observes the state — the current situation, the context it finds itself in right now. Then, based on that state, it chooses an action that changes the environment. Observe, act. Observe, act. That back-and-forth is what we mean by interaction.

Picture a warehouse robot. Its environment is everything around it — the walls, the shelves, the obstacles scattered on the floor, the other robots trundling past. The robot can't change the warehouse by thinking about it; it changes things only by acting inside it.

Its state is a snapshot of its current situation: where it is on the floor, where the nearby obstacles are, and whether it's currently carrying a load. Notice how much richer that is than "just my position" — the state is the whole relevant picture the robot needs to decide well.

And its actions are the moves available to it: move forward, turn left, lift an object. Each action nudges the environment into a new configuration — the robot rolls forward and now faces a shelf; it lifts a box and is now carrying a load. Crucially, the action the agent takes changes the state it sees next. The agent isn't a passive observer; it's a participant shaping its own future situations.

📖 Interaction

The ongoing exchange in which the agent observes the state of the environment and takes an action that changes it — producing the next state. RL learning happens entirely inside this loop; there is no learning outside of interacting.

INTUITIONYou, behind the wheel

Here's the analogy I keep coming back to, because you've lived it: learning to drive. When you drive, you are the agent and the road is the environment. You're constantly observing the state — your speed, the cars around you, the stretch of road ahead — and constantly taking actions: you accelerate, you brake, you steer. And just like the warehouse robot, your actions change the state. Press the accelerator and a moment later your speed is higher and the gap to the car ahead is smaller. That new situation is the state you must now respond to.

🚗 Agent (you, driving) 🛣️ Environment (the road) action: accelerate / brake / steer feedback: reward / penalty · new state
One turn of the loop: you observe the state, take an action, and the road answers with feedback and a fresh state. Round and round, for the whole drive.

This is exactly the warehouse robot in different clothes. Agent and environment, locked in a loop of observe-and-act. The beauty of RL is that the same picture describes a self-driving car, a chess engine, a trading bot, and your little sister wobbling down the street on her first bike — they all live inside this one exchange.

CORE CONCEPTFeedback: the world answers back

An action on its own teaches nothing. What turns interaction into learning is the second half of the loop: after the agent acts, the environment responds with feedback. That feedback comes as a number — a reward when the action was beneficial, or a penalty (a negative reward) when it was poor.

For the warehouse robot, a positive reward follows beneficial actions: navigating toward its goal without clipping an obstacle, say, or placing a load exactly where it belongs. A penalty follows poor ones: crashing into a wall, or trundling down a needlessly long, inefficient route that wastes time and battery. The robot doesn't get a lecture explaining what it did wrong — it just gets the number, and the number is enough.

Back behind the wheel, you feel this constantly, even if you've never called it "reward". Braking smoothly feels good — passengers relaxed, coffee unspilled — and that comfortable, in-control feeling is a positive signal. Jerky, over-eager acceleration feels uncomfortable; it might even trigger a warning chime or a glare from the passenger seat — a clear negative signal. Over many drives, those signals teach you which actions lead to better outcomes, long before anyone hands you a rulebook.

💡 The tell

Feedback in RL is evaluative, not instructive. The environment tells you how good what you did was — it never tells you what you should have done instead. Figuring that out is the agent's job. That single distinction is what separates RL from supervised learning.

So the full loop now has both halves. The agent observes the state, chooses an action, and the environment returns feedback (a reward or penalty) along with the next state. Interaction supplies the experience; feedback supplies the lesson.

SEE ITAct → Feedback

Time to drive the loop yourself. Pick a scenario, choose an action, and watch the environment answer — the reward or penalty for that action, a one-line consequence, and your running cumulative reward. Good actions stack up; poor ones drain the total. The agent has no rulebook; the only thing teaching it is this stream of feedback.

🔁 Interactive · One turn of the loop, your choice

Choose a scenario, then take an action. The environment responds. Your cumulative reward is the running total — exactly what the agent is trying to grow.

This action's feedback
—
Cumulative reward
0

Pick an action above to take one step of the loop.

Feel the rhythm? Each click is one turn of the loop: an action goes out, feedback comes back, and the total shifts. Nobody told you which button was "correct" — you learned that brake smoothly beats slam the gas purely from the numbers. That's reinforcement learning in miniature, and that growing total is the cumulative reward the whole field is built to maximise.

FRAMEWORKHow reward shapes learning

Here's the part that ties the loop together. The agent's goal is not to grab the biggest reward on any single step. Its goal is to maximise the cumulative reward — the total piled up over the whole episode. You met this idea in Lesson 1.3 under its formal name, the return. Feedback shapes learning precisely because the agent is always weighing each action against that long-run total.

Why does the distinction matter? Because the action that pays best right now is often not the action that pays best over time — and the agent has to learn the difference from feedback alone.

Picture the warehouse robot at a fork. One path is direct but risky: it cuts straight through a crowded aisle, saving time if nothing goes wrong — but one misjudged turn means crashing into a shelf and a hefty penalty. The other path is longer but safer: a clear, roundabout route that takes more steps yet almost never ends in a collision. Chasing only the immediate "shortest distance" reward, the robot takes the risky path. Chasing the cumulative reward — counting the occasional crash penalty into the average — it often learns the longer path wins.

Driving makes the same trade-off visceral. Driving fast is immediately satisfying — you get there sooner, and that's a real, felt reward in the moment (short-term). But speed carries a long-term tail of penalties: accidents, tickets, wear, the occasional very large negative. So you learn, over many trips, to balance speed with caution — not because anyone forbade fast driving, but because the cumulative reward of steady, careful driving comes out ahead.

🔑 The shaping principle

Over time, feedback teaches the agent which actions lead to better long-run outcomes — not which feel best in the instant. Maximising cumulative reward is what quietly forces the agent to trade a tempting short-term gain for a larger payoff later.

COMPARISONShort-term vs long-term

Because this trade-off is the engine behind so much of what's coming in Stage 2, it's worth seeing it laid out plainly. The same action can look great on the immediate scoreboard and terrible on the cumulative one.

ActionShort-term (immediate reward)Long-term (cumulative reward)What the agent learns
Robot: risky shortcutfewer steps, quick winoccasional big crash penalty drags the average downoften prefer the safer route
Robot: safe detourmore steps, slower nowrarely crashes — higher reliable totalsteady reliability pays off
Driver: speedarrive sooner, feels greatrisk of accidents & ticketstemper speed with caution
Driver: smooth & steadyslightly slower tripsafe, comfortable, no penaltiescaution wins the long game

Look down the two reward columns and the lesson jumps out: the immediate column and the cumulative column disagree. An agent that optimises the first becomes reckless; an agent that optimises the second becomes wise. The whole art of the algorithms you'll meet next — value functions, Q-learning, policy methods — is really the art of estimating that cumulative column so the agent can act on it.

WEAKNESSCommon mistakes to avoid

✗ "Only the immediate reward matters."

No — the agent is after the cumulative reward (the return). An action with a small or zero payoff now can be the smart move because it sets up a far bigger total later. Optimising only the next step is how agents become greedy and shortsighted.

✗ "Feedback is just a supervised label."

A label says "the right answer was X." A reward only says "what you did was worth +3" — it never reveals the correct action. RL feedback is evaluative, not instructive, it can be delayed, and the agent must work out for itself which action earned it.

✗ "The state is just the agent's position."

Position is only one piece. The state is the whole relevant situation — for the robot that includes nearby obstacles and whether it's carrying a load; for the driver, your speed and the cars around you. Strip the state down to position alone and the agent can't tell apart situations that demand different actions.

KEY INSIGHTWhere this connects in your course

  • The cast (1.1) — agent, environment, state, action, reward: this lesson zooms into the loop they all live inside.
  • The RL framework & loop (1.2) — the formal state → action → reward → next-state cycle that interaction-and-feedback makes concrete.
  • The return (1.3) — the cumulative reward you watched grow here, made precise with discounting by γ.
  • Policy — the agent's brain (2.2) — how the agent decides which action to take in each state.
  • Value functions (2.3) — how the agent estimates that long-run cumulative reward so it can choose wisely.

Hold the heartbeat — act, get feedback, adjust, repeat — and everything in Stage 2 is just teaching the agent to do that exchange better and better.

GLOSSARYKey terms introduced

Interaction
The exchange in which the agent observes a state and takes an action that changes the environment, producing the next state.
State (St)
The whole relevant snapshot of the current situation the agent observes — not just position.
Action (At)
A choice the agent makes that affects the environment and changes the next state.
Feedback
The environment's response after an action — an evaluative signal, not an instruction.
Reward vs penalty
A positive reward follows a beneficial action; a penalty (negative reward) follows a poor one. Both are the same signal with opposite sign.
Cumulative reward
The total reward piled up over time — what the agent actually tries to maximise (formally, the return).
Short-term vs long-term
The trade-off between the reward of an action right now and its effect on the long-run cumulative total.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1Describe one full turn of the interaction-and-feedback loop.›

The agent observes the state (the current situation), chooses an action that changes the environment, and the environment returns feedback — a reward or penalty — along with the next state. Then it repeats.

2Why is RL feedback different from a supervised-learning label?›

A label gives the correct answer; reward feedback is only evaluative — it says how good the action was, never what the right action was. It can also be delayed, leaving the agent to figure out which action earned it.

3The robot can take a direct-but-risky path or a longer-but-safer one. Which should it learn, and why?›

It depends on the cumulative reward, not the immediate one. If the risky path's occasional crash penalty drags the long-run total below the safe path's, the agent should learn to prefer the longer, safer route — trading a short-term gain for a bigger long-term total.

4Why isn't "my current position" a good enough definition of the state?›

Because two situations at the same position can demand totally different actions. The state must capture the whole relevant context — nearby obstacles, whether the robot is carrying a load, the driver's speed and surrounding cars — so the agent can choose well.

You've found the heartbeat. Next: the policy — the brain that picks the action →