ProjekinAja. Reinforcement Learning Stage 1 · Foundations Lesson 1.2
● Stage 1 · Lesson 1.2

The RL Framework & Loop

The precise, repeating cycle every RL agent runs — state, action, reward, next state — turning the bike intuition into a formal loop you can name and follow.

⏱ ~13 min read 🔁 The core cycle 🧱 The framework

🎯 By the end you'll understand

The four building blocks of every RL system (agent, environment, policy, reward signal), the exact loop they run at each time step — observe St, choose At, receive Rt+1 and land in St+1 — and how that loop unrolls into a trajectory the agent learns from.

KEY INSIGHTThe one idea to take away

Reinforcement learning isn't one big decision — it's the same tiny loop, run over and over: see where you are, pick a move, see what happens, repeat.

In Lesson 1.1 your little sister learned to ride a bike by trying, falling, and adjusting. That story felt like a loop — and it was. This lesson gives that loop its real shape. Once you can see the cycle clearly, the rest of the course is just learning how the agent gets better at one step of it.

🔑 Hold onto this

At every moment the agent does exactly four things: observe a state, choose an action, receive a reward, and land in a new state. That four-beat rhythm — St → At → Rt+1 → St+1 — is the entire RL framework. Everything else is detail on top of it.

FRAMEWORKThe four elements of an RL system

Before we set the loop spinning, let's name the cast clearly. Every RL system — whether it's a robot in a maze, a program playing chess, or your sister on her bike — is built from the same four elements. You met the first few in Lesson 1.1; here they are gathered as the working framework.

🤖

The agent

The learner and decision-maker — the thing we're training. It's the only part of the system that chooses.

🌍

The environment

Everything outside the agent that it senses and affects. The agent acts on the environment, and the environment answers back.

🧠

The policy π(a|s)

The agent's rule or strategy for choosing an action in each state. It's the brain of the operation — and the thing that gets better as the agent learns.

🍌

The reward signal

The single number the environment sends after each action, telling the agent whether that move was good or bad. It defines the goal — and nothing else does.

Notice the division of labour. The agent owns the policy — it decides. The environment owns the reward signal and the rules of what happens next — it reacts. The agent can change its policy through learning, but it can never change the rules of the environment or hand itself a reward. That clean split is what makes the loop honest: the only way to earn more reward is to choose better.

📖 Policy

A policy π(a|s) is a mapping from states to actions — a rule that says, "in this state, take this action." The whole goal of RL is to learn the policy that earns the most reward over time. We'll study policies in depth in Lesson 2.2; for now, just think of it as the agent's playbook.

CORE CONCEPTThe interaction loop, step by step

Now let's set those four elements in motion. The magic of RL is that they interact on a clock. Time ticks forward in discrete time steps, and at each step t the same exchange happens between the agent and the environment.

Here's the exchange, beat by beat. The agent observes the current state St. Using its policy, it selects an action At. The environment takes that action, transitions to a new state St+1, and hands back a reward Rt+1. Then the clock ticks: the new state becomes the current state, and it all happens again.

Press play and watch the index. The reward and the next state come back together, and only then does the clock tick.

The loop, three turns
Agent sees state St Environment action At reward Rt+1 · next state St+1 step t
Trajectory · what has happened so far
S₀A₀R₁S₁A₁R₂S₂A₂R₃S₃
The agent sends an action; the environment returns a reward and the next state. The next state becomes the new "current" state, and the loop turns again.

Why does the reward carry the subscript t+1 instead of t? Because the reward is a consequence — it arrives after the action, together with the next state. At step t the agent only knows St and picks At; the environment's response — Rt+1 and St+1 — belongs to the next tick. It's a small bit of bookkeeping, but it keeps cause and effect in the right order: act first, find out after.

St → At → Rt+1, St+1 → At+1 → Rt+2, St+2 → … the interaction loop, unrolled in time

The agent's job across all of this is to learn a policy π(a|s) that maps each state to a good action — good meaning it leads to the most cumulative reward over the whole sequence, not just the next step. We'll make "cumulative reward" precise in Lesson 1.3. For now, hold the loop.

CORE CONCEPTHow the cycle works together

It helps to slow the loop down and watch it as six distinct moves. Every single time step, the agent and environment hand off to each other in this order:

1

Observation

The agent observes the current state St — a snapshot of the situation right now.

2

Decision

Using its policy, the agent selects an action At for that state.

3

Interaction

The agent takes the action, affecting the environment.

4

Feedback

The environment returns a reward Rt+1 and the next state St+1.

5

Update

The agent uses the reward and next state to update its strategy — nudging the policy toward actions that paid off.

6

Repeat

The next state becomes the current state, and the cycle begins again at step 1.

Steps 1 through 4 are pure interaction — they'd happen even if the agent never learned a thing. Step 5 is where the learning lives: it's the moment the agent looks at what just happened and adjusts. Everything you'll study in Stage 2 — value functions, Q-learning, SARSA — is really just different recipes for that one step. The loop stays the same; only the update gets smarter.

💡 The tell

If you can spot the four-beat exchange — observe, decide, act, get feedback — repeating on a clock, you're looking at a reinforcement learning problem. If the data arrives all at once with answers attached, it isn't.

EXAMPLEThe robot in the maze

Let's swap the bike for a fresh example you'll see again and again: a robot finding its way out of a maze. It's the perfect teaching case because every element of the framework has an obvious, physical meaning.

RL elementIn the maze
Agentthe robot
Environmentthe maze
State Stthe robot's current position (which cell it's in)
Action Atmove forward / turn left / turn right
Reward Rt+1+10 for getting closer to the exit · −1 for hitting a wall · +50 for finding the exit
Next state St+1the robot's new position after moving
🤖 START 🚪 EXIT +50
A simple maze. The robot starts top-left and must reach the exit bottom-right. Dark blocks are walls; bumping one costs −1. Each step closer to the exit earns +10; reaching it earns +50.

Watch how the loop runs here. The robot observes its position (state). It decides to move forward (action). It takes the move (interaction). The maze responds — it either advanced toward the exit (+10), bumped a wall (−1), or reached the door (+50) — and reports the robot's new cell (feedback). The robot updates its strategy with that result, then repeats from the new position. No one ever hands it the correct path. It discovers the way out from rewards alone — exactly like your sister with her bananas, just in a grid.

SEE ITRun the RL loop

Here is the loop made literal. Below is one hand-authored run of the robot escaping the maze. Press ▶ Step to advance the cycle one iteration — the four beats observe → decide → act → feedback light up, the readout shows the current tuple (St, At, Rt+1, St+1), and the cumulative reward grows. Watch the total climb to the big +50 payoff at the exit.

🤖 Interactive · One trip through the maze, step by step

Each ▶ Step is one full turn of the loop. The robot follows a fixed route through the maze drawn above — your job is to watch the tuple and the running return. Cells are written (row, column), counted from the start at (0,0) in the top-left; the exit is (3,3). The bar under the readout lets you pause, step back or drag to any moment of the trip.

1

Move forward — toward the exit

From (0,0) down to (1,0). One cell closer. Reward +10

2

Turn left, then move forward

From (1,0) to (1,1). Closer again. Reward +10

3

Move forward — into a wall

Blocked at (1,1) by the wall in (1,2). The robot stays put. Reward −1 💥

4

Turn right, then move down

From (1,1) to (2,1). Closer again. Reward +10

5

Turn left, move toward the door

From (2,1) to (2,2). Reward +10

6

Move forward — closer still

From (2,2) to (2,3). Almost there. Reward +10

7

Turn right, move forward — through the exit!

From (2,3) to (3,3). Goal reached. Reward +50 🚪

State Sₜ
—
Action Aₜ
—
Reward Rₜ₊₁
—
Next state Sₜ₊₁
—
Cumulative reward
0

The robot is at the start, knowing nothing. Press ▶ Step to run one turn of the loop.

Notice that one bad move (the wall, −1) didn't derail the trip — what mattered was the total, dominated by getting closer and finally that +50 for the exit. The agent never saw "the correct path." It just ran the loop, collected rewards, and the sequence of states and actions you watched is the thing it learns from. That sequence has a name.

FORMALThe loop becomes a trajectory

Run the loop from the very beginning and write down everything that happens, in order. You get an alternating chain of states, actions, and rewards:

S0, A0, R1, S1, A1, R2, S2, A2, R3, … a trajectory (the recorded history of one run)

This chain is called a trajectory — sometimes a rollout. It always starts with a state S0, because the agent has to see something before it can act. Then it strictly alternates: a state is followed by the action chosen there, which is followed by the reward and next state that action produced. Read it left to right and you're literally re-watching the loop turn.

When a trajectory has a natural ending — the robot reaches the exit, the game ends, your sister arrives — we call that finished run an episode. The agent typically runs many episodes, and each one is a fresh trajectory it can learn from. Some problems never end (a thermostat, a stock-trading bot); those are continuing tasks, and we handle them with the discounting you'll meet in Lesson 1.3.

📖 Trajectory (episode)

A trajectory is the full sequence S0, A0, R1, S1, A1, R2, … produced by running the loop. A trajectory that runs to a terminal state is an episode. Trajectories are the raw experience RL algorithms learn from — RL generates its own training data by living through them.

And that's the whole framework. Four elements, one four-beat loop, repeated until you have a trajectory. The state tells you where you are, the policy picks the action, the environment answers with a reward and the next state, and the agent updates. Hold this picture — every algorithm in this course is a different, smarter way of doing the same loop.

WEAKNESSCommon mistakes to avoid

✗ "The loop runs once and the agent has learned."

No — a single turn of the loop is one tiny experience. Learning emerges from running the loop thousands of times, across many episodes. One trip through the maze barely teaches anything; the hundredth trip is where good behaviour shows up.

✗ "The state is the whole world."

The state is the snapshot the agent observes — usually just what it needs to decide, like the robot's cell. It isn't every atom of the environment. Choosing what goes into the state is a real design decision (more in Lesson 1.4 on the Markov property).

✗ "The reward and next state come before the action."

The order is fixed: observe → act → then receive reward and next state. That's why reward carries the t+1 subscript — it's a consequence of the action, never a hint handed out beforehand.

KEY INSIGHTWhere this connects in your course

  • What RL is (1.1) — the bike story and trial-and-error learning you just formalised into a loop.
  • The agent's goal (1.3) — turning the rewards along a trajectory into a single number, the return, and discounting future rewards by γ.
  • Markov Decision Processes (1.4) — the formal frame that makes "state," "transition," and "reward" mathematically precise.
  • Value functions (2.3) — how the agent judges how good a state is, so step 5 of the loop (the update) can actually improve the policy.

The loop is the skeleton of the whole field. From here on, every new idea hangs on one of its parts: a better way to value a state, a better policy for choosing actions, a smarter update. Same loop, sharper pieces.

GLOSSARYKey terms introduced

State (St)
The snapshot of the situation the agent observes at time step t.
Action (At)
The choice the agent makes in state St, selected by its policy.
Reward (Rt+1)
The numeric feedback the environment returns after an action — arriving after the act, hence the t+1.
Next state (St+1)
The new state the environment transitions to after the agent acts.
Time step
One tick of the loop's clock — one full observe-act-feedback exchange, indexed by t.
Trajectory / episode
The recorded sequence S0, A0, R1, S1, … from a run; a run that ends in a terminal state is an episode.
Policy (π(a|s))
The agent's strategy — a mapping from each state to the action to take there.
Transition
The environment's move from St to St+1 in response to action At.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1Name the four beats of the RL loop in order, with their symbols.›

The agent observes state St, chooses action At, the environment returns reward Rt+1 and lands the agent in next state St+1. Then the loop repeats from St+1.

2Why does reward carry the subscript t+1 and not t?›

Because the reward is a consequence of the action. At step t the agent only knows St and picks At; the reward and next state arrive after — at the next tick. The subscript keeps cause and effect in order: act first, find out after.

3Map the maze example onto the four elements: agent, environment, state, action, reward, next state.›

Agent = the robot; environment = the maze; state = the robot's current cell; action = move forward / turn left / turn right; reward = +10 closer, −1 hitting a wall, +50 finding the exit; next state = the robot's new cell.

4What is a trajectory, and how does it relate to the loop?›

A trajectory is the recorded sequence the loop produces: S0, A0, R1, S1, A1, R2, … — it always starts with a state and alternates state, action, reward. It's just the loop written down in order, and it's the raw experience the agent learns from. A trajectory that ends in a terminal state is an episode.

Loop locked in. Next: turning rewards into a single goal →