ProjekinAja. Reinforcement Learning Stage 4 · Advanced & Applied Lesson 4.1
● Stage 4 · Lesson 4.1

Model-Based RL

Everything so far was learned the hard way — by trial and error in the real environment. What if the agent could learn a model of the world and practise inside its own imagination? That's model-based RL.

⏱ ~15 min read 🧠 Planning & imagination 🚀 Stage 4 begins

🎯 By the end you'll understand

The difference between model-free and model-based RL, what it means for an agent to learn the environment's dynamics P(s'|s,a) and reward R, how a model lets the agent plan by simulating futures, the big payoff (sample efficiency), the classic Dyna idea, modern world models like PlaNet and Dreamer, and the honest catch — a learned model is never perfect.

KEY INSIGHTThe one idea to take away

A model-free agent only learns from experience it actually lives. A model-based agent learns a little model of the world — and then practises inside its own head, for free.

Every algorithm you met in Stages 2 and 3 — Q-learning, SARSA, DQN, PPO, SAC — learned by acting in the real environment and reacting to what came back. None of them ever built a picture of how the world works. Model-based RL adds exactly that: a learned model of the environment's dynamics. Once you have a model, you can imagine what would happen next without paying the real cost of finding out.

🔑 Hold onto this

Think of a chess player. A beginner only improves from games actually played. A master thinks several moves ahead in their head — running futures on a mental model of the board before touching a piece. Model-free is the beginner; model-based is the master who plans. The model turns expensive real experience into cheap imagined experience.

CORE CONCEPTTwo ways to learn

Picture two people learning to drive to a new office across town.

The first just drives it, over and over. Wrong turn? They feel the pain, remember it, and turn earlier next time. After fifty real drives they're great — but those fifty drives cost fifty tanks of attention, time, and the odd near-miss. That's model-free learning: improve a policy or value estimate directly from real experience, never asking "how does traffic actually work?"

The second person studies a map first. They learn the road network — which streets connect to which, roughly how long each takes. Now, before driving, they can trace routes in their head, compare them, and pick the best one. They still drive to confirm, but they need far fewer real trips because most of the figuring-out happened on the map. That's model-based learning: learn a model of the environment, then plan against it.

📖 Model-free RL

Learning a value function or policy directly from experience, with no explicit model of the environment's dynamics. The agent never predicts the next state — it just learns what to do. (Q-learning, SARSA, DQN, REINFORCE, PPO, SAC.)

📖 Model-based RL

The agent learns (or is given) a model of the environment — its transition dynamics P(s'|s,a) and reward R(s,a) — and uses that model to plan: simulate possible futures and choose actions before committing to them in the real world.

Neither is "the right one." They are two answers to the same question — how do I turn experience into good behaviour? — and they trade off in opposite directions. We'll lay out that trade-off carefully later in the lesson.

FORMALWhat "a model" actually means

Back in Lesson 1.4 you met the Markov Decision Process: states, actions, a reward, and the rule that says where you land next. A model is precisely that last part made explicit and usable by the agent. It has two halves.

P(s'|s,a) = probability of landing in state s' after taking action a in state s transition (dynamics) model
R(s,a) = the reward you expect for taking action a in state s reward model

Put plainly: a model is a little simulator the agent carries around. Feed it a state and an action, and it answers "here's what would probably happen next, and here's the reward you'd get." If the agent has that, it can run a whole episode in its imagination — state to action to predicted next state to predicted reward, and repeat — without ever touching the real environment.

Where does the model come from? Two cases:

🎁

Given (a known model)

Sometimes the rules are handed to you. The rules of chess or Go are perfect, exact dynamics — you know what every move does. This is where planning shines and where AlphaGo lives.

📚

Learned (a learned model)

Usually you don't know the physics of a robot arm or the dynamics of a game from pixels. So the agent learns the model from real experience — fitting P(s'|s,a) and R as it goes — then plans against its best guess.

The given-model case is clean but rare. The learned-model case is where most of the interesting (and dangerous) work happens — because a learned model can be wrong, a point we'll return to.

CORE CONCEPTPlanning: thinking ahead with a model

A model is only useful because of what it unlocks: planning. Planning is using the model to look into the future before acting, so you can choose the action that leads somewhere good.

📖 Planning

Using a model of the environment to simulate future trajectories (imagined rollouts) and improve a decision — without taking those steps for real. The model produces synthetic experience; planning turns it into a better action or value estimate.

The simplest version: from where you stand, imagine taking each available action, then imagine a few steps further along each branch, add up the predicted rewards, and pick the first action on whichever imagined path looks best. You only commit to that one first action in the real world — then you re-plan from the new state. This re-plan-every-step rhythm is the core idea behind model-predictive control (MPC).

The famous example is AlphaGo / AlphaZero. Go has a perfect known model — the rules — so the agent can plan by searching a tree of possible futures. The search is Monte Carlo Tree Search (MCTS): roll out promising lines of play in imagination, guided by a learned value network (how good is this position?) and a learned policy network (which moves are worth exploring?). The model tells it what each move does; the search figures out which sequence is best; the learned networks keep the search from drowning in the astronomical number of possibilities. That combination — model + tree search + learned value/policy — is the blueprint for superhuman game play.

Press play to watch one planning decision being made, and count the imagined steps against the real ones.

One planning decision
now s +4 +5 Σ +9 best up +3 +1 Σ +4 right −1 −1 Σ −2 down
Imagined steps
6
Real steps
0
imagined step, inside the model real step, in the environment
Planning: imagine each action a few steps ahead using the model, score the branches, then take the first action of the best one — all without spending a single real step. Press play to watch it happen.

SEE ITLearn vs imagine

Here's the whole payoff in one picture. Both agents are solving the same task and reach the same final skill. The only difference is how many real environment steps it takes them to get there — and real steps are the expensive thing (a robot wearing out, real money, real time). Toggle between the two and watch the curve.

🧠 Interactive · Sample efficiency, real steps vs imagined ones

Same task, same ceiling. The question is only how fast performance rises per real step taken in the environment.

Model-free: learns only from real steps — slow but simple and robust.

Real steps to reach "good"
—
Imagined steps used
—

And here's one planning decision from the current state. The model rolls out three imagined futures; the agent commits only to the first action of the best one. Press to plan.

↑ Go up
—
→ Go right
—
↓ Go down
—

No plan yet. The model hasn't been consulted.

The model-based curve climbs faster per real step because most of the practice happened in imagination — cheaply, inside the learned model. That single property, sample efficiency, is the entire reason to bother with the extra machinery of learning a model.

FRAMEWORKDyna: blend both worlds

You don't have to choose. The classic Dyna architecture (Sutton, 1990) is the bridge: keep doing model-free learning from real experience, and learn a model on the side, and use that model to generate extra imagined experience to learn from too. The model-free updates and the model-based updates feed the same value function.

1

Act for real

Take a real action, see the real next state and reward — a genuine bit of experience.

2

Learn from it (model-free)

Do a normal Q-learning update from that real transition, exactly like Lesson 2.6.

3

Update the model

Record what just happened so the model's P(s'|s,a) and R get a little more accurate.

4

Imagine & learn again (model-based)

Now run, say, n extra updates on transitions sampled from the model — free practice. Each real step buys you many learning updates.

This is why Dyna is so instructive: it shows model-free and model-based aren't enemies but two taps feeding one bucket. A single real step might be followed by 5, 10, or 50 imagined updates — so the value function improves far faster per real interaction than pure model-free learning would allow. Crank the imagination dial up, and you lean harder on the model; crank it to zero, and you're back to plain model-free.

Every real step is precious. A model lets you squeeze many lessons out of each one — by replaying it, and variations of it, in your head.

CORE CONCEPTWorld models & learning in a dream

The modern, deep-learning version of this idea is the world model. Instead of a tidy table of probabilities, the agent learns a compact neural model that predicts how the world evolves — and crucially, it often does this in a latent space: a small learned summary of the raw observation, not the full pixels.

📖 World model

A learned, usually deep and often latent, model of the environment's dynamics. It compresses high-dimensional observations (pixels) into a compact state and predicts how that state changes — so the agent can plan or train a policy almost entirely inside the model.

Why latent? Predicting full-resolution future frames is wasteful and hard. Predicting a small summary of the world — enough to decide what to do — is much cheaper, and it's what these methods do. Two well-known examples from the literature:

🛰️

PlaNet — the Deep Planning Network

Learns a latent dynamics model from images, then plans directly in latent space — searching over imagined action sequences — to control tasks from pixels, with far fewer real interactions than model-free baselines.

🌙

Dreamer — "dream to control"

Builds on the same latent world model, but instead of planning by search it trains a policy by latent imagination: it learns behaviours almost entirely from imagined rollouts inside the model. The agent literally learns to act by dreaming.

Keep the picture conceptual: a world model is just a learned simulator small enough to dream in. PlaNet plans inside it; Dreamer learns a policy inside it. Both are chasing the same prize — squeeze the most behaviour out of the fewest real steps.

COMPARISONModel-free vs model-based, side by side

Here is the trade-off in one table. Read it as two columns of opposite strengths, not a winner and a loser.

 Model-freeModel-based
What's learneda value function and/or policy, directlya model of dynamics P(s'|s,a) & reward R, then plan/learn from it
Sample efficiencylow — needs lots of real experiencehigh — imagined experience is cheap
Complexitysimpler — one thing to learnharder — must learn a model and plan with it
Robustnessrobust — no model to be wrongfragile to model error — bad model misleads planning
Compute at decision timecheap — just read off the policycan be heavy — searching/rolling out futures
ExamplesQ-learning, DQN, PPO, SACDyna, AlphaZero (given model), PlaNet, Dreamer

The headline: model-based buys sample efficiency by paying in complexity and model-error risk. If real experience is cheap and plentiful (a fast simulator), model-free's simplicity and robustness often win. If real experience is expensive or dangerous (a physical robot, a real patient, real money), the effort of learning a model pays for itself many times over.

WEAKNESSThe catch: a learned model is never perfect

Now the honest part. A learned model is a guess, and planning is only as good as the model you plan against. Plan brilliantly inside a wrong model and you'll act brilliantly on a fantasy.

⚠️ Model bias & compounding error

Small prediction errors compound over a rollout: the model's guess for step 1 is slightly off, step 2 is built on that wrong state and is more off, and by step 20 the imagined future has drifted into nonsense. The agent then optimises for rewards that do not exist in the real world — it has learned to game its own flawed simulator. This is model bias, and it is the central failure mode of model-based RL.

This is why real systems hedge. They keep rollouts short (don't dream too far before the error blows up), they model their own uncertainty and distrust confident-but-shaky predictions, and they keep gathering real data to correct the model where it's wrong. Dyna's blend — real updates anchoring the imagined ones — is itself a guard against runaway model bias.

💡 The mental rule

A model is a map, not the territory. Maps are wonderful — they let you plan a route without walking every street. But trust a stale or sloppy map too far and you'll confidently march into a wall. Plan with the map; verify on the ground.

WEAKNESSCommon mistakes to avoid

✗ "Model-based RL is always better — it's smarter."

No. It's more sample-efficient, but it adds the burden of learning a model and the risk that the model is wrong. When a fast, accurate simulator already exists, simple model-free methods (PPO, SAC) are often more robust and easier to get working. "Better" depends entirely on how expensive real experience is.

✗ "A learned model is basically the true environment."

It's an approximation fitted from limited data. Its errors compound over long rollouts, and an agent that plans too far ahead will optimise against a fantasy. Treat the model as a useful but fallible map — keep rollouts short and keep correcting it with real data.

✗ "Planning is what makes it model-based — you don't really need a model."

Planning requires a model. To "imagine the next state" you need something that predicts the next state — that something is the model, whether given (chess rules) or learned (a world model). No model, no planning; you'd be back to pure trial-and-error model-free learning.

KEY INSIGHTWhere this connects in your course

  • MDPs & transitions (1.4) — the model P(s'|s,a) and R are exactly the MDP's dynamics, now made explicit and put to work.
  • Q-learning (2.6) — the model-free workhorse Dyna keeps running, now topped up with imagined experience from a learned model.
  • DQN (3.2) — the model-free deep-RL baseline that world-model methods like PlaNet and Dreamer aim to beat on sample efficiency.
  • Reward shaping & reward hacking (4.2) — model bias is a close cousin: an agent gaming a flawed model is gaming a flawed reward picture. Both are "optimising the wrong thing brilliantly."

Hold the one sentence — learn a model, then practise and plan inside it — and the rest of Stage 4 is about what powerful agents do once they can imagine.

GLOSSARYKey terms introduced

Model-based RL
Learning or using a model of the environment, then planning/learning from simulated experience.
Model-free RL
Learning a value function or policy directly from real experience, with no model of dynamics.
Transition model (P(s'|s,a))
A predictor of the next state given the current state and action — the dynamics half of a model.
Reward model (R(s,a))
A predictor of the reward for taking an action in a state.
Planning
Using a model to simulate future trajectories and improve a decision before acting for real.
Dyna
A classic architecture that learns model-free from real experience and also generates extra imagined experience from a learned model.
World model
A learned, often latent, deep model of dynamics inside which an agent can plan or train a policy (e.g. PlaNet, Dreamer).
MCTS
Monte Carlo Tree Search — planning by searching a tree of imagined futures, used with a model in AlphaGo/AlphaZero.
Model bias
Error in a learned model that compounds over rollouts and misleads planning toward rewards that don't exist in reality.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1In one sentence, what's the core difference between model-free and model-based RL?›

Model-free learns a value function or policy directly from real experience with no model of the world; model-based learns a model of the dynamics P(s'|s,a) and reward R, then plans or learns from imagined experience generated by that model.

2What is the main payoff of model-based RL, and why does it happen?›

Sample efficiency — reaching good performance with far fewer real environment steps. It happens because the agent can generate cheap imagined experience from its model and plan ahead, instead of paying the real cost (a robot wearing out, time, money) for every lesson.

3What does Dyna do that's clever, and how do PlaNet and Dreamer differ?›

Dyna blends both worlds: it learns model-free from real transitions and learns a model that generates extra imagined transitions to learn from — many updates per real step. PlaNet learns a latent world model and plans in latent space; Dreamer uses a similar latent model but trains a policy by imagination ("dream to control") rather than by search.

4Why isn't model-based RL simply the winner? Name the catch.›

A learned model is imperfect. Its errors compound over long rollouts (model bias), so the agent can end up optimising for rewards that don't exist in reality. It also adds complexity — you must learn the model and plan with it. Model-free is simpler and more robust; model-based trades that away for sample efficiency.

Stage 4 begins. Next: how rewards can help — and how agents cheat them →