ProjekinAja. Reinforcement Learning Stage 4 · Advanced & Applied Lesson 4.4
● Stage 4 · Lesson 4.4

Offline RL & Imitation Learning

Trial and error is risky and expensive in the real world. What if the agent could learn from a fixed dataset, or by copying an expert, instead of exploring? Two ways to learn without (much) interaction.

⏱ ~15 min read 🧭 Learning without exploring 🏥 Applied RL

🎯 By the end you'll understand

Two ways to learn a policy without live trial and error: imitation learning (copy an expert's demonstrations) and offline RL (squeeze a policy out of a fixed dataset of past experience). You'll see why both keep tripping over the same villain — distribution shift — and how each one tries to tame it.

KEY INSIGHTThe one idea to take away

Sometimes the cheapest, safest teacher isn't the environment — it's a pile of data someone already collected. The whole challenge is learning from it without wandering off the edge of what that data actually covers.

Everything in Stages 2 and 3 assumed the agent could act in the world — try things, fail, and learn from the consequences. But in healthcare, finance, or a real robot, every clumsy exploratory action has a cost (sometimes a human one). So a different question becomes urgent: can we learn a good policy without exploring? Yes — two ways. Copy an expert, or mine logged experience. Both work, and both share one nemesis.

🔑 Hold onto this

Online RL collects its own data by exploring. Imitation learning and offline RL learn from data they did not collect — expert demos or logged history. The price of skipping exploration is distribution shift: the moment your policy strays into situations the data never showed, it's flying blind.

CONTEXTWhy you'd ever avoid trial and error

Remember teaching your little sister to ride a bike? That worked because falling over costs a scraped knee and a banana, not a catastrophe. Trial and error is wonderful when mistakes are cheap.

But now imagine the "agent" is a system recommending insulin doses for diabetic patients, or trading a retirement fund, or steering a two-tonne car. You cannot let it "explore" — try a wild dose just to see what happens — because the cost of a bad action is measured in harm, money, or lives. Exploration here is dangerous, expensive, or simply not allowed.

And yet, in all these settings, data already exists: hospital records of what doctors did and how patients fared, trading logs, millions of miles of human driving, billions of past clicks. The dream is to turn that existing data into a good policy — no risky live experiments required.

💡 Two flavours of "learn from old data"

Imitation learning assumes the old data came from a good expert and just copies it. Offline RL makes no such assumption — it takes a possibly-mediocre dataset and tries to extract the best policy it can, using rewards, the way ordinary RL would.

CORE CONCEPTPart A — Imitation learning

Here's the most human way to learn a skill: watch someone good at it, and copy them. You didn't learn to drive by randomly mashing pedals until the reward signal sorted you out — you watched a parent drive for years, then imitated. Imitation learning gives an agent that same shortcut.

📖 Imitation learning

Learning a policy from a set of expert demonstrations — recorded (state, action) pairs showing what a skilled agent did — instead of from a reward signal. The goal is to reproduce the expert's behaviour.

Notice what's missing: there's no reward function at all. The teaching signal isn't "this outcome was good (+1)"; it's "in this situation, the expert chose this action." That looks a lot like supervised learning — and the simplest method leans into exactly that.

Behavioral cloning — supervised learning in disguise

📖 Behavioral cloning (BC)

Treat imitation as plain supervised learning: collect the expert's (state, action) pairs as a labelled dataset, then train a model to predict the expert's action from the state. The state is the input, the expert's action is the label.

That's genuinely it. If your sister had a camera on her shoulder recording a champion cyclist — "when the bike leans left, steer slightly left", "when a curb appears, slow down" — she could fit a model mapping each scene to the rider's response. Simple, fast, no environment interaction needed during training. So what's the catch?

WEAKNESSBehavioral cloning & the drift problem

The catch is subtle and it's the heart of this whole lesson. Behavioral cloning only ever saw the states the expert visited — and a good expert mostly stays on the road, in the middle of the lane, doing sensible things. The dataset is a thin ribbon of "good situations."

Now the cloned agent drives. It's good, but not perfect, so it makes a tiny error — drifts six inches toward the lane edge. The expert never drifted there, so there are no demonstrations for "slightly off-centre." The model has no idea what to do. It guesses, drifts a little more, and now it's further from anything it was trained on. Each mistake pushes it into less-familiar territory, where it makes bigger mistakes. The errors snowball — and the car drives off the road.

📖 Distribution shift (compounding error)

When a learned policy acts, it visits states drawn from its own behaviour — which differs from the distribution of states in the training data. Small errors move the agent into unseen states, where it errs more, moving it further still. The error compounds over a trajectory instead of staying small.

✗ Why BC's errors snowball but supervised learning's don't

In ordinary supervised learning, each prediction is independent — one wrong label doesn't change the next input. In control, your action determines your next state. So an error doesn't just cost you one mistake; it changes the world you face next, dragging you off the data manifold where every future prediction also degrades.

Theory makes this precise: if BC has even a small per-step error rate, the total error over a horizon of T steps can grow like T2 in the worst case — far worse than the T you'd hope for. That quadratic blow-up is the compounding-error problem.

DAgger — let the expert label the agent's own mistakes

The classic fix is DAgger (Dataset Aggregation). The idea is delightfully direct: let the cloned agent drive, record the new (off-distribution) states it stumbles into, then ask the expert what they'd do in those states. Add those fresh labels to the dataset, retrain, repeat. Over a few rounds the dataset grows to cover exactly the recovery situations BC never saw — so the agent learns how to get back on the road. The cost: you need ongoing access to the expert to label new states.

Inverse RL — recover the why, not just the what

Inverse RL (IRL) takes a deeper route: instead of copying actions, it tries to infer the reward function that the expert seems to be optimising, then runs ordinary RL on that recovered reward. Learn why the expert acts, and you can generalise to new situations they never demonstrated — at the price of a much harder, more expensive learning problem.

🚗 The driving analogy, summed up

Behavioral cloning = memorise "in this exact view, do what the instructor did." DAgger = also ask the instructor what to do when you've drifted. Inverse RL = figure out what the instructor was trying to achieve (stay centred, smooth, safe) and drive toward that goal yourself.

CORE CONCEPTPart B — Offline RL (batch RL)

Imitation learning has one big assumption baked in: the demonstrations are good. But what if your data is just… history — a mixed bag of decisions, some smart, some dumb, each with a recorded reward? You don't want to copy it. You want to do the RL thing: find the policy that earns the most reward, learning from those logged consequences. That's offline RL.

📖 Offline RL (a.k.a. batch RL)

Learning a good policy purely from a fixed dataset of logged transitions (s, a, r, s′), with no new interaction with the environment during training. Unlike imitation, it uses the reward and aims to improve on the behaviour that generated the data, not merely reproduce it.

This is the holy grail for high-stakes domains. A hospital's records become training data for a treatment policy; a recommender's click logs train a better recommender; the robot learns from yesterday's runs overnight, with zero risk. No exploration. No environment. Just the dataset.

You might think: "We already have off-policy algorithms — Q-learning learns about one policy from data generated by another. Just point Q-learning at the logged dataset and we're done!" It's a completely reasonable thought. It also fails, often spectacularly, and understanding why is the key insight of offline RL.

FORMALThe distribution-shift trap (extrapolation error)

Recall Q-learning's update from Stage 2. The bootstrap target uses the value of the best next action:

Q(St,At) ← Q(St,At) + α[Rt+1 + γ maxa Q(St+1,a) − Q(St,At)]Q-learning update — the highlighted term is the problem offline

That maxa Q(St+1,a) quietly asks: "what's the value of the best possible action in the next state?" Online, if the agent gets seduced by an action with an inflated value, it simply tries it, sees a low reward, and corrects the estimate. The environment is a constant fact-checker.

Offline, there is no fact-checker. Suppose the dataset never contains action a* in state St+1. The network has never been corrected there, so its estimate Q(St+1, a*) is pure extrapolation — and neural nets extrapolate optimistically, often wildly so. The max then seeks out exactly these over-valued, never-seen actions and feeds their fantasy values back into the targets. Errors don't get corrected; they get amplified and propagated across the whole value function.

✗ Extrapolation error — the silent killer of naive offline Q-learning

The policy is drawn toward out-of-distribution (OOD) actions precisely because their values are unrealistically high — and offline there's no way to check. This is distribution shift again, wearing a value-function costume: the policy you're learning wants to do things the dataset never did, and your value estimates for those things are fiction.

The fix: stay close to the data

Every offline RL method is, at heart, one instruction: don't trust values for actions the data doesn't support. Two broad strategies:

🚧

Policy constraint

Force the learned policy to stay close to the data-generating (behaviour) policy — only pick actions the dataset actually contains in that state. No straying onto unseen actions, so no fantasy values to chase.

⬇️

Conservative value estimates

Deliberately penalise the values of OOD actions — push them down so the max can't be fooled by them. CQL (Conservative Q-Learning) is the headline example: it learns a Q-function that lower-bounds the truth, refusing to be optimistic about the unseen.

🔑 The unifying idea

Imitation learning fails when the agent drifts off the data. Offline RL fails when the value estimates drift off the data. Same disease — distribution shift — and the same cure: stay in-distribution. The art of offline RL is improving on the data while never wandering past its edges.

SEE ITWhy copying breaks

Let's watch behavioral cloning drift. Below, the expert's path is the green ribbon of states the demonstrations covered. The cloned agent tries to follow it. Toggle between a tiny early slip and a clean run, then press play, or step forward one square at a time, and watch what happens once it leaves the ribbon. You can step back or drag the slider to see any step again.

🛣️ Interactive · Behavioral cloning, step by step

The agent copies the expert state-by-state. Inside the green band it's fine. The moment it slips off, it has no demonstrations to copy — and the error compounds.

Expert demonstrations (the data)
Cloned agent's path
Distance from data
in-distribution
Step error
0

Press play, or step forward, to roll the agent along the road.

In the slip scenario, one small early error puts the agent on a square the expert never visited (red). With no demonstration to copy there, its next guess is worse, the error grows, and it tumbles off the road — pure compounding error. In the clean run it never leaves the green ribbon, so every step has a demonstration to copy and it sails through. Same policy, same expert; the only difference is whether it stayed in-distribution.

COMPARISONOnline RL vs imitation vs offline RL

Three ways to get a policy, side by side. The columns that matter most are the last two: what each one requires, and what bites you if you're careless.

ApproachNeeds live interaction?Needs a reward signal?Learns fromKey risk
Online RL
Stages 2–3
Yes — explores & collects its own data Yes its own trial-and-error experience exploration is slow, costly, or unsafe
Imitation (BC) No (training); but DAgger needs the expert No — uses expert actions, not rewards expert (state, action) demos distribution shift / compounding drift
Offline RL No — a fixed logged dataset Yes — uses logged rewards logged (s, a, r, s′) transitions extrapolation error on OOD actions

One more distinction worth nailing, because it trips people up: off-policy is not the same as offline. Off-policy methods (like Q-learning) learn about one policy from data generated by another — but they can still keep interacting with the environment to fix their mistakes. Offline RL is the strict case where interaction is switched off entirely. Every offline method is off-policy; not every off-policy method is offline.

💡 Which should you reach for?

Have a great expert and just want to match them fast? Imitation. Have a mixed dataset of past decisions with rewards and want to do better than the average behaviour in it? Offline RL. Can you safely experiment and have time to burn? Online RL still gives the strongest policies — it's the gold standard when you can afford it.

WEAKNESSCommon mistakes to avoid

✗ "Behavioral cloning generalizes perfectly — it learned from a perfect expert."

The expert may be perfect; the coverage isn't. BC only saw the states the expert visited, so it has no idea how to recover once it drifts somewhere new. A flawless expert plus a tiny agent error still equals a car off the road. That's distribution shift, and it's why DAgger and inverse RL exist.

✗ "You can just run normal Q-learning on a fixed dataset."

Naive off-policy Q-learning on logged data usually fails badly. The max in the target chases out-of-distribution actions whose values are wildly over-estimated and can never be corrected without interaction. You need offline-specific fixes — policy constraints or conservative methods like CQL — to keep the values honest.

✗ "Offline RL means you never touch an environment, ever."

Offline RL needs no environment during training — that's the whole point. But you still must evaluate the final policy before trusting it, and off-policy evaluation from a fixed dataset is notoriously hard. Most real deployments do a careful, limited online validation before going live. "No exploration while learning" ≠ "never check your work."

KEY INSIGHTWhere this connects in your course

  • Q-learning & off-policy (2.6) — the very update whose max term breaks offline; offline RL is what happens when you take its data away.
  • Deep Q-Networks (3.2) — the function approximation that makes extrapolation error so dangerous: neural nets confidently guess values for actions they've never seen.
  • RLHF — RL from human feedback (4.5) — the next lesson takes this idea further: learn from human preference data (which answer is better), a cousin of imitation and offline learning.
  • Real-world deployment (4.6) — where safety, evaluation, and "you can't just explore" stop being theory and become the whole job.

Both ideas in this lesson exist for the same reason: live exploration is a luxury you often can't afford. Imitation copies an expert; offline RL squeezes a policy from the past. Keep them in-distribution and they're powerful. Let them wander, and they fall apart in exactly the same way.

GLOSSARYKey terms introduced

Imitation learning
Learning a policy by copying expert demonstrations — (state, action) pairs — instead of from a reward signal.
Behavioral cloning (BC)
The simplest imitation method: supervised learning that predicts the expert's action from the state.
DAgger
Dataset Aggregation — iteratively collect the agent's own visited states and have the expert label them, fixing distribution shift.
Inverse RL (IRL)
Infer the reward function the expert is optimising, then run RL on it — recover the why, not just the what.
Offline RL (batch RL)
Learning a policy purely from a fixed logged dataset with rewards and no new environment interaction.
Distribution shift
The mismatch between the states/actions in the data and those the learned policy actually produces — the root cause of both methods' failures.
Extrapolation error
Over-optimistic value estimates for out-of-distribution actions that offline methods can never correct without interaction.
Conservative methods
Offline RL techniques (e.g. CQL) that penalise OOD-action values so the policy stays close to the data.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1What is the single problem that breaks both behavioral cloning and naive offline Q-learning?›

Distribution shift. In BC, the agent drifts into states the expert never demonstrated and has nothing to copy, so errors compound. In offline Q-learning, the max chases out-of-distribution actions whose values are over-estimated and can't be corrected. Both fail when learning leaves the region the data covers — the cure for both is to stay in-distribution.

2How does behavioral cloning differ from offline RL — and when would you pick each?›

BC uses no reward; it just copies expert (state, action) pairs, so it can only be as good as the expert. Offline RL uses logged rewards and tries to find a policy better than the data's average behaviour. Pick BC when you have a great expert and want speed; pick offline RL when you have a mixed dataset with rewards and want to improve on it.

3Why does the maxa Q(St+1,a) term cause trouble offline but not online?›

The max seeks the highest-valued next action. Online, if that value is an over-optimistic fantasy, the agent tries the action, gets a real reward, and corrects the estimate. Offline there's no interaction to correct it, so the fantasy value gets fed into targets and amplified across the value function — extrapolation error.

4Is "off-policy" the same as "offline"? Name a fix for offline RL's core problem.›

No. Off-policy means learning about one policy from another's data, but you can still interact. Offline means no interaction at all. Fixes: policy constraints (keep the policy close to the behaviour policy) or conservative value methods like CQL, which push down the values of out-of-distribution actions so the max can't be fooled.

Next: learning from human preferences instead of demonstrations →