Offline RL & Imitation Learning
Trial and error is risky and expensive in the real world. What if the agent could learn from a fixed dataset, or by copying an expert, instead of exploring? Two ways to learn without (much) interaction.
🎯 By the end you'll understand
Two ways to learn a policy without live trial and error: imitation learning (copy an expert's demonstrations) and offline RL (squeeze a policy out of a fixed dataset of past experience). You'll see why both keep tripping over the same villain — distribution shift — and how each one tries to tame it.
KEY INSIGHTThe one idea to take away
Sometimes the cheapest, safest teacher isn't the environment — it's a pile of data someone already collected. The whole challenge is learning from it without wandering off the edge of what that data actually covers.
Everything in Stages 2 and 3 assumed the agent could act in the world — try things, fail, and learn from the consequences. But in healthcare, finance, or a real robot, every clumsy exploratory action has a cost (sometimes a human one). So a different question becomes urgent: can we learn a good policy without exploring? Yes — two ways. Copy an expert, or mine logged experience. Both work, and both share one nemesis.
Online RL collects its own data by exploring. Imitation learning and offline RL learn from data they did not collect — expert demos or logged history. The price of skipping exploration is distribution shift: the moment your policy strays into situations the data never showed, it's flying blind.
CONTEXTWhy you'd ever avoid trial and error
Remember teaching your little sister to ride a bike? That worked because falling over costs a scraped knee and a banana, not a catastrophe. Trial and error is wonderful when mistakes are cheap.
But now imagine the "agent" is a system recommending insulin doses for diabetic patients, or trading a retirement fund, or steering a two-tonne car. You cannot let it "explore" — try a wild dose just to see what happens — because the cost of a bad action is measured in harm, money, or lives. Exploration here is dangerous, expensive, or simply not allowed.
And yet, in all these settings, data already exists: hospital records of what doctors did and how patients fared, trading logs, millions of miles of human driving, billions of past clicks. The dream is to turn that existing data into a good policy — no risky live experiments required.
Imitation learning assumes the old data came from a good expert and just copies it. Offline RL makes no such assumption — it takes a possibly-mediocre dataset and tries to extract the best policy it can, using rewards, the way ordinary RL would.
CORE CONCEPTPart A — Imitation learning
Here's the most human way to learn a skill: watch someone good at it, and copy them. You didn't learn to drive by randomly mashing pedals until the reward signal sorted you out — you watched a parent drive for years, then imitated. Imitation learning gives an agent that same shortcut.
Learning a policy from a set of expert demonstrations — recorded (state, action) pairs showing what a skilled agent did — instead of from a reward signal. The goal is to reproduce the expert's behaviour.
Notice what's missing: there's no reward function at all. The teaching signal isn't "this outcome was good (+1)"; it's "in this situation, the expert chose this action." That looks a lot like supervised learning — and the simplest method leans into exactly that.
Behavioral cloning — supervised learning in disguise
Treat imitation as plain supervised learning: collect the expert's (state, action) pairs as a labelled dataset, then train a model to predict the expert's action from the state. The state is the input, the expert's action is the label.
That's genuinely it. If your sister had a camera on her shoulder recording a champion cyclist — "when the bike leans left, steer slightly left", "when a curb appears, slow down" — she could fit a model mapping each scene to the rider's response. Simple, fast, no environment interaction needed during training. So what's the catch?
WEAKNESSBehavioral cloning & the drift problem
The catch is subtle and it's the heart of this whole lesson. Behavioral cloning only ever saw the states the expert visited — and a good expert mostly stays on the road, in the middle of the lane, doing sensible things. The dataset is a thin ribbon of "good situations."
Now the cloned agent drives. It's good, but not perfect, so it makes a tiny error — drifts six inches toward the lane edge. The expert never drifted there, so there are no demonstrations for "slightly off-centre." The model has no idea what to do. It guesses, drifts a little more, and now it's further from anything it was trained on. Each mistake pushes it into less-familiar territory, where it makes bigger mistakes. The errors snowball — and the car drives off the road.
When a learned policy acts, it visits states drawn from its own behaviour — which differs from the distribution of states in the training data. Small errors move the agent into unseen states, where it errs more, moving it further still. The error compounds over a trajectory instead of staying small.
In ordinary supervised learning, each prediction is independent — one wrong label doesn't change the next input. In control, your action determines your next state. So an error doesn't just cost you one mistake; it changes the world you face next, dragging you off the data manifold where every future prediction also degrades.
Theory makes this precise: if BC has even a small per-step error rate, the total error over a horizon of T steps can grow like T2 in the worst case — far worse than the T you'd hope for. That quadratic blow-up is the compounding-error problem.
DAgger — let the expert label the agent's own mistakes
The classic fix is DAgger (Dataset Aggregation). The idea is delightfully direct: let the cloned agent drive, record the new (off-distribution) states it stumbles into, then ask the expert what they'd do in those states. Add those fresh labels to the dataset, retrain, repeat. Over a few rounds the dataset grows to cover exactly the recovery situations BC never saw — so the agent learns how to get back on the road. The cost: you need ongoing access to the expert to label new states.
Inverse RL — recover the why, not just the what
Inverse RL (IRL) takes a deeper route: instead of copying actions, it tries to infer the reward function that the expert seems to be optimising, then runs ordinary RL on that recovered reward. Learn why the expert acts, and you can generalise to new situations they never demonstrated — at the price of a much harder, more expensive learning problem.
Behavioral cloning = memorise "in this exact view, do what the instructor did." DAgger = also ask the instructor what to do when you've drifted. Inverse RL = figure out what the instructor was trying to achieve (stay centred, smooth, safe) and drive toward that goal yourself.
CORE CONCEPTPart B — Offline RL (batch RL)
Imitation learning has one big assumption baked in: the demonstrations are good. But what if your data is just… history — a mixed bag of decisions, some smart, some dumb, each with a recorded reward? You don't want to copy it. You want to do the RL thing: find the policy that earns the most reward, learning from those logged consequences. That's offline RL.
Learning a good policy purely from a fixed dataset of logged transitions (s, a, r, s′), with no new interaction with the environment during training. Unlike imitation, it uses the reward and aims to improve on the behaviour that generated the data, not merely reproduce it.
This is the holy grail for high-stakes domains. A hospital's records become training data for a treatment policy; a recommender's click logs train a better recommender; the robot learns from yesterday's runs overnight, with zero risk. No exploration. No environment. Just the dataset.
You might think: "We already have off-policy algorithms — Q-learning learns about one policy from data generated by another. Just point Q-learning at the logged dataset and we're done!" It's a completely reasonable thought. It also fails, often spectacularly, and understanding why is the key insight of offline RL.
FORMALThe distribution-shift trap (extrapolation error)
Recall Q-learning's update from Stage 2. The bootstrap target uses the value of the best next action:
That maxa Q(St+1,a) quietly asks: "what's the value of the best possible action in the next state?" Online, if the agent gets seduced by an action with an inflated value, it simply tries it, sees a low reward, and corrects the estimate. The environment is a constant fact-checker.
Offline, there is no fact-checker. Suppose the dataset never contains action a* in state St+1. The network has never been corrected there, so its estimate Q(St+1, a*) is pure extrapolation — and neural nets extrapolate optimistically, often wildly so. The max then seeks out exactly these over-valued, never-seen actions and feeds their fantasy values back into the targets. Errors don't get corrected; they get amplified and propagated across the whole value function.
The policy is drawn toward out-of-distribution (OOD) actions precisely because their values are unrealistically high — and offline there's no way to check. This is distribution shift again, wearing a value-function costume: the policy you're learning wants to do things the dataset never did, and your value estimates for those things are fiction.
The fix: stay close to the data
Every offline RL method is, at heart, one instruction: don't trust values for actions the data doesn't support. Two broad strategies:
Policy constraint
Force the learned policy to stay close to the data-generating (behaviour) policy — only pick actions the dataset actually contains in that state. No straying onto unseen actions, so no fantasy values to chase.
Conservative value estimates
Deliberately penalise the values of OOD actions — push them down so the max can't be fooled by them. CQL (Conservative Q-Learning) is the headline example: it learns a Q-function that lower-bounds the truth, refusing to be optimistic about the unseen.
Imitation learning fails when the agent drifts off the data. Offline RL fails when the value estimates drift off the data. Same disease — distribution shift — and the same cure: stay in-distribution. The art of offline RL is improving on the data while never wandering past its edges.
SEE ITWhy copying breaks
Let's watch behavioral cloning drift. Below, the expert's path is the green ribbon of states the demonstrations covered. The cloned agent tries to follow it. Toggle between a tiny early slip and a clean run, then press play, or step forward one square at a time, and watch what happens once it leaves the ribbon. You can step back or drag the slider to see any step again.
In the slip scenario, one small early error puts the agent on a square the expert never visited (red). With no demonstration to copy there, its next guess is worse, the error grows, and it tumbles off the road — pure compounding error. In the clean run it never leaves the green ribbon, so every step has a demonstration to copy and it sails through. Same policy, same expert; the only difference is whether it stayed in-distribution.
COMPARISONOnline RL vs imitation vs offline RL
Three ways to get a policy, side by side. The columns that matter most are the last two: what each one requires, and what bites you if you're careless.
| Approach | Needs live interaction? | Needs a reward signal? | Learns from | Key risk |
|---|---|---|---|---|
| Online RL Stages 2–3 |
Yes — explores & collects its own data | Yes | its own trial-and-error experience | exploration is slow, costly, or unsafe |
| Imitation (BC) | No (training); but DAgger needs the expert | No — uses expert actions, not rewards | expert (state, action) demos | distribution shift / compounding drift |
| Offline RL | No — a fixed logged dataset | Yes — uses logged rewards | logged (s, a, r, s′) transitions | extrapolation error on OOD actions |
One more distinction worth nailing, because it trips people up: off-policy is not the same as offline. Off-policy methods (like Q-learning) learn about one policy from data generated by another — but they can still keep interacting with the environment to fix their mistakes. Offline RL is the strict case where interaction is switched off entirely. Every offline method is off-policy; not every off-policy method is offline.
Have a great expert and just want to match them fast? Imitation. Have a mixed dataset of past decisions with rewards and want to do better than the average behaviour in it? Offline RL. Can you safely experiment and have time to burn? Online RL still gives the strongest policies — it's the gold standard when you can afford it.
WEAKNESSCommon mistakes to avoid
The expert may be perfect; the coverage isn't. BC only saw the states the expert visited, so it has no idea how to recover once it drifts somewhere new. A flawless expert plus a tiny agent error still equals a car off the road. That's distribution shift, and it's why DAgger and inverse RL exist.
Naive off-policy Q-learning on logged data usually fails badly. The max in the target chases out-of-distribution actions whose values are wildly over-estimated and can never be corrected without interaction. You need offline-specific fixes — policy constraints or conservative methods like CQL — to keep the values honest.
Offline RL needs no environment during training — that's the whole point. But you still must evaluate the final policy before trusting it, and off-policy evaluation from a fixed dataset is notoriously hard. Most real deployments do a careful, limited online validation before going live. "No exploration while learning" ≠ "never check your work."
KEY INSIGHTWhere this connects in your course
- Q-learning & off-policy (2.6) — the very update whose max term breaks offline; offline RL is what happens when you take its data away.
- Deep Q-Networks (3.2) — the function approximation that makes extrapolation error so dangerous: neural nets confidently guess values for actions they've never seen.
- RLHF — RL from human feedback (4.5) — the next lesson takes this idea further: learn from human preference data (which answer is better), a cousin of imitation and offline learning.
- Real-world deployment (4.6) — where safety, evaluation, and "you can't just explore" stop being theory and become the whole job.
Both ideas in this lesson exist for the same reason: live exploration is a luxury you often can't afford. Imitation copies an expert; offline RL squeezes a policy from the past. Keep them in-distribution and they're powerful. Let them wander, and they fall apart in exactly the same way.
GLOSSARYKey terms introduced
- Imitation learning
- Learning a policy by copying expert demonstrations — (state, action) pairs — instead of from a reward signal.
- Behavioral cloning (BC)
- The simplest imitation method: supervised learning that predicts the expert's action from the state.
- DAgger
- Dataset Aggregation — iteratively collect the agent's own visited states and have the expert label them, fixing distribution shift.
- Inverse RL (IRL)
- Infer the reward function the expert is optimising, then run RL on it — recover the why, not just the what.
- Offline RL (batch RL)
- Learning a policy purely from a fixed logged dataset with rewards and no new environment interaction.
- Distribution shift
- The mismatch between the states/actions in the data and those the learned policy actually produces — the root cause of both methods' failures.
- Extrapolation error
- Over-optimistic value estimates for out-of-distribution actions that offline methods can never correct without interaction.
- Conservative methods
- Offline RL techniques (e.g. CQL) that penalise OOD-action values so the policy stays close to the data.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What is the single problem that breaks both behavioral cloning and naive offline Q-learning?›
Distribution shift. In BC, the agent drifts into states the expert never demonstrated and has nothing to copy, so errors compound. In offline Q-learning, the max chases out-of-distribution actions whose values are over-estimated and can't be corrected. Both fail when learning leaves the region the data covers — the cure for both is to stay in-distribution.
2How does behavioral cloning differ from offline RL — and when would you pick each?›
BC uses no reward; it just copies expert (state, action) pairs, so it can only be as good as the expert. Offline RL uses logged rewards and tries to find a policy better than the data's average behaviour. Pick BC when you have a great expert and want speed; pick offline RL when you have a mixed dataset with rewards and want to improve on it.
3Why does the maxa Q(St+1,a) term cause trouble offline but not online?›
The max seeks the highest-valued next action. Online, if that value is an over-optimistic fantasy, the agent tries the action, gets a real reward, and corrects the estimate. Offline there's no interaction to correct it, so the fantasy value gets fed into targets and amplified across the value function — extrapolation error.
4Is "off-policy" the same as "offline"? Name a fix for offline RL's core problem.›
No. Off-policy means learning about one policy from another's data, but you can still interact. Offline means no interaction at all. Fixes: policy constraints (keep the policy close to the behaviour policy) or conservative value methods like CQL, which push down the values of out-of-distribution actions so the max can't be fooled.