Where RL Wins & The Road Ahead
You started at "what is a reward?" and ended at deep, multi-agent, human-aligned RL. This final lesson zooms all the way out: where RL genuinely shines, where it quietly struggles, and where you go from here.
🎯 By the end you'll understand
The whole arc of what you learned in one mental map, the real domains where RL is the right tool (games, robotics, operations, aligning LLMs), the honest places it falls down (sample cost, reward design, sim-to-real), where the field is heading, and a concrete step-by-step ladder to take you from reader to practitioner.
KEY INSIGHTThe one idea, one last time
Everything you learned — value functions, Bellman, Q-learning, PPO, RLHF — is one idea wearing different clothes: learn by trial and error to maximise reward.
Back in Lesson 1.1 you taught your little sister to ride a bike. Bananas were +1, falling was −1, and nobody told her the "correct" move — she discovered it from the reward alone. Every algorithm since has just been a sharper, more scalable way of doing exactly that. AlphaGo and a person-following robot and ChatGPT's helpfulness training are all the bike, scaled up.
RL is not magic and it is not a hammer for every nail. It is a single, powerful idea — maximise long-run reward through interaction — that shines brilliantly on a specific shape of problem (sequential decisions, a clear reward, lots of experience available) and struggles badly elsewhere. Knowing which shape you have is the real skill.
FRAMEWORKThe journey you took
Before we look forward, let's look back — because you covered a lot of ground, and seeing it as one connected map is itself a skill. Here is the whole course in four stages.
| Stage | The question it answered | What you learned |
|---|---|---|
| 1 · Foundations | What is this, and what are the pieces? | Agent, environment, state, action, reward; the RL loop; the return and discounting by γ; the Markov Decision Process that frames it all. |
| 2 · How the agent learns | How does it actually get better? | Policy π and value functions V(s)/Q(s,a); the Bellman equations; TD learning; Q-learning (off-policy) & SARSA (on-policy); the exploration vs exploitation trade-off. |
| 3 · Deep RL | How does it scale to real, huge problems? | Function approximation with neural nets; DQN; policy gradients; actor-critic; PPO; continuous control for robots and physics. |
| 4 · Advanced & applied | What does grown-up RL look like? | Model-based RL & world models; reward hacking & safety; multi-agent RL; offline RL & imitation; RLHF — aligning large language models. |
Read that table top to bottom and you can feel the thread: we asked what is the goal (Stage 1), then how do we reach it (Stage 2), then how do we scale it (Stage 3), then how do we make it safe and useful in the messy real world (Stage 4). The one sentence never changed — learn by trial and error to maximise reward — we just kept making it more precise and more powerful.
EXAMPLEWhere RL genuinely wins
RL earns its reputation on problems that are sequential (a chain of decisions, not one), where there's a clear reward to chase, and where you can gather a lot of experience — either from a fast simulator or from a system you're allowed to poke at. Four domains fit that mould beautifully.
1 · Games. This is where RL first dazzled the world, because games come with a built-in reward (the score, or win/lose) and a perfect simulator (the game itself). DeepMind's DQN learned to play dozens of Atari games from raw pixels. AlphaGo beat a world champion at Go in 2016, and its successor AlphaZero learned Go, chess, and shogi to superhuman level from self-play alone, no human games. OpenAI Five reached top-tier play in Dota 2, and DeepMind's AlphaStar hit Grandmaster in StarCraft II — both messy, real-time, partially observed games. These were the proofs of concept that made everyone take RL seriously.
2 · Robotics & control. Continuous control (Stage 3) lets RL move bodies. RL has trained locomotion (legged robots learning to walk and recover from shoves), manipulation (grasping and rearranging objects), and remarkable dexterity — OpenAI famously trained a robot hand to solve a Rubik's cube. Control problems like this are a natural fit because a physics simulator can generate the millions of trials RL needs. This is one of your own areas: a person-following robot — keeping a target in view and at a safe distance using LIDAR and camera — is a sequential control problem with a clear reward (stay locked on, don't collide), exactly RL's home turf.
3 · Recommendation & operations. Anywhere a system makes a long stream of decisions to optimise a long-run outcome, RL fits. DeepMind used RL to cut data-centre cooling energy at Google. RL has been applied to chip design (placing components to optimise a layout), to logistics and scheduling, and to recommendation (optimising long-term engagement, not just the next click). The common thread: a sequence of decisions with a measurable objective. Another of your areas lives here too: finance — the FinRL framework applies RL to trading and portfolio management, treating "buy / sell / hold over time to maximise risk-adjusted return" as exactly the sequential decision problem RL was built for.
4 · Aligning large language models. The newest and arguably most consequential win is the one you met in Lesson 4.5: RLHF. A pretrained language model is fine-tuned with RL so that its outputs maximise a reward learned from human preferences. This is how raw text predictors became helpful, honest, harmless assistants. It's RL on a problem with no score function at all — the reward had to be learned from people, which is a beautiful illustration of how far the one idea stretches.
Sequential decisions · a reward you can define or learn · enough experience to learn from (a simulator, a game engine, logged data, or human feedback). When a problem has that shape, RL is often the best tool there is. When it doesn't, read the next section carefully.
WEAKNESSWhere RL struggles — honestly
It would be a disservice to send you off thinking RL is a finished, plug-and-play technology. It is powerful but finicky, and a huge part of being a real practitioner is knowing its failure modes. Here are the big ones, side by side with where RL wins.
Clear, cheap reward signal · a fast simulator or game engine · millions of trials are affordable · sequential decisions where long-term planning matters · mistakes during training are harmless (it's just a simulation).
Reward is vague or must be hand-crafted · real-world trials are slow, costly, or dangerous · you only have a fixed log of data · a simple supervised model or rule would do · mistakes during learning hurt real people or hardware.
Sample inefficiency. This is the headline weakness. RL agents often need millions to billions of interactions to get good. That's fine in a simulator that runs faster than real time, but brutal anywhere each trial costs real money, real time, or real wear on a robot. Humans learn to drive in ~20 hours; an RL agent might need centuries of equivalent experience.
Reward design is hard — and dangerous. You saw this in Lesson 4.2: reward hacking. An agent maximises exactly what you wrote down, not what you meant. A cleaning robot rewarded for "no visible mess" might learn to switch off its camera. The reward function is your steering wheel, and a tiny mis-specification sends the agent confidently in the wrong direction.
The sim-to-real gap. Train a robot policy in a perfect simulation and it can fail the instant it meets the real world — real friction, sensor noise, latency, and lighting are never quite what the simulator modelled. Closing this gap (with domain randomisation, better simulators, fine-tuning on real data) is one of the central headaches of applied robotics RL.
Instability & reproducibility. Deep RL is notoriously sensitive. The same algorithm with a different random seed, a slightly different reward scale, or a small hyperparameter change can swing from "solved it" to "learned nothing." Published results can be hard to reproduce — a real, documented pain point in the field.
Safety in high-stakes settings. RL learns by trying things. In a game, a bad action costs a point. In a hospital, a power grid, or a self-driving car, a bad action during exploration can be catastrophic. This is why offline RL and careful constraints matter so much for real deployment.
Sparse, long-horizon credit assignment. When the reward arrives only at the very end (you won the game / the molecule worked), the agent must figure out which of thousands of earlier actions deserve the credit. The longer the horizon and the sparser the signal, the harder this gets.
RL is the right tool for a specific class of problems, and a frequent over-reach for everything else. A well-tuned supervised model, a bandit, or even a hand-written rule will often beat RL with a fraction of the effort and risk. Reach for RL when the problem is genuinely sequential, the reward is real, and you can afford the experience — not by default.
CONTEXTThe frontier — measured, not hyped
RL is a fast-moving field, so treat this as a snapshot that will keep evolving. But several directions are clearly where the energy is going, and each one is essentially an attack on a weakness from the section above.
Better sample efficiency & world models
Model-based RL (Lesson 4.1) lets an agent imagine rollouts inside a learned model of the world instead of paying for every real trial — directly fighting sample inefficiency.
Offline RL for real-world data
Learning good policies from fixed logged data — no risky live exploration — which is what makes RL usable in healthcare, finance, and other high-stakes domains.
RL + foundation models / LLMs
Agents that reason and act: language models that use tools, plan, and take actions, with RL refining their behaviour. RLHF was the opening move; "agentic" RL is the active frontier.
Generalist agents
Single agents that handle many tasks and environments rather than one narrow game — moving from a Go specialist toward something broadly capable.
Better exploration
Smarter ways to seek out informative experiences (curiosity, intrinsic motivation) so agents stop flailing randomly on sparse, long-horizon problems.
Safer reward & alignment
Methods to specify what we actually want and to keep agents within safe bounds — the research response to reward hacking and high-stakes risk.
Notice that the frontier isn't a list of flashy demos — it's the field maturing, methodically turning each of RL's weaknesses into an open research problem. That's a healthy sign, and a great map of where a curious newcomer can contribute.
SEE ITYour next RL project
Reading is not the same as doing — and the single best thing you can do now is build something small that runs. Tell the widget what you're drawn to and how confident you feel, and it'll hand you a concrete first project, the lessons worth re-reading, and the exact tool to start with.
Whatever it suggested, the rule is the same: start smaller than feels satisfying, get it running, then make it harder. A CartPole that balances is worth ten architecture diagrams you never coded.
FRAMEWORKA practice ladder — reader to practitioner
Here is the path I'd give anyone leaving this course. Climb it in order; each rung makes the next one make sense.
Implement tabular Q-learning on a gridworld — by hand
A tiny grid, a Q-table, and the update from Stage 2. No libraries. When you watch the Q-values fill in and a path emerge, the whole of Stage 2 clicks permanently. This is the most valuable hour you'll spend.
Meet the standard tools: Gymnasium + Stable-Baselines3
Gymnasium is the standard library of RL environments (a common env.step(action) interface). Stable-Baselines3 gives you battle-tested DQN, PPO, and SAC implementations. Together they let you train a real agent in a dozen lines.
Reproduce a classic: CartPole → Atari
Solve CartPole with PPO (it trains in minutes — your "hello world"). Then climb to an Atari game with DQN. Reproducing known results teaches you the unglamorous real skills: tuning, debugging, and patience with instability.
Build a project in your domain
Now make it yours. Robotics? Try a continuous-control task (e.g. a locomotion env) and read up on sim-to-real. Finance? Explore FinRL and frame a trading/portfolio task. This is where it stops being a course and starts being a skill.
Keep your own knowledge base
Lean on your own RL knowledge base as a study resource and add to it as you build. Re-deriving Bellman or the Q-update from your own notes — not from memory of a video — is how understanding turns into mastery.
It's tempting to jump straight to rung 3 with a fancy environment. Don't skip rung 1. The people who truly understand RL almost all wrote a from-scratch Q-learning loop at some point — it's where the abstractions become real.
WEAKNESSCommon mistakes to avoid
It isn't. RL shines on sequential decision problems with a real reward and affordable experience. For most prediction tasks, a supervised model is simpler, cheaper, and better. Diagnose the shape of your problem before reaching for RL — over-using it is the most common beginner error.
The sim-to-real gap says otherwise. Real friction, noise, latency, and lighting break simulator-trained policies all the time. Plan for domain randomisation, real-world fine-tuning, and safety margins — never deploy a sim policy to hardware and assume it'll just work.
Finishing means you now understand RL end to end — which is genuinely a lot. But mastery comes from implementing and experimenting: writing the Q-loop, fighting the instability, watching CartPole finally balance. The course was the map; the territory is the code you haven't written yet.
KEY INSIGHTHow it all connects
- Where it began (1.1) — What Is Reinforcement Learning?: the bike, the bananas, and the one sentence that runs through everything you just finished.
- Exploration (2.8) — Exploration vs Exploitation: the trade-off that becomes "better exploration" on the frontier and underlies every hard sparse-reward problem.
- Continuous control (3.6) — Continuous Control: the deep-RL machinery that powers the robotics and locomotion wins above.
- RLHF (4.5) — RL from Human Feedback: aligning LLMs, the newest and most consequential application — and the launch pad for agentic RL.
- Your domains — robotics (your person-following robot is a sequential control problem RL was built for) and finance (FinRL frames trading/portfolio management as exactly the kind of long-horizon reward maximisation you now understand). Both are excellent first real projects.
And the thread that ties all of it together is still the very first thing you learned: learn by trial and error to maximise reward. You can now follow that one idea from a child on a bike all the way to a language model aligned to human values. That's the whole field, and you understand it.
GLOSSARYKey terms introduced
- Sample efficiency
- How much interaction (experience) an algorithm needs to learn well. RL is often sample-inefficient — it needs huge amounts of data — which is a central weakness and research focus.
- Sim-to-real gap
- The mismatch between a simulator and the real world that causes a policy trained in simulation to fail on real hardware. A core challenge in applied robotics RL.
- Gymnasium
- The standard Python library of RL environments, providing a common interface (reset / step) for training and benchmarking agents. (The maintained successor to OpenAI Gym.)
- Stable-Baselines3
- A popular open-source library of reliable, ready-to-use implementations of RL algorithms (DQN, PPO, SAC, and more) built on PyTorch.
- Generalist agent
- An agent designed to perform many different tasks or operate in many environments, rather than mastering a single narrow one — a frontier goal of the field.
- Credit assignment
- The problem of figuring out which earlier actions deserve credit (or blame) for a later reward — especially hard when rewards are sparse or arrive at the very end of a long episode.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What "shape" of problem is RL the right tool for — and name one place it's the wrong tool?›
RL fits sequential decision problems with a clear (or learnable) reward and affordable experience (a simulator, game engine, logged data, or human feedback). It's the wrong tool for a plain prediction task with labelled answers — a supervised model is simpler, cheaper, and usually better there.
2Give three honest weaknesses of RL and, for each, the frontier direction tackling it.›
Sample inefficiency → model-based RL / world models (imagine rollouts instead of paying for them). Risky live exploration → offline RL (learn from fixed logged data). Reward hacking / unsafe behaviour → safer reward design and alignment research. (Also valid: sim-to-real → better simulators & domain randomisation; sparse credit assignment → better exploration.)
3Why is "implement tabular Q-learning on a gridworld by hand" the recommended first rung — before using a library?›
Because writing the Q-table and the update yourself makes the abstractions of Stage 2 concrete: you watch the values propagate and a good path emerge from nothing but the reward. Libraries hide that machinery; doing it by hand once is where the real understanding locks in, and it makes every later (library-based) step make sense.
4Why is finishing this course not the same as mastering RL — and what closes the gap?›
The course gave you genuine end-to-end understanding — the map. Mastery is the territory: implementing and experimenting. Writing the Q-loop, wrestling with instability, tuning a PPO run, watching CartPole finally balance, then building a project in your own domain (robotics or finance). Understanding plus repeated building is what turns into mastery.