Deep Q-Networks (DQN)
The breakthrough that learned to play Atari from raw pixels — Q-learning with a neural network, plus two clever tricks that made it actually work.
🎯 By the end you'll understand
How DQN swaps the Q-table for a neural network Q(s,a;θ), the squared-error loss it trains on (the TD target turned into a regression problem), and the two stabilising tricks — experience replay and a target network — that turned an unstable idea into the algorithm that mastered Atari.
KEY INSIGHTThe one idea to take away
DQN is Q-learning where the Q-table becomes a neural network — and two tricks, a replay buffer and a frozen target, are what keep that network from spiralling out of control.
In Lesson 2.6 you learned Q-learning: keep a table of Q(s,a) values and nudge each one toward a TD target. In Lesson 3.1 you saw why that table breaks the moment states get large — a screen of pixels has more configurations than atoms in the universe. DQN's answer is simple to say and hard to make stable: replace the table with a neural network. The whole lesson is about that swap, and the two repairs it needed.
Plain Q-learning + a neural network = a beautiful idea that diverges. DQN's contribution wasn't "use a network" — that was obvious. It was the two tricks that make the network actually converge: experience replay and a separate target network. Remember the idea and the repairs, or you've only got half of DQN.
CORE CONCEPTFrom a lookup table to a function
Think back to the cleaning robot's Q-table from Stage 2: a grid with one cell for every (state, action) pair, each holding a number you looked up and updated. That works beautifully for three rooms. It is hopeless for an Atari screen, where the "state" is 84×84 pixels — there is no table big enough, and you'd never visit the same screen twice anyway.
So instead of storing a value for every pair, DQN learns a function that computes one. A neural network takes the state as input and outputs a Q-value for each possible action at once. The network's weights — written θ — are the thing we train. We write the network's estimate as Q(s,a;θ): "the value of action a in state s, according to a network with weights θ."
A reinforcement-learning algorithm that approximates the action-value function with a neural network, Q(s,a;θ), and trains it with Q-learning-style updates. Introduced by DeepMind in 2015, it learned to play dozens of Atari games at human level directly from raw pixels, using the same network and hyperparameters for every game.
The shape matters. The network doesn't take an action as input — it takes only the state, and emits a whole vector of Q-values, one per action. That's a deliberate design choice: it means a single forward pass gives you the value of every move, so picking the best action (or computing maxa Q) is just reading off the largest output. One pass, all the answers.
FORMALThe loss it minimises
Here's the bridge from Stage 2. The tabular Q-learning update you already know is:
Read that bracket again. It's the difference between where we think we are, Q(St,At), and where the evidence says we should be — the TD target, Rt+1 + γ maxa Q(St+1,a). The update drags our estimate a fraction α toward that target.
A neural network can't be "nudged" cell by cell — it's trained by gradient descent on a loss. So we turn the same idea into a regression problem: make the network's output match the TD target, and measure the gap with squared error.
That's the whole training signal. The label we regress toward is y; the prediction is Q(s,a;θ); the loss is how far apart they are. Run gradient descent on L(θ) and the network learns to predict TD targets — which is exactly what Q-learning always did, now expressed as supervised-looking regression.
Look closely: the prediction uses the live weights θ, but the target uses a different set, θ⁻ (theta-minus). That tiny notation difference is one of the two tricks — a separate target network. We'll see in a moment why using the live θ on both sides is a recipe for disaster.
Notice the loss maxes over actions to build the target. That's the off-policy, value-iteration spirit of Q-learning carried straight into the deep setting — and, as we'll note at the end, it's also why DQN only works for discrete action sets.
WEAKNESSWhy the naive version falls apart
You might think we're done: build a network, compute the loss, run gradient descent. People tried exactly that for years, and it kept blowing up. In Lesson 3.1 you met the culprit by name — the deadly triad: combining function approximation, bootstrapping (targets built from your own estimates), and off-policy learning. Stack all three and value estimates can diverge to infinity. DQN has all three. Two specific symptoms cause most of the pain:
If you train on transitions in the order you experience them, frame 1001 looks almost identical to frame 1000. The network sees a long run of near-identical, highly correlated examples, overfits to whatever it's doing right now, and forgets everything else — like cramming for an exam by reading the same paragraph ten thousand times.
The target R + γ maxa′ Q(s′,a′;θ) depends on the very weights θ we're updating. So every gradient step moves the prediction and the target at the same time. You're a dog chasing its own tail: the goalposts move every step, and the whole thing can spiral away instead of settling.
Both problems have the same flavour — instability from feedback loops. DQN's two tricks are precise antidotes, one for each symptom.
FRAMEWORKThe two clever tricks
Here is the heart of the lesson. Two ideas, each fixing one of the symptoms above.
1 · Experience replay
Instead of learning from each transition once, in order, and throwing it away, DQN stores every transition (s, a, r, s′) in a big circular memory called the replay buffer (think: the last million transitions). Each training step samples a random minibatch from this buffer and learns from that.
Experience replay is the technique of saving past transitions and training on random samples of them rather than only the latest experience. The replay buffer is the fixed-size memory (a queue that overwrites its oldest entries) that holds them.
This one trick buys three things at once:
- Breaks correlation. A random minibatch mixes transitions from many different times and situations, so consecutive gradient steps no longer see near-identical data. The training distribution looks far more like the independent samples that gradient descent assumes.
- Reuses data. A single hard-won transition can be replayed in many minibatches, so the agent squeezes far more learning out of every interaction — crucial when real experience is expensive. This is what people mean by DQN being more sample-efficient than learning online once.
- Smooths the distribution. Averaging over a buffer of past behaviour stops sudden shifts in what the agent is currently doing from yanking the network around, which also prevents nasty oscillations.
2 · The target network
To stop the target from chasing itself, DQN keeps a second copy of the network — the target network, with weights θ⁻. The target y = R + γ maxa′ Q(s′,a′;θ⁻) is computed with these frozen weights, while gradients only ever update the live weights θ. Every C steps (typically a few thousand), the target network is refreshed: θ⁻ ← θ.
A slowly-updated copy of the Q-network used only to compute TD targets. Its weights θ⁻ are held fixed for a stretch of training and periodically synced to the live weights θ. This gives the network a stationary target to aim at for a while, instead of a target that moves every step.
The analogy: imagine practising archery at a target that lurches sideways every time you release an arrow — you'd never group your shots. Freeze the target between rounds and you can actually zero in. Periodically the instructor moves the target to a new (better) spot, and you re-zero. Stable target, then re-aim, repeat.
Experience replay fixes correlated, wasteful data by learning from a random sample of a memory. The target network fixes the moving target by freezing the weights used in the target. Together they tame the deadly triad enough for deep Q-learning to converge.
SEE ITWhy the tricks matter
Talk is cheap — watch the difference. Below is a hand-authored picture of training loss over time. Toggle the two tricks off and the loss is jagged and diverging: the network never settles. Toggle them on and the same training smooths out and converges. This is the qualitative behaviour every deep-RL practitioner has seen with their own eyes.
The shape is the lesson. Without the tricks, the feedback loops in the deadly triad amplify themselves and the loss climbs and thrashes. With them, each gradient step sees decorrelated data aimed at a stable target, so the error shrinks and stays down. Neither curve is real measured data — they're deterministic illustrations of behaviour that's been reproduced thousands of times.
EXAMPLEThe DQN training loop
Now assemble everything into the actual algorithm. Read it once for shape, then again noticing exactly where each trick appears.
# --- setup --- init Q-network with random weights θ init target network θ⁻ ← θ # identical copy at the start init replay buffer D (capacity ~1,000,000) ε = 1.0 # start fully exploratory for step in range(num_steps): # 1. act with ε-greedy exploration if rand() < ε: a = random_action() else: a = argmaxa Q(s, a; θ) # greedy w.r.t. live net # 2. step the environment, store the transition s′, r, done = env.step(a) D.store( (s, a, r, s′, done) ) # <-- experience replay s = s′ # 3. learn from a RANDOM minibatch states, actions, rewards, next_states, dones = D.sample(minibatch_size) # <-- decorrelated samples y = rewards + (1 − dones) · γ · maxa′ Q(next_states, a′; θ⁻) # target net! (0 for terminal) L = mean( (y − Q(states, actions; θ))² ) # squared TD error over the whole batch θ ← θ − lr · ∇θ L # gradient descent step # 4. periodically sync the target network if step % C == 0: θ⁻ ← θ # <-- target network refresh ε = max(ε_min, ε · decay) # anneal exploration over time
Four numbered beats, and two of them are the tricks: step 2–3 are experience replay (store, then sample randomly), and the highlighted target line plus step 4 are the target network (compute targets with θ⁻, refresh it every C steps). Strip those out and you have the naive version that diverges; keep them and you have DQN.
Press play to follow one transition through those four beats. Watch which set of weights each number comes from, and which set the gradient step is allowed to move.
Notice the target is just r when done is true: there's no future after the episode ends, so the bootstrapped γ max Q term is dropped. Forgetting this is a classic bug — it leaks phantom future value into terminal transitions.
CONTEXTThe DQN family — famous improvements
The 2015 DQN was the start, not the end. A wave of refinements each fixed one weakness. You don't need the math now — just recognise the names, because you'll meet them constantly:
| Variant | What it fixes (one line) |
|---|---|
| Double DQN | The max in the target systematically overestimates Q-values; it decouples action selection from evaluation to cut the bias. |
| Dueling DQN | Splits the network into a state-value stream and an advantage stream, so it learns which states are good even when the action barely matters. |
| Prioritized Experience Replay | Samples the replay buffer by TD error instead of uniformly, replaying surprising, informative transitions more often. |
| Rainbow | Combines all of the above (plus more) into one agent — the kitchen-sink DQN, and a strong Atari baseline. |
If Prioritized Experience Replay rings a bell — yes, it's the natural upgrade to the uniform replay buffer you just learned: not all experiences are equally worth replaying, so weight the surprising ones higher.
WEAKNESSCommon mistakes to avoid
That naive combination diverges. The entire reason DQN is a landmark is the two tricks: experience replay and the target network. Drop them and watch the loss spiral, exactly as in the widget above.
It can't. The target needs maxa′ Q(s′,a′), which means enumerating every action — fine for "left / right / fire", impossible for a steering angle that's any real number. Continuous control needs different methods (policy gradients, actor-critic, DDPG — coming in Lesson 3.6).
That's precisely the moving-target bug the target network exists to prevent. Targets must be computed with the frozen θ⁻; using the live θ on both sides reintroduces the self-chasing instability.
KEY INSIGHTWhere this connects in your course
- Q-learning (2.6) — DQN is this algorithm; the squared TD error in the loss is the tabular update reborn as gradient descent.
- Exploration & ε-greedy (2.8) — DQN still explores with ε-greedy, annealing ε from 1 down toward a small floor as it learns.
- When tables break (3.1) — the deadly triad you met there is exactly what the two tricks are built to tame.
- Policy gradients & REINFORCE (3.3) — next, a completely different family that learns the policy directly instead of values.
- Continuous control (3.6) — where the max-over-actions limitation finally forces us beyond DQN.
Hold the shape: Q-learning + a network + replay + a frozen target. That's DQN, and it's the foundation every value-based deep RL method builds on.
GLOSSARYKey terms introduced
- Deep Q-Network (DQN)
- Q-learning with the action-value function approximated by a neural network Q(s,a;θ), stabilised by experience replay and a target network.
- Experience replay
- Storing past transitions and training on random samples of them, to decorrelate data and reuse experience.
- Replay buffer
- The fixed-size memory (a queue that overwrites oldest entries) holding past (s,a,r,s′) transitions.
- Target network (θ⁻)
- A slowly-updated copy of the Q-network used only to compute TD targets, giving a stationary aim point.
- TD target
- The bootstrapped label the network regresses toward: R + γ maxa′ Q(s′,a′;θ⁻).
- Minibatch
- A small random set of transitions sampled from the replay buffer for one gradient step.
- Discrete actions
- A finite, enumerable action set — required by DQN because it must take a max over all actions.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What single change turns tabular Q-learning into DQN, and what is the resulting loss?›
Replace the Q-table with a neural network Q(s,a;θ). Training becomes gradient descent on the squared TD error, L(θ) = (y − Q(s,a;θ))2, where the target y = R + γ maxa′ Q(s′,a′;θ⁻).
2Name the two stabilising tricks and say which problem each one solves.›
Experience replay solves correlated, wasteful data — it samples random minibatches from a buffer of past transitions, decorrelating samples and reusing experience. The target network solves the moving-target problem — it computes targets with frozen weights θ⁻ so the prediction isn't chasing a target that shifts every step.
3Why can't DQN handle continuous action spaces?›
The target requires maxa′ Q(s′,a′), which means evaluating every possible action. With a continuous action (any real number) there are infinitely many, so the max isn't computable. Continuous control needs policy-based or actor-critic methods instead (Lesson 3.6).
4In the training loop, what value should the target take for a terminal transition, and why?›
Just the reward, y = r — no γ max Q term. There is no next state after the episode ends, so there is no future value to bootstrap from. Including it would leak phantom future reward into terminal states.