The Agent's Goal — Return & Discounting
The agent doesn't chase the next reward. It chases the biggest total over time. Meet the return — and discover why the future is worth a little less than the present.
🎯 By the end you'll understand
What the return Gt is and why the agent maximises it instead of the next reward, why we maximise the expected return when the world is uncertain, and how the discount factor γ shapes whether an agent is short-sighted or far-sighted — with a worked example you can compute by hand.
KEY INSIGHTThe one idea to take away
An RL agent isn't greedy for the next point. Its true goal is the biggest total reward over time — and it values rewards sooner more than rewards later.
Think of RL like a video game. You score points for doing good things and lose points for mistakes. The agent's job isn't to grab whatever point is closest — it's to rack up the highest score across the whole game. That running, cumulative score has a name: the return. And because the future is never guaranteed, the agent quietly prefers a point now to the same point later.
The reward Rt+1 is the immediate feedback for one action. The return Gt is the sum of all future rewards from now on. The agent maximises the return — and to keep things sane and realistic, it discounts rewards that arrive further in the future by a factor γ.
INTUITIONScoring points over time
Remember your little sister learning to ride the bike? Every smooth metre earned her a banana — a +1 — and every fall cost her a −1. We said her goal was to "collect as many bananas as possible." Let's take that seriously, because it's the whole point of this lesson.
Her goal was never a single banana. If chasing one banana right now meant swerving wildly and crashing on the next metre, that's a terrible trade — one banana, then a fall, then probably more falls. What she really wants is the biggest pile of bananas by the end of the ride. Sometimes that means going a little slower now (no banana this instant) so she can ride smoothly for the next ten metres (ten bananas later).
That's the same logic in every game you've ever played. In chess you'll happily sacrifice a pawn — a small loss now — to win the game later. In a racing game you brake before the corner — losing speed now — to take the whole lap faster. The score you care about is the final score, the accumulation of everything, not the reward of the single move in front of you.
Whenever a good decision "costs" something now but pays off later, you're seeing the difference between immediate reward and return. RL is built around that difference.
CORE CONCEPTThe return: Gt
Let's give that "running total" a precise name and symbol. Starting from some time step t, the return is the sum of all the rewards the agent collects from then on:
Read it left to right: from where the agent stands now, add up the reward it gets next step, plus the step after that, plus the one after that, all the way to the end. That single number — the future total — is what the agent is actually trying to make as large as possible.
The cumulative reward the agent collects from time step t onward. It's the quantity the agent maximises — not any single reward, but the whole future sum.
So when the author asked, "Why does the agent aim to maximise the return?", the answer is now clear: because the return captures the best long-term result. Maximising the return forces the agent to balance immediate gains against future opportunities, instead of grabbing whatever looks shiniest this second. A move worth nothing now but setting up a big payoff later still increases the return — so the agent learns to make it.
One small but important detail about the indices. The reward for the action taken at step t is written Rt+1 — it arrives on the next tick of the clock, because the environment has to respond first. That's why the sum starts at Rt+1 and not Rt. (We met this convention back in the RL loop.)
FORMALWhy we maximise the expected return
Here's a wrinkle the clean formula above hides. The future is uncertain. Your sister doesn't know in advance exactly how the next ten metres will go — sometimes a gust of wind, a bump in the road, or a slightly different action sends things one way or another. The same action in the same state can lead to different rewards on different days.
When the outcome is uncertain, you can't maximise a single, known total — there isn't one. Instead, the agent maximises the average return it can expect, over all the ways the future might unfold. We write that with the expectation symbol 𝔼 ("the expected value of"):
In words: "choose behaviour that makes the expected total future reward as large as possible." If a risky move sometimes wins huge and sometimes crashes, the expectation weighs both outcomes by how likely they are. This is why RL agents can look so sensible — they're optimising an average over an uncertain future, not gambling everything on a single lucky run.
The agent maximises the return (a sum over time, not one reward) and, because the future is uncertain, it specifically maximises the expected return 𝔼[Gt] (an average over all possible futures).
CORE CONCEPTDiscounting the future
Now the most important idea in this lesson. Ask yourself an honest question: would you rather have a banana now, or the exact same banana next week?
You'd take it now — and you'd be right to. A banana now is certain; a banana next week might never arrive (you could be away, it could spoil, plans change). The future is always a little less trustworthy than the present. RL bakes that intuition directly into the goal using a discount factor called γ (gamma), a number between 0 and 1.
Each step you look further into the future, you multiply the reward by another factor of γ. The reward next step counts fully; the one after counts γ times as much; the one after that γ2; and so on. The result is the discounted return:
A number γ ∈ [0, 1] that shrinks the value of future rewards. A reward arriving k steps from now is worth only γk of its face value to the agent today.
Why bother discounting at all? Three solid reasons, and they reinforce each other:
Uncertainty
The further ahead a reward is, the less sure we are it'll actually happen. Discounting says "trust the near future more than the far future" — exactly the banana-now instinct.
The maths stays finite
Some tasks never end (a robot that runs forever). Adding up 1 + 1 + 1 + … blows up to ∞. But with γ < 1 the geometric sum 1 + γ + γ2 + … converges to a finite number, so "the return" is always well-defined.
A preference for sooner
All else equal, getting reward earlier is genuinely better — it's available, reinvestable, less risky. Discounting encodes "sooner beats later," the same reason a dollar today beats a dollar next year.
That convergence point deserves a number. With a steady +1 reward every step forever and discount γ, the geometric series adds up to a tidy closed form:
So an endless stream of +1s with γ = 0.9 is worth 1 / (1 − 0.9) = 10 — a finite, sensible number, even though the rewards never stop. That's the magic of γ.
COMPARISONWhat γ actually does
The discount factor is a single dial that sets the agent's personality — how far ahead it cares to look. Slide it from 0 to 1 and you change the whole character of the agent.
| γ value | Personality | What it cares about | Real-world feel |
|---|---|---|---|
| γ → 0 | Myopic (greedy) | only the very next reward Rt+1 | impulsive — grab the closest banana, ignore tomorrow |
| γ ≈ 0.5 | balanced-ish | near rewards matter a lot, far ones fade fast | thinks a few moves ahead |
| γ → 1 | Far-sighted | the whole future, almost undiscounted | patient planner — willing to wait for a big payoff |
When γ = 0, every term except the first vanishes (0k = 0 for k ≥ 1), so Gt = Rt+1 — the agent literally cannot see past the next step. When γ creeps toward 1, distant rewards keep almost their full value, and the agent will gladly forgo points now to chase a jackpot many steps away.
Low γ = short-sighted, reacts to what's immediately in front of it. High γ = long-term thinker, plans toward distant goals. Most real RL uses something like γ = 0.9 to 0.99 — far-sighted, but not blind to risk.
SEE ITThe discount explorer
Here's the dial in your hands. Below is a small, fixed sequence of future rewards. Drag γ and watch two things change at once: the discounted return Gt, and the height of each bar — each bar is one step's contribution γk·R to the total. See how a low γ crushes the far-future bars to almost nothing, while a high γ keeps the whole future alive.
Play with the two patterns at the extremes. At γ near 0, "steady +1s" wins easily — its reward shows up now, while the big late reward is discounted into dust. Crank γ toward 1 and the "big reward at the end" pulls ahead, because the patient agent finally values that distant jackpot. The same agent, the same rewards — only γ changed — and the best strategy flipped. That's the discount factor doing its quiet, powerful work.
EXAMPLEComputing a discounted return by hand
Let's nail it down with arithmetic you can check on paper. Suppose from time t the agent will receive these three rewards, in order: 0, then 0, then +10. (Think: two nothing steps, then a treasure.) Take γ = 0.9.
The treasure is two steps away, so it's discounted by γ2 = 0.81. The agent still values it highly: 8.1 out of a possible 10. A far-sighted agent will happily wait two steps for that reward.
Now turn the dial down to a more impatient γ = 0.5 and recompute the very same rewards:
Same treasure, same two-step wait — but now it's worth only 2.5. To this impatient agent, a reward of +3 available right now would actually look better than the +10 two steps out. Different γ, different priorities, possibly a completely different choice. That, in two short calculations, is why γ is one of the most consequential knobs in all of RL.
Rewards [2, 2, 2] with γ = 0.9. Compute Gt. (Answer: 2 + 0.9·2 + 0.81·2 = 2 + 1.8 + 1.62 = 5.42.) Notice each banana is worth a little less than the last.
WEAKNESSCommon mistakes to avoid
No — it maximises the return Gt, the discounted sum of all future rewards. Chasing the next +1 is exactly the greedy, short-sighted behaviour you get only in the degenerate case γ = 0. Real agents trade small losses now for big gains later.
Not quite. γ = 1 is fine for tasks that are guaranteed to end (episodic tasks), but for never-ending tasks it lets the return blow up to ∞, and it can make learning unstable. It also ignores real-world uncertainty about the far future. Discounting (γ < 1) is usually the safer, more realistic default.
They're different. The reward (and the reward function that produces it) is the immediate signal for a single step. The return is the accumulation of those rewards over time. One reward feeds into the return; the return is the goal built out of many rewards.
KEY INSIGHTWhere this connects in your course
- What RL is (1.1) — we promised the agent's aim was the biggest total, not the next reward. This lesson is that promise made precise: the total is the return.
- The RL loop (1.2) — the loop produces a stream of rewards Rt+1, Rt+2, …; the return is what you get when you sum that stream (with discounting).
- Markov Decision Processes (1.4) — the MDP gives the formal frame where γ and the return live as part of the problem definition.
- Value functions (2.3) — a value V(s) is literally the expected return from a state. Everything you just learned about 𝔼[Gt] becomes the definition of "value."
Hold this one line: the agent maximises the expected, discounted return — 𝔼[Gt] with γ. Almost every algorithm in the rest of the course is a clever way to estimate or improve that quantity.
GLOSSARYKey terms introduced
- Return (Gt)
- The cumulative (discounted) reward from time t onward — what the agent maximises.
- Cumulative reward
- Another name for the total reward summed over time; the undiscounted return.
- Expected return (𝔼[Gt])
- The average return over all the ways an uncertain future might unfold — the agent's true objective.
- Discount factor (γ)
- A number in [0,1] that shrinks future rewards; a reward k steps away is worth γk of its value today.
- Horizon
- How far into the future the agent effectively cares about. Finite/episodic horizons end; infinite horizons rely on γ < 1 to stay finite.
- Myopic agent
- A short-sighted agent (small γ) that cares mostly about immediate reward.
- Far-sighted agent
- A patient agent (large γ) that values long-term reward and plans toward distant payoffs.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What is the return Gt, and why does the agent maximise it instead of the next reward?›
The return is the sum of all future rewards from step t on: Gt = Rt+1 + Rt+2 + … (discounted by γ). The agent maximises it because it captures the best long-term result — balancing immediate gains against future opportunities, instead of greedily grabbing the next point.
2Why do we maximise the expected return rather than just "the return"?›
Because the future is uncertain — the same action can lead to different rewards. There's no single guaranteed total, so the agent maximises 𝔼[Gt], the average return over all possible futures, weighting each outcome by how likely it is.
3Give three reasons we discount future rewards with γ.›
(1) Uncertainty — far-off rewards are less sure to arrive. (2) Convergence — for never-ending tasks, γ < 1 keeps the infinite sum finite (it converges, e.g. Σγk = 1/(1−γ)). (3) Preference for sooner — a reward now is genuinely better than the same reward later.
4Rewards [0, 0, 10]. Compute Gt for γ = 0.9 and for γ = 0.5. What does the difference tell you?›
γ = 0.9: 0 + 0.9·0 + 0.81·10 = 8.1. γ = 0.5: 0 + 0.5·0 + 0.25·10 = 2.5. The patient agent (γ=0.9) values the distant treasure highly; the impatient agent (γ=0.5) barely cares about it. Higher γ = more far-sighted.