ProjekinAja. Reinforcement Learning Stage 1 · Foundations Lesson 1.3
● Stage 1 · Lesson 1.3

The Agent's Goal — Return & Discounting

The agent doesn't chase the next reward. It chases the biggest total over time. Meet the return — and discover why the future is worth a little less than the present.

⏱ ~14 min read 🎯 What the agent maximises γ Discounting

🎯 By the end you'll understand

What the return Gt is and why the agent maximises it instead of the next reward, why we maximise the expected return when the world is uncertain, and how the discount factor γ shapes whether an agent is short-sighted or far-sighted — with a worked example you can compute by hand.

KEY INSIGHTThe one idea to take away

An RL agent isn't greedy for the next point. Its true goal is the biggest total reward over time — and it values rewards sooner more than rewards later.

Think of RL like a video game. You score points for doing good things and lose points for mistakes. The agent's job isn't to grab whatever point is closest — it's to rack up the highest score across the whole game. That running, cumulative score has a name: the return. And because the future is never guaranteed, the agent quietly prefers a point now to the same point later.

🔑 Hold onto this

The reward Rt+1 is the immediate feedback for one action. The return Gt is the sum of all future rewards from now on. The agent maximises the return — and to keep things sane and realistic, it discounts rewards that arrive further in the future by a factor γ.

INTUITIONScoring points over time

Remember your little sister learning to ride the bike? Every smooth metre earned her a banana — a +1 — and every fall cost her a −1. We said her goal was to "collect as many bananas as possible." Let's take that seriously, because it's the whole point of this lesson.

Her goal was never a single banana. If chasing one banana right now meant swerving wildly and crashing on the next metre, that's a terrible trade — one banana, then a fall, then probably more falls. What she really wants is the biggest pile of bananas by the end of the ride. Sometimes that means going a little slower now (no banana this instant) so she can ride smoothly for the next ten metres (ten bananas later).

That's the same logic in every game you've ever played. In chess you'll happily sacrifice a pawn — a small loss now — to win the game later. In a racing game you brake before the corner — losing speed now — to take the whole lap faster. The score you care about is the final score, the accumulation of everything, not the reward of the single move in front of you.

💡 The tell

Whenever a good decision "costs" something now but pays off later, you're seeing the difference between immediate reward and return. RL is built around that difference.

CORE CONCEPTThe return: Gt

Let's give that "running total" a precise name and symbol. Starting from some time step t, the return is the sum of all the rewards the agent collects from then on:

Gt = Rt+1 + Rt+2 + Rt+3 + … the (undiscounted) return

Read it left to right: from where the agent stands now, add up the reward it gets next step, plus the step after that, plus the one after that, all the way to the end. That single number — the future total — is what the agent is actually trying to make as large as possible.

📖 Return (Gt)

The cumulative reward the agent collects from time step t onward. It's the quantity the agent maximises — not any single reward, but the whole future sum.

So when the author asked, "Why does the agent aim to maximise the return?", the answer is now clear: because the return captures the best long-term result. Maximising the return forces the agent to balance immediate gains against future opportunities, instead of grabbing whatever looks shiniest this second. A move worth nothing now but setting up a big payoff later still increases the return — so the agent learns to make it.

One small but important detail about the indices. The reward for the action taken at step t is written Rt+1 — it arrives on the next tick of the clock, because the environment has to respond first. That's why the sum starts at Rt+1 and not Rt. (We met this convention back in the RL loop.)

FORMALWhy we maximise the expected return

Here's a wrinkle the clean formula above hides. The future is uncertain. Your sister doesn't know in advance exactly how the next ten metres will go — sometimes a gust of wind, a bump in the road, or a slightly different action sends things one way or another. The same action in the same state can lead to different rewards on different days.

When the outcome is uncertain, you can't maximise a single, known total — there isn't one. Instead, the agent maximises the average return it can expect, over all the ways the future might unfold. We write that with the expectation symbol 𝔼 ("the expected value of"):

maximise   𝔼[Gt] the agent's true objective

In words: "choose behaviour that makes the expected total future reward as large as possible." If a risky move sometimes wins huge and sometimes crashes, the expectation weighs both outcomes by how likely they are. This is why RL agents can look so sensible — they're optimising an average over an uncertain future, not gambling everything on a single lucky run.

🔑 Two upgrades in one sentence

The agent maximises the return (a sum over time, not one reward) and, because the future is uncertain, it specifically maximises the expected return 𝔼[Gt] (an average over all possible futures).

CORE CONCEPTDiscounting the future

Now the most important idea in this lesson. Ask yourself an honest question: would you rather have a banana now, or the exact same banana next week?

You'd take it now — and you'd be right to. A banana now is certain; a banana next week might never arrive (you could be away, it could spoil, plans change). The future is always a little less trustworthy than the present. RL bakes that intuition directly into the goal using a discount factor called γ (gamma), a number between 0 and 1.

Each step you look further into the future, you multiply the reward by another factor of γ. The reward next step counts fully; the one after counts γ times as much; the one after that γ2; and so on. The result is the discounted return:

Gt = Rt+1 + γRt+2 + γ2Rt+3 + … = Σk=0∞ γk Rt+k+1 the discounted return
📖 Discount factor (γ)

A number γ ∈ [0, 1] that shrinks the value of future rewards. A reward arriving k steps from now is worth only γk of its face value to the agent today.

Why bother discounting at all? Three solid reasons, and they reinforce each other:

🎲

Uncertainty

The further ahead a reward is, the less sure we are it'll actually happen. Discounting says "trust the near future more than the far future" — exactly the banana-now instinct.

∞

The maths stays finite

Some tasks never end (a robot that runs forever). Adding up 1 + 1 + 1 + … blows up to ∞. But with γ < 1 the geometric sum 1 + γ + γ2 + … converges to a finite number, so "the return" is always well-defined.

⏳

A preference for sooner

All else equal, getting reward earlier is genuinely better — it's available, reinvestable, less risky. Discounting encodes "sooner beats later," the same reason a dollar today beats a dollar next year.

That convergence point deserves a number. With a steady +1 reward every step forever and discount γ, the geometric series adds up to a tidy closed form:

1 + γ + γ2 + … = 1 / (1 − γ) geometric sum (for γ < 1)

So an endless stream of +1s with γ = 0.9 is worth 1 / (1 − 0.9) = 10 — a finite, sensible number, even though the rewards never stop. That's the magic of γ.

COMPARISONWhat γ actually does

The discount factor is a single dial that sets the agent's personality — how far ahead it cares to look. Slide it from 0 to 1 and you change the whole character of the agent.

γ valuePersonalityWhat it cares aboutReal-world feel
γ → 0Myopic (greedy)only the very next reward Rt+1impulsive — grab the closest banana, ignore tomorrow
γ ≈ 0.5balanced-ishnear rewards matter a lot, far ones fade fastthinks a few moves ahead
γ → 1Far-sightedthe whole future, almost undiscountedpatient planner — willing to wait for a big payoff

When γ = 0, every term except the first vanishes (0k = 0 for k ≥ 1), so Gt = Rt+1 — the agent literally cannot see past the next step. When γ creeps toward 1, distant rewards keep almost their full value, and the agent will gladly forgo points now to chase a jackpot many steps away.

💡 Rule of thumb

Low γ = short-sighted, reacts to what's immediately in front of it. High γ = long-term thinker, plans toward distant goals. Most real RL uses something like γ = 0.9 to 0.99 — far-sighted, but not blind to risk.

SEE ITThe discount explorer

Here's the dial in your hands. Below is a small, fixed sequence of future rewards. Drag γ and watch two things change at once: the discounted return Gt, and the height of each bar — each bar is one step's contribution γk·R to the total. See how a low γ crushes the far-future bars to almost nothing, while a high γ keeps the whole future alive.

γ Interactive · How discounting reshapes the return

Slide the discount factor, then try switching the reward pattern. Notice which pattern "wins" at low vs high γ.

0.90

Pattern: a steady +1 reward at every one of the next 8 steps.

Discounted return Gt
—
Undiscounted total (γ=1)
—
Personality
—

Play with the two patterns at the extremes. At γ near 0, "steady +1s" wins easily — its reward shows up now, while the big late reward is discounted into dust. Crank γ toward 1 and the "big reward at the end" pulls ahead, because the patient agent finally values that distant jackpot. The same agent, the same rewards — only γ changed — and the best strategy flipped. That's the discount factor doing its quiet, powerful work.

EXAMPLEComputing a discounted return by hand

Let's nail it down with arithmetic you can check on paper. Suppose from time t the agent will receive these three rewards, in order: 0, then 0, then +10. (Think: two nothing steps, then a treasure.) Take γ = 0.9.

Gt = 0 + 0.9 · 0 + 0.92 · 10 = 0 + 0 + 0.81 · 10 = 8.1 γ = 0.9 → patient agent

The treasure is two steps away, so it's discounted by γ2 = 0.81. The agent still values it highly: 8.1 out of a possible 10. A far-sighted agent will happily wait two steps for that reward.

Now turn the dial down to a more impatient γ = 0.5 and recompute the very same rewards:

Gt = 0 + 0.5 · 0 + 0.52 · 10 = 0.25 · 10 = 2.5 γ = 0.5 → impatient agent

Same treasure, same two-step wait — but now it's worth only 2.5. To this impatient agent, a reward of +3 available right now would actually look better than the +10 two steps out. Different γ, different priorities, possibly a completely different choice. That, in two short calculations, is why γ is one of the most consequential knobs in all of RL.

✏️ Try it yourself

Rewards [2, 2, 2] with γ = 0.9. Compute Gt. (Answer: 2 + 0.9·2 + 0.81·2 = 2 + 1.8 + 1.62 = 5.42.) Notice each banana is worth a little less than the last.

WEAKNESSCommon mistakes to avoid

✗ "The agent should maximise the immediate reward."

No — it maximises the return Gt, the discounted sum of all future rewards. Chasing the next +1 is exactly the greedy, short-sighted behaviour you get only in the degenerate case γ = 0. Real agents trade small losses now for big gains later.

✗ "γ = 1 is always the right choice."

Not quite. γ = 1 is fine for tasks that are guaranteed to end (episodic tasks), but for never-ending tasks it lets the return blow up to ∞, and it can make learning unstable. It also ignores real-world uncertainty about the far future. Discounting (γ < 1) is usually the safer, more realistic default.

✗ "The return is the same thing as the reward function."

They're different. The reward (and the reward function that produces it) is the immediate signal for a single step. The return is the accumulation of those rewards over time. One reward feeds into the return; the return is the goal built out of many rewards.

KEY INSIGHTWhere this connects in your course

  • What RL is (1.1) — we promised the agent's aim was the biggest total, not the next reward. This lesson is that promise made precise: the total is the return.
  • The RL loop (1.2) — the loop produces a stream of rewards Rt+1, Rt+2, …; the return is what you get when you sum that stream (with discounting).
  • Markov Decision Processes (1.4) — the MDP gives the formal frame where γ and the return live as part of the problem definition.
  • Value functions (2.3) — a value V(s) is literally the expected return from a state. Everything you just learned about 𝔼[Gt] becomes the definition of "value."

Hold this one line: the agent maximises the expected, discounted return — 𝔼[Gt] with γ. Almost every algorithm in the rest of the course is a clever way to estimate or improve that quantity.

GLOSSARYKey terms introduced

Return (Gt)
The cumulative (discounted) reward from time t onward — what the agent maximises.
Cumulative reward
Another name for the total reward summed over time; the undiscounted return.
Expected return (𝔼[Gt])
The average return over all the ways an uncertain future might unfold — the agent's true objective.
Discount factor (γ)
A number in [0,1] that shrinks future rewards; a reward k steps away is worth γk of its value today.
Horizon
How far into the future the agent effectively cares about. Finite/episodic horizons end; infinite horizons rely on γ < 1 to stay finite.
Myopic agent
A short-sighted agent (small γ) that cares mostly about immediate reward.
Far-sighted agent
A patient agent (large γ) that values long-term reward and plans toward distant payoffs.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1What is the return Gt, and why does the agent maximise it instead of the next reward?›

The return is the sum of all future rewards from step t on: Gt = Rt+1 + Rt+2 + … (discounted by γ). The agent maximises it because it captures the best long-term result — balancing immediate gains against future opportunities, instead of greedily grabbing the next point.

2Why do we maximise the expected return rather than just "the return"?›

Because the future is uncertain — the same action can lead to different rewards. There's no single guaranteed total, so the agent maximises 𝔼[Gt], the average return over all possible futures, weighting each outcome by how likely it is.

3Give three reasons we discount future rewards with γ.›

(1) Uncertainty — far-off rewards are less sure to arrive. (2) Convergence — for never-ending tasks, γ < 1 keeps the infinite sum finite (it converges, e.g. Σγk = 1/(1−γ)). (3) Preference for sooner — a reward now is genuinely better than the same reward later.

4Rewards [0, 0, 10]. Compute Gt for γ = 0.9 and for γ = 0.5. What does the difference tell you?›

γ = 0.9: 0 + 0.9·0 + 0.81·10 = 8.1.  γ = 0.5: 0 + 0.5·0 + 0.25·10 = 2.5. The patient agent (γ=0.9) values the distant treasure highly; the impatient agent (γ=0.5) barely cares about it. Higher γ = more far-sighted.

You now know what the agent is really chasing. Next: the formal frame — MDPs →