Context — The Agent's Working Memory
An agent only knows what's in its context window right now. Understand this one thing and 90% of agent weirdness suddenly makes sense.
🎯 By the end you'll understand
What the context window is and why it's finite, how an agent's loop fills it up fast, what happens when it overflows (the agent literally forgets), why huge tool outputs are dangerous, and why context is not long-term memory — it's wiped every session unless you re-supply it.
KEY INSIGHTThe one idea to take away
An agent doesn't "know" anything. On each turn it only sees what's written on its context window — and that whiteboard has a fixed size.
There's no hidden brain quietly remembering your conversation. Every time the model takes a turn, it is handed a block of text — the system prompt, the goal, the tool definitions, and the whole history so far — and it responds to only that. Nothing else exists to it. That block is the context window, and it is the single most important mechanic to internalize in this entire stage.
The model is stateless. It has no memory between turns of its own. The "memory" you perceive is just the running transcript being re-sent every single turn. When that transcript gets too big to fit, the agent doesn't get a warning — it just silently loses the parts that fell off.
DEFINITIONWhat context actually is
You met the loop in Lesson 2.1 and tools in Lesson 2.2. The context window is the surface all of that is written onto. Picture a whiteboard of fixed size: everything the model needs to reason about must fit on it, and when it's full, you have to erase something to write something new.
On any given turn, that whiteboard holds four kinds of thing:
The system prompt
The standing instructions — who the agent is, its rules, its style. Written once, re-sent every turn.
The goal
The task you actually asked for. "Fix the failing test in auth.py."
The tool definitions
The menu of tools it can call, each with a name, description, and input schema. The model has to read these to know what's available.
The history so far
Every action it has taken and every observation it got back — the entire loop transcript, accumulating turn by turn.
The context window is the total span of text a model can "see" and reason over in a single turn. It includes the system prompt, the goal, the tool definitions, and the full running history. It has a hard maximum size, measured in tokens, and that size is finite.
A token is the chunk of text a model actually counts — roughly ¾ of a word in English (so ~750 words ≈ 1,000 tokens). The context window's size is a token budget: a model with a "200K context" can hold about 200,000 tokens of system prompt + goal + tools + history at once, and not one more.
So when someone says a model has a "200K context window," they mean the whiteboard holds about 200,000 tokens — perhaps 150,000 words. Big, but not infinite. And in an agent, you fill it faster than you'd think.
One more thing worth burning in: tokens cost money and time. Every token in the window is re-processed on every turn, so a bloated context isn't just a memory problem — it makes each step slower and more expensive. A tidy window is a faster, cheaper, more accurate agent. That's three good reasons to care, not one.
CORE CONCEPTWhat fills it up — and why it's fast
A chatbot conversation grows slowly: a question, an answer, a question, an answer. An agent is different, because every pass through the loop dumps two new things onto the whiteboard:
- The action — the model's reasoning plus the tool call it decided to make. Usually small.
- The observation — whatever the tool handed back. This can be enormous: a full web page, a 500-line file, a giant JSON blob, a directory listing, a stack trace.
And here's the part that surprises people: the whole history is re-sent on every turn. The model is stateless, remember — so to let it "see" turn 1 when it's on turn 8, the framework pastes turns 1 through 7 back in, every time. Context doesn't grow linearly with what's interesting; it grows with everything that ever happened.
Turn 1: read auth.py (480 lines) → 6,000 tokens of file land in context. Turn 2: run the test suite → 4,000 tokens of output. Turn 3: grep the repo → 3,000 tokens of matches. Before it has changed a single line, ~13,000 tokens are gone — and all of it gets re-sent on turn 4, and turn 5, and turn 6.
This is why long agent runs feel like they "slow down" or "lose the plot" near the end. The whiteboard is getting crowded, and the original goal — written way up at the top — is now buried under a mountain of tool output.
WEAKNESSWhat happens when it's full
The context window is a hard ceiling. You cannot exceed it — the request simply won't fit. So when an agent's running history grows past the budget, something has to give. Two things typically happen, neither of them gentle:
Truncation / eviction
The framework drops the oldest content to make room — usually the start of the transcript. The agent literally forgets the beginning of the task: the original instructions, early decisions, why it was doing this at all.
Summarization
A smarter system compresses old turns into a short summary before dropping them — "I read these files, made these changes." Better, but lossy: detail is gone, and summaries can quietly distort what really happened.
There's a subtler failure even before you hit the ceiling. Research has consistently found that models attend best to the beginning and end of their context, and worst to the middle. Bury a crucial instruction in the middle of a huge transcript and the model may glide right past it — a phenomenon nicknamed "lost in the middle."
When content is evicted, the model isn't told. It just sees a shorter transcript and acts confidently on incomplete information. A coding agent might re-edit a file it already fixed, or contradict a constraint you gave it at the start — not because it's "dumb," but because that instruction no longer exists in its world.
So much of what looks like an agent "going rogue," "ignoring instructions," or "losing focus" is really just this: the relevant text fell off the whiteboard. Once you see context, those mysteries dissolve.
SEE ITContext Window Filler
This whiteboard holds 1,000 tokens (tiny, so it fills fast — real ones are far bigger, but the dynamic is identical). Add items and watch the budget. When you go over capacity, the oldest items at the top get evicted — greyed out and struck through. The agent can no longer see them.
Try it: add the system prompt, goal, and tool defs, then keep clicking Add a loop turn. Notice how fast tool output eats the budget. Then hit Paste a huge document on a nearly-full window and watch the top of the task vanish.
PRACTICALContext engineering, in one breath
If context is finite and precious, then managing what goes in it is a real skill — arguably the skill for getting agents to work reliably. It even has a name now: context engineering. You'll go deep on it in Lesson 3.3; here's the shape of it:
Put the right things in
The instructions, examples, and facts the agent genuinely needs for this step — and a crisp restatement of the goal so it stays near the top, where attention is strong.
Leave the junk out
Don't paste a whole 50-page PDF when one paragraph matters. Don't dump a 5,000-line log when the agent needs the last error. Filter before it hits the window.
Summarize as you go
Compress old, settled turns into short notes so the window keeps room for what's next. This is how long-running agents survive past their token budget.
Stop thinking "the model is smart, just tell it everything." Start thinking like a director of attention: what does it need to see, right now, to take the next good step? Everything else is noise competing for a finite budget.
CORE CONCEPTContext is not long-term memory
This is the trap that catches almost everyone. Because the agent seems to "remember" your conversation, it's natural to assume it remembers you. It doesn't. The context window is working memory, not storage — like the few papers on your desk, not the filing cabinet behind you.
When a session ends, the whiteboard is wiped clean. Start a new chat tomorrow and the agent has no idea who you are, what you worked on, or what it just spent an hour doing — unless something re-supplies that information into the fresh window.
It didn't. Nothing the model "experienced" in one session changes the model. The only way it "knows" yesterday's facts is if a system fetches them and pastes them back into today's context. That fetching-and-pasting is exactly what memory and retrieval (RAG) are — your next lesson.
This is also why the same agent can feel brilliant in one session and clueless the next. Nothing changed about its intelligence — what changed is what got loaded onto the whiteboard before it started. Give it the right context and it shines; start it blank and it gropes. The variable is almost never the model; it's the window.
So the bridge to Lesson 2.4 is this: long-term memory is the machinery that decides what to load back into the context window, so a stateless model can act like it remembers. Context is the stage; memory is the crew deciding what props to put on it.
WEAKNESSCommon mistakes to avoid
Only what currently fits on the whiteboard. Older turns get evicted or summarized away, and a new session starts blank. It remembers what's in the window now — nothing more.
Bigger budgets help, but they're still finite — and "lost in the middle" means a huge window crammed full can actually reduce accuracy. More room is not a substitute for putting the right things in it.
A giant document or raw tool dump can blow the entire budget in one shot, evicting the goal and instructions you actually care about. Big outputs are dangerous, not free. Filter and summarize first.
KEY INSIGHTWhere this connects in your course
- The loop (2.1) — every pass adds an action + observation to context; that's why it fills up turn by turn.
- Tools (2.2) — tool definitions take up space, and tool outputs are the biggest thing flooding the window.
- Memory & RAG (2.4) — the machinery that re-supplies the window so a stateless model can "remember" across sessions.
- Reasoning & self-correction (2.6) — gets harder as the window fills and the goal sinks toward the lossy middle.
- Context engineering (3.3) — the practical craft of curating what's in the window, previewed here.
- The LLM engine (1.3) — tokens and statelessness come from how the model itself works.
Hold the picture: a finite whiteboard, re-read every turn, that the loop keeps filling. Once you see context, agents stop being magic and start being mechanism.
GLOSSARYKey terms introduced
- Context window
- The total span of text a model can see and reason over in a single turn — system prompt, goal, tool definitions, and full history. Finite, measured in tokens.
- Token
- The unit of text a model counts; roughly ¾ of a word in English. Window size is a token budget.
- System prompt
- The standing instructions defining who the agent is and how it behaves, re-sent in context every turn.
- Eviction / truncation
- Dropping the oldest content from the window when it overflows, so new content fits. The agent silently loses the dropped part.
- Lost in the middle
- The tendency of models to attend well to the start and end of context but poorly to the middle, so buried instructions get missed.
- Context engineering
- The skill of curating what goes into the window — adding what's needed, leaving junk out, summarizing — to keep an agent reliable (Lesson 3.3).
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What four kinds of thing live in an agent's context window?›
The system prompt (standing instructions), the goal (the task), the tool definitions (its menu of tools), and the full history of actions and observations so far. All of it is re-sent on every turn.
2Why does an agent's context fill up so much faster than a normal chat?›
Every loop turn adds an action and an observation, and tool outputs (files, web pages, logs) can be huge. Because the model is stateless, the entire history is re-sent each turn — so context grows with everything that ever happened, not just what's relevant.
3What happens when the window overflows, and why is it dangerous?›
The oldest content is evicted or summarized away — often the original goal and instructions. The agent isn't told, so it acts confidently on incomplete information. Combined with "lost in the middle," this is why agents seem to forget or ignore instructions on long runs.
4Why is context not the same as long-term memory?›
Context is working memory, wiped at the end of each session. The model is stateless — it doesn't retain anything between sessions. To "remember" across sessions, a separate memory/RAG system must re-supply the relevant facts back into a fresh context window (Lesson 2.4).