Giving Your Agent Memory
Your agent forgets everything the moment the program ends. Memory fixes that — from simply saving the conversation to a lightweight RAG you can build yourself, in plain Python, no library required.
🎯 By the end you'll understand
The difference between context (the messages list, gone on restart) and memory (persistence across runs), and three concrete ways to add memory in code: save the conversation to a file, keep a facts store you inject into the system prompt, and build a minimal keyword-RAG retriever yourself — plus the retrieve → augment → generate flow and its honest limits.
KEY INSIGHTThe one idea to take away
Memory isn't a feature the model has — it's a habit you build: write the right things down, and read the right things back into context before you ask.
In Lesson 4.4 you saw the model is stateless: it remembers a conversation only because you keep re-sending the messages list. But that list lives in memory and vanishes when the program exits. Restart, and your agent has amnesia — it has never met you.
Real memory means writing something to disk and reading it back next run. That's it. There's no magic store inside the model — there's a file you save, and a decision about what to load back into the limited context window (2.3) when you do. This lesson makes the abstract memory ideas from Lesson 2.4 concrete, in code you can run today.
Memory is two moves: save (persist something useful across runs) and retrieve (pull the relevant bit back into context before the model call). Everything from a JSON file to a billion-document vector database is a fancier version of those same two moves.
DEFINITIONContext is short-term; memory is long-term
These two words get muddled constantly, so let's pin them down — they're the spine of the whole lesson.
| Context (short-term) 🧮 | Memory (long-term) 💾 | |
|---|---|---|
| What it is | the messages list you send each call | data saved to disk, outside the program |
| Lives where | in RAM, while the program runs | in a file (or database) that outlives the run |
| Survives a restart? | no — gone the moment it exits | yes — that's the whole point |
| Size limit | the context window (2.3) — small, finite | effectively unlimited (it's just disk) |
| The catch | everything in it costs tokens, every call | you must choose what to load back in |
Persistence means data outlives the process that created it. A variable in your program is not persistent — kill the program and it's gone. Write that variable to a file and it is persistent: it's still there next run, next week, next reboot. Giving an agent memory is, at bottom, just making the right things persistent.
So the design question is never "does my agent have memory?" — it's "what do I persist, and what do I load back into the limited context, and when?" We'll answer that in three escalating levels.
PRACTICALLevel 1 — Persist the conversation
The simplest possible memory: take the messages list you already have and save it to a file. Next run, load it back before the loop starts. Now the agent continues the conversation instead of starting blank.
The tool for this is serialization — turning an in-memory object into text you can write to disk, then back again. For a list of dicts, Python's built-in json module does it in two lines: json.dump to write, json.load to read.
import json, os PATH = "history.json" def load_messages(): # read history back at startup if os.path.exists(PATH): # first run? no file yet with open(PATH) as f: return json.load(f) # JSON text → list of dicts return [] # start empty if nothing saved def save_messages(messages): # write history at the end (or each turn) with open(PATH, "w") as f: json.dump(messages, f, indent=2) # list of dicts → JSON text on disk
Now wrap your chat loop from 4.4 with these two calls: load at the top, save at the bottom.
messages = load_messages() # ← was: messages = [] (now it remembers) while True: user = input("You: ") if user.lower() in ("quit", "exit"): save_messages(messages) # persist before we leave break messages.append({"role": "user", "content": user}) response = client.messages.create( model="claude-sonnet-5", max_tokens=1024, messages=messages, ) reply = response.content[0].text print("Bot:", reply) messages.append({"role": "assistant", "content": reply})
That single edit — messages = load_messages() instead of messages = [] — is the difference between an agent that has met you and one that hasn't. Simple, and genuinely effective for a personal assistant.
Reloading the entire history works until it doesn't: a months-long conversation will overflow the context window and cost a fortune in tokens every call (2.3). Level 1 is a great start, not a final answer. That tension — keep everything vs. keep only what matters — is exactly what Levels 2 and 3 solve.
CORE CONCEPTLevel 2 — A facts store
Most of a conversation is throwaway. What you actually want to carry forward are a handful of durable facts: the user's name, their timezone, their preferences, the project they're working on. Instead of replaying the whole transcript, keep a tiny facts store — a dictionary saved to disk — and inject only those facts into the system prompt at the start of each run.
import json, os FACTS_PATH = "facts.json" def load_facts(): if os.path.exists(FACTS_PATH): with open(FACTS_PATH) as f: return json.load(f) # e.g. {"name": "Ada", "tz": "GMT"} return {} def remember_fact(key, value): # call this when you learn something durable facts = load_facts() facts[key] = value with open(FACTS_PATH, "w") as f: json.dump(facts, f, indent=2) def facts_prompt(): # turn the dict into a system-prompt line facts = load_facts() if not facts: return "You are a helpful assistant." lines = [f"- {k}: {v}" for k, v in facts.items()] return "You are a helpful assistant. Known facts about the user:\n" + "\n".join(lines)
Now you pass system=facts_prompt() to the model call. The agent starts every run already knowing "the user is Ada, timezone GMT" — using a few dozen tokens instead of a whole replayed transcript. You decide what's worth remembering; the store stays small and sharp.
Level 1 persists the raw conversation. Level 2 persists distilled facts. Both are "save to a file, load it back" — the difference is how much you distill before saving. The more you distill, the cheaper and sharper the memory. That instinct — distill, don't dump — carries straight into Level 3.
FRAMEWORKLevel 3 — Build your own RAG
What if there's too much to fit in the system prompt — a notebook of meeting notes, a wiki, a year of documents? You can't inject all of it; the context window won't hold it and you'd pay for thousands of tokens you don't need. So you store everything on disk, and at query time you retrieve only the few pieces relevant to the current question. That is RAG — Retrieval-Augmented Generation — which you met conceptually in Lesson 2.4. Here you build a working one yourself.
Keep a knowledge store outside the model. On each query, retrieve the most relevant chunks, augment the prompt by pasting them into context, then let the model generate an answer grounded in them. The model's "memory" becomes "whatever you retrieved this turn." No retraining, no fine-tuning — just smart loading.
The only hard part is retrieval: given a question, which documents are relevant? The simplest scorer that works is keyword overlap — count how many words the question and a document share. More shared words, more likely relevant. It's crude, but it's real, it's deterministic, and you can write it from scratch:
import re def words(text): # lowercase, strip punctuation → set of words return set(re.findall(r"[a-z0-9]+", text.lower())) def score(query, doc): # how many words do they share? return len(words(query) & words(doc)) # & = set intersection def retrieve(query, docs, top_k=2): # return the best-matching docs scored = [(score(query, d), d) for d in docs] scored.sort(reverse=True) # highest score first return [d for s, d in scored[:top_k] if s > 0] # top_k, drop zero-overlap
Thirteen lines, no dependencies, and it genuinely retrieves. retrieve("what's my dog's name?", docs) ranks every note by shared words and hands back the top two that actually overlap. Drop in your documents and it works:
docs = [ "The user's dog is named Pixel, a border collie.", "The user prefers meetings in the afternoon, never before 11am.", "The user is allergic to shellfish and peanuts.", ] query = "what is my dog called?" hits = retrieve(query, docs) # → ["The user's dog is named Pixel, ..."] context = "\n".join(hits) # only the relevant note(s) system = f"Use these notes to answer:\n{context}" response = client.messages.create( model="claude-sonnet-5", max_tokens=512, system=system, # augment: the retrieved notes go here messages=[{"role": "user", "content": query}], ) print(response.content[0].text) # "Your dog is named Pixel." — grounded in the note
Notice the system prompt holds only the matching note, not all three. That's the entire discipline of managing context (3.3): retrieve the right bit, don't dump everything.
Keyword overlap misses meaning. Ask "what's my pup's name?" and it scores zero against "dog" — different words, same idea. The grown-up fix is embeddings: turn each document and the query into a vector of numbers that captures meaning, then retrieve by closeness in that space (this is the vector search from 2.4). Same retrieve → augment → generate flow — just a smarter score function. Production RAG swaps the scorer; the shape you just built stays identical.
SEE ITBuild-Your-Own RAG
Here's the retrieve() function from above, running live. Pick a question and watch the keyword-overlap score computed against each of six notes — exactly as the Python does — then see the top matches retrieved and the answer generated from only those. This is RAG, end to end, with nothing hidden.
Try "Where do I live?" — none of the six notes share useful words with it, so every score is 0, nothing is retrieved, and the agent honestly says it doesn't know. That's RAG working correctly: it answers only from what it actually found, instead of bluffing.
CORE CONCEPTThe retrieve → augment → generate flow
Every level of memory in this lesson is a version of the same three-beat flow. Once you see it, you'll spot it inside every RAG system, no matter how big:
Retrieve
Score your stored documents against the query and pick the top few. In our code that's retrieve(query, docs). The scorer can be keyword overlap (simple) or embeddings (smart) — the step is the same.
Augment
Paste those retrieved chunks into the prompt — usually the system message. You're augmenting the model's context with just-in-time knowledge it didn't have.
Generate
Call the model. It answers grounded in what you retrieved — and only that. Good retrieval in, good answer out; garbage in, garbage out.
That's why it's called retrieval-augmented generation: you augment the generation step with retrieved memory. The model never changed. You just got better at feeding it.
WEAKNESSHonest limits of homemade memory
This stuff works, but don't oversell it to yourself. Three real limits to respect:
- Keyword retrieval is crude. It matches words, not meaning. Synonyms, paraphrases, and typos slip right past it — "pup" won't find "dog." It's a fine starting point and a terrible finishing point. Embeddings (above) are the upgrade when keyword overlap starts missing obvious matches.
- Memory goes stale. A fact you saved months ago ("the user works at Acme") may now be wrong, but it'll keep getting retrieved and injected as if it were true. Stored memory has no expiry unless you give it one — and a confidently-retrieved stale fact is worse than no fact.
- The context budget is finite. Every retrieved chunk costs tokens and competes for room in the window (2.3). Retrieve too much and you blow the budget, slow the call, and bury the signal. The discipline of managing context (3.3) applies directly: a small
top_kof the right chunks beats a big dump of vaguely-related ones.
RAG's answer can only be as good as what it retrieved. If the right document isn't in the top hits, the model never sees it and will either guess or say it doesn't know. When a RAG agent gives a bad answer, suspect the retriever first — far more often than the model.
WEAKNESSCommon mistakes to avoid
Reloading every message from every session (Level 1, taken too far) eventually overflows the context window and burns tokens on ancient, irrelevant chatter every single call. Past a point you must distill (Level 2) or retrieve selectively (Level 3), not just accumulate. Memory should get sharper over time, not heavier.
A keyword retriever silently misses anything phrased differently from the stored note. Don't assume "it's in the store, so it'll be found." If a synonym or paraphrase can hide a document, your agent will confidently answer as if that document didn't exist. Know the gap, and reach for embeddings when it bites.
A memory store that only ever grows fills with outdated and contradictory facts. Old preferences, changed jobs, abandoned projects — all still retrievable, all still injected as gospel. Build in a way to update, expire, or remove facts, or your "memory" slowly becomes a source of confident wrong answers.
KEY INSIGHTWhere this connects in your course
- Context window (2.3) — the limited, costly space memory must feed into; the reason you retrieve instead of dump.
- Memory & RAG (2.4) — the concept you met there (persistence, retrieval, embeddings/vector search) is now real, runnable code.
- Managing context (3.3) — "retrieve the right bit, don't dump everything" is that discipline applied to memory.
- The chat loop (4.4) — Level 1 just wraps that loop with
load_messages()/save_messages(). - Tools (4.5) —
remember_fact()andretrieve()can themselves be tools the agent calls when it decides to. - A real working agent (4.7) — next you'll assemble loop + tools + memory into one agent that actually gets work done.
You can now give an agent a past. Combine the loop (4.4), tools (4.5), and memory (this lesson) and you have every ingredient of a genuine agent — which is exactly what you'll build next.
GLOSSARYKey terms introduced
- Persistence
- Data that outlives the program that made it — written to disk so it survives a restart. The foundation of all memory.
- Serialization (JSON)
- Turning an in-memory object (like a list of dicts) into text you can save, and back again. Python's
json.dumpwrites it;json.loadreads it. - Facts store
- A small persisted dictionary of durable facts about the user, injected into the system prompt so the agent starts each run already knowing them.
- Retriever
- The function that, given a query, picks the most relevant documents from the store —
retrieve(query, docs)in our code. - Keyword overlap
- A simple, deterministic relevance score: how many words a query and a document share. Crude but real; matches words, not meaning.
- Embedding (upgrade)
- A vector of numbers capturing a text's meaning, so retrieval can match by semantic closeness (vector search, 2.4) instead of shared words — the grown-up replacement for keyword overlap.
- Retrieve-augment-generate
- The RAG flow: retrieve relevant chunks, augment the prompt with them, generate a grounded answer. The model is unchanged; only its context improves.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What's the difference between context and memory, and why does it matter?›
Context is the messages list sent each call — short-term, in RAM, gone on restart, and limited by the context window. Memory is data saved to disk that outlives the run (persistence). It matters because the model is stateless: to remember across runs you must save something and load the relevant bit back into the limited context.
2What's the smallest change that gives a chat loop memory across runs, and what's its downside?›
Replace messages = [] with messages = load_messages() (and save on exit) — serialize the list to JSON with json.dump/json.load. The downside: reloading the entire history eventually overflows the context window and costs tokens on every call, so you must distill (facts store) or retrieve selectively (RAG) instead of accumulating forever.
3Explain the three beats of RAG, and what our keyword retriever actually computes.›
Retrieve the most relevant docs for the query, augment the prompt by pasting them into context, generate a grounded answer. Our retriever computes keyword overlap — the number of words the query and a document share (set(query) & set(doc)) — sorts by that score, and returns the top k with score > 0.
4Name two honest limits of homemade keyword RAG and the fix for the first.›
(1) It matches words, not meaning, so synonyms/paraphrases ("pup" vs "dog") slip past — fix it with embeddings / vector search, which score by semantic closeness. (2) Memory goes stale and is still confidently retrieved unless you prune/expire it. (Also valid: retrieval costs tokens against a finite context budget, so retrieve few right chunks, not many vague ones.)