ProjekinAja. Agentic AI Stage 4 · Building Agents Lesson 4.6
● Stage 4 · Lesson 4.6

Giving Your Agent Memory

Your agent forgets everything the moment the program ends. Memory fixes that — from simply saving the conversation to a lightweight RAG you can build yourself, in plain Python, no library required.

⏱ ~15 min read 🧠 Memory, for real 🐍 Three levels of code

🎯 By the end you'll understand

The difference between context (the messages list, gone on restart) and memory (persistence across runs), and three concrete ways to add memory in code: save the conversation to a file, keep a facts store you inject into the system prompt, and build a minimal keyword-RAG retriever yourself — plus the retrieve → augment → generate flow and its honest limits.

KEY INSIGHTThe one idea to take away

Memory isn't a feature the model has — it's a habit you build: write the right things down, and read the right things back into context before you ask.

In Lesson 4.4 you saw the model is stateless: it remembers a conversation only because you keep re-sending the messages list. But that list lives in memory and vanishes when the program exits. Restart, and your agent has amnesia — it has never met you.

Real memory means writing something to disk and reading it back next run. That's it. There's no magic store inside the model — there's a file you save, and a decision about what to load back into the limited context window (2.3) when you do. This lesson makes the abstract memory ideas from Lesson 2.4 concrete, in code you can run today.

🔑 Hold onto this

Memory is two moves: save (persist something useful across runs) and retrieve (pull the relevant bit back into context before the model call). Everything from a JSON file to a billion-document vector database is a fancier version of those same two moves.

DEFINITIONContext is short-term; memory is long-term

These two words get muddled constantly, so let's pin them down — they're the spine of the whole lesson.

Context (short-term) 🧮Memory (long-term) 💾
What it isthe messages list you send each calldata saved to disk, outside the program
Lives wherein RAM, while the program runsin a file (or database) that outlives the run
Survives a restart?no — gone the moment it exitsyes — that's the whole point
Size limitthe context window (2.3) — small, finiteeffectively unlimited (it's just disk)
The catcheverything in it costs tokens, every callyou must choose what to load back in
📖 Persistence

Persistence means data outlives the process that created it. A variable in your program is not persistent — kill the program and it's gone. Write that variable to a file and it is persistent: it's still there next run, next week, next reboot. Giving an agent memory is, at bottom, just making the right things persistent.

So the design question is never "does my agent have memory?" — it's "what do I persist, and what do I load back into the limited context, and when?" We'll answer that in three escalating levels.

PRACTICALLevel 1 — Persist the conversation

The simplest possible memory: take the messages list you already have and save it to a file. Next run, load it back before the loop starts. Now the agent continues the conversation instead of starting blank.

The tool for this is serialization — turning an in-memory object into text you can write to disk, then back again. For a list of dicts, Python's built-in json module does it in two lines: json.dump to write, json.load to read.

memory.pypython
import json, os

PATH = "history.json"

def load_messages():                          # read history back at startup
    if os.path.exists(PATH):                  # first run? no file yet
        with open(PATH) as f:
            return json.load(f)              # JSON text → list of dicts
    return []                                  # start empty if nothing saved

def save_messages(messages):                  # write history at the end (or each turn)
    with open(PATH, "w") as f:
        json.dump(messages, f, indent=2)   # list of dicts → JSON text on disk

Now wrap your chat loop from 4.4 with these two calls: load at the top, save at the bottom.

chat.pypython
messages = load_messages()                     # ← was: messages = []  (now it remembers)

while True:
    user = input("You: ")
    if user.lower() in ("quit", "exit"):
        save_messages(messages)              # persist before we leave
        break
    messages.append({"role": "user", "content": user})
    response = client.messages.create(
        model="claude-sonnet-5", max_tokens=1024, messages=messages,
    )
    reply = response.content[0].text
    print("Bot:", reply)
    messages.append({"role": "assistant", "content": reply})

That single edit — messages = load_messages() instead of messages = [] — is the difference between an agent that has met you and one that hasn't. Simple, and genuinely effective for a personal assistant.

💡 But you can't do this forever

Reloading the entire history works until it doesn't: a months-long conversation will overflow the context window and cost a fortune in tokens every call (2.3). Level 1 is a great start, not a final answer. That tension — keep everything vs. keep only what matters — is exactly what Levels 2 and 3 solve.

CORE CONCEPTLevel 2 — A facts store

Most of a conversation is throwaway. What you actually want to carry forward are a handful of durable facts: the user's name, their timezone, their preferences, the project they're working on. Instead of replaying the whole transcript, keep a tiny facts store — a dictionary saved to disk — and inject only those facts into the system prompt at the start of each run.

facts.pypython
import json, os

FACTS_PATH = "facts.json"

def load_facts():
    if os.path.exists(FACTS_PATH):
        with open(FACTS_PATH) as f:
            return json.load(f)              # e.g. {"name": "Ada", "tz": "GMT"}
    return {}

def remember_fact(key, value):                # call this when you learn something durable
    facts = load_facts()
    facts[key] = value
    with open(FACTS_PATH, "w") as f:
        json.dump(facts, f, indent=2)

def facts_prompt():                            # turn the dict into a system-prompt line
    facts = load_facts()
    if not facts:
        return "You are a helpful assistant."
    lines = [f"- {k}: {v}" for k, v in facts.items()]
    return "You are a helpful assistant. Known facts about the user:\n" + "\n".join(lines)

Now you pass system=facts_prompt() to the model call. The agent starts every run already knowing "the user is Ada, timezone GMT" — using a few dozen tokens instead of a whole replayed transcript. You decide what's worth remembering; the store stays small and sharp.

▸ The pattern under both levels

Level 1 persists the raw conversation. Level 2 persists distilled facts. Both are "save to a file, load it back" — the difference is how much you distill before saving. The more you distill, the cheaper and sharper the memory. That instinct — distill, don't dump — carries straight into Level 3.

FRAMEWORKLevel 3 — Build your own RAG

What if there's too much to fit in the system prompt — a notebook of meeting notes, a wiki, a year of documents? You can't inject all of it; the context window won't hold it and you'd pay for thousands of tokens you don't need. So you store everything on disk, and at query time you retrieve only the few pieces relevant to the current question. That is RAG — Retrieval-Augmented Generation — which you met conceptually in Lesson 2.4. Here you build a working one yourself.

📖 RAG (Retrieval-Augmented Generation), recap

Keep a knowledge store outside the model. On each query, retrieve the most relevant chunks, augment the prompt by pasting them into context, then let the model generate an answer grounded in them. The model's "memory" becomes "whatever you retrieved this turn." No retraining, no fine-tuning — just smart loading.

The only hard part is retrieval: given a question, which documents are relevant? The simplest scorer that works is keyword overlap — count how many words the question and a document share. More shared words, more likely relevant. It's crude, but it's real, it's deterministic, and you can write it from scratch:

retriever.pypython
import re

def words(text):                              # lowercase, strip punctuation → set of words
    return set(re.findall(r"[a-z0-9]+", text.lower()))

def score(query, doc):                         # how many words do they share?
    return len(words(query) & words(doc))   # & = set intersection

def retrieve(query, docs, top_k=2):           # return the best-matching docs
    scored = [(score(query, d), d) for d in docs]
    scored.sort(reverse=True)              # highest score first
    return [d for s, d in scored[:top_k] if s > 0]  # top_k, drop zero-overlap

Thirteen lines, no dependencies, and it genuinely retrieves. retrieve("what's my dog's name?", docs) ranks every note by shared words and hands back the top two that actually overlap. Drop in your documents and it works:

use_rag.pypython
docs = [
    "The user's dog is named Pixel, a border collie.",
    "The user prefers meetings in the afternoon, never before 11am.",
    "The user is allergic to shellfish and peanuts.",
]

query = "what is my dog called?"
hits = retrieve(query, docs)            # → ["The user's dog is named Pixel, ..."]

context = "\n".join(hits)                  # only the relevant note(s)
system = f"Use these notes to answer:\n{context}"

response = client.messages.create(
    model="claude-sonnet-5", max_tokens=512,
    system=system,                       # augment: the retrieved notes go here
    messages=[{"role": "user", "content": query}],
)
print(response.content[0].text)        # "Your dog is named Pixel." — grounded in the note

Notice the system prompt holds only the matching note, not all three. That's the entire discipline of managing context (3.3): retrieve the right bit, don't dump everything.

💡 The upgrade path: embeddings

Keyword overlap misses meaning. Ask "what's my pup's name?" and it scores zero against "dog" — different words, same idea. The grown-up fix is embeddings: turn each document and the query into a vector of numbers that captures meaning, then retrieve by closeness in that space (this is the vector search from 2.4). Same retrieve → augment → generate flow — just a smarter score function. Production RAG swaps the scorer; the shape you just built stays identical.

SEE ITBuild-Your-Own RAG

Here's the retrieve() function from above, running live. Pick a question and watch the keyword-overlap score computed against each of six notes — exactly as the Python does — then see the top matches retrieved and the answer generated from only those. This is RAG, end to end, with nothing hidden.

🔎 Interactive · keyword-overlap retrieval over a tiny doc set

Score = number of words the question shares with a note. Top 2 with score > 0 are retrieved.

Knowledge store · scored against your question
Retrieved into context

Generated answer (grounded in the retrieved notes)
Agent

Try "Where do I live?" — none of the six notes share useful words with it, so every score is 0, nothing is retrieved, and the agent honestly says it doesn't know. That's RAG working correctly: it answers only from what it actually found, instead of bluffing.

CORE CONCEPTThe retrieve → augment → generate flow

Every level of memory in this lesson is a version of the same three-beat flow. Once you see it, you'll spot it inside every RAG system, no matter how big:

query the question ① retrieve score the store memory on disk ② augment paste top hits in ③ generate grounded answer
①

Retrieve

Score your stored documents against the query and pick the top few. In our code that's retrieve(query, docs). The scorer can be keyword overlap (simple) or embeddings (smart) — the step is the same.

②

Augment

Paste those retrieved chunks into the prompt — usually the system message. You're augmenting the model's context with just-in-time knowledge it didn't have.

③

Generate

Call the model. It answers grounded in what you retrieved — and only that. Good retrieval in, good answer out; garbage in, garbage out.

That's why it's called retrieval-augmented generation: you augment the generation step with retrieved memory. The model never changed. You just got better at feeding it.

WEAKNESSHonest limits of homemade memory

This stuff works, but don't oversell it to yourself. Three real limits to respect:

⚠️ Retrieval quality is everything

RAG's answer can only be as good as what it retrieved. If the right document isn't in the top hits, the model never sees it and will either guess or say it doesn't know. When a RAG agent gives a bad answer, suspect the retriever first — far more often than the model.

WEAKNESSCommon mistakes to avoid

✗ Keeping the whole history forever

Reloading every message from every session (Level 1, taken too far) eventually overflows the context window and burns tokens on ancient, irrelevant chatter every single call. Past a point you must distill (Level 2) or retrieve selectively (Level 3), not just accumulate. Memory should get sharper over time, not heavier.

✗ Trusting keyword RAG to have perfect recall

A keyword retriever silently misses anything phrased differently from the stored note. Don't assume "it's in the store, so it'll be found." If a synonym or paraphrase can hide a document, your agent will confidently answer as if that document didn't exist. Know the gap, and reach for embeddings when it bites.

✗ Never pruning stale memory

A memory store that only ever grows fills with outdated and contradictory facts. Old preferences, changed jobs, abandoned projects — all still retrievable, all still injected as gospel. Build in a way to update, expire, or remove facts, or your "memory" slowly becomes a source of confident wrong answers.

KEY INSIGHTWhere this connects in your course

  • Context window (2.3) — the limited, costly space memory must feed into; the reason you retrieve instead of dump.
  • Memory & RAG (2.4) — the concept you met there (persistence, retrieval, embeddings/vector search) is now real, runnable code.
  • Managing context (3.3) — "retrieve the right bit, don't dump everything" is that discipline applied to memory.
  • The chat loop (4.4) — Level 1 just wraps that loop with load_messages() / save_messages().
  • Tools (4.5) — remember_fact() and retrieve() can themselves be tools the agent calls when it decides to.
  • A real working agent (4.7) — next you'll assemble loop + tools + memory into one agent that actually gets work done.

You can now give an agent a past. Combine the loop (4.4), tools (4.5), and memory (this lesson) and you have every ingredient of a genuine agent — which is exactly what you'll build next.

GLOSSARYKey terms introduced

Persistence
Data that outlives the program that made it — written to disk so it survives a restart. The foundation of all memory.
Serialization (JSON)
Turning an in-memory object (like a list of dicts) into text you can save, and back again. Python's json.dump writes it; json.load reads it.
Facts store
A small persisted dictionary of durable facts about the user, injected into the system prompt so the agent starts each run already knowing them.
Retriever
The function that, given a query, picks the most relevant documents from the store — retrieve(query, docs) in our code.
Keyword overlap
A simple, deterministic relevance score: how many words a query and a document share. Crude but real; matches words, not meaning.
Embedding (upgrade)
A vector of numbers capturing a text's meaning, so retrieval can match by semantic closeness (vector search, 2.4) instead of shared words — the grown-up replacement for keyword overlap.
Retrieve-augment-generate
The RAG flow: retrieve relevant chunks, augment the prompt with them, generate a grounded answer. The model is unchanged; only its context improves.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1What's the difference between context and memory, and why does it matter?›

Context is the messages list sent each call — short-term, in RAM, gone on restart, and limited by the context window. Memory is data saved to disk that outlives the run (persistence). It matters because the model is stateless: to remember across runs you must save something and load the relevant bit back into the limited context.

2What's the smallest change that gives a chat loop memory across runs, and what's its downside?›

Replace messages = [] with messages = load_messages() (and save on exit) — serialize the list to JSON with json.dump/json.load. The downside: reloading the entire history eventually overflows the context window and costs tokens on every call, so you must distill (facts store) or retrieve selectively (RAG) instead of accumulating forever.

3Explain the three beats of RAG, and what our keyword retriever actually computes.›

Retrieve the most relevant docs for the query, augment the prompt by pasting them into context, generate a grounded answer. Our retriever computes keyword overlap — the number of words the query and a document share (set(query) & set(doc)) — sorts by that score, and returns the top k with score > 0.

4Name two honest limits of homemade keyword RAG and the fix for the first.›

(1) It matches words, not meaning, so synonyms/paraphrases ("pup" vs "dog") slip past — fix it with embeddings / vector search, which score by semantic closeness. (2) Memory goes stale and is still confidently retrieved unless you prune/expire it. (Also valid: retrieval costs tokens against a finite context budget, so retrieve few right chunks, not many vague ones.)

Your agent has a past now. Next: put it all together →