Memory — Short-Term, Long-Term & RAG
Context is what an agent holds right now. Memory is how it remembers across time — and RAG is how it looks things up without trying to memorize everything.
🎯 By the end you'll understand
The difference between short-term memory (the context window) and long-term memory (stored outside the model), what RAG really means in plain words, how embeddings let a system find things by meaning instead of exact words, the retrieve → augment → generate flow, the three practical memory types — and the honest limits of all of it.
KEY INSIGHTThe one idea to take away
An agent doesn't have a memory — it has a notebook it can search. Remembering isn't recall from a brain; it's looking the right thing up and pasting it back into context.
In Lesson 2.3 you learned that the context window is the agent's working memory — finite, and wiped clean when the session ends. So the obvious question is: how does an agent remember anything from yesterday, or know facts that were never in its training? The answer is almost boringly mechanical. It writes things down somewhere outside itself, and when it needs them, it fetches the relevant bits and drops them back into the context window.
The model itself never changes and never "learns" your documents. All memory lives outside the model. Remembering = retrieving the right text and putting it into context so the model can read it this turn. That single trick is the whole of long-term memory and RAG.
CORE CONCEPTShort-term memory: the context window
You already met this in Lesson 2.3, so just a quick recap. Short-term memory is the context window — everything the model can "see" on this turn: the system prompt, the conversation so far, tool results, the documents you pasted in. It's the agent's desk. While the session is live, it can refer back to anything on that desk.
But two things make it short-term:
- It's finite. The desk has an edge. Past a certain size, older items fall off (or get summarized away). Context is a budget, not infinity.
- It's volatile. When the session ends, the desk is cleared. Start a new chat tomorrow and the agent has no idea who you are or what you discussed — unless something put that information back on the desk.
Short-term memory is a whiteboard. Powerful while you're in the room, but wiped when you leave. Everything else in this lesson is about how an agent saves a snapshot of the whiteboard and re-loads the useful parts later.
CORE CONCEPTLong-term memory: storing outside the model
If the whiteboard gets wiped, you need a filing cabinet that survives. That's long-term memory: facts, notes, and past interactions saved in a store outside the model — a database, a set of files, a vector store. Crucially, the model is not modified. We don't retrain it on your data. We just keep your data nearby and hand it the relevant pages when needed.
So a long-term-memory loop looks like this:
Write
During or after a session, save useful things to the store — "the user prefers metric units," "project deadline is June 20," a summary of what happened.
Persist
That store lives outside any single conversation. It's still there tomorrow, next week, next session.
Retrieve
When a new task starts, fetch the relevant saved bits and load them back into the context window — onto today's whiteboard.
This is why a well-built assistant can "remember" your name across sessions. It didn't grow a memory. On each new session it queried its store and re-injected the facts into context before answering. The remembering happens in the plumbing, not the brain.
"Long-term memory" is a slightly grand name for save-then-look-up. The hard part isn't storing — it's retrieving the right thing at the right time. Which is exactly the problem RAG was built to solve.
DEFINITIONWhat RAG actually is
Here's the problem. You have a huge pile of knowledge — a 400-page handbook, ten thousand support tickets, your company wiki. The context window can't hold all of it (it's finite — see 2.3), and dumping everything in would be slow, expensive, and would bury the answer in noise. So how do you let the model use knowledge that's far too big to fit?
You don't put the knowledge in the model. You put it in a searchable store, and at question time you fetch only the few passages that matter, drop those into context, and let the model answer using them. That pattern has a name.
RAG is a pattern where, instead of relying only on what the model already knows, you retrieve relevant text from an external store, augment the prompt by adding that text to the context, and then let the model generate an answer grounded in it. Retrieval-Augmented Generation — the name is literally the three steps.
The mental shift is important. A plain chatbot answers from its training (frozen, general, sometimes out of date or hallucinated). A RAG system answers from specific text you handed it this turn — your docs, today's data, the relevant ticket. It's the difference between "tell me what you remember from school" and "here's the page, now answer."
RAG is how almost every "chat with your documents" product works, and how serious agents access private or fresh knowledge. It's cheaper than retraining, it updates instantly (change a file, the next answer reflects it), and it lets you point at the source. You'll build a tiny one yourself in Lesson 4.6.
CORE CONCEPTFinding by meaning, not by words
RAG only works if retrieval is good. So how does the system find the relevant passages? The naive way is keyword search: match the words in the question to words in the documents. But that breaks constantly. Ask "how do I get my money back?" and the relevant doc says "refund policy" — zero shared keywords, yet they mean the same thing.
The fix is embeddings. An embedding model reads a piece of text and turns it into a long list of numbers — a vector — that acts like a fingerprint of its meaning. The trick: texts that mean similar things get vectors that sit close together, even if they share no words. "Get my money back" lands right next to "refund policy."
An embedding is a numeric vector — a fingerprint — that represents the meaning of a piece of text. Similar meanings produce nearby vectors. Semantic (vector) search finds matches by measuring which vectors are closest, so it can find the right passage by meaning rather than by exact wording.
You don't need the math. The intuition is enough: every note in your store has a meaning-fingerprint, the question gets a fingerprint too, and retrieval just grabs the notes whose fingerprints are closest to the question's. That's why a good RAG system can answer a casually-worded question from a formally-worded document — it's comparing meaning, not spelling.
Embedding = meaning as numbers. Semantic search = find the nearest meanings. Close vectors ≈ similar ideas.
SEE ITRAG in action
Here's a tiny "knowledge base" of six notes about a fictional product. Pick a question. Watch the system retrieve only the one or two notes whose meaning matches, drop the rest, and then generate an answer using only what it retrieved. Notice: it never reads all six. It fetches the relevant bit.
Note: a real system scores by comparing embedding vectors. Here the matches are hand-authored so the mechanism is clear — but the shape is exactly right: score every note for relevance, keep the top one or two, ignore the rest.
FRAMEWORKThe retrieve → augment → generate flow
Every RAG system, from a toy to a production agent, runs the same three-beat loop. Here it is as a diagram:
- Retrieve. Take the question, find the few passages in the store whose meaning is closest (semantic search over embeddings). Toss everything else.
- Augment. Paste those passages into the context window alongside the question — literally "here are the relevant notes; use them."
- Generate. The model answers using the supplied passages. Because it's reading real text rather than recalling from training, the answer is grounded and you can often cite the source.
That's it. RAG is not a different kind of model — it's a data-fetching step bolted in front of a normal model. The cleverness is all in the retrieve step.
PRACTICALThree kinds of memory in practice
When people build agents with memory, they usually split it into three flavors. It's borrowed loosely from how psychologists describe human memory, and it's a genuinely useful way to decide what to store:
| Type | What it holds | Example |
|---|---|---|
| Episodic | what happened — past events and interactions | "Last Tuesday the user asked to cancel order #88." |
| Semantic | facts and knowledge, timeless | "The user lives in Manila." · "Refunds take 5–7 days." |
| Procedural | how to do things — skills and routines | "To file a refund: open billing → select order → submit." |
In practice an agent might keep a running log (episodic), a fact store about the user and domain (semantic), and a set of playbooks or tool routines (procedural). All three are just stored text retrieved back into context when relevant — the same trick, sorted by purpose.
Be honest about the limits. Retrieval can fetch the wrong chunk (a near-meaning that isn't actually the answer). Stored facts go stale (you saved "deadline June 20," then it moved). And if the right passage isn't in the store, RAG can't conjure it — it'll answer from general training and may sound confident while being wrong. Memory here is a searchable archive, not a flawless brain.
WEAKNESSCommon mistakes to avoid
It doesn't. Your docs live in an external store; the model only sees the few chunks retrieved this turn, then forgets them. Nothing about the model changes. Delete the doc and the knowledge is gone next query.
RAG is only as good as its retrieval. Fetch the wrong chunk and you get a confident wrong answer. Miss the chunk entirely and the model falls back to (possibly outdated) training. Stale or badly-chunked stores quietly poison results.
No — context is finite (2.3) and cluttered. Stuffing in twenty loosely-related chunks buries the right one in noise, raises cost, and can lower answer quality. Retrieve few and precise, not many and vague.
KEY INSIGHTWhere this connects in your course
- Context window (2.3) — short-term memory is the context window, and its finiteness is the whole reason RAG exists. Retrieval's job is to spend that scarce context wisely.
- Planning & decomposition (2.5) — an agent that remembers past steps and retrieved facts can plan over many turns instead of starting blind each time.
- Tools (2.2) — "search the knowledge base" is itself a tool the agent calls. Retrieval is a tool, and memory is wired in through tool use.
- Build a RAG agent (4.6) — you'll assemble a tiny store, embed a few notes, retrieve, and generate — turning this lesson's diagram into running code.
Hold the through-line: memory is plumbing, not magic. Save text outside the model, find the relevant bit by meaning, paste it back into context, answer. Every "the AI remembered" moment is that loop, running quietly.
GLOSSARYKey terms introduced
- Short-term memory
- The context window — what the agent holds this session. Finite and wiped when the session ends.
- Long-term memory
- Facts and past interactions stored outside the model (a database/file), retrieved back into context when needed.
- RAG
- Retrieval-Augmented Generation: retrieve relevant text from a store, add it to the prompt, then generate a grounded answer.
- Embedding
- A numeric vector that fingerprints the meaning of text; similar meanings produce nearby vectors.
- Semantic / vector search
- Finding matches by comparing meaning-vectors — closest vectors win — rather than matching exact words.
- Retrieval
- The step of fetching the most relevant passages from the store for a given question.
- Chunk
- A small piece of a larger document, stored and retrieved as a unit so only relevant bits load into context.
- Episodic memory
- Stored record of what happened — past events and interactions.
- Semantic memory
- Stored facts and knowledge, independent of when they were learned. Don't confuse it with semantic search: "semantic memory" is a type of memory (what's stored — facts), while "semantic search" is a retrieval method (finding text by meaning). The names share a word but mean different things.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1How does an agent "remember" something from a past session if the context window gets wiped?›
It saved the information to a store outside the model (long-term memory). At the start of a new session it retrieves the relevant saved bits and loads them back into the context window. The model didn't recall anything — the plumbing re-injected it.
2Explain RAG in one sentence, using its three steps.›
Retrieve the most relevant passages from an external store, augment the prompt by adding them to context, then generate an answer grounded in those passages — instead of relying only on the model's training.
3Why use embeddings instead of plain keyword search for retrieval?›
Embeddings capture meaning, not spelling. "Get my money back" and "refund policy" share no keywords but have nearby vectors, so semantic search finds the right note. Keyword matching would miss it.
4Give two honest reasons RAG is not "perfect memory."›
Retrieval can fetch the wrong chunk (a close-but-incorrect meaning), and stored facts can go stale. Also, if the right passage isn't in the store at all, the model falls back to general training and may sound confident while being wrong. And stuffing in more chunks isn't a fix — it adds noise.