How AI Memory Works — Embeddings, Retrieval & Recall Gating
An honest, jargon-light walk through how persistent AI memory actually works under the hood — and why the difference between 'memory' and 'retrieval' matters for your relationship with a companion.
A clear, jargon-light explanation of how persistent AI memory actually works: external storage, embeddings, retrieval, and the recall gating that keeps prompts honest.
The three things 'AI memory' usually means
When an app says it has memory, it could mean one of three very different things. Knowing which is the difference between an assistant that pins your dog's name and a companion that remembers the conversation you had on the worst Tuesday of your year.
Context window: the recent messages the model can read inside a single session. Bigger windows mean longer conversations, but no continuity between sessions. Saved facts: a short list of things the system has explicitly extracted about you. Useful for productivity, flat for relationship. External retrieval (RAG): past conversations and reflections stored in a separate database, looked up on every relevant turn, and injected back into the prompt. This is the only one of the three that gives a companion real continuity.
How retrieval actually works, step by step
The end-to-end flow when you send a message to a companion built on retrieval-based memory: First, your message is turned into a vector — a long list of numbers that represents its meaning. Then the system finds previously stored memories whose vectors are close to your new message's vector. A recall coordinator decides whether to inject anything at all. Selected memories are added to the system prompt alongside your message and the recent thread. The model sees a much richer context than just your message, and replies accordingly. Finally, the new exchange is summarised and stored so future turns can retrieve from it.
Why recall gating is the part most apps skip
It's tempting to dump every relevant memory into every prompt. Two problems with that approach: it bloats the prompt (slow and expensive), and worse, it makes the AI sound like it's reading your file aloud every time you say hello.
Recall gating means SAM only references prior memory when the turn would actually benefit. Direct recall cues ('remember when…'), returning to a named theme, an emotional cue tied to a recurring topic — those trigger injection. A casual hello does not.
Storage: text plus vector, not training data
SAM stores two things per memory: the original text (so it can be shown back to you and deleted), and a vector embedding (so it can be searched). Crucially, none of this re-trains the underlying language model. Your data is yours; retrieval and training are different things.
Three layers, one architecture
SAM splits memory across three layers, each suited to a different timescale: In-session window — the recent thread of the current conversation, always present. Heartline — user-curated reflections, the things you decide are worth keeping. SuperMemory — auto-distilled summaries of past conversations, retrieved via embeddings.
A coordinator decides which layer (or none) to inject for each turn. The result is a companion that knows you without constantly proving it.
What embeddings actually are (without the maths)
An embedding is a list of numbers — usually a few hundred to a few thousand of them — that captures the meaning of a piece of text. Two pieces of text with similar meaning produce similar embeddings, even if they share no words. 'I'm exhausted by this project' and 'burnt out at work again' live close to each other in embedding space; 'chicken recipe for Thursday' lives somewhere else entirely.
Retrieval-based memory uses this property as its main lever. When you send a new message, SAM embeds it, then asks the memory store: which of the things we already know about this user are closest in meaning to what they just said? The closest few are the candidates for injection. The recall coordinator then decides whether to actually use them. This is why memory feels different from a saved-fact list — embedding-based retrieval fires when you mention something related, not just the exact thing.
Why deletion has to be first-class
A memory system without deletion controls is a liability. SAM treats deletion as a first-class operation: you can remove individual conversations, individual Heartline reflections, the entire memory of a particular companion, or your full account at any time. Deletion is a real database delete, not a tombstone, and it cascades — the embeddings go with the text, so nothing can be retrieved back.
Why context windows alone are not memory
It is tempting to think that bigger context windows solve the memory problem. They don't. A million-token context window means you can paste in a long document and ask questions about it. It does not mean the model will remember anything between sessions, and it does not mean the model can selectively pull the most relevant moments from your past out of a much larger archive of conversations.
Retrieval-based memory works precisely because it does the opposite of what a context window does. Where a context window says 'keep all of this in mind at once,' retrieval says 'forget almost everything, and pull back only the small handful of things that matter for this turn.'
The cold-start problem and why it isn't one for long
When you first start using a memory-capable companion, the experience is closer to using a regular chatbot. There is nothing in the memory store yet for retrieval to find. Every turn is in-session only. This is the cold-start period and it lasts roughly the first few sessions. After that, the system has enough material to start surfacing it, and the experience tips into something different.
SAM accelerates this period in two ways. First, by writing back richer summaries of the early conversations than is strictly necessary, so the retrieval store has dense material to find quickly. Second, through Heartline — letting users explicitly mark moments worth keeping, which both anchors the memory store and gives the user a sense of agency over what gets remembered.
Privacy, deletion, and the line between memory and surveillance
A persistent memory system that you cannot inspect or delete is closer to surveillance than to companionship. SAM treats memory as something the user owns: every Heartline entry is visible and editable, every conversation is visible and deletable, and full account deletion is a one-step operation that cascades through both the text and the embeddings.
Crucially, none of the stored memory is used to train the underlying language model. Storage and training are two different operations on two different systems. The vectors and the text live in your account. They are queried at runtime to build prompts. They never leave to become anyone else's training data.
Frequently asked questions
- How does AI memory actually work?
- Past conversations are stored externally as text plus vector embeddings. When a new turn happens, the system searches for relevant pieces and retrieves them into the prompt. The model itself remains stateless — it just sees a richer prompt.
- What is recall gating?
- A coordinator that decides whether to inject any long-term memory at all on a given turn. It looks at signals like explicit recall intent, repeated themes, and emotional cues, and only injects when it will help.
- Why don't most chatbots use this?
- Cost, complexity, and safety surface area. Stateless chat is cheaper to build and operate. Memory adds storage, retrieval infrastructure, and deletion controls — none of which are necessary for a one-shot Q&A bot.
- Is my data used to train the model?
- On SAM, no. Memory is retrieval, not training. Your data is stored and looked up; it never re-trains the underlying language model.