How AI Memory Works — Embeddings, Retrieval & Recall Gating

An honest, jargon-light walk through how persistent AI memory actually works under the hood — and why the difference between 'memory' and 'retrieval' matters for your relationship with a companion.

A clear, jargon-light explanation of how persistent AI memory actually works: external storage, embeddings, retrieval, and the recall gating that keeps prompts honest.

The three things 'AI memory' usually means

When an app says it has memory, it could mean one of three very different things. Knowing which is the difference between an assistant that pins your dog's name and a companion that remembers the conversation you had on the worst Tuesday of your year.

Context window: the information the model can consider in a single request. A larger context window lets a model consider more information at once. It does not itself save information between sessions, although an application can store a conversation and load it again later.

Saved facts (a.k.a. “memories”): a short list of things the system has explicitly extracted about you, such as “works as a designer” or “has a dog named Toby”.

External retrieval (RAG): past conversations and reflections stored separately, looked up when relevant, and supplied as context in the prompt. Saved facts, maintained summaries and retrieved passages can all support continuity between conversations. A companion can combine these approaches; retrieval-based memory is one part of the design, not the only way to preserve context.

How retrieval actually works, step by step

A companion such as SAM can combine several kinds of context. Saving information is separate from choosing what a later response should use.

1. Capture useful information. Conversation records and saved chat history are separate from derived memories. Stable facts can be extracted separately; background processing can turn selected conversation material into episodes. This does not mean every exchange immediately becomes a lasting memory.

2. Build context for a later message. Recent conversation, saved facts and available relationship background can supply continuity. They do not all depend on an explicit “remember when” request.

3. Find historical evidence. Episodes can be searched using embeddings, alongside other routes that consider what the question asks about, such as people, topics or time.

4. Select what fits. Relevance, recency, episode state, duplication and context limits can affect which material is included. The closest semantic match is not automatically the best memory to use.

5. Generate the reply. Selected evidence becomes part of the model’s input. It can influence the response, but inclusion does not guarantee accurate recall or a particular answer.

A detail shared with one companion should not be assumed to be available automatically to another.

What recall gating selects

Recall gating is selection, not a single switch that turns all memory on or off. Saved facts and relationship background may already be available during an ordinary exchange. A direct recall question can change how historical evidence is sought and selected.

There is also a difference between supplying a memory to the model and mentioning it in the reply. Useful background can shape an answer quietly. A greeting does not prove that no earlier information was supplied, and a recall request does not guarantee that the right detail will be found.

See AI Companion With Memory for how SAM approaches continuity in conversation.

Storage: memory records, not model training

Memory does not consist of one identical text-and-vector pair for every detail. Conversation records, saved facts, episode narratives and relationship background are different kinds of information. Episodes can have embeddings for search; an episode narrative is derived material, not necessarily the original message.

Supplying stored context during response generation is different from retraining model weights. That architectural distinction does not, by itself, establish how an external provider retains or uses data.

Memory sources, one architecture

It helps to distinguish recent conversation, durable facts, relationship background and selected historical evidence. These are explanatory categories, not four mutually exclusive storage systems. Different records and processing paths can contribute to the context assembled for a response. A product feature such as Heartline should not be treated as a complete description of the underlying memory architecture.

What embeddings actually are (without the maths)

An embedding represents text as numbers that can help compare semantic similarity. For illustration, “exhausted by this project” and “burnt out at work again” may be more closely related than either is to a chicken recipe. This is an illustration, not a measured retrieval result. Similarity helps identify candidates; it does not establish that a memory is correct, current or useful.

Saved facts can also influence an answer without an exact-word match. Embeddings are one retrieval tool, not the dividing line between systems that can and cannot preserve context.

What this looks like in practice

A worked example: preparing for an interview

This fictional example illustrates how information could move through SAM’s memory design. It is not a transcript, a measured test or a promise of recall.

In an earlier conversation, you tell one companion: “I have a job interview next Thursday. I prefer preparing quietly on my own rather than doing mock interviews. Also, I don’t like olives.”

What might be retained? The conversation may remain in saved chat history, while background processing may create an episode about the interview and your preparation preference. Fact extraction is a separate process; this exact wording is not guaranteed to become a saved fact. The olive preference might be retained separately, remain only in conversation history or not become a derived memory.

Later, you ask the same companion: “Do you remember how I wanted to prepare for that interview?”

How could relevant material be found? The question can lead the system to look for historical evidence about the interview and preparation. Embedding-based search can help find related wording, while other retrieval routes may contribute. Recent conversation and available background context can also help.

Why might something be left out? A separate olive-related memory adds little to an interview-preparation question. It may fail relevance selection or lose space to more useful evidence. This is not guaranteed: if the preference is bundled with relevant material, it could still enter the context. Leaving it out of this response does not delete it.

How could it affect the answer? If the preparation preference is supplied and used correctly, a possible reply is: “You said you preferred quiet preparation on your own. Would you like to make a short checklist?” The model should not turn that preference into an unsupported assumption that you are anxious or dislike people.

What can go wrong? The earlier detail may not have become a usable memory, retrieval may miss it, or selected information may not be used correctly. “Next Thursday” also belongs to the earlier conversation: a useful response must not assume the interview is still upcoming. Saving, selecting and successfully using a memory are separate steps.

Why context windows alone are not memory

A context window limits what the model can consider in one request. Persistence is a separate application responsibility: an app can save conversation text, facts or summaries and supply them again later. Retrieval helps select from a larger history without placing all of it into every request. Information omitted from a prompt has not necessarily been deleted.

The cold-start problem

A new relationship may offer little earlier material to draw on. Saved facts can begin accumulating early, while useful historical episodes depend on what is shared and successfully processed. There is no fixed number of sessions after which accurate recall is guaranteed.

Latency, cost, and why “fast cheap memory” is harder than it looks

Storing, processing and selecting memory adds work. The effect on response time depends on the services used, the amount of material considered and which operations run together. Timing claims need measurements from the deployed system; they should not be inferred from an architectural diagram.

Privacy, deletion, and the line between memory and surveillance

Different deletion actions have different scopes

Clearing a chat, removing a Heartline entry and deleting an account are different operations. Removing a source item does not, by itself, demonstrate that all separately derived memories have also been erased. The scope of each control matters.

SAM’s privacy policy explains that third-party providers may process conversation content to generate responses and that provider-side collection and training are disabled where supported. Do not confuse account-scoped storage with a promise that data never reaches an external processor.

The policy also distinguishes a deletion request from completed deletion: a failed step can leave a request pending. A statement about deleting account data should not be read as a guarantee of immediate erasure from all provider logs and backups.

Frequently asked questions

How does AI memory actually work?
AI memory can combine saved conversation history, durable facts, relationship background and retrieved historical evidence. Embeddings can help search episode narratives, but not every stored record is an embedding or a copy of an original message.
What is recall gating?
Recall gating selects historical evidence for a response. It does not mean that all saved facts and background context are absent until you explicitly ask the companion to remember something.
Why don't most chatbots use this?
Cost, complexity, and safety surface area. Stateless chat is cheaper to build and operate. Memory adds storage, retrieval infrastructure, and deletion controls — none of which are necessary for a one-shot Q&A bot.
Is my data used to train the model?
Retrieving stored information for a response is different from retraining a model. Provider processing, retention and training restrictions are separate privacy-policy questions; retrieval alone does not prove that no external provider can retain or use data.

Meet these companions

  • Echo — Empathic Mirror
  • Selene — Mysterious Truth-Teller
  • Morgan — Cerebral Siren