Fundamentals

What Is AI Agent Memory? A Complete Guide for Developers

Devasheesh Mishra··10 min read

The Stateless Problem

Large language models have no memory. Every API call starts from zero. The model does not know who you are, what you said yesterday, or what changed since last week. Whatever appears in the prompt is all the model knows — and when the conversation ends, everything is gone.

This is fine for one-shot tasks. It is catastrophic for agents that interact with the same user over days, weeks, or months. A tutoring agent that forgets the student’s learning history. A health companion that loses track of medication changes. A customer support bot that asks the same onboarding questions every session.

The industry calls this “conversational amnesia,” and it is the default behavior of every LLM in production today.

What Agent Memory Actually Is

Agent memory is a system that gives an LLM persistent state across conversations. Instead of re-sending the entire conversation history on every API call, a memory layer extracts, structures, and stores important information — then retrieves only what is relevant when the user returns.

The distinction from a database is important. Memory is not just storage. It involves:

  • Extraction: Identifying what matters in a conversation (facts, preferences, events) and discarding noise
  • Structuring: Organizing extracted information into queryable formats — entities, relationships, temporal metadata
  • Evolution: Tracking how facts change over time, not just what is currently true
  • Retrieval: Finding the right memories at the right time, with the right level of depth

A flat key-value store or a vector database can handle storage. Memory requires all four.

The Four Types of Memory

Drawing from cognitive science (Tulving, 1972) and the CoALA framework, AI agent memory falls into four categories:

Working Memory

The LLM’s context window. Everything currently in the prompt — system instructions, tool definitions, recent messages. It is volatile, limited by token count, and wiped at the end of the session. Think of it as RAM: fast but temporary.

Episodic Memory

Timestamped records of what happened. “The user mentioned moving to Delhi on March 15.” “In our last session, the user asked about Python web frameworks.” Episodic memory captures events — who said what, when, and in what context. It answers questions like “what did we discuss last Tuesday?”

Semantic Memory

Durable facts about the user. “The user is a vegetarian.” “Their preferred language is Python.” “They work at Acme Corp.” Semantic memory is context-free — it represents what is true, not when it became true. It answers questions like “what does this user prefer?”

Procedural Memory

Learned behaviors and patterns. “When this user asks about deployment, they usually mean AWS.” “This user prefers concise answers over detailed explanations.” Procedural memory captures how to interact with a specific user, not what facts are known about them.

Most production memory systems focus on episodic and semantic memory. Procedural memory remains an active research area.

Why Context Windows Are Not Memory

The most common objection: “Gemini has a 1M token context window. Why do I need memory?”

Three reasons:

Cost scales quadratically. Re-sending full conversation history on every call means Session 50 includes the content of Sessions 1–49. The math is brutal — a single user can cost $540+ over 60 days with naive context stuffing.

Accuracy degrades with length. The “lost in the middle” phenomenon is well-documented. Models struggle to attend to information buried in long contexts. A user fact mentioned in session 3 may be effectively invisible when the context window contains 50 sessions of irrelevant history.

Latency grows with history. Every token in the context window adds inference latency. A 200-token prompt returns in milliseconds. A 500,000-token prompt with 50 sessions of history takes significantly longer — and you pay that latency on every single exchange. For voice agents with sub-second budgets, this is a hard blocker. Even for text-based agents, the cumulative slowdown across thousands of users makes context stuffing impractical at scale.

How Modern Memory Layers Work

All modern memory systems share a basic architecture: an ingestion pipeline that processes conversations into stored memories, and a retrieval pipeline that finds relevant memories when needed. The differences lie in how each pipeline is designed.

Append-Only Systems

The simplest approach: extract facts from conversations and store them as vector embeddings. When the user returns, run a semantic similarity search against the stored facts. Systems like basic RAG setups and early memory layers use this model.

The problem: no mechanism for handling facts that change. When a user moves from Delhi to Mumbai, the system stores both “lives in Delhi” and “lives in Mumbai” with no link between them. This is what we mean by the overwrite problem.

Graph-Based Systems

More sophisticated systems organize memories as knowledge graphs — entities connected by typed relationships. This enables richer queries: “who does the user work with?” can be answered by traversing the graph from the user node to connected entities.

The challenge is keeping the graph consistent as information evolves. Most graph-based systems still lack explicit version control, meaning updates either overwrite nodes or create duplicate entries.

Versioned Memory Systems

The most advanced approach treats memory as a versioned, evolving data structure. When a fact changes, the old version is preserved in a version chain, linked to the new version with metadata explaining the transition (temporal update, correction, preference change). This enables queries like “where did the user live before Delhi?” — which neither append-only nor basic graph systems can answer reliably.

What Makes a Good Memory System

Based on evaluating production memory layers against the LongMemEval benchmark, the capabilities that separate high-performing systems from low-performing ones are:

  1. Temporal reasoning: Can the system understand how a user’s information has evolved over time? If a user was vegetarian in January but started eating fish in June, the system should know both the current preference and the history behind it. If the user asks “what did I used to eat?”, the system needs to trace that evolution — not just return the latest fact.

  2. Non-destructive updates: When facts change, does the system preserve history or silently overwrite? Systems without version control fail on knowledge update tracking.

  3. Conflict detection: When contradictory information arrives, does the system recognize the contradiction — or store both versions without flagging it?

  4. Latency flexibility: Different queries need different retrieval depths. A simple “what’s my name?” query should not take 7 seconds. A complex “summarize my health history over the last 6 months” query should not be limited to sub-second retrieval.

  5. Decoupled ingestion and retrieval: Systems that process memories synchronously (during the conversation) add latency to the user experience. Systems that process asynchronously (after the conversation) can afford more thorough curation without impacting response time.

How Recallr Approaches This

Recallr is built around a versioned memory graph with asynchronous curation. The core design:

  • Asynchronous curation pipeline: After each session, conversations are processed into structured knowledge — entities, relationships, temporal metadata — without adding latency to the agent’s responses.
  • Git-inspired version control: When facts change, old versions are archived in a linked chain. The system detects whether a change is a temporal update, a correction, or a genuine contradiction that requires user clarification.
  • Merge conflict resolution: When the system detects a genuine contradiction — say, a user said “I’m allergic to peanuts” in one session and “I have no allergies” in another — it does not silently pick one. Instead, Recallr generates clarifying questions and delivers them via webhooks to your application. The user answers at their own pace, the system verifies whether the answers fully resolve the ambiguity, and then updates the memory graph. The graph stays eventually consistent — contradictions are surfaced, never buried.
  • Adaptive recall strategies: Three retrieval modes — Low-Latency (median 299ms), Balanced (median 1.2s), and Agentic (median 7s) — let developers match retrieval depth to their use case.

On LongMemEval, this architecture achieves 97.5% accuracy, compared to 65.4% for Mem0 and 29.4% for Supermemory. Full benchmark breakdown here.

Getting Started

If your agent has more than a few sessions of history per user, it needs a memory layer. The question is what kind.

For static knowledge retrieval (documents, FAQs), RAG is the right tool. For persistent, evolving user state across conversations, you need a dedicated memory system.

See how Recallr integrates in two lines of code.

← Back to all posts