Fundamentals

What Is Context Engineering for AI Agents?

Devasheesh Mishra··8 min read

Beyond Prompt Engineering

Prompt engineering is about crafting the right words. Context engineering is about controlling the right information.

When your agent fails in production, the cause is rarely a bad prompt. It is usually a bad context: irrelevant memories polluting the prompt, critical facts missing from retrieval, stale information overriding current state, or the model drowning in 200k tokens when it needed 2k.

Context engineering is the discipline of managing what information reaches the LLM — what gets included, what gets excluded, in what order, and at what depth. It operates across the full information stack, not just the system prompt.

The Five Layers of Context

Every LLM prompt is assembled from multiple sources. Context engineering means managing all of them:

1. System Instructions

The base layer. Role definitions, behavioral constraints, output format requirements. This is what most people think of as “prompt engineering.” It is the smallest and least variable part of the context.

2. Tool Definitions

Function schemas, API descriptions, tool usage examples. In agentic systems, this layer can consume thousands of tokens — and grows with every tool you add. Poorly documented tools waste context budget on ambiguous schemas the model has to guess about.

3. Retrieved Knowledge

Memories, RAG results, knowledge base entries. This is where context engineering intersects directly with memory architecture. The question is not just “what do we retrieve?” but “how much is too much?” and “what happens when retrieved facts contradict each other?”

4. Conversation History

Recent messages in the current session. The naive approach includes everything. The engineered approach includes a window of recent messages plus summaries of older exchanges — enough for conversational coherence without quadratic cost growth.

5. User State

Persistent user profile information: preferences, past interactions, known facts. This is the domain of long-term memory — information that persists across sessions and evolves over time.

Four Failure Modes

Context engineering matters because bad context fails silently. The model does not tell you it is confused — it just produces worse output. Four common failure modes:

Context poisoning. Irrelevant or outdated information in the prompt actively misleads the model. A memory system that retrieves a user’s old address alongside their current one does not give the model “more context” — it creates ambiguity.

Context distraction. Too much context dilutes attention. The “lost in the middle” phenomenon shows that models struggle to attend to information buried in long prompts. More context is not always better context.

Context confusion. Contradictory information without resolution. The prompt says the user is vegetarian in one memory and had steak last night in another. Without explicit conflict resolution, the model picks one at random — or worse, ignores both.

Context starvation. Critical information missing from the prompt because retrieval was too shallow, too slow, or used the wrong search strategy. The model produces a plausible but wrong answer because it never saw the relevant fact.

Context Engineering vs. RAG vs. Memory

These three concepts are related but distinct:

RAG (Retrieval-Augmented Generation) is a technique — retrieve documents, stuff them into the prompt. It is one component of context engineering, focused on static knowledge retrieval. It was not designed for conversational memory.

Memory is a capability — persistent, evolving user state that spans sessions. Memory systems handle extraction, versioning, conflict resolution, and temporal reasoning. Memory is what makes context personal and dynamic.

Context engineering is the discipline that encompasses both. It is the system-level design of how all information sources are assembled, prioritized, and formatted for the LLM. A well-engineered context pipeline manages the interplay between RAG, memory, conversation history, and system instructions.

The Assembly Problem

The hardest part of context engineering is not retrieval — it is assembly. When you have memories, RAG results, conversation history, and tool definitions all competing for a finite context window, how do you prioritize?

A production-grade context assembly follows a strict priority schedule:

  1. System instructions — Always present, minimal tokens
  2. Tool definitions — Available capabilities and function schemas
  3. Retrieved memories — Most relevant persistent facts, structured with version history and temporal metadata
  4. Session summaries — Compressed episodic context from past sessions
  5. Pending conflicts — Unresolved contradictions that need user attention
  6. Recent conversation history — Last N messages for conversational coherence

The key insight: memory quality determines context quality. If your memory system stores noisy, unstructured, contradictory facts, no amount of prompt engineering will fix the downstream output. Investing in ingestion-time curation pays dividends at retrieval time.

How Memory Layers Enable Context Engineering

The best context engineering happens when the memory layer does the hard work before retrieval:

Deduplication at ingestion. When the user mentions the same fact across multiple sessions, the memory system should merge these into a single structured entry — not store five copies for the retrieval pipeline to sort out later.

Conflict detection at ingestion. When contradictory facts arrive, they should be flagged during curation — not surfaced as ambiguous context in the prompt.

Temporal grounding. Every memory should carry dual timestamps — when the event occurred and when the system learned about it. This lets the retrieval pipeline filter by time range and present only temporally relevant context.

Adaptive retrieval. Different queries need different context depths. A factual recall (“what’s my email?”) needs a sub-second vector search. A complex temporal query (“how has my treatment plan changed over the last 6 months?”) needs deep graph traversal. The memory layer should route queries automatically — not force one retrieval mode for everything.

Recallr’s architecture is built around this principle. The asynchronous curation pipeline handles deduplication, conflict detection, and temporal grounding after each session — so that by the time retrieval happens, the context is clean, structured, and ready to assemble. Three recall strategies (Low-Latency at 299ms, Balanced at 1.2s, Agentic at 7s) let developers match retrieval depth to query complexity.

The result on LongMemEval: 97.5% accuracy — because context quality was engineered at ingestion time, not patched at retrieval time. See the benchmark results.

Getting Started with Context Engineering

Three practical steps:

  1. Audit your current context. Print the full prompt your agent sends to the LLM. Measure how many tokens are consumed by each layer. Identify noise, redundancy, and missing information.

  2. Separate memory from RAG. If you are using RAG as a memory hack, you are paying more for worse results. Use RAG for static knowledge bases. Use a dedicated memory layer for persistent user state.

  3. Measure retrieval quality, not just accuracy. Track how many retrieved memories are actually relevant to the query. A memory system that retrieves 20 facts and 15 are irrelevant is actively harming your agent’s output — even if the 5 relevant ones are correct.

See how Recallr integrates in two lines of code.

← Back to all posts