How We Scored 97.5% on LongMemEval
A technical breakdown of the temporal memory architecture behind Recallr's private-capital decision graph and benchmark performance.
The Benchmark
LongMemEval is an oracle-based evaluation framework for long-term conversational memory. It tests 500 questions across six task types:
- Single-Session User (n=70): Can the system recall facts from a single conversation?
- Single-Session Preference (n=30): Does it remember stated preferences?
- Knowledge Update (n=78): When facts change, does it track the update?
- Temporal Reasoning (n=133): Can it reason about *when* things happened?
- Multi-Session (n=133): Does memory persist across separate conversations?
- Single-Session Assistant (n=56): Can it recall what *it* said previously?
Why Existing Solutions Struggle
Most memory systems treat all information as equal — a flat key-value store or a vector database with no temporal model. This works for simple recall but fails catastrophically on temporal reasoning and knowledge updates.
Mem0 achieves 76.9% overall, dropping to 48.9% on temporal reasoning. Supermemory scores ~30% overall, with 27.1% on temporal reasoning. The failure mode is consistent: without a temporal model, these systems can't distinguish between "when something happened" and "when it was mentioned."
Our Architecture
Recallr's versioned knowledge graph is the key differentiator:
1. Dual Timestamps Every memory entity carries two timestamps: - Event time: When the fact was actually true - Ingestion time: When the system learned about it
This distinction is what enables temporal reasoning. When a user says "I moved to Delhi last month," the event time is last month, not today.
2. Version Chains Each entity maintains a linked-list of versions. When information updates: - A new version node is created - A "supersedes" edge connects it to the previous version - The old version is archived, not deleted
This is why we score 97.4% on Knowledge Update — we never lose history.
3. Conflict Classification When contradictory information arrives, we classify it into four types: - Temporal update (facts changed over time) - Correction (user is fixing a mistake) - Preference change (opinions evolved) - True contradiction (needs clarification)
Each type has a different resolution strategy.
4. Graph Traversal for Recall Instead of flat vector similarity, recall traverses the knowledge graph — following relationships, respecting temporal boundaries, and surfacing connected memories that vector search would miss.
The Results
| Task Type | Recallr | Mem0 | Mem0 (Graph) | Supermemory |
|---|---|---|---|---|
| Single-Session User | 100% | 90% | 90% | 30% |
| Single-Session Pref. | 100% | 90% | 90% | 20% |
| Knowledge Update | 97.4% | 76.9% | 75.6% | 60.3% |
| Temporal Reasoning | 97.0% | 48.9% | 50.4% | 27.1% |
| Multi-Session | 89.5% | 65.4% | 63.2% | 35.3% |
| Single-Session Asst. | 100% | 19.6% | 19.6% | 3.6% |
The full benchmark code is open source: github.com/recallrai/benchmarks
What This Means In Practice
97.5% accuracy means the memory layer rarely loses context, rarely confuses timelines, and almost never silently overwrites important information. For private capital teams working across diligence history, IC decisions, and portfolio context, this level of reliability is the minimum bar. View the full benchmark results.