# Memory Evaluation: Benchmarks and Custom Evals

How public benchmarks and custom evals measure memory recall, updates, temporal reasoning, and cost.

- Author: Jatin Bansal (https://jatinbansal.com/)
- Published: 2026-05-19
- Canonical: https://jatinbansal.com/ai-engineering/memory-evaluation/
- Topic: AI Engineering

---

Memory dashboards can report writes, reads, latency, and store size without measuring whether the system recalls the right fact. Evaluation needs multi-session traces, later questions with known answers, relevant episode IDs, and separate scores for temporal updates and abstention.

## The memory-eval contract

A memory eval ingests a fixed corpus of multi-session interactions before presenting its questions. Questions are tagged by ability, including single-hop recall, multi-hop reasoning, temporal reasoning, knowledge updates, and abstention. Reports should include retrieval and answer quality alongside tokens, latency, and cost.

[RAG evaluation](/ai-engineering/rag-evaluation/) scores retrieval against a mostly static corpus. Memory evaluation scores a corpus the system created during prior interactions. A [needle-in-a-haystack test](https://github.com/gkamradt/LLMTest_NeedleInAHaystack) asks whether a model can find a literal fact already present in its prompt; a memory test asks whether the full system can recover a paraphrased fact across sessions.

## Public benchmarks

Five benchmarks define the field in 2026. Each probes a different facet. A defensible memory system has numbers on at least the first two; serious teams run all five.

**LoCoMo** ([Maharana et al., 2024](https://snap-research.github.io/locomo/)). Fifty multi-session conversations between two fictional speakers, each spanning up to 32 sessions and averaging ~600 turns (~16K tokens), with timestamps and multimodal context. The 1,540-question QA set covers five categories: single-hop (intra-session recall), multi-hop (cross-session synthesis), temporal reasoning, open-domain, and adversarial (questions that should be refused). The paper reports GPT-4 at ~32.1 F1 against a human ceiling of ~87.9; the field's headline gap. Recent framework numbers: Mem0's 2026 token-efficient algorithm reports [92.5% on LoCoMo](https://mem0.ai/research); the 2025 [Mem0 paper](https://arxiv.org/abs/2504.19413) reports 66.9% vector-only and 68.4% with the Mem0g graph extension, both versus 52.9% for OpenAI's built-in memory feature. The wide range reflects different LLM judges, prompts, and ingest pipelines; read the protocol before comparing.

**LongMemEval** ([Wu et al., ICLR 2025](https://arxiv.org/abs/2410.10813)). Five hundred manually-curated questions embedded in freely-scalable chat histories, probing five memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. The construction is more rigorous than LoCoMo's; each question is human-authored, with histories scalable to test ingest-time behavior at multiple sizes. The paper documents commercial chat assistants dropping ~30% on sustained interactions versus single-session baselines. [Zep's January 2025 paper](https://arxiv.org/abs/2501.13956) reports 71.2% overall on LongMemEval with GPT-4o (versus 60.2% for vanilla full-context at 29s latency), with the largest gains in temporal-reasoning (+17.3pp) and multi-session (+13.6pp). Mem0's 2026 algorithm reports [94.4% on LongMemEval](https://mem0.ai/research).

**MemoryAgentBench** ([Hu et al., 2025](https://arxiv.org/abs/2507.05257)). Probes four cognitive competencies; accurate retrieval, test-time learning, long-range understanding, and *selective forgetting*. The selective-forgetting category finally exposed the contradiction-resolution failure mode every framework had been papering over: *all* paradigms fail dramatically on multi-hop conflict resolution, with best accuracy at or below 6% on CR-MH (Contradiction Resolution, Multi-Hop) even in the most advanced models. Run it if your product depends on the agent updating beliefs as facts change.

**BEAM** ([Mem0's State of AI Agent Memory 2026](https://mem0.ai/blog/state-of-ai-agent-memory-2026)). Scales to 1M and 10M token regimes, with ten categories including preference following, knowledge update, event ordering, abstention, and contradiction resolution. Mem0's 2026 numbers: 64.1 and 48.6 on BEAM 1M/10M. The contradiction-resolution and event-ordering categories are where production systems silently regress as corpora grow.

**∞Bench** ([Zhang et al., 2024](https://arxiv.org/abs/2402.13718)), **NoLiMa** ([Modarressi et al., ICML 2025](https://arxiv.org/abs/2502.05167)), and **RULER** ([Hsieh et al., 2024](https://arxiv.org/abs/2404.06654)) are long-context evals, not memory evals; they measure whether a model can find a fact buried in a 100K+ or 32K+ token prompt, not whether a system can recall across sessions. They calibrate the *long-context fallback*: NoLiMa's finding that 11 of 13 tested 128K-claiming models drop below 50% of their short-context baseline at 32K is the empirical case for memory-as-a-system over context stuffing.

A defensible benchmark story: numbers on LoCoMo and LongMemEval for the headline; MemoryAgentBench or BEAM if contradiction and selective forgetting matter; ∞Bench/NoLiMa/RULER numbers to justify why the system uses retrieval rather than bigger context.

## Retrieval and answer metrics

Two metric families do the work. **Precision/recall/F1 over retrieved memories** measures the *retrieval substrate*; given a query, how many of the top-K retrieved episodes are actually relevant, and how many of the relevant episodes in the store made it into the top-K? **End-to-end answer correctness** ([LLM-as-judge](/ai-engineering/llm-as-judge/), sometimes string-match against a normalized gold) measures whether the *whole system*; retrieval plus generation; answers right.

Both matter and fail differently. A retrieval substrate at 90% precision/recall with a generator that fumbles synthesis scores low end-to-end. A substrate that retrieves garbage paired with a generator whose parametric memory carries the answer scores high. The [Memory for Autonomous LLM Agents survey (2025)](https://arxiv.org/abs/2504.15965) lays out the taxonomy; the production rule is *measure both, separately, and pay attention when they diverge*.

The precision/recall mechanics, for episodic recall against a known gold:

- Top-K retrieval: the system returns K episodes for a query.
- Gold-relevant set: the eval pre-computes which episodes are relevant. For LoCoMo this is human-annotated; for a custom eval, you build it.
- Precision@K: (relevant retrieved) / K. **Recall@K:** (relevant retrieved) / (total relevant in store). **F1@K:** harmonic mean.

Pair these with **temporal-correctness** for the temporal category; "what did the agent believe last Tuesday?" should return the belief that held *as of last Tuesday*, not the current one. [Bi-temporal stores](/ai-engineering/temporal-reasoning-provenance/) like Graphiti score this category much higher than vector-only stores.

For *contradiction resolution*, the metric is [supersession behavior](/ai-engineering/memory-conflict-and-forgetting/): when two episodes claim contradictory facts, does the answer reflect the most recent (or most-trusted) claim, or average / pick at random? MemoryAgentBench CR-MH scores this; for a custom eval, the simplest version is hand-built "fact A at session N; ¬A at N+5; question at N+10" tuples.

## Code: a precision/recall harness (Python)

Phase-separated ingest-then-query, category-tagged questions, per-category metrics with token-cost alongside. Store-interchangeable across [Mem0](https://docs.mem0.ai/), [LangGraph store](https://docs.langchain.com/oss/python/langgraph/persistence), [Letta](https://docs.letta.com/), or a homegrown vector store.

```python
# pip install mem0ai openai
from dataclasses import dataclass, field
from typing import Any, Callable, Literal
import json, time
from collections import defaultdict

Category = Literal["single_hop", "multi_hop", "temporal", "knowledge_update", "abstention"]

@dataclass
class EvalQuestion:
    qid: str
    user_id: str
    question: str
    gold_answer: str | None  # None for abstention category
    relevant_episode_ids: set[str]  # ground truth for retrieval metrics
    category: Category

@dataclass
class IngestEvent:
    user_id: str
    text: str
    episode_id: str
    timestamp: float  # seconds since epoch; preserves multi-session structure

@dataclass
class EvalRun:
    by_category: dict[Category, list[dict]] = field(default_factory=lambda: defaultdict(list))

    def add(self, cat: Category, **fields):
        self.by_category[cat].append(fields)

    def summary(self) -> dict:
        out = {}
        for cat, rows in self.by_category.items():
            if not rows: continue
            n = len(rows)
            out[cat] = {
                "n": n,
                "precision_at_k": sum(r["precision"] for r in rows) / n,
                "recall_at_k": sum(r["recall"] for r in rows) / n,
                "f1_at_k": sum(r["f1"] for r in rows) / n,
                "answer_correct": sum(r["answer_correct"] for r in rows) / n,
                "avg_tokens": sum(r["tokens"] for r in rows) / n,
                "avg_latency_ms": sum(r["latency_ms"] for r in rows) / n,
            }
        return out


def f1(p: float, r: float) -> float:
    return 0.0 if (p + r) == 0 else 2 * p * r / (p + r)


def run_eval(
    ingest_events: list[IngestEvent],
    questions: list[EvalQuestion],
    memory_write: Callable[[IngestEvent], None],
    memory_search: Callable[[str, str, int], list[dict]],     # (user_id, query, k) -> [{episode_id, text, score}]
    answer_with_context: Callable[[str, list[str]], tuple[str, int]],  # (q, retrieved_texts) -> (answer, tokens_used)
    judge_correct: Callable[[str, str | None, Category], bool],  # (pred, gold, category) -> bool
    k: int = 5,
) -> EvalRun:
    # Phase 1: ingest. The memory system has no access to the questions here.
    for ev in sorted(ingest_events, key=lambda e: e.timestamp):
        memory_write(ev)
    # Phase 2: query. Per-question retrieval, scoring, and category attribution.
    run = EvalRun()
    for q in questions:
        t0 = time.perf_counter()
        retrieved = memory_search(q.user_id, q.question, k)
        retrieved_ids = {r["episode_id"] for r in retrieved}
        relevant_retrieved = retrieved_ids & q.relevant_episode_ids
        # Edge: abstention questions have no relevant episodes; recall is 1 by convention,
        # precision is 0 if any episodes were retrieved (system should have abstained on empty).
        if q.category == "abstention":
            precision = 0.0 if retrieved else 1.0
            recall = 1.0
        else:
            precision = (len(relevant_retrieved) / len(retrieved_ids)) if retrieved_ids else 0.0
            recall = (len(relevant_retrieved) / len(q.relevant_episode_ids)) if q.relevant_episode_ids else 0.0
        answer, tokens = answer_with_context(q.question, [r["text"] for r in retrieved])
        correct = judge_correct(answer, q.gold_answer, q.category)
        run.add(
            q.category,
            qid=q.qid,
            precision=precision,
            recall=recall,
            f1=f1(precision, recall),
            answer_correct=1.0 if correct else 0.0,
            tokens=tokens,
            latency_ms=(time.perf_counter() - t0) * 1000.0,
        )
    return run
```

The `memory_write` and `memory_search` callables are the only contract; the same harness scores Mem0, LangGraph, and a homegrown vector store identically. For abstention questions, the judge should accept any refusal phrasing as correct when the gold is `None`. `EvalRun.summary()` is the publishable artifact: per-category precision, recall, F1, end-to-end correctness, and token/latency budget in one dict.

## Designing a custom eval

Public benchmarks are necessary, not sufficient. They probe generic conversational memory; they don't probe your product's recall patterns. A customer-support agent has return-policy preferences and account-tier context that LoCoMo doesn't touch; a medical-history agent has drug interactions and prior diagnoses that LongMemEval doesn't touch. The custom eval is where workload-specific failure modes show up.

The minimum shape:

1. 20-50 real or near-real multi-session traces: Real if privacy permits; otherwise synthetic in the product's actual conversational shape. Curated 20-50 beats lazy 500.
2. Per-trace question set, 10-20 questions: Tagged by the LongMemEval five-ability frame. Each has a gold answer (or `None` for abstention) and relevant-episode IDs for retrieval metrics.
3. An ingest-then-query harness: The Python/TS code above is the template.
4. A weekly cron run: Every memory-policy change passes the eval before shipping. A per-category regression is blocking even if the aggregate holds.
5. A growing adversarial subset: Every production incident caused by a memory bug becomes a regression question. Over time this set becomes more useful than the seed corpus.

The trap to avoid: scoring only end-to-end answer correctness and ignoring retrieval precision/recall. If retrieval is at 30% recall but the generator's parametric knowledge masks it for 80% of questions, the system will fail catastrophically the moment the [question distribution shifts](/ai-engineering/drift-detection/). Retrieval metrics are the leading indicator; end-to-end is the lagging one.

For [Ragas-style](https://docs.ragas.io/) metrics applied to memory: context precision and context recall translate directly, and faithfulness catches "generator confabulates a memory." The multi-session-specific categories (temporal correctness, contradiction resolution) diverge; Ragas doesn't have those out of the box.

## Further reading

- [LongMemEval (Wu et al., ICLR 2025)](https://arxiv.org/abs/2410.10813); the most rigorously-constructed public memory eval. The five-ability framing is the right shape to copy for a custom eval; pair with the [GitHub repo](https://github.com/xiaowu0162/LongMemEval) for prompt templates and judge protocol.
- [LoCoMo (Maharana et al., 2024)](https://snap-research.github.io/locomo/); the canonical multi-session benchmark. The 32.1 F1 for GPT-4 vs 87.9 human ceiling is the field's headline gap; recent framework numbers (Mem0 92.5%) only make sense against that baseline.
- [Memory for Autonomous LLM Agents (2025)](https://arxiv.org/abs/2504.15965); comprehensive survey with the single best treatment of the metric taxonomy and LLM-as-judge bias.
- [State of AI Agent Memory 2026 (mem0.ai)](https://mem0.ai/blog/state-of-ai-agent-memory-2026); production-framework view on current numbers across LoCoMo, LongMemEval, BEAM, with explicit protocol notes. Pair with [Letta's controlled-benchmark post](https://www.letta.com/blog/benchmarking-ai-agent-memory) for a different vendor's read.
