Benchmark · last updated 2026-06-19

LoCoMo Benchmark

1,540 QA pairs. 10 real conversations. gpt-4o-mini judge. Methodology published. Run it yourself.

Result

MemHQ scored 83.2% on LoCoMo — 8 points above Zep (75.1%) and 16 above Mem0's published paper result (67.1%). The jump from our previous 70.6% comes from one architectural change: a chunk-direct evidence lane that retrieves verbatim source dialogue alongside extracted memories. Ingest hash: b3f7e9cad827.

Scores

LoCoMo has four question categories. Our per-category breakdown is below. Competitor per-category numbers are not published by their teams — we only report what is independently verifiable.

System	Overall	Cat 1 Single-hop	Cat 2 Temporal	Cat 3 Multi-hop	Cat 4 Open-domain
MemHQus	83.2%	83.7%	68.9%	64.6%	90.6%
Mem0	67.1%	—	—	—	—
Zep	75.1%	—	—	—	—

Where we fall short

Cat 3 (Multi-hop) is now our weakest category at 64.6% — chained inference across sessions benefits least from verbatim evidence. Cat 2 (Temporal) remains below our other categories at 68.9% despite a +14pp jump. Both gaps are stated here because you should not have to diff our marketing against our data.

Methodology

LoCoMo (arxiv:2402.17753) is a long-context memory benchmark built from real multi-session conversations. It tests whether a memory system can answer questions that require tracking facts across sessions, resolving contradictions, and reasoning about time.

Dataset: 10 conversations, 1,540 QA pairs across 4 categories.
Judge: gpt-4o-mini — scores each answer on a three-level scale (complete / partial / insufficient). A "correct" answer requires a complete or partial score.
Ingest: all 10 conversations ingested in a single batch. No per-question fine-tuning or cherry-picking.
Ingest hash: b3f7e9cad827 — SHA prefix of the ingestion snapshot. Pins the exact data used for QA so results are reproducible against the same snapshot. The report also records the live read-path flag state (CHUNK_DIRECT_K=8, ASK_ANCHOR_VALIDFROM=0) behind the headline number.
Model: MemHQ retrieval + synthesis uses gpt-4o-mini for synthesis and text-embedding-3-large for embeddings.
Competitor scores: Mem0 (67.1%) and Zep (75.1%) are taken from their respective published results. We have not independently reproduced their runs. We report them as-published, not as a direct head-to-head under identical conditions.

Run it yourself

The full evaluation harness — conversation data, judge prompts, scoring rubric, and ingest pipeline — is published on GitHub. You can reproduce our 83.2% result or run it against your own memory system.

GitHub: github.com/MemHQ/memhq/tree/main/bench
Run command: pnpm locomo:parallel (full ingest + QA) or INGEST_HASH=b3f7e9cad827 pnpm locomo:qa-only (QA only, reuses our snapshot).

Honest caveats

Benchmarks measure one dataset at one point in time. LoCoMo is the best public long-term memory benchmark we know of, but it is not a complete picture of real-world performance.
Our score is self-reported. We publish the harness so you do not have to take our word for it.
The gpt-4o-mini judge introduces variance. We estimate ±0.5–1pp run-to-run noise from model non-determinism.
Genesys reports 89.9% with a full-context-style reader — a different architecture class. Among constant-size memory systems (Mem0, Zep, MemHQ) we lead, and we say so with per-category numbers attached.
A previous version of this page reported 70.6% against snapshot 1620eb0ea5c4. That result predates the chunk-direct evidence lane; the delta was measured within a single snapshot with a paired per-question diff (237 fixed vs 24 regressed).