Benchmarks · LoCoMo
Measured honestly, end to end.
We run the LoCoMo long-term conversational memory benchmark against the deployed Cerebro MCP endpoint — the exact surface a real agent uses. No direct-DB shortcuts, no retrieval bypass: every number includes the edge function, RLS, embedding, and search ranking.
| Category | Accuracy |
|---|---|
| Single-hop recall | 42.9% |
| Multi-hop | 25% |
| Temporal reasoning | 18.9% |
| Open-domain | 46.2% |
| Adversarial | 48.9% |
Judged 2026-07-21 · answerer claude-haiku-4.5 (via OpenRouter) · judge claude-sonnet-5, temperature 0 (via OpenRouter)
Methodology first
Ingest. Each of LoCoMo's multi-session conversations becomes one Cerebro project. Every turn is written through the public memory_write MCP tool as an observation (speaker, session, timestamp preserved).
Answer. For each benchmark question we call memory_search (top 10) on the deployed endpoint, then ask claude-haiku-4.5 (via OpenRouter) to answer using only the retrieved memories. All five LoCoMo question categories are included — nothing excluded.
Judge. claude-sonnet-5, temperature 0 (via OpenRouter) grades each prediction against gold with this verbatim prompt — any reply that isn't exactly CORRECT counts as WRONG:
You grade QA. Given QUESTION, GOLD, PREDICTED, reply exactly CORRECT or WRONG. Semantic match counts; missing or contradicting gold is WRONG.
The full harness — ingest, runner, judge, raw per-question results — is committed in our repo under scripts/locomo/, with each run's summary.json as the audit trail behind the figures on this page.
Caveats — read these
- Vendor LoCoMo numbers are disputed. Zep revised its headline 84% claim down to ~75% after methodology criticism; Mem0's independent reproduction of Zep's setup landed near 58%. Treat every vendor's single figure — including ours — as a claim to re-run, not a fact.
- Ours is deliberately conservative. We measure the full deployed path with a cheap answerer model and a strict judge. Skipping the production surface or upgrading the answerer would inflate the number; we'd rather publish the one an actual agent gets.
- This run is preliminary. Figures above cover 1 of 10 locomo conversations (preliminary — full run pending). The page updates — with the committed raw results — as the full run lands.