ADEBENCH — agent memory benchmark and leaderboard
A benchmark for agent memory. It scores the text a memory actually delivers to the model, on one golden set, with no LLM judge.
| # | Memory | Core · 55 | public2 · 55 | Full |
|---|---|---|---|---|
| 1 | Dakera |
94% 51.9 / 55 · 1 run |
waits for a Docker host |
97% 96.9 / 100 |
| 2 | ADE Brain the memory of adebench's author, and its code is private: nobody else can rerun this row. It runs on the same set and rules as every other memory |
89% 48.8 / 55 · same in 5 runs |
public2 run in progress |
94% 93.8 / 100 |
| 3= | Aionforge |
76% 41.8 / 55 · 1 run |
waits for a Docker host |
80% 51.8 / 65 |
| 3= | supermemory |
74% 40.5 / 55 · 38.1–43.0 · 5 runs |
public2 run in progress |
78% 50.5 / 65 · 48.1–52.9 |
| 3= | gbrain |
73% 40.4 / 55 · same in 3 runs |
78% 43.1 / 55 · same in 3 runs |
84% 75.4 / 90 |
| 5 | Hindsight |
66% 36.3 / 55 · 1 run |
waits for a Docker host |
66% 36.2 / 55 |
| 6= | agentmemory |
65% 35.5 / 55 · same in 3 runs |
70% 38.6 / 55 · same in 3 runs |
66% 42.6 / 65 |
| 6= | agentmemory (Gemini) |
65% 35.5 / 55 · same in 5 runs |
public2 run in progress |
66% 42.6 / 65 |
| 6= | engram |
64% 35.2 / 55 · same in 3 runs |
44% 24.4 / 55 · same in 3 runs |
70% 45.2 / 65 |
| 6= | Memoose |
63% 34.6 / 55 · same in 3 runs |
56% 30.7 / 55 · same in 3 runs |
76% 68.3 / 90 |
| 7= | cognee |
63% 34.4 / 55 · 33.7–35.4 · 5 runs |
public2 run in progress |
68% 44.4 / 65 · 43.7–45.4 |
| 7= | mem0 |
62% 34.2 / 55 · 30.4–35.5 · 5 runs |
public2 run in progress |
68% 44.2 / 65 · 40.4–45.4 |
| 12= | memU |
59% 32.2 / 55 · same in 5 runs |
public2 run in progress |
65% 42.1 / 65 |
| 12= | Nemp |
58% 32.1 / 55 · 1 run |
runs inside Claude Code; its author can run it |
56% 36.4 / 65 |
| 12= | Jev-Mem |
58% 31.9 / 55 · 28.9–33.0 · 6 runs |
67% 37.1 / 55 · 36.6–37.7 · 5 runs |
64% 41.9 / 65 · 39.0–42.9 |
| 12= | TokenMizer |
57% 25.6 / 45 · same in 5 runs · no time |
58% 32.0 / 55 · 27.5–39.5 · 5 runs |
57% 25.6 / 45 |
Core is what every memory can be measured on, a write and a read through the door: door 25, updates 10, time 10, abstention 10. The ranking is on it. Full adds what a memory has behind the door (cards, live state, file search, graph), over the points it could be measured on. Chars per answer is what the model receives for one question: the same score at 300 characters and at 20,000 is not the same memory. Write → visible is how long a value just written takes to reach the door. Scores are the mean of the runs kept for each memory, with their range; two memories closer than the wider of their ranges (never less than one probe, 1.8 points) share a rank, marked =, and no memory ranks above one with a higher mean.
public2 is the core on a second public set, three times larger: 24 door questions and 12 invented entities, written by another model (Gemini) from a fixed specification and accepted only by a mechanical validator. It is shown beside the board as each memory is run on it, and does not change the ranking. A private set of the same size, which no session that develops a memory can read, checks each memory's public2 number: its outcome is on the memory's page.
| Memory | Score | Probes |
|---|---|---|
ADE Brain |
98% 98.0 / 100 |
165 set 9a276b567d |
The golden set is small on purpose, so it runs anywhere. The bench was built for something else: a memory measured on its owner's data, with probes written on the facts it really holds, rerun after every change. Not comparable across memories, and the number that tells an owner whether a change helped. Totals only: the probes are the owner's facts.
Method
Eight sections: door, cards, updates, time, live state, abstention, file search, graph. A point is earned when the expected words are in the text the model receives; a retired value next to the current one is a failure.
Run it
No dependencies. Every adapter and importer is in the repository, and each memory's page names the configuration that produced its numbers.
python -m adebench --help
Your memory
Write your probes, point the harness at your memory, send the totals with a pull request. The set's hash ties a number to the probes that produced it.
GitHub
adebench is MIT. Adapters, sets, reports: adecubed/adebench. Each memory's author is told when its results go up; if we ran it wrong, we fix it and rerun.