github-featured/rohitg00--agentmemory
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity

docs(benchmarks): correct coding-agent-life-v1 P@5 numbers (#796) (#805)

Browse Source 
* docs(benchmarks): correct coding-agent-life-v1 P@5 numbers

* Update 2026-05-20-coding-agent-life-v1.md
This commit is contained in:
Rohit Ghumare
2026-06-03 10:51:50 +01:00
committed by GitHub
parent 7446bded53
commit e5c9dc68e4
1 changed files with 49 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/benchmarks/2026-05-20-coding-agent-life-v1.md
+49 -24
View File
@@ -1,51 +1,76 @@
# 2026-05-20 — coding-agent-life-v1 (v0.9.21)
# 2026-05-20 — coding-agent-life-v1 (v0.9.26)
**Commit:** `e9dc710`
**Bench:** coding-agent-life-v1 (15 sessions, 15 queries)
**N:** 15
**K:** 5
**Hardware:** macOS 15 (Apple Silicon)
**agentmemory:** v0.9.21
**agentmemory:** v0.9.26
**iii-engine:** v0.11.2
**Embedding provider:** local default
**Sandbox:** isolated data dir at `/tmp/agentmemory-eval-sandbox/`, ports 3411/3412
## Math ceiling on this dataset
12 of 15 questions have 1 gold session, 3 have 2 gold sessions. Per
`scoreQuestion` in `eval/runner/score.ts`, P@K = `hits / k` averaged
across questions, so the **maximum achievable P@5** is:
```
(12 * 1/5) + (3 * 2/5)) / 15 = (2.4 + 1.2) / 15 = 0.240
```
R@5 ceiling is **1.000** (every gold session found in top-5).
The benchmark is **small** (15 questions) and **gold-sparse** (mostly
single-gold). It's tuned for fast iteration on the retrieval stack,
not for headline P@K comparisons. **Recall** + per-question-type
**P@5** are the signals; aggregate P@5 saturates at 0.240 so it can't
differentiate top-tier adapters.
## Headline
`agentmemory-hybrid` hits **100% top-5 hit rate**, R@5 = **0.967**, P@5 = **0.578**.
`agentmemory-hybrid` hits **100% top-5 hit rate**, R@5 = **1.000**,
P@5 = **0.240** (at the math ceiling — every gold session retrieved
in top-5).
Same corpus, grep baseline: R@5 = 0.967, P@5 = 0.267 — same recall, but **2.2× worse precision**. Hybrid's top-5 is mostly gold; grep's top-5 is half noise.
grep baseline: R@5 = **0.967**, P@5 = **0.227** — missed one gold
session in one multi-gold question. Lift is **recall**, not aggregate
precision.
## Per-adapter
| Adapter | P@5 | R@5 | Hit rate | p50 latency |
|---|---|---|---|---|
| grep (tokenized substring) | 0.267 | 0.967 | 15 / 15 | 0 ms |
| `agentmemory-hybrid` | **0.578** | **0.967** | **15 / 15** | 14 ms |
| grep (tokenized substring) | 0.227 | 0.967 | 15 / 15 | 0 ms |
| `agentmemory-hybrid` | **0.240** | **1.000** | **15 / 15** | 14 ms |
`agentmemory-hybrid` runs through the production smart-search endpoint (`POST /agentmemory/smart-search`) so it exercises the full BM25 + embedding + reranker stack.
## Per-question-type
P@5, grep vs `agentmemory-hybrid`:
At K=5 with 1 gold per single-session question, the P@5 ceiling per
question is **0.20**; with 2 gold the ceiling is **0.40**. Both
adapters saturate the per-type ceiling on most types, so the per-type
table primarily exposes where one adapter **misses** gold (failing
the recall side).
| Type | grep | hybrid | hybrid lift |
|---|---|---|---|
| single-session-bug | 0.20 | 0.33 | 1.7× |
| single-session-infra (n=2) | 0.20 | 0.50 | 2.5× |
| single-session-refactor | 0.20 | 0.50 | 2.5× |
| single-session-feature | 0.50 | 0.50 | tie |
| single-session-test | 0.20 | 0.33 | 1.7× |
| single-session-perf | 0.20 | 0.50 | 2.5× |
| single-session-api | 0.20 | 0.50 | 2.5× |
| single-session-db | 0.20 | 0.50 | 2.5× |
| single-session-release | 0.20 | 0.33 | 1.7× |
| multi-session-causal | 0.40 | 0.40 | tie |
| preference (n=2) | 0.20 | 0.42 | 2.1× |
| multi-session-review | 0.40 | 0.67 | 1.7× |
| temporal (R@5 = 0.50 grep / 1.00 hybrid) | 0.50 | 0.67 | 1.3× |
| Type | grep P@5 | grep R@5 | hybrid P@5 | hybrid R@5 |
|---|---|---|---|---|
| single-session-bug | 0.20 | 1.00 | 0.20 | 1.00 |
| single-session-infra (n=2) | 0.20 | 1.00 | 0.20 | 1.00 |
| single-session-refactor | 0.20 | 1.00 | 0.20 | 1.00 |
| single-session-feature | 0.20 | 1.00 | 0.20 | 1.00 |
| single-session-test | 0.20 | 1.00 | 0.20 | 1.00 |
| single-session-perf | 0.20 | 1.00 | 0.20 | 1.00 |
| single-session-api | 0.20 | 1.00 | 0.20 | 1.00 |
| single-session-db | 0.20 | 1.00 | 0.20 | 1.00 |
| single-session-release | 0.20 | 1.00 | 0.20 | 1.00 |
| multi-session-causal (2 gold) | 0.40 | 1.00 | 0.40 | 1.00 |
| preference (n=2) | 0.20 | 1.00 | 0.20 | 1.00 |
| multi-session-review (2 gold) | 0.40 | 1.00 | 0.40 | 1.00 |
| temporal (2 gold) | 0.20 | 0.50 | 0.40 | 1.00 |
Temporal queries (`What was shipped on April 8th 2026?`) need both gold sessions to score full recall. grep finds 1/2; hybrid finds 2/2.
The differentiator at this corpus size is **temporal** (`What was shipped on April 8th 2026?`): grep finds 1 of 2 gold sessions, hybrid finds both. Per-type R@5 saturates everywhere else.
## Methodology