Research and benchmarks

Storage and indexing measured at scale, and whether any of it helps an agent actually answer correctly — LongMemEval, end to end.

Two different questions, measured separately, because a fast index that retrieves the wrong thing is not a working memory. Storage and indexing below answers "does one repository scale." Memory quality answers "does an agent that uses this actually answer better" — the real LongMemEval benchmark, run end to end: ingest, retrieve, answer, grade.

Storage and indexing

The harness generates synthetic corpora at multiple scales with realistic tier proportions, and measures corpus import, cold build, incremental sync, and query latency (p50/p99) for both tree navigation and selective full-text search:

go run ./bench -n 1000,10000,100000 -changed 50 -queries 500

Import and cold-build are go-git/SQLite bulk work and swing with machine load; the query and incremental-apply numbers are the meaningful ones — search stays in the hundreds of microseconds and navigation in the tens, flat across two orders of magnitude of corpus size, because both are O(direct children)/O(matches), not O(corpus).

On a DGX Spark (GB10, arm64), the same CGo-free binary cross-compiles with a plain GOARCH=arm64 go build and runs untouched, faster on the import/build stages thanks to unified LPDDR5X memory:

What actually scales with corpus size, and what does not: the index apply is genuinely O(changed) — 50 changed files out of 100k costs ~132ms of SQLite work regardless of corpus size. What does grow is git's own tree diff (~3.4s of a 100k sync), which scales with tree bushiness and object-store packing, not file count — in real use (small commits, packed objects, diffing against the immediately prior commit) this is small; the synthetic worst case is a wide tree of loose objects diffed against a 100k-file batch in one go.

Vector search at scale

Brute-force cosine scan over 384-dimension vectors, one query against the whole corpus:

Linear and I/O-bound, as expected for an exact scan — not a design flaw, a known trade for staying CGo-free (see How GitLoom works for why). ~100k–250k vectors per namespace keeps the scan under 30–70ms and is the practical ceiling before an ANN index would start paying for itself — the reason the recommended shape is one namespace per end user rather than one shared corpus, see Namespaces and multi-user patterns.

Memory quality (LongMemEval)

cmd/gitloom-longmem runs the real LongMemEval oracle set (500 instances, 6 question types) end to end: ingest each instance's chat history into a fresh GitLoom memory, answer the question with the tool-using agent retrieving from that memory, judge the answer with an abstention-aware LLM grader. Every round below was diagnosed from the previous round's actual failures, pulled from real tool traces — not guessed.

(v2 isn't a column — a stratified 48-instance diagnostic sample, not a full run. v7 measured n=499, one short of the full 500-instance oracle set, recorded as observed rather than squared to 500. v6 is a separate, non-comparable run — see v6 — Bedrock, multi-judge panel below.)

44% → 91.4%, driven by treating the harness's own per-instance tool traces as the input to the next fix rather than a scoreboard:

• v1 → v3 (+28 pts). The baseline told extraction to "ignore assistant chatter," which starved single-session-assistant to 12%; v3 captures assistant-provided facts atomically, rewrites retrieval into a thorough multi-search protocol, and injects a topic map so the agent starts oriented.
• v3 → v4 (+8 pts). Classifying all 116 non-abstention failures found 45% had already retrieved the right memory and answered "I don't have that information" anyway — a prompt bug, not a retrieval failure. v4 forbids declining when search returned results, adds semantic search (below), and a parallel prefetch that fires the topic map, a lexical search, and a vector search concurrently before the agent's first turn.
• v4 → v5 (+3 pts). Most remaining failures were counting and ordering slips on evidence already found — off-by-one counts, wrong "most recent." v5 adds an explicit evidence-table protocol: enumerate every candidate with its date, dedupe, filter, sort, and only then read off the answer.
• v5 → v7 (+8.4 pts, no harness or model change). Same model (deepreinforce-ai/Ornith-1.0-35B-FP8), same judge, same hybrid semantic search (bge embeddings) as v5 — nothing about the benchmark changed. What changed is two weeks of engine work landing in between: deterministic retrieval fixes, cue-based and entity-aware retrieval, and relationship edges gaining a validity window so an update to a fact supersedes what it contradicts instead of returning both. The gain is broad rather than concentrated in one category, consistent with a retrieval correctness fix rather than a prompt tweak.

Embeddings, isolated

Same binary and prompts, only the embedding endpoint toggled — an honest A/B, not two different runs compared after the fact:

61 instances were fixed by embeddings, 41 were broken by them (McNemar χ²=3.54, short of the 3.84 needed for p<0.05 at this n — real but not yet statistically proven). The gains concentrate where vocabulary mismatch is expected (multi-session +11, knowledge-update +8, preference +7); the losses are lexically-easy cases where the fused ranking occasionally displaces an exact keyword hit.

Retrieval latency, inside the QA loop

Measured on the v5 run (500 instances, concurrency 48):

Retrieval is 0.11% of the QA turn. Each additional search/recall round-trip costs ~1.6s of LLM latency, not retrieval time — retrieval itself stays single-digit milliseconds regardless of call count. Ingest, not QA, dominates end-to-end cost (63s vs 13.4s mean) and is the right target for further latency work.

v6 — Bedrock, multi-judge panel

100 instances, stratified across all six question types, run entirely on Amazon Bedrock (the configuration the hosted platform actually runs) with a cross-family judge panel — Haiku 4.5, Ministral 3 14B, Ministral 3 8B, majority vote — so one grader's quirks do not read as a memory-quality difference.

The 10-point gap between "evidence supported" and "answer-only strict" is a summarizer problem, not a retrieval one — the panel grades the two independently for exactly this reason. A follow-up run at adequate token settings (the first pass used an undersized -max-tokens that truncated extraction and starved multi-session specifically) recovered most of that category to be in line with the rest; the corrected headline number is not yet re-measured end to end. Not comparable to v5's 83% — different model, different scale, lexical-only retrieval (no embedding endpoint was available for this run), different judge panel. The matching headline number is a coincidence.

Reproducing these numbers

go test ./...      # unit + integration tests across every package
go vet ./...

wget -P data https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_oracle.json
gitloom-longmem -dataset data/longmemeval_oracle.json -concurrency 40 \
  -embed-endpoint http://localhost:8001/v1 -embed-model bge \
  -outdir runs/my-run -keep-repos

Every run persists a crash-safe results.jsonl — per-instance tool traces, timings, verdicts — plus, optionally, every instance's memory as a real kept git repository: the raw material every table on this page was built from. Nothing here is an estimate presented as a measurement; where a figure is a range, it is written as one.