Benchmark · LongMemEval-S 2026-05-18 · Audit SC-HEK-2026-05-17-KNM
knowmind compared.
Three of the most common search architectures — knowmind, plain full-text search and naive substring search — against an identical test corpus, the same questions, the same environment. Supplemented with reference figures other vendors publish in their whitepapers.
Important for context: this is not a direct head-to-head benchmark against Mem0, Zep, Cognee or Letta. The competitor figures come from their own published benchmarks (LongMemEval, LOCOMO, DMR) with different corpora and methods. The knowmind figures are measured by us. Direct comparison is therefore only approximate. Raw data and benchmark script are available on request for reproduction.
Own benchmark, as of 23 May 2026
Test corpus of 1,219 memories from the internal workspace of Schübeler Consulting. 20 held-out questions that were not used while tuning the pipeline. Recall@5 means: at least one of the five best hits contains the right source according to the gold standard. Scripts, gold standard and raw data are published reproducibly at github.com/Schubeler-Consulting/knowmind-benchmark. With a sample of 20, the 95 per cent confidence interval for 80 per cent recall works out at roughly 56 to 94 per cent. Your own corpus may land above or below that.
| System | Strategy | Recall@5 | Latency | Hosting | Licence | Own AI subscription |
|---|---|---|---|---|---|---|
| knowmind | Combines full text, semantic search and knowledge graph; names and people are resolved deliberately (identity routing) | 80.0 % (n=20) | 475 ms | DE (Hetzner) | Proprietary, SaaS | optional |
| Classic vector RAG | Pure embedding retrieval without hybrid stages | 65.0 % (n=20) | 491 ms | identical | Open source | n/a |
| File-name search | Token matches in the file name, top-K by number of matches | 55.0 % (n=20) | 2 ms | identical | Open source | n/a |
| Full-text search (ripgrep) | Keyword matches across all files, top-K by number of matches | 55.0 % (n=20) | 27 ms | identical | Open source | n/a |
Reference figures from published benchmarks.
Other memory vendors and search stacks — figures from their own whitepapers or from independent benchmarks. The corpora differ (LongMemEval, LOCOMO, DMR); direct comparison is only approximate.
| System | Strategy | Recall@5 | Latency | Hosting | Licence | Own AI subscription |
|---|---|---|---|---|---|---|
| Zep | Knowledge graph with a time axis | ~76 % (LongMemEval, vendor figure) | — | US (AWS) | Apache 2.0 + SaaS | partly |
| Cognee | Graph-based retrieval | ~75 % (own evals) | — | US / EU | MIT + SaaS | partly |
| Letta (MemGPT) | Hierarchical memory | ~70 % | — | US | Apache 2.0 + SaaS | yes |
| Mem0 | Fact extraction by an LLM | ~70 % (LongMemEval, vendor figure) | — | US | OSS + cloud | yes |
| Pinecone (plain) | Pure vector search | ~60 % | — | US (AWS/GCP) | Proprietary, SaaS | n/a |
| Elasticsearch | Keyword search | ~25–45 % | — | self-hosted | Elastic License | n/a |
| ChatGPT long context | Whole corpus in the prompt | limited | — | US | Proprietary | — |
Method
- 20 typical knowledge-work questions in German, gold standard with two to five expected keywords per question.
- Recall@5: one of the five best hits contains at least one expected keyword.
- knowmind called through its official interface, the comparison methods run on the same test corpus.
- Reference figures for other vendors taken from their published whitepapers.
- Response time: mean after a warm-up phase, measured at the Hetzner site in Nuremberg.
Raw data and script on request. The figures shown here come from an internal test corpus; all content is internal and not public. For an M&A diligence we make the full data set available under NDA.
Measure it against your own knowledge
Put your own notes in and ask your own questions. Whether the right source is among the first hits, you can see on your own material in a few minutes.