Blog/Benchmarks
Benchmark methodology: what we measure and what we don't
Our benchmark runs the engine's real code on a fixed, synthetic corpus. Here is exactly what that proves, and the claims it does not support.
| Group | Share of cases |
|---|---|
| Answer in top highlight | 86.2% |
| Random chunk | 18.0% |
The ContextFetch team
Numbers on a pricing page are easy to write and hard to check. Ours come from one script, benchmarks/run_benchmark.py. It imports the engine's real modules, with no copies and no mocks, works offline by default, and writes everything, including per-case detail, to results.json. We publish that file as is.
The corpus is synthetic, and we say so
The test documents are 50 Markdown files written for this repository about fictional projects: READMEs, API references, changelogs and tutorials, about 2.9 KB each. They are not crawled web pages. Every figure that uses them, for compression, vector recall and pruning, is a measurement on that corpus only. Every fifth file is held out, the same split the engine tests use.
Pruning: the headline number
For 29 fixed question and document pairs, each with a phrase that answers the question and occurs in exactly one chunk, we run the production pruning function with its API defaults and count tokens with the engine's own estimate, a quarter of the characters.
| Measure | Result |
|---|---|
| Tokens saved, mean | 69.5% |
| Tokens saved, median | 79.0% |
| Answer in the top highlight | 86.2% |
| Answer in the top 3 | 96.6% |
| Answer kept in the pruned context | 93.1% |
| Random-chunk baseline | 18.0% |
Two things follow. The site says “69% fewer tokens”, the mean rounded down, never up, and always next to “29 synthetic docs; not yet measured on live pages”. And in 2 of the 29 cases the saving came from dropping the answer, which is why we report the hit rates next to the savings.
Storage and vectors
- zstd with a trained dictionary: 2.64x on held-out docs, against 2.06x without one. What storage actually achieves is 1.98x, because the compressed bytes are base64-encoded into the index payload. We quote 1.98x for capacity, not 2.64x.
- 1-bit vectors: 48 bytes instead of 1,536. Binary-only recall@10 is 0.52 and 0.63 on our two query sets; rescoring the binary top-40 with full vectors lifts it to 0.86 and 0.90. The corpus is small, so a top-40 shortlist covers 15 to 19% of it. Expect lower recall on millions of documents.
- Extraction is not compression. Raw HTML to stored bytes looks like 179.74x on our fixtures, but most of that is the extractor throwing markup away. We label it extraction plus compression, lossy, and never call it a compression ratio.
What we don't measure
- Competitor latency and quality. The script can time Exa and Tavily, but only with your keys and your credits. Without them it reports “not measured”, and so does our site. We compare competitors on list price and features only.
- Token savings on live pages. Not measured yet. Real pages are longer than our 761-token average, so the number could go either way.
- Latency on an idle machine. The final run started at a load average of 7.7 on 8 cores. Treat its component timings as upper bounds for that laptop.
End to end
Search latency through the whole stack comes from a separate five-query script, run before and after each change and against production. The raw files are on the benchmarks page, and the latency post walks through them.
More from the blog
| Group | Cold p50 |
|---|---|
| Before | 4,140 ms |
| After | 1,644 ms |
How we cut cold search from 4.1 s to 1.6 s
The ContextFetch team
HTTP/2 402 x-payment-url: https://api.contextfetch.dev/buy/instant {"error": "keyless_limit", "action": {"type": "payment_link", ...}, "packs": [...]}
Keyless MCP and the 402 payment loop for agents
The ContextFetch team