How we cut cold search from 4.1 s to 1.6 s

A search API for agents sits inside someone else's loop. Every second we spend is a second the agent waits. So before launch we set four targets for one fixed set of five queries, measured on our Apple M2 test machine through the full Docker stack and our HTTPS proxy:

• Recall: every query returns at least 8 relevant results with at least 500 characters of fetched page Markdown.
• Cold latency: under 1.2 s with text and highlights.
• Warm latency: the same request repeated in under 35 ms.
• Responsiveness: /health p95 under 50 ms while 16 searches run at once.

The baseline was slow. Cold p50 was 4,140 ms and the worst query took 6,356 ms. Repeating a query took 2.8 to 7.3 s, because there was no cache at all.

What we changed

Recall in parallel, under a budget. Every source is queried at once: the metasearch layer (pages 1 and 2), Wikipedia full-text search, Hacker News, Stack Exchange and GitHub (only for technical queries), arXiv (only for academic ones, at most one request every 3 s) and our own index. Results are fused with reciprocal rank fusion. A source that misses the budget is dropped for that request, never awaited.

A fetch phase with a deadline. Fetching starts as soon as the first source answers, highest fused rank first, with up to 18 requests in flight and a 1,200 ms cap per URL. The phase ends when numResults + 3 usable pages are in, or 950 ms after the first fetch, and anything still running is cancelled. Error pages are rejected before parsing, and HTML extraction runs on a process pool so it never blocks the event loop.

One batched rerank. All page-ranking and highlight pairs for a request go to the cross-encoder in a single call, at most 3 candidate chunks per page, with input capped at 256 tokens. In the engine container that costs 21 ms per pair on 4 threads and 33 ms on 2.

Two caches, kept apart from billing. A 15-minute query cache and a 48-hour URL content cache, compressed with zstd, live in a separate Redis that is allowed to evict. Credit balances live elsewhere, so eviction can never touch money. A cache hit takes 1 to 4 ms at the server.

The numbers

Where the time goes now, from the server timings in the response for the slowest cold query (“python asyncio timeout”): recall 490 ms, fetch 635 ms, rerank 1,400 ms. The floor on this machine is recall plus 0.6 to 1.4 s of cross-encoder time.

The honest part

Not all of that drop is engineering. Between the two runs our web recall changed and fewer web pages came back from the metasearch layer, so there was less to fetch and less to rerank. Our own write-up says plainly that this explains most of the gap. The last tuning run with fuller web recall had a cold p50 of 2,723 ms: still a real improvement on 4,140 ms, but not 1,644 ms.

Against the four targets:

• Recall: missed. Results with full Markdown were 10, 7, 0, 9, 5. One query returned nothing.
• Cold latency: missed. p50 1,644 ms is 0.44 s over the 1.2 s target.
• Warm latency: met for the 4 queries with results (6 to 14 ms at the client). An empty response is not cached, so the fifth repeat took 1,842 ms.
• Responsiveness: met in the load run: /health p95 of 35.6 and 31.1 ms with 16 concurrent searches.

In production

On 2026-10-17 we ran the same five queries against the production API from a client in Florida, across the Atlantic from the server. Every query returned 10 results. Cold requests took 2.2 s to 3.5 s (p50 3,141 ms), and repeats, answered from the query cache, took 143 ms to 282 ms including the network. Cold search is still the number we most want to bring down, and recall quality comes first.

Every response carries these timings in its contextfetch field, so you can check them yourself in the playground.