How we know it works

method

A held-out set of real queries with recorded expected behaviour, scored per change rather than per release. Answers are graded on whether the cited source supports them, not on whether they read well.

what it tells us

The gap between models on a public benchmark rarely survives contact with a specific corpus. We have had a smaller model win on a client's documents against a larger one that beat it everywhere else, which is why the set has to be yours.

method

Recall measured at fixed cut-offs against a labelled set, with the keyword and vector halves scored separately before the hybrid result is scored at all.

what it tells us

Pure vector retrieval loses exact matches: part numbers, invoice references, surnames. Those are a large share of what people actually search for, which is why every retrieval system we ship is hybrid.

method

The same corpus indexed several ways, then run against the same question set, so chunk size stops being a matter of opinion.

what it tells us

Chunk boundaries do more damage than chunk size. A table split across two chunks answers confidently and wrongly, and no amount of reranking recovers it.

method

The evaluation set runs on a schedule against production, not only at release, with the score kept next to latency and cost per call.

what it tells us

AI systems fail quietly. Nothing throws an error; answers just get worse as documents age and providers update models underneath you. Without a scheduled run, the first report of a problem comes from a customer.

method

Every tool an agent can reach is written down with its blast radius, then tested with cases designed to make the agent overreach.

what it tells us

The useful question is not what an agent can do but what it can do wrongly at three in the morning with nobody watching. Most agent designs answer the first question and never ask the second.