The identical evaluation, re-run
Same 12 questions, same ground truth, same k. The only thing
that changed is the pipeline. Both columns were produced by the same code path — the precomputed
numbers on this page and the numbers from the live re-run button come from one implementation of
run_eval.
Baseline against all three improvements
Which change earned which movement
The ladder tells you the total. It cannot tell you the attribution, because each rung inherits
the ones below it. The table below runs each branch alone against baseline, which is the
only way to say honestly that hybrid search bought this and reranking bought that.
Two of these branches do not do what the room expects.
Layout-aware chunking on its own makes the wrong-machine rate worse, because finer
chunks mean more passages from the same page competing for the same five slots; its duplicate-slot
count is the highest of any configuration. And adding hybrid search to it lowers hit rate while
raising source correctness, because BM25 pulls the right document to rank 1 on model codes and
displaces some gold passages out of the top five at the same time. The cross-encoder is doing
most of the work here. That is not the story anyone plans to tell, and it is what the numbers
say.
Attribution: each branch on its own
| Branch | Hit rate | Source correctness | MRR | Wrong machine's manual at rank 1 |
|---|---|---|---|---|
| baseline none of the three |
10% | 20% | 10% | 40% |
| structure Layout-aware extraction + parent-child chunks + family/revision metadata | 20% +10 | 30% +10 | 15% +5 | 50% +10 |
| hybrid BM25 fused with dense retrieval (reciprocal rank fusion) | 30% +20 | 40% +20 | 18% +7 | 30% -10 |
| rerank Cross-encoder reranking + revision and variant guards | 50% +40 | 70% +50 | 43% +33 | 20% -20 |
| improved all three together |
30% | 80% | 25% | 0% |
The cumulative ladder
Re-run it now
Runs the same question set through the same harness in this process. Retrieval metrics only, so no model call and no API key is required.
Evaluation computed by scripts/run_evals.py at 2026-08-10 07:29:29 over 12 questions at k=5, generation off. Re-runnable live from this page.