Baseline, measured
The same naive pipeline, run over a fixed set of expert-written questions with ground truth located in the corpus itself. Retrieval metrics need no model and no judge; they are deterministic and they are the ones that diagnose the failure.
Hit rate
10%
a passage that answers the question arrived at all, k=5
Source correctness
20%
rank 1 is the right machine and a current revision
Wrong machine at rank 1
40%
rank 1 is another machine's manual
Correct manual anywhere in top-5
70%
the ceiling on what any model could do
This is the number to show first
On 12 questions the correct manual reaches the context window
70% of the time, and the passage at rank 1 is the right machine on
a current revision 20% of the time. No prompt, no model and no
temperature setting can raise correctness above the evidence that reached the context window. A
room that believes it has a model problem is usually looking at this.
Reading context recall honestly. Raw context recall here is
1.4%, and on its own that number is misleading. Several of these
questions are answered in a dozen or more places in their own manual, so retrieving five
passages has an arithmetic ceiling of k divided by the size of the gold set — a mean ceiling of
86.9% on this set. Against that ceiling the baseline
achieves 10%. Both numbers are in the table below.
Baseline metrics
Per question
| ID | Question | Recall | MRR | Rank-1 document | Verdict | |
|---|---|---|---|---|---|---|
| Q1 | What is the torque spec for the compressor connecting-rod bearing cap nuts on the 5 CFM 17 | 0.00 | 0.00 | tm-5-4310-339-15 | wrong source | trace |
| Q2 | How often should the compressor oil filter be cleaned on the 125 CFM 100 PSI skid-mounted | 0.00 | 0.00 | doe-sourcebook | wrong source | trace |
| Q3 | What is the lubricant level check interval for a reciprocating air compressor under the cu | 0.00 | 0.00 | doe-sourcebook | wrong source | trace |
| Q4 | Give me the crankcase oil change procedure for the 15 CFM 175 PSI tank-mounted Bauer compr | 0.00 | 0.00 | tm-5-4310-360-14 | wrong source | trace |
| Q5 | What is the intake valve part number for the 1986 hand-truck mounted 5 CFM 175 PSI compres | 0.00 | 0.00 | tm-5-4310-378-14 | correct source | trace |
| Q6 | How often should compressor pressure gauges be calibrated? | 0.00 | 0.00 | fist-2-6-2026 | wrong source | trace |
| Q7 | What is the safe operating pressure and relief-valve setting for the receiver on our 15 CF | 0.00 | 0.00 | fist-4-1a-2020 | wrong source | trace |
| Q8 | Which model number does the 5 CFM 175 PSI Champion compressor use, OEH-34-60 or LP-512? | 0.14 | 1.00 | tm-9-4310-241-15 | correct source | trace |
| Q9 | What does Change 1 say about the 125 CFM rotary compressor? | 0.00 | 0.00 | tm-5-4310-221-15 | wrong source | trace |
| Q10 | What is the air intake filter service interval for the Davey 6M125 125 CFM diesel compress | 0.00 | 0.00 | tm-9-4310-241-15 | wrong source | trace |
| Q11 | What is the torque spec for the Atlas Copco GA 30 VSD+ gearbox? | — | — | — | abstention item | trace |
| Q12 | What is the recommended PID tuning for the variable-speed drive controller on a modern oil | — | — | — | abstention item | trace |
How the ground truth was established
Each question carries a regular expression and the document that should answer
it. pipeline/questions.py runs those patterns over the built index and records the
chunk ids that actually contain the answer. A pattern that matches nothing is reported as
unresolved and excluded from scoring rather than counted as a failure. The resolved set is
written to data/build/goldset.json.
| ID | Question | Correct source | Gold chunks | State |
|---|---|---|---|---|
| Q1 | What is the torque spec for the compressor connecting-rod bearing cap nuts on the 5 CFM 175 PSI hand-truck compressor? | tm-9-4310-241-15 | 2 | resolved |
| Q2 | How often should the compressor oil filter be cleaned on the 125 CFM 100 PSI skid-mounted gasoline compressor? | tm-5-4310-221-15 | 1 | resolved |
| Q3 | What is the lubricant level check interval for a reciprocating air compressor under the current maintenance schedule? | fist-4-1a-2026 | 6 | resolved |
| Q4 | Give me the crankcase oil change procedure for the 15 CFM 175 PSI tank-mounted Bauer compressor. | tm-5-4310-363-14 | 4 | resolved |
| Q5 | What is the intake valve part number for the 1986 hand-truck mounted 5 CFM 175 PSI compressor unit, model ZPC175/5? | tm-5-4310-378-14 | 1 | resolved |
| Q6 | How often should compressor pressure gauges be calibrated? | fist-4-1a-2026 | 3 | resolved |
| Q7 | What is the safe operating pressure and relief-valve setting for the receiver on our 15 CFM 175 PSI tank-mounted compressor? | tm-5-4310-363-14, tm-5-4310-277-14, tm-9-4310-227-15, tm-5-4310-339-15 | 3 | resolved |
| Q8 | Which model number does the 5 CFM 175 PSI Champion compressor use, OEH-34-60 or LP-512? | tm-9-4310-241-15, tm-9-4310-241-35p | 39 | resolved |
| Q9 | What does Change 1 say about the 125 CFM rotary compressor? | tm-5-4310-354-14 | 12 | resolved |
| Q10 | What is the air intake filter service interval for the Davey 6M125 125 CFM diesel compressor? | tm-5-4310-354-14 | 3 | resolved |
| Q11 | What is the torque spec for the Atlas Copco GA 30 VSD+ gearbox? | — | 0 | resolved |
| Q12 | What is the recommended PID tuning for the variable-speed drive controller on a modern oil-injected screw compressor? | — | 0 | resolved |
Evaluation computed by scripts/run_evals.py at 2026-08-10 07:29:29 over 12 questions at k=5, generation off. Re-runnable live from this page.