The model was not the problem

Every retrieval, rerank and generation emits a span using the OpenTelemetry GenAI attribute names, so the question "was this the model or the retrieval?" is answered by reading the trace rather than by arguing about it.

Trace 6c77c7208da5433e

ask · 14535 ms · status ok

Question: What is the relief-valve setting on the 4 CFM 3000 PSI charging compressor, and how do I adjust it?
Mode: improved
Documents returned
Answer refused: True
Cost: $0.01480

Span tree

retrieval retrieve

3608.2 ms

9 attributes

atlas.doc_ids ['fist-2-9-2022', 'tm-5-4310-275-14', 'fist-2-9-2022', 'tm-5-4310-275-14', 'tm-5-4310-275-14']

atlas.families ['D', 'F']

atlas.k 5

atlas.mode improved

atlas.n_results 5

atlas.pages [34, 14, 32, 16, 1]

atlas.reranked True

atlas.top_score 1.9655

atlas.withheld 13

llm answer

10926.5 ms

15 attributes

atlas.attempts 1

atlas.context_chars 4668

atlas.cost_usd 0.014796

atlas.degraded False

atlas.latency_ms 10926.4

atlas.n_passages 5

gen_ai.operation.name chat

gen_ai.request.max_tokens 700

gen_ai.request.model claude-sonnet-5

gen_ai.request.temperature 0.0

gen_ai.response.finish_reasons ['length']

gen_ai.response.model claude-sonnet-5

gen_ai.system opusmax

gen_ai.usage.input_tokens 1432

gen_ai.usage.output_tokens 700

The counterfactual: same model, correct passages

The argument that the bottleneck is retrieval is only worth making if it can be falsified. This runs the identical question through the identical model and the identical prompt, changing one thing: the second run is handed the passages the gold set says contain the answer. If the answer becomes correct, the model was never the constraint. The gold passages are the ones pipeline/questions.py located in the corpus, not passages chosen to make the point.

Index built by scripts/build_index.py at 2026-08-10 01:44:34 in 711s: 25 documents, 3,146 pages, 12,590 improved chunks / 5,434 naive chunks. Embeddings: Snowflake/snowflake-arctic-embed-s (int8 ONNX, Apache-2.0). Reranker: cross-encoder/ms-marco-MiniLM-L-6-v2 (ONNX, Apache-2.0).