RAG Reliability study

I created RAG Reliability to examine retrieval failures and answer/defer decisions. The study compares lexical and dense retrieval, hybrid fusion and status-aware reranking under a fixed protocol with a frozen development/holdout split.

A relevant-looking document is not always enough to justify an answer. I wanted to test a stricter rule for cases where lexical and dense retrieval disagree, current and obsolete documents compete, a query needs clarification or the available evidence is not strong enough to answer safely.

• A BM25 lexical retrieval baseline.
• A dense MiniLM retrieval baseline.
• Equal-weight reciprocal-rank fusion (RRF).
• Status-aware current-first reranking.
• Consensus, selective-risk and signal-separability analysis.
• A versioned synthetic benchmark with a frozen development/holdout split.
• A top-1 consensus gate and three success criteria registered before the holdout.
• A one-time unblinded holdout evaluation.
• Deterministic reports and a SHA-256 artifact registry.

• I used only a synthetic benchmark: no production traffic and no real enterprise corpus.
• The holdout stayed fixed at 36 queries with a fixed answer, clarify and abstain composition.
• I froze the gate rules and three success criteria before any holdout evaluation.
• I kept development and holdout separated until the one-time evaluation.
• The holdout is unblinded, so it cannot be reused as unseen evidence.
• I do not retune the gate on this benchmark after observing the holdout.
• Integrity regeneration checks reproducibility only; it does not create new evaluation evidence.
• These results do not support population-level claims or production deployment claims.

1. 01

   Lexical retrieval

   BM25 provides exact-term and token-overlap evidence.

2. 02

   Dense retrieval

   Pinned MiniLM embeddings provide semantic retrieval evidence.

3. 03

   Hybrid fusion

   Reciprocal-rank fusion combines lexical and dense rankings with equal weights.

4. 04

   Status-aware reranking

   Current documents are prioritised over obsolete evidence.

5. 05

   Top-1 consensus gate

   The system answers only when BM25, dense retrieval and the final current-first ranking agree on the same top document; otherwise it defers.

1. 01

   Establish baselines

   I started with lexical BM25 and dense MiniLM retrieval baselines on the synthetic corpora.

2. 02

   Add fusion and document status

   I combined lexical and dense rankings with equal-weight reciprocal-rank fusion and applied status-aware current-first reranking.

3. 03

   Analyse failure signals

   I examined consensus failures, selective risk and signal separability to see when retrieval evidence was strong enough to answer.

4. 04

   Freeze the benchmark

   I built a versioned synthetic benchmark with a frozen development/holdout split before the final evaluation.

5. 05

   Pre-register the gate and criteria

   I registered the top-1 consensus gate and three success criteria before any holdout access.

6. 06

   Evaluate once

   I ran a one-time unblinded holdout evaluation without changing the gate or thresholds afterwards.

7. 07

   Prepare the evidence package

   I assembled deterministic JSON reports, a 33-artifact SHA-256 integrity registry and a versioned study package.

• Versioned synthetic corpora.
• A typed retrieval and evaluation implementation.
• Deterministic JSON reports.
• A pre-registration artefact.
• A one-time holdout report.
• A final study report.
• A 33-artifact integrity registry across 12 study stages.
• Automated integrity and reproducibility checks.
• A versioned study package with SHA-256 verification.

• Python
• uv
• pytest
• BM25
• sentence-transformers
• all-MiniLM-L6-v2
• reciprocal-rank fusion
• SHA-256 artifact verification

• Pre-registered synthetic holdout result

  Frozen success criteria

  PASS 3/3

  PASS 3/3 means the three pre-registered criteria were met on the 36-query holdout: 17/20, 17/22 and 3/14. Those three criteria are the only frozen success criteria. I froze the gate and thresholds before seeing the holdout and did not change them afterwards.

• Synthetic holdout metric

  Answered-query precision

  17/20 = 0.850

  The frozen threshold required at least 0.75.

• Synthetic holdout metric

  Answerable-query coverage

  17/22 = 0.773

  The frozen threshold required at least 0.70.

• Synthetic holdout metric

  False-answer rate

  3/14 = 0.214

  The frozen threshold allowed at most 0.50 for queries expected to defer.

• Descriptive synthetic holdout metric

  Decision accuracy

  28/36 = 0.778

  28/36 = 0.778 is descriptive context reported alongside precision, coverage and deferral errors. It was not one of the three frozen success criteria.

• Reproducibility evidence

  Evidence package

  33 SHA-256-registered artifacts across 12 study stages

  I registered 33 artefacts across 12 study stages with SHA-256. The registry records package contents and integrity; it is not a performance metric.

• This is a single synthetic benchmark with a small fixed 36-query holdout.
• The answer, clarify and abstain composition was fixed.
• I did not evaluate a real enterprise corpus or real support traffic.
• The system does not produce a calibrated probability or confidence score.
• I did not evaluate answer generation.
• Three abstain queries were answered incorrectly.
• Five answerable queries were deferred.
• There is no evidence of generalisation to other corpora, domains or languages.
• The holdout is unblinded and cannot be reused as unseen evidence.
• These results do not show production performance or guaranteed answer correctness.

• Before using this approach with real data, I would create a new representative holdout.
• I would estimate prevalence and the cost of false answers and unnecessary deferrals.
• I would define document lifecycle ownership and status responsibility.
• I would measure retrieval quality, deferral behaviour, latency and cost.
• I would define access controls for sensitive material.
• I would include human escalation for deferred or uncertain cases.
• I would evaluate answer generation separately from retrieval consensus.
• If the gate changed, I would require new confirmatory data before relying on it.

• RAG Reliability Audit of existing systems.
• Answerability and abstention policy design.
• False-answer and evidence-conflict analysis.
• Synthetic and representative evaluation design.
• Status-aware knowledge retrieval.
• Evaluation protocol design.
• Technical evidence and decision reporting.

• View the public repository

  Source code, synthetic corpora, tests, reports and reproducibility controls for the study.