RAG Reliability study

RAG Reliability is a proprietary product direction for deciding when a RAG system should answer or defer. Study v1 uses a synthetic Atlas Support Operations corpus with a frozen holdout so that evaluation, failure analysis and artifact integrity remain reviewable. Public results are summarised on this page; implementation details and full artifacts remain private. A controlled public demonstration of the Explorer is linked from this page.

High retrieval relevance alone does not guarantee that a system should answer. The evaluated pipeline must also handle conflicting lexical and dense rank-1 results, obsolete versus current documents, answerable queries, queries that require clarification, queries where abstention is appropriate, and the precision-coverage trade-off introduced by deferral.

• Lexical BM25 baseline.
• Dense MiniLM baseline.
• Equal-weight hybrid reciprocal-rank fusion.
• Status-aware current-first reranking.
• Retrieval-consensus analysis.
• Selective-risk analysis.
• Signal-separability analysis.
• Versioned v2 synthetic benchmark.
• Frozen development/holdout split.
• Top-1 gate pre-registration.
• One-time holdout evaluation.
• Deterministic artifact registry and SHA-256 evidence package.

• Synthetic benchmark only; no production traffic or real enterprise corpus.
• 36-query holdout with fixed query composition.
• Gate rules and success criteria were frozen before holdout evaluation.
• Development and holdout were separated until the one-time evaluation.
• Holdout status: unblinded. The holdout cannot be reused as unseen evidence.
• Post-holdout gate tuning on this benchmark is prohibited.
• Integrity regeneration is a reproducibility check only.
• Results cannot support population-level or production deployment claims.
• Source code, tests and full study artifacts are not publicly distributed.

1. 01

   Lexical retrieval

   BM25 provides exact-term and token-overlap evidence.

2. 02

   Dense retrieval

   Pinned MiniLM embeddings provide semantic retrieval evidence.

3. 03

   Hybrid fusion

   Reciprocal-rank fusion combines lexical and dense rankings with equal weights.

4. 04

   Status-aware reranking

   Current documents are prioritised over obsolete evidence.

5. 05

   Top-1 consensus gate

   Answer only when BM25, dense and final current evidence share the same rank-1 document; otherwise defer.

1. 01

   Establish baselines

   Define lexical BM25 and dense MiniLM retrieval baselines on the synthetic corpora.

2. 02

   Evaluate hybrid retrieval

   Combine lexical and dense rankings with equal-weight reciprocal-rank fusion and apply status-aware current-first reranking.

3. 03

   Analyse gate failures

   Examine consensus-gate failures and signal separability to understand when retrieval evidence supports an answer decision.

4. 04

   Create frozen benchmark

   Build the versioned v2 synthetic benchmark with a fixed development/holdout split.

5. 05

   Pre-register gate and criteria

   Freeze the top-1 consensus gate rules and three success criteria before any holdout access.

6. 06

   Run holdout evaluation

   Execute the one-time unblinded holdout evaluation without changing the gate or thresholds.

7. 07

   Consolidate evidence package

   Assemble deterministic JSON reports, a 33-artifact integrity registry and a versioned study package.

• Versioned synthetic corpora.
• Typed retrieval and evaluation implementation.
• Deterministic JSON reports.
• Pre-registration artifact.
• One-time holdout report.
• Final study report.
• 33-artifact integrity registry.
• Automated integrity and reproducibility checks.
• Versioned study package with SHA-256 verification.

• Python
• uv
• pytest
• BM25
• sentence-transformers
• all-MiniLM-L6-v2
• reciprocal-rank fusion
• SHA-256 artifact verification

• Pre-registered synthetic holdout result

  Frozen success criteria

  PASS 3/3

  All three criteria were frozen before holdout evaluation and passed without changing the gate or thresholds.

• Synthetic holdout metric

  Answered-query precision

  17/20 = 0.850

  Required value was at least 0.75.

• Synthetic holdout metric

  Answerable-query coverage

  17/22 = 0.773

  Required value was at least 0.70.

• Synthetic holdout metric

  False-answer rate

  3/14 = 0.214

  Required value was at most 0.50 for queries expected to defer.

• Descriptive synthetic holdout metric

  Decision accuracy

  28/36 = 0.778

  Accuracy was reported alongside precision, coverage and deferral errors; it was not a frozen success criterion.

• Reproducibility evidence

  Evidence package

  33 registered artifacts across 12 stages

  The study is based on a reproducible internal benchmark over a synthetic support corpus. Public results are summarised here; source code and full artifacts remain private.

• Single synthetic benchmark with a small fixed 36-query holdout.
• Fixed answer, clarify and abstain query balance.
• No evidence about real support workloads or enterprise corpora.
• No calibrated probability or confidence score is produced.
• No generative-answer evaluation was performed.
• Three false answers occurred on abstain queries.
• Five answerable queries were deferred.
• No generalisation evidence for other corpora, domains or languages.
• The holdout is unblinded and cannot be reused as unseen evidence.
• The study does not claim production validation or guaranteed answer correctness.

• Independently designed real-corpus holdout with representative traffic and prevalence estimates.
• Document lifecycle and status governance.
• Retrieval and deferral monitoring with latency and cost measurement.
• Access control and sensitive-data handling.
• Human escalation paths for deferred or low-confidence queries.
• Answer-generation evaluation beyond retrieval consensus.
• Incident review and rollback procedures.
• New confirmatory data for any modified gate before deployment.

• RAG Reliability Audit of existing systems.
• Answerability and abstention policy design.
• False-answer and evidence-conflict analysis.
• Synthetic and representative evaluation design.
• Status-aware knowledge retrieval.
• Evaluation protocol design.
• Technical evidence and decision reporting.