Protect / Release Protection
Method demonstration for release protection
Did this change make the system worse enough that we should block release?
What you are looking at
One Release Protection instrument. A synthetic change is evaluated against a baseline, then against multiple agreed release conditions (not a single magic metric) until the gate resolves.
What you can do
Pick a change under review, follow the causal order from delta to threshold to gate, inspect multiple release conditions, and read the Decision Record: RELEASE, BLOCK, or INVESTIGATE.
What it means
Regression is crossing an agreed release boundary, not merely that a metric moved. A gate without an owner and an action path is not useful.
Mental model
- CHANGE
- EVALUATE
- COMPARE TO BASELINE
- THRESHOLD
- PASS/FAIL
- RELEASE/BLOCK
Release under review
Assistant answer path (synthetic CI scenario)
A frozen synthetic eval set sits behind the gate. Fast checks run on every change; a broader behavioural suite runs on a slower cadence. Crossing a threshold is what matters, not noise in a single chart.
Causal order
- Name the change under review
- Evaluate behaviour on versioned fixtures
- Compare observed values to the agreed baseline
- Resolve each release condition against its threshold
- Aggregate condition outcomes into a gate state
- Act: RELEASE, BLOCK, or INVESTIGATE with an owner path
Release / state instrument
One main instrument. Select a change; the timeline reveals delta → threshold → gate. Multiple conditions stay first-class.
SYNTHETIC / ILLUSTRATIVE METHOD DEMO. Invented for teaching.
Causal timeline
1 · Change
Prompt refine (looks fine)
commit synth-a1c3 · system prompt wording only
Copy tweak in the system prompt. Behaviour barely moves; no release boundary crossed.
2 · Evaluate
Fast-gate suite on frozen fixtures; broader suite not required for this class of change.
3 · Compare to baseline
Each row is an agreed release boundary. A metric moving is not enough; the threshold must resolve.
4 · Threshold / release conditions
Release conditions (not one magic metric)
Grounding completeness
PASSUnsupported claim
- Baseline
- 92%
- Observed
- 91%
- Delta
- −1 pp
- Threshold
- ≥ 88%
- Cadence
- Fast gate
Harmful overconfidence
PASSConfident wrong answer
- Baseline
- 3.1%
- Observed
- 3.0%
- Delta
- −0.1 pp
- Threshold
- ≤ 5.0%
- Cadence
- Fast gate
Latency p95
PASSSlow assist
- Baseline
- 1.4 s
- Observed
- 1.5 s
- Delta
- +0.1 s
- Threshold
- ≤ 2.0 s
- Cadence
- Fast gate
Deferral discipline
PASSFailed abstain
- Baseline
- 0.8%
- Observed
- 0.9%
- Delta
- +0.1 pp
- Threshold
- ≤ 2.0%
- Cadence
- Broader eval
5 · Gate state
RELEASE
All agreed release conditions stay inside their thresholds. A small metric move is not a regression here.
Active record: RELEASE
Technical Inspect / Semantic Zoom (illustrative): Reveal fixtures, baseline, metrics, thresholds, CI job, report
All inspect fields are synthetic fixtures for teaching the method.
- Versioned fixtures / eval set
- eval-set@v0.4-synth · 120 frozen tickets
- Baseline
- baseline@main-synth-2026-07-01
- Metrics / failure classes
- grounding · overconfidence · latency p95 · deferral
- Thresholds
- team-owned release.yaml (illustrative)
- CI runner / job
- fast-gate · pr-check (synthetic runner)
- Report artifact
- report/synth-a1c3.json (illustrative artifact)
Causal timeline
1 · Change
Retrieval tweak (crosses boundary)
commit synth-b7e2 · retrieval top-k + chunk overlap
Top-k and chunking change. Grounding completeness crosses the agreed release boundary.
2 · Evaluate
Fast-gate catches the grounding fail; broader suite would deepen failure taxonomy later.
3 · Compare to baseline
Each row is an agreed release boundary. A metric moving is not enough; the threshold must resolve.
4 · Threshold / release conditions
Release conditions (not one magic metric)
Grounding completeness
FAILUnsupported claim
- Baseline
- 92%
- Observed
- 81%
- Delta
- −11 pp
- Threshold
- ≥ 88%
- Cadence
- Fast gate
Harmful overconfidence
FAILConfident wrong answer
- Baseline
- 3.1%
- Observed
- 6.4%
- Delta
- +3.3 pp
- Threshold
- ≤ 5.0%
- Cadence
- Fast gate
Latency p95
PASSSlow assist
- Baseline
- 1.4 s
- Observed
- 1.3 s
- Delta
- −0.1 s
- Threshold
- ≤ 2.0 s
- Cadence
- Fast gate
Deferral discipline
PASSFailed abstain
- Baseline
- 0.8%
- Observed
- 1.1%
- Delta
- +0.3 pp
- Threshold
- ≤ 2.0%
- Cadence
- Broader eval
5 · Gate state
BLOCK
Two release conditions failed. Faster latency does not redeem a crossed grounding boundary.
Active record: BLOCK
Technical Inspect / Semantic Zoom (illustrative): Reveal fixtures, baseline, metrics, thresholds, CI job, report
All inspect fields are synthetic fixtures for teaching the method.
- Versioned fixtures / eval set
- eval-set@v0.4-synth · 120 frozen tickets
- Baseline
- baseline@main-synth-2026-07-01
- Metrics / failure classes
- grounding · overconfidence · latency p95 · deferral
- Thresholds
- team-owned release.yaml (illustrative)
- CI runner / job
- fast-gate · pr-check (synthetic runner)
- Report artifact
- report/synth-b7e2.json (illustrative artifact)
Causal timeline
1 · Change
Model bump (needs owner)
commit synth-c9d4 · model tag bump (synthetic vendor id)
Model version bump. Quality holds; latency sits in a caution band that needs an owner before RELEASE.
2 · Evaluate
Fast-gate latency caution; broader eval still green. Gate does not auto-clear without investigation.
3 · Compare to baseline
Each row is an agreed release boundary. A metric moving is not enough; the threshold must resolve.
4 · Threshold / release conditions
Release conditions (not one magic metric)
Grounding completeness
PASSUnsupported claim
- Baseline
- 92%
- Observed
- 93%
- Delta
- +1 pp
- Threshold
- ≥ 88%
- Cadence
- Fast gate
Harmful overconfidence
PASSConfident wrong answer
- Baseline
- 3.1%
- Observed
- 2.9%
- Delta
- −0.2 pp
- Threshold
- ≤ 5.0%
- Cadence
- Fast gate
Latency p95
CAUTIONSlow assist
- Baseline
- 1.4 s
- Observed
- 1.95 s
- Delta
- +0.55 s
- Threshold
- ≤ 2.0 s (caution ≥ 1.8 s)
- Cadence
- Fast gate
Deferral discipline
PASSFailed abstain
- Baseline
- 0.8%
- Observed
- 0.7%
- Delta
- −0.1 pp
- Threshold
- ≤ 2.0%
- Cadence
- Broader eval
5 · Gate state
INVESTIGATE
No hard FAIL, but latency entered the caution band. INVESTIGATE: an owner must decide before RELEASE.
Active record: INVESTIGATE
Technical Inspect / Semantic Zoom (illustrative): Reveal fixtures, baseline, metrics, thresholds, CI job, report
All inspect fields are synthetic fixtures for teaching the method.
- Versioned fixtures / eval set
- eval-set@v0.4-synth · 120 frozen tickets
- Baseline
- baseline@main-synth-2026-07-01
- Metrics / failure classes
- grounding · overconfidence · latency p95 · deferral
- Thresholds
- team-owned release.yaml (illustrative)
- CI runner / job
- fast-gate · nightly broader-eval (synthetic)
- Report artifact
- report/synth-c9d4.json (illustrative artifact)
Decision Record
How this work resolves: RELEASE, BLOCK, or INVESTIGATE.
RELEASE
No agreed release boundary was crossed. Shipping remains defensible for this change.
BLOCK
At least one release condition failed. Do not ship until the boundary is restored or renegotiated.
INVESTIGATE
Something needs an owner before a clean RELEASE/BLOCK: caution without automatic clearance.
Natural next step
If release risk is live, discuss the real change, baselines, and who owns the gate. Published regression-gate material remains method evidence only.