Synthetic study
A comparison that recommended nobody.
Which setup clears a quality bar fixed in advance?
Three setups ran the same synthetic ticket workflow. The quality bar was fixed before the result was read. Exact match was 11/30, 10/30, and 10/30. None was eligible. A slightly higher score does not make a setup acceptable when it missed the bar. A lower cost did not change that either. The holdout for that run is open, so it cannot be reused as unseen. This page has no interactive comparator.
The lesson is the bar, not a winner. A relative score is not a reason to accept a setup that missed the criteria.
Bar fixed in advance: 24 of 30
| Configuration A | Configuration B | Configuration C |
|---|---|---|
| 11/30 | 10/30 | 10/30 |
The same metric and the same 30 cases for all three configurations. The names A, B, and C keep the order of the recorded result. They are not model names. The holdout for that run is open.
Configuration A has the higher relative result, 11 of 30, and it is still short of 24. None was eligible. This is not a model recommendation.