Results
The full matrix, with intervals, breakdowns, and the comparisons that survive correction.
Every configuration ran the same stratified sample of tasks. That is what makes the comparisons paired, and paired is what makes the intervals honest.
The headline
The column that matters is dollars per success, not per task. They rank configurations differently, and only one of them is honest: a model that costs a third as much and fails twice as often is not cheaper, because you paid for the failure and then paid again.
| Configuration | pass@1 | 95% CI | pass^k | $ / task | $ / success | vs frontier | Pareto |
|---|---|---|---|---|---|---|---|
| Local only (Qwen3.5-4B, 4-bit) | 0.439 | [0.367, 0.511] | 0.122 | $0.00000 | $0.00000 | - | |
| Oracle (ceiling) | 1.000 | [1.000, 1.000] | 1.000 | $0.00000 | $0.00000 | - | |
| Coin flip (floor) | 0.033 | [0.011, 0.061] | 0.011 | $0.00000 | $0.00000 | - | |
| All cheap (gpt-oss-20b) | 0.622 | [0.550, 0.694] | 0.267 | $0.00070 | $0.00113 | 0.02x | * |
| Cascade (fully open weights) | 0.828 | [0.772, 0.883] | 0.633 | $0.00566 | $0.00683 | 0.13x | * |
| Cascade (gpt-5.4-mini executor) | 0.956 | [0.922, 0.983] | 0.867 | $0.01399 | $0.01464 | 0.28x | * |
| Ablation: no escalation | 0.700 | [0.633, 0.767] | 0.406 | $0.01381 | $0.01973 | 0.38x | |
| Ablation: no compaction | 0.883 | [0.833, 0.928] | 0.744 | $0.01743 | $0.01973 | 0.38x | |
| Cascade (Qwen executor, OSS verifier) | 0.883 | [0.833, 0.928] | 0.744 | $0.01743 | $0.01973 | 0.38x | |
| Ablation: executor reviews itself | 0.833 | [0.778, 0.883] | 0.656 | $0.01765 | $0.02118 | 0.40x | |
| Single model (Sonnet 5) | 0.883 | [0.833, 0.928] | 0.722 | $0.02012 | $0.02278 | 0.43x | |
| Ablation: all tools in prompt | 0.883 | [0.833, 0.928] | 0.744 | $0.02285 | $0.02587 | 0.49x | |
| Naive role split (frontier bookends) | 0.639 | [0.567, 0.706] | 0.311 | $0.01751 | $0.02741 | 0.52x | |
| Cascade (frontier verifier) | 0.917 | [0.872, 0.956] | 0.817 | $0.02637 | $0.02877 | 0.55x | |
| All frontier (Opus 5) | 0.950 | [0.917, 0.978] | 0.850 | $0.04980 | $0.05242 | 1.00x |
Rows in bold are on the computed Pareto frontier: nothing else is both cheaper per success and at least as reliable. Controls and unbilled rows are excluded from the frontier, because a local model has no marginal token cost and would dominate the dollars axis by construction at 44% pass@1.
Quality, ranked
By tier
The tiers are designed to separate configurations that look identical on easy work. T1 is a single lookup and everything is near ceiling; T3 is conditional policy reasoning; T4 is traps, where the correct answer is often a refusal, an abstention or a question.
Show the numbers
| Configuration | T1 | T2 | T3 | T4 | T5 |
|---|---|---|---|---|---|
| Oracle (ceiling) | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| Cascade (gpt-5.4-mini executor) | 0.974 | 0.947 | 0.972 | 0.968 | 0.882 |
| All frontier (Opus 5) | 0.974 | 0.965 | 0.917 | 0.968 | 0.882 |
| Cascade (frontier verifier) | 0.974 | 0.877 | 0.833 | 1.000 | 0.941 |
| Ablation: no compaction | 0.949 | 0.877 | 0.806 | 0.903 | 0.882 |
| Ablation: all tools in prompt | 0.949 | 0.877 | 0.806 | 0.903 | 0.882 |
| Cascade (Qwen executor, OSS verifier) | 0.949 | 0.877 | 0.806 | 0.903 | 0.882 |
| Single model (Sonnet 5) | 0.872 | 0.895 | 0.917 | 0.839 | 0.882 |
| Ablation: executor reviews itself | 0.897 | 0.825 | 0.750 | 0.871 | 0.824 |
| Cascade (fully open weights) | 0.949 | 0.754 | 0.778 | 0.935 | 0.706 |
| Ablation: no escalation | 0.795 | 0.719 | 0.583 | 0.742 | 0.588 |
| Naive role split (frontier bookends) | 0.846 | 0.509 | 0.583 | 0.742 | 0.529 |
| All cheap (gpt-oss-20b) | 0.692 | 0.526 | 0.667 | 0.839 | 0.294 |
| Local only (Qwen3.5-4B, 4-bit) | 0.513 | 0.439 | 0.361 | 0.516 | 0.294 |
| Coin flip (floor) | 0.000 | 0.000 | 0.000 | 0.194 | 0.000 |
| Configuration | T1 | T2 | T3 | T4 | T5 |
|---|---|---|---|---|---|
| Oracle (ceiling) | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| Cascade (gpt-5.4-mini executor) | 0.974 | 0.947 | 0.972 | 0.968 | 0.882 |
| All frontier (Opus 5) | 0.974 | 0.965 | 0.917 | 0.968 | 0.882 |
| Cascade (frontier verifier) | 0.974 | 0.877 | 0.833 | 1.000 | 0.941 |
| Ablation: no compaction | 0.949 | 0.877 | 0.806 | 0.903 | 0.882 |
| Ablation: all tools in prompt | 0.949 | 0.877 | 0.806 | 0.903 | 0.882 |
| Cascade (Qwen executor, OSS verifier) | 0.949 | 0.877 | 0.806 | 0.903 | 0.882 |
| Single model (Sonnet 5) | 0.872 | 0.895 | 0.917 | 0.839 | 0.882 |
| Ablation: executor reviews itself | 0.897 | 0.825 | 0.750 | 0.871 | 0.824 |
| Cascade (fully open weights) | 0.949 | 0.754 | 0.778 | 0.935 | 0.706 |
| Ablation: no escalation | 0.795 | 0.719 | 0.583 | 0.742 | 0.588 |
| Naive role split (frontier bookends) | 0.846 | 0.509 | 0.583 | 0.742 | 0.529 |
| All cheap (gpt-oss-20b) | 0.692 | 0.526 | 0.667 | 0.839 | 0.294 |
| Local only (Qwen3.5-4B, 4-bit) | 0.513 | 0.439 | 0.361 | 0.516 | 0.294 |
| Coin flip (floor) | 0.000 | 0.000 | 0.000 | 0.194 | 0.000 |
By world, and the transfer number
ops is the primary domain: tasks are generated there, thresholds are tuned there. clinic is never trained or tuned on, and its schema is deliberately not a relabelling: coverage is keyed by two dimensions rather than one, so a system that pattern-matched “read the tier row” reads the wrong row rather than no row.
The last column is the honest generalisation number.
| Configuration | clinic | doc | ops | ops to clinic |
|---|---|---|---|---|
| Oracle (ceiling) | 1.000 | 1.000 | 1.000 | +0.0 pp |
| Cascade (gpt-5.4-mini executor) | 0.961 | 0.946 | 0.960 | +0.1 pp |
| All frontier (Opus 5) | 0.941 | 0.946 | 0.960 | -1.8 pp |
| Cascade (frontier verifier) | 0.882 | 0.927 | 0.932 | -5.0 pp |
| Ablation: no compaction | 0.843 | 0.909 | 0.892 | -4.9 pp |
| Ablation: all tools in prompt | 0.843 | 0.909 | 0.892 | -4.9 pp |
| Cascade (Qwen executor, OSS verifier) | 0.843 | 0.909 | 0.892 | -4.9 pp |
| Single model (Sonnet 5) | 0.882 | 0.891 | 0.878 | +0.4 pp |
| Ablation: executor reviews itself | 0.784 | 0.891 | 0.824 | -4.0 pp |
| Cascade (fully open weights) | 0.784 | 0.818 | 0.865 | -8.1 pp |
| Ablation: no escalation | 0.686 | 0.746 | 0.676 | +1.1 pp |
| Naive role split (frontier bookends) | 0.667 | 0.545 | 0.689 | -2.3 pp |
| All cheap (gpt-oss-20b) | 0.745 | 0.527 | 0.608 | +13.7 pp |
| Local only (Qwen3.5-4B, 4-bit) | 0.412 | 0.400 | 0.486 | -7.5 pp |
| Coin flip (floor) | 0.039 | 0.000 | 0.054 | -1.5 pp |
Safety and grounding
Abstention alone is not a virtue. A system that abstains on everything scores perfectly on abstain recall and is useless, which is why over-refusal sits in the next column. An unsanctioned privileged action is the most serious thing this harness can record, and it is a separate column for that reason.
| Configuration | Abstain recall | Over-refusal | Injection resisted | Unsanctioned actions | Citation recall |
|---|---|---|---|---|---|
| Oracle (ceiling) | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| Cascade (gpt-5.4-mini executor) | 0.968 | 0.000 | 0.600 | 0.000 | 1.000 |
| All frontier (Opus 5) | 1.000 | 0.000 | 1.000 | 0.000 | 1.000 |
| Cascade (frontier verifier) | 1.000 | 0.000 | 0.400 | 0.000 | 1.000 |
| Ablation: no compaction | 0.903 | 0.000 | 0.400 | 0.000 | 0.941 |
| Ablation: all tools in prompt | 0.903 | 0.000 | 0.400 | 0.000 | 0.941 |
| Cascade (Qwen executor, OSS verifier) | 0.903 | 0.000 | 0.400 | 0.000 | 0.941 |
| Single model (Sonnet 5) | 0.839 | 0.000 | 0.800 | 0.000 | 1.000 |
| Ablation: executor reviews itself | 0.903 | 0.000 | 0.400 | 0.000 | 1.000 |
| Cascade (fully open weights) | 0.968 | 0.000 | 0.300 | 0.000 | 0.941 |
| Ablation: no escalation | 0.806 | 0.000 | 0.400 | 0.000 | 0.824 |
| Naive role split (frontier bookends) | 0.806 | 0.000 | 0.300 | 0.000 | 0.941 |
| All cheap (gpt-oss-20b) | 0.871 | 0.000 | 0.500 | 0.000 | 0.882 |
| Local only (Qwen3.5-4B, 4-bit) | 0.613 | 0.000 | 0.200 | 0.000 | 0.647 |
| Coin flip (floor) | 0.387 | 0.000 | 0.100 | 0.000 | 0.059 |
Injection resisted is the share of injected tasks where the agent did not reach for the planted action. It used to mean something else: the world came out unchanged. That version read 1.000 for every configuration in this table, including the coin flip, which is not a fact about the agents. The policy function refuses the planted call server-side before anything can be written, so the world is always unchanged and the column was measuring the sandbox. An agent that reaches for the injected action and is refused has not resisted it, it was stopped. Counting attempts instead, the column runs from 0.100 on the coin flip to 1.000 on the frontier row, which is the ordering you would expect and the one it previously had no way to express.
The gate’s own contribution is a single number rather than a column, because it is the same for every row: across all 450 injected runs in this matrix, the policy function refused every attempted privileged call and the world changed zero times. That is the point of putting it server-side. It is also why publishing it per configuration was misleading, and why an attempt-rate column beside this one would be pure redundancy: it is exactly one minus this, on all fifteen rows.
Which differences are real
Fifteen configurations would be 105 pairwise comparisons. The family is 91, because one row is not a configuration: ablation_no_compaction turns off a mechanism that never engages on this workload, where trajectories peak around twelve thousand tokens and the compaction threshold sits far above that. It reproduces its baseline task for task and dollar for dollar. Left in the family it would have contributed fourteen pairs, each of them the same configuration compared against itself under two names, and every one of those free non-significant results makes the survivors look more selective than they are by enlarging the number Holm divides by.
At alpha 0.05 across 91 comparisons you would expect four or five spurious wins from noise alone, so the whole family is Holm-Bonferroni corrected. McNemar’s exact test is used rather than a chi-square approximation, because only the discordant pairs carry information and there are often few of them.
The twenty largest differences that survive correction:
| A | B | A - B | 95% CI | Discordant | McNemar p |
|---|---|---|---|---|---|
| oracle_ceiling | random_floor | +0.967 | [+0.939, +0.989] | 174 | 0.00e+00 |
| cascade_mini_executor | random_floor | +0.922 | [+0.883, +0.961] | 166 | 0.00e+00 |
| frontier_all_opus | random_floor | +0.917 | [+0.878, +0.956] | 165 | 0.00e+00 |
| cascade_frontier_verify | random_floor | +0.883 | [+0.833, +0.928] | 159 | 0.00e+00 |
| ablation_no_tool_search | random_floor | +0.850 | [+0.794, +0.900] | 155 | 0.00e+00 |
| cascade_default | random_floor | +0.850 | [+0.794, +0.900] | 155 | 0.00e+00 |
| random_floor | single_sonnet | -0.850 | [-0.900, -0.794] | 153 | 0.00e+00 |
| ablation_self_review | random_floor | +0.800 | [+0.733, +0.861] | 148 | 0.00e+00 |
| cascade_all_open | random_floor | +0.794 | [+0.733, +0.856] | 145 | 0.00e+00 |
| ablation_no_escalation | random_floor | +0.667 | [+0.594, +0.739] | 124 | 0.00e+00 |
| naive_role_split | random_floor | +0.606 | [+0.528, +0.678] | 113 | 0.00e+00 |
| budget_floor_oss20b | random_floor | +0.589 | [+0.517, +0.661] | 108 | 0.00e+00 |
| local_only | oracle_ceiling | -0.561 | [-0.633, -0.489] | 101 | 0.00e+00 |
| cascade_mini_executor | local_only | +0.517 | [+0.439, +0.594] | 97 | 0.00e+00 |
| frontier_all_opus | local_only | +0.511 | [+0.433, +0.589] | 100 | 0.00e+00 |
| cascade_frontier_verify | local_only | +0.478 | [+0.394, +0.561] | 102 | 0.00e+00 |
| ablation_no_tool_search | local_only | +0.444 | [+0.361, +0.528] | 98 | 0.00e+00 |
| cascade_default | local_only | +0.444 | [+0.361, +0.528] | 98 | 0.00e+00 |
| local_only | single_sonnet | -0.444 | [-0.528, -0.361] | 96 | 0.00e+00 |
| local_only | random_floor | +0.406 | [+0.328, +0.483] | 79 | 0.00e+00 |
The models under test
| Key | Provider | Tier | $ in/M | $ out/M | Cache | Training use | Verified |
|---|---|---|---|---|---|---|---|
claude-haiku-4-5 |
anthropic | mid | 1.000 | 5.000 | 90% | forbidden | 2026-07-26 |
claude-opus-5 |
anthropic | frontier | 5.000 | 25.000 | 90% | forbidden | 2026-07-26 |
claude-sonnet-5 |
anthropic | frontier | 2.000 | 10.000 | 90% | forbidden | 2026-07-26 |
coinflip |
simulated | local | 0.000 | 0.000 | 0% | allowed | 2026-07-26 |
gemini-20-flash |
cheap | 0.075 | 0.300 | 90% | forbidden | 2026-07-26 | |
gemini-31-pro |
frontier | 2.000 | 12.000 | 90% | forbidden | 2026-07-26 | |
gpt-54-mini |
openai | mid | 0.600 | 2.400 | 90% | forbidden | unverified |
gpt-54-nano |
openai | cheap | 0.200 | 1.250 | 90% | forbidden | 2026-07-26 |
gpt-56-sol |
openai | frontier | 2.500 | 15.000 | 90% | forbidden | 2026-07-26 |
groq-oss-120b |
groq | mid | 0.150 | 0.600 | 50% | allowed | 2026-07-26 |
groq-oss-20b |
groq | cheap | 0.075 | 0.300 | 50% | allowed | 2026-07-26 |
groq-prompt-guard-86m |
groq | guard | 0.000 | 0.000 | 0% | allowed | 2026-07-26 |
groq-qwen36-27b |
groq | mid | 0.600 | 3.000 | 0% | allowed | 2026-07-26 |
groq-safeguard-20b |
groq | guard | 0.075 | 0.300 | 50% | allowed | 2026-07-26 |
local-mlx-qwen-4b |
mlx | local | 0.000 | 0.000 | 0% | allowed | 2026-07-26 |
mistral-medium |
mistral | mid | 0.400 | 2.000 | 0% | allowed | unverified |
oracle |
simulated | local | 0.000 | 0.000 | 0% | allowed | 2026-07-26 |
Two rows carry **unverified** in the last column. Their prices are estimates and the report flags them everywhere they appear. toolsmith probe models checks every id against the provider’s live catalogue and fails the build on drift; a weekly CI job runs it and opens an issue.
Reproducing this page
uv run toolsmith world build --all
uv run toolsmith tasks build
uv run toolsmith matrix run --provider simulated --n 180 --trials 3 --seed 20260726
uv run toolsmith report buildCI runs exactly that on every push and fails if the committed artifacts differ by a byte.