Which model should run which part of your agent?
A control plane that answers it with receipts, for under twenty dollars.
You cannot buy a frontier model for every step of an agent. You also cannot trust a cheap one without proof. This is the harness that decides, per role and per task, which model to use, and then shows its working in dollars, milliseconds, and failure transcripts.
The 30-second version
Pick an agent task. It needs a planner, an executor that loops, and a reviewer. The obvious move is to put a frontier model on the ends and something cheap in the middle, because the ends are short calls and the middle is where the tokens go.
That obvious move is the worst of the fifteen configurations measured here, and the reason is arithmetic rather than taste. The executor is the only role that runs N times, so its per-step reliability is raised to the Nth power. Put your weakest model in the exponent and you have put it in the worst place available.
The intuitive split
0.639
pass@1, at $0.02741 per success
A cascade instead
0.956
pass@1, at $0.01464 per success
All frontier
0.950
pass@1, at $0.05242 per success
Real money spent
$0.00
of a $20 cap, enforced in code
The cascade matches the all-frontier row’s quality at 0.28x its cost per success, and beats the intuitive split on both axes at once. That is the finding. The rest of this site is how it was measured and where it fails.
What is actually here
- 8,100 graded runs across 15 configurations on one stratified sample of 180 tasks, 3 trials each.
- The suite is 7,756 verified tasks, of which 1,280 are traps and 375 carry an indirect prompt injection.
- Best value on the frontier: Cascade (gpt-5.4-mini executor) at 0.956 pass@1 and $0.01464 per success, which is 0.28x the all-frontier row’s cost at 0.950 pass@1.
- The intuitive role split (frontier bookends, cheap executor) reaches 0.639 pass@1 at $0.02741 per success: worse than the cascade on both axes.
- 64 of 91 pairwise comparisons survive Holm-Bonferroni correction.
- Input tokens are 82% of the token bill, averaged across configurations.
- Real money spent: $0.00 of a $20 cap, enforced in code before every call.
Provenance. Every number on this site was produced by the production code path over a deterministic behavioural simulator, seeded from published model-card priors. No API call was made and nothing was spent. The simulator is a provider, not a test fixture: the executor loop, the gates, the cascade, the ledger, the statistics and this report are the code that would run live. Adding a key and running --provider live regenerates every table here for real, through the identical path. See ADR 0001 for why that trade was made and what it costs the reader.
The cost-quality frontier
Show the numbers
| Configuration | pass@1 | 95% CI | pass^k | $ / task | $ / success | vs frontier | Pareto |
|---|---|---|---|---|---|---|---|
| Local only (Qwen3.5-4B, 4-bit) | 0.439 | [0.367, 0.511] | 0.122 | $0.00000 | $0.00000 | - | |
| Oracle (ceiling) | 1.000 | [1.000, 1.000] | 1.000 | $0.00000 | $0.00000 | - | |
| Coin flip (floor) | 0.033 | [0.011, 0.061] | 0.011 | $0.00000 | $0.00000 | - | |
| All cheap (gpt-oss-20b) | 0.622 | [0.550, 0.694] | 0.267 | $0.00070 | $0.00113 | 0.02x | * |
| Cascade (fully open weights) | 0.828 | [0.772, 0.883] | 0.633 | $0.00566 | $0.00683 | 0.13x | * |
| Cascade (gpt-5.4-mini executor) | 0.956 | [0.922, 0.983] | 0.867 | $0.01399 | $0.01464 | 0.28x | * |
| Ablation: no escalation | 0.700 | [0.633, 0.767] | 0.406 | $0.01381 | $0.01973 | 0.38x | |
| Ablation: no compaction | 0.883 | [0.833, 0.928] | 0.744 | $0.01743 | $0.01973 | 0.38x | |
| Cascade (Qwen executor, OSS verifier) | 0.883 | [0.833, 0.928] | 0.744 | $0.01743 | $0.01973 | 0.38x | |
| Ablation: executor reviews itself | 0.833 | [0.778, 0.883] | 0.656 | $0.01765 | $0.02118 | 0.40x | |
| Single model (Sonnet 5) | 0.883 | [0.833, 0.928] | 0.722 | $0.02012 | $0.02278 | 0.43x | |
| Ablation: all tools in prompt | 0.883 | [0.833, 0.928] | 0.744 | $0.02285 | $0.02587 | 0.49x | |
| Naive role split (frontier bookends) | 0.639 | [0.567, 0.706] | 0.311 | $0.01751 | $0.02741 | 0.52x | |
| Cascade (frontier verifier) | 0.917 | [0.872, 0.956] | 0.817 | $0.02637 | $0.02877 | 0.55x | |
| All frontier (Opus 5) | 0.950 | [0.917, 0.978] | 0.850 | $0.04980 | $0.05242 | 1.00x |
The three things worth taking away
Finding 1
Input tokens are 82% of the bill, and nobody optimises them
Output tokens are what people price a model on. Input is what they actually buy. The executor re-reads its whole transcript every turn, so input grows with the square of the turn count while output grows linearly.
The lever is therefore context, not generation. Retrieving tool schemas through a search_tools meta-tool instead of inlining every tool the world binds cuts input by 48% per task at no measurable cost in accuracy, which is a 23% reduction in dollars per success for an afternoon of engineering and no tokens.
Finding 2
The executor is the only role whose errors compound, so it is the wrong place to economise
A model with per-step reliability q completes an N-step trajectory cleanly at roughly q^N. The planner and the reviewer are single shots. Measured here: the all-cheap row passes 71.4% of one-step tasks and 56.5% of two-step ones. Bare q² would say 51.0%; the gap is the reviewer and the escalation catching part of what the executor drops, which is the whole reason those two roles exist. The all-frontier row gives up under four points across the same step.
This inverts the usual intuition. The planner is cheap to upgrade because it runs once, not because its input is cheap.
Finding 3
Escalation is a second independent attempt, and it is worth 18 points
Removing escalation from the cascade, changing nothing else, costs 18.3 points of pass@1. The retries add 26% to the bill, and buy back exactly enough successes to leave cost per success unchanged at $0.01973. Two attempts beat one even when the first is weaker, which is how a cascade reaches the frontier row’s success rate at a fraction of its cost.
The best cascade scores 0.956 against the frontier row’s 0.950. That is not a win and this page will not call it one: the McNemar test on the paired outcomes gives p = 1.0, so the difference is nothing. The result is that they are indistinguishable on quality and a factor of four apart on cost, which is the more useful sentence anyway.
The variable that governs it is not the executor’s accuracy but the verifier’s detection rate: escalation only fires on what the verifier noticed.
And one where the source spec was wrong
The design document this project was built from predicted the executor at 73% of spend and the planner at 9%, on a six-turn loop. At the 2.4 turns these tasks actually take, the planner’s single frontier call is 48% of cascade spend and the executor 30%.
The claim is turn-count dependent, and the spec did not say so. It is reported here rather than smoothed over, because a harness that only confirms its own brief is not a harness.
How to check any of this
git clone https://github.com/hoomanesteki/toolsmith-small-llm-agent-uplift
cd toolsmith-small-llm-agent-uplift
uv sync --all-extras
make all # worlds, tasks, matrix, report, site: about six minutes, $0No API keys, no network beyond the package index. Every table on this site is written by toolsmith report build from eval/results/results.jsonl, and CI fails if a fresh run produces different bytes.
To see it move rather than read about it:
make serve # http://127.0.0.1:7860Four screens: the comparison surface, one run watched frame by frame, an interactive chat where the model selector changes the whole pipeline, and the human-in-the-loop review queue.