Findings
Four claims, each with the number that supports it and the caveat that limits it.
Every claim below is a row in the results file. Where the evidence is weaker than the claim would like, the section says so rather than rounding up.
1. Input is the bill
Across all fifteen configurations, input tokens are 82% of the token count. The executor drives it: it re-reads the whole transcript on every turn, so its input grows quadratically in turns while its output grows linearly.
This is why context engineering is the first lever and not the last. It is also why per-token price comparisons mislead: a model that is cheaper per output token and more verbose in its tool calls can cost more per task.
What the lever is worth
Track C measured tool retrieval against the all-in-prompt baseline with everything else held identical:
- input tokens per task: -48%
- dollars per success: -23%
- pass@1: no measurable change
The cost of the lever is inside those numbers rather than excluded from them. Retrieved schemas persist for the rest of the run, so the stable prefix grows once and the prefix cache misses once. A version of this claim that reported only the token saving would be the flattering half.
2. Error compounds in the executor and nowhere else
A model with per-step reliability q finishes an N-step trajectory cleanly at roughly q^N. The planner runs once. The reviewer runs once. The executor runs N times.
Measured on tasks grouped by gold-program length, holding the configuration fixed:
| Gold program | All cheap | All frontier |
|---|---|---|
| 1 step | 0.714 | 0.982 |
| 2 steps | 0.565 | 0.944 |
The cheap executor loses 15 points going from one step to two. The frontier executor loses 4. That gap is the whole argument against putting the cheapest model in the loop, and it is also why the planner is cheap to upgrade: it runs once, so there is no exponent on it.
What this does and does not show. The compounding is a property of the system, and it reproduces exactly because the simulator is seeded from per-step priors that compound by construction. What a live run would add is whether the published priors are right. The shape of the finding does not depend on that; the absolute rates do.
3. Escalation buys quality, and whether it pays depends on your objective
Removing escalation from the cascade and changing nothing else costs 18.3 points of pass@1. The retries are not free: they add 26% to the bill per task. What makes the trade so clean is where those two numbers land. Cost per success is $0.01973 either way, to the fifth decimal. The extra spend and the extra successes cancel, and you are left choosing between 0.883 and 0.700 at the same price per thing that worked.
Whether it pays is a different question, and Track B answers it honestly by sweeping the threshold on validation and reporting on test:
- On a cost-per-success objective, the optimal threshold is 0.00. The verifier fires often enough that the retries cost more than the successes they buy.
- On a quality objective, always escalating reaches 96.7% pass@1, 2.5 points above the sweep’s low end, at 57% more per success.
The low end of that sweep is not the ablation row above it. At tau=0.00 the cascade still escalates whenever the verifier explicitly rejects a trajectory, which is most of the escalations it ever performs; the threshold only starts binding once it is high enough to override an accepted one. The ablation removes escalation altogether. That is why one says 0.700 and the other 0.908, and the difference between them is a measure of how much work the explicit reject does on its own.
Both ends are published. Which is right is a product decision, and the point of the track is that it is now priced rather than assumed.
The sweep is flat below about tau=0.65 because the combined confidence score rarely lands that low: the verifier’s explicit reject is doing the work, and the threshold only binds once it is high enough to override an accepted trajectory. That flatness is a finding about the signal, not a gap in the grid.
4. Cross-model review is not free, and neither is self-review
The matrix includes a configuration whose reviewer shares the executor’s family, identical in every other respect. Self-review costs 5 points of pass@1, and the mechanism shows up in a number that looks like good news: the escalation rate falls, from 23.9% to 18.9%. A reviewer from the same family agrees with the executor more often, so fewer runs get a second attempt, so more bad trajectories ship. A guardrail that fires less is not a guardrail working better. The literature puts the effect at roughly a quarter of a model’s error-catching ability (GPT-4 on GSM8K falling 95.5 to 91.5 to 89.0 across self-correction rounds, arXiv 2310.01798); this harness applies that penalty and then measures what it does to the system.
The practical consequence is a configuration rule rather than a research finding: never let the reviewer share the executor’s family, and if you must, say so on the row. Five configurations here do it deliberately and carry a self-review tag for exactly that reason.
The four optimisation tracks
| Track | Lever | Verdict | What it found |
|---|---|---|---|
| Context engineering | tool retrieval and transcript compaction | gain | Retrieving tool schemas instead of inlining them cuts input tokens by 48% per task and costs 0.0% of pass@1, for a 23% reduction in dollars per success. |
| Routing and escalation | tau, the confidence below which a trajectory is retried on a stronger model | null | On a cost-per-success objective the optimal threshold is tau=0.00: escalation does not pay for itself, because the verifier fires often enough that the retries cost more than the successes they buy. It does buy quality if you want it: tau=1.00 reaches 96.7% pass@1, 2.5% above never escalating, for 57% more per success. Which of those is right is a product decision, and the point of this track is that it is now priced. |
| Prompt compilation | the executor system prompt | unmeasurable | Prompt compilation cannot be scored under the simulator: it samples behaviour from published model priors and never reads the prompt. What is measurable is the token cost, which ranges 81 to 237 tokens per turn. The dearest prompt costs 2.9x the cheapest, on every turn of every task. |
| Fine-tuning (LoRA) | adapter weights on a strong-mid open executor | null | Not run, and the reasoning is the result. Routing and context engineering already deliver 88 to 96% of frontier pass@1 at 0.13 to 0.38 times the cost per success without training anything, the residual gap is in abstention judgement rather than tool-calling syntax, and the recurring cost of SERVING an adapter exceeds the entire project budget while the training run itself costs a few dollars. |
Two of these are nulls and one is unmeasurable. That is the honest distribution for four levers tried in one project, and publishing it is more useful than publishing only the one that worked.
Track A is unmeasurable by construction here, and saying so is the finding. The simulator samples behaviour from capability priors and never reads the prompt, so any accuracy difference reported under it would be an artifact of the harness. What is real is the token cost: five candidates spanning 81 to 237 tokens per turn, so the dearest prompt costs 2.9x the cheapest, paid on every turn of every task. A prompt that buys two points of accuracy at three times the prefix may still lose on dollars per success, and that is what a live run would settle.
Track D was not run, and the reasoning is the result. Routing and context already deliver 88 to 96% of frontier pass@1 at 0.13 to 0.38 times the cost per success with no training at all; the residual gap is in abstention judgement rather than tool-calling syntax, which is the part fine-tuning on benign data is least likely to fix and most likely to damage; and serving an adapter costs more than the entire project budget while training one costs a few dollars. The safety gate that would apply is implemented rather than described, so running it later needs compute rather than design.