flowchart LR S([World seed]) --> W(World) W --> T(Task) T --> O(Oracle program) O --> GT(Ground truth) W --> TR(Trajectory) T --> TR TR --> GV(Gate verdicts) GT --> RUN(Run) GV --> RUN RUN --> M([Published metric])
Governance
Where every row came from, and what the build refuses to let through.
The part that reads senior is not the model choice. It is being able to say where a number came from, and having a check that fails when the answer stops being true.
Five gates that fail the build
Each of these is a claim this project makes, turned into an exit code. A claim that is not a gate is a claim that decays.
| Gate | What it asserts | Command |
|---|---|---|
| Licence firewall | No model whose terms forbid it appears as the generator of a training row | toolsmith ci firewall |
| Model agnosticism | No model identifier is hardcoded in src/ |
toolsmith ci model-agnostic |
| Decontamination | No test task duplicates a training task on all three axes | toolsmith ci decontam |
| Hidden split | The sealed split is byte-identical to the committed hash | toolsmith ci hidden-split |
| Budget | Cumulative live spend is under the cap | toolsmith ci budget |
The licence firewall
Training data may originate only from oracle programs written in this repository, self-hosted permissively-licensed weights, or providers whose terms permit it. Anthropic, OpenAI and Google are inference-only: they appear in the evaluation matrix and as escalation targets, and never in a file a trainer reads.
That is not a promise in a README. It is a pydantic field on every dataset row, a training_data_use column in the model registry, and a CI job that scans data/train/** and fails on violation. Evaluation directories are explicitly exempt, because evaluation is inference and frontier models belong there.
The firewall currently reports zero files scanned. That is the expected state: it is armed and there is nothing to scan, because nothing was trained.
Model agnosticism, as a check
The headline architectural property is that every model is configuration and never code. That is easy to say and easy to break: one convenience default and the claim is quietly false while the README still makes it.
So it is a gate. The check parses every Python file under src/ with ast, ignores docstrings and comments (documentation may name models; code may not), and fails if a model identifier appears in executable code. Twenty-eight identifiers, because models.yaml gives every model two names: the key this repository uses, groq-oss-120b, and the string the vendor answers to, openai/gpt-oss-120b. The gate checked only the second for a while, and a judge panel written with the first sat in harness/judges.py scoring zero offences. Both are portability leaks. Code that spells either one has picked a model.
Provenance on every row
Every dataset row, trajectory and judgment carries a provenance record: source_type, generator_model, generator_version, prompt_hash, license, created_at, seed, and the content hashes of its parents. The parents build an append-only lineage DAG:
Reading it backwards from a cell in the results table tells you which task produced it, which oracle program defined its ground truth, which seed built the database that oracle ran against, and which configuration was in force. Append-only matters: a lineage file you can rewrite proves nothing.
One honest caveat. The log, the traversal and the append-only guarantee are implemented and tested; the pipeline does not yet write to them. Every fact the graph would carry is recoverable today from the provenance record stamped on each row, which is written on every run. The graph would turn that walk from a join into a lookup. It is listed here as built rather than as running, because the distinction is the sort of thing this page exists to keep straight.
The budget, enforced rather than intended
CostLedger.check_affordable runs before every live request and raises if the call would push cumulative spend past the cap in configs/budget.yaml. Not after the invoice arrives. Simulated calls are priced from the same catalogue and totalled into the run manifest, so the report can say what a run would have cost live, but they never consume the real budget because they never leave the process.
Current state: $0.00 spent against a $20 cap. Buying the published matrix from the vendors, at the catalogue prices in configs/models.yaml, would have cost $121.68. That is a useful number in its own right: it is the price of this evidence, and it is why the simulator is a provider rather than a test fixture.
Compliance and data
- 100% synthetic by construction. Every value comes from a seeded RNG over literal word lists committed to this repository. No real personal data exists here and none could.
- The PII detector runs over generated rows as a proof rather than an assumption, and its result is published.
- Card-number matching is Luhn-validated, so the detector does not fire on every order total. A guardrail that cries wolf is a guardrail that gets switched off, and a guardrail that is switched off is not a guardrail.
Continuous integration
| Job | What it does |
|---|---|
quality |
ruff format, ruff lint, mypy |
test |
356 tests on two operating systems and two Python versions |
gates |
the five governance gates above |
reproduce |
the whole pipeline end to end, then asserts the committed artifacts are byte-identical to a fresh run |
drift |
weekly, checks every declared model against the provider’s live catalogue and opens an issue on drift |
The reproduce job is the one that makes the rest mean anything. Two reproducibility bugs were found by running the matrix twice and diffing: measured wall-clock leaking into the results file, and gzip stamping both a timestamp and the output filename into its header. Both are fixed, and the job would catch a third.
A gate that could not fail
For a while the gates job was green on every push and checking almost nothing. The task suite is derived from a seed rather than committed, so a checkout does not have it, and two of the gates treated a missing dataset as an early return that printed PASS. Decontamination and the hidden-split seal, the two claims the whole argument rests on, were being reported as verified on evidence that was not there.
The repair has two halves, and only having both makes it a check again. CI now generates the dataset before the gates run, so there is something to verify. And a gate that cannot verify no longer says PASS: it says SKIP when a project has genuinely generated nothing, and it fails outright when a seal is committed but the data it seals is missing, which is the case that was quietly passing. Two tests pin the behaviour, and both of them fail against the old implementation.