Dataset card
The task suite: how it was made, what is in it, and what it is not.
Summary
A generated suite of agent tasks across three synthetic domains, each carrying an executable oracle program. Ground truth is computed by running that program in a sandbox, never written by hand and never judged by a model.
| Tier | Tasks | What it measures |
|---|---|---|
| T1 | 1,899 | Single lookup. The floor: every model should pass these. |
| T2 | 2,400 | Multi-hop, two to four calls. The bulk of the suite. |
| T3 | 1,440 | Conditional and policy reasoning. Where cheap models start failing. |
| T4 | 1,280 | Traps. The correct answer is often ‘I cannot’, ‘no’, or a question. |
| T5 | 737 | Retrieval-grounded. Citation and faithfulness. |
How it was made
- Three worlds are built from a seeded RNG over literal word lists committed to this repository. No
faker, no network, no wall clock. Build twice and the SHA-256 over every row is identical. - Task templates are written in a shared thirteen-verb grammar and phrase their questions with each world’s own lexicon, so one template produces a sensible retail question, a sensible clinical question and a sensible documentation question.
- Each draft’s gold program is executed in a fresh sandbox. The task is kept only if the execution agrees with what the template claimed.
- Splits are assigned by a hash of the task’s content, so a task keeps its split when the suite is regenerated at a different size. Adding tasks does not silently move existing ones from train into test.
- The hidden split’s hash is sealed in a git tag before any optimisation run.
Composition
- Three domains: retail operations (primary), outpatient clinic (transfer, never trained on), documentation estate (grounding).
- Five tiers: single lookup, multi-hop, conditional policy reasoning, traps, retrieval-grounded.
- 1,280 traps across four kinds: unanswerable, policy-violating, ambiguous, and indirect prompt injection.
- 375 tasks carrying a planted injection inside a tool result.
- Roughly 870 tasks legitimately mutate world state; the rest are read-only, and a read-only task that mutates anything has failed even if its answer is right.
Provenance and licence
Every row carries source_type: oracle_program and generator_model: null. Nothing in this dataset was generated by a language model, which means no provider’s terms of service constrain its use. It is MIT licensed with the rest of the repository.
Privacy
100% synthetic by construction. Names are drawn from a committed, deliberately international list and combined freely, so the population encodes no assumption about who works where. Email addresses use reserved example domains. No real personal data exists here and none could.
Known limitations and biases
- Template-generated. Prompts are drawn from a fixed set of phrasings per template. Real user language is messier, and a system tuned on this suite would be tuned on that regularity.
- English only.
- Bounded task space. Some templates have small unique task spaces (a policy-fact question over four tiers and three fields has twelve distinct instances), so the generator retires them once exhausted. The suite is 7,756 tasks against a target of 8,000 for exactly this reason, and the shortfall is reported rather than padded with duplicates.
- Decontamination is three-axis and could be fooled. A pair that differs in its gold program but is semantically identical would not be caught. The three axes were chosen after two weaker definitions produced false positives on a generated suite; a fourth axis may be needed for a suite built differently.
- Grounding facts are structured. docworld’s prose is rendered from a structured fact, which is what makes citation checkable and also what makes its documents more regular than real ones.
Regenerating it
uv run toolsmith world build --all
uv run toolsmith tasks build --total 8000 --seed 20260726Deterministic: the same command produces the same suite, and toolsmith ci hidden-split fails if it does not.