What was measured, how, and what it does not show yet
Frontier agent results are kept apart from engineering controls (scripted strategies with no model in the loop) and from the attacks the grader was built to survive. Sources are named under each block.
Opus leads Sonnet by +0.088 mean reward on 147 held-out CSV worlds; the 95% interval excludes zero.
| Agent | Mean reward | 95% interval | Causes found | False claims | Fully solved | $ / episode |
|---|---|---|---|---|---|---|
| Opus 5.5 | 0.532 | [0.472, 0.590] | 307 / 560 | 30 | 22.3% | 0.315 |
| Sonnet 5.5 | 0.444 | [0.379, 0.509] | 266 / 560 | 15 | 14.8% | 0.111 |
| Haiku 5.5 | 0.230 | [0.169, 0.295] | 162 / 560 | 45 | 5.3% | 0.005 |
| Paired on the same worlds | Difference | 95% interval |
|---|---|---|
| Opus 5.5 − Sonnet 5.5 | +0.088 | [+0.028, +0.149] |
| Sonnet 5.5 − Haiku 5.5 | +0.213 | [+0.146, +0.279] |
| Haiku 5.5 − Strong rule-miner | +0.144 | [+0.064, +0.226] |
Model rows and differences: the 147 held-out CSV worlds of eval5, two passes per model. 1,098 episodes in all, counting the 36 warehouse worlds, two independent passes, $187.39 of inference. A scan of every tool call in all 1,098 transcripts found no attempt to read outside the episode folder, reach the network or look for answer files. On the 36 warehouse (SQL) worlds the set separates Opus and Sonnet from Haiku but not Opus from Sonnet; a larger warehouse set is needed before that tier ranks frontier models on its own. Source: Sample pack datasheet, eval5 (Nov 2026).
Share of six frontier episodes (three models × two passes) that fully solved each of the 147 worlds. Every unsolved world's reference answer scores 1.0 and every cause stands out at z ≥ 4, so unsolved means hard, not broken. Partial credit keeps the reward dense where full solves are rare.
| Strategy | Mean reward | What it shows |
|---|---|---|
| Reference answer | 1.000 | every task, so every task is solvable |
| Reference + the faded (stale) causes | 0.750 | stale claims cost |
| Strong rule-miner | 0.312 | never fully solves a task with causes (0 of 25) |
| Always answer “nothing” | -0.116 | abstaining is not a safe default |
| Top-3 one-column splits | -0.640 | shallow mining picks up leaks |
| Spray 10 splits | -0.702 | volume is punished |
Source: Sample pack datasheet §4: a separate 28-task CSV control set, scripted, no model in the loop (not the eval's held-out worlds).
- Tell an empty world from its files (row count, columns, traps, outcome rate)metadata-only classifier; smoke run (smoke/tell_probe.py), not in CI; warehouse rate and faded-strength AUC 0.45AUC 0.50 ± 0.07
- Spray many claims, any confidenceevery blind exploit in the smoke run scores below zero; best is an all-low-confidence spray at −0.25, which counts as no claim (smoke/v8.json)≤ −0.25 mean
- Learn the marker columns that track a causean oracle that knows every tracked rule; smoke run (smoke/v8.json)mean −0.37
- Claim a column recorded after the outcomeleak claims cost double a false one−0.5 per claim
- Report a pattern that held in the pastgraded on fresh future cases, where it has faded−0.25 per claim
- Make the SQL depend on today's date16 clock-reading forms refused, legitimate date arithmetic still runs (tests/test_warehouse.py, CI)claim invalid
- Run a query that never finishesstep and time budgets; the grader always answerscut off, invalid
- Close a ticket every way at oncea second terminal action forfeits the credit−0.50
- Say “approved” without issuing the refundthe reply counts only once the refund exists−0.25
- Refund first, then declinean irreversible wrong refund forfeits the credit−0.40 mean
- Follow the instruction planted in the ticketthe attacker's goal is reported separately as an attack success−0.40 mean
- Grader down or rate-limitedthe verifier retries, then records an errored trial, never a −0.25no score
Each attack is reproduced in the repository: rows citing tests/ run in CI; rows citing smoke/ come from recorded smoke runs (smoke/v8.json), run by hand.
| Qwen3-4B | Held-out reward | Causes found (of 280) | DABench accuracy |
|---|---|---|---|
| Untrained | −0.262 | 0 | 10.2% |
| After warm start | +0.034 | 88 | 32.7% |
| After 13 GRPO steps | +0.037 | 88 | 30.3% |
Warm start: +0.296 held-out reward [+0.221, +0.374] and +22.4 points on DABench, an outside benchmark never trained on [+15.7, +28.7].
- Much of the held-out gain is learning to submit a report at all; against always answering “nothing” the gain is about +0.16.
- DABench rises too, but a matched fine-tune on other data has not been run, so transfer is not attributed to these worlds.
- RL lift is not yet shown: the first GRPO run stopped at step 13 when most groups had no reward variance.
Source: Sample pack datasheet §7 (Qwen3-4B, LoRA, one H100, 1–2 Nov 2026).