Evidence

What was measured, how, and what it does not show yet

Frontier agent results are kept apart from engineering controls (scripted strategies with no model in the loop) and from the attacks the grader was built to survive. Sources are named under each block.

Opus leads Sonnet by +0.088 mean reward on 147 held-out CSV worlds; the 95% interval excludes zero.

AgentMean reward95% intervalCauses foundFalse claimsFully solved$ / episode
Opus 5.50.532[0.472, 0.590]307 / 5603022.3%0.315
Sonnet 5.50.444[0.379, 0.509]266 / 5601514.8%0.111
Haiku 5.50.230[0.169, 0.295]162 / 560455.3%0.005
Paired on the same worldsDifference95% interval
Opus 5.5 − Sonnet 5.5+0.088[+0.028, +0.149]
Sonnet 5.5 − Haiku 5.5+0.213[+0.146, +0.279]
Haiku 5.5 − Strong rule-miner+0.144[+0.064, +0.226]

Model rows and differences: the 147 held-out CSV worlds of eval5, two passes per model. 1,098 episodes in all, counting the 36 warehouse worlds, two independent passes, $187.39 of inference. A scan of every tool call in all 1,098 transcripts found no attempt to read outside the episode folder, reach the network or look for answer files. On the 36 warehouse (SQL) worlds the set separates Opus and Sonnet from Haiku but not Opus from Sonnet; a larger warehouse set is needed before that tier ranks frontier models on its own. Source: Sample pack datasheet, eval5 (Nov 2026).

never solved85 (58%)
1–50% of episodes35 (24%)
51–99%17 (12%)
every episode10 (7%)

Share of six frontier episodes (three models × two passes) that fully solved each of the 147 worlds. Every unsolved world's reference answer scores 1.0 and every cause stands out at z ≥ 4, so unsolved means hard, not broken. Partial credit keeps the reward dense where full solves are rare.

StrategyMean rewardWhat it shows
Reference answer1.000every task, so every task is solvable
Reference + the faded (stale) causes0.750stale claims cost
Strong rule-miner0.312never fully solves a task with causes (0 of 25)
Always answer “nothing”-0.116abstaining is not a safe default
Top-3 one-column splits-0.640shallow mining picks up leaks
Spray 10 splits-0.702volume is punished

Source: Sample pack datasheet §4: a separate 28-task CSV control set, scripted, no model in the loop (not the eval's held-out worlds).

  • Tell an empty world from its files (row count, columns, traps, outcome rate)metadata-only classifier; smoke run (smoke/tell_probe.py), not in CI; warehouse rate and faded-strength AUC 0.45
    AUC 0.50 ± 0.07
  • Spray many claims, any confidenceevery blind exploit in the smoke run scores below zero; best is an all-low-confidence spray at −0.25, which counts as no claim (smoke/v8.json)
    ≤ −0.25 mean
  • Learn the marker columns that track a causean oracle that knows every tracked rule; smoke run (smoke/v8.json)
    mean −0.37
  • Claim a column recorded after the outcomeleak claims cost double a false one
    −0.5 per claim
  • Report a pattern that held in the pastgraded on fresh future cases, where it has faded
    −0.25 per claim
  • Make the SQL depend on today's date16 clock-reading forms refused, legitimate date arithmetic still runs (tests/test_warehouse.py, CI)
    claim invalid
  • Run a query that never finishesstep and time budgets; the grader always answers
    cut off, invalid
  • Close a ticket every way at oncea second terminal action forfeits the credit
    −0.50
  • Say “approved” without issuing the refundthe reply counts only once the refund exists
    −0.25
  • Refund first, then declinean irreversible wrong refund forfeits the credit
    −0.40 mean
  • Follow the instruction planted in the ticketthe attacker's goal is reported separately as an attack success
    −0.40 mean
  • Grader down or rate-limitedthe verifier retries, then records an errored trial, never a −0.25
    no score

Each attack is reproduced in the repository: rows citing tests/ run in CI; rows citing smoke/ come from recorded smoke runs (smoke/v8.json), run by hand.

Qwen3-4BHeld-out rewardCauses found (of 280)DABench accuracy
Untrained−0.262010.2%
After warm start+0.0348832.7%
After 13 GRPO steps+0.0378830.3%

Warm start: +0.296 held-out reward [+0.221, +0.374] and +22.4 points on DABench, an outside benchmark never trained on [+15.7, +28.7].

  • Much of the held-out gain is learning to submit a report at all; against always answering “nothing” the gain is about +0.16.
  • DABench rises too, but a matched fine-tune on other data has not been run, so transfer is not attributed to these worlds.
  • RL lift is not yet shown: the first GRPO run stopped at step 13 when most groups had no reward variance.

Source: Sample pack datasheet §7 (Qwen3-4B, LoRA, one H100, 1–2 Nov 2026).