How steady are these results?

A single attempt can make a coin-flip behavior look like a fixed trait, in either direction — that's what the four runs per task are for. The table below is the complete record: all twenty-five tasks, every model, every run. Each cell is how many of that model’s runs were a full pass (every hidden check) on that task; the small line in the Task 19 row breaks that task’s runs down score by score. Task 24, the free drawing, has no hidden checks; its row shows elements found on my checklist (out of 12) and my rank.

A companion check: two tasks (the long-document recall and the closures/caches task) were also re-run for the three Claude models in a fully sealed setup — the model got the prompt text and nothing else, with no tools, no files, and no possible route to the checks or answer keys. All three scored exactly the same sealed as unsealed — evidence, not proof, that the numbers here reflect real capability rather than lucky access.

Reading the table: a cell is the count of runs that passed every hidden check for that task. — means no runs have been done yet for that model on that task. Every cell rests on four complete runs, with one stated exception: Qwen's Task 8 is three. Three samples had earlier been lost to a saving problem on our side and were left out of the totals; they were re-collected on 2026-07-29, so nothing is excluded any more — and none of the published totals moved when they came back.

Behind the partial cells: Flash-Lite’s contradiction-hunt runs (Task 23) always find the device and timeout conflicts but dismiss at least one of the other two; its acrostic misses (Task 20) usually drop the seven-words-per-line rule, while Qwen’s one acrostic miss writes two six-word lines; Gemini 3.6 Flash’s one strict-drawing miss (Task 25) is a 9 of 10.