The one kind of task that told them apart
Eight of the eighteen coding tasks were passed by every model tested, on every run, including the genuinely tricky ones. The gap opened on the tasks that mimic real data — a bank statement, a shared-format ledger — where the input is deliberately messy. Two traps were hidden in that data, and each model's fate came down to a single line.
Trap 1 — numbers with commas in them
A money amount like 1,200.50 contains a comma. If you split a
date,description,amount line on every comma, that amount
shatters into two pieces, the row no longer has the right shape, and it silently
vanishes. The fix is one word — tell the split to stop after the first two
commas:
line.split(',', 2) # split on the first two commas only
line.split(',') # splits 1,200.50 apart, row is lost
Trap 2 — dates that can't exist
The data also plants impossible dates like 2026-02-29 (2026 isn't
a leap year) and 2026-04-31 (April has 30 days), which are supposed
to be thrown out. Checking against a real calendar catches them. Checking only
that the day is "between 1 and 31" does not:
date(year, month, day) # a real calendar refuses Feb 29
1 <= day <= 31 # "31 is a valid number", so Feb 31 passes
The run-by-run record has this task's run-by-run rates. What the runs reveal underneath: Grok's one failing run trips all three thousands-comma cases at once — the identical comma trap that catches both Gemini models in every single run. DeepSeek's two failing runs are two different multi-case misses, not one repeatable slip. And Sonnet and Qwen clear this task in every run — the only two models that do.
Trap 3 — a decade-old Python subtlety, not a new one (added 21 July)
A harder task (Task 12) asks for the real elapsed seconds between two local
wall-clock times in a timezone that observes daylight saving, across the exact
moment the clock jumps. Python's own documentation contains the trap: when you
subtract two aware datetimes that happen to share the identical
tzinfo object, Python silently takes a shortcut and compares the
base (naive) datetimes, discarding any difference in UTC offset. Diffing
.timestamp() values instead sidesteps the shortcut entirely:
return e.timestamp() - s.timestamp() # sidesteps the shortcut entirely
return (end_dt - start_dt).total_seconds() # same tzinfo object → naive subtraction, DST offset silently dropped
What makes this one worth calling out: three completely
independent, differently-shaped pieces of code fall into the identical trap.
Sonnet's and Qwen's are simple one-line subtractions. Grok's is nothing like
that — it's an elaborate 15-line routine that manually walks backwards
second-by-second to correctly resolve the spring-forward gap, gets that hard
part right, and still ends on the exact same naive subtraction as the others. Sophistication in the gap-handling logic did not protect it from a
bug in the very last line. DeepSeek fails all 8 cases too, but for an unrelated
reason: its code imports NonExistentTimeError from
zoneinfo, a name that does not exist there, so the function never
even runs.
The run-by-run record has the exact run-by-run rates — for most of the field this trap is close to a coin-flip. What's telling is how each model fails. Grok and DeepSeek each invented a nonexistent zoneinfo exception in one run — different names, same fabrication. Sonnet, Qwen and both Gemini models reproduce the plain naive-subtraction bug in almost every failing run (Gemini 3.5 Flash-Lite identically in all four). Fable clears the task every run, and Opus's .timestamp()-based habit is the only other approach that shows real, if imperfect, resistance.
The question almost every model gets wrong — and the one model that doesn't. A non-code task (Task 19) asks five character-counting questions, including: count the words in a sentence where a hyphenated compound like "twenty-first-century" counts as ONE word and a standalone em dash does not count as a word at all. The correct count is 13. Twelve of the thirteen models — Sonnet 5, Opus 4.8, Opus 5, Fable, Grok, DeepSeek V4, DeepSeek V4-Flash, Qwen, Gemini 3.6 Flash, and all three of GPT-5.6 Sol, Terra and Luna — get this wrong in every one of their four runs, answering 12 instead. The sole exception is Gemini 3.5 Flash-Lite, which gets it right in 3 of its 4 runs — the only model, at any sample size, to answer this question correctly at all. Every other question on the task, every model gets right almost every run: Opus 4.8 also misses the vowel-count question in all four of its runs (answering 6 where the correct count is 8, so it never scores better than 3 of 5); Sonnet fumbles the string-reversal question in one of its four runs; Gemini 3.6 Flash fumbles the same string-reversal question in two of its four runs. It's the single most consistent finding on this page: thirteen different systems, from six different labs, almost all landing on the identical wrong number for the same sentence, run after run — and the newest and strongest column, Opus 5, misses it exactly like the rest.
A misdirection-reasoning task (Task 21) reworded the classic fox/chicken/grain river-crossing puzzle with a boat that holds two items instead of one. Eleven of the thirteen models solve it correctly (3 crossings) in every one of their four runs — Sonnet 5, Opus 4.8, Opus 5, Fable, Grok, DeepSeek V4, Qwen, Gemini 3.6 Flash and all three of GPT-5.6 Sol, Terra and Luna, each via a valid 3-crossing route. DeepSeek V4-Flash misses it once in four. The outlier is Gemini 3.5 Flash-Lite, which gets it wrong in all 4 runs — the only model in this comparison, at any sample size, to fail the river-crossing question consistently rather than as an occasional miss. Worth noting against an earlier single-sample impression: Sonnet 5 was once recorded as failing this puzzle. At four runs it is clean in all of them, which is exactly the kind of verdict a single sample gets wrong.
That is the shape of the gap between the two hardest parsing tasks — the table above has every rate. The details worth keeping: Grok's two ledger misses are two different single cases (a comma-bearing date, a basic INR row), not one repeatable weakness; Fable's ledger failures are the same single INR case in three of four runs; DeepSeek never clears the ledger's one bad-date case even as DeepSeek-V4-Pro (the model its Expert + DeepThink setting serves); and Gemini 3.6 Flash sails the ledger every time while failing the messy-statement parser every time — the mirror image of most of the field. It is not about how deeply a model can think. It is about whether it reaches for the careful tool when told, plainly, that the data is messy — and how consistently.