Model Face-Off ▮
← back to AI Signal

A few honest notes

  • Defaults, on purpose — by four different routes. Five of the thirteen models ran at their own chat site's default setting; DeepSeek ran with Expert + DeepThink on (the setting that serves the DeepSeek-V4-Pro model), the two Gemini models ran through Google's API at its defaults, Claude Opus 5 ran through the Claude command-line tool at its default, the three GPT-5.6 models ran through the Codex command-line tool at its default, and DeepSeek V4-Flash ran through OpenCode's free route. Those settings differ from each other, so this isn't a perfectly controlled lab experiment — it's a snapshot of what a real person gets by pasting the same request into each.
  • One shot each. I took the first answer, with no retries or hints. A model might well fix its slip if you point it out — but the whole point was to see who gets it right unprompted on messy data.
  • The scoring checks itself. Before trusting the automated checks, I ran them against a known-correct set of answers (everything should pass) and a deliberately-broken set (the exact traps should fail) — and they behaved as expected. The checks stay private so no model can memorise them.
  • A standing comparison, not a one-time test. The tasks and the checks never change, so any model can be compared with any other by reading their rows, and notable new models get added over time. No row is a final verdict — fifteen of the twenty-five tasks show at least one dropped run for at least one model (see “How steady are these results?” for exactly which ones), which is why every score here is an average across four runs with the worst run shown beside it, never a single sitting.

If you take one thing away: these models are separated less by intelligence than by carefulness. Five of them, across four labs, share one identical decade-old Python bug; twelve of the thirteen answer the same counting question the same wrong way, run after run — including the newest and strongest column; and several results that look like fixed traits in a single sitting turn out to be coin-flips across four. A one-run comparison of models — anyone's — deserves deep suspicion.

Related writing

Longer pieces that lean on these results, or that this page grew out of. Each is on the site and on Medium.