A better model will not save you from a bad rubric

Roboflow published a piece this week arguing that GPT-5.6 Sol is the strongest vision model OpenAI has shipped. Their evidence is specific, their test set is public, and that already puts the claim ahead of most of what gets posted about model quality.

It is a good result. It is also, read too quickly, a procurement decision heading for trouble.

The word doing all the work in that sentence is "strongest".

Vision covers many tasks

Vision covers reading a dense table, counting objects in a crowded frame, spotting a small defect on a large surface and describing a scene.

Those are four tasks with four different failure modes, and one word covers all of them. A model can move decisively ahead on three and stay flat on the fourth, and every summary of that situation is a choice about which task the summariser happened to care about.

Usually that choice is invisible, including to the person making it.

The problem is old, and it applies to people as much as to models. "Strong analyst" is also a compression of several judgements against an implicit standard, and the standard is where all the disagreement lives. We do not notice, because with people we at least argue about the standard out loud.

Write the rubric first

The discipline that helps here is unglamorous and slightly annoying. Define what a good answer looks like before you see any answers.

Rubrics written after the outputs arrive tend to describe the outputs, through the ordinary human tendency to find the criteria that the impressive thing satisfies. Nobody has to be dishonest for it to happen. You will do it, I will do it, and neither of us will notice.

A rubric written in advance also survives the model changing underneath you, and a leaderboard position lasts only until the next release.

The chart above is invented, and its shape carries the argument. The spread on cost is enormous, and for one particular task the spread on score is often small. So the useful question is "which is good enough here, and what does the gap cost me per thousand calls".

Where a public benchmark earns its keep

Aggregate benchmarks are excellent at exclusion. They tell you a model is not in contention, and in a field shipping weekly, cheap exclusion is worth real money.

They are poor at selection, because selection depends on a task distribution the benchmark has never seen. Roughly:

Fifty examples

So use the leaderboard to build a shortlist, then decide with your own evaluation.

Assemble fifty examples from your real workload, including the ones your current process gets wrong. Score them against the definition you wrote first.

Fifty is small enough to build in an afternoon and large enough to embarrass a confident assumption. That set will outlive several model generations, which is more than most benchmarks manage, and it turns "which model is best" into a question you can answer for yourself.

Roboflow's write-up is worth reading. Then run your own fifty and decide from those.