Listen to this article
AI may have been used to generate this audio.
Press Space or K to play or pause, and J or L to go back or forward 15 seconds.Qwen3.8 27B posted a 52 on Artificial Analysis this week. For a model that size that is a strong number, and the obvious story to write is the comparison to models several times larger.
It is a real result. It is also one number standing in for a basket of very different things, and that is where it starts costing people money.
Two models, same score, opposite decision
A composite score sums performance across a set of tasks. Two models with the same total can be unalike in every way that would matter to you.
If your product summarises hundred-page documents, Model B is the only candidate on that chart, and the leaderboard position will never tell you so. If you write code, the reverse holds. The same number leads to opposite decisions, and the number is what goes in the board pack.
The average is real, and it says nothing about your workload.
This is the oldest problem in assessment
None of this is new, and it applies well beyond models.
A single number gets adopted because it enables ranking, and ranking is what committees need, because it converts a judgement into a procedure. A procedure can be defended in a meeting. A judgement has to be argued.
Then the number quietly redefines the thing it was meant to measure. People optimise toward the index, the index stops carrying the information it originally carried, and everyone keeps reporting it anyway, because the alternative is admitting the last two years of reporting meant less than it looked.
The identical shape shows up in learning programmes. "Completion" is a composite too: attendance, clicks, time-on-page. Once it becomes the reported number, the programme reorganises itself around producing completions. It will keep hitting target while the thing you wanted stays where it was.
What a useful evaluation looks like instead
The fix is a smaller, uglier, more specific index that you own.
Leaderboard score | Your own evaluation |
|---|---|
Fixed public task mix | Sampled from your actual traffic |
One number, ranked | Per-task pass rates, unranked |
Updated when the vendor ships | Updated when your product changes |
Answers "which model is best?" | Answers "is this good enough for us?" |
Cannot be gamed by you | Can be gamed by you, so don't |
A hundred real examples from your own logs, graded by someone who knows what a correct answer looks like, beats every public benchmark for the specific purpose of deciding what to ship. It takes an afternoon. Most teams never spend it.
So use the leaderboard for what it is good at
Aggregate benchmarks are good at exclusion. They tell you a model is not in contention, and in a field shipping weekly, cheap exclusion is valuable.
They are poor at selection, because selection depends on a task distribution the benchmark does not know about.
Use the leaderboard to build the shortlist. Then decide on your own numbers. The organisations getting real value out of small models are, almost without exception, the ones that measured on their own work and found the frontier model was solving a problem they did not have.
The Qwen numbers are here. Read them as a filter for your shortlist.
One email when we publish. Research, product decisions, and what teams report back.
