What 240 local-model benchmarks say
We ran the same sixteen-category evaluation across five models, qwen2.5-coder, deepseek-r1, gemma3, phi4-mini and the qwen3 family, on three Apple Silicon machines we keep around for exactly this kind of test. Multiply that out and you land on 240 separate runs, which is where the number in the title comes from. What came back said less about which model is best and more about how much your own hardware decides the answer.
The three machines: a base M1 with 16GB of unified memory, an M2 Pro with 32GB, and an M4 Max with 64GB. Same prompts, same scoring, same sixteen categories, run once per model per machine. The only variable we changed on purpose was the hardware underneath.
The scores move more than the models do
qwen2.5-coder’s code generation score on the M1 sits meaningfully lower than the same model on the M4 Max. Not because the weights changed. Quantization, available RAM, and how much the machine throttles under a long run all shift the number, and they shift it by more than the gap between two different models on the same machine. deepseek-r1 shows the same pattern in reasoning, gemma3 in translation.
The ranking moves too. A model that leads on one machine slips behind another on a second, in the same category. On the M1, phi4-mini and gemma3 trade places in summarisation depending on how much memory pressure the rest of the system is under at the time. That is not noise we filtered out. It is the whole point.
A published score is a snapshot of someone else’s machine
Public leaderboards run once, on one machine, and publish a single number per model per category. That number is real, but it describes conditions you don’t share: someone else’s RAM, someone else’s quantization choices, someone else’s thermal envelope. Nothing about it tells you what qwen2.5-coder will score on the machine sitting in front of you.
This is why Soriku doesn’t import a public benchmark table and call it done. When you add a model, Soriku benchmarks it on your hardware, in your sixteen categories, and shows the spread alongside the score. A number without its uncertainty is a number you can’t trust yet.
What we changed because of it
We stopped trusting a category score without also reading its spread. A tight cluster of results across three test runs means something different from three runs that landed nowhere near each other, even when the average is identical. The capability map shows both now, and a score with a wide spread gets treated as provisional until it’s re-run.
None of this makes benchmarking easier. It makes the number closer to true. A score that only holds on the machine that produced it was never really a score, it was a guess wearing a decimal point.