What Your LLM Benchmark Is Actually Measuring

A benchmark can steer you toward the wrong model even when you test every model with exactly the same settings.

On a 200-question sample from GSM8K, a benchmark of grade-school math word problems, my zero-shot test made Llama look much better than Granite: 68.5% versus 33.5%. After giving both models more room to answer and using an AI-assisted audit to check what they actually said, Granite came out ahead on that sample, 93.5% to 88.0%. I’d been cutting off its answers and then misreading some of the ones it finished.

A shared token limit can cut off a wordier model, and a shared grader can reject answers in a format it doesn’t recognize, sometimes enough to decide the winner. Carry that ranking into an application with different requirements, and you can reject a model before testing the conditions where it might work better.

If you really need an answer inside that limit and in that exact format, those are useful failures to measure. But I’d been reading the score as “how good is this model at math?” without checking how much of it meant “did my setup let it finish, and did my code understand the answer?” That’s a fairly expensive assumption to build a model recommendation on, and checking it changed mine.

I was cutting off the answer

These runs used lm-evaluation-harness, or lm-eval, a public evaluation toolkit, on HOMER, my Strix Halo miniPC. The two models were granite4-h-small and llama31-8b, and the chat adapter in my installed lm-eval version, 0.4.9.1, gave them a default 256-token generation limit. Both models got the same allowance, but I hadn’t checked what that allowance did to each model under each prompt.

The choice of prompt recipe changed which model came out ahead, so I focused the follow-up on two recipes supplied by the toolkit. The zero-shot recipe asks the model to “think step by step” without showing any worked examples, and 123 of Granite’s 200 responses were unfinished before resolving the answer. The Llama-style eight-shot recipe shows eight worked examples, and only four responses were unfinished. The much shorter answers suggest those examples also helped it get to the point, though I haven’t isolated that effect from the recipes’ other differences.

I replayed the saved prompts on HOMER with a 1,024-token allowance, keeping the same Q4_K_M weights, llama.cpp binary, and request settings apart from the cap. I also used Codex to audit the answers in both sets of responses, accepting equivalent numeric notation and rejecting numbers that happened to match before the model had actually answered. Since this is an AI-assisted audit rather than independent human grading, I’ve kept the original extractor scores alongside it.

Each cell below covers the same 200 questions, and the column numbers are the output token limits. “Extractor” is the toolkit’s automatic score. “Audited” is the score from that answer review, which checked whether the response gave the requested answer against the benchmark reference. It didn’t require every reasoning step to be sound, and the reviewer can make mistakes too.

Read down the original extractor column, and the recipe decides which model wins: Llama leads in zero-shot, while Granite leads in eight-shot. Read down the final audited column, and Granite leads in both. Its own gap between recipes has also shrunk from 53 points to one, though I wouldn’t read anything into that one-point difference on this sample.

The control reruns didn’t reproduce every original response exactly, so I can’t attribute every changed answer to extra tokens. But 50 of Granite’s longer zero-shot responses do start with the entire unfinished original, byte for byte, and then continue to a correct answer. For those questions, I can see the answer arriving after the point where I’d previously stopped it.

At 1,024 tokens, none of Granite’s responses hit the limit. Its zero-shot answers averaged 326.47 generated tokens, compared with 110.91 for the eight-shot answers. Llama still hit the limit on six zero-shot questions and three eight-shot questions. More room helped, but it didn’t make this an unlimited-budget test.

And sometimes I wasn’t reading it properly

The longer run still leaves a fourteen-point gap between Granite’s two recipes under automatic scoring, 75.0% versus 89.0%, so the token limit doesn’t explain the whole result. The next problem was how the toolkit read the answers.

The flexible extractor takes the last matching number, so when one archived Granite answer correctly said a wire made 8 pieces, then mentioned that each was 6 inches long, the grader took 6. Another gave 26.00, which failed comparison with 26. It could also award a pass when an unfinished response stopped at step 4, on a question whose answer happened to be four.

This particular decimal problem wasn’t even waiting for me to discover it. huaanrui reported it in lm-eval issue #3214 in August 2025, including the same Rory sandwich question that appears in my sample. The issue is closed, but as of this writing, September 5, 2026, the decimal bug still reproduces on upstream main. I checked its actual extractor and scoring code: $29. passes against 29, while $29.00. fails. It removes the dollar sign and trailing period, then compares 29.00 with 29 as strings. The same happens with 26.00 versus 26. That’s the behavior at commit b954108, using the shipped GSM8K task settings.

Once the audit reads the requested answer instead, Granite’s zero-shot rerun goes from 75.0% to 93.5%. The eight-shot rerun goes from 89.0% to 92.5%. The grading error was much more expensive for one recipe than the other.

Sclar, Choi, Tsvetkov, and Suhr’s FormatSpread work had already established that prompt formats can affect models differently enough to reverse comparisons. Their experiments preserve meaning more carefully than my shipped recipes, which change examples, wording, and stop sequences together. Hua and colleagues’ Flaw or Artifact? supplies another reason to investigate: much of their measured prompt sensitivity shrank when they replaced heuristic scoring with a model judge.

Those papers prompted me to check how much of my own spread survived when the model could finish and the answer got read correctly, and for these two recipes on Granite, very little did. If your application needs an answer inside 256 tokens, Granite’s unfinished responses are real failures. If you can give it more room, my original zero-shot comparison would have sent you toward Llama on evidence that doesn’t survive the change.

Consistent settings tell you which model worked better under those settings. Before you build around that recommendation, check whether it survives the settings your application will actually use. I’d skipped that part.

“Let it think” is a setting to test

The budget problem also showed up in my own 95-task suite, which compares the cost of getting correct answers on this box. This is a different workload from GSM8K, so its percentages aren’t comparable to the ones above. It runs each task twice: once asking for an immediate answer, called terse mode, and once allowing the model to work through the problem, called free mode. Both have a 4,096-token generation cap, and these three models show why the choice matters:

On these tasks, Mistral scores much better when I let it work through the problem. Gemma’s score falls, partly because it runs out of tokens. Models that already reason can use up the whole allowance continuing to reason, leaving my harness waiting for an answer.

If you need a finished answer within that allowance, running out of tokens is a real failure. Your application doesn’t get much use out of “hang on, I’m still thinking.” These runs don’t establish what the model could solve with more room, though. You lose that distinction when all you get is an accuracy score.

I also tested reasoning on and off, using --reasoning off on three Gemma 4 sizes. Within each pair, the weights and tasks stayed the same. These scores combine both prompting modes, so they are not directly comparable to the separate mode scores above.

Turning the headline feature off helped the two smaller models substantially, and after the 26B result, “turn reasoning off” looks like excellent advice for that model on this workload. It’s a short hop from there to telling Gemma users generally to do it, except the 31B model got slightly worse. I’d want more evidence before making much of that small decline, but it gives me no reason to recommend turning reasoning off across the family.

If your application disables reasoning or imposes a tighter output budget, check whether a recommendation based on reasoning-enabled tests still holds. You can change the outcome without touching the model file.

Srivastava and colleagues’ LLMThinkBench study also finds that longer reasoning can accompany lower accuracy on basic math, and that restricting the token budget can cause large drops. My Gemma rows show what that tuning question looks like within one family on this workload.

My grader had a preference for model families

Within that same custom suite, the grader caused a separate problem. Prompts and reasoning flags are at least things I’d expect to tune, but checking whether a list of numbers is correct felt like a rather less exciting part of the project. I wrote it, so surely I could trust it to recognize a list of numbers… right??

Mine objected to this:

Expected: [33, 15]
Received: [33,15]

I collapsed runs of whitespace but didn’t remove the spaces around separators, so all the numbers were right, and my code was objecting to the spaces.

Fixing spacing rescued 30 answers for qwen3-30b-thinking and 23 for qwen3-4b-thinking. Their instruct counterparts recovered 2 and 7 respectively. gemma-3-4b recovered none.

The thinking variants tended to write compact lists. I was comparing instruct and thinking models using a grader that disproportionately rejected the thinking models’ formatting. After re-grading the stored outputs, the comparison looked like this:

“Reasoning training costs twenty accuracy points at 30B” looked like a result worth investigating, and most of it turned out to be my grader objecting to [33,15]. Naming a unit caught a different set of models: llama31-8b answered 221 units. when the expected string was 221, and correcting numeric extraction moved its overall score from 45.3% to 60.0%, while phi-4 moved from 58.9% to 73.2%. The spacing fix had rescued no answers for either of them.

Each formatting convention caught a different set of models. In the Qwen comparison, the error followed the very distinction I was trying to measure, so even the instruct-versus-thinking comparison within a family was affected. Adding more test questions won’t fix a rule like that, but they will give you a more precise estimate of how often each model writes a list without spaces.

A public harness can make the same distinction

“Damen, maybe your grader is just bad” is a fair objection, but back in lm-eval, there’s another comparison you can make without generating a single new answer. For its five-shot gsm8k task, a separate recipe from the two in the opening, lm-eval reports two readings of the generated answers. strict-match looks for a number after ####, the convention demonstrated in the examples, while flexible-extract takes the last matching number in the response. The model doesn’t even have to run again, because both filters grade the output it already produced.

Four of the six models change position, with Llama moving from second-last to third. phi-4’s score changes by almost sixty points, while qwen3-4b-instruct barely moves in accuracy at all.

In the logged responses, phi-4 emits #### only 13% of the time, while gemma-3-12b does so 88% of the time. Requiring that marker accounts for much of their difference in sensitivity to the grading rule.

If your application needs that exact format, you absolutely care whether the model supplies it. But if you’re choosing a model for arithmetic, how much of its apparent struggle is with writing ####? Check the flexible extractor’s passes, too. Finding the right number somewhere convenient isn’t enough to establish that the model answered the question correctly.

I’m glad the harness prints both. The trouble starts when one column becomes “GSM8K accuracy” with everything else stripped off, and you carry that number into a decision about which model to use.

Jo and colleagues examine this in Finding Answers in Thought Matters. They show extraction methods changing model rankings and inspect the output conventions responsible. They also examine incomplete reasoning caused by repetition, which is relevant to the unfinished responses I discuss below.

Fine, I’ll make it write the right format

That leaves an obvious question: could you make the model produce the format you need in the first place? Back on the custom suite, I tried llama.cpp’s grammars, which restrict the text a model is allowed to generate. If I made the model write the format my strict grader expected, would those answers start passing without another patch to the grader?

Will Kurt, Remi Louf, and Clémentine Fourrier had already tested structured generation as a way to reduce prompt sensitivity and ranking changes on GSM8K and GPQA. Their GSM8K constraint allowed reasoning before the answer, a detail that becomes rather relevant to my attempt below.

I used llama.cpp grammars to constrain answer shapes: brackets and separators for a list, for example, without supplying the values or the expected list length. For the formatting tasks, the generator read the expected answer to select separators and character classes, so it did use answer-key information about the format. Control and constrained runs used the same prompts on the same loaded server. These are 84 terse-mode tasks, excluding the code-execution tasks. Here, strict grading requires an exact final-line match, while tolerant grading accepts variations in spacing, punctuation, and surrounding text. Neither score includes a separate answer audit like the one in the opening.

For phi-4, Llama, and Qwen Coder, the successful task sets matched exactly between tolerant grading of the original output and strict grading of the constrained output. That supports the explanation that these particular failures were avoidable formatting failures. But I’ve changed generation too, and some models now solve different tasks.

qwen3-4b-instruct, for example, keeps the same total but swaps two task verdicts. For phi-4-reasoning, tolerant control and strict constrained scores both come out at 45.2%, but each condition solves 15 tasks the other fails. Look only at the totals, and you’ll miss that entirely.

Then I applied the grammar from the first generated token to models whose templates open a thinking block. qwen3-4b-thinking went from 98.8% under tolerant control grading to 0.0% under strict constrained grading. qwen3.6-35b-a3b went from 86.9% to 0.0%.

My grammar excluded the characters needed to close the thinking block, and neither model emitted a closing tag in any constrained response. I’d restricted the output before they could get to the part I wanted to restrict. Some kept going as long strings of digits and commas until the budget ran out.

And yes, I ran this before reading the relevant work. Tam and colleagues had already shown that format restrictions can degrade reasoning, and The Format Tax finds that separating reasoning from formatting recovers much of the loss in its tests. Reading first would have been a cheaper way to find that out. I haven’t run that separation here, so what I’ve established is where my particular constraint setup fails, and I can’t recommend it to someone using those thinking models, even though it recovered correct answers for the bare-answering ones.

What can you trust a comparison to tell you?

The useful comparison depends on how much of your application you’re willing to change to accommodate a model. If you need a drop-in replacement behind an existing API, the prompt, output schema, and budget are real requirements. “It works great if you change all of those” may be true and still be no help to you. Testing the models under those fixed conditions answers the question you actually have.

If you can adapt the application, give each model a comparable tuning budget, and check the selected setup on tasks you didn’t use for tuning. Pick a winner before doing that, and you risk selecting whichever model happened to like your first attempt at a prompt.

If you’re making a broader claim about model ability, show results across plausible configurations, including the ones that reverse your preferred ranking.

The lm-eval maintainers’ Lessons from the Trenches is a good reference for this: inspect outputs, document the setup, and be careful about comparisons across implementations. Here’s how I’d apply that advice to the failures I found:

• Keep the responses and their termination information. Inspect examples of wrong, correct, and unfinished output. In my custom suite, all 15 free-mode passes for phi-4-reasoning were truncated responses: the correct number survived at the end of a repetition loop. If the requirement is a completed answer, those are failures. An answer-string score alone concealed that distinction.
• Try the settings likely to change in actual use. Prompt mode, reasoning switch, generation budget, and output constraints belong next to the model name. Check each candidate. A good setting for one size does not establish a family default.
• Grade the stored output more than one defensible way. On the five-shot GSM8K task, the two extractor scores already provide a useful first check. Investigate the disagreements before interpreting either column as ability.
• Check whether the decision survives. If a reasonable configuration change reverses the winner, report the reversal. If it changes a score but leaves the decision intact, report that too. Your readers can use a result with known limits.

And use a public harness where it covers the workload. Seriously. I set out to test models and ended up maintaining rules about how to read commas and units. Public tooling still needs checking, as GSM8K demonstrates, but at least I can compare notes with other people using the same implementation.

If picking another shipped recipe changes the winner, you want to find that in the writeup, before you’ve built an application around the recommendation.

Scope and methodology

These are local runs on quantized open weights through a chat endpoint on the Strix Halo box. The GSM8K sample is 200 questions, not the full benchmark. The budget follow-up covers two models and two untuned shipped recipes. The separate extractor comparison uses the five-shot gsm8k task. These runs do not establish definitive model rankings or optimized scores.

The follow-up completed 800 requests at 1,024 tokens and 80 predetermined controls at 256, with temperature zero, seed 1234, and four concurrent requests. Codex reviewed the original and new answers, without blinding, and ambiguous or unresolved answers count as incorrect. The new logs retain the termination metadata that the original logs omitted. The 80 predetermined controls, replayed at the original 256-token limit, reproduced 78 pass/fail verdicts and 40 response strings exactly. After 679 requests, the thermal guard interrupted the run. I had left the automatic fan curve running instead of pinning the benchmark fans. The remaining 201 requests ran with full fans and unchanged inference settings. No throughput comparison is being made from this run.

My custom suite has 95 tasks run in two modes. The prompt and reasoning-switch results shown here use one run per condition, temperature zero, and a 4,096-token cap. Percentages and point changes are rounded separately from the underlying counts. That cap is part of the result, especially where reasoning fails to finish. I retained historical re-grading evidence for eight complete runs. The other runs were measured after the comparator fixes. The grammar experiment covers 84 terse tasks per model, and tests one constraint design.

I checked the tables against the stored results. The custom suite retained output tails, which support answer-extraction checks but not a complete audit of every generation. Generated source for the code-execution slice was not retained and cannot be re-graded.