On a 200-question sample from GSM8K, a benchmark of grade-school math word problems, my zero-shot test made Llama look much better than Granite: 68.5% versus 33.5%. After giving both models more room to answer and using an AI-assisted audit to check what they actually said, Granite came out ahead on that sample, 93.5% to 88.0%. I’d been cutting off its answers and then misreading some of the ones it finished.

A shared token limit can cut off a wordier model, and a shared grader can reject answers in a format it doesn’t recognize, sometimes enough to decide the winner. Carry that ranking into an application with different requirements, and you can reject a model before testing the conditions where it might work better.

If you really need an answer inside that limit and in that exact format, those are useful failures to measure. But I’d been reading the score as “how good is this model at math?” without checking how much of it meant “did my setup let it finish, and did my code understand the answer?” That’s a fairly expensive assumption to build a model recommendation on, and checking it changed mine.

I was cutting off the answer

These runs used lm-evaluation-harness, or lm-eval, a public evaluation toolkit, on HOMER, my Strix Halo miniPC. The two models were granite4-h-small and llama31-8b, and the chat adapter in my installed lm-eval version, 0.4.9.1, gave them a default 256-token generation limit. Both models got the same allowance, but I hadn’t checked what that allowance did to each model under each prompt.

The choice of prompt recipe changed which model came out ahead, so I focused the follow-up on two recipes supplied by the toolkit. The zero-shot recipe asks the model to “think step by step” without showing any worked examples, and 123 of Granite’s 200 responses were unfinished before resolving the answer. The Llama-style eight-shot recipe shows eight worked examples, and only four responses were unfinished. The much shorter answers suggest those examples also helped it get to the point, though I haven’t isolated that effect from the recipes’ other differences.

I replayed the saved prompts on HOMER with a 1,024-token allowance, keeping the same Q4_K_M weights, llama.cpp binary, and request settings apart from the cap. I also used Codex to audit the answers in both sets of responses, accepting equivalent numeric notation and rejecting numbers that happened to match before the model had actually answered. Since this is an AI-assisted audit rather than independent human grading, I’ve kept the original extractor scores alongside it.

Each cell below covers the same 200 questions, and the column numbers are the output token limits. “Extractor” is the toolkit’s automatic score. “Audited” is the score from that answer review, which checked whether the response gave the requested answer against the benchmark reference. It didn’t require every reasoning step to be sound, and the reviewer can make mistakes too.

Read down the original extractor column, and the recipe decides which model wins: Llama leads in zero-shot, while Granite leads in eight-shot. Read down the final audited column, and Granite leads in both. Its own gap between recipes has also shrunk from 53 points to one, though I wouldn’t read anything into that one-point difference on this sample.

The control reruns didn’t reproduce every original response exactly, so I can’t attribute every changed answer to extra tokens. But 50 of Granite’s longer zero-shot responses do start with the entire unfinished original, byte for byte, and then continue to a correct answer. For those questions, I can see the answer arriving after the point where I’d previously stopped it.

At 1,024 tokens, none of Granite’s responses hit the limit. Its zero-shot answers averaged 326.47 generated tokens, compared with 110.91 for the eight-shot answers. Llama still hit the limit on six zero-shot questions and three eight-shot questions. More room helped, but it didn’t make this an unlimited-budget test.

And sometimes I wasn’t reading it properly

The longer run still leaves a fourteen-point gap between Granite’s two recipes under automatic scoring, 75.0% versus 89.0%, so the token limit doesn’t explain the whole result. The next problem was how the toolkit read the answers.

The flexible extractor takes the last matching number, so when one archived Granite answer correctly said a wire made 8 pieces, then mentioned that each was 6 inches long, the grader took 6. Another gave 26.00, which failed comparison with 26. It could also award a pass when an unfinished response stopped at step 4, on a question whose answer happened to be four.

This particular decimal problem wasn’t even waiting for me to discover it. huaanrui reported it in lm-eval issue #3214 in August 2025, including the same Rory sandwich question that appears in my sample. The issue is closed, but as of this writing, September 5, 2026, the decimal bug still reproduces on upstream main. I checked its actual extractor and scoring code: $29. passes against 29, while $29.00. fails. It removes the dollar sign and trailing period, then compares 29.00 with 29 as strings. The same happens with 26.00 versus 26. That’s the behavior at commit b954108, using the shipped GSM8K task settings.

Once the audit reads the requested answer instead, Granite’s zero-shot rerun goes from 75.0% to 93.5%. The eight-shot rerun goes from 89.0% to 92.5%. The grading error was much more expensive for one recipe than the other.

Sclar, Choi, Tsvetkov, and Suhr’s FormatSpread work had already established that prompt formats can affect models differently enough to reverse comparisons. Their experiments preserve meaning more carefully than my shipped recipes, which change examples, wording, and stop sequences together. Hua and colleagues’ Flaw or Artifact? supplies another reason to investigate: much of their measured prompt sensitivity shrank when they replaced heuristic scoring with a model judge.

Those papers prompted me to check how much of my own spread survived when the model could finish and the answer got read correctly, and for these two recipes on Granite, very little did. If your application needs an answer inside 256 tokens, Granite’s unfinished responses are real failures. If you can give it more room, my original zero-shot comparison would have sent you toward Llama on evidence that doesn’t survive the change.

Consistent settings tell you which model worked better under those settings. Before you build around that recommendation, check whether it survives the settings your application will actually use. I’d skipped that part.

“Let it think” is a setting to test

The budget problem also showed up in my own 95-task suite, which compares the cost of getting correct answers on this box. This is a different workload from GSM8K, so its percentages aren’t comparable to the ones above. It runs each task twice: once asking for an immediate answer, called terse mode, and once allowing the model to work through the problem, called free mode. Both have a 4,096-token generation cap, and these three models show why the choice matters:

On these tasks, Mistral scores much better when I let it work through the problem. Gemma’s score falls, partly because it runs out of tokens. Models that already reason can use up the whole allowance continuing to reason, leaving my harness waiting for an answer.

If you need a finished answer within that allowance, running out of tokens is a real failure. Your application doesn’t get much use out of “hang on, I’m still thinking.” These runs don’t establish what the model could solve with more room, though. You lose that distinction when all you get is an accuracy score.

I also tested reasoning on and off, using --reasoning off on three Gemma 4 sizes. Within each pair, the weights and tasks stayed the same. These scores combine both prompting modes, so they are not directly comparable to the separate mode scores above.

Turning the headline feature off helped the two smaller models substantially, and after the 26B result, “turn reasoning off” looks like excellent advice for that model on this workload. It’s a short hop from there to telling Gemma users generally to do it, except the 31B model got slightly worse. I’d want more evidence before making much of that small decline, but it gives me no reason to recommend turning reasoning off across the family.

If your application disables reasoning or imposes a tighter output budget, check whether a recommendation based on reasoning-enabled tests still holds. You can change the outcome without touching the model file.

Srivastava and colleagues’ LLMThinkBench study also finds that longer reasoning can accompany lower accuracy on basic math, and that restricting the token budget can cause large drops. My Gemma rows show what that tuning question looks like within one family on this workload.

My grader had a preference for model families

Within that same custom suite, the grader caused a separate problem. Prompts and reasoning flags are at least things I’d expect to tune, but checking whether a list of numbers is correct felt like a rather less exciting part of the project. I wrote it, so surely I could trust it to recognize a list of numbers… right??

Mine objected to this:

Expected: [33, 15]
Received: [33,15]

I collapsed runs of whitespace but didn’t remove the spaces around separators, so all the numbers were right, and my code was objecting to the spaces.

Fixing spacing rescued 30 answers for qwen3-30b-thinking and 23 for qwen3-4b-thinking. Their instruct counterparts recovered 2 and 7 respectively. gemma-3-4b recovered none.

The thinking variants tended to write compact lists. I was comparing instruct and thinking models using a grader that disproportionately rejected the thinking models’ formatting. After re-grading the stored outputs, the comparison looked like this:

“Reasoning training costs twenty accuracy points at 30B” looked like a result worth investigating, and most of it turned out to be my grader objecting to [33,15]. Naming a unit caught a different set of models: llama31-8b answered 221 units. when the expected string was 221, and correcting numeric extraction moved its overall score from 45.3% to 60.0%, while phi-4 moved from 58.9% to 73.2%. The spacing fix had rescued no answers for either of them.

Each formatting convention caught a different set of models. In the Qwen comparison, the error followed the very distinction I was trying to measure, so even the instruct-versus-thinking comparison within a family was affected. Adding more test questions won’t fix a rule like that, but they will give you a more precise estimate of how often each model writes a list without spaces.

A public harness can make the same distinction

“Damen, maybe your grader is just bad” is a fair objection, but back in lm-eval, there’s another comparison you can make without generating a single new answer. For its five-shot gsm8k task, a separate recipe from the two in the opening, lm-eval reports two readings of the generated answers. strict-match looks for a number after ####, the convention demonstrated in the examples, while flexible-extract takes the last matching number in the response. The model doesn’t even have to run again, because both filters grade the output it already produced.

Four of the six models change position, with Llama moving from second-last to third. phi-4’s score changes by almost sixty points, while qwen3-4b-instruct barely moves in accuracy at all.

In the logged responses, phi-4 emits #### only 13% of the time, while gemma-3-12b does so 88% of the time. Requiring that marker accounts for much of their difference in sensitivity to the grading rule.

If your application needs that exact format, you absolutely care whether the model supplies it. But if you’re choosing a model for arithmetic, how much of its apparent struggle is with writing ####? Check the flexible extractor’s passes, too. Finding the right number somewhere convenient isn’t enough to establish that the model answered the question correctly.

I’m glad the harness prints both. The trouble starts when one column becomes “GSM8K accuracy” with everything else stripped off, and you carry that number into a decision about which model to use.

Jo and colleagues examine this in Finding Answers in Thought Matters. They show extraction methods changing model rankings and inspect the output conventions responsible. They also examine incomplete reasoning caused by repetition, which is relevant to the unfinished responses I discuss below.

Fine, I’ll make it write the right format

That leaves an obvious question: could you make the model produce the format you need in the first place? Back on the custom suite, I tried llama.cpp’s grammars, which restrict the text a model is allowed to generate. If I made the model write the format my strict grader expected, would those answers start passing without another patch to the grader?

Will Kurt, Remi Louf, and Clémentine Fourrier had already tested structured generation as a way to reduce prompt sensitivity and ranking changes on GSM8K and GPQA. Their GSM8K constraint allowed reasoning before the answer, a detail that becomes rather relevant to my attempt below.

I used llama.cpp grammars to constrain answer shapes: brackets and separators for a list, for example, without supplying the values or the expected list length. For the formatting tasks, the generator read the expected answer to select separators and character classes, so it did use answer-key information about the format. Control and constrained runs used the same prompts on the same loaded server. These are 84 terse-mode tasks, excluding the code-execution tasks. Here, strict grading requires an exact final-line match, while tolerant grading accepts variations in spacing, punctuation, and surrounding text. Neither score includes a separate answer audit like the one in the opening.

For phi-4, Llama, and Qwen Coder, the successful task sets matched exactly between tolerant grading of the original output and strict grading of the constrained output. That supports the explanation that these particular failures were avoidable formatting failures. But I’ve changed generation too, and some models now solve different tasks.

qwen3-4b-instruct, for example, keeps the same total but swaps two task verdicts. For phi-4-reasoning, tolerant control and strict constrained scores both come out at 45.2%, but each condition solves 15 tasks the other fails. Look only at the totals, and you’ll miss that entirely.

Then I applied the grammar from the first generated token to models whose templates open a thinking block. qwen3-4b-thinking went from 98.8% under tolerant control grading to 0.0% under strict constrained grading. qwen3.6-35b-a3b went from 86.9% to 0.0%.

My grammar excluded the characters needed to close the thinking block, and neither model emitted a closing tag in any constrained response. I’d restricted the output before they could get to the part I wanted to restrict. Some kept going as long strings of digits and commas until the budget ran out.

And yes, I ran this before reading the relevant work. Tam and colleagues had already shown that format restrictions can degrade reasoning, and The Format Tax finds that separating reasoning from formatting recovers much of the loss in its tests. Reading first would have been a cheaper way to find that out. I haven’t run that separation here, so what I’ve established is where my particular constraint setup fails, and I can’t recommend it to someone using those thinking models, even though it recovered correct answers for the bare-answering ones.

What can you trust a comparison to tell you?

The useful comparison depends on how much of your application you’re willing to change to accommodate a model. If you need a drop-in replacement behind an existing API, the prompt, output schema, and budget are real requirements. “It works great if you change all of those” may be true and still be no help to you. Testing the models under those fixed conditions answers the question you actually have.

If you can adapt the application, give each model a comparable tuning budget, and check the selected setup on tasks you didn’t use for tuning. Pick a winner before doing that, and you risk selecting whichever model happened to like your first attempt at a prompt.

If you’re making a broader claim about model ability, show results across plausible configurations, including the ones that reverse your preferred ranking.

The lm-eval maintainers’ Lessons from the Trenches is a good reference for this: inspect outputs, document the setup, and be careful about comparisons across implementations. Here’s how I’d apply that advice to the failures I found:

• Keep the responses and their termination information. Inspect examples of wrong, correct, and unfinished output. In my custom suite, all 15 free-mode passes for phi-4-reasoning were truncated responses: the correct number survived at the end of a repetition loop. If the requirement is a completed answer, those are failures. An answer-string score alone concealed that distinction.
• Try the settings likely to change in actual use. Prompt mode, reasoning switch, generation budget, and output constraints belong next to the model name. Check each candidate. A good setting for one size does not establish a family default.
• Grade the stored output more than one defensible way. On the five-shot GSM8K task, the two extractor scores already provide a useful first check. Investigate the disagreements before interpreting either column as ability.
• Check whether the decision survives. If a reasonable configuration change reverses the winner, report the reversal. If it changes a score but leaves the decision intact, report that too. Your readers can use a result with known limits.

And use a public harness where it covers the workload. Seriously. I set out to test models and ended up maintaining rules about how to read commas and units. Public tooling still needs checking, as GSM8K demonstrates, but at least I can compare notes with other people using the same implementation.

If picking another shipped recipe changes the winner, you want to find that in the writeup, before you’ve built an application around the recommendation.

Scope and methodology

These are local runs on quantized open weights through a chat endpoint on the Strix Halo box. The GSM8K sample is 200 questions, not the full benchmark. The budget follow-up covers two models and two untuned shipped recipes. The separate extractor comparison uses the five-shot gsm8k task. These runs do not establish definitive model rankings or optimized scores.

The follow-up completed 800 requests at 1,024 tokens and 80 predetermined controls at 256, with temperature zero, seed 1234, and four concurrent requests. Codex reviewed the original and new answers, without blinding, and ambiguous or unresolved answers count as incorrect. The new logs retain the termination metadata that the original logs omitted. The 80 predetermined controls, replayed at the original 256-token limit, reproduced 78 pass/fail verdicts and 40 response strings exactly. After 679 requests, the thermal guard interrupted the run. I had left the automatic fan curve running instead of pinning the benchmark fans. The remaining 201 requests ran with full fans and unchanged inference settings. No throughput comparison is being made from this run.

My custom suite has 95 tasks run in two modes. The prompt and reasoning-switch results shown here use one run per condition, temperature zero, and a 4,096-token cap. Percentages and point changes are rounded separately from the underlying counts. That cap is part of the result, especially where reasoning fails to finish. I retained historical re-grading evidence for eight complete runs. The other runs were measured after the comparator fixes. The grammar experiment covers 84 terse tasks per model, and tests one constraint design.

I checked the tables against the stored results. The custom suite retained output tails, which support answer-extraction checks but not a complete audit of every generation. Generated source for the code-execution slice was not retained and cannot be re-graded.

If you’re shopping for a hybrid inverter you can actually drive from a script, this one qualifies now. And if you already own one, everything below is the map, the traps, and the working setup, so you don’t have to go find them yourself.

What the system looks like

The setup: solar array, two Eco-Worthy 48V/100Ah LFP packs in parallel (~10kWh, 200Ah at 48V), one Ampinvt TEL-48502M100 inverter/charger, and the rack it powers. All of it lives in the same shed in my back yard. The inverter has an LCD with numbers on it, which is handy right up until you want those numbers from indoors, or in a graph, or from ten minutes ago.

What it grew into is three processes on one small VM, I’ll swap it to a container later probably, talking over a local MQTT broker:

┌──────────────┐  Modbus-RTU   ┌──────────┐   MQTT    ┌──────────┐   REST   ┌───────────┐
│  inverter    │──────────────▶│  daemon  │──────────▶│   shim   │─────────▶│ dashboard │
│   (CH340T)   │ 9600 8N1 s1   │ pyserial │  topics   │ FastAPI  │ /status  │   rich    │
└──────────────┘               └──────────┘           └──────────┘          └───────────┘
                                     ▲                     ▲
┌──────────────┐  BLE / GATT   ┌──────────┐              MQTT
│  BMS packs   │──────────────▶│   ble    │──────────────▶ broker network
│ (JBD 0xFF00) │  notify 0xFF01│  poller  │  derived      (HA / Grafana)
└──────────────┘               └──────────┘  topics

• daemon owns the serial port. Polls registers, decodes scale and sentinels, publishes per-field MQTT topics, integrates the energy ledger and the coulomb counter, and is the only process allowed to write setpoints.
• ble poller owns the Bluetooth adapter. Reads each battery pack’s BMS, publishes per-pack telemetry, and runs the load-routing guard.
• shim subscribes to everything, caches the latest values, and serves them as JSON over HTTP for the dashboard and anything else that doesn’t speak MQTT.

Splitting the two pollers isn’t just for giggles. BLE is slow and flaky next to RS-485, and I didn’t want a Bluetooth timeout to be able to stall inverter polling, so they’re separate processes under separate units. The poller doesn’t write to the daemon’s state either - it subscribes to what the daemon publishes and puts its own derived topics back on the bus.

Talking to the inverter

The inverter has a USB-B port with an internal CH340T behind it, and it speaks bog-standard Modbus-RTU: function 0x03 to read holding registers, 0x06 to write one, 9600 8N1, slave address 1. That part takes ten minutes to establish.

Making it reliable took a lot longer, and comes down to three things the firmware cares about.

Chunk reads at 32 registers. Anything much over 32 registers times out. Not an error, not an exception response, just silence until the timeout expires. 88 fails, 32 is comfortable.

Leave ~50ms between requests. The firmware ignores follow-up reads that arrive too fast, and it does it silently. Enforce it inside the client instead of trusting callers, or someone eventually writes a tight loop and spends an afternoon wondering where half their reads went.

Pin the line flags off. rtscts, dsrdtr and xonxoff all explicitly False - the CH340 drops reads if pyserial so much as toggles RTS on open.

Each of those fails partially rather than cleanly, which is what made them expensive to find. If you’re pointing a stock Modbus library at this device and getting intermittent reads, it’s one of the three.

Above that, read_blocks() groups a sparse address set into ≤32-register runs and returns how many runs succeeded, so the caller can tell “the bus is down” from “here’s a snapshot” instead of an empty reading that looks like the real thing. One subtlety in the watchdog: a hung serial read blocks the poll thread while holding the lock, so the watchdog needs a way to close the FD without taking that lock, or it blocks on the thing it’s supposed to be rescuing.

Building the map

I scanned 0x0000 through 0xFFFF. Everything that responds lives in 0x0400 to 0x21xx, in three clusters:

There is nothing above 0x30FF. The SRNE family keeps daily PV/grid/load energy counters up at 0xF0xx; this box has no equivalent I could find, which quietly decides what you can and can’t measure for the rest of the project.

Finding addresses that respond is easy. Working out what they mean is the whole job, and only two approaches ever proved anything.

For settings: photograph the LCD. Walk every parameter screen [01] through [63], photograph each one, and value-match against a register dump. If LCD parameter [09] reads 58.4V and 0x1007 holds raw 146, you have the address and the scale from a single photo.

That’s also how the weirdest convention on this device falls out. Most values are raw × 0.1, but charge-curve setpoints are stored per-12V-cell at 0.1V resolution, so on a 48V (4×12V) config the scale is ×0.4:

raw 146 → 14.6 V/cell → 58.4 V system

Assume ×0.1 like everything else, get 14.6V, conclude it isn’t a voltage and move on. The photo is what breaks the tie.

For live values: the delta test. Change exactly one physical condition and watch which raw register moves. This is the only thing that counts as an identification.

The delta test is also what settles sign conventions, which on this device are not what you’d guess: battery_current is negative when charging (current into the pack), positive when discharging. Confirmed against an LCD showing an unsigned “12A” with the PV→Battery arrow lit while the register read -12.3A. A backwards sign in a control rule does the catastrophic opposite of what you intended, so this one lives in a comment, in the docs, and in the field definition itself.

The map is a list of frozen dataclasses - address, name, scale, unit, signed, sentinel, notes - and the decode is about as boring as it should be:

def decode(field: Field, raw: int) -> float | int | None:
    """Apply scale + sentinel handling. Returns None if raw is a sentinel."""
    sentinels = {field.sentinel} if field.sentinel is not None else DEFAULT_SENTINELS
    if raw in sentinels:
        return None
    if field.signed and raw >= 0x8000:
        raw = raw - 0x10000
    return raw * field.scale if field.scale != 1.0 else raw

Raw 4096 and 0xFFFF are the device’s “no value” sentinels and decode to None, which propagates as an empty MQTT payload instead of a plausible-looking zero. A zero meaning “no reading” and a zero meaning “no current” look identical on a dashboard, and only one of them should ruin your evening.

The notes field is the one that earns its keep. It carries the evidence for every identification: which LCD photo it matched, which delta test moved it, and what I had it confidently mislabeled as beforehand. It’s why I can still trust a field I confirmed two months ago and haven’t thought about since.

Don’t trust the 0x20xx block

This is the most useful thing to know before you point anything at this inverter. The 0x2000-0x212c region responds to reads and looks fantastic - a value that reads 8 when the load is light, one that reads 640W, a VA/W pair sitting adjacent at 0x2030/0x2031 exactly how other OEMs lay out apparent and active power, a battery SoC at 0x2049 that matched the app on the nose first time I looked.

Almost none of it is live. The numbers are plausible, stale, and in some cases just firmware defaults that never change. Every address in the known-dead list at the bottom of this post came out of that block, and I misidentified three of them before I worked out what was going on.

Two exceptions, and they’re the useful ones. 0x203c and 0x203d are live, and they’re the only place this inverter tells you where your load is actually running:

Across 771 settled samples - 610 on grid, 161 on battery, spanning three real transfers - the two value sets are completely disjoint. They lag the actual transfer by one poll, so allow a sample either side of a changeover before reading them. They hold those constants while battery power swings -1447→+1324 W and PV runs 0→1825 W, so they report state, not magnitude: compare against the constants, don’t read the number as a measurement.

Also worth knowing: 0x0509 is PV power in watts, not a DC bus voltage. It reads ~154 at 1540W, which makes a 48V→120V inverter look like it’s reporting a ~154V internal bus, and I had it mislabeled that way for weeks. It tracks pv_voltage × pv_current at r=1.000 over 118 distinct values and reads 0 whenever PV current is 0. There is no DC-bus-voltage register on this device.

If you want to check a candidate register yourself, the quick version is: log it every 20-30 seconds through a window where the real quantity is swinging hard - dawn recharge is ideal - then count how many distinct values it took. Live registers took 70-108 distinct values across 180 samples for me. The dead ones took one to four. There’s no gray zone, and it takes one morning.

A couple of cautions on that. A low count alone isn’t proof, because 0x203c above only ever takes two values and is perfectly live - if a register describes a state rather than a quantity, ask how many states the machine has. And a register that only changes when the inverter changes mode will look dead unless your window contains a mode change, which is how I had 0x204b pegged as the operating-state enum for two months before testing it across a real transfer.

Whatever you disprove, keep it written down with the evidence attached instead of deleting it:

Field(0x2127, "cand_pv_power_0x2127",    scale=10.0, unit="W",
      notes="STALE - frozen at 1700W across the 2026-05-25 capture "
            "(read 1700W at dawn with PV≈0). Use derived "
            "pv_voltage × pv_current instead."),

That note has stopped me re-investigating 0x2127 at least twice, because 1700 is such a believable number for PV power.

Deriving what the map doesn’t have

The direct consequence of the stale-cache block: the only trustworthy power numbers on this inverter are derived, not read.

pv_power      = pv_voltage      × pv_current
battery_power = battery_voltage × battery_current    # signed, negative = charging
load_balance  = pv_power + battery_power             # what the load is drawing

Everything downstream - grid offset, charge, discharge, the per-day ledger priced against my time-of-use rates - comes off those two products. Nothing reads a “power” register, because every power register on this device is a fiction.

There’s a second consequence: total grid usage is not measurable from this inverter. No AC-input power or energy register exists and the likely-looking candidates are all stale. The only honest grid quantity available is grid→battery charging, max(0, battery_charge - PV). True household grid draw needs an external CT clamp, and the software says so instead of inventing a self-sufficiency percentage.

Writing setpoints safely

Reading is free. Writing to a reverse-engineered address on a device rated for 5kW, with a lithium bank hanging off it, deserves more care - so every write goes through five gates. Three of them are boring and I’ll do them quickly:

• An allow-list. A field is unwritable until someone registers it explicitly, with bounds and a note saying where those bounds came from.
• Engineering-unit bounds. The API and CLI take volts, amps and percent, and conversion to raw happens once via the field’s own scale, so nobody hand-computes a ×0.4 at midnight.
• An audit log. Every attempt appends to JSONL with timestamp, initiator and before/after values, fsync‘d, because an audit line that only reached the page cache is not evidence. An unclean reboot right after a setpoint change is the moment you want to know what was written.

The other two are worth more words.

Cross-field rules are where the domain knowledge lives. float_voltage ≤ boost_voltage, so the bank is never parked at absorption. mains_to_battery_v > battery_to_mains_v, or the transfer thrashes. And a minimum band between the transfer and handback thresholds:

# A grid→battery handback threshold this close to the battery→grid threshold is
# crossed by IR rebound alone when the load drops away, so the transfer never
# settles. Measured 2026-07-11: the two sat 0.4V apart (one raw step) and the
# rack flipped ~95×/hour.
MIN_TRANSFER_BAND_V = 1.0

That 0.4V is smaller than the bank’s IR rebound when the load lifts, so grid takes the load, the voltage springs back over the handback point, and the load gets handed straight back. Ninety-five relay flips an hour is audible, which is a diagnostic technique I don’t recommend but which is how I first noticed.

Read-back verification is the gate I’d keep if you made me give up the other four, and it’s specific to reverse-engineered maps: Modbus 0x06 echoes the frame back whether or not the address is real. The inverter will happily “accept” a write to a register it doesn’t own, with a well-formed echo and a valid CRC, and the value simply doesn’t stick. The echo proves nothing, so every write reads itself back:

readback = self.read_holding(address, 1)[0]
if readback != value:
    raise ModbusError(
        f"write verify failed at 0x{address:04x}: "
        f"wrote {value} (0x{value:04x}), read back {readback} (0x{readback:04x})"
    )

Measuring charge state: coulomb counting

Everything else in this system is shaped by one physical fact: LFP has no voltage curve to speak of. From roughly 20% to 90% state of charge the cell just sits on a plateau. I measured it under a full-rack ~26A draw and the bank held at 51.8V, dead flat, for nine straight minutes while true per-pack SoC fell from 44% to 42%. Two percent of the bank drained and the terminal voltage did not move!

Which makes any voltage-derived SoC fiction across the entire range you care about, and the inverter’s single SoC register is voltage-derived. The only honest way to get the number is to integrate current - amp-hours in, amp-hours out, against a known capacity. That’s soc.py.

def _integrate_locked(self, battery_current: float, gap_s: float) -> None:
    # negative current = charging (into pack); positive = discharging.
    amp_hours = -(battery_current + self.parasitic_a) * (gap_s / SECONDS_PER_HOUR)
    if amp_hours >= 0:  # charging: derate by coulombic efficiency
        self._charge_ah += amp_hours * self.charge_efficiency
    else:               # discharging: full coulombs leave
        self._charge_ah += amp_hours

A raw integrator drifts forever, so it needs propping up.

Anchor it on physics. When the bank reaches float voltage (≥56.4V) and current has tapered into the CV-phase tail (<5A) and both hold for 10 minutes, the bank is full by definition, so snap to 100%. Standard LFP recalibration, fires on every real full charge.

Make that anchor condition something you continuously observe. If samples stop - daemon restart, inverter dropout - the “at full” hold clock has to reset, or an outage that happens to bridge one high-voltage sample counts the entire blind period as “held at full” and fires the anchor on a half-empty bank the moment samples resume. Took me a while to work out why the counter kept jumping to 100% after restarts. Same applies to the integration itself: gaps longer than max_gap get dropped rather than integrated against stale current. I’d rather under-count than invent amp-hours.

Meter what you’d otherwise clamp. The count clamps at capacity, obviously, but a bank that keeps absorbing charge after you’ve declared it 100% was never full. So instead of throwing that away:

# The clamp would bin the evidence that we were never actually full.
if self._charge_ah > self.capacity_ah:
    self._overflow_ah += self._charge_ah - self.capacity_ah

That meter paid for itself almost immediately. It caught the bank absorbing 51.2Ah over 16 hours past a verified anchor, then 65.3Ah over 21.4 hours on the next cycle. A healthy 200Ah bank cannot do that. Two cycles, one division:

51.2 Ah / 16.0 h  = 3.20 A
65.3 Ah / 21.4 h  = 3.05 A

Roughly 3.1A that the battery_current register never sees, because it’s the inverter’s own DC-bus self-consumption - about 160W of overhead sitting upstream of the shunt. Left unmetered it was inflating SoC by 40-50Ah a day, a quarter of the bank, which explains a lot of the dusk deaths that happened at a comfortable-looking “23%”. It’s a config constant now, derating charge and inflating discharge during integration. The estimator also refuses to claim a number it hasn’t earned: while it’s carrying unexplained overflow it caps itself at 99.5%, and only a real float-and-taper anchor clears the books.

It publishes battery_soc_last_full so the dashboard can show when the count was last anchored, and “never” means it’s still running on its initial seed and you shouldn’t believe it yet. A bare percentage looks equally confident whether it was anchored an hour ago or never, which is how you end up trusting one that’s been drifting since Tuesday.

Ground truth: BLE into the BMS

Counting coulombs at the bank level has one blind spot. A single measurement sees two parallel packs as one, and on the flat part of the LFP curve two packs at identical terminal voltage can be holding quite different amounts of charge. The weaker one’s BMS cuts out first and takes the rack with it, and none of that is visible from any register on the Modbus bus.

The packs know, though. Each has a BMS, and the BMS talks Bluetooth, because that’s how the phone app works. So: go around the inverter entirely. A TP-Link UB500 dongle (2357:0604, RTL8761BU) went straight into the Proxmox host and got passed through to the monitoring VM. The one gotcha is a spectacularly unhelpful error: the VM is minimal and has no linux-firmware, so hci0 refused to come up with rtl8761bu_fw.bin not found, error -2. Rather than drag a multi-gigabyte firmware package onto a small VM, drop in the two files it actually wants - rtl8761bu_fw.bin (44 KB) and rtl8761bu_config.bin (6 bytes, yes really, six) - reload btusb, and you have a controller.

A GATT dump confirmed these are JBD/Xiaoxiang boards, the most common Chinese LFP BMS family and well enough documented by the community to work from. Dump the table, don’t trust the brand on the label - Eco-Worthy is a case around somebody else’s board, and whose board it is turns out to be the only question that matters.

After Modbus-with-caveats it’s a genuinely pleasant protocol. Service 0xFF00, write commands to 0xFF02, replies arrive as notifications on 0xFF01. Same frame shape both directions:

DD A5      77

Two commands cover everything worth having: DDA50300FFFD77 for basic info and DDA50400FFFC77 for per-cell millivolts. The checksum is 0x10000 - sum(cmd, len), and len is 0 for reads.

Replies arrive chunked across MTU-sized notifications, so you buffer and ask “is that a whole frame yet” after each one. Check the declared length byte and not just the trailing 0x77, or the first payload containing a stray 0x77 will hand you a truncated frame that decodes perfectly and means nothing:

def frame_complete(buf: bytes) -> bool:
    if len(buf) < 7 or buf[0] != 0xDD:
        return False
    payload_len = buf[3]
    return len(buf) >= 4 + payload_len + 3 and buf[-1] == 0x77

The basic-info decode is the prize. SoC, true pack current, residual capacity, cycles, protection flags, FET state and temperatures, all out of one 0x03 reply:

def decode_basic(frame: bytes) -> dict:
    d = _validate(frame, 0x03)
    ntc = d[22]
    temps = [
        (struct.unpack(">H", d[23 + 2*i:25 + 2*i])[0] - 2731) / 10.0
        for i in range(ntc)
    ]
    return {
        "voltage":    struct.unpack(">H", d[0:2])[0] / 100.0,
        "current":    struct.unpack(">h", d[2:4])[0] / 100.0,   # signed, +charge
        "resid_ah":   struct.unpack(">H", d[4:6])[0] / 100.0,
        "nominal_ah": struct.unpack(">H", d[6:8])[0] / 100.0,
        "cycles":     struct.unpack(">H", d[8:10])[0],
        "protection": struct.unpack(">H", d[16:18])[0],
        "soc_pct":    d[19],
        "fet":        d[20],
        "ncell":      d[21],
        "temps_c":    temps,
    }

Temperatures are decikelvin, hence the 2731. SoC is a single byte at offset 19. Cell voltages come back from 0x04 as a big-endian u16 of millivolts per cell, which gets you cell spread for free - a useful early warning that a pack is coming apart internally.

One trap when you bolt this onto something that already exists: the BMS says positive is charging, the inverter says negative is charging. Same current, same bank, opposite signs. Rather than rely on remembering that at every call site, the reading object carries an explicit current_inverter_convention() that re-signs it, and the convention is written on the dataclass where it can’t be missed.

The frame decoders are pure functions with bleak deferred into the one async call that needs it, so the whole protocol layer unit-tests against captured frames on a machine with no Bluetooth stack at all.

Fusing the two signals

Per-pack telemetry gets you two things neither source produces alone.

The imbalance floor is min(pack_socs), not the average. The weak pack is the one that cuts out, so the weak pack is the number the system runs on. It publishes the spread too and warns above 10 points, because two packs on the same bus drifting apart is something you want to hear about early rather than at dusk.

The parasitic cross-check falls out for free. The BMS measures true current at the pack terminals; the inverter register can’t see the inverter’s own bus draw. Subtract one from the other and there it is. First time I ran it the register said -12.7A “charging” while the BMS said -10.0A at the terminals, and that 2.7A gap is the same self-consumption I’d fitted weeks earlier from overflow accumulation - completely different data, agreeing to within half an amp. I was quite pleased with that one!

It runs continuously now and warns if the implied value drifts more than 1.5A from the configured constant, so a change in the inverter’s idle draw shows up as a log line rather than as mysterious SoC drift three weeks later. Gated on |current| ≥ 3A, because at rest you’re dividing noise by noise.

Closing the loop: routing the load

With a trustworthy charge number, the last piece is acting on it: keeping the load on the battery while there’s charge, and moving it to grid before the bank gets low enough for a pack to cut out.

The obvious approach is to set the inverter’s own thresholds and let its routing logic get on with it. I spent weeks on that. It does not work on this hardware, and not for want of trying values:

So instead of nudging thresholds and hoping, drive the load source directly, from the signal that knows the charge state, using the control that moves the load immediately: supply_priority_mode, the UTI/SBU/SOL enum at 0x1102.

The decision itself is a pure function, which makes it trivial to test every path:

def decide(state, soc_pct, soc_age_s, cfg) -> str:
    """Next state, given the current one and the freshest SoC we have."""
    trusted = soc_pct is not None and soc_age_s <= cfg.stale_seconds

    if not trusted:
        # Cannot see the battery. Never switch back to it; if we were already on
        # mains, stay there. On first run with no reading, assume the worst.
        return GRID if state in (None, GRID) else state

    if soc_pct <= cfg.floor_pct:
        return GRID
    if soc_pct >= cfg.resume_pct:
        return BATTERY
    # Between the rails: hold.
    return state or BATTERY

Weakest-pack SoC at or below the floor (35%) writes mode 1, UTI: load runs on mains, battery stops draining, PV keeps refilling it. At or above resume (55%) it writes mode 2, SBU, and the load goes back to solar and battery. The config refuses to start if resume ≤ floor, because a guard with no hysteresis will just sit there flapping the relay all evening.

What makes it safe to leave running unattended is that every failure mode resolves toward grid. Grid is the backstop; the battery is the thing I can destroy. Stale SoC, dead BLE, first run with no reading - all route to mains and stay there, and it will never switch back to a battery it can’t currently measure. It writes through the same SafeWriter as everything else, so an automated actor gets no privileged bypass, and it reads back the live mode first, so a landed write goes quiet while a rejected one keeps retrying in the audit log instead of just sitting there doing nothing.

The switch moves the load in about 15 seconds, against the minute-long lag the voltage levers gave me. In service it cycles battery→grid around 35% and back around 55% through each dusk.

There are two easy ways to get the verification wrong, and I found both of them:

Working out which source is carrying the load needs an energy balance, not the charge sign. In daylight the bank charges whether the load is on grid or on battery, because PV surplus spills in either direction, so battery_current < 0 tells you precisely nothing. I burned about three hours on this and confidently reported a “hysteresis works” result that was entirely an artifact of it. What does discriminate, at any hour:

load + parasitic ≈ 475 W       # measured at dawn: PV ≈ 0, bank supplies everything
implied_mains = bank_charge_W + 475 - PV_W
                                # ≈0 ⇒ on battery;  >250 ⇒ on grid

The software coulomb counter is not in the routing path, and that was a deliberate demotion. The guard gates on min(pack_socs) from the BMSes and nothing else. My estimator is a model integrating a register that’s missing 3.1A, leaning on a float anchor that might not have fired in days; each JBD pack meanwhile does its own hardware coulomb count against its own shunt. Where those disagree the hardware wins, and it took a bad night to accept that - the estimator once read 33% while both packs sat at ~76%, and dragged the guard into a pointless grid transfer at dusk on a three-quarters-full bank. It’s a fine dashboard number. It has no business steering a contactor.

How it’s deployed

Three systemd units on a Rocky 9 VM with the CH340 and the BT dongle passed through: solar-monitor-daemon owns the serial port and everything derived from it, solar-monitor-ble owns the Bluetooth adapter and the routing guard, and solar-monitor-shim turns MQTT into REST for the dashboard. The BLE unit runs as root because BlueZ’s D-Bus policy denies org.bluez to everyone else. Both pollers are Type=notify with a systemd watchdog, and the daemon carries its own on top for the hung-serial-read case.

Couple of things worth stealing. A udev rule pins the inverter to /dev/solar-inverter and kills USB autosuspend for the CH340, because a suspended adapter looks just like a dead inverter and I only wanted to have that particular afternoon once. And the BLE poller must not inherit the shared mqtt.client_id - two clients with the same id collide on the broker, kick each other in a reconnect storm, and take the daemon’s publishing down along with the poller’s.

Config is one YAML file; the only values you’d have to think about are capacity_ah (must match the real bank or the percentage is meaningless), parasitic_a, the float-anchor pair full_voltage/taper_current_a, and the guard’s floor_pct/resume_pct.

The full register map

Everything below came off the live device. Bus is Modbus-RTU, function 0x03/0x06, 9600 8N1, slave 1, ≤32-register block reads, ≥50ms between requests. Raw 4096 (0x1000) and 0xFFFF are “no value” sentinels.

Live readings - these track reality and cross-check against the LCD:

Settings - every row value-matched against a photograph of the corresponding LCD parameter. Charge-curve voltages use the ×0.4 per-12V-cell scale:

Known-dead, published so nobody re-investigates them: 0x203f (looks like load %), 0x204c and 0x2031 (look like load watts), 0x2030 (looks like VA), 0x2127 and 0x2125 (look like PV), 0x2049 and 0x2034 (look like SoC), 0x111f (looks like a second SoC - reads 40% on a near-empty bank), 0x203e (looks like AC-in volts), 0x2121, 0x2123, 0x2037, 0x2044, 0x204f. All frozen across long captures, including across real grid↔battery load transfers.

0x204b is dead, and it’s the one I’d most expected to be alive. An operating-state enum observed at 5, with an SRNE analog, plausibly “on grid / on battery / standby” - the one bit of routing visibility this map otherwise lacks. It sits at 5 across real grid↔battery transfers in both directions, so whatever it is, it isn’t that. The load-source signal is at 0x203c/0x203d instead.

Still open: 0x2116 and 0x2041, candidate temperatures. They stay flat for a legitimate reason - temperatures move slowly and nothing thermal happened while I was polling - so they need a real thermal swing before anyone can say anything about them. And 0x0510, which is unambiguously live (142 distinct values) but whose best correlate is battery power at r=-0.876 with a ±40% ratio spread. A ±40% spread is what made me throw out 0x203c the first time, and I’d rather not repeat that particular mistake, so it stays unnamed until PV and charge power drift far enough apart to tell them apart.

Limits worth knowing before you start

• ac_charge_current [28] is DC-side, not AC draw. The manual calls it “Maximum AC charging current.” It’s the DC current into the battery, so 15A DC is roughly 7.5A AC at the wall, and it does not cap total AC draw - the inverter still pulls whatever the load needs on top of it. Worth knowing before you use it as a budget for anything.
• The coulomb estimator has one open issue. Its float anchor needs the bank to reach 56.4V with tapered current, and under the SoC-guard regime the bank cycles roughly 35-80% and rarely gets there, so last_full can sit at “never” and the count free-drifts. It’s decoupled from routing and the dashboard flags the low confidence, but the energy ledger leans on it. On the list.
• Which lever governs depends on whether BMS comms are live. If the battery is talking to the inverter, switch_to_mains_soc is the real governor and the voltage thresholds do nothing - I held battery_to_mains_v a full volt above the bank for three minutes and the load never moved. With no BMS link it falls back to voltage, which on LFP means it fires far too late. Check which regime you’re in before tuning anything, because the same setpoint does opposite amounts of nothing in each.
• The BLE side is JBD-generic. If your pack answers on service 0xFF00, the decoder above should work as-is - but dump the GATT table and confirm before you trust the badge on the case.

The code that drives all this isn’t public - it’s a personal thing held together with optimism and systemd units, and it assumes my exact bank and my exact power rates. But the map is all up there, dead registers included. If you’ve got the same box, that should be enough to skip most of what I went through.

So here we go, a list of failures sorted by how they failed, because on this hardware the how is the useful part.

The failures modes fall into three buckets:

• Type 1, it ran and was slower or worse.
• Type 2, it silently did nothing, which is the failure mode that ate the most days.
• Type 3, it worked and I measured it wrong.

Reminder, because it drives everything below: this box is not bandwidth-starved in the abstract, it is bandwidth-pinned. I measured decode reading memory at ~95% of the read ceiling (242.7 of 256 GB/s). We cannot stream faster. So every real win has to move fewer bytes, and most of what follows is me learning that (over and over again) the hard way.

Type 1: it ran, and it was worse

The first entry is a bit of a cheat, it never even got to run, but it’s the wall the whole project is pinned against, so it goes first.

Memory and fabric overclock: architecturally locked

The obvious first move on a bandwidth-pinned box is to raise the memory clock. You can’t. Infinity Fabric on this part is rate-matched to the memory clock, there’s no independent FCLK knob, and memory frequency is SKU-binned at 8000 (AMD ships 8533 as separate part numbers). I went as far as the AMI/AGESA BIOS unlock that others have tried on 395 boxes; it fails, and recovery needs a hardware SPI programmer. The clocks were already pinned at max, not thermally throttled. Revive this only with different silicon, an 8533 SKU or an LPDDR6 next-gen part. On this box it’s closed.

rocWMMA flash-attention regresses past 32k

The community-tuned ROCm numbers looked great, so I chased a ROCm + rocWMMA flash-attention build. It helps at short context and, by community guidance and my own A/B against RADV, gives ground back at depth past ~32k, which is exactly the range this whole project cares about. The strixhalo.wiki maintainers now say it outright: as of ROCm 7.0.2+ the rocWMMA path shouldn’t be used with upstream llama.cpp, it’s slower than the plain ROCm/HIP path as context grows. RADV/Vulkan was the faster path at depth in my runs, so the ROCm detour was a net loss for the long-context case. The tuned 13.3 tok/s-at-130k number floating around comes from a patched branch (lhl’s Strix Halo benchmarking), not a stock build, which is worth knowing before you go replicate it.

MALL residency is too small to matter

gfx1151 has a 32MB L3/MALL that is genuinely strong when your working set fits, 3.4-9.6x DRAM bandwidth. A 35B MoE’s working set does not fit in 32MB. Measured, cache-residency bought about 1.1x on decode, and it’s conditional and it costs you recall to arrange. The cache is real and the win is not, for this model. This could be fun to revisit with smaller models, however! I really wanted to find a way to make this one work out.

KV-cache quant tax

q4_0 KV quant hurts prefill at mid context on this box (131k: 393 tok/s vs f16’s 473, a 0.83x loss), because the dequant overhead outweighs the bandwidth it saves at that depth. It flips to a win only at extreme context (256k: q4_0 240 vs f16 223, 1.08x). And once you’re running the sliding-window recipe, quant loses again (256k: q4_0 767 vs SWA-f16 855, 0.90x), because the window already bounded the KV to a size where there’s no bandwidth pressure left for quant to feed on. SWA and KV-quant are substitutes, not complements, you pick the bigger lever, you don’t stack them. I spent… a while trying to stack them.

Type 2: it silently did nothing

This is the long one, and the pile that cost me the most days, because a silent no-op is the only failure that doesn’t have the decency to announce itself. The run finishes, nothing errors, and the numbers come back looking exactly as plausible as a real result. You can burn a whole experiment on one and not know until you go digging.

The “content-aware” sparse mask was never content-aware

The tell, in hindsight, is embarrassingly clean. I had a tau threshold knob on the selector, and sweeping it across its entire range (0.01 to 0.99) moved neither prefill speed nor perplexity by a hair. At a 16k window every setting landed on ~753 tok/s and PPL 1.3512; at a 2k window every setting landed on ~903 tok/s. The absolute numbers move with the window, but tau itself was inert at both. A knob that changes nothing isn’t tuning anything, so my content selector was selecting no content. The reason is a softmax over all blocks peaked so hard on recent tokens that far-back salient blocks never clear any threshold worth setting, which quietly collapsed the whole thing into a plain sink-plus-sliding-window. I chased that for a week before I caught it, and the full autopsy (the mechanism, the max-pool swap that changed nothing, the DeltaNet layers that were actually carrying the needle) is in the 256k post. The Type 2 lesson is small and I won’t forget it again: if a feature has an off switch, run it. If “off” looks identical to “on,” it was never on.

The ggml step() bug: the mask that never masked

This one lived underneath the last one, and it came first: before my mask was even a lazy sliding window, it was nothing at all. It leaned on ggml’s step(), and step(0) disagrees with itself across backends: 0 on CPU, 1 on Vulkan. On the GPU path that meant my mask never dropped a tile, flash-attention never skipped anything, and I ran a whole recall curve that was silently dense. I chased it down, filed it as issue #25027 (accepted and fixed upstream), and wrote the whole saga up separately, so I’ll spare you the forensics here. It earns a mention because it’s the purest Type 2 specimen I have: my code was correct, the framework disagreed with itself at exactly one input value, and the only symptom was “the optimization does nothing,” with no error anywhere.

The additive-mask hack was O(N^2) in bookkeeping

This one ran, and ran correctly, which is exactly why it lands in this bucket: the mask masked, the outputs were right, and the speedup it was supposed to buy quietly never showed up. The tell was that it kept eroding the deeper I went. A sliding window applied as an additive -inf mask still makes flash-attention walk every query-tile / key-tile pair and pay a per-tile check on the all-masked tiles it then throws away: O(N) in real attention work, O(N^2) in tile bookkeeping, and at depth the bookkeeping is most of the bill. The 256k post has the ladder; the lesson for this list is that “it’s masked” and “it’s cheap” are separate claims, and a mask riding on top of a dense kernel only ever buys you the first one. The actual win needed native sliding-window attention with a bounded rolling KV cache (flat 850-890 tok/s from 32k to 1M, a 32x range that sags 4.8%), which is O(N) for real. The mask was a scaffold that looked like the answer.

Env-gated patches silently no-op on the wrong binary

This is the one I’d carve into the case. HOMER has two llama-server binaries: the system one at /usr/local/bin (no patches) and my build-vk/bin build (the SWA and per-layer-quant patches). My patches are environment-gated, so on the unpatched binary they don’t error, they just quietly do nothing, n_swa stays 0 and the model runs dense. I ran a windowing sweep against the system binary once and got a clean, plausible, completely dense curve, and nothing in the output said a word. The only defense is behavioral: an out-of-window needle must miss, and if it doesn’t, either you’re on the wrong binary or the flag never took. I hit this same wall on the 256k work too, which is why “it ran” stopped counting as “it ran patched” for me. Every grid now starts with a needle the feature is supposed to drop, and if it survives, I throw the grid out before I read a single number.

Kernel-level knobs that measured to nothing

A cluster of small ones, grouped because they share a shape: I turned them on, measured, and found no signal. Transparent hugepages, a measured no-op (the GART already uses large pages). amd_iommu=off, neutral, because iommu=pt overrides it and there was near-zero headroom anyway. Borrowing gfx1100 kernels via HSA_OVERRIDE_GFX_VERSION=11.0.0 looked incredible on a GEMM microbench (5.6x) and produced no end-to-end LLM win I could measure. Microbenchmark speedups that don’t survive contact with a real forward pass are their own kind of silent nothing.

Type 3: it worked, I read it wrong

The NIAH “5/5” that was measuring thinking budget

qwen3.6 is a reasoning model, and for a while I measured needle-in-a-haystack recall as “did it answer within the -n token budget,” not “could it retrieve.” Those aren’t the same thing. A too-short budget makes the model look like it failed to recall when it just ran out of room to think, and a generous one flatters recall for reasons that have nothing to do with the retrieval mechanism I thought I was testing. Several of my early clean 5/5 numbers were contaminated by this, including a “win4096 5/5 but +74% PPL” that I’d been treating as a real recall win. I now measure recall with greedy decode and a stop token, deterministic and exact, and I re-ran the suspect numbers.

The temp-0 degeneracy attractor that fooled me three times

The single most expensive misread of the project. At temp-0 greedy decode, deep-context generation on this model falls into a degeneracy attractor, it starts emitting bare “3” or single-character garbage, and it does this non-monotonically: a keep-budget sweep goes hit / miss / hit / miss with no pattern, and the baseline itself flips run to run. I read this as “eviction is dropping the needle” at least three separate times before pinning it. It is not an eviction failure, the scorer keeps the needle at global rank #1; it is a decode-side collapse from attention-sink deprivation, and bumping temperature to 0.3 partially stabilizes it. I burned days attributing a generation-side attractor to my byte-reduction code because the symptom (wrong answer at depth) looked identical to the thing I was afraid of.

The NPU that was the right answer to the wrong model

Not a mistake so much as a scope misread. The NPU’s flat-prefill win is real, I measured it, on a dense proxy model (Qwen3-8B), where dense attention collapses with context and the NPU’s steady prefill overtakes the iGPU around 16-32k. My actual hero model is an MoE whose iGPU prefill is robust and defers that collapse past the range I care about, so the crossover never appears and the NPU stops being a lever. The measurement was right; treating it as a general win for this model was the error. But turn that around, because it’s not a dead end so much as an open door I walked past. On a dense long-context model the crossover is real: the NPU genuinely overtakes the iGPU on prefill past ~16-32k, and the only reason I didn’t chase it further is that my hero model defers the collapse the NPU was going to rescue me from. The stack is already standing on HOMER. So if you’re running a dense model at depth on one of these boxes, this is the thread I’d pull, and I’d love to hear where it lands, because I think there’s a real win sitting there that I just wasn’t the right person to find.

The pattern

The Strix box is pinned at the bandwidth wall, so saving bytes is the only thing that helps, and I spent most of these dead-ends discovering that a thing I thought was saving bytes wasn’t doing anything at all.

• A mask that masked nothing.
• A patch on the wrong binary.
• A kernel knob that microbenched beautifully and moved no real tokens.

The dangerous failures aren’t the slow ones, they’re the ones where the run completes, the numbers look plausible, and you’re one behavioral check away from finding out you measured a no-op. The single habit that came out of all of this: before trusting any result, prove the thing you think you turned on is the thing that ran. An out-of-window needle has to miss. Or you’re measuring dense and calling it a win.

Where the last post left off

In the 256K post I got usable long context out of my Strix Halo box by marking the hybrid’s ten full-attention layers sliding-window, keeping a few of them full as a far-recall backstop. Bounded KV, flat prefill, decode that holds up at depth. That post deliberately left one thread on the table: whether the other decode lever still works on top of it. I mean one win is cool but two is better, right?

That other lever is MTP - multi-token prediction, llama.cpp’s self-speculative decode - which I previously got working on this box, then gave bad advice about, then apologized for. The short version of those two posts: MTP drafts a few tokens ahead with a lightweight head and verifies them in one pass, the payoff is governed by the Leviathan et al. formula, and whether it helps depends on the acceptance rate α, the draft depth γ, and the platform cost ratio c. On this box, with the right --spec-draft-n-max, it’s a real win.

So: SWA bounds the KV and keeps decode fast at depth. MTP amortizes the weight reads. They attack different costs so they should stack, you’d think. You can see where this is going.

(A confession: if you applied the full fork patch from the 256K post, or read its diff unusually closely, you may have noticed a small MTP fix sitting in there that the post never mentioned. That was this. It shipped quietly because it belonged to a post I hadn’t written yet - this one - and because I was too lazy to rip it out just for that post lol)

The crash

They did not stack. They didn’t even underperform together, which would have at least been interesting data. Turning both on aborted at load:

GGML_ASSERT(hparams.swa_type == LLAMA_SWA_TYPE_NONE
            && "Use llama_kv_cache_iswa for SWA") failed
  src/llama-graph.cpp:2704, via llama_model_qwen35moe::build_arch_graph

Well that is annoying. Here I finally remembered to do the smart thing: google before hand-patching. The search turned up issue #23322 (people combining SWA and MTP on the Qwen3.6 family, hitting a different problem) and a pair of upstream iSWA crash fixes (#24294, #23131) that my fork’s base predated. Which raised an awkward possibility: maybe upstream had already fixed my crash while I wasn’t looking.

Llama.cpp moves FAST

My fork was 638 commits behind master… not ideal. There was no way to find out whether the crash was already fixed except to pay the debt, so I rebased the whole patch set onto current upstream - through an API migration that had renamed or restructured basically everything my patches touched. Two of my patches turned out to be obsolete and got dropped outright: the original env-gated sparse-mask hack (superseded by the SWA recipe it grew into) and my old MTP graph routing (upstream had refactored MTP properly in #23643).

The answer, after all that: no. Current upstream, same crash. The two upstream iSWA fixes were real but covered different holes. This one was still my problem, boo.

The actual fix

The bug is a divergence between two graphs that are supposed to agree. When the SWA recipe is on, the main graph correctly routes attention through the iSWA hybrid cache. But MTP builds its own little sub-graph for the draft head, and that sub-graph built its attention input with the non-iSWA builder - the one that opens with the assert above. The main graph and the MTP graph disagreed about what kind of cache the model was running on, and the assert is what disagreement looks like.

The fix mirrors the main graph: when swa_type != NONE, build the MTP sub-graph’s attention input through the iSWA builder too. The MTP layer’s index isn’t marked as a sliding-window layer, so it lands on the full-attention sub-cache - the draft head attends the full KV, which is what you want from the thing whose guesses get verified. The fix is tiny and the non-SWA path is byte-identical to before.

Best case: super-additive at 123K

With both levers finally running in the same process, the 2x2 at ~123K context (predictable log-continuation generation, greedy, q4_1 KV; full config in the caveats):

Separately the levers buy +14% and +11%. Together they buy +46%, well past their sum, and the acceptance column says why. Under SWA, the draft head’s guesses went from being accepted 80% of the time to 96% of the time. SWA isn’t just contributing its own speedup next to MTP’s - it’s making MTP better at its job.

(Prefill note: MTP costs you some prefill wherever it runs, 7-15% in this 2x2 - the draft head is extra graph. SWA-only is the prefill champion. This post is about decode.)

Why SWA lifts acceptance

The mistakes post was all about the denominator of the Leviathan formula - the cost ratio c that the hardware sets, the thing I’d ignored while staring at acceptance. This result is the numerator’s revenge: same box, same c, and the speedup moved anyway, because SWA moved α.

The mechanism is almost embarrassingly simple once you see it. The draft head is small and shallow: its whole job is guessing what the big model will say next. Bound the big model’s effective context with a sliding window and you’ve lowered its next-token entropy - it has less history to condition on, so it becomes more predictable. A more predictable target is an easier target to draft for.

The depth behavior fits: at 20K context, dense acceptance is already ~0.88 (short contexts are easy to draft for), and MTP+SWA (70.3 t/s) just tracks MTP-only (73.8) - no lift, but no harm either. The lift only appears where the dense model’s conditioning gets long and the window’s pruning of it starts to matter.

It also matches something the long-context speculative-decoding literature already knows from the other direction. MagicDec (Together AI’s writeup) attacks long-context decode by giving the draft a StreamingLLM-style window while the target stays full-attention, and reports that a windowed draft holds high acceptance out to 100K: windowed and full-context distributions agree on most tokens. Mine is the mirror image. Here the target is the windowed one, because bounding its KV is the whole point of the SWA recipe, and the acceptance gain lands on the stock MTP head as a side effect.

The lift showed up everywhere I measured, in the same direction. Not in the same size though, and that’s where the good news stops.

Forcing the worst case

That 0.96 acceptance came from continuing a repetitive maintenance log - about the most predictable generation task that exists. Of course the draft head aced it! This series has burned me enough times that an exciting best-case number now triggers a panic reflex instead of a victory lap: go find the workload that hates it.

The test: a diverse ~79K context (combinatorial prose, nothing repeating) and a genuinely open-ended analytical generation task, with generation diversity measured on both sides so I could confirm the two configs were doing comparably hard work rather than one of them quietly degenerating into an easy loop. Same 2x2:

On hard generation, MTP does not pay off. Not dense, and not with SWA either. The acceptance lift is still there (0.51 to 0.55, same direction as always) but it’s small, and 0.55 is nowhere near the ~0.8 acceptance this box needs before the draft-and-verify overhead nets out ahead. SWA shrank MTP’s loss from -11% to -2.6%. Shrinking a loss is nice but not exactly much of a win.

SWA lifts MTP acceptance everywhere - the mechanism is real - but the size of the lift tracks how predictable the generation already was: 0.80 to 0.96 on the log, 0.51 to 0.55 on the analysis. SWA lowers the predictability threshold at which MTP breaks even. It does not make MTP universal, because nothing does - the acceptance gate never goes away, and your workload is what sets it.

Meanwhile, look at the SWA-only rows in both tables. +14% at 123K, +8.6% at 79K, no acceptance column, no conditions. That’s the lever that doesn’t care what you’re generating.

The rule

• Take the SWA recipe’s decode win always. It’s bounded-KV physics, generation-agnostic, and it’s the reason the 256K post exists.
• Add MTP on top only when the generation is predictable or structured: code, formatted continuation, extraction, log-like output. There it’s spectacular: +46% at 123K.
• Leave MTP off for open-ended, creative, or analytical generation, even with SWA. It was net-negative dense and it’s still net-negative stacked; SWA just made it hurt less.
• If you only remember one thing: acceptance is workload, cost ratio is hardware, and you need both on your side.

Getting the patch

As confessed at the top: if you’re running the full fork patch from the 256K post, you’ve had this fix all along. It’s also available standalone as mtp-iswa.patch; it applies clean to current upstream.

I’m not sending this one upstream, at least for now, because stock Qwen3.6 GGUFs don’t set swa_type on this arch, so a vanilla model never hits this assert. You need a downstream SWA config, like mine, to reach the path. That makes it a latent-inconsistency fix - the MTP sub-graph diverges from the main graph in exactly the handling the assert guards - rather than a crash any stock user is living with today, and “apply my SWA patch first” is a weak reproduction story for a maintainer to review against.

If you’re stacking SWA on this family some other way and hit the assert, the patch is right there. And if anyone feels like carrying it through the upstream contribution process, PLEASE be my guest - the diff is small, the reasoning is in this post, and I’d be happy to see it land. (Related upstream: #23322 is the same feature combination but a different problem - a runtime acceptance issue, not this build crash.)

Caveats

• One model, one architecture. Everything here is Qwen3.6-35B-A3B, the qwen35moe hybrid (10 full-attention GQA + 30 Gated-DeltaNet layers), UD-Q4_K_M with the bundled MTP head. The crash and fix are specific to that arch’s MTP graph. The acceptance-lift mechanism (a bounded context lowers the target’s entropy, which makes drafting easier) has no model-specific step in the argument, but I’ve measured it exactly once, on one family. Treat the direction as an argument and the magnitudes as this box, this model.
• Config, for reproducibility: Vulkan/RADV, --dynsparse-swa 16384 --dynsparse-swa-full 27,31,35,39, q4_1 KV, -ub 512, greedy decode, --spec-type draft-mtp. The SWA flags are from my fork (patches shipped with the 256K post); everything else is stock.
• The worst-case comparison is the diversity-matched one. I actually ran two hard tasks at 79K, and in the other one the two configs went different directions off the same prompt: the SWA run kept continuing the document it was given (word-diversity 0.29, i.e. it settled into something predictable) while the dense run veered off into meta-reasoning about the task (0.68). Different generations means their acceptance numbers aren’t comparable, so that run doesn’t get to be the headline - though for what it’s worth, it showed the same shape: the moment the generation turned predictable, MTP+SWA won big (+32%). The -2.6% comes from the task where both configs measured equally diverse (~0.72). Raw numbers for everything in this post: the result JSON.
• Best case and worst case are different context depths (123K vs 79K), because they’re built from different source text. The dense baselines bracket them consistently, but don’t read the +46%-vs-(-2.6%) pair as a single controlled variable flip; it’s two points on the same curve.
• Single-stream, batch-1, as always in this series. Greedy decode, so MTP output is byte-identical to MTP-off and the comparison is a pure speed test.

The trick is giving the hybrid model’s attention layers a bounded sliding window, so prefill goes flat (~855 tokens/s from 32K to 1M) and the KV cache stops growing. Then, because a windowed layer can only retrieve what falls inside its window, you keep 4 of the 10 attention layers un-windowed. I was wrong four times on the way here, and the week I lost to a one-character cross-backend bug is its own post.

A tuned box that is not yet a useful agent

This didn’t start as a sparse-attention project. It started as dissatisfaction with a box I’d already tuned. If you’ve been following along, you’ve watched me get Qwen running well on my Strix Halo mini-PC and then squeeze it from every direction I could find: quant ladders, speculative decoding, a thermal daemon so it could sustain load. Decode sits in the high 30s to mid 40s of tokens per second and short prompts turn around instantly. A nice local chatbot.

I don’t want a chatbot, though. I want a local agent, and agents eat context. So of course context is where the box fell over. A dense 256K-token prefill (the model’s full trained window) took about 20 minutes, per query, before the first token of the answer. A million tokens took 4 hours and 37 minutes. That’s not an estimate, I ran it once early on and watched it crawl. “Supports 256K context” was technically true and practically a lie. So the question stopped being “can I make it faster” and became: can I make a much larger context window usable on this box? I aimed at a frankly silly number, a million tokens. The intro already spoiled how that ends.

The box is HOMER, the same Strix Halo mini-PC this whole series is built on (introduced here, benchmarked against real hardware here). What matters for this post: a unified-memory APU has frontier-class memory and laptop-class compute. 128 GB of UMA will hold a multi-million-token KV cache without flinching. It just can’t feed the math fast enough (decode on these models is memory-bandwidth-bound, not compute-bound - that’s the roofline at work).

The model is Qwen3.6-35B-A3B at Q4_K_M, and it stays the model. I’m not swapping down to something sub-quadratic I’d like less. It’s a hybrid: of its 40 layers, 10 are full-attention GQA and the other 30 are Gated-DeltaNet (linear attention). That detail matters later, because only those 10 full-attention layers grow a KV cache, and only those 10 are somewhere attention sparsity can buy anything. It runs coherent on Vulkan/RADV llama.cpp (b9204), which I confirmed before trusting any throughput numbers.

The earlier tuning work also told me what kind of solution could work at all. I’d measured the box’s real memory bandwidth with a little HIP microbench: the read path hits about 242.7 GB/s out of a theoretical 256. The bus is maxed. No clever kernel is going to stream faster, and the memory overclock door is welded shut on this SKU. So the bandwidth-regime thesis from the rest of this series collapses to one rule here:

Every remaining win has to move fewer bytes.

Anything whose only benefit is more FLOPS is DOA since I’m not compute-starved at decode and I can’t go faster on the bus. What survives are tricks that skip reading some of the data. At long context, the biggest pile of skippable bytes is the KV cache during prefill, and the tool for skipping it is sparse attention.

The free win hiding in llama.cpp

llama.cpp’s Vulkan FlashAttention has a mask-aware tile-skip optimization (PR #19281, later hardened by a bounds fix in #20296): before it loads a tile of K and V, it checks whether that tile’s slice of the attention mask is entirely -inf. If it is, the query can’t attend there anyway, so the shader continues without loading the tile. No K read, no V read, no compute.

So I didn’t need a sparse-attention kernel, just a sparse-attention mask. Mark the blocks I don’t care about as -inf and the existing FlashAttention skips the memory traffic for me, on hardware it already runs on. All that’s left to build is the decision of what to drop.

I wasn’t the first one here either. DeepSeek’s own sparse attention (their DSA, from the V3.2 work) landed upstream as PR #23346, doing this same thing: fill a mask with -inf, unmask the selected positions, add it to the causal mask, hand it to the stock FlashAttention. The maintainers’ guidance on sparse attention was that it should reuse FA and the mask-skip rather than add a new op, so the route was sanctioned. (One AMD landmine worth flagging once: the argsort kernel hits a shared-memory assert on AMD GPUs, filed against top-k sampling in #24177, and it’s the same kernel an in-graph top-k block-selection would call. I pick blocks with a plain threshold instead. I wanted a threshold anyway.)

Attempt 1: the dumb fast version

Before trying to get too fancy, I built the dumbest possible mask to prove the skip even fires: keep a few “sink” tokens at the very start, keep a recent window of the last N tokens, drop everything in between. This is the classic StreamingLLM shape. Purely position-based, no knowledge of the content, a handful of lines in set_input_kq_mask_impl.

On the hero model at a 131K-token prefill, baseline throughput is 473 t/s. With a 256-token sink and a 4096-token window it’s 945 t/s, so 2.0x. At 262K it’s 223 -> 890, which is 4.0x. The speedup grows with depth, since the O(n²) attention term it’s killing is a bigger slice of prefill the deeper you go. In wall-clock terms a cold 256K prefill drops from about 19.6 minutes toward 5.

Two side measurements worth keeping. The window is a gentle knob: across windows from 1K to 32K at 131K context, throughput only slides from ~1008 to ~702 t/s, so a bigger, safer window is cheap. And there’s an Amdahl ceiling: at 131K the speedup caps around 2.1x even with a tiny window, because the MoE FFNs and those 30 DeltaNet layers are about half of prefill and sparse attention doesn’t touch them. At 256K attention is a larger fraction of the work, so the ceiling rises toward 4x.

The catch is that this is lossy. A fixed window drops the middle of your context. I ran a needle-in-a-haystack test (a planted string, TURQUOISE-7731, at a known depth) and got a depressingly clean result: you retrieve the needle if and only if it falls inside the window. Put it 20K tokens deep with a 16K window and it’s gone. The text stays fluent either way (sparsity doesn’t make the model babble), and a window big enough to never miss is a window big enough to give back the speedup. I also tried a dilated variant, keeping every Nth block on top of the window. It catches a deep needle if and only if the needle lands on the stride grid. Better than a pure window, still a dice roll. Position-based masks are a dead end for retrieval.

Why the dumb version doesn’t work

The fix has to be content-aware: keep the blocks that matter, wherever they sit, not the blocks that happen to be recent. The cheap version of that would be a static calibrated pattern, i.e. figure out once, offline, which columns are “heavy hitters” for this model and always keep those. Before building anything dynamic I measured whether static could work, with a little capture tool built on llama.cpp’s eval-callback, watching the 10 full-attention layers.

The obvious cheap proxy, key magnitude, is dead on this model. The Qwen3 line QK-normalizes its keys (an RMSNorm on Q and K before the dot product, a trick that goes back to ViT-22B), and per-position key norm came back flat: max over median 1.0x. Actual attention mass (run the softmax with FA off and watch where the weight goes) is another matter, max-over-median 47x. The weight piles onto the sink, onto a recent local window, and onto about 6.6% of columns scattered through the middle that are content-salient. The planted needle showed up as one of those heavy columns, so “keep the high-attention columns” preserves retrieval by construction.

Then I moved the needle (token ~350 to ~541) and re-measured, and the heavy mid-context cluster moved with it. The salient columns follow the content. A static calibration learned on one prompt can’t generalize to the next, because the important columns are wherever this particular document put them. So selection has to be dynamic: per-prompt, in the graph, from the actual keys.

Attempt 2: it retrieves, and it’s slower

Attempt 2 does its scoring dynamically, inside the compute graph right before the FlashAttention call. Pool the queries and keys down to one representative vector per block, score block against block, softmax over the key-blocks, threshold to a keep/drop decision, and OR that with “always keep the sink and the recent window.” Turn the drops into an additive -inf block mask and hand it to the stock FlashAttention. Same tile-skip as attempt 1, but the kept set follows the content.

Retrieval-wise it delivered. The first cut scored blocks off a single strided token and retrieved erratically (2 of 5 needle depths), but mean-pooled block representatives unioned with the sink+window fixed that: 5 out of 5 needle depths, at two different thresholds. The needle’s block gets kept wherever it lands.

......... aaaaaaaaand it was slower than doing nothing. 0.79x. At 131K, baseline 473 t/s became 372. What the heck?

The mechanism is strictly subtractive: I’m dropping blocks, FlashAttention should read less. Chasing down how dropping work produced a slowdown ate the better part of a week and became a post of its own. The short version: a one-character cross-backend bug in ggml. The step function I built my keep/drop mask on returns a different value at exactly zero on the Vulkan backend than on the CPU (step(0) is 1 there, 0 on the CPU), so my “drop this block” decision quietly evaluated to “keep” for every block, and FlashAttention never skipped a thing. The full debugging war story, and the upstream fix, are here.

What it does, and what it turned out to be

With the mask finally dropping the blocks I told it to (huzzah!), the code delivered the number I’d been chasing: at a 131K prefill, baseline 473 t/s became 880, a clean 1.86x, with the needle still retrieved at 5 of 5 depths. I started writing the blog post and getting ready to accept the recognition of my genius.

Then I measured it properly, and it just became two more things I’d been wrong about. Blarg.

First, the 1.86x was graded against the wrong quality bar. The 880 t/s came at a 4096-token window, and the only perplexity I’d checked was at a 32K context, where 4096 is still ~12% of everything. At the 131K context where the speed number lives, a 4096 window is ~3% of the prompt. So I did the boring thing I should have done first: same-context perplexity at 131K, swept across windows, on two corpora (Moby Dick for prose, the llama.cpp source tree for code).

The 1.86x headline is unusable at real depth. A +74% prose perplexity is not a model you’d want to talk to. Prose has a cliff-then-recovery shape, since it leans on local context and a moderate absolute window restores it. The 16K window at +2.2% is the knee, and it’s what I’d ship for prose. Code never recovers: still +2.5% at a half-context window that’s barely faster than baseline, because dropping any blocks hurts long-range structure (a matched delimiter, a scope opened a thousand lines up). For code, don’t sparsify.

Notice that needle-in-a-haystack passed 5/5 on every row of that table, including the +74% disaster at the top. This is one of those things I would REALLY watch out for, and is why I do so much validation/testing. One planted token survives heavy sparsity while the model’s general grip on the context craters. This is the complaint RULER and NoLiMa make about single-needle retrieval, and if I’d trusted NIAH I’d have shipped the +74% config as “perfect.” Perplexity was the honest measure here. Though perplexity has a blind spot of its own, which bites me later on.

The second thing I was wrong about retired the word “content-aware.” Quality and speed barely moved no matter how I set the content threshold, so I ablated it directly: tau from 0.01 to 0.99 at a fixed window gave identical speed and perplexity at every value. tau was a no-op. I had built a content selector that selected no content, at any setting, the entire time. .... Yeah. Egg on my face.

The mechanism is simple once you look. The block score is a softmax over all ~1000 key-blocks, and that softmax is sharply peaked on the recent blocks. A far-back salient block, normalized against the dominant recent ones, comes out with a probability near zero, below any threshold worth using. The keep-cliff sits between tau=0 and tau=0.0005, nowhere near the 1/1024 you’d see if the distribution were remotely flat. I wondered whether mean-pooling 128 tokens into one representative was washing out a single sharp needle key, so I swapped in max-pooling. It behaved identically, because the problem is the softmax normalization, not the pooling. The only blocks my selector wanted were the recent ones already inside the window. Net new blocks kept: zero.

So attempt 2 had silently collapsed back into attempt 1, a plain sink-plus-sliding-window. Every row in that table is a sliding-window result. The content-aware machinery I spent a week debugging was, once it finally ran, computing an elaborate score and then keeping the same blocks a three-line window mask would have kept.

Why did it retrieve, then? If the selector keeps nothing far-back, the needle’s block gets dropped on the windowed attention layers, and yet NIAH was 5/5. At the time I had an explanation I liked: only 10 of the 40 layers are full-attention and those are the only ones I masked, so the 30 unmasked Gated-DeltaNet layers, which see every token, must be carrying the needle. A tidy story that fit every data point I had. It’s also totally wrong, which makes it mistake number four, and I wouldn’t find out until I measured recall properly, two sections down.

The moonshot: a million tokens, prefilled but not usable

Even with the content-aware story dead there’s a real result here, the one I’d been chasing since I set the silly target, because the cold prefill was always the wall on this box. Dense prefill goes super-quadratic past ~131K: the KV cache for the full-attention layers outgrows the 32 MB Infinity Cache, and attention starts re-reading it from LPDDR5 at the 256 GB/s ceiling. Windowing the attention layers should bound each query’s KV slice and flatten the curve. My first cut at that was the mask, and it half-worked. Dense versus my 16K-window mask:

The speedup grows with depth. At a million tokens the dense side is not a guess: that’s the 4h37m run from the intro, and fitting a curve to this ladder lands right on it (4.6-5.2h depending on the fit). The mask extrapolates to ~40 minutes there, a raw ~6.6x, but I never ran the mask at 1M, and it has two structural leftovers that made me not want to. It still allocates the full KV cache, so the memory win is zero. And llama.cpp’s FlashAttention still loops over every query-tile by key-tile pair, paying a per-tile check even on the all--inf tiles it skips, so the work is O(N) but the bookkeeping stays O(N²).

The fix is to stop faking the window and ask for a real one. llama.cpp already ships sliding-window attention as a first-class path (Mistral, Gemma, and the hybrid lfm2 models all use it), so I marked the 10 attention layers as sliding-window through that path. They get a rolling KV cache bounded to the window: it never grows, and the kernel never touches tiles outside it. Prefill becomes genuinely linear:

~855 tokens a second from 32K to 1M, a 4.8% drift across a 32x range, while dense falls off a cliff. Both million-token prefills are real wall-clock numbers, and the ratio is ~13x. (*The dense 1M is the earlier run from the intro: Q8_0 weights and q8_0 KV via llama-bench, which is why it’s starred rather than sitting in the same Q4_K_M ladder as the rest of the column. It also ran without a RoPE override, so it’s a throughput measurement, not a coherence-valid million.) One thing that run settled, to be fair to dense: memory was never the killer on this box. The dense 1M fit in the 128 GB and even decoded at 10.6 tokens/s once the 4.6 hours were paid. The windowed cache staying bounded at a few hundred MB, where dense’s grows linearly with context, is still the difference between a box with headroom and a box running one job, and it matters a lot more on hardware with less RAM than this.

Here’s the half of the moonshot I didn’t get. The quality is the same windowed quality from the last section, because it’s the same window done properly (perplexity at 131K is 1.3718 versus the mask’s 1.3512 - the 1.5% is the attention sink the mask kept and the stock path leaves out). At a million tokens that window table stops being a caveat. A 16K window is 1.6% of a 1M context, the same fraction as a 2K window at 131K, and that configuration measured +57% code and +199% prose perplexity. Unusable. Retrieval is somehow even worse: the next section measures it properly, and a windowed layer’s recall reach is its window, so nothing more than 16K tokens back is findable. On top of which, Qwen3.6’s trained window is 256K. You can stretch a model past its training with RoPE tricks (on the big MI300X box earlier in this series I validated a freq-base override out to needle retrieval at ~634K), but nothing in these runs does, so past 256K there was never a usability claim on the table anyway. The million-token run is a prefill-and-memory demo. It proves the machinery stays flat and bounded as deep as you push it, and the window is a dial rather than a wall (at 256K, a 32K window holds 699 t/s at +5.1% code perplexity, a 64K window 522 t/s at +2.5%). It is not a usable context.

Prefill wall down, memory wall down, usability not achieved. The useful move from there was to point the same machinery at the deepest context where usability is winnable, and find out where that is.

Recall, measured properly: the four-layer backstop

First I had to fix my own measurement. The 5/5 retrieval results earlier in this post came from a harness I’d stopped questioning: sampled decoding, no stop token, long rambling generations scored by substring match, and a rambling generation can blunder into a substring match it didn’t earn. The corrected harness is greedy decoding, stop at end-of-generation, exact match, plus one behavioral check before trusting any grid: an out-of-window needle has to actually miss on the binary under test. That last check is not paranoia. I burned a full run measuring an unpatched binary that silently ignored the window setting and ran dense. If your feature is env-gated, “it ran” is not evidence it ran.

Measured that way, the sliding-window path gives a blunt answer: single-needle retrieval reach is the window. In a ~98K context with needles planted ~88K, ~49K, and ~10K tokens back, dense finds all three, a 64K window finds the two inside it and misses the 88K one, a 16K window finds only the last. The needles the window misses stay missed, which kills my DeltaNet explanation. The 30 linear-attention layers see every token, and they’re why the model stays fluent about the far context, why perplexity degrades gently under windowing instead of collapsing the way a windowed pure-attention model does. They do not do precise retrieval. On this model, precise retrieval lives in the full-attention layers, all ten of which I had just windowed.

The fix: don’t window all ten. I swept “keep the last K attention layers full” at a 16K window, same three needles. K=2 buys nothing, still blind past the window. K=4 brings all three needles back. K=6 matches K=4 and is just slower. There’s a critical mass somewhere between 2 and 4 layers, and 4 of the 10 is the cheapest configuration that clears it. The retrieval-heads literature says why this shape appears: Wu et al. showed NIAH-style retrieval is done by a small set of attention heads scattered across layers, and RazorAttention / DuoAttention keep those heads full while windowing the rest. Keeping whole layers is the blunt version of the same idea. You keep enough layers full to contain the heads that matter.

That is the recipe, and on my fork it graduated from env hack to a real CLI flag. Here’s the entire mechanism, a loader-time helper shared by the hybrid architectures (each one’s graph then routes through llama.cpp’s existing iSWA hybrid cache when swa_type is set):

// models.h (my fork) - shared by qwen35moe, qwen3next, granite-hybrid.
// Window every full-attention layer; un-window a chosen few as the recall backstop.
static inline void apply_dynsparse_swa(llama_hparams & hparams, const llama_model_params & mp) {
    // ... resolve W / backstop list from --dynsparse-swa* flags (env vars as fallback) ...
    hparams.swa_type = LLAMA_SWA_TYPE_STANDARD;
    hparams.n_swa    = swa_w;                                   // the window W
    for (uint32_t i = 0; i < n_main; ++i) {
        hparams.is_swa_impl[i] = hparams.is_recr(i) ? 0u : 1u;  // attention layers -> windowed
    }
    // ... then un-window the backstop set (--dynsparse-swa-full 27,31,35,39)
    //     and print the resolved windowed/kept-full split at load ...
}

Which makes the whole recipe a command line: window to 16K, keep four layers full, q4_1 KV cache.

build-vk/bin/llama-server -m Qwen3.6-35B-A3B-Q4_K_M.gguf \
    --dynsparse-swa 16384 --dynsparse-swa-full 27,31,35,39 \
    -ctk q4_1 -ctv q4_1 -fa on -c 262144

None of this is upstream currently, so don’t expect the flag in a stock build. The whole fork is one patch on top of upstream llama.cpp commit 6f4f53f2b: grab the patch, then

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 6f4f53f2b
git apply 395aimax-fork-full.patch
cmake -B build-vk -DGGML_VULKAN=ON && cmake --build build-vk --config Release

That patch also carries the measurement tooling behind this post’s numbers (the attention-capture tool and the KV-eviction harness). A smaller extract with just the SWA feature is here if you’d rather read than build. Run end-to-end at real depths, with the needle planted outside the window so it can only be reached through the backstop:

At 131K the recipe beats dense on all three axes at once (1.89x smaller KV, 1.33x faster prefill, 1.14x faster decode - fewer bytes helps decode too). And the 256K row is the model’s entire trained context. Where the million was the stunt, this row is the everyday claim: the whole native window, ingested in ten and a half minutes at the throughput dense manages at half that depth, answering at 36 tokens/s with retrieval intact, on the “cheap” box. FINALLY! We did it, Reddit!

A single planted needle is the easiest retrieval task there is (Paulsen’s MECW work measured effective context across task types and found single-needle by far the most flattering probe), so I graded harder ones too. Multi-fact retrieval (five codes scattered through the context, “list all five”) is where the recipe hits its first real boundary: 5/5 at 128K but 1/5 at 256K, where dense still scores 5/5, so it’s my recipe’s limit rather than the model’s. Doubling the window to 32K restores 5/5 at 256K for +8% KV and +8% prefill. Oddly, four of the five needles sit far outside even the doubled window (the deepest is 216K tokens back), so the kept-full layers were never the bottleneck. The narrow window was degrading the model’s ability to bind multiple facts together at depth, and more local context fixes it.

I also checked that the model uses a long context rather than just fishing strings out of it: plant counterfactuals that contradict strong priors (water boils at 157C, the capital of France is Oslo) and ask it to answer from the document. The recipe matches dense, 5/5 at both 128K and 256K, zero reversions to what the model knows is really true. The one task family that still needs dense is aggregation, where the answer is spread diffusely over the whole context. Windowing breaks that gather and no backstop saves it, so know which kind of question you’re serving.

One model is cool but I wanted to know how portable this was, so I ported the recipe to two more hybrids. On Qwen3-Next-80B (another DeltaNet hybrid) the whole pattern transfers at 128K: 2.07x smaller KV, 1.37x faster prefill, 1.16x faster decode, far needle retrieved, multi-fact 5/5. On Granite-4.0-H-Small, a Mamba2-heavy hybrid with only 4 attention layers out of 40, it also works, with the critical-mass lesson in miniature: one kept-full layer holds recall at 64K but breaks at 128K, and two of its four hold 128K at a 1.46x KV shrink. The constant that moves between models is which layers you keep full, so measure that per model rather than trusting mine. The scope boundary: this is a hybrid-model technique. A pure-attention model has no recurrent layers to keep the far context fluent, windowing it at prefill collapses quality (SWAA documents this), and the recipe degrades into plain StreamingLLM with recall capped at the window. The hybrid’s linear layers make the windowing survivable, and the kept-full attention layers make it retrieve.

Where this sits, and what’s actually mine

Credit where it’s due, because almost none of the algorithm here is new, and the part that was supposed to be mine didn’t pan out. Training-free, dynamic, content-aware sparse prefill is a well-trodden line, and attempt 2 was my try at joining it:

• MInference (Microsoft, NeurIPS 2024) defined the problem: classify each head’s sparse pattern, build the indices per prompt, run an optimized sparse prefill. It’s the anchor everyone benchmarks against.
• FlexPrefill (ICLR 2025) does the part I leaned on hardest: per-prompt, training-free, selecting blocks by a cumulative-attention threshold rather than a fixed top-k. That’s my threshold, arrived at separately for the same reasons.
• SeerAttention is the closest thing to my scorer: pool Q and K down to block representatives, multiply for block scores, run block-sparse attention. They learn a gate for it, where I threw the learning away and thresholded the raw pooled score.
• XAttention and SpargeAttention are training-free block-sparse cousins with different cheap importance proxies (antidiagonal sums, block self-similarity).
• DeepSeek’s NSA is the trained-from-scratch version of the idea, and their DSA is the inference-time one whose llama.cpp mask idiom I copied outright.
• My fixed sink+window is straight out of StreamingLLM, and the attention sink it leans on is a studied phenomenon, not folklore.
• The keep-some-attention-full backstop has its own literature: retrieval heads (a small, sparse set of heads does the retrieving), plus RazorAttention / DuoAttention, which keep those heads full and window the rest. My kept-full layers are the coarse version of their kept-full heads. SWAA studies windowing full-attention models at prefill and finds it collapses on pure-attention models, the boundary my hybrid results sit just inside.

So what’s mine? Less than I thought when I started writing. I set out to join that club, training-free content-aware block selection, and on this model and these corpora I couldn’t, because there was no cheap salient subset for a selector to find. The quality need is broad: code wants ~25% of the context back before it’s within 5% of dense, prose wants a moderate absolute window. With no handful of heavy blocks to keep, my selector reduced to a sliding window, and the window is StreamingLLM. What’s left:

1. It runs on a “cheap” consumer iGPU, through Vulkan, with no new kernel. The papers above ship custom CUDA. Here the win was noticing the box already had the machinery, first a mask riding the stock FlashAttention tile-skip, then marking the attention layers sliding-window through llama.cpp’s existing hybrid-SWA path. Nobody had ported the RazorAttention/DuoAttention idea into this stack either, and llama.cpp’s knob is per-layer, so the layer-level version is what’s implementable today. I implemented it.
2. The measured recipe on hybrids: window the attention layers, keep a critical mass full (4 of 10 here - 2 buys nothing, 6 buys nothing more), with operating points from grading it against real tasks. 16K windows for single-fact, 32K for multi-fact, dense for aggregation. Validated on three hybrid families, scoped to hybrids on purpose.
3. The bug, which is its own post and the reason any of this took as long as it did.

One anti-result, because these rarely get written up: pooled-QK threshold selection with a global softmax does not surface far-back salient blocks on this model. The normalization drowns them, and max-pooling doesn’t save it. If you’re building content-aware sparse attention, ablate your selector against a plain window early. Mine looked like it was working for a week and wasn’t.

Takeaways

The most expensive lesson was the one-character bug, and it has its own post. Short version: when a compute graph validates on the CPU but misbehaves on a GPU backend, suspect a per-op semantic mismatch before anything clever about scheduling or memory. The bug only hid the work, though. The work itself taught me two things.

Ablate your own knob. If a feature has a setting that turns it off, run that setting, because if “off” looks identical to “on” then your feature was never on. One tau sweep would have caught the inert selector in an afternoon instead of an article. I believed in it for a week because I never pointed it at its own off switch.

And don’t trust a needle. It passed 5/5 at every window I tried, including the one at +74% perplexity, and then it turned out even those passes were partly my harness being generous. Perplexity lies in the other direction. It barely notices retrieval, which is how the recall boundary hid inside decent-looking curves. Measure both, strictly, on the content you actually serve, and pit whatever you built against the dumb baseline it’s supposed to beat. Mine turned out to be the dumb baseline, which I’d have known on day two if I’d run the comparison.

What’s left is smaller than I set out to build, and I like it more for being true. The cold prefill wall comes down the way the bandwidth thesis said it had to: stop reading bytes you don’t need, and the bytes turned out to be most of the KV cache on most of the attention layers. The linear-attention layers keep the far context fluent for free, four kept-full attention layers keep it retrievable, the million-token moonshot lands as a 20-minute prefill instead of a 4.6-hour one but nothing more, and the model’s whole native 256K context becomes something a local agent can live in, which is what I wanted from this box all along. I just had to be wrong four times to get there, and teach the GPU that zero is not a positive number along the way.

I found it the hard way: building sparse-attention prefill on my “cheap” AMD Strix Halo miniPC, I hit a slowdown where there should have been a 2x speedup, and spent the better part of a week blaming the graph scheduler, the allocator, an O(n^2) re-sweep, and buffer aliasing, before realizing my “drop this block” mask, built out of step(), was quietly keeping every block because step(0) was wrong on the GPU, so FlashAttention read all of it anyway.

Just enough setup to make the bug land

To explain what the bug was I have to back up to what I was building…

I’ve been trying to make a long context usable on a “cheap” Strix Halo box (a Ryzen AI Max+ 395 with the gfx1151 iGPU) - yes I know, I am a crazy person. That box is bandwidth-bound so the only real approach is moving fewer bytes, and during a long prefill the way you do that is sparse attention: don’t read the parts of the KV cache that don’t matter.

The neat thing about doing this in llama.cpp is that you don’t have to write a kernel. The Vulkan FlashAttention already has a tile-skip (PR #19281): before it loads a tile of K and V, it checks whether that tile’s slice of the attention mask is entirely -inf, and if so it skips the tile, the load, and the compute. The entire feature reduces to one thing, computing a good mask. Mark a block -inf and its bytes never move.

To decide which blocks to keep I built a boolean: keep a block if it scores as content-salient or it falls in a recent window. And I built that OR the way you build booleans when you’re working in a tensor graph that has no if: out of arithmetic, with the Heaviside step function. step(x) is 1 for positive x and 0 otherwise, so keep = step(content_score + in_window) is a serviceable OR, and step of zero is supposed to be the “neither” case that evaluates to drop.

Turns out that “supposed to” in the previous sentence was doing a lot of heavy lifting.

The symptom: drop work, get… slower???

With the mask in place, the thing got slower. What the heck? At a 131K-token prefill, baseline throughput was 473 t/s; my sparse version did 372. That’s confusing, because the mechanism is supposed to be strictly subtractive. I’m dropping blocks. FlashAttention should be reading less. Dropping work and getting a slowdown means something is very wrong, and I had a mask that looked correct, output that was coherent, retrieval that passed my tests, and a number going the wrong way.

A week of being wrong

None of these were dumb guesses. Every one had a real mechanism behind it, got measured, and got fixed, and not one of them was the problem (RIP me, I guess, but will be useful later!). The graveyard, in order:

Wrong theory #1: a CPU fallback. My block-scoring step pooled the keys with ggml_pool_1d, and the Vulkan backend has no 1D pooling kernel, only 2D. So every micro-batch, that op silently fell back to the CPU, with a full GPU->CPU->GPU round trip and a sync each time. That’s real, and I confirmed it three ways (source, a decomposition that showed a fixed per-ubatch tax independent of how much I dropped, and a scheduler trace). I rewrote it as a ggml_pool_2d over the token axis. It helped: 0.79x became 0.94x. Still a slowdown.

Wrong theory #2: an O(n^2) re-sweep. The graph is stateless per micro-batch, so each of the ~256 prefill ubatches re-pools all of the keys accumulated so far. Sum that over the prefill and you get an O(n^2) re-sweep of K, the same order as FlashAttention’s own read. I built a model that fit all my data points and it was beautiful. The fix is to amortize: cache each block’s representative once, when its tokens are written to the KV cache, and read the cache instead of re-pooling. A persistent per-layer rep buffer parallel to the KV cache, incremental updates on the write path, the whole thing. It compiled, it retrieved correctly, and it bought zero speed. The beautiful model was wrong.

Wrong theory #3: FlashAttention isn’t skipping at all. By now I’d stopped trusting my assumptions and turned on the Vulkan performance logger. It settled it: FlashAttention was dominating the trace, reading the full K, every layer, the entire time. The sparse machinery I’d been agonizing over wasn’t even in the top 25 ops. The keep-nothing case (an all--inf mask, which should be the maximum possible skip) ran at 390 t/s, slower than baseline. If the skip were firing, that case would be 2x. It wasn’t firing. At all. For any version. Every “sparse” run I’d done was running dense attention plus the overhead of computing a mask nobody used.

That reframed the hunt. The mask was being computed and added, and I could prove it was logically -inf, because a keep-nothing mask makes the model attend over nothing and produce NaNs. But FlashAttention wouldn’t tile-skip it. Meanwhile a simpler fixed-window mask, the one I’d built first and fed in as a plain input tensor, skipped fine. Same kernel, same model, one mask skips and one doesn’t.

Wrong theory #4: the graph scheduler / buffer aliasing. I narrowed it hard. A diagnostic that forced the mask to all--inf through a short op chain skipped cleanly (>2x, the win was provably there). My long dynamic chain produced a mask that matched the short one in shape, stride, type, and contiguity, and as far as I could tell held the same -inf values, and yet FlashAttention skipped one and not the other. The only difference was the graph that produced the tensor. From there it really looked like a scheduler or graph-allocator bug: a missing barrier between my long producer chain and the FlashAttention consumer, or a buffer getting aliased and overwritten before FA’s skip pre-pass read it. I had a whole “the deeply-derived tensor lands in a reused buffer” story, complete with a clean-buffer workaround that happened to make it skip. It was, again, plausible, and again not the cause.

The pattern across all of it: I spent days on re-sweep math, op fusion, mask layout, sync barriers, and buffer lifetimes. Every one of those was a sophisticated guess about systems behavior, and every one was wrong, because the bug wasn’t in the systems. It was in arithmetic.

The actual bug

What all my workarounds had in common, the reason “force the mask through a short chain” or “drop into a freshly-filled buffer” made it skip, is that they all bypassed the math that built the keep/drop decision. I’d been treating that math as incidental, when it was the whole bug.

Recall the OR: keep = step(content_score + in_window), where step is 1 for positive and 0 otherwise. The intent is “keep this block if it’s content-salient or it’s in the window.” The “neither” case, score zero and not in the window, is supposed to land on step(0) and evaluate to drop.

step(0) does not return 0 on the Vulkan backend. It returns 1.

Here are the two implementations, which are supposed to compute the same function:

// CPU reference: ggml/src/ggml-cpu/vec.h, ggml_vec_step_f32
y[i] = (x[i] > 0.f) ? 1.f : 0.f;     // step(0) = 0

// Vulkan: ggml/src/ggml-vulkan/vulkan-shaders/unary.comp, op_step
return x >= 0.0f ? 1.0f : 0.0f;      // step(0) = 1

> on the CPU, >= on the GPU. At exactly zero they disagree by the maximum possible amount. And boolean-indicator math sits on exactly-zero constantly, because that’s what “this condition is false” is. So on the GPU, my “neither” case, the blocks I most wanted to drop, evaluated to step(0) = 1 = keep. Every block was kept. The mask was effectively all-zeros. FlashAttention dutifully read every tile, because I’d told it to, and I’d spent a week blaming the scheduler for honoring my own broken arithmetic.

It survives undetected because you only hit it at an input of exactly 0.0, and the upstream test for this op never does. test-backend-ops initializes its inputs with uniform random floats in [-150, 150], which never land precisely on zero, so the STEP test passes green while the op is wrong. I forced the input to exactly 0 in a one-line edit and reran it:

[STEP] ERR = 1.000000000 > 0.000000100   STEP(type=f16,...,v=0): FAIL
[STEP] ERR = 1.000000000 > 0.000000100   STEP(type=f32,...,v=0): FAIL
0/4 tests passed   Backend Vulkan0: FAIL

ERR of exactly 1.0: CPU says 0, Vulkan says 1. Both precisions. The fix is one character, >= to >, to match the CPU reference. My local workaround, until that lands, is to bias the OR by 0.5 so the boundary is never at zero: step((a + b) - 0.5), which is robust no matter which way step(0) falls. I filed it upstream with the repro and the one-line fix: ggml-org/llama.cpp#25027. A Vulkan-backend maintainer picked it up and wrote the actual patch, PR #25036, >= to >, approved by two maintainers and waiting to merge as I write this.

“step(0) is off by one” sounds like a corner case, and for most llama.cpp users it is one: step barely appears in mainstream model graphs, so nobody’s chat session was silently wrong because of this. The cost is narrower and meaner than that. If you build boolean or gating logic out of step on the Vulkan backend, the way I did, your graph silently disagrees with the identical graph on the CPU, at exactly the input boolean logic lands on constantly: the false case, where the indicators sum to zero. Nothing crashes and nothing warns. It just quietly does the opposite of what you wrote, on one backend only.

That is also what fixing it means. Once the patch lands, step on Vulkan agrees with the CPU, I drop the -0.5 hack, and the mask works without a workaround. The broader win is quieter than a speedup: ggml has one fewer place where the same graph returns two different answers on two backends. A portable op that isn’t actually portable is worse than no op at all, because you trust it. Now it matches the CPU, like it always should have.

The thing that still stings: every fix I built in the graveyard above was correct. The pool_2d rewrite, the rep cache, the clean mask construction, they were all better, and they all paid off the instant the mask started dropping blocks. I just built them to fix problems that were downstream of a one-character lie about whether zero is positive.

Exactly how much each of those buys, now that the skip actually fires, is a writeup of its own. I’ll dig into the pool_1d-to-pool_2d fix, the amortized rep cache, and the rest of the speed work in a follow-up post. This one’s about the bug.

The lesson I’ll actually remember

If a compute graph validates fine on the CPU and misbehaves only on a GPU backend, suspect a per-op semantic mismatch first, before scheduling, fusion, barriers, or memory. Backends are supposed to compute the same function and almost always do, which is exactly why the rare place they don’t is so expensive: every instinct sends you to the systems layer, where the behavior is, instead of the arithmetic, where the bug is. The tell, in hindsight, was that all my workarounds worked by avoiding the math, and I read that as “the systems path is fragile” when it meant “the math is wrong.” A week, for >= versus >.

It generalizes past this one shader. The same class of thing, the same op computing subtly different numbers depending on where it runs, is exactly what the determinism people keep finding (Thinking Machines’ writeup on batch-variance is the recent canonical one), and it’s why the boring discipline pays: when you build boolean logic out of arithmetic, the boundary value is a real input, not an edge case, and it’s the one a random test generator never produces on its own. The whole thing was avoidable, and I’d have caught it on day one if I’d doubted the easy part instead of the hard part.

The daemon is the boring part, a poll loop and four numbers. The rest of this is how I ended up writing it, and why the off-the-shelf tools were no help.

A tradition of cooking this box

I have now lost a workload to thermals on this miniPC three separate times (four crashes, counting the sitting where I did it twice), across two (soon to be three) posts.

1. The first time, I learned the hard way that a UMA APU in a miniPC chassis will cook itself to a hard shutdown the instant CPU and GPU spike together - a 165W burst the little cooler can’t clear, and the firmware just cuts power. The fix that time was a ryzenadj power cap.
2. The second time, I literally opened the relevant section with the words “don’t re-make the same thermal mistakes as last time,” and then re-made them - forgot to pin the fans before a multi-hour sweep and crashed the box twice in one sitting. After that I wrote a wrapper that refused to start a benchmark unless the fans were confirmed spinning at full tilt. Surely that was the end of it.
3. It was not the end of it, because the wrapper only helped if I remembered to route a run through it. So I kicked off a 262K-token benchmark the quick way, walked away, and it went dark at 102°C partway through the run. This post isn’t live yet, hence the lack of a link, but it’s coming.

The root problem here was clear: any system built on me remembering was doomed to failure.

Driving the fans

None of this part is my discovery - I got into Strix Halo thermals the way most people do - by finding the strixhalo.wiki fan-and-power-control guide after the box started misbehaving (it first cooked itself in my very first post on this machine), which is where both the ec_su_axb35 tooling and the “the standard Linux fan stack won’t bind here” situation come from. The part that’s actually mine is the specific curve and the daemon that runs it, further down.

The short version: lm-sensors + fancontrol won’t bind here, because there’s no standard hwmon PWM (pulse-width modulation, the variable signal that normally sets fan speed) channel for it to grab. The fans sit behind a vendor EC (embedded controller) exposed by a custom out-of-tree driver, ec_su_axb35, under its own sysfs class:

/sys/class/ec_su_axb35/
├── apu/power_mode          # balanced | performance
├── fan1/{mode,level,rpm,rampup_curve,rampdown_curve}
├── fan2/...
├── fan3/...
└── temp1/{temp,min,max}

You set a fan by writing fixed to mode and a level of 1-5, or auto to hand control back to the EC. No PWM percentage, no hwmon binding. (For the record, at level=5 the three fans read 4227 / 4321 / 1876 RPM - two big fans near 4300, one smaller one near 1900. Worth knowing, because “max” isn’t one number.)

The daemon hinges on one detail here: the EC’s own temp1 is a conservative sensor that tops out around 75°C, but the temperature that actually kills the box - the one that read 102 at the crash - is Tctl, from k10temp. Tctl is the control temperature AMD’s chips report for thermal management: effectively the CPU die/junction temperature, and the number the hardware itself watches when it decides to cut power. So the daemon watches Tctl, not the EC’s sensor (and finds k10temp by name rather than by hwmon index, because those indices renumber across reboots - exactly the kind of thing that turns “works on my machine” into “works until the next reboot”).

Damen’s daemon

It’s a graded fan curve on Tctl. Below 75°C the fans go back to auto and the box stays quiet; from 75°C up it steps through fan levels, hitting full blast by 85°C - comfortably under the 100°C TjMax, the junction-temperature limit where the chip starts cutting power to save itself:

CURVE=(75:2 79:3 82:4 85:5)   # Tctl °C : fan level; below 75 -> auto
HYST=3                        # downward deadband so it doesn't flap at a boundary
INTERVAL=3                    # seconds between polls

That’s the entire configuration. The rest is plumbing: poll Tctl, compute the target level, and if it changed, write the level to all three fans (or hand them back to auto when it cools off). A 3°C downward deadband stops it oscillating when the temperature parks on a band edge; warming ramps immediately, cooling waits a few degrees before stepping down. Simulating a sweep makes the hysteresis obvious:

WARMING:  auto ... 75->lvl2  79->lvl3  82->lvl4  85->lvl5
COOLING:  lvl5 held to 82 ... 81->lvl3  75->lvl2  71->auto

It runs as a systemd service with Restart=always, and it deliberately does not restore auto when it exits - if the daemon dies, the fans hold where they were rather than dropping mid-load, and systemd just brings it back. Fail toward cold, not quiet.

Here it is doing exactly that. I installed the service, then drove a sustained 35B benchmark and watched its journal:

strix-fan-daemon started; watching hwmon3/temp1_input; curve=75:2 79:3 82:4 85:5 interval=3s hyst=3C
Tctl 50C -> fans AUTO
Tctl 75C -> fans level 2
Tctl 79C -> fans level 3
Tctl 63C -> fans AUTO

Idle, the fans sit on auto. The bench drives the die temp up, and at 75C the daemon takes over: level 2, then level 3 at 79C. Level 3 turned out to be enough to hold the box in the high 70s for the rest of the run, so it never had to reach for 4 or 5. When the load stopped and the die fell back through the floor, it handed the fans back to auto. And the whole time, power_mode stayed exactly where I left it (performance) - the daemon only ever touches the fans, never the power profile.

The code

Here’s everything: the three files plus an installer. They assume the ec_su_axb35 driver is already installed and loaded - if it isn’t, set that up from the strixhalo.wiki guide first. If you have the same box you can lift these as-is; on a different EC the shape carries over even though the sysfs paths won’t.

The daemon itself, strix-fan-daemon.sh:

#!/bin/bash
# strix-fan-daemon - graded Tctl-driven fan curve for the Strix Halo.
#
# Automates, as a service, what fan-inference.sh does by hand: it watches the CPU
# die temperature and ramps the fans so the box stops cooking itself when someone
# forgets to crank them before a benchmark. It touches ONLY the fans
# (fanN/{mode,level}); it never changes apu/power_mode, ryzenadj, or any power limit.
#
# The spec:
#   - graded curve (not binary)
#   - driven by CPU die temp, k10temp / Tctl
#   - full fans (level 5) at >=85C, release to auto below 75C
#   - fans only, no ryzenadj
#
# Runs as root via systemd (strix-fan-daemon.service).

set -u

EC="/sys/class/ec_su_axb35"

# ---- tunables (the whole config is these four things) ----------------------
INTERVAL=3            # seconds between Tctl polls
HYST=3               # downward deadband, in C, to stop the fans flapping at a boundary
HEARTBEAT=20         # log a heartbeat every N polls even when nothing changes

# Graded curve. Each entry "MIN_TEMP:LEVEL": at Tctl >= MIN_TEMP (and below the
# next entry) the fans are pinned to LEVEL (1-5) in fixed mode. Below the lowest
# MIN_TEMP they drop back to auto (quiet idle). power_mode is never touched. The 75
# floor and the 85->5 top are the anchors; 79/82 are the ramp in between.
CURVE=(75:2 79:3 82:4 85:5)
# ----------------------------------------------------------------------------

log() { echo "$(date '+%F %T') $*"; }

if [ "${EUID:-$(id -u)}" -ne 0 ]; then
  echo "strix-fan-daemon must run as root (it writes $EC)." >&2
  exit 1
fi

# Resolve the k10temp Tctl input by hwmon *name* (indices move across reboots).
find_tctl() {
  local h lab
  for h in /sys/class/hwmon/hwmon*; do
    [ -r "$h/name" ] || continue
    [ "$(cat "$h/name")" = "k10temp" ] || continue
    for lab in "$h"/temp*_label; do
      [ -r "$lab" ] || continue
      if [ "$(cat "$lab")" = "Tctl" ]; then echo "${lab%_label}_input"; return 0; fi
    done
    [ -r "$h/temp1_input" ] && { echo "$h/temp1_input"; return 0; }
  done
  return 1
}

# Desired fan level for a given temp (0 = auto/idle).
desired_level() {
  local t=$1 lvl=0 entry min level
  for entry in "${CURVE[@]}"; do
    min=${entry%:*}; level=${entry#*:}
    [ "$t" -ge "$min" ] && lvl=$level
  done
  echo "$lvl"
}

apply_level() {            # $1 = level (0 = auto). Fans only - never touches apu/power_mode.
  local f
  if [ "$1" -eq 0 ]; then
    for f in fan1 fan2 fan3; do echo auto > "$EC/$f/mode"; done
  else
    for f in fan1 fan2 fan3; do
      echo fixed > "$EC/$f/mode"; echo "$1" > "$EC/$f/level"
    done
  fi
}

TCTL=$(find_tctl) || { log "FATAL: could not find k10temp/Tctl in /sys/class/hwmon"; exit 1; }
log "strix-fan-daemon started; watching $TCTL; curve=${CURVE[*]} interval=${INTERVAL}s hyst=${HYST}C"

cur=-1            # last level actually applied (-1 = unknown/forces first apply)
ticks=0
while :; do
  if ! milli=$(cat "$TCTL" 2>/dev/null) || [ -z "$milli" ]; then
    log "WARN: Tctl read failed; leaving fans unchanged"; sleep "$INTERVAL"; continue
  fi
  t=$(( milli / 1000 ))

  want=$(desired_level "$t")
  if [ "$want" -lt "$cur" ]; then
    # Cooling: only step down if we'd still want a lower level HYST degrees hotter
    # (otherwise hold, to avoid flapping at a band edge).
    want_warm=$(desired_level "$(( t + HYST ))")
    [ "$want_warm" -ge "$cur" ] && want=$cur
  fi

  if [ "$want" -ne "$cur" ]; then
    apply_level "$want"
    if [ "$want" -eq 0 ]; then log "Tctl ${t}C -> fans AUTO"; else log "Tctl ${t}C -> fans level $want"; fi
    cur=$want
    ticks=0
  else
    ticks=$(( ticks + 1 ))
    if [ "$ticks" -ge "$HEARTBEAT" ]; then
      log "Tctl ${t}C (holding $( [ "$cur" -eq 0 ] && echo AUTO || echo "level $cur" ))"; ticks=0
    fi
  fi
  sleep "$INTERVAL"
done

The systemd unit, strix-fan-daemon.service. The comment is the one deliberate choice in here: if the daemon dies, do not drop the fans back to auto.

[Unit]
Description=Strix Halo graded Tctl fan-curve daemon
Documentation=/blog-posts
After=multi-user.target

[Service]
Type=simple
ExecStart=/usr/local/sbin/strix-fan-daemon.sh
Restart=always
RestartSec=2
# Fail safe: if the daemon dies we do NOT restore auto (that would drop the fans
# mid-load); systemd just restarts it, and fans hold wherever they were.
Nice=-5

[Install]
WantedBy=multi-user.target

And the idempotent installer, install.sh:

#!/bin/bash
# Idempotent installer for the Strix fan-curve daemon. Run as root on the system:
#   sudo ./install.sh
# Safe to re-run; it just refreshes the files and reloads the service.
set -euo pipefail

SRC="$(cd "$(dirname "$0")" && pwd)"

install -m 0755 "$SRC/strix-fan-daemon.sh"      /usr/local/sbin/strix-fan-daemon.sh
install -m 0644 "$SRC/strix-fan-daemon.service" /etc/systemd/system/strix-fan-daemon.service

systemctl daemon-reload
systemctl enable strix-fan-daemon.service
systemctl restart strix-fan-daemon.service   # restart so a re-install actually picks up script changes
sleep 1
systemctl status --no-pager strix-fan-daemon.service || true
echo
echo "Installed. Follow it live with:  journalctl -fu strix-fan-daemon"

Drop those three in a directory, then:

sudo ./install.sh
journalctl -fu strix-fan-daemon      # watch it work

The real bug was man all along

Here’s the part I’d want past-Damen to read. Every one of these three deaths already had a fix written down - ryzenadj after the first, fan-pinning and then a whole bench wrapper after the second. What kept failing was process: a fix you have to remember to apply is a fix that eventually doesn’t get applied.

So now, the protection runs as a service the box starts itself - watching temperature continuously rather than waiting on me!

The usual caveats

• Fans only - this does not cover the burst-spike death. The very first thermal crash wasn’t a slow cook; it was a 165W CPU+GPU transient that tripped a hard shutdown faster than any fan can respond. Fans can’t save you from that - only a power cap can. This daemon is deliberately fans-only, so for burst protection you still want the ryzenadj cap from the first post. What it fixes is the sustained-load death (the one that ate the 262K bench), which is the one that actually keeps happening to me.
• It’s reactive. The curve engages as Tctl climbs past 75, so there’s a brief warm-up transient at the start of a heavy load before the fans spin up - unlike pinning them by hand before you start. Fine for sustained benchmarks; if you wanted to close that gap you’d add a load-aware floor on top of the curve.
• One box, one EC. All of this is specific to the ec_su_axb35 driver on this particular miniPC. The shape of the solution generalizes; the sysfs paths and fan levels may not.

General neat learnings

• If your fans have no hwmon PWM, the standard tooling won’t help - find the vendor EC’s sysfs and drive it yourself. It’s less work than it sounds: this whole daemon is one poll loop and four config values.
• Watch the temperature that kills the box, not the convenient one. Tctl, not the EC’s polite low-reading sensor.

The least reliable component in this miniPC was always the human holding the SSH session. It’s a hundred lines of bash’s job now, and bash doesn’t forget to turn the fans on.

The MI300X behaved as expected: MTP made qwen3.6 slower at every setting. And the part that should have tipped me off sooner is that all three machines - Strix, 3090, MI300X - accepted the draft head’s guesses about 90% of the time. Same trick, same acceptance rate, and the outcome ran from a 41% win to a loss at every setting based on nothing but which GPU it ran on. My mistake was reasoning about memory bandwidth, when bandwidth is only a proxy for the variable that actually decides this - a variable that’s been sitting in the original speculative-decoding paper since 2023, which I’d have known if I’d googled before I started giving advice ;)

A brief refresher

Speculative decoding speeds up generation by having a small, fast predictor guess several tokens ahead, then letting the full model verify the whole batch in one forward pass. Every accepted guess is a token you got nearly for free. MTP is the self-speculative version: the model ships with a lightweight extra head trained to predict a few tokens out, so the draft and the verify come from the same model. The knob that matters is --spec-draft-n-max, how many tokens the head drafts before the model checks its work. On the miniPC the sweet spot was 2-4; the default of 16 was a disaster. (All of that is in the first post; this one assumes it.)

The catch with spec decoding is that every rejected guess is wasted compute, both the draft’s and the verify’s. The whole thing only pays off if the guesses are accepted often enough that the saved steps outweigh the wasted work. So you check the acceptance rate. Which is where these results stop making sense.

The MI300X: slower everywhere

Same model (qwen3.6-35B-A3B), same llama.cpp, single MI300X, decode tokens/sec, higher is better. Methodology: a fixed ~80-token prompt, 256 tokens generated, 3 reps averaged with a warm-up discarded, via llama-server’s /completion endpoint. I swept --spec-draft-n-max from 1 to 16 against the MTP-off baseline, for both a Q8_0 and a Q4_K_M trunk:

No wins there. The least-bad config (Q8_0 at n_max=3) still loses 4%, and that one had enough run-to-run variance that I’d call it a wash at best. The default n_max=16 is a 20-26% regression, reproducing the same “the default is a trap” finding from the miniPC, just without any good configuration on the other side of it to redeem it. On this hardware there is no setting I’ve found where MTP comes out ahead.

The 3090: where Damen was wrong

And now for something completely different. RTX 3090, 24 GB, CUDA build of the same llama.cpp, same qwen3.6-35B-A3B at Q4_K_M with the matching MTP head. Greedy decode so MTP-on produces byte-identical output to MTP-off (with greedy, speculative decoding is exact, so this is a controlled same-workload speed test), 256 tokens, three reps each. Decode tokens/sec:

That is a real, large win, and the reason I’d bet against it is worth digging into a little because past me wasn’t REALLY just being lazy. I’d cited an early community writeup that swept 19 spec-decode configs on a 3090 and found none faster than baseline, and that’s what I leaned on when I told you to skip it. In fairness to “past Damen”, that writeup wasn’t even testing the same mechanism: it swept a separate draft model plus ngram methods, not the self-speculative MTP head this post is about, and its own author was careful to call the negative result engine-specific to llama.cpp’s draft path - not a property of the card or the model - having seen vLLM’s MTP come out ahead. Run through the draft-mtp path, MTP on this 3090 turns out to be the best result in the series. I’ll happily take being wrong in this direction.

The peak is +41% at n_max=2, with a comfortable plateau across 2 to 4. The acceptance rate at the peak was 90.6% (164 of 181 drafted tokens accepted, measured server-side; the throughput numbers above come from the greedy llama-cli runs, so acceptance and speed are from separate invocations of the same config rather than one combined measurement). The off baseline (167 tok/s) matches an independent llama-bench run to within noise, so the speedup is genuine and not a harness artifact. Two things jump out against the MI300X. The win is bigger than the miniPC’s +18-26%, on a card with almost four times the bandwidth, which is backwards from any simple bandwidth story. And the n_max=16 default tanks it here too, down 24%.

The part that didn’t add up (until it did)

The strange part was the acceptance. It was excellent everywhere: 82-93% on the MI300X, 90.6% on the 3090, similarly high on the Strix. And that same near-90% bought a 41% speedup on the 3090, +18-26% on the miniPC, and a slowdown on the MI300X. So “is the speculation working?” (yes, accepted everywhere) turns out to be a completely different question from “is it helping?” Acceptance tells you the head is good. It says nothing about whether the saved steps are worth the added compute, and that depends entirely on the hardware.

Why: bandwidth was a proxy

This is the part I got wrong, and it’s textbook, which somehow makes it worse. I went looking in memory bandwidth, and there is a bandwidth story here, but it’s a stand-in for something more precise that’s been written down since the technique was invented.

Speculative decoding has a closed-form speedup model, from Leviathan et al. (2023), the paper that introduced it. The expected wall-clock speedup is:

       1 - α^(γ+1)
S = ──────────────────
     (1 - α)(γc + 1)

Three knobs. α is the acceptance rate - how often a drafted token survives verification. γ is how many tokens you draft per step (our --spec-draft-n-max). And c is the one I had been ignoring: the cost ratio, how long one draft pass takes relative to one verify pass. The catch is that it’s platform-dependent, a property of the model and the hardware it runs on, not the model alone.

That fact is… kind of the whole thing here. Acceptance only sets the numerator - how many tokens a good round buys you; c sets the denominator, what that round costs. I’d been obsessing over the numerator and ignoring the denominator the hardware quietly controls.

So I did the thing I should have done first: I took our measured α, γ, and speedup on each card and solved the formula backwards for the c that each result implies. At the matched setting - γ=2, acceptance ~91% on both - the cost ratio is the entire story:

There it is. At the same acceptance the 3090’s draft costs under half a verify pass while the MI300X’s effective cost is more than a whole one. A draft that is cheap relative to verification (c well under 1) leaves room for the accepted tokens to net out ahead. A draft that costs as much as the verification it is trying to save (c near or above 1) can never come out ahead no matter how good the acceptance - you can read it straight off the formula, where once γc + 1 outgrows the numerator, S drops below 1. The break-even c at ~91% acceptance is around 0.86; the 3090 sits comfortably under it, the MI300X over it.

So why is the MI300X’s c so much higher? Bandwidth comes back in here, not as the answer but as the thing that sets c. Verification batches γ+1 tokens into one forward pass, and that pass is only cheap (only about one decode step instead of γ+1 of them) if its dominant cost amortizes across the batch. Streaming the weights is a per-pass cost: load them once, verify the whole batch, done. So when decode is memory-bandwidth-bound, a γ-token verify costs about what a 1-token decode costs, the denominator’s “1” holds, and c stays small. That is the 3090. But MoE routing and attention are per-token costs - a four-token verify does four tokens’ worth of both, batched or not. On the MI300X, single-stream decode of a 3B-active MoE is not bandwidth-bound at all (it sits under 10% of the card’s 5.3 TB/s, bottlenecked on that per-token overhead instead), so verification does not amortize, the effective “1” in the denominator behaves more like γ+1, and that violation gets folded straight into c. The MI300X is just paying per-token where the 3090 pays per-pass.

And this is why a result that looks like a flat contradiction isn’t one. AMD has published up to 3× from speculative decoding on the very same MI300X, at batch size 1 - the exact configuration where I measured a loss. The difference is the model. Their wins are on dense 34B-to-70B models, and a dense 70B at batch-1 genuinely is bandwidth-bound: streaming seventy billion parameters per token is an enormous per-pass cost that amortizes beautifully, so its c stays small and the formula pays out. qwen3.6 fires only ~3B parameters per token, so it never gets bandwidth-bound on a 5.3 TB/s card, its verify will not amortize, and its c balloons. The model’s active size - not the card - decides whether you are paying per-pass or per-token. (The MoE speculative-decoding literature describes the same mechanism from the other side: under speculation a γ-token verify can activate the union of all experts those tokens route to, inflating exactly the per-token cost that will not amortize - see e.g. MoE-Spec.)

One caveat on that c column before I move on: I solved for c rather than measuring it directly, so it is an effective ratio. It lumps a genuinely expensive draft together with verification that simply didn’t amortize, and only a draft-pass/verify-pass timing breakdown, which I haven’t run, would separate the two. (More on the cross-backend caveats at the end.)

None of this is new theory. The Leviathan formula is the founding paper, and the roofline reading of it - amortize per-pass costs, pay per-token costs in full - is well-trodden. What three boxes running the identical model show is the half that’s easy to forget: at fixed acceptance, the speedup is whatever the cost ratio says, and that ratio swings by more than 2× from card to card. Bandwidth was never the variable. It was a proxy for c, and the Strix is proof the proxy leaks.

What to take from it

• The speedup has a formula, and acceptance is only one of its three terms. Whether MTP helps is set by acceptance α, draft depth γ, and the platform cost ratio c = draft-pass time over verify-pass time. c is the one that varies by hardware, and it is what decides whether the same ~90% acceptance turns into +41% or -14%. If you remember one thing past “tune n_max,” remember that the cost ratio, not the bandwidth, is what you are really up against.
• Don’t trust acceptance rate as a go/no-go signal. Roughly 90% acceptance bought +41% on one card and a net loss on another, because acceptance only sets the formula’s numerator. Measure end-to-end tokens/sec against the MTP-off baseline; that is the only number that decides it.
• The n_max=16 default is bad everywhere. It was a 3x regression on the miniPC and a 20-26% one on both the MI300X and the 3090. Wherever MTP runs, the sweet spot was 2 to 4 and the default was wrong. If you turn spec decoding on, that is the first knob to move.

The card that benefitted most was the mid-range one, because its draft was cheap relative to its verify - which bandwidth only loosely predicts. It was never about small versus big. It’s the cost ratio, and that’s a thing you can measure instead of guess.

So: if you read the last post and skipped MTP on your 3090 - sorry, go turn it on.

Caveats

• Single-stream, batch-1. This is the latency case. Under heavy concurrent batching the decode regime changes (it becomes compute-bound for a different reason), and the spec-decode tradeoff would need its own measurement. These conclusions are specific to the one-request-at-a-time case.
• The workload was CoT-heavy, which flatters MTP. All three machines were measured on a thinking-model prompt that generates long, structured chains of reasoning, and that kind of text is unusually self-predictable, which pushes acceptance toward 90% and the speedup with it. The relative comparison across machines holds, because they all saw the same favorable workload, but the absolute gains would shrink on less predictable output. Read +41% as the top of the range for this card, not a universal number.
• The three rows are not a controlled cross-hardware A/B, and c lives in the software stack as much as the silicon. Strix ran Vulkan, the MI300X ROCm, the 3090 CUDA, at differing quants, so the cross-card magnitudes carry the usual cross-backend caveat; what holds up is the sign and rough size of each result plus the within-machine off-vs-on comparisons, which were measured identically per card. The Strix is the live example of the backend mattering: an open llama.cpp issue (#23126) reports its Vulkan path on unified-memory iGPUs pays a draft/target synchronization penalty that inflates c for backend reasons, not hardware ones (it’s filed for a separate draft model, not MTP, so it may not transfer). It’s also why I don’t quote a back-computed c for the Strix at all: I never captured a clean acceptance number for it, so any c would be invented.
• One model, one architecture. qwen3.6 is a 3B-active MoE, which is part of why single-stream decode is so overhead-bound on the MI300X. A dense model would shift the compute-versus-bandwidth balance, and where it landed on each card is its own measurement.

So what happens when you go the other direction, from the miniPC to the biggest GPU you can get your hands on? What happens when VRAM, bandwidth, and “what fits” all stop being the bottleneck? How much quality do you actually lose when you quantize a model, and is the speed worth it?

The quality answer is the dull one, so let’s get it out of the way: from BF16 down to Q4_K_M I couldn’t find a difference worth reporting (on either model, whether the benchmark was code, math, instruction-following, or knowledge).

The speed answer is why I bothered. Most of what we learned about quantizing on the miniPC was a bandwidth artifact: a set of lessons that were only the case because that little machine was starving for memory, and they cease to be true on a card with bandwidth to spare. The cleanest case: quantizing harder used to make a model generate faster, and here it does nothing at all, Q8 and Q4 decoding at the same speed. A few of the old rules go further than flat and reverse outright, and chasing down why is most of what follows.

There’s one practical takeaway worth pulling right to the front (which I..... have not done here, sorry, I’m a jerk). FP8 is the format you’d actually deploy on an MI300X, and served the naive/obvious way - --quantization fp8 and nothing else - it is slower than not quantizing at all, losing to plain BF16 at the low concurrency a single self-hosted endpoint actually runs at, and only inching ahead once you’re pushing 128 streams. One environment variable that, outside AMD’s and vLLM’s own posts and threads, I’ve barely seen mentioned anywhere, VLLM_ROCM_USE_AITER=1, swings it up to 46% the other way: a bigger lever than the entire gap between FP8 and full precision. If you serve models on AMD and that’s news, then you are welcome.

Why a “quant ladder” at all??

A quick refresher: A model’s weights are stored at some numeric precision. Train and release at BF16 (16 bits per weight), then quantize down - Q8 (~8 bits), Q6, Q5, Q4 - to make the file smaller and, on most hardware, faster. The catch is supposed to be quality: fewer bits per weight means a coarser approximation of what the model learned, and at some point it should start writing worse code.

The miniPC posts established one half of this: quality is hardware-agnostic. Run the same GGUF on a Strix Halo iGPU or an A30 and you get the same benchmark score within noise, because the hardware decides how fast, not how smart. This post is the other half: hold the hardware fixed, walk down the quant ladder, and watch what happens to both quality and speed. BF16 sits at the top as ground truth - the precision the model was actually released at - and each rung below trades bits for bytes.

The MI300X is the right place to run this cleanly, for two dumb reasons: 1) I never have to offload and 2) ... I mean someone gave me access to an MI300X! On the miniPC, “what fits” is the entire story. Here, a 35B or 80B model fits in a single GPU’s 192 GiB with room to spare at every quant including BF16, so nothing spills, nothing streams over PCIe, and the only variable that moves is the precision.

The models on the ladder

Two models, both of which have appeared in this series, both Mixture-of-Experts (total parameters large, active-per-token small):

Picking these two on purpose: one mid-size reasoning model with a thinking mode, one larger coder model without. If the quant curve looks the same on both, the finding generalizes past a single architecture.

How it was served

Two serving stacks, because no single one does everything I needed cleanly on ROCm:

• GGUF K-quants via llama.cpp built for ROCm/gfx942 - this is the ladder proper (Q8_0, Q6_K, Q5_K_M, Q4_K_M), and it’s the same quant format I ran on Strix Halo, so the numbers are directly comparable across the whole series. Plain llama-quantize, no imatrix, to keep it honest and reproducible.
• vLLM (AMD’s rocm/vllm-dev container - bare-metal vLLM on ROCm 7.x is its own saga, container-only is the path) for two reference points the GGUF ladder can’t give me: the BF16 anchor and a native FP8 run, FP8 being the format you’d actually deploy in production.

The one wrinkle worth flagging up front: the BF16/FP8 reference points come from vLLM, while the Q8-to-Q4 ladder comes from llama.cpp. So a BF16-vs-Q8 comparison crosses a serving-stack boundary, and I won’t lean on it. The clean apples-to-apples quant curve is Q8 down to Q4, all llama.cpp, all the same harness - and that’s the one I draw conclusions from. BF16 and FP8 are there as sanity anchors, not as rungs.

A note on what I didn’t run: HumanEval+. I left it on the shelf on purpose, because I’ve watched my own HumanEval+ harness score the same model file anywhere from 37% to 92% just by rephrasing the prompt (nested code fences in the prompt template were enough to make a weaker quant emit a fragment instead of a function). A benchmark that swings 55 points on prompt formatting isn’t measuring the model, it’s measuring my harness, so I trust Polyglot and lm_eval here and treat HumanEval+ numbers, mine or anyone’s, with suspicion.

The big risk going in was narrower than any of this, and specific to this box. I’d been running qwen3-coder-next on llama.cpp for months on Strix Halo and the A30, so the model and its GGUFs were known-good; what was new here was the backend, llama.cpp built for ROCm/gfx942. qwen3-coder-next isn’t a plain transformer, it runs on the Qwen3-Next architecture with a hybrid/linear-attention design whose unusual kernels have needed their own fixes in llama.cpp before, and the Vulkan and CUDA paths I already trusted were no guarantee the HIP/ROCm path would compute those layers correctly. It’s the kind of model that can load up looking fine and still quietly emit garbage. So the de-risk gate was: build for gfx942, load, run one Polyglot exercise to prove the harness could still tell a right answer from a wrong one before committing to multi-hour runs. It cleared, which I don’t take for granted when an odd architecture meets a backend it hasn’t been road-tested on.

Quality: the ladder is flat

This is the half I expected to be dull, and it obliged. That quantizing down to Q4 barely touches quality is close to received wisdom by now, so I walked the ladder to confirm it on these two models, just in case of a surprise.

Here’s the coding result - Aider Polyglot, 225 multi-language exercises, pass@2 (the model gets a second shot after seeing test failures, which is how you’d actually use it):

Take a long look at the numbers and let them sink in. The qwen3-coder-next ladder spans 1.8 points top to bottom - Q5 (the 5-bit quant!) scores higher than Q8. qwen3.6 thinking-on bounces between 61 and 65 with no downward trend; Q8 and Q5 tie at the top, Q6 dips below both. These aren’t degradation curves, they’re noise. On a 225-exercise benchmark, the run-to-run wobble is bigger than anything the quantization is doing. Down to Q4_K_M - a 4-bit approximation of the weights - there is no measurable coding-quality cost on either model.

And it’s not just coding. The whole point of running lm_eval on top was to check quality from angles Polyglot can’t see - math, instruction-following, broad knowledge:

Every delta from Q8 to Q4 is under one point. (One thing I watched for, since these are reasoning-capable models: if you grade a thinking model wrong, its chain-of-thought eats the answer field and gsm8k craters to single digits - a config bug that looks like a quality cliff. gsm8k landing at 87-90% is how I know the reasoning was handled correctly and these scores are real, not artifacts of a misgraded run.) Across four benchmarks and two models, going from 16 bits down to 4 costs you essentially nothing I can measure. That’s the result everyone hopes quantization gives and rarely gets to state this cleanly, so I’ll take the clean win before the speed section complicates things.

A notion that didn’t pan out: the thinking gap doesn’t widen

qwen3.6 has a thinking on/off switch, and I ran both modes partly to chase a notion: maybe chain-of-thought compensates for quantization damage, so the gap between thinking-on and thinking-off should widen as you quantize harder (thinking would have more to fix). The first two data points teased it - the gap looked like it grew from ~9pp at Q8 to ~12pp at Q4.

Then I filled in the middle rungs, and the pattern evaporated. The thinking-on-minus-off gap across BF16/Q8/Q6/Q5/Q4 goes 5.4 / 9.3 / 4.0 / 12.9 / 11.6 points, a random walk between 4 and 13. The apparent widening was just two cherry-picked endpoints. Thinking doesn’t measurably rescue a more-quantized model, boooo. Oh well, worth a try!

Speed: where the miniPC intuition breaks

This is the part that surprised me a little. Here’s llama-bench on qwen3.6 across the ladder - prefill (processing your prompt) and decode (generating the response), tokens/sec, higher is better:

The decode column is the one to watch, and it refuses to move. Halving the bytes per weight from Q8 to Q4 takes generation from 154 tok/s to 153, a gap the run-to-run noise swallows whole. Even full F16 only sags to 139. For everything the ladder does to file size, decode speed sits still.

Strix Halo behaved nothing like this, and this is exactly the rule I opened with coming due. There, generation is bandwidth-bound, so dropping down the quant ladder genuinely speeds it up: in the MTP post this same qwen3.6 ran about 50 tok/s at Q8 and 65 at Q4 with MTP off, a clean ~30% win for nothing but going smaller. The two boxes side by side:

The same change that bought 30% on the miniPC buys nothing here. The MI300X carries so much HBM3 bandwidth that decode never waits on memory, even at Q8, and trimming the model further only frees up headroom the GPU was never short on. The “smaller model generates faster” logic I leaned on for three straight posts was really a fact about starved hardware all along, and it doesn’t survive contact with a card sitting this far below its bandwidth limit.

Prefill tells the other half of it. F16 prefill collapses, 1,303 tok/s against roughly 5,500 for every quant, so moving off full precision is a real ~4x win there (prefill leans on compute and bandwidth in a way decode doesn’t). Once you’re actually on the ladder, though, Q8 to Q4 prefill flattens out the same way decode does. The whole jump lives between F16 and Q8; everything below Q8 is rounding error.

The 80B coder-next shows the identical shape, just slower in absolute terms - F16 prefill/decode of 717/106, Q4 of 3,527/121. Same story: decode flat across quants, F16 prefill collapses, file size drops 3.3x (148 to 45 GiB) for free.

One weird rung: Q6_K is slower than Q8_0

Look again at the qwen3.6 table: Q6_K prefills at 4,374 tok/s, behind the larger Q8_0 at 5,473, even though its file is 8 GB smaller. coder-next repeats the trick (Q6_K 2,840 against Q8_0 3,425), so it isn’t a one-off. Q8_0 unpacks almost for free, a scale factor per block and not much else, while the K-quants carry a fussier block structure that costs real compute to dequantize on the way to the matmul. When bandwidth is the bottleneck, as on the miniPC, the smaller file still wins, because the bytes you save outweigh the compute you spend getting at them. Lift the bottleneck and the bill for that extra dequant work comes due, which is how the smaller quant ends up the slower one.

So when does quantizing a big GPU ever pay off?

If decode speed is flat and quality is flat, the obvious reaction is “so don’t bother quantizing on a card like this.” That’s the wrong lesson, and it’s worth going into why, because the benchmark above is rigged against quant in a way that’s easy to miss.

My llama-bench runs are batch-1: one request, one stream, nobody else on the GPU. That’s the latency case, and it’s exactly the regime where quant on this hardware does nothing. Here’s the math: At batch-1, generating a token means reading the active weights once, so the speed ceiling is roughly (memory bandwidth) / (bytes of active weights). For a 3B-active MoE at Q8, that’s about 3 GB of weights against the MI300X’s 5,325 GB/s of HBM3: a ceiling near 1,700 tok/s. I measured 154. We’re under a tenth of the way to the bandwidth wall, which means single-stream decode here is bound by overhead (kernel launches, MoE routing, attention), not by hauling weights. Halve the weight bytes with a heavier quant and you’ve cut a cost that wasn’t the bottleneck, so nothing moves.

The miniPC lives on the far side of that line. Its ~256 GB/s of LPDDR5, roughly 21x less bandwidth than the MI300X, drops the same Q8 ceiling to about 85 tok/s, and in practice it generates around 50. That machine is pinned against the wall, so shaving bytes per token nearly doubles its ceiling and you feel every byte of it.

So back to the point: when does it pay off on a big GPU? Three real cases, none of which my batch-1 latency test can see:

1. To fit on fewer GPUs, ideally one. This is about capacity, not speed, and for a big enough model it can be the biggest practical win on the list, even if neither of my two ever triggers it: qwen3.6 is 66 GB at F16 and coder-next is 148 GiB, so both fit a single 192 GB card at full precision and I never had to quantize either just to make it fit. It bites a tier up, on models too large for one card, where halving the bytes per weight is the difference between needing two cards and one. Collapsing a model onto a single GPU sidesteps tensor-parallel entirely, no cross-GPU comms on the critical path, which is usually simpler and faster per request than splitting it across several, though a future post will hopefully demonstrate that with real numbers (the multi-GPU story on this box could be its own writeup). For the models in this post it’s moot; once you’re past what a single card holds, quantizing to fit is often the highest-value move available.
2. To free memory for KV cache, which is long context and high concurrency. Weights and the KV cache share the same 192 GB. A 35B model barely dents it, but a 120B-plus model serving many users at long context becomes KV-bound, not weight-bound, and every gigabyte you don’t spend on weights is a gigabyte you can spend on more concurrent sequences or more context. Single-stream latency stays flat, but aggregate throughput climbs because you’re fitting more work on the card at once. One sharp caveat, though, and it’s the same bandwidth lesson again: this means quantizing the weights to make room for an f16 KV cache, not quantizing the KV cache itself. I checked that separately, and on this hardware quantizing the cache is its own “smaller is slower” pitfall: a q8_0 KV cache costs you ~27% of decode at 256K context and a q4_0 cache ~59%, because attention has to dequantize the cache on every step and the bandwidth that would make that free is exactly what this card has to spare. Quantize KV to fit something that otherwise wouldn’t; never quantize it expecting speed.
3. For throughput at scale, a native low-precision format like FP8, not a K-quant. As you batch up requests, decode stops being overhead-bound and becomes compute-bound, the matmuls get big enough to saturate the cores. On paper FP8 wins there on raw math: the MI300X does FP8 matrix multiply at 2,615 TFLOPS against 1,307 for BF16, a clean 2x. That’s a compute win with nothing to do with memory, and GGUF K-quants can’t touch it because they dequantize back to fp16 before the matmul. So I measured it, and there’s a real gotcha hiding in the word “FP8,” see the next section.

Measured: FP8 wastes most of its advantage by default

I ran a concurrency sweep instead of asserting claim #3: the same qwen3.6-35B, a fixed 512-in/128-out workload, closed-loop at 1/8/32/64/128 concurrent requests. BF16 and FP8 on vLLM, Q4_K_M on llama.cpp. The first FP8 column is the obvious invocation, --quantization fp8. The second adds one environment variable, VLLM_ROCM_USE_AITER=1, which turns on AMD’s optimized kernels. Output tokens/sec, higher is better:

Default FP8 is a trap on this hardware. The obvious invocation, --quantization fp8 and nothing else, is slower than just running BF16 across the low and middle of the range a single self-hosted endpoint actually runs at: 20% slower single-stream, 9% slower at eight concurrent, dead even at 32, and a measly 7% ahead even at 128. That’s a thoroughly underwhelming showing for a format with twice BF16’s theoretical FLOPS. It matches open reports of FP8 underperforming BF16 on MI300X (vLLM #31475): the stock FP8 path on ROCm leaves the win on the floor.

One flag recovers most of it. Flip on AMD’s AITER kernels (VLLM_ROCM_USE_AITER=1) and FP8 goes from “why did I bother” to 18% faster than BF16 at eight concurrent, 30% at 32, and 46% at 64. Same weights, same precision, same vLLM, one env var, up to a 46% throughput difference. If you run an FP8 endpoint on an MI300X without AITER, you are leaving roughly a quarter of your serving capacity unused and would have been faster staying on BF16 at low load. (I also tried a pre-quantized static-scale FP8 checkpoint, Qwen’s official one, in case dynamic scaling was the culprit. It made no difference. AITER was the whole story.)

So what is the flag actually doing? AITER is AMD’s pile of hand-tuned inference kernels for ROCm, a mix of assembly, Composable Kernel, and Triton; the ones that matter here are its FP8 GEMM and fused-MoE kernels. Stock vLLM on ROCm runs FP8 through generic kernels that don’t squeeze the MI300X’s FP8 throughput, and AITER swaps in kernels actually tuned for this silicon, for the matmuls and, on an MoE, the expert routing. It’s a kernel change, not a precision change, so it’s free on the quality side: you’re running the identical FP8 weights, just through math that fits the hardware (the same FP8 that lands at 59.1 Polyglot, within noise of BF16).

The catch, before you build on it: AITER is a moving target, not a set-and-forget switch. Coverage is uneven, the tuned kernels exist for some models and ops and not others (AMD’s own launch numbers were scoped to DeepSeek), and which ones are fast shifts version to version with the ROCm/vLLM container; and the FP8 win in particular only lands on CDNA3-class cards like this one, which have the native FP8 units to begin with. The honest read is “free speed for this model, on this container, today,” which is why I measured it on qwen3.6 instead of assuming the flag behaves the same everywhere.

But even tuned, it’s not the 2x in my usecase. AITER FP8 tops out around +46%, not the +100% the FLOPS spec implies. The reason is the model: a 35B-A3B MoE only fires ~3B parameters per token, so the FP8 matmul speedup applies to a sliver of the work while routing and expert-selection overhead, which FP8 does nothing for, takes a bigger share of each token. A dense model would likely see more of the 2x. So the honest ceiling for this class of model is “a third to a half faster, with the right kernels,” which is still well worth having.

And llama.cpp is a single-stream tool, not a serving stack. At one request it’s competitive (114 tok/s, ahead of default FP8). The instant you add concurrency it collapses: throughput flatlines around 270 tok/s no matter the load (vLLM scales past 3,600 with AITER, more than 10x), it starts dropping requests, and tail latency falls apart, mean time-to-first-token at 128 concurrent was over twelve seconds. GGUF on llama.cpp is the right tool for the single-user, fits-on-my-box case this whole series is about; put it under a production serving load and you want vLLM. (The failed-request counts wobble enough that I read them as “it fell over,” not a precise capacity number, but the throughput plateau and the latency blowup are unambiguous. This makes me sad, because honestly I kind of hate using vLLM hah.)

So claim #3 holds, with a big asterisk: a native format does win the compute regime where a K-quant can’t, but only if you turn on the kernels that make it work. The default doesn’t, and the gap between the default and the tuned path is bigger than the gap between FP8 and BF16 in the first place.

The thing that genuinely does not help, on any of these axes, is grabbing a bigger GGUF K-quant and expecting single-stream generation to speed up. It won’t, and as the Q6_K rung showed, it can go backwards.

So what do you actually pick

If you’re running one of these models on an MI300X (or anything with comparable HBM bandwidth), the takeaways are clean:

• Quantize to Q4_K_M without quality guilt. Two models, four benchmarks, BF16 to Q4 is flat. You’re not trading smarts for size in any way I could measure.
• Quantize for capacity, not latency. On this hardware the reason to shrink a model is to fit it on fewer GPUs (ideally one, dodging tensor-parallel) or to free memory for KV cache, not to make a single stream generate faster. Q4 gets you a 3.3x smaller model and ~4x faster prefill versus F16, but Q8-to-Q4 will not move your decode speed, because decode was never bandwidth-bound here. If single-stream tokens/sec is all you care about, Q8 and Q4 are a wash; pick on footprint.
• Don’t assume “smaller quant = faster” transfers from consumer hardware. It’s a bandwidth-starvation effect, and this box isn’t starved. Q6_K being slower than Q8_0 here is the canary: dequant compute can cost more than the bytes you saved.
• FP8 for footprint always, for throughput only with AITER. FP8 lands within noise of BF16 on quality (59.1 vs 59.6 Polyglot) and halves the model’s memory, reason enough on its own. For speed, the issue is the kernels: stock --quantization fp8 is slower than BF16 below ~32 concurrent, but VLLM_ROCM_USE_AITER=1 flips it to 30-46% faster under load. If you serve FP8 on MI300X, turn AITER on, it’s the difference between FP8 being a downgrade and a real win. And if you’re low-concurrency, BF16 is still faster single-stream, so don’t reach for FP8 expecting latency.

The throughline for the series holds and extends: quality is hardware-agnostic and quant-agnostic down to Q4. What’s hardware-specific is the speed you get for quantizing - and on a GPU that isn’t starved for bandwidth, that speed is mostly already on the table at Q8. The interesting question was never “how smart is the smaller model” (just as smart). It’s “what does the smaller model buy you on this box” - and the answer flips depending on whether the box was ever hungry for bandwidth in the first place.

Caveats and fine print

• TP=1 throughout. Everything here ran on a single MI300X, which is all these models need (they fit one card at every precision). Multi-GPU on this box is its own saga - 2- and 4-way tensor parallel do work once you get the boot flags and RCCL shared memory sorted, and the full multi-GPU story is a post of its own. Nothing here needed more than one GPU.
• The BF16/FP8 anchors are vLLM, the ladder is llama.cpp. As noted up top, I only draw the quant-curve conclusions from the Q8 to Q4 llama.cpp runs, which are internally consistent. Treat the BF16 and FP8 points as reference, not as rungs on the same ruler.
• Polyglot noise is real. 225 exercises is enough to rank models a tier apart but not enough to resolve sub-2-point quant differences - which is exactly why I’m calling the ladder “flat” rather than reading its wiggles as signal. The lm_eval breadth (sub-1pp everywhere) is what makes me confident it’s genuinely flat and not just Polyglot being coarse.
• This is two MoE models. Both are A3B (~3B active). A dense model, or a much larger active-parameter count, could push decode back toward bandwidth-bound even on this hardware. If you’re on different architecture, measure your own decode column before trusting the “flat” claim.

Here's where it landed.

Topology

flowchart TD
    classDef core fill:#1d3557,stroke:#a8dadc,color:#f1faee
    classDef infra fill:#457b9d,stroke:#a8dadc,color:#f1faee
    classDef wan fill:#2a9d8f,stroke:#1d3557,color:#f1faee
    classDef leaf fill:#f4a261,stroke:#1d3557,color:#1d3557

    ISP[/"Fiber ISP"/]:::wan
    TMO[/"T-Mobile 5G
(backup)"/]:::wan

    ISP -->|2.5 GbE| PF
    TMO -.->|failover| PF

    subgraph HOUSE [" HOUSE "]
        PF["pfSense
router / firewall
dual-WAN"]:::core
        XG8["USW Pro XG 8
core switch"]:::core
        FLEX["UniFi Flex XG
leaf switch"]:::leaf
        APH["U7 Pro AP
Wi-Fi 7"]:::leaf

        PF -->|10G DAC| XG8
        XG8 -->|10G| FLEX
        XG8 --- APH
        FLEX --- HV["Plex · Sonos
Hue · AV"]:::leaf
    end

    XG8 ==>|"10G fiber
(only house↔shed link)"| JUN

    subgraph SHED [" SHED "]
        JUN["Juniper EX4550
core switch"]:::core
        PROX["Proxmox
Dell R740"]:::infra
        NAS["TrueNAS SCALE
Dell R730xd
ZFS RAID10 (striped mirrors)"]:::infra
        APS["U7 Pro AP
Wi-Fi 7"]:::leaf
        IDRAC["iDRAC
out-of-band mgmt"]:::infra

        JUN --- PROX
        JUN ===|2x 10G LACP| NAS
        JUN --- APS
        JUN --- IDRAC
    end

The house↔shed backbone runs over OS2 fiber: a 12-strand pull, with one pair lit at 10GBASE-LR today and plenty of spare strands waiting to be put to work.

VLANs

Everything used to live on one flat /8 (yes, all of it, on the same broadcast domain, don't @ me). It's now carved into zones, each its own /16 in private address space:

Inter-zone traffic is default-deny. IoT in particular only gets DNS and NTP and is otherwise walled off from everything. The smart plugs do not need to phone home, and they certainly don't need to see the NAS.

The router is a Ship of Theseus

The firewall is still, technically, the home theater PC I bought in 2015. Eleven years of transplants later, the only original parts left are the case and (maybe) the power supply. New CPU, new board, new NICs, everything that actually does the work has been swapped out. It's still the same machine the way my grandfather's hammer is still my grandfather's hammer.

Current internals:

• CPU: Intel Core i5-12400 (it started life with a Core i7-4790K)
• Board: ASUS PRIME B760M-A D4
• RAM: 64 GB
• WAN: Intel I225-V 2.5GbE (fiber) + Realtek RTL8125 2.5GbE (T-Mobile 5G failover)
• LAN: Mellanox ConnectX-4 Lx, dual SFP28 (the 10G uplink to the rest of the network)
• Disk: NVMe SSD

It started life playing movies in a living room. It now routes two WAN links, terminates a 10G trunk, and runs pfSense for the whole property. Same hammer.

The rest of the hardware

• Core switching: UniFi USW Pro XG 8 PoE (house) and a Juniper EX4550 (shed), joined by 10G fiber, with a UniFi Flex XG as a downstream leaf.
• Hypervisor: Dell R740 running Proxmox.
• Storage: Dell R730xd plus an MD1400 SAS shelf running TrueNAS SCALE. ZFS RAID10 (striped mirrors), dual-10G LACP to the switch.
• Wi-Fi: UniFi U7 Pro (Wi-Fi 7), one per building, plus an outdoor AP for the observatory gear.

Not shown on the diagram, because it would turn into spaghetti: the Strix Halo node, a handful of Playwright test nodes, a couple of build hosts, and the usual rotating cast of throwaway VMs.

What runs on it

A mix of personal and small-business services, all self-hosted: a PeerTube instance, a couple of websites (including the one you're reading), Plex, Immich, Forgejo, Home Assistant, a PBX, authoritative DNS (PowerDNS + Kea for DHCP), and HAProxy out front doing TLS termination, plus a rotating cast of dev and AI VMs.

Pain

• commit confirmed is not optional. A plain commit on the Juniper once took down the entire network. Every change that touches the trunk now gets a rollback timer.
• sshguard cannot tell Claude Code from a brute-force bot. Point an agent at your infra and it will open a fresh SSH session for nearly every command, sshguard reads the rapid reconnects as an attack, and bans the host. Congratulations, your AI ops helper has now locked both of you out of the router. Ask me how I know.
• Lack of proper out-of-band management means you REALLY want to make sure you've got a keyboard/monitor available for when you break your firewall/router.

The three-sentence version: turned on with its default settings, MTP made generation 3x slower! Tuned for this model, it’s about 18-26% faster. The difference between the two is a single number.

What the heck is MTP??

Normally a model generates one token per forward pass: run the whole network, get one token, repeat. That’s slow. Speculative decoding speeds it up by first having a small, fast predictor - something far cheaper to run than the full model - guess the next several tokens, and then letting the full model verify that whole batch of guesses in a single forward pass. The trick is that checking several tokens at once costs the big model about the same as generating one. Every guess it accepts is a token you got essentially for free. The catch, of course, is that every rejected guess is wasted compute, both the draft’s and the verify’s. The math only pays off if the guesses are accepted often.

MTP (Multi-Token Prediction) is the self-speculative version: instead of running a second small “draft” model alongside the big one, the model ships with an extra lightweight head trained to predict a few tokens ahead. The draft and the verify come from the same model. llama.cpp added support in PR #22673 (merged 2026-05-16, build b9180 or later), exposed as --spec-type draft-mtp.

Qwen3.6 here is a 35B-A3B Mixture-of-Experts model: 35B total parameters, but only ~3B are active per token. That “A3B” part turns out to matter a lot for whether MTP helps.

The wrong path

The first thing I did, which was the first thing and not the smart thing, was the naive run: flip MTP on, leave everything at defaults, and see what stock settings buy you. It’s what most people will reach for, and the articles all quote ~2x, so why not? Generation dropped to 18.6 tok/s, down from ~60 with MTP off. So, the opposite of 2x. Definitely didn’t seem right lol.

The culprit was --spec-draft-n-max, the number of tokens the head is allowed to draft ahead before the model checks its work. It defaults to 16. Here’s what it gets you at a range of values (tok/s, higher is better):

A caveat on these numbers: they were quick and exploratory. I killed off any stray model servers but skipped the full services-down, fans-pinned protocol I used for the recommended config below, so read the absolute baseline loosely. (It also idles at ~60 here rather than the ~65 you’ll see later because this early run used a slightly slower quant upload. Why two “Q4_K_M” files run at different speeds is its own rabbit hole, maybe a future companion post.) A 3x regression dwarfs either effect, which is the whole point.

So there we have it: for me, the default was a roughly 3x regression, and the line between useless and useful is narrow. n=8 is still slower than no MTP at all, and only by n=4 does it pull ahead. I almost wrote the whole thing up as “MTP doesn’t work on Strix Halo MoE” right there. Then I did the thing I should’ve led with if I’d actually wanted good numbers: went and read how MTP works.

The knob that matters: --spec-draft-n-max

The default of 16 is calibrated for dense, instruction-tuned models, which accept long draft runs. An A3B MoE is the opposite: only ~3B parameters fire per token, the head’s predictions get rejected sooner, and every rejected draft past the acceptance point is pure waste. The community guidance for A3B-class MoEs converges on n=2 or 3. Even llama.cpp’s own MTP pull request reports its best results around 3 draft tokens at roughly 75% steady-state acceptance, nowhere near the default 16, and acceptance only falls faster as you push past that.

The other thing worth saying plainly: the reported 2x speedup you may have seen for “Qwen3.6 + MTP” comes from other setups (the PR author quotes >2x, testing on a different stack than a power-capped Strix Halo). On this box, for the 35B-A3B MoE on Vulkan, the realistic ceiling I measured is more like 1.2x. Lower acceptance means less free speed. That’s not a failure, it’s just the honest number for this class of model on this hardware, and it’s still worth having.

There’s also a second knob, --spec-draft-p-min (the minimum probability the head needs before a draft is even attempted, default 0.75). Some guides call it the most impactful parameter. On my hardware, sweeping it from 0.5 to 0.9 stayed within sampling noise, so I left it at the default. Your mileage may vary; it’s worth a quick sweep, but n_max is the one that actually moved my numbers.

The config I'd actually use

Two setups: pick by whether you care more about disk or about the last drop of quality. Both are gfx1151 on Vulkan (see the caveat at the end if you’re on something else), same clean-bench methodology as the last post.

The build. You need llama.cpp with PR #22673, so b9180 or later. Check with llama-server --help and look for draft-mtp in the --spec-type modes. Build it Vulkan-only: there’s a known foot-gun (issue #23199) where, in a dual Vulkan+ROCm build, the MTP tensors get placed on the ROCm device and MTP is silently disabled even when you asked for Vulkan. I didn’t run into this thankfully (my build is Vulkan-only), but watch out.

Q4_K_M (smaller, faster, what I’d default to). Two files from bartowski/Qwen_Qwen3.6-35B-A3B-GGUF: the model (Qwen_Qwen3.6-35B-A3B-Q4_K_M.gguf, ~20 GiB) and the MTP head as a separate file (mtp-Qwen_Qwen3.6-35B-A3B-Q4_0.gguf, ~1 GiB).

llama-server \
  --model Qwen_Qwen3.6-35B-A3B-Q4_K_M.gguf \
  --model-draft mtp-Qwen_Qwen3.6-35B-A3B-Q4_0.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  -ngl 999 -c 8192 -fa on --parallel 1 \
  -t 32 -tb 32 -ub 2048 \
  -ctk q8_0 -ctv q8_0 \
  --reasoning off \
  --host 0.0.0.0 --port 8089 --alias qwen3.6

That gets ~77 tok/s, versus ~65 with the --model-draft and --spec-type lines removed. About +18%.

Q8_0 (more disk, a hair more quality). Same flags, the Q8_0 trunk and head (Qwen_Qwen3.6-35B-A3B-Q8_0.gguf ~35 GiB, mtp-Qwen_Qwen3.6-35B-A3B-Q8_0.gguf ~2 GiB), and one important change: --spec-draft-n-max 2 instead of 3. Expected: ~63 tok/s, versus ~50 with MTP off. About +26%.

The sweet spot drops to 2 at Q8 because the trunk’s own predictions are more confident, so it accepts fewer drafts from the head, and drafting a third token just wastes work. Sweep n_max (1, 2, 3, 4) on whatever model and quant combo you land on and use whichever wins.

The head quant matters less than you’d think, and lighter is better. A Q4_0 head on the Q4_K_M trunk hit 77 tok/s; a Q8_0 head on the same trunk did 74. The bigger head’s better predictions don’t pay for the extra bandwidth they cost.

Configs that lose, for completeness: the default --spec-draft-n-max 16 (~19 tok/s, a 3x regression) and --spec-draft-n-max 8 (~28 tok/s, still slower than no MTP). The default is the trap.

If you’d rather not juggle two files, unsloth/Qwen3.6-35B-A3B-MTP-GGUF bundles the trunk and head into one (drop --model-draft, keep --spec-type draft-mtp). I measured ~75 tok/s at n=3, about 2 tok/s behind the bartowski split, with identical quality on Polyglot, gsm8k, and ifeval. Use whichever workflow you prefer.

What about the faster IQ4_XS build?

There’s an IQ4_XS build with the MTP head baked in, and a pretty wild-sounding claim for IQ4_XS+MTP on Strix Halo was making the rounds: 90.8 tok/s average, 110.6 peak. IQ4_XS is a smaller quant than Q4_K_M (about 4.25 bits per weight versus ~4.8), and since generation is bandwidth-bound, smaller CAN mean faster, so it’s a plausible claim. I downloaded it and benched it the same way as everything above, but I was not able to reproduce it.

Apples-to-apples, at the q8_0 KV cache my recommended config uses, IQ4_XS lands around 78 tok/s, a hair over the Q4_K_M + MTP setup above (~77) but inside the noise. Switching IQ4_XS to an f16 KV cache pushes it to ~81. That is a real few percent, but it is the cache talking, not the quant: f16 would lift the Q4_K_M numbers the same way. So IQ4_XS earns you a little (a smaller file), and a little more if you spend the extra memory on an f16 cache, but it is not the different league the 90+ figure implies. A raw llama-bench pass on the file came in at 73, so that is not where the headline comes from either.

My best guess for the gap to 90.8: I cap this box at 100W with ryzenadj for round-the-clock thermal stability (the “I melted it twice” saga from the last post). Run the chip hotter, or on a newer build, and you would probably claw some of it back. Worth a shot if you have the thermal headroom. But on a power-limited Strix Halo it is a modest step over the Q4_K_M + MTP config, not a leap, and nothing I measured got close to 90. There are some thermal upgrades I may attempt to make, and maybe I’ll revisit this with a higher cap if I do.

This is a Strix Halo / Vulkan result

The numbers above are gfx1151 on Vulkan. On CUDA, an early community writeup for this same model found no net speedup from llama.cpp’s speculative-decoding paths: it tested 19 configurations on an RTX 3090 and found none faster than baseline (the same author’s HackMD notes lay out the detail). Note that’s a llama.cpp-specific result, the same author found vLLM’s MTP faster on the same card. So the speedup here may be specific to the Vulkan path or to the unified-memory architecture. If you’re on a 3090 or an A100, don’t expect these numbers (Damen's Update: This is TERRIBLE ADVICE!!!), and if you measure your own, publish them: there’s a bit of public A3B-MoE spec-decode data now (mostly CUDA, plus a Strix Halo ROCm run or two), but I couldn’t find a single A3B-on-gfx1151-via-Vulkan benchmark out there, so that corner is wide open.

The takeaway

MTP on this hardware is a real but modest gain: ~18-26%, not the 2x you’ll see quoted for the dense model. And the default config is actively harmful on an A3B MoE. If you take one thing from this: turn --spec-draft-n-max down to 2 or 3 before you decide whether MTP works for you.

The short version: I didn't lie! It is roughly like Sonnet when it's a bit slow. BUT, things have changed and now you should use qwen3.6. Also there are some other neat findings ahead ;)

The longer version: quality is hardware-agnostic, which is exactly what you'd expect (run the same model on a Strix Halo APU or a datacenter A30 and you get the same benchmark score within noise, because the hardware should decide how fast, not how smart). The interesting questions were always about speed and fit, and that is where the miniPC shines!

Locally-hosted qwen3.6-thinking (a 35B-A3B Mixture-of-Experts (MoE) model, that is, 35B total parameters with only ~3B activated per token) scores 62.2% on the Aider Polyglot benchmark, sitting right between Claude Sonnet 4 thinking (61.3%) and Claude 3.7 Sonnet thinking (64.9%), and it does that at 45 tok/s with a 65K-token context. That is Sonnet-class capability, at usable speeds, on a ~$2.5K box. It is not Opus-class (72.0%) or GPT-5-class (88.0%), but Sonnet-class coding on hardware I own, at no per-token cost and with nothing leaving my network.

The surprises were both about memory, not compute. First: even when a model fits comfortably in the A30's VRAM, the A30 can still lose at long context. The A30 wins at short context, but on qwen3-30b-a3b at 65K depth the miniPC outruns it by 36% (28.1 vs 20.6 tok/s), because the A30's 24 GiB runs out of room for the KV cache (the attention key/value store that grows with every token of context) while the miniPC's 128 GiB of unified memory shrugs it off (the crossover lands somewhere between 8K and 32K). qwen3-30b-a3b is only 17 GiB, so this isn't even an offload problem, it's all running on the GPU, the A30 just can't hold a big model and a big KV cache at once.

The bigger surprise was the hybrid cliff. For any model that doesn't fit in 24 GiB at all, the miniPC wins outright. qwen3-coder-next (80B-A3B) has to be split across GPU and CPU on the A30, and once half its layers (plus their KV) live in slow DDR4 system memory and get streamed over PCIe every token, it crawls: 3.9 tok/s at 65K against Strix Halo's 38, almost 10x. That gap, a unified-memory box versus a GPU forced into hybrid offload, is the whole reason I bought the thing.

The recommendation has also moved since the last post. Back then, qwen3-coder-next was the best local-fitting coding model I had, and I called its quality "something like Sonnet 4.5 on a slow day." That was ... close enough that I won't say I was wrong, hah (it lands about 9pp below baseline Sonnet 4), but qwen3.6 has since released, and on the same Strix Halo hardware it is straightforwardly better: higher quality, faster, and a smaller VRAM footprint. If you have been running qwen3-coder-next since my last post, switch to qwen3.6.

The contestants

The "what fits" row is the most important and most under-discussed difference, with one clarification: it means fits entirely in VRAM, at full speed. You can push a bigger model onto the A30 with hybrid GPU/CPU offload (I do exactly that later in the post), it just runs much slower once part of it spills out of the 24 GiB. The miniPC's unified memory has no such cliff: anything up to 128 GiB loads and runs on the iGPU at full speed, including a 120B model the A30 can't hold in VRAM at all.

It's worth noting what's doing the spilling on the A30 side: its host is an older Dell R740 on DDR4, so when a model overflows into system RAM, DDR4 bandwidth is the bottleneck. A newer DDR5 host with more memory channels would lift the hybrid numbers, but it would also cost more, which is sort of the point: the unified-memory miniPC sidesteps the spill entirely, for the price of a single mid-range box.

The benchmarks

Four things, each measuring something different:

• Aider Polyglot, 225 multi-language Exercism coding exercises, the model is asked to edit existing files to make tests pass. This is the only benchmark on the list that resembles real-world agentic coding work, and it's the one frontier models actually struggle with. Not saturated.
• HumanEval+, function-level code generation, 164 problems. Top models all score 90%+. Saturated.
• lm_eval (gsm8k, ifeval, mmlu_pro), knowledge and instruction-following at single-prompt level. Frontier models saturate this too.
• llama-bench, pure throughput, no quality signal. Two numbers matter, both in tokens/sec: pp (prompt processing, how fast the model ingests your prompt) and tg (token generation, how fast it writes the reply). I report the defaults pp512 / tg128 (a 512-token prompt, 128 generated tokens) plus depth tests for long-context behavior.

I treat Polyglot as the load-bearing quality metric because (a) it actually discriminates, and (b) it's what I care about, agentic coding is what these boxes get used for in practice. If there's a benchmark I didn't run, it is because I don't know about it or didn't think of it.

One reading convention for every table below: higher is better, unless I explicitly say otherwise.

Quality: Strix Halo vs A30

Nobody expects a model to get smarter or dumber depending on whether it runs on AMD or NVIDIA silicon, and it doesn't. This was never really an open question. But I had both boxes and was running the benchmarks anyway, so I figured I'd confirm it and see where any real differences showed up. The answer: quality is the same within noise, with one model-specific surprise. Boring but necessary setup for everything that follows.

("p@1+" is pass@1 on EvalPlus's extended test set, meaning the model's first answer has to pass every test.)

Within noise on HumanEval+ across the board, and on the qwen2.5-coder Polyglot row. The qwen3.6 Polyglot row shows a 6.7pp cross-host gap (53.8% Strix Halo vs 47.1% A30), which is larger than I'd expect from pure sampling noise; possibly a real CUDA-vs-Vulkan difference for that specific model and harness, or a build-version skew between the two boxes. The HumanEval+ gemma/qwen2.5/qwen3-30b rows on the same model files agree exactly cross-host, so it isn't a general "the A30 produces worse logits" pattern; it's a qwen3.6-Polyglot-specific finding I'd want to dig into in a future bench.

So the model is mostly the model, and hardware doesn't make it dumber in any general sense. There can be model-specific cross-host quirks worth checking (this one came as a surprise to me), but for the typical case, once you've picked a model that fits, the hardware question reduces to how fast and can it even fit.

Quality: Local vs the Frontier

Here's where it gets fun. I'm going to split this into coding and non-coding because they behave very differently.

Coding-specific: Aider Polyglot

Polyglot is the benchmark where frontier models still have headroom, and the one that tracks "how good is this thing as a coding agent." Here's the comparison (Aider leaderboard scores for the API models, my results for local):

* The 23× gap between local gpt-oss-120B (1.8%) and the same model's API leaderboard score (41.8%) is almost certainly the reasoning_effort parameter not wiring through to llama.cpp's gpt-oss path: low/medium/high produce near-identical outputs within sampling noise. For a model whose top-line capability is its reasoning depth, a broken reasoning knob is a broken model. Full discussion in item 3 below.

(All API model scores in this table come from the Aider Polyglot leaderboard, last updated 2025-11-20. A few newer frontier releases (Google's Gemini 3 and Anthropic's Claude Opus 4.5 / 4.7) exist but haven't been scored by the Aider team yet, so they aren't represented above. The most recent Gemini and Opus variants the leaderboard does have are Gemini 2.5 Pro 32k-think at 83.1% and Claude Opus 4 32k-think at 72.0%.)

What this shows:

1. Sonnet-class is achievable locally, in both thinking and no-think modes. My best local model (qwen3.6-thinking, a 35B-A3B MoE) sits right in the Claude Sonnet thinking band (62.2% vs Sonnet 4 thinking 61.3%). And on the apples-to-apples no-think comparison, qwen3.6 with thinking off (53.8%) is just 2.6pp under Claude Sonnet 4 no-think (56.4%); effectively tied within Polyglot's noise floor. So it's not just "Sonnet-class when allowed to think"; it's "Sonnet-class without needing to think." That second result was the bigger surprise.

2. The recommendation has moved since my last post. Back then, qwen3-coder-next (80B-A3B) was the best local-fitting coding model I had, and the explicit subject of the previous post. qwen3.6 didn't exist yet. Now it does, and it's straightforwardly better: 53.8% Polyglot at thinking-off (vs qwen3-coder-next's 47.6%), 62.2% at thinking-on, smaller VRAM footprint, faster throughput. If you've been running qwen3-coder-next on Strix Halo since my last post: try qwen3.6.

3. The real gap is to GPT-5, Gemini 2.5 Pro, and Claude Opus. Those three are ~10-26pp ahead of my best local model. The Anthropic ladder is worth calling out specifically: qwen3.6-thinking (62.2%) is essentially tied with Sonnet 4 thinking (61.3%), but Anthropic's actual flagship is Opus, which scores 72.0%, about 10pp ahead of local. Then GPT-5 (88.0%) and Gemini 2.5 Pro thinking (83.1%) are the real top of the leaderboard. DeepSeek V3.2 Reasoner (74.2%) is the closest open-weight to that band, but at ~700B parameters it won't fit on either of my boxes.

4. Local quants underperform their API counterparts catastrophically on some models. My local gpt-oss-120B Q4_K_M scored 1.8%; the leaderboard's gpt-oss-120b (high) scored 41.8%. That's a 23x gap, not a small one. Three things contribute: quantization, the reasoning_effort parameter doesn't actually wire through to the model on llama.cpp (I verified this; low/medium/high produce near-identical outputs within sampling noise), and I used Aider's whole edit format vs the leaderboard's diff. The reasoning-effort issue is probably the biggest factor; gpt-oss is essentially a reasoning model, and if the reasoning depth knob is broken, the model is operating in something close to a "low effort" mode regardless of what you pass in.

5. Thinking mode is meaningful when measured correctly. qwen3.6 without thinking: 53.8%. With thinking: 62.2%. That's 8.4 pp of capability sitting behind a flag.

A caveat about polyglot versioning

My test harness ran 225 exercises on most models, the same set as Aider's leaderboard. A few runs got 289 or 450 (multiple attempts per exercise from a config tweak); rates are still computed as passed/total. Edit-format matters too: I used whole because it's more robust to weaker models, while Aider's leaderboard uses diff because it gets better scores from the top models. whole is generally a slight handicap. Treat the comparisons as directional, not exact.

Methodology note: all HumanEval+ numbers in this post come from evalplus.codegen, the canonical scorer behind EvalPlus's published leaderboard.

Saturated benchmarks: HumanEval+ and lm_eval

These are the benches where frontier and local models all score in the same 85-95% range, they don't discriminate well anymore. Quick look:

Cross-host gsm8k + ifeval, identical Q4_K_M quantization, identical chat-completions API, 200 items each:

The cross-host rows agree within 1-2 percentage points across the board. Same model, same quantization, same prompt, same score within noise.

Notable observations:

• qwen3.6-thinking tops gsm8k at 96.5%, better than any A30 result in this set, and in the same ~95-97% band frontier models hit on saturated math benches before the frontier moved on to AIME / FrontierMath. On a 35B-A3B MoE running on a miniPC.
• gpt-oss-20b ifeval at 25.5% is shockingly low for a model that hits 87.5% on gsm8k in the same run. This is the --reasoning off configuration. The other gpt-oss-20b runs in my data, reasoning on variants, also fall in the 25-36% ifeval band, so this isn't a reasoning-flag artifact; gpt-oss-20b just struggles with strict prompt-following regardless. Worth knowing if you were planning to deploy it for instruction-bound tasks.
• phi-4 inverted profile: 90.5% gsm8k but only 56.5% ifeval. It's a math-strong, instruction-weaker model. Useful data point for choosing models by use case.

Same story as the coding benches: local models are basically tied with frontier on saturated benchmarks. A Qwen3-30B-A3B on a miniPC scores 95.5% on gsm8k, comfortably in the same band as any frontier model that's still being measured against gsm8k. The frontier moat still exists, but it's on real-world agentic coding (Polyglot).

Quality at depth: NIAH

Throughput at 65K context is meaningless if the model can't actually find anything at 65K. I tested needle-in-haystack retrieval (single-needle: ask "what's the best thing to do in San Francisco?" with a sandwich-and-Dolores-Park needle planted at 10%, 50%, or 90% depth in a haystack of Paul Graham essays):

Top-row finding: Strix Halo's qwen3.6-thinking and qwen3-coder-next both score perfect retrieval at 60K context, with response times of 1-2 min per query. The model isn't just running with that context, it's actually using it. Combined with the throughput numbers, this is what makes the miniPC a real coding-agent target rather than a benchmark curiosity.

Speed: default throughput

Quality matters; speed matters more than people think. A 62% model running at 1 tok/s is unusable. A 50% model at 80 tok/s is a daily driver.

(Methodology note before the tables: every Strix Halo throughput number below was collected with no other model servers running, fans pinned to max, and free memory verified before each run. There's a bench wrapper now that refuses to start without those conditions met. I ended up writing it after melting the poor machine twice, details in What I'd do differently below.)

Default pp512 / tg128 numbers (Q4_K_M, -fa 1, Strix Halo on q8_0 KV / A30 on q4_0 KV, see longctx section for the protocol note). Throughput is in tokens/sec, so higher is better. The last column is the one place bigger isn't better: it's the A30/miniPC tg ratio, where above 1.0 means the A30 is faster and below 1.0 means the miniPC wins (I flag those rows inline).

† The A30 row for qwen3-coder-next is hybrid GPU/CPU offload (22 of 49 layers on GPU, the rest on CPU/RAM). The 45 GiB Q4_K_M model can't fit fully in 24 GiB VRAM, so this is what you get if you force it onto the A30 anyway, the apples-to-apples speed cost of exceeding the VRAM ceiling on a dedicated GPU.

Two stories here:

1. A30 wins at default by 2-3x. Expected, a dedicated GPU with proper VRAM and CUDA kernels should beat an APU running Vulkan. The factor is consistent across dense models in the 2.2-2.8x range.

2. MoE narrows the gap and makes the miniPC viable. Look at qwen3-30b-a3b-2507: A30/Strix Halo ratio is just 1.56x for tg, the smallest gap in the table among the bigger models. That's because the model only activates ~3B params per token. Memory bandwidth matters more than raw compute for tg, and Strix Halo's UMA gives it surprisingly good bandwidth for active-parameter-light workloads. (The 4B models also show ratios below 2x, small models stop benefiting from the A30's compute headroom because they're already bandwidth-bound on both boxes.)

Compare that to the dense qwen2.5-coder-32b: 11.1 tok/s on Strix Halo vs 24.1 on A30, still a 2.18x gap but the absolute number is terrible on Strix Halo. I don't know about the rest of you, but 11 tok/s on a 32B dense model is not exactly what I'd call "usable". I'd never reach for the dense coder if a comparable-quality MoE exists.

Speed at long context

Now the fun part. Wait, I already said that. Another fun part! Coding agents send long context (the codebase, the test results, previous turns), so what happens when you push the depth?

I ran the same pp512 / tg128 test at depths 0 / 8K / 32K / 65K. Strix Halo is benched with q8_0 KV cache (matches how the production llama-servers are deployed). A30's previously-collected longctx sweep was at q4_0 KV; the small protocol asymmetry is mildly conservative for Strix Halo at depth (q4_0 saves a bit of KV bandwidth at the cost of dequant overhead, within MC noise on this hardware, but if anything it shaves a few percent off the Strix Halo side at deep contexts).

gpt-oss-20b (fits both)

A30 tg dropped 58% from default to 65K depth (130 to 55 tok/s). Strix Halo tg dropped 44% over the same range (81 to 46 tok/s). A30 still wins on this model at every depth, but the lead shrinks dramatically as context grows, the A30/Strix Halo ratio compresses from 1.61x at default to 1.20x at 65K.

qwen3-30b-a3b-2507 (fits both)

This is where it gets spicy. A30 tg dropped 85% from default to 65K (136 to 21 tok/s), the 24 GiB VRAM ran out of room for a meaningful KV cache at depth. Strix Halo tg dropped 68% over the same range (87 to 28 tok/s), painful but consistent. Crossover happens between 8K and 32K context. At 32K the miniPC is already faster; at 65K it's 36% faster than the dedicated GPU.

The model itself is 17 GiB Q4_K_M. The A30 has 24 GiB of VRAM. At 65K context the KV cache plus activations plus the model are competing for that 7 GiB headroom, and CUDA's memory management gets bottlenecked. Strix Halo's 128 GiB UMA doesn't care, there's so much memory headroom that the only constraint is compute and bandwidth, both of which degrade gracefully.

qwen3.6-thinking (Strix Halo only, Sonnet-tier model)

This is the model I'd actually use for coding. The numbers are remarkable:

tg drops 32% from default to 65K depth (67.1 to 45.5 tok/s). A Sonnet-class model running locally at 45 tok/s with a 65K-token context window. That's actually usable for serious agentic coding, you can pack a meaningful chunk of a codebase into the context and not pay a brutal speed tax for it.

A note on Q8_0: I also ran the no-think qwen3.6 at Q8_0 (38 GiB on disk vs Q4_K_M's 20 GiB). Polyglot moved from 53.8% to 56.9%, a ~3 pp gain. Throughput dropped from 65 tok/s to 50 tok/s at default and is similarly proportional at depth. So if you have the disk and want every last point of Polyglot, Q8_0 is a real upgrade. If you'd rather have the speed, Q4_K_M is the right call, the quality gap is small relative to the speed cost.

qwen3-coder-next 80B-A3B (Strix Halo UMA vs A30 hybrid offload)

The 80B-A3B that motivated the last post. At 45 GiB Q4_K_M it doesn't fit in 24 GiB VRAM, so the A30 column here is hybrid GPU/CPU offload (-ngl 22, 22 of 49 layers on GPU, the rest streamed from system RAM). Strix Halo's 128 GiB UMA swallows the full model and runs entirely on the iGPU:

This is the clearest "wrong tool for the job" result I had. The A30 is a good card, it just doesn't have enough VRAM to hold the model, and PCIe bandwidth between GPU and host RAM is roughly 30x slower than the A30's own HBM2. So every token has to drag activations across that bottleneck.

The math: A30 hybrid tg falls from 12.2 to 3.9 tok/s (a 68% drop) over the depth sweep, while Strix Halo's UMA tg falls from 56.4 to 38.1 (only 32%). The A30 falls off twice as steeply because attention has to read the full KV cache to produce each new token, and on hybrid mode roughly half the model's layers, plus their slice of the KV cache, live in CPU RAM (DDR4, on this server). Each token's attention op pays PCIe-bandwidth overhead, and that overhead scales with context length. So 4.6× at default and 9.8× at 65K.

On the Strix system the story is the other way around: the iGPU has the same bandwidth to all 128 GiB as it does to the first 16 GiB. There's no VRAM cliff to fall off because there's no VRAM/RAM distinction at all. tg drops 32% from default to 65K (56.4 to 38.1 tok/s), painful but consistent, and at 38 tok/s with 65K of context loaded it's still... not fast, but usable.

(I also tried to run Aider Polyglot on A30 hybrid for a quality cross-check; the harness's per-call timeout repeatedly fired against the 3.9-9.3 tok/s hybrid response rate, and I abandoned the run after 9 of 225 exercises in ~5 hours. Throughput data above is from llama-bench directly, which doesn't have that problem.)

DeepSeek-Coder-V2-Lite, the bonus weird result

I benchmarked this one for completeness, expecting nothing exciting. Instead I found one of the clearest "the dedicated GPU is broken here" results in the whole sweep. DeepSeek-V2's Multi-head Latent Attention (MLA) uses a low-rank-projected KV cache that's smaller than standard MHA but requires a different attention kernel. The CUDA implementation in llama.cpp build 9064 falls off a cliff once any KV is present:

The A30 bench actually wedged my harness, at d=32K, the CUDA kernel grinds at ~3-5 tok/s prefill, which means a single measurement of the 32K-token prefill would take 100+ minutes. I killed it after 17 minutes of no progress.

Strix Halo's Vulkan path handles MLA at depth normally, degrading from 106 to 17 tok/s tg is a real cliff, but it's a finite one and the bench actually finishes. Even at d=0 Strix Halo is 4× faster on pp512 (1641 vs 408), and that's before any KV is in play. The CUDA backend isn't just slow at depth on this architecture, it's just slow on this architecture.

This isn't a hardware issue, I don't think, it's a software bug. Presumably to be fixed in some future llama.cpp release lol. But for anyone considering DeepSeek-V2-family models for coding right now the miniPC is the only sensible target. A 24 GiB A30 will load the model just fine and then be fairly unusable.

Bonus models (Strix Halo coverage only)

For completeness, three more models I benchmarked on Strix Halo to fill out the table:

What each box is actually best for

• Strix Halo as a coding agent: qwen3.6 with thinking on when I want quality, qwen3.6 with thinking off when I want speed/quality balance. Same model file, same throughput, just flip the --reasoning flag.
• A30 for serving small concurrent requests: gpt-oss-20b at 130 tok/s or qwen3-30b-a3b at 136 tok/s is great for embeddings, rerank, and utility models in a stack.

These are different jobs. The boxes aren't substitutes; they're complements.

What I'd do differently

1. Update everything to the latest first. I spent a week chasing scores that looked too low only to realize my llama.cpp was 700 commits behind on reasoning-channel handling. Thinking models scored 0% on lm_eval because the reasoning content was consuming the entire context budget. A rebuild fixed it. This stuff moves fast, llama.cpp lands fixes weekly, so pull and rebuild to the latest before you trust a single number.

2. Bench with -d from the start, not -c. The -c arg got removed from llama-bench in recent builds; the replacement is -d for testing tg at a given KV depth. My first A30 long-context sweep died at parse time. Trivial fix in retrospect, but it cost me half a day.

3. Don't trust HumanEval+ as a discriminator. Everything competent scores 85%+. The bench doesn't separate "okay" from "great." Polyglot is what actually matters; I should have run it first.

4. Run whole and diff edit formats both. I ran everything in whole because it's robust for weak models. That makes the strong-model comparisons against Aider's leaderboard (which uses diff) slightly unfair to the local models. Doing both would have given a cleaner local-vs-API comparison.

5. Treat thermals and bench cleanliness as first-class concerns. Two specific traps cost me roughly a week of redo work:

   • Don't re-make the same thermal mistakes as last time. I already worked this box's thermals out in the last post: sustained GPU load trips it unless you cap power with ryzenadj and pin the fans manually, because the stock fan curve is tuned for desktop bursts, not back-to-back benchmarks holding the GPU near 100% for minutes at a time. Then I forgot to actually turn any of that on before kicking off a multi-hour sweep, and crashed the box twice (no kernel log, just unreachable until a power-cycle) rediscovering a lesson I'd already written down. The fix was the one I already had on the shelf: mode=fixed level=5 on all three fans (under /sys/class/ec_su_axb35/fan*/) before any sustained workload. The wrapper now refuses to start a bench unless the fans are confirmed above 3500 RPM.
   • Keep other model servers cleared out the whole time, not just at the start. Any concurrent llama-server process --mlock's its model into RAM and steals memory bandwidth from the bench. I caught this when a spot-check tg128 re-run came in 5% higher than the recorded number with everything else stopped. Five percent is small enough to miss in a single run and big enough to materially change rankings across models. The real trap is that it's easy to start clean and then let stray servers creep back in over a long session, so the fix isn't a one-time cleanup, it's re-verifying nothing else is loaded before every single run. Every Strix Halo throughput number in this post was collected that way, and the wrapper enforces it as a precondition.

   The meta lesson: a bench harness that requires you to remember the discipline will eventually run dirty. Make the harness refuse to run unless the conditions are met.

   The end result

   Use the right tool for the job. Shocking, I know.

   The miniPC can be a Sonnet-tier coding agent (when running the right model) that costs about $2,500 once and never sends my code anywhere. The A30 box is for smaller task-specific models that need high throughput.

   The local-vs-frontier gap is still real on the hardest problems and on real agentic Polyglot work, but it's roughly Sonnet-class for daily-driver coding tasks, and the gap is closing. The next time someone benchmarks this, I expect the frontier-API moat to be at least a little bit smaller.

   ---

   *Footnote: the reasoning_effort parameter not wiring through to llama.cpp's gpt-oss path is documented elsewhere; I verified by running effort=low/medium/high through lm_eval gsm8k and getting near-identical scores (90% / 86% / 86%), within the sampling noise band for a 200-item subset. If the flag were actually doing anything, I'd expect monotonic improvement from low to high; instead "high" is the same as "medium" and "low" comes out higher than both, which only makes sense if all three are effectively the same configuration plus sampling noise. A separate post about this might be coming.*

   ]]> https://damenknight.com/running-frontier-coding-model-mini-pc/ https://damenknight.com/running-frontier-coding-model-mini-pc/ Thu, 12 Mar 2026 12:00:00 GMT Homelab Projects AI I got Qwen3-Coder-Next (80B MoE) running at 46 tok/s on a under-$3K mini PC. It took a full OS reinstall, a firmware downgrade, kernel parameter archaeology, a… TL;DR: I got Qwen3-Coder-Next (80B MoE) running at 46 tok/s on an under-$3K mini PC. It took a full OS reinstall, a firmware downgrade, kernel parameter archaeology, a thermal crisis, and throwing out about half the tuning advice I found online. Here's everything I learned the hard way.

   Why This Hardware

   My existing GPU setups didn't have enough VRAM to run some of the larger models I was interested in testing. Discrete GPUs with 48+ GB of VRAM are absurdly expensive, and splitting a model across multiple consumer cards comes with its own headaches and PCIe bottleneck tax. So I started looking into UMA (Unified Memory Architecture) systems - where the CPU and GPU share the same memory pool - as a significantly more affordable way to get a ton of usable memory for inference.

   That led me to the Ryzen AI MAX+ 395. It's a weird chip - a laptop/mini-PC APU with 32 Zen 5 cores, a 40-CU RDNA 3.5 iGPU, and support for up to 128 GB of LPDDR5 unified memory. Since the CPU and GPU share the same pool, the GPU can address all 128 GB without PCIe bottlenecks. For LLM inference, where model weights need to stream through the compute units every single token, that's a huge deal.

   The theoretical memory bandwidth is 256 GB/s (LPDDR5X-8000 on a 256-bit bus). In practice I measured around 212-215 GB/s - about 82% efficiency. That's slower than an M4 Max (~546 GB/s) but faster than trying to cram a 70B model across two consumer GPUs and eating the PCIe tax.

   The GMKtec NucBox EVO-X2 packages this chip into a mini PC chassis for under $3K with 128 GB RAM - though with the way LPDDR5 prices have been going lately, check current pricing before you get too excited. There are a few other options with this chip: Framework makes a Desktop, ASUS has the ROG Flow Z13 tablet, and Minisforum has the EliteMini AI Max. The GMKtec was the best price-to-performance option I found at the time, but it's worth shopping around.

   The OS: Rocky Linux 9.7

   I'm running Rocky Linux 9.7 - enterprise stability, good package ecosystem, SELinux actually works properly. Any RHEL 9 derivative should work similarly.

   The Three Things That Must Be Right

   After the base OS was clean, I hit a wall. A really frustrating wall. Getting this hardware working properly requires three specific things to be correct - the right kernel, the right firmware, and thermal power limits that won't let the system cook itself to death. I'm going to cover all three here because skipping any one of them will ruin your day.

   1. Kernel 6.18.4 or newer

   The KFD (Kernel Fusion Driver) in older kernels has a page table bug specific to gfx1151. Any GPU tensor allocation triggers "Memory access fault: Page not present" errors. This was fixed upstream in kernel 6.18.4. Rocky 9's stock kernel is 6.12, which is too old.

   I tried AMD's amdgpu-dkms package first (which backports the amdgpu driver to older kernels), but the DKMS version is pre-6.18 and doesn't include the KFD fix. No combination of kernel parameters - HSA_ENABLE_SDMA=0, amd_iommu=off, amdgpu.noretry=0, amdgpu.cwsr_enable=0 - works around it. Trust me, I tried them all. You need the actual kernel fix.

   The solution: ELRepo's kernel-ml package, which provides mainline kernels packaged for RHEL/Rocky. I installed 6.19.6 and it just worked.

   sudo dnf install -y elrepo-release
   sudo rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
   sudo dnf --enablerepo=elrepo-kernel install -y kernel-ml
   

   2. MES firmware version 0x80

   Even with kernel 6.19.6, I was still getting page faults. Cool. The second half of the puzzle is the MES (Micro Engine Scheduler) firmware. Rocky's linux-firmware-20260130 package ships MES version 0x83, which is known to cause ROCm page faults on Strix Halo. The upstream linux-firmware repository explicitly reverted it with the commit message: "MES FW 0x83 is reported to cause ROCm page faults."

   Rocky hadn't picked up the revert yet, and AMD's own amdgpu-dkms-firmware package also ships 0x83. So the fix is manual:

   # Download good firmware (version 0x80) from upstream revert commit
   curl -sL -o /tmp/gc_11_5_1_mes1.bin \
     "https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/amdgpu/gc_11_5_1_mes1.bin?id=c092c7487eb7c3d58697f490ff605bc38f4cc947"
   curl -sL -o /tmp/gc_11_5_1_mes_2.bin \
     "https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/amdgpu/gc_11_5_1_mes_2.bin?id=c092c7487eb7c3d58697f490ff605bc38f4cc947"
   
   # Install to updates dir (takes priority over base firmware)
   sudo cp /tmp/gc_11_5_1_mes1.bin /lib/firmware/updates/amdgpu/
   sudo cp /tmp/gc_11_5_1_mes_2.bin /lib/firmware/updates/amdgpu/
   
   # Rebuild initramfs and reboot
   sudo dracut --force /boot/initramfs-$(uname -r).img $(uname -r)
   sudo reboot
   

   Verify after reboot:

   sudo cat /sys/kernel/debug/dri/0/amdgpu_firmware_info | grep MES
   # Good: firmware version: 0x00000080
   # Bad:  firmware version: 0x00000083
   

   Once both pieces were in place, PyTorch passed all validation checks: tensor operations, all data types (fp32, fp16, bf16, int8), 4 GiB memory allocation, and ~1.05 TFLOPS on a 4096x4096 FP32 matmul. Finally.

   Lesson learned the hard way: Pin your firmware. I added exclude=linux-firmware* amdgpu-dkms-firmware* to /etc/dnf/dnf.conf to prevent package updates from sneaking MES 0x83 back in. Ask me how I know.

   3. Thermal Power Limits

   This one might be the most important of the three, so don't skip it.

   While setting up a PyTorch benchmarking suite, the system started dying on me. At first I figured "oh weird, the host crashed" - but when I went to check on it, it wasn't just locked up. It was fully powered off. That's... not normal. Then it happened again. And again. Full hard power-off events with no warning, no logs, nothing.

   I set up thermal monitoring logging every 5 seconds and caught the cause:

   19:00:07  Tctl=71°C   pwr=92W    ← normal inference
   19:00:12  Tctl=91°C   pwr=165W   ← torch.compile spike
   19:00:22  Tctl=93°C   pwr=164W   ← approaching TjMax (100°C)
   19:00:27  Tctl=61°C   pwr=30W    ← thermal shutdown
   

   torch.compile triggers Triton/Inductor kernel compilation that simultaneously hammers all 32 CPU cores and the GPU. On a UMA APU where everything shares one thermal envelope in a mini PC chassis, that produces a 165W power spike - way past the 120W PPT Fast limit and far more than the little cooler can handle. The firmware thermal protection kicks in and just kills power. No graceful shutdown, just off.

   Normal LLM inference is totally fine - 73-75W, 76-80°C, perfectly stable all day long. But the moment you hit a mixed CPU+GPU burst workload, you're rolling the dice. And it's not just torch.compile - anything that pegs the CPU and GPU simultaneously in this chassis can trigger it. I lost count of how many times the system just cut out on me before I got this sorted.

   The fix is ryzenadj, a tool that lets you adjust AMD mobile power limits from Linux:

   sudo ryzenadj --fast-limit=100000 --tctl-temp=88
   

   This caps burst power to 100W and sets the thermal target to 88°C, giving 12°C of headroom before TjMax. Do this immediately after your first boot, before you run anything heavy. I created a systemd service to persist these limits across reboots so they're always active. The GMKtec ships with BIOS 1.12 / EC 1.10 (the latest available), so there's no firmware fix coming - you've gotta manage this in software.

   Other thermal improvements people recommend but I haven't tried yet: replacing the stock thermal paste with PTM7950 phase-change material, and the ec_su_axb35 kernel module for Linux fan control. Maybe I'll get to those at some point.

   Understanding Unified Memory (It's Unintuitive)

   The BIOS has a "UMA Frame Buffer Size" setting that defaults to 64 GB. Your instinct says "big number = more GPU memory = good." Yeah, your instinct is wrong here.

   On a traditional discrete GPU, VRAM is physically separate from system RAM. On Strix Halo, there's only one pool of LPDDR5. The BIOS carveout reserves a chunk of that pool as dedicated VRAM - the OS can't see it, can't use it for anything else, and the GPU doesn't even need it because it can access system RAM at the same speed through GTT (Graphics Translation Table).

   The optimal configuration is:

   • BIOS VRAM: 2 GB (the minimum on the GMKtec's current BIOS 1.12 - you'll see guides online saying to set this to 512 MB, but that was only possible on earlier BIOS versions. 2 GB is as low as it goes now.)
   • GTT: 124 GB (dynamically mapped, shared between CPU and GPU)

   This gives you ~124 GB usable for both CPU and GPU workloads, instead of 64 GB locked to GPU + 64 GB for CPU.

   The kernel parameters to make this work:

   amdgpu.gttsize=126976          # 124 GiB GTT
   ttm.pages_limit=29360128       # Allow TTM to manage 112 GiB of pages
   ttm.page_pool_size=29360128    # Matching pool size
   amdgpu.no_system_mem_limit=1   # Disable SVM resident memory cap
   amd_iommu=off                  # Fully disable IOMMU (~4% bandwidth gain)
   

   The ttm.pages_limit parameter is particularly sneaky. Without it, you can set GTT to 124 GB and the kernel will report 124 GB, but HIP/ROCm applications will only see ~62 GiB. The TTM subsystem has its own page limit that must match. And it has to be set at boot - runtime changes don't take effect. That one took a while to figure out.

   On Rocky 9, updating kernel parameters has its own gotcha: editing /etc/default/grub and running grub2-mkconfig doesn't work. Rocky 9 uses BLS (Boot Loader Specification) entries, which have their own options line. Use grubby instead:

   grubby --update-kernel=DEFAULT --args="amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=29360128 ttm.page_pool_size=29360128 amdgpu.no_system_mem_limit=1"
   

   Building and Running llama.cpp

   Ok, with the hardware finally cooperating, I built llama.cpp. I started with ROCm/HIP since that's what everyone recommends for AMD GPUs:

   cmake -B build \
     -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 \
     -DGGML_NATIVE=OFF -DCMAKE_C_FLAGS='-march=znver4' -DCMAKE_CXX_FLAGS='-march=znver4' \
     -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON \
     -DGGML_HIP_ROCWMMA_FATTN=ON -DGGML_CUDA_FA_ALL_QUANTS=ON \
     -DGGML_LTO=ON -DCMAKE_BUILD_TYPE=Release
   cmake --build build -j$(nproc)
   

   A few build notes:

   • -DGGML_NATIVE=OFF with explicit -march=znver4 is required because GCC 11 on Rocky 9 emits VNNI instructions that the system's binutils can't assemble. Specifying znver4 explicitly avoids the problematic auto-detection.
   • The AVX512 flags enable SIMD for CPU-side tensor ops. Zen 5 has full AVX-512 support.
   • GGML_HIP_ROCWMMA_FATTN enables wave matrix multiply for flash attention.

   Critical for APUs: You must set GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 before running. Without it, llama.cpp tries to allocate in the 2 GB dedicated VRAM carveout and fails for any model larger than 2 GB. With it, allocations go through the full GTT pool. Don't skip this or you'll be very confused.

   The Model

   I'm running Qwen3-Coder-Next Q4_K_M - an 80B parameter Mixture-of-Experts model with 3B active parameters, purpose-built for coding agents. At Q4_K_M quantization it's about 46 GiB across 4 GGUF shards, fitting comfortably in 128 GB with room for a 65K token context window.

   The Mixture-of-Experts architecture is what makes this hardware viable. An 80B MoE model only needs to stream the active expert weights each token - roughly 3B parameters - not the full 80B. Dense 70B models? They crawl at 5-7 tok/s on this hardware. This 80B MoE? 46 tok/s. Same memory, same bandwidth - the model architecture makes all the difference.

   This model scored #1 on SWE-rebench Pass@5 at 64.6%, beating Claude Opus 4.6 (58.3%). Running it locally at interactive speeds on a sub-$3K box (give or take, depending on what RAM prices are doing this week) is... pretty nuts.

   Runtime Configuration

   I run llama-server as a systemd service with these flags:

   -fa on              # Flash attention (smaller KV cache, faster attention)
   --parallel 1        # Single slot - all memory for one user
   -t 32 -tb 32       # All 32 CPU cores
   -ub 2048            # Large ubatch for GPU utilization during prompt processing
   -ctk q8_0 -ctv q8_0  # Quantized KV cache (~2x smaller than f16, minimal quality loss)
   --mlock             # Pin model in RAM
   -c 65536            # 65K context window
   

   Two things I learned about GPU power modes: profile_peak sounds good but actually causes thermal throttling on an integrated GPU sharing the SoC thermal envelope. Generation dropped from 37.9 to 26.9 tok/s. Ouch. Use high instead - it clocks up aggressively but lets the thermal controller do its job.

   Tuning: What the Internet Got Wrong

   With the system stable, I went through every tuning recommendation I could find - a comprehensive "definitive guide" document and the strixhalo.wiki llama.cpp performance page. I benchmarked each claim individually. A lot of them were wrong, at least for this hardware.

   Things that didn't matter

   --no-mmap vs --mlock: Identical performance. pp=219.5/tg=37.7 vs pp=218.7/tg=38.0. On a UMA APU where GPU memory is system memory, both approaches effectively do the same thing. Pick whichever you prefer.

   -b 256 batch size: Slightly worse than the default -ub 2048. The claimed jump from 70 to 591 tok/s was for Qwen3-30B-A3B, a much smaller model with different memory access patterns. Don't copy batch size settings across models.

   ROCBLAS_USE_HIPBLASLT=1: No measurable effect on gfx1151 with this model. The "mandatory" claim may apply to other GPU architectures.

   Things that helped a little

   amd_iommu=off: Real. Generation speed went from 38.0 to 39.4 tok/s - a 3.7% improvement. Not the claimed 6%, but free performance. I also bumped GTT from 112 GiB to 124 GiB in the same change.

   The big discovery: Vulkan beats ROCm

   Then I built llama.cpp with Vulkan instead of HIP, just to see what would happen:

   cmake .. -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
   

   The results were... not subtle:

   Context	Vulkan pp (tok/s)	Vulkan tg (tok/s)	HIP pp (tok/s)	HIP tg (tok/s)
   Default (512)	548	45.9	336	40.8
   32K	394	36.8	91	29.7
   65K	305	32.2	54	23.5
   100K	213	28.2	36	18.7

   Vulkan with RADV (Mesa's open-source Vulkan driver) was 63% faster at prompt processing and 12% faster at generation at default context. The gap widens with context length - at 100K tokens, Vulkan is nearly 6x faster at prompt processing and 51% faster at generation.

   This directly contradicts the common advice that "ROCm is better for long-context work." That may be true on datacenter GPUs (MI300X) or older desktop GPUs (gfx1100), but on gfx1151, the HIP compute kernels are known to run 2-6x slower than expected. Vulkan's cooperative matrix support through RADV doesn't have the same problem.

   The guides also recommended AMDVLK (AMD's proprietary Vulkan driver) over RADV for 10-15% better performance. I investigated and found that AMD discontinued AMDVLK in September 2025, going all-in on RADV. The strixhalo.wiki's own benchmarks actually show RADV beating AMDVLK even before they killed it. Just use RADV.

   One nice bonus: the Vulkan build doesn't need the GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 environment variable. That's a HIP/ROCm-specific workaround.

   The Boring But Important Stuff

   A handful of other things that aren't exciting but tripped me up:

   DNF firmware pinning: Added exclude=linux-firmware* amdgpu-dkms-firmware* to /etc/dnf/dnf.conf. Without this, a routine dnf update can reintroduce MES 0x83 and break GPU compute.

   EPEL rocminfo conflict: EPEL ships rocminfo 5.4.4 which conflicts with the ROCm 7.2 version from AMD's repo. Fixed with dnf config-manager --save --setopt=epel.excludepkgs=rocminfo.

   SELinux and systemd: The llama-server binary must live in /usr/local/bin (not ~/) for SELinux to allow systemd to execute it. Run restorecon -v after copying.

   WiFi: The MediaTek MT7925 (Wi-Fi 7) works with WPA2 networks but fails on WPA2/WPA3 mixed-mode SSIDs. Suspected mt7925e driver bug. If your router broadcasts both, you may need a WPA2-only SSID.

   GPU performance mode: Set via udev rule to persist across reboots:

   echo 'ACTION=="add", SUBSYSTEM=="drm", KERNEL=="card0", ATTR{device/power_dpm_force_performance_level}="high"' \
     | sudo tee /etc/udev/rules.d/99-gpu-perf.rules
   

   What I'd Do Differently

   If I was setting this up again from scratch:

   1. Start with Vulkan, not ROCm/HIP. I spent way too much time optimizing the HIP build before discovering Vulkan was faster at everything. Just build llama.cpp with -DGGML_VULKAN=ON from the start.

   2. Install ELRepo kernel immediately. Don't waste time trying to make the stock 6.12 kernel work with DKMS. It can't. I tried.

   3. Check MES firmware before debugging anything else. If rocminfo hangs or GPU compute produces page faults, check MES version first. It's the most common cause and the least obvious one.

   4. Set BIOS VRAM to minimum and maximize GTT from day one. The default 64 GB carveout wastes half your memory for no reason.

   5. Install ryzenadj before you do literally anything else. Seriously. The thermal shutdowns caught me completely off guard and happened repeatedly. The stock power limits on this chassis are not safe for sustained workloads. Cap power first, then start playing with models.

   The End Result

   My final configuration:

   Component	Setting
   OS	Rocky Linux 9.7, kernel 6.19.6 (ELRepo)
   GPU driver	Mesa RADV 25.0.7 (Vulkan)
   MES firmware	0x80 (manually installed)
   BIOS VRAM	2 GB (minimum)
   GTT	124 GiB
   IOMMU	Fully disabled
   Power limits	100W burst / 88°C target (ryzenadj)
   llama.cpp	Vulkan build, flash attention, q8_0 KV cache
   Model	Qwen3-Coder-Next Q4_K_M (80B MoE, 46 GiB)
   Context	65K tokens

   Performance:

   Metric	Speed
   Token generation (short context)	45.9 tok/s
   Token generation (32K context)	36.8 tok/s
   Token generation (65K context)	32.2 tok/s
   Token generation (100K context)	28.2 tok/s
   Prompt processing (short context)	548 tok/s

   For a mini PC that cost me under $3K - though good luck getting that price if LPDDR5 keeps doing what it's been doing - running a frontier-class 80B coding model entirely locally, with 65K context and no API costs? I'm pretty happy with that.

   ---

   Tested on: GMKtec NucBox EVO-X2, AMD Ryzen AI MAX+ 395, 128 GB LPDDR5, Rocky Linux 9.7, kernel 6.19.6, llama.cpp build f90bd1dd8, Mesa RADV 25.0.7. March 2026.

   ]]> https://damenknight.com/rescuing-unsupported-enterprise-ssds-megaraid-tools/ https://damenknight.com/rescuing-unsupported-enterprise-ssds-megaraid-tools/ Sat, 31 Jan 2026 09:00:00 GMT Homelab I picked up a pair of refurbished Samsung PM1643a SSDs for my Dell R740 Proxmox server, but my PERC H330 showed them as "Unsupported" with 0 KB size. They had… I picked up a pair of refurbished Samsung PM1643a SSDs (3.84TB each) for my Dell R740 Proxmox server. Great deal on enterprise drives, right? Except when I installed them, my PERC H330 controller showed them as "UGUnsp" (Unconfigured Good Unsupported) with a size of... 0 KB.

   The drives had been pulled from an enterprise storage array (likely Hitachi or EMC) and were formatted with 520-byte sectors instead of the standard 512. Those extra 8 bytes per sector are used for T10-DIF data integrity protection - useful in big SANs, useless for my homelab.

   The usual fixes wouldn't work. sg_format? Can't see the drives. Samsung DC Toolkit? Nope. Perccli format/erase commands? "Operation not allowed." The controller refused to expose them to Linux at all - no /dev/sd*, no /dev/sg*. My only option seemed to be flashing the H330 to IT-mode, which meant unacceptable downtime.

   Then I noticed something: smartctl -d megaraid,4 -i /dev/sda could actually talk to the drives via MegaRAID passthrough. The controller wouldn't expose them, but it would relay SCSI commands to them. That was my way in.

   With some help from Claude Code, I dug into smartctl's source code and reverse-engineered the MegaRAID IOCTL interface. The result is a set of small C tools that send SCSI FORMAT UNIT and MODE SELECT commands directly through the MegaRAID passthrough - no HBA flash required, no downtime.

   Both drives are now happily running at 512-byte sectors, showing their full 3.49 TiB each, and working perfectly as JBOD in Proxmox.

   I've open-sourced the tools in case anyone else runs into this: github.com/filthyrake/megaraid_format_tools

   ]]> https://damenknight.com/vlog/ https://damenknight.com/vlog/ Wed, 03 Dec 2025 21:07:35 GMT Uncategorized Once upon a time I had a youtube channel. I mean I still do but most of my videos are now private and I stopped posting. I dont really want to reactivate my… Once upon a time I had a youtube channel. I mean I still do but most of my videos are now private and I stopped posting. I dont really want to reactivate my channel and be a YouTuber again, but I also dont want to just toss all that stuff out, so I’m standing up vlog.damenknight.com and migrating MOST of my old YouTube content over.

   Now I DO want to create at least some content still – I clearly enjoy it – but doing it here instead of on YT will hopefully let me keep it a bit more chill and maybe more consistent. Head on over, check it out as I get things migrated, and keep your eyes out in the future for all NEW content! More car stuff, more astro stuff, more tech, you name it!

   ]]> https://damenknight.com/astrophotography-datasets-site/ https://damenknight.com/astrophotography-datasets-site/ Mon, 05 May 2025 00:13:28 GMT Astrophotography For a while now, I’ve been sharing my astrophotography datasets on a really basic mid-90’s-looking site I threw together. It was ugly, it was hard to use, it… For a while now, I’ve been sharing my astrophotography datasets on a really basic mid-90’s-looking site I threw together. It was ugly, it was hard to use, it sucked. But it served its purpose and I was ok with it. I had bigger plans though.

   You see, there aren’t a ton of sites that share astrophotography datasets – especially not for free. Many people sell theirs, and the best free options have historically been places like NASA and the ESA. I absolutely love those free resources, but I wanted something for the rest of us and with more variety.

   So I’ve spent the past… many many many months working hard to build something new. This was a pain, since I am not a web guy or a software guy and this involved both – and I’m far far from done still – but it is in a place where I’m happy to talk about it and start showing it off.

   If you havent yet, head over to check it out: Miscellaneous Datasets. It is fairly filterable by what kind of data you want. All datasets can be downloaded in all the major formats. Right now it is limited to data I’ve personally captured, but I’m hard at work getting more contributors on board. My dream is that someday this will be the largest free non-government dataset resource in the world.

   ]]> https://damenknight.com/warewulf-home-lab-setup/ https://damenknight.com/warewulf-home-lab-setup/ Tue, 25 Mar 2025 19:13:18 GMT Homelab Projects This cluster is something I setup to learn more about HPC. My initial project is using the cluster to do astrophotography image…

   

   Control Node

   VM running on PowerEdge R730
   Rocky Linux 9.5
   Warewulf 6.4

   Compute Nodes

   1x MeLE Quieter 3C
   * Intel N100
   * 16GB RAM
   
   3x MeLE Quieter 4C
   * Intel N100
   * 16GB RAM

   Connectivity

   TrendNet Switch

   This cluster is something I setup to learn more about HPC. My initial project is using the cluster to do astrophotography image pre-processing/calibrating/integrating.

   ]]> https://damenknight.com/obligatory-homelab-writeup/ https://damenknight.com/obligatory-homelab-writeup/ Sat, 15 Mar 2025 16:41:12 GMT Homelab Projects Don’t judge my local fire-hazard. I’ll get this out of the way first: You do not want rack servers in your home. They’re *really* loud. I am just a crazy…

   Don’t judge my local fire-hazard.

   I’ll get this out of the way first: You do not want rack servers in your home. They’re *really* loud. I am just a crazy person.

   Ok, now that that’s out of the way, let’s talk about what we’ve got. Starting with the items actually installed in the rack from the bottom and working our way up:

   Synology RS2421+

   Hardware:
   12x 18TB Seagate Exos HDDs (SHR)
   2x 1TB SSD Cache (RAID 1)
   32GB RAM
   10Gbe NIC
   
   Services:
   DHCP
   Temporary Web Hosting
   NAS

   Dell PowerEdge R730

   Hardware:
   2x 1TB SAS SSDs (RAID 1)
   6x 1TB SAS SSDs (RAID 10)
   768GB RAM
   2x Xeon E5-2699(v4)
   2x 10Gbe NIC
   2x 1Gbe NIC
   1x nVidia Tesla P4
   1x nVidia Tesla T4
   
   Services:
   Proxmox
   Ollama
   Prod Web Hosting
   Media Management
   Frigate
   WareWulf Cluster Management

   Dell PowerEdge R730xd

   Hardware:
   2x 1TB SATA SSDs (ZFS Mirror)
   12x 16TB Seagate Exos SATA HDDs (ZFS Striped Mirror)
   768GB RAM
   2x Xeon E5-2680(v3)
   2x 10Gbe NIC
   2x 1Gbe NIC
   1x LSI SAS9305-16e HBA (attached to the PowerVault MD1400)
   
   Services:
   TrueNas
   NAS
   Immich
   Minio
   Tailscale
   Uptime-Kuma
   Postgres

   Dell PowerVault MD1400

   Hardware:
   6x 18TB Seagate Exos SAS HDDs (ZFS Striped Mirror)

   Juniper EX4550-32T-AFI

   10Gbe Managed Switch

   CyberPower CPS1215RM 

   It’s a PDU, what do you want?

   On top of the rack things get a little spicier

   

   WareWulf Cluster

   Hardware:
   1x MeLE Quieter 3C MiniPC
   3x MeLE Quieter 4C MiniPC
   1x TRENDnet Switch (connects to 1Gbe NIC in the PowerEdge R730 for the WareWulf Controller)
   
   Services:
   Test environment for HPC
   
   Details here.

   Plex Server

   Hardware:
   BeeLink EQ12 MiniPC
   
   Services:
   Plex

   Home Assistant

   Hardware:
   1x Home Assistant Green
   
   Services:
   Home Assistant

   Moving beyond the rack entirely we have have the router/firewall

   Custom SFF PC

   Hardware:
   1x Core i5-12400
   64GB RAM
   1x 2.5Gbe NIC (to internet)
   1x 10Gbe NIC (to LAN)
   
   Services:
   PFSense
   DNS
   pfBlockerNG
   

   Finally we have the ridiculous power distribution/UPS setup. I wont go into too much detail here, but there are 2x 1500VA APC UPSes, and everything with redundant power is split between the two. Those two UPSes are themselves plugged into a Jackery 5000 power station.

   The rack itself is just a VEVOR 12U open frame rack.

   I left out a few things – I’ve got a raspberry pi 5 running an allsky camera in my backyard (https://allsky.miscellaneousnerdery.com/), there’s a sonos node, a smart things node, a philips node, a govee node, etc etc etc. I may go into detail on those at some point, if I ever build out a full network map to share – it has gotten a lot more complicated since my last post.

   In my next post I’ll go into the various services I’m running in more detail, along with how they’re all setup and what I’m using them for.

   ]]> https://damenknight.com/mounting-my-minipc-and-powerbox/ https://damenknight.com/mounting-my-minipc-and-powerbox/ Mon, 29 Jan 2024 14:43:16 GMT Astrophotography I recently got a Pegasus Pocket PowerBox Advanced for my astro setup, and while I was at it I moved my miniPC up to attach on the telescope instead of onto my… I recently got a Pegasus Pocket PowerBox Advanced for my astro setup, and while I was at it I moved my miniPC up to attach on the telescope instead of onto my mount.

   I did this using the accessories from BuckEye Stargazer. Unfortunately, there are not a ton of guides or instructions available for how everything goes together so I went ahead and filmed a quick video going over the process.

   ]]> https://damenknight.com/achieving-better-telescope-balance/ https://damenknight.com/achieving-better-telescope-balance/ Thu, 25 Jan 2024 17:25:34 GMT Astrophotography On the astrophotography Discord server I hang out on I’ve seen lots and lots of folks struggle with balancing their telescope. I think a big part of the reason…

   On the astrophotography Discord server I hang out on I’ve seen lots and lots of folks struggle with balancing their telescope. I think a big part of the reason why this is so difficult is that a lot of the descriptions of what good balance is are vague or non-existent. So I went ahead and made a quick video to help show what “good” balance looks like.

   ]]> https://damenknight.com/network-setup/ https://damenknight.com/network-setup/ Mon, 22 Jan 2024 18:39:44 GMT Homelab Projects WiFi: NetGear Orbi RBRE960 (AP mode, 3AP Mesh)Router: Custom MiniPC pfSense Router (10GBit LAN, 2.5GBit WAN)

   WiFi: NetGear Orbi RBRE960 (AP mode, 3AP Mesh)
   Router: Custom MiniPC pfSense Router (10GBit LAN, 2.5GBit WAN)
   

   ]]>