Requantizing Qwen3.8-Flash-Next for my own machine
| Q4_K_XL (the first build) | |
|---|---|
| Base model | Qwen3.8-Flash-Next |
| File | 102.5 GB, 4 shards |
| Context | 262,144 tokens |
| Quality | |
| Perplexity | 2.9165 ± 0.0363 |
| KLD vs UD-Q4_K_XL | 0.0282 |
| Down error vs BF16 | 0.00667 |
| Speed | |
| Prefill, first 33k | 128 tok/s |
| Prefill at 262k | 57 tok/s |
| Decode at 262k | 14.4 tok/s |
| Setup | |
| Machine | AMD Strix Halo Ryzen AI MAX+ 395 |
| GPU | Radeon 8060S |
| Memory | 128 GB DDR5, 8000 MT/s |
| System | Windows 11, Vulkan |
| Engine | llama.cpp b10715 |
I know Qwen4 is just around the corner, but I decided to give my Qwen3.8-Flash-Next GGUF another go. It's quantized specifically for my machine: 128 GB of DDR5 at 8000 MT/s on an AMD Strix Halo (Ryzen AI MAX+ 395, Radeon 8060S), running Windows 11.
With the release of llama.cpp 1.0.0 I reran my personal benchmarks and noticed there were gains to be had. I wanted them, because I'd sometimes run out of memory with my current GGUF: the model would complain that it couldn't run a build or a test for lack of RAM. So I went for it.
Naturally, everything was built on my daily driver, running my everyday Windows, so the speed and performance you see here come from the actual machine, not from some impossible sandboxed environment. The card on the right is where I started.
And then, of course, the journey begins.
Where the memory goes
A 128 GB Strix Halo on Windows can give its GPU memory in one of two ways: a fixed carve-out of 96 GB, or dynamic allocation with the shared-memory slider at its 124 GB maximum, where one process can put about 80 GiB on the GPU. This model's 28.8 GB n-gram table never leaves system RAM, so only the second mode works.[1]
llama-server -m Q4_K_XL-IQ4NL/Qwen3.8-Flash-Next-Q4_K_XL-IQ4NL-00001-of-00004.gguf \
-c 262144 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -ub 512 -np 1 \
-lm mmap --lazy-mode on --cache-ram 0 --ctx-checkpoints 8 --jinja \
-md mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf --spec-type draft-mtp,ngram-mod
Speed across the context
- unsloth b10715
- b11160-mix, int8 on
- b11160-mix, int8 off (mine)
The builds compared
In daily use, with gatto
Old and new, side by side
| Q4_K_XL-IQ4NL (the new build) | |
|---|---|
| Base model | Qwen3.8-Flash-Next |
| File | 102.5 GB, 4 shards |
| Context | 262,144 tokens |
| On the GPU | 78.3–78.6 GiB |
| Quality | |
| Perplexity | 2.9092 ± 0.0362 |
| KLD vs UD-Q4_K_XL | 0.0304 |
| Same top token | 94.40 % |
| Down error vs BF16 | 0.00490 |
| Speed | |
| Prefill, first 33k | 159 tok/s |
| Prefill at 262k | 67 tok/s |
| Decode at 262k | 15.8 tok/s |
| Decode in daily use | 28.4 / 20.1 / 17.0 tok/s under 25k / 25–50k / 50–100k |
| Get it | |
| Hugging Face | bitlamas/Qwen3.8-… |
| First build | New build | Change | |
|---|---|---|---|
| File size | 102.5 GB | 102.5 GB | same |
| Squared error vs BF16 | 0.00689 | 0.00583 | ▼ −15.4 % |
| Weighted error vs BF16 | 0.00667 | 0.00490 | ▼ −26.5 % |
| Perplexity | 2.9165 | 2.9092 | ▼ −0.25 % |
| Prefill, first 33k | 128 tok/s | 159 tok/s | ▲ +24 % |
| Decode at 262k | 14.4 tok/s | 15.8 tok/s | ▲ +10 % |
Conclusion
The new build is on Hugging Face as bitlamas/Qwen3.8-Flash-Next-GGUF, with the raw logs for every number.[2]
References
- ^ Qwen3.8-Flash-Next GGUF, model card, “Why this size, on Windows”.
- ^ logs/, the raw measurements.