Quantization quality loss is the part of local AI where folklore has replaced measurement. "Q4 is fine" gets repeated until it sounds like a law, usually with no numbers attached. The honest position is more interesting: at 4-bit the loss really is small, the drop-off below that is sharp, and the size of the loss depends more on what you are asking the model to do than on which quantization you picked.
Quantization Quality Loss: Where the Cliff Actually Is#
The "cliff" is a useful image — quality holding steady as files shrink, then falling away quickly at some point.
The measurements in circulation put that point at around 3-bit. One benchmark of Qwen3.6-27B across the full quantization range found that from Q3_K_XL downwards, KLD rose above 0.1 and token selection began to diverge materially. Above that level, the same tests keep finding differences small enough to argue about.
KLD — Kullback–Leibler divergence — measures how far the quantized model's probability distribution has drifted from the original's. Lower is better. It has become the metric of choice in this niche precisely because it is more sensitive than asking "did it get the answer right".
The Threshold Everyone Quotes, and Its Limit#
Here is the problem with that 0.1 figure, stated by the people who measure it.
A thread comparing KV quantization levels reported differences of 0.005 and 0.01 in KLD, then admitted: "These numbers are hard to understand for me. What does a 0.005 increase in KLD [mean]?" Another noted that Q6 unquantized sat "0.01 units away" without being able to say what a user would notice.
So the community has a sensitive metric and no agreed translation from that metric into experience. KLD reliably tells you the ordering — Q8 is closer to the original than Q4, which is closer than Q2. It does not tell you the point at which a human reading the output would object. Anyone quoting a KLD delta as though it settles the question is over-reading it.
That is not a reason to ignore the numbers. It is a reason to treat "0.1 is the cliff" as a useful landmark rather than a physical constant.
Four-Bit Quants Land Within Two Points of Each Other#
The most useful benchmark in the material tested twelve GGUF quantizations of Qwen3.6-27B on a single RTX 3090, scoring them on HumanEval+, MBPP+, perplexity, throughput and needle-in-a-haystack retrieval.
The headline result: every 4-bit variant finished within about two percentage points of the others.
| Model and quantization | HumanEval base | HumanEval+ | Run time |
|---|---|---|---|
| Gemma-4-31B UD-Q4_K_XL | 97.0% | 94.5% | 34:02 |
| Qwen3.6-27B UD-Q4_K_XL (no MTP) | 95.7% | 92.1% | 19:17 |
| Qwen3.6-27B IQ4_NL-mtp | 95.1% | 91.5% | 9:39 |
| Qwen3.6-27B UD-Q4_K_XL-MTP | 95.1% | 90.9% | 11:07 |
| Qwen3.6-27B IQ4_XS-Q8nextn | 93.9% | 90.2% | 9:01 |
Table: one tester, one GPU. The spread across four-bit options is small; the spread in run time is not.
Note the last column. The highest-scoring Qwen configuration took twice as long as the fastest, for 1.6 points of HumanEval+. Which of those you want depends entirely on whether you are waiting for the output.
The Same Quant, Two Very Different Scores#
This is the finding that should change how you read quantization claims.
The same setups were also scored on MBPP+, a different coding benchmark. Where HumanEval+ held up in the low 90s, MBPP+ fell to 78%. Gemma-4-31B went from 94.5% on one to 78.6% on the other. The quantization did not change between those two numbers — only the task did.
A single "quality" number for a quantization level is therefore hiding a range. When someone says a quant retains 99% of quality, the fair question is: 99% of what, measured how? The answer is usually one benchmark on one model — real evidence, but narrower than the claim implies.
Where the Differences Stop Being Real#
The same test found that all Q5 variants tied at 90.9% on HumanEval+, and the tester's own conclusion was that "the differences between them are noise."
This is worth internalising, because a lot of effort in local AI goes into hunting for a marginally better quantization of the same model. Below a certain gap, you are not measuring quality any more — you are measuring the variance of the test. Time spent picking between Q5 variants is better spent on a larger model, more context, or a better prompt.
One related trap from the same run: an abliterated finetune at Q5_K_M scored 75% with multiple timeouts. That is a far bigger drop than any quantization level caused, and it came from the finetune, not the compression.
How to Read the Next Quantization Benchmark#
Four questions, and they will handle almost everything you meet:
- Which model, and which size? A cliff located on a 27B model says little about a 4B one.
- Which benchmark? HumanEval+ and MBPP+ disagreed by 16 points on the same setup.
- How many samples? The RTX 3090 test screened on 20 tasks and confirmed on 164 — those are very different signals.
- Is the gap bigger than the noise? If variants tie, they tie.
For the practical version of all this — which file to actually download for your card — see which model quant to download. If you are writing code with a local model, the task-specific picture is in quantization for coding. If the terminology here was unfamiliar, quantization explained covers the names first, and what hardware you need to run AI models locally covers the memory side.
Sources#
- The Quantization Cliff: When Does Compression Break Code Intelligence?
- Do Qwen3.6 27B quantizations break the pelican? (Quesma Blog, 27 Jul 2026)
- Qwen 3.6 27B GGUF quality benchmark (LocalBench)
- Qwen3.6-27B quantization benchmark writeup (huy.rocks, 29 May 2026)
- KLD-guided quantization (Hugging Face blog)
- Qwen3.6-27B quantization benchmark (r/LocalLLaMA)
- Qwen3.5-9B quantization comparison (r/LocalLLaMA)
- Unsloth Dynamic GGUFs on Aider Polyglot (Unsloth docs)
Last reviewed: 23 August 2026. Every benchmark cited is one or two models on one tester's hardware; no controlled study across model families exists in the sources. Re-checked quarterly.