Quantization for coding behaves differently from quantization for everything else, and the standard advice does not account for it. "Q4 is fine" is a reasonable summary if you are writing emails. If you are generating code, the same compression shows up in ways you will notice — not as garbled output, but as subtly wrong logic that costs you time. Here is what the measurements show.
Quantization for Coding: Why Code Is Different#
Prose is forgiving. Swap a word for a near-synonym and the meaning survives. Code is not: one wrong variable name, one off-by-one boundary, one hallucinated method, and it does not run.
That asymmetry is why compressing a model hurts code more than conversation. The model's output distribution shifts slightly at every token — which is what quantization does — and code has far fewer acceptable next tokens than a sentence does.
Creative and conversational work sits at the tolerant end — one thread on roleplay found heavy quantization broadly acceptable. Exact recall and code generation sit at the other.
The 16-Point Gap Between Two Coding Benchmarks#
The strongest evidence in the material comes from a test of twelve GGUF quantizations of Qwen3.6-27B on a single RTX 3090, which scored each on both HumanEval+ and MBPP+.
The two benchmarks disagreed sharply on identical setups:
| Model and quantization | HumanEval+ | MBPP+ | Gap |
|---|---|---|---|
| Gemma-4-31B UD-Q4_K_XL | 94.5% | 78.6% | −15.9 pp |
| Qwen3.6-27B UD-Q4_K_XL-MTP | 90.9% | 78.0% | −12.9 pp |
Table: the same files, two coding benchmarks. Source: r/LocalLLM, "Qwen3.6-27B on RTX 3090: tested 12 GGUF quants across HumanEval+, MBPP+, perplexity, throughput and needle-in-haystack".
Nothing changed between those columns except the task. That is the finding: a single "quality retained" figure for a quantization is not meaningful for coding, because the number you get depends heavily on which coding problems you measure.
It also means you should be sceptical of anyone quoting one benchmark as proof a quant is safe for development work. The quantization cliff is not one edge — it moves with the workload.
Time Costs More Than Quality Here#
Within the 4-bit range, the quality differences for code are small and the speed differences are not.
The spread from best to worst on HumanEval+ was about four percentage points. The spread in run time was nearly four-fold — from 9 minutes to 34. The tester's own verdict for a 24 GiB card was that IQ4_NL-mtp offered the best overall balance, with UD-Q4_K_XL without MTP as "the quality king if you don't mind ~2×" the time.
For coding specifically that trade matters more than it looks, because you iterate. A model that is 1.6 points better but takes twice as long per attempt is not obviously better when you are making ten attempts.
One more finding worth carrying: all Q5 variants tied at 90.9% in that test, and the differences were explicitly called noise. Hunting for a better Q5 build is not where coding quality comes from.
Leave the KV Cache Alone#
KV cache quantization is one of the best-value settings in local AI — it can buy large amounts of context for free. For code, it is the one place the community advice turns cautious.
The thread asking the question directly, "For coding — is it OK to quantize KV cache?", does not produce a confident yes. Given that code is already the least tolerant task, stacking cache compression on top of model compression is the wrong place to economise.
The sensible order: quantize the model to 4-bit, keep the cache at full precision, and if you run out of context, shorten the context rather than compress it.
Check the Finetune Before You Blame the Quant#
A useful warning from the same test run. An abliterated finetune at Q5_K_M scored 75% with multiple timeouts — against 90.9% for standard Q5 builds.
That is a 16-point drop, far larger than anything quantization level caused in the same test. If a local model is writing bad code, the compression is not the first thing to suspect — check what variant of the model you actually downloaded. Uncensored and abliterated builds trade capability for compliance-removal, and coding is where that shows.
What to Actually Run#
Concretely, for local coding work:
- Stay at 4-bit or above. Below roughly 3-bit, divergence rises sharply for every task, and code has least room for it.
- Pick for speed within that band, since quality differences at 4-bit are small and iteration count is not.
- Keep the KV cache unquantized, unlike your chat setup.
- Use a standard build, not an abliterated or heavily modified finetune.
- Test on your own code, because the benchmarks measure narrow puzzle-style problems, not your codebase.
If you are still deciding which file to fetch for your card, which quant to download has the by-VRAM version. And to run any of this locally, llama.cpp is where these quantization settings actually live.
Sources#
- The Quantization Cliff: When Does Compression Break Code Intelligence?
- Unsloth Dynamic GGUFs on Aider Polyglot (Unsloth docs)
- Qwen 3.6 27B GGUF quality benchmark (LocalBench)
- Qwen3.6-27B quantization benchmark writeup (huy.rocks, 29 May 2026)
- Quality comparison between Qwen 3.6 27B quantizations (r/LocalLLaMA)
- Quantization notes (bartowski)
- GGUF format documentation (Hugging Face)
Last reviewed: 23 August 2026. The coding benchmarks cited are one tester's runs on one GPU; HumanEval+ and MBPP+ measure narrow puzzle-style problems, not production code. Re-checked quarterly.