--- title: "Quantization for Coding: 5 Hard Truths About Q4 and Code" url: "https://learnaitodayonline.com/quantization-for-coding/" description: "Quantization for coding is not the same as for chat: MBPP+ fell 13–16 points where HumanEval+ held. What to run, and what to leave alone." author: "Robert Waithaka" published: "2026-08-31" last_reviewed: "2026-08-23" categories: ["Apps"] tags: ["tool-guide", "level-intermediate"] series: "Quantization, end to end" series_part: 4 site: "Learn Artificial Intelligence" approx_tokens: 2087 --- # Quantization for Coding: 5 Hard Truths About Q4 and Code Quantization for coding behaves differently from quantization for everything else, and the standard advice does not account for it. "Q4 is fine" is a reasonable summary if you are writing emails. If you are generating code, the same compression shows up in ways you will notice — not as garbled output, but as subtly wrong logic that costs you time. Here is what the measurements show. ## Quantization for Coding: Why Code Is Different Prose is forgiving. Swap a word for a near-synonym and the meaning survives. Code is not: one wrong variable name, one off-by-one boundary, one hallucinated method, and it does not run. That asymmetry is why compressing a model hurts code more than conversation. The model's output distribution shifts slightly at every token — which is what [quantization does](https://learnaitodayonline.com/quantization-explained/) — and code has far fewer acceptable next tokens than a sentence does. ![How much quantization each task will tolerate](https://cdn.sanity.io/images/gfihpee1/production/a34a8637984440449c4f2b237ea82377c6a691d7-1173x800.png) _Figure: tolerance by task, ranked from the community evidence. Sources: the MBPP+/HumanEval+ gap in the RTX 3090 test; "For coding — is it ok to quantize KV Cache?"; "Is using a heavily quantized model bad for rp?" This is a ranking from reported evidence, not a measured scale._ Creative and conversational work sits at the tolerant end — one thread on roleplay found heavy quantization broadly acceptable. Exact recall and code generation sit at the other. ## The 16-Point Gap Between Two Coding Benchmarks The strongest evidence in the material comes from a test of twelve GGUF quantizations of Qwen3.6-27B on a single RTX 3090, which scored each on both HumanEval+ and MBPP+. The two benchmarks disagreed sharply on identical setups: | Model and quantization | HumanEval+ | MBPP+ | Gap | | --- | --- | --- | --- | | Gemma-4-31B UD-Q4_K_XL | 94.5% | 78.6% | −15.9 pp | | Qwen3.6-27B UD-Q4_K_XL-MTP | 90.9% | 78.0% | −12.9 pp | _Table: the same files, two coding benchmarks. Source: r/LocalLLM, "Qwen3.6-27B on RTX 3090: tested 12 GGUF quants across HumanEval+, MBPP+, perplexity, throughput and needle-in-haystack"._ Nothing changed between those columns except the task. That is the finding: **a single "quality retained" figure for a quantization is not meaningful for coding**, because the number you get depends heavily on which coding problems you measure. It also means you should be sceptical of anyone quoting one benchmark as proof a quant is safe for development work. The [quantization cliff](https://learnaitodayonline.com/quantization-quality-loss/) is not one edge — it moves with the workload. ## Time Costs More Than Quality Here Within the 4-bit range, the quality differences for code are small and the speed differences are not. ![Quantization for coding — run time against HumanEval+ score](https://cdn.sanity.io/images/gfihpee1/production/ee94dfb8410daa9fb3ec2eff0e7bd83521597a4b-1235x850.png) _Figure: HumanEval+ against wall-clock run time across the tested configurations. Source: r/LocalLLM RTX 3090 test._ The spread from best to worst on HumanEval+ was about four percentage points. The spread in run time was nearly four-fold — from 9 minutes to 34. The tester's own verdict for a 24 GiB card was that **IQ4_NL-mtp offered the best overall balance**, with UD-Q4_K_XL without MTP as "the quality king if you don't mind ~2×" the time. For coding specifically that trade matters more than it looks, because you iterate. A model that is 1.6 points better but takes twice as long per attempt is not obviously better when you are making ten attempts. One more finding worth carrying: **all Q5 variants tied at 90.9%** in that test, and the differences were explicitly called noise. Hunting for a better Q5 build is not where coding quality comes from. ## Leave the KV Cache Alone [KV cache quantization](https://learnaitodayonline.com/kv-cache-quantization/) is one of the best-value settings in local AI — it can buy large amounts of context for free. For code, it is the one place the community advice turns cautious. The thread asking the question directly, "For coding — is it OK to quantize KV cache?", does not produce a confident yes. Given that code is already the least tolerant task, stacking cache compression on top of model compression is the wrong place to economise. The sensible order: quantize the model to 4-bit, keep the cache at full precision, and if you run out of context, shorten the context rather than compress it. ## Check the Finetune Before You Blame the Quant A useful warning from the same test run. An **abliterated finetune at Q5_K_M scored 75% with multiple timeouts** — against 90.9% for standard Q5 builds. ![A finetune, not a quant, causing the real drop](https://cdn.sanity.io/images/gfihpee1/production/c22eb099f618dc2f5400f4b2f064c6641b8bcf12-1605x746.png) _Figure: the abliterated finetune against standard Q5 builds. Source: r/LocalLLM RTX 3090 test — "abliterated finetunes appear to hurt code"._ That is a 16-point drop, far larger than anything quantization level caused in the same test. If a local model is writing bad code, the compression is not the first thing to suspect — check what variant of the model you actually downloaded. Uncensored and abliterated builds trade capability for compliance-removal, and coding is where that shows. ## What to Actually Run ![Four rules for quantization and coding](https://cdn.sanity.io/images/gfihpee1/production/3533e7f1a904a4666d3e460045a6b48a26c8c0aa-1167x635.png) _Figure: the four rules the benchmarks support. Sources: RTX 3090 12-quant test; "For coding — is it ok to quantize KV Cache?"._ Concretely, for local coding work: - **Stay at 4-bit or above.** Below roughly 3-bit, divergence rises sharply for every task, and code has least room for it. - **Pick for speed within that band**, since quality differences at 4-bit are small and iteration count is not. - **Keep the KV cache unquantized**, unlike your chat setup. - **Use a standard build**, not an abliterated or heavily modified finetune. - **Test on your own code**, because the benchmarks measure narrow puzzle-style problems, not your codebase. If you are still deciding which file to fetch for your card, [which quant to download](https://learnaitodayonline.com/which-quant-to-download/) has the by-VRAM version. And to run any of this locally, [llama.cpp](https://learnaitodayonline.com/llama-cpp-run-local-ai-models/) is where these quantization settings actually live. --- ## Sources - [The Quantization Cliff: When Does Compression Break Code Intelligence?](https://www.promptinjection.net/p/ai-llm-the-quantization-cliff-when-does-compression-break-code) - [Unsloth Dynamic GGUFs on Aider Polyglot (Unsloth docs)](https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs/unsloth-dynamic-ggufs-on-aider-polyglot) - [Qwen 3.6 27B GGUF quality benchmark (LocalBench)](https://localbench.substack.com/p/qwen-3-6-27b-gguf-quality-benchmark) - [Qwen3.6-27B quantization benchmark writeup (huy.rocks, 29 May 2026)](https://www.huy.rocks/everyday/05-29-2026-ai-qwen3-6-27b-quantization-benchmark) - [Quality comparison between Qwen 3.6 27B quantizations (r/LocalLLaMA)](https://www.reddit.com/r/LocalLLaMA/comments/1t53dhp/quality_comparison_between_qwen_36_27b/) - [Quantization notes (bartowski)](https://github.com/bartowski1182/llm-knowledge/blob/main/quantization/quantization.md) - [GGUF format documentation (Hugging Face)](https://huggingface.co/docs/hub/gguf) _Last reviewed: 23 August 2026. The coding benchmarks cited are one tester's runs on one GPU; HumanEval+ and MBPP+ measure narrow puzzle-style problems, not production code. Re-checked quarterly._