Quantization Explained: The 5 Terms You Actually Need

Quantization explained, in one line: it is the trick that lets a model far too big for your graphics card run on it anyway. Download any open-weight model and you will hit a wall of filenames — Q4_K_M, IQ4_XS, UD-Q8_K_XL — with no explanation of which one you want. The same 27-billion-parameter model ships anywhere from 9.6 GB to 54.7 GB depending on which you pick. This is what those names mean, what they cost you, and how to choose.

Quantization Explained in One Sentence#

A model is a very large pile of numbers. Quantization stores each of those numbers less precisely, so the pile takes up less room.

That is the whole idea. A 16-bit number becomes an 8-bit number, or a 4-bit one. You lose a little accuracy in each individual value, and in exchange the file shrinks and it fits in memory you actually own.

Here is what that looks like in practice. Quesma tested every published quantization of Qwen3.6 27B in July 2026 and the ladder runs from 54.7 GB down to 9.6 GB — the same model, a sixth of the size.

Table: selected rungs from the Qwen3.6 27B ladder. Sizes scale with parameter count, so treat the shape as the lesson rather than the exact numbers.

What Precision Actually Means#

"Precision" here means how many bits are used to store one number. More bits, finer detail.

The nearest everyday comparison is rounding money. Writing £12.99 as £13 loses almost nothing on one item. Do it across a shopping list and the total drifts slightly. Do it across billions of values and the question becomes whether the drift is small enough not to matter — which is exactly what the benchmarks in the next section measure.

The rounding is not applied bluntly. Modern methods group weights and spend more bits where the model is most sensitive, which is why a well-made 4-bit file can hold up far better than the raw bit count suggests.

Reading a GGUF Filename: Q4_K_M Decoded#

GGUF is the file format used by llama.cpp and the tools built on it. The name after it tells you how the model was compressed.

Three prefixes and suffixes cover most of what you will meet:

• Q4, Q5, Q8 — bits per weight. Lower is smaller and rougher.
• _K — K-quants, which group weights more cleverly than the original method. _S, _M, _L then mark small, medium and large within that family.
• IQ — "importance" quants, which use a calibration pass to decide where precision matters. Often smaller than a plain quant at similar quality.
• UD- — Unsloth Dynamic, which mixes precision across layers rather than applying one level everywhere.

If you remember nothing else: the number is the size, the letters are the method.

How Much Quality Do You Actually Lose?#

Less than the size reduction suggests, and this is where the folklore turns out to be roughly right.

One tester benchmarked every quantization of Qwen3.8-27B that fits in 16 GB of VRAM, using Q8_0 as a near-lossless reference. Their finding: Q4_K_M was indistinguishable from Q8 within the error bars of the test (±0.045) while being 10 GB smaller. Their conclusion was blunt — there is no reason to run Q8 on a 16 GB card.

IQ4_XS held 99.2% of the reference at 14.6 GB. The one disappointment was NVFP4, the same size as IQ4_XS but 3–5% worse in that tester's runs.

Table: single-tester measurements on named hardware. Real numbers, not a controlled study — one model each.

Why "Percent Match" Flatters Quantization#

Now the number that should make you cautious about every figure above.

Quesma also compared the unquantized model against itself and got 48.5% agreement. Because generation involves random sampling, the original model does not reproduce its own output either. So a "99.3% match" is not 99.3% of the way to perfect — it is measured against a target that only matches itself half the time.

This does not mean quantization is secretly terrible. It means percentage-match figures compress a fuzzy thing into a crisp-looking number, and you should read them as ordering rather than as scores. Q8 beats Q4 beats Q2. How much, in a way you would notice? The honest answer from the material is: less than people fear at 4-bit, and nobody has measured it cleanly across model families.

The same caution applies to KLD, the other metric these tests use. One benchmark thread put it plainly: "These numbers are hard to understand for me. What does a 0.005 increase in KLD [mean]?" The measurements are real. Their interpretation is not settled.

So Which File Should You Download?#

For most people, on most hardware, running a 27B-class model: Q4_K_M, or IQ4_XS if you are tight on memory. That is where the benchmarks land and where community advice has converged.

The full answer depends on your VRAM, and it is worth getting right — so it has its own guide. If you want to know where quality genuinely breaks down rather than where it holds, that is the quantization cliff. If you are choosing a model rather than a file, start with the best free local AI models, and if you are still working out what your machine can handle, what hardware you need to run AI models locally covers the three numbers that matter.

To actually download and run one of these files, our Ollama guide walks through it step by step.

Sources#

• Do Qwen3.6 27B quantizations break the pelican? (Quesma Blog, Piotr Migdał, 27 Jul 2026)
• Introduction to Weight Quantization (Maxime Labonne)
• GGUF format documentation (Hugging Face)
• Quantization notes (bartowski)
• Quality comparison between Qwen 3.6 27B quantizations (r/LocalLLaMA, 2026)
• Lessons from GGUF evaluations (The Kaitchup)
• llama.cpp quantization development (GitHub)

Last reviewed: 23 August 2026. Quantization naming and defaults change with each llama.cpp release; file sizes are for one 27B model. Re-checked quarterly.