Quantization Explained: Q4, Q8, GGUF and Which to Download

Quantization explained in one line is the trick that lets a model far too big for your graphics card run on it anyway, by storing its numbers less precisely. For most people with a mid-range card, the file to download is Q4_K_M, or IQ4_XS when memory is tight. Download any open-weight model and you meet a wall of filenames such as Q4_K_M, IQ4_XS and UD-Q8_K_XL, with no word on which one you want. The same 27-billion-parameter model ships anywhere from 9.6 GB to 54.7 GB depending on the choice. The rest of this post explains what the names mean, what they cost, and how to pick one for your card.

What Is Quantization?#

Quantization is the process of storing each of a model's numbers with fewer bits, so the whole model takes up less memory. A model is a very large pile of numbers, and a 16-bit number can become an 8-bit or a 4-bit one. Each value loses a little accuracy, and in exchange the file shrinks until it fits in memory you actually own.

The size range is wider than most people expect. According to Migdał (2026), writing for Quesma, every published quantization of Qwen3.6 27B runs from 54.7 GB down to 9.6 GB, which is the same model at about a sixth of the size. The chart below shows the full ladder.

The table picks out the rungs you are most likely to see on a download page.

Table: selected rungs from the Qwen3.6 27B ladder. Sizes scale with parameter count, so treat the shape as the lesson rather than the exact numbers.

What Does Precision Mean in a Model?#

Precision is the number of bits used to store one number, and more bits keep finer detail. The nearest everyday comparison is rounding prices. Writing £12.99 as £13 loses almost nothing on one item, while doing it across a long shopping list makes the total drift. Across billions of values, the real question is whether that drift is small enough not to matter, which is exactly what the benchmarks later in this post try to measure.

The rounding is not applied bluntly. Modern methods group weights together and spend more bits where the model is most sensitive. This is why a well-made 4-bit file holds up far better than the raw bit count suggests.

How Do You Read a GGUF Filename Like Q4_K_M?#

A GGUF filename tells you how many bits each weight uses and which method compressed it. GGUF is the file format used by llama.cpp and the tools built on it, and the part after the model name is the recipe.

The number after Q is the bits per weight, so Q4 is smaller and rougher than Q8. A _K means K-quants, which group weights more cleverly than the original method, and _S, _M and _L then mark small, medium and large within that family. An IQ prefix marks an "importance" quant, which uses a calibration pass to decide where precision matters and is often smaller than a plain quant at similar quality. A UD- prefix is Unsloth Dynamic, which mixes precision across layers instead of applying one level everywhere.

In practice four names cover most downloads. Q4_K_M is the safe default. IQ4_XS is the one to try when memory is tight. UD-Q4_K_XL scored highest of the Qwen 4-bit variants in one RTX 3090 test, and UD-IQ2_XXS is a last resort for when nothing else fits. If you remember one thing, the number is the size and the letters are the method.

The letters matter as much as the number. In one tester's runs, NVFP4 files came out the same size as IQ4_XS but 3 to 5 percent worse, so two files of equal size are not equal in quality.

Which File Should You Download?#

The file to download depends mostly on one number you already know, your card's VRAM. The table below gives a starting point for a 27B-class model, the size most local setups aim at in 2026. The site's VRAM calculator does the same arithmetic for a specific model, card and context length, using the real file sizes.

Table: starting points, not laws. Model size matters as much as quantization level.

A bigger model at 4-bit usually beats a smaller model at 8-bit. In the 16 GB benchmark, Q4_K_M was indistinguishable from Q8 within the test's error bars of ±0.045 while saving 10 GB, and the tester saw no reason to run Q8 on a 16 GB card. Ten gigabytes buys a lot more model. The rule stops at about 3-bit, though, because in one full-range test KLD, a measure of how far a quant's output drifts from the original, rose above 0.1 from Q3_K_XL downwards.

Sixteen gigabytes is the awkward case, because a 27B model at IQ4_XS only just fits. In the r/LocalLLaMA thread, mradermacher's IQ4_XS at 15.1 GB was judged "a bit tight", while cHunter's at 14.7 GB was "also very good". Those 400 megabytes decide whether a usable context window fits beside the model.

The same thread expected no more than about 65k of context on 16 GB with this model, because the KV cache claims the rest, and it advised against the --pure option, which forces one uniform level and gives up the mixed-precision gains.

A smaller file is not automatically a faster one. In that comparison a non-MTP IQ4_XS build produced 25.3 tokens per second, while a smaller IQ3 build with multi-token prediction reached 33.7. The speed came from MTP, not from the compression. When two files differ in more than the quant, check what else changed before crediting the size.

How Much Quality Do You Lose?#

At 4-bit, the measured loss is small, and below about 3-bit it grows quickly. In one tester's 16 GB benchmark, IQ4_XS kept 99.2 percent of the Q8 reference at 14.6 GB, and Q4_K_M could not be told apart from Q8 within the error bars. Coding tasks are less forgiving than chat, and the full comparison, including where the cliff sits, is in how much quality quantization loses.

Why Do Percent Match Scores Flatter Quantization?#

Percent match scores flatter quantization because the original model does not match itself either. According to Migdał (2026), Q8_0 agreed with the full BF16 output 99.3 percent of the time, which sounds close to perfect. However, Quesma also compared the unquantized model with itself and got only 48.5 percent agreement. This is because generation involves random sampling, so a fresh run of the same model picks different words.

As a result, a "99.3 percent match" is measured against a target that only matches itself about half the time, and it should be read as an ordering, not a score. Q8 beats Q4, which beats Q2. How much of that a person would notice has not been measured cleanly across model families. The same caution applies to KLD, where one benchmark thread admitted that a 0.005 change was hard to interpret.

None of this means quantization is secretly worse than the tests say. It means the figures are real and their meaning is still loose, so the practical advice holds. Start at Q4_K_M, drop to IQ4_XS only when memory forces it, and spend any memory you save on a larger model. If you are still choosing that model, the best free local AI models sorts the options by hardware tier, and what hardware you need to run AI models locally explains the three numbers that decide what will run.

Sources#

• Do Qwen3.6 27B quantizations break the pelican? (Quesma Blog, Piotr Migdał, 27 Jul 2026)
• Introduction to Weight Quantization (Maxime Labonne)
• GGUF format documentation (Hugging Face)
• Quantization notes (bartowski)
• Qwen3.6-27B quantization benchmark (r/LocalLLaMA)
• Quality comparison between Qwen 3.6 27B quantizations (r/LocalLLaMA, 2026)
• Qwen 3.6 27B GGUF quality benchmark (LocalBench)
• Lessons from GGUF evaluations (The Kaitchup)
• llama.cpp quantization development (GitHub)

Last reviewed: 18 October 2026. This post now includes the former guide to which quant to download. File sizes are for one 27B model and change with each llama.cpp release, and most quality figures are single-tester measurements on named hardware. Re-checked quarterly.