---
title: "Quantization Explained: The 5 Terms You Actually Need"
url: "https://learnaitodayonline.com/quantization-explained/"
description: "Quantization explained in plain English: what Q4, Q8 and GGUF mean, how much quality you really lose, and which file to download for your GPU."
author: "Robert Waithaka"
published: "2026-08-30"
last_reviewed: "2026-08-23"
categories: ["Apps"]
tags: ["tool-guide", "level-intermediate"]
series: "Quantization, end to end"
series_part: 1
site: "Learn Artificial Intelligence"
approx_tokens: 2329
---

# Quantization Explained: The 5 Terms You Actually Need

Quantization explained, in one line: it is the trick that lets a model far too big for your graphics card run on it anyway. Download any open-weight model and you will hit a wall of filenames — `Q4_K_M`, `IQ4_XS`, `UD-Q8_K_XL` — with no explanation of which one you want. The same 27-billion-parameter model ships anywhere from 9.6 GB to 54.7 GB depending on which you pick. This is what those names mean, what they cost you, and how to choose.

## Quantization Explained in One Sentence

A model is a very large pile of numbers. Quantization stores each of those numbers less precisely, so the pile takes up less room.

That is the whole idea. A 16-bit number becomes an 8-bit number, or a 4-bit one. You lose a little accuracy in each individual value, and in exchange the file shrinks and it fits in memory you actually own.

Here is what that looks like in practice. Quesma tested every published quantization of Qwen3.6 27B in July 2026 and the ladder runs from 54.7 GB down to 9.6 GB — the same model, a sixth of the size.

![Quantization explained — the same 27B model at twelve different file sizes](https://cdn.sanity.io/images/gfihpee1/production/7a7c9b4946e4d1237177d6862193385e882b80b6-1306x870.png)

_Figure: the full size ladder for one 27B model. Source: Quesma Blog, "Do Qwen3.6 27B quantizations break the pelican?", Piotr Migdał, 27 July 2026._

| Quantization | File size | What it is |
| --- | --- | --- |
| BF16 | 54.7 GB | the original weights, full precision |
| Q8_0 | 29.0 GB | 8-bit, treated as near-lossless in most tests |
| Q6_K | 22.9 GB | 6-bit |
| Q5_K_M | 19.8 GB | 5-bit, medium |
| Q4_K_M | 17.1 GB | 4-bit — the usual default |
| IQ4_NL | 16.3 GB | 4-bit "importance" quant |
| UD-IQ2_XXS | 9.6 GB | 2-bit, smallest published |

_Table: selected rungs from the Qwen3.6 27B ladder. Sizes scale with parameter count, so treat the shape as the lesson rather than the exact numbers._

## What Precision Actually Means

"Precision" here means how many bits are used to store one number. More bits, finer detail.

The nearest everyday comparison is rounding money. Writing £12.99 as £13 loses almost nothing on one item. Do it across a shopping list and the total drifts slightly. Do it across billions of values and the question becomes whether the drift is small enough not to matter — which is exactly what the benchmarks in the next section measure.

![What quantization does — fewer bits per number](https://cdn.sanity.io/images/gfihpee1/production/d216593dc06addf61611197af3f44f79b934fb59-1140x514.png)

_Figure: 16-bit, 8-bit and 4-bit storage for the same model, with the resulting file sizes. Sources: Quesma; Maxime Labonne's weight quantization primer._

The rounding is not applied bluntly. Modern methods group weights and spend more bits where the model is most sensitive, which is why a well-made 4-bit file can hold up far better than the raw bit count suggests.

## Reading a GGUF Filename: Q4_K_M Decoded

GGUF is the file format used by [llama.cpp](https://github.com/ggml-org/llama.cpp/pull/21029) and the tools built on it. The name after it tells you how the model was compressed.

![Reading a GGUF filename — Q4_K_M decoded](https://cdn.sanity.io/images/gfihpee1/production/1f62ad31f005855a533411e2e37888402c85e712-1140x540.png)

_Figure: the four parts of a GGUF quantization name. Sources: Hugging Face GGUF documentation; bartowski's quantization notes._

Three prefixes and suffixes cover most of what you will meet:

- **Q4, Q5, Q8** — bits per weight. Lower is smaller and rougher.
- **\_K** — K-quants, which group weights more cleverly than the original method. **\_S, \_M, \_L** then mark small, medium and large within that family.
- **IQ** — "importance" quants, which use a calibration pass to decide where precision matters. Often smaller than a plain quant at similar quality.
- **UD-** — Unsloth Dynamic, which mixes precision across layers rather than applying one level everywhere.

If you remember nothing else: the number is the size, the letters are the method.

## How Much Quality Do You Actually Lose?

Less than the size reduction suggests, and this is where the folklore turns out to be roughly right.

One tester benchmarked every quantization of Qwen3.8-27B that fits in 16 GB of VRAM, using Q8_0 as a near-lossless reference. Their finding: **Q4_K_M was indistinguishable from Q8 within the error bars of the test (±0.045) while being 10 GB smaller.** Their conclusion was blunt — there is no reason to run Q8 on a 16 GB card.

IQ4_XS held 99.2% of the reference at 14.6 GB. The one disappointment was NVFP4, the same size as IQ4_XS but 3–5% worse in that tester's runs.

![Quality retained by quantization level against a Q8 reference](https://cdn.sanity.io/images/gfihpee1/production/6a37636de59f8de88520144810bc62fdedde280b-2002x850.png)

_Figure: quality retained, and the caveat that undercuts the whole metric. Sources: Quesma; r/LocalLLM, "I benchmarked every Qwen 3.8 27B quant that fits in 16GB VRAM"._

| Finding | Figure | Source |
| --- | --- | --- |
| Q8_0 output agreement with the original | 99.3% | Quesma (Jul 2026) |
| Q4_K_M vs Q8 reference | indistinguishable within ±0.045 | r/LocalLLM 16 GB benchmark |
| IQ4_XS vs Q8 reference | 99.2% at 14.6 GB | r/LocalLLM 16 GB benchmark |
| NVFP4 vs IQ4_XS at equal size | 3–5% worse | r/LocalLLM 16 GB benchmark |

_Table: single-tester measurements on named hardware. Real numbers, not a controlled study — one model each._

## Why "Percent Match" Flatters Quantization

Now the number that should make you cautious about every figure above.

Quesma also compared the unquantized model against **itself** and got 48.5% agreement. Because generation involves random sampling, the original model does not reproduce its own output either. So a "99.3% match" is not 99.3% of the way to perfect — it is measured against a target that only matches itself half the time.

This does not mean quantization is secretly terrible. It means percentage-match figures compress a fuzzy thing into a crisp-looking number, and you should read them as ordering rather than as scores. Q8 beats Q4 beats Q2. How much, in a way you would notice? The honest answer from the material is: less than people fear at 4-bit, and nobody has measured it cleanly across model families.

The same caution applies to KLD, the other metric these tests use. One benchmark thread put it plainly: _"These numbers are hard to understand for me. What does a 0.005 increase in KLD [mean]?"_ The measurements are real. Their interpretation is not settled.

## So Which File Should You Download?

For most people, on most hardware, running a 27B-class model: **Q4_K_M, or IQ4_XS if you are tight on memory.** That is where the benchmarks land and where community advice has converged.

The full answer depends on your VRAM, and it is worth getting right — so it has [its own guide](https://learnaitodayonline.com/which-quant-to-download/). If you want to know where quality genuinely breaks down rather than where it holds, that is [the quantization cliff](https://learnaitodayonline.com/quantization-quality-loss/). If you are choosing a model rather than a file, start with [the best free local AI models](https://learnaitodayonline.com/best-free-local-ai-models/), and if you are still working out what your machine can handle, [what hardware you need to run AI models locally](https://learnaitodayonline.com/what-hardware-do-you-need-to-run-ai-models-locally/) covers the three numbers that matter.

To actually download and run one of these files, our [Ollama guide](https://learnaitodayonline.com/ollama-download-install-run-local-ai-models/) walks through it step by step.

---

## Sources

- [Do Qwen3.6 27B quantizations break the pelican? (Quesma Blog, Piotr Migdał, 27 Jul 2026)](https://quesma.com/blog/qwen-quantization-quality/)
- [Introduction to Weight Quantization (Maxime Labonne)](https://mlabonne.github.io/blog/posts/Introduction_to_Weight_Quantization.html)
- [GGUF format documentation (Hugging Face)](https://huggingface.co/docs/hub/gguf)
- [Quantization notes (bartowski)](https://github.com/bartowski1182/llm-knowledge/blob/main/quantization/quantization.md)
- [Quality comparison between Qwen 3.6 27B quantizations (r/LocalLLaMA, 2026)](https://www.reddit.com/r/LocalLLaMA/comments/1t53dhp/quality_comparison_between_qwen_36_27b/)
- [Lessons from GGUF evaluations (The Kaitchup)](https://kaitchup.substack.com/p/lessons-from-gguf-evaluations-ternary)
- [llama.cpp quantization development (GitHub)](https://github.com/ggml-org/llama.cpp/pull/21029)

_Last reviewed: 23 August 2026. Quantization naming and defaults change with each llama.cpp release; file sizes are for one 27B model. Re-checked quarterly._