Which quant to download is the first real decision you face after picking a model, and model pages rarely help — you get a list of twenty files and no guidance. The good news is that the answer is mostly determined by one number you already know: how much VRAM your graphics card has. Four rules cover nearly every case.
Which Quant to Download, by VRAM#
Start here, then read the rules that follow for the exceptions. These assume a 27B-class model, which is the size most local setups aim at in 2026.
| Your VRAM | Reasonable pick | Why |
|---|---|---|
| 8 GB | a smaller model at Q4_K_M | a 27B model will not fit; drop parameter count, not precision |
| 12 GB | a mid-size model at Q4_K_M | still the best quality-per-gigabyte point |
| 16 GB | IQ4_XS (~14.6–15.1 GB) | fits, but the margin is thin — see rule two |
| 24 GB | IQ4_NL-mtp or UD-Q4_K_XL | the tester's "best overall balance" at this tier |
| 32 GB+ | Q6_K if it fits | one commenter's rule: stay at Q6 unless it will not load |
Table: starting points, not laws. Model size matters as much as quantization level.
Rule One: Bigger Model Beats Higher Precision#
If you have to choose between a larger model at 4-bit and a smaller model at 8-bit, the local AI community has converged fairly firmly on the larger model.
The reason shows up in the benchmarks. Going from Q8 to Q4_K_M cost one tester nothing they could measure — the two were indistinguishable within the error bars of the test (±0.045) — while saving 10 GB. Their conclusion was that there is "no reason to run Q8 on a 16GB card". Ten gigabytes buys a great deal more model.
That does not extend downwards forever. Below roughly 3-bit, quality falls away sharply — KLD crossed 0.1 from Q3_K_XL in one full-range test. The rule is "prefer size at 4-bit", not "quantize as hard as possible".
Rule Two: On 16GB, Watch the Margin#
Sixteen gigabytes is the most common awkward case, because a 27B model at IQ4_XS almost fits comfortably.
The same thread compared specific community builds: mradermacher's IQ4_XS at 15.1 GB was judged "a bit tight" on a 16 GB card, while cHunter's at 14.7 GB was "also very good". Four hundred megabytes decides whether you have room for a usable context window.
Two practical notes from that thread. On 16 GB you should not expect more than roughly 65k context with this model — the KV cache claims the rest. And for this particular model at this size, the advice was not to use the --pure option, which forces a uniform quantization and gives up the mixed-precision gains.
Rule Three: Read the Whole Filename#
The number in the filename is the headline, but the letters change the answer.
- Q4_K_M — standard 4-bit, K-quant, medium. The safe default.
- IQ4_XS — an "importance" quant, calibrated so precision goes where it matters. Often smaller than Q4_K_M at similar quality, which is why it wins on 16 GB.
- UD-Q4_K_XL — Unsloth Dynamic, mixing precision across layers. Scored highest of the Qwen 4-bit variants in the RTX 3090 test.
- UD-IQ2_XXS — 2-bit, the smallest published. A last resort, not a choice.
One warning from the measurements: NVFP4 files came out the same size as IQ4_XS but 3–5% worse in one tester's runs. Same size does not mean same quality, so the format matters as much as the bit count. Quesma's July 2026 comparison is a good place to see the full ladder side by side.
Rule Four: Speed Is Not the Quant's Job Alone#
It is tempting to assume a smaller file always runs faster. Other settings interfere.
In that comparison, a non-MTP IQ4_XS build produced 25.3 tokens/sec while a smaller IQ3 build with multi-token prediction enabled hit 33.7 tokens/sec. The faster one was not faster because it was smaller — MTP changed the throughput independently of the quantization level.
The practical consequence: when you compare two files, check whether they differ in anything besides the quant. Otherwise you will credit the compression for a gain that came from elsewhere.
The Short Version#
If you want one line: download Q4_K_M, or IQ4_XS if you are tight on memory, and spend the memory you saved on a bigger model.
That covers most people on most cards. If a name in this guide was unfamiliar, quantization explained decodes the terminology properly. If you write code with local models, read quantization for coding before settling — the answer shifts. And if you are choosing which model to run in the first place, the best free local AI models works through the current options by hardware tier.
Sources#
- Do Qwen3.6 27B quantizations break the pelican? (Quesma Blog, 27 Jul 2026)
- GGUF format documentation (Hugging Face)
- Quantization notes (bartowski)
- Qwen3.6-27B quantization benchmark (r/LocalLLaMA)
- Qwen 3.6 27B GGUF quality benchmark (LocalBench)
- Lessons from GGUF evaluations (The Kaitchup)
- Unsloth Dynamic GGUFs on Aider Polyglot (Unsloth docs)
Last reviewed: 23 August 2026. File sizes are for one 27B model and change with each release; every figure here is a single-tester measurement on named hardware. Re-checked quarterly.