--- title: "Which Quant to Download: 4 Essential Rules for Your VRAM" url: "https://learnaitodayonline.com/which-quant-to-download/" description: "Which quant to download for your GPU: the 4 rules that cover 8GB to 32GB, how to read a GGUF filename, and the trap on 16GB cards." author: "Robert Waithaka" published: "2026-08-30" last_reviewed: "2026-08-23" categories: ["Apps"] tags: ["tool-guide", "level-beginner"] series: "Quantization, end to end" series_part: 2 site: "Learn Artificial Intelligence" approx_tokens: 1934 --- # Which Quant to Download: 4 Essential Rules for Your VRAM Which quant to download is the first real decision you face after picking a model, and model pages rarely help — you get a list of twenty files and no guidance. The good news is that the answer is mostly determined by one number you already know: how much VRAM your graphics card has. Four rules cover nearly every case. ## Which Quant to Download, by VRAM Start here, then read the rules that follow for the exceptions. These assume a 27B-class model, which is the size most local setups aim at in 2026. ![Which quant to download — recommendations by VRAM tier](https://cdn.sanity.io/images/gfihpee1/production/bac64787973ed97742807cfcfc7b198ca46b9eed-1220x872.png) _Figure: the starting point by card. Sources: r/LocalLLaMA "Qwen3.6-27B Quantization Benchmark" for the 16GB guidance; the RTX 3090 12-quant test for 24GiB; a 32GB commenter in the same thread._ | Your VRAM | Reasonable pick | Why | | --- | --- | --- | | 8 GB | a smaller model at Q4_K_M | a 27B model will not fit; drop parameter count, not precision | | 12 GB | a mid-size model at Q4_K_M | still the best quality-per-gigabyte point | | 16 GB | IQ4_XS (~14.6–15.1 GB) | fits, but the margin is thin — see rule two | | 24 GB | IQ4_NL-mtp or UD-Q4_K_XL | the tester's "best overall balance" at this tier | | 32 GB+ | Q6_K if it fits | one commenter's rule: stay at Q6 unless it will not load | _Table: starting points, not laws. Model size matters as much as quantization level._ ## Rule One: Bigger Model Beats Higher Precision If you have to choose between a larger model at 4-bit and a smaller model at 8-bit, the local AI community has converged fairly firmly on the larger model. The reason shows up in the benchmarks. Going from Q8 to Q4_K_M cost one tester nothing they could measure — the two were **indistinguishable within the error bars of the test (±0.045)** — while saving 10 GB. Their conclusion was that there is "no reason to run Q8 on a 16GB card". Ten gigabytes buys a great deal more model. That does not extend downwards forever. Below roughly 3-bit, [quality falls away sharply](https://learnaitodayonline.com/quantization-quality-loss/) — KLD crossed 0.1 from Q3_K_XL in one full-range test. The rule is "prefer size at 4-bit", not "quantize as hard as possible". ## Rule Two: On 16GB, Watch the Margin Sixteen gigabytes is the most common awkward case, because a 27B model at IQ4_XS _almost_ fits comfortably. ![IQ4_XS file sizes against a 16GB card](https://cdn.sanity.io/images/gfihpee1/production/a09c68a219632a236b2742e0ad97ed20e1dfa4d6-1522x773.png) _Figure: two published IQ4_XS builds against the card's ceiling. Source: r/LocalLLaMA, "Qwen3.6-27B Quantization Benchmark"._ The same thread compared specific community builds: mradermacher's IQ4_XS at **15.1 GB** was judged "a bit tight" on a 16 GB card, while cHunter's at **14.7 GB** was "also very good". Four hundred megabytes decides whether you have room for a usable context window. Two practical notes from that thread. On 16 GB you should not expect more than roughly **65k context** with this model — the [KV cache](https://learnaitodayonline.com/kv-cache-quantization/) claims the rest. And for this particular model at this size, the advice was **not** to use the `--pure` option, which forces a uniform quantization and gives up the mixed-precision gains. ## Rule Three: Read the Whole Filename The number in the filename is the headline, but the letters change the answer. ![Four GGUF filenames decoded](https://cdn.sanity.io/images/gfihpee1/production/7a6c099c1faba313a2ba5cbc1d21897424a81ad7-1140x589.png) _Figure: the four patterns you will actually meet. Sources: Hugging Face GGUF docs; bartowski's quantization notes._ - **Q4_K_M** — standard 4-bit, K-quant, medium. The safe default. - **IQ4_XS** — an "importance" quant, calibrated so precision goes where it matters. Often smaller than Q4_K_M at similar quality, which is why it wins on 16 GB. - **UD-Q4_K_XL** — Unsloth Dynamic, mixing precision across layers. Scored highest of the Qwen 4-bit variants in the RTX 3090 test. - **UD-IQ2_XXS** — 2-bit, the smallest published. A last resort, not a choice. One warning from the measurements: NVFP4 files came out the same size as IQ4_XS but **3–5% worse** in one tester's runs. Same size does not mean same quality, so the format matters as much as the bit count. [Quesma's July 2026 comparison](https://quesma.com/blog/qwen-quantization-quality/) is a good place to see the full ladder side by side. ## Rule Four: Speed Is Not the Quant's Job Alone It is tempting to assume a smaller file always runs faster. Other settings interfere. ![A smaller quant with MTP outrunning a larger one](https://cdn.sanity.io/images/gfihpee1/production/a8930563960106d88dd06485bce08e0c8f0e63e6-1141x778.png) _Figure: generation speed at context 0. Source: r/LocalLLaMA, "Qwen3.6-27B Quantization Benchmark"._ In that comparison, a non-MTP IQ4_XS build produced **25.3 tokens/sec** while a smaller IQ3 build with multi-token prediction enabled hit **33.7 tokens/sec**. The faster one was not faster because it was smaller — MTP changed the throughput independently of the quantization level. The practical consequence: when you compare two files, check whether they differ in anything besides the quant. Otherwise you will credit the compression for a gain that came from elsewhere. ## The Short Version If you want one line: **download Q4_K_M, or IQ4_XS if you are tight on memory, and spend the memory you saved on a bigger model.** That covers most people on most cards. If a name in this guide was unfamiliar, [quantization explained](https://learnaitodayonline.com/quantization-explained/) decodes the terminology properly. If you write code with local models, read [quantization for coding](https://learnaitodayonline.com/quantization-for-coding/) before settling — the answer shifts. And if you are choosing which model to run in the first place, [the best free local AI models](https://learnaitodayonline.com/best-free-local-ai-models/) works through the current options by hardware tier. --- ## Sources - [Do Qwen3.6 27B quantizations break the pelican? (Quesma Blog, 27 Jul 2026)](https://quesma.com/blog/qwen-quantization-quality/) - [GGUF format documentation (Hugging Face)](https://huggingface.co/docs/hub/gguf) - [Quantization notes (bartowski)](https://github.com/bartowski1182/llm-knowledge/blob/main/quantization/quantization.md) - [Qwen3.6-27B quantization benchmark (r/LocalLLaMA)](https://www.reddit.com/r/LocalLLaMA/comments/1tr9vzn/qwen3627b_quantization_benchmark/) - [Qwen 3.6 27B GGUF quality benchmark (LocalBench)](https://localbench.substack.com/p/qwen-3-6-27b-gguf-quality-benchmark) - [Lessons from GGUF evaluations (The Kaitchup)](https://kaitchup.substack.com/p/lessons-from-gguf-evaluations-ternary) - [Unsloth Dynamic GGUFs on Aider Polyglot (Unsloth docs)](https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs/unsloth-dynamic-ggufs-on-aider-polyglot) _Last reviewed: 23 August 2026. File sizes are for one 27B model and change with each release; every figure here is a single-tester measurement on named hardware. Re-checked quarterly._