Home›Quantization

Quantization

How models are shrunk to fit your hardware, and what it costs in quality.

Quantization is how a large model is shrunk to fit on an ordinary graphics card or laptop, by storing its numbers with less precision. It is the reason model files have names like Q4_K_M, and picking the wrong file is a common reason a local model runs slowly or gives poor answers.

Read the full introduction

The explainer decodes the file names and the terms around them, then turns that into rules for each amount of video memory, and the VRAM calculator checks a specific model. The quality post looks at what is lost at each level, using published benchmark results, and explains why coding suffers more than chat. The last post covers quantizing the KV cache, which buys a longer context window on the same hardware.

Much of the evidence in this area comes from community testing rather than formal studies, so the posts say who measured what and how far the results can be trusted.