What is quantization in LLMs, and what do Q4_K_M and Q8_0 mean?

Quantization stores a model's weights with fewer bits, so the model takes much less memory and disk space at the cost of some accuracy. It is the reason an 8-billion-parameter model that needs about 15 GiB at 16 bits fits in about 4.6 GiB as Q4_K_M, and why you can run it on an 8 GB graphics card. llama.cpp's documentation says quantization "may introduce some accuracy loss", usually measured as perplexity or KL divergence. This guide explains what the names Q4_K_M, Q8_0 and MXFP4 mean, how much memory and quality each level trades, and how to choose, using the documentation of llama.cpp and Hugging Face and the original measurements by the author of the k-quants, all checked on October 6, 2026.

What is quantization?

A language model is mostly a very large set of numbers, its weights. Trained models usually keep them as 16-bit or 32-bit floating-point numbers. Quantization stores each weight with fewer bits, for example 4 or 8, plus a small amount of extra data that lets the computer rebuild approximate values. The llama.cpp quantize documentation describes it as reducing the precision of the model weights, for example from 32-bit floats to 4-bit integers, which shrinks the model's size and can speed up inference.

The saving is large. For Llama 3.1 8B, llama.cpp's table gives these sizes, with the 16-bit size from our own calculation:

Table: size of Llama 3.1 8B by quantization (llama.cpp measurements, October 6, 2026)

The bits per weight are above the nominal 4, 5 or 8 because each block of weights also stores scaling factors. To see what a given size means for your graphics card, use our VRAM calculator or the GPU guide.

What do Q4_K_M, Q8_0 and the other names mean?

The names come from llama.cpp and the GGUF file format, described by Hugging Face as a binary format optimized for quick loading and saving of models. Read them from left to right:

• The number is the nominal bit width: Q4 is 4-bit, Q8 is 8-bit.
• K marks the "k-quants", a family added to llama.cpp in June 2023, which store weights in super-blocks of smaller blocks with their own scales. Hugging Face's table gives Q4_K at 4.5 bits per weight and Q6_K at 6.5625.
• The last letter (S, M, L) is a "quantization mix", as the k-quants pull request explains: it decides which tensors get more bits. Q4_K_S uses Q4_K for all tensors, while Q4_K_M uses the higher-precision Q6_K for half of the attention.wv and feed_forward.w2 tensors and Q4_K for the rest; Q3_K_L uses Q5_K for those tensors and Q3_K for the others.
• Q8_0 and the other _0 and _1 types are older, simpler formats. Hugging Face lists them as legacy methods that are not widely used today, although llama.cpp still offers Q8_0 and it is the 8-bit option in our memory tables.
• IQ types (IQ4_XS, IQ3_S and so on) use an importance matrix to preserve the most important weights at low bit widths.
• MXFP4 is a 4-bit "microscaling block floating point" format. OpenAI says its gpt-oss models were post-trained with MXFP4 quantization of the mixture-of-experts weights, which is how gpt-oss-20b runs within 16 GB of memory.

How much quality do you lose?

The most direct evidence is the original measurement published with the k-quants in 2023, on the LLaMA 7B model: perplexity (lower is better) on the wikitext dataset, against the 16-bit model's 5.9066.

Table: quality and size of k-quants on LLaMA 7B (original measurements, 2023)

The shape is what matters: from 16 bits to Q5_K_M the loss in this test is a quarter of a percent, around Q4_K_M it is under 1%, and below 4 bits it grows quickly. The author's own summary is that perplexity is a fairly smooth function of model size, so you can pick the best model your memory allows. Two caveats: these numbers are for a 2023 model, so they show the trend rather than predict a current model, and perplexity is not the same as how good answers feel on your task. llama.cpp says the loss is usually measured with perplexity and KL divergence. If a model is borderline, try the next format up before buying more memory.

How do you choose a quantization?

A practical rule, which is our own reading of the data above:

1. Start with Q4_K_M. In the original test it keeps the loss under 1% at about 29% of the 16-bit size, and it is the main 4-bit format in our memory tables.
2. Go up to Q5_K_M, Q6_K or Q8_0 if you have the memory. The loss shrinks, but the file grows, and your context window also needs memory.
3. Go down to Q3 or below only if the model will not fit otherwise. A larger model at Q3 is often better than a smaller one at Q8, but test it on your own tasks.
4. Check that the model fits. With Ollama, run ollama ps and look for "100% GPU", as explained in our Ollama commands cheat sheet.

Ollama does not quantize GGUF models when you import them: its documentation says to prepare and quantize them first with a GGUF tool such as llama.cpp's llama-quantize. Ollama's library pages show the download size of each model tag, which is the memory the weights will need.

Can the cache be quantized too?

Yes. The key-value cache that holds the conversation can be stored with fewer bits. Ollama's OLLAMA_KV_CACHE_TYPE accepts f16 (the default), q8_0, which uses about half the memory, and q4_0, which uses about a quarter; it requires Flash Attention. This does not change the weights, only the memory used by long contexts.

Quantization and cloud APIs

Quantization is mostly a topic for models you run yourself. When you use a cloud API, the provider decides how the model is served and you pay per token, as in LLM API pricing compared. Running a quantized model locally has no per-token price, only the cost of the machine; see how to run an LLM locally for the steps.