LLM Quantization Explained: 4-Bit, 8-Bit and What You Lose

Llama 3.1 8B stored at 16 bits per weight is a 14.96 GiB file. The same model at the popular Q4_K_M level is 4.58 GiB, about a third of the size, according to the llama.cpp quantization README. Nothing was retrained; every weight was rounded to a coarser value. That is LLM quantization explained in one sentence: fewer bits per number, a smaller and usually faster model, and a small, measurable change in what it says. The useful questions are how small, and where it lands.

What quantization does to a weight

Weights are ordinary decimal numbers, usually 16-bit floats. Quantization replaces each with a small integer plus a shared scale factor, and the runtime rebuilds an approximate value.

Take a simple 4-bit scheme. Four bits give 16 possible integers, from -8 to 7. Weights are handled in blocks of 32, and each block gets one scale, chosen so the largest weight in the block maps to the edge of the range. Suppose the largest weight in a block is 0.44. The scale is 0.44 / 7 = 0.0629. Now round three weights from that block:

Original	Divided by scale	Stored integer	Rebuilt value	Error
0.44	7.00	7	0.440	0%
-0.31	-4.93	-5	-0.314	1.4%
0.05	0.79	1	0.063	26%

Large weights come back almost exactly; the small one is off by a quarter. Every method is a way of spending few bits where error hurts most.

The storage cost is visible too. Thirty-two weights at 4 bits each is 128 bits, plus a 16-bit scale makes 144 bits, or 144 / 32 = 4.5 bits per weight. This is why a "4-bit" file is never exactly a quarter of the 16-bit size. The scales ride along.

How to read the level names

GGUF names: Q4_K_M and friends

A name like Q4_K_M has three parts:

• Q4 is the nominal bits per weight for most of the model.
• K marks a "k-quant", introduced in llama.cpp pull request 1684. Weights are grouped into super-blocks of 256, split into smaller blocks that each get their own scale, and the scales themselves are quantized to save space.
• S, M or L is the mix. The plain variants keep everything at the base level. The larger mixes keep certain sensitive tensors at a higher precision. The original k-quant pull request says Q4_K_M uses 6-bit Q6_K for half of the attention.wv and feed_forward.w2 tensors and Q4_K for the rest, while Q4_K_S uses Q4_K throughout.

Names starting with IQ (IQ2_XS, IQ3_M, IQ4_XS) are a newer family designed to hold up better at low bit counts, and the 2- and 3-bit ones are normally built with an importance matrix (more on that below). Q8_0 and Q4_0 are older, simpler block formats; F16 and BF16 are the 16-bit originals.

The nominal number in the name is not the real size. Because of the scales and the higher-precision tensors in each mix, Q4_K_M averages 4.89 bits per weight and Q3_K_M lands at 4.00, according to the llama.cpp quantize README. The table in the next section lists the effective figure for each common level; use those, not the nominal ones, when you work out how much VRAM a model needs.

GPU engine names: GPTQ, AWQ, FP8

On Hugging Face you will also see repositories labeled GPTQ, AWQ or FP8. These name the method used to choose the rounded values, and the files are usually served by GPU engines such as vLLM rather than llama.cpp. AWQ, for example, finds the weights that matter by watching activations on sample text; its authors report that protecting about 1% of salient weights greatly reduces error. Which runtime loads which file is covered in the guide to model file formats and runtimes.

One more flag to know: llama.cpp's --imatrix option feeds an importance matrix, computed from sample text, into quantization. The README says a suitable importance matrix can reduce quality loss. Many published GGUF files are built this way.

How much quality is lost, by the numbers

The llama.cpp perplexity tool compares a quantized model against the 16-bit original token by token. Two of its statistics are easy to read. KL divergence measures how far apart the two probability distributions are, where 0 means identical. "Same top p" is the share of positions where both models rank the same token first. The divergence and top-token figures below are for LLaMA 3 8B on the Wikitext-2 test set, from the perplexity README, quantized without an importance matrix. The bits per weight come from the quantize README's Llama 3.1 8B measurements.

Level	Bits per weight	KL divergence	Same top token
Q8_0	8.50	0.0014	97.7%
Q6_K	6.56	0.0055	96.0%
Q5_K_M	5.70	0.0108	not reported
Q4_K_M	4.89	0.0313	91.9%
Q3_K_M	4.00	0.1019	not reported
Q2_K	3.16	0.4451	71.1%

Two patterns stand out. Each step down multiplies the divergence by roughly two to four, and the curve bends hard at the bottom: Q2_K diverges about 14 times more than Q4_K_M (0.4451 / 0.0313 = 14.2). An importance matrix narrows the gap but does not close it; the same README lists Q2_K built with one at 0.3322, still about ten times Q4_K_M. And even Q8_0 is not identical; it disagrees on the top token about 2 times in 100.

In working terms, 91.9% means about 8 positions in 100 where the quantized model's first choice differs, measured with both models reading the same text. Over a 500-token answer that is around 40 chances to take a different word. Most are harmless, a "but" instead of a "however". The ones that matter decide the rest of the answer.

Where the loss shows up first

Three areas break before general chat does.

Long reasoning. A chain of steps compounds small errors: one wrong token early in a calculation and every later step builds on it. A 2024 study of multilingual models, Marchisio et al., found that challenging tasks such as mathematical reasoning were the first to decline under quantization. A 2025 study of reasoning models by Liu et al. reported near-lossless results at 8-bit weights and activations, or 4-bit weights with 16-bit activations, but significant risk at lower bit widths, with model size and task difficulty among the deciding factors.

Exact figures. An invoice total, a part number or a dosage has exactly one correct answer. In prose a near-miss token is a synonym; in a number it is a wrong digit that reads just as confidently. Extraction and copy tasks deserve their own test cases.

Less common languages. The same Marchisio study found non-Latin-script languages were hit hardest. It also found that automatic metrics badly understated the harm: for Japanese, automatic tasks showed an average drop of 1.7%, while human evaluators on realistic prompts reported 16.0%. A model that looks fine on English benchmarks may write noticeably worse Japanese, Arabic or Hindi at the same level.

Summarizing an English email or classifying a ticket usually survives 4-bit with little visible change, which is why Q4_K_M is a common default.

How to test the loss on your own task

The tables above are averages on Wikipedia text with one 8B model. Only your own examples show where your task sits on the curve, and the method takes an afternoon.

1. Collect 30 to 50 real examples of the job: actual documents, actual questions, and the answer a competent person would accept. Include long documents, tables of figures and non-English material on purpose. The same set serves later when you pick between open-source models.
2. Pick a reference. Run the examples through the highest precision you can load, ideally F16 or BF16, otherwise Q8_0, whose divergence in the table above is very small.
3. Run two or three candidate levels, for example Q6_K, Q5_K_M and Q4_K_M, with the temperature set to 0 so differences come from the weights, not random sampling.
4. Score by task type. Exact match for extracted figures and codes. Pass or fail against your accepted answer for reasoning questions. A blind side-by-side read for drafted text, where the reviewer does not know which level wrote which answer.
5. Choose the smallest level that ties the reference on the cases you care about, then spend the saved memory on context length or a larger model.

For a quick numeric check without scoring answers by hand, llama.cpp can compute KL divergence on your own text. Run llama-perplexity on the 16-bit model with -f pointing at a text file of your documents and --kl-divergence-base naming an output file, which saves its logits. Then run the quantized model with the same --kl-divergence-base file plus --kl-divergence. The README warns the logits file is large, 37 GiB for LLaMA 3 on Wikitext-2, so use a modest sample, and check llama-perplexity --help since flags change between releases.

One trade-off often decides the question: a bigger model at 4 bits frequently beats a smaller model at 8 bits in the same memory. A 14B model at Q4_K_M takes about 14 x 4.89 / 8 = 8.6 GB, close to an 8B model at Q8_0 at 8 x 8.50 / 8 = 8.5 GB. Run both through the same test set before deciding, since the answer depends on the task.

Frequently asked questions

Does quantization make a model faster?

Usually, for generating text, because generation speed is limited by how many bytes the GPU must read per token, and smaller weights mean fewer bytes. In the llama.cpp README's 8B measurements, Q8_0 generated about 51 tokens per second against about 29 for F16. Prompt processing did not speed up in the same table; F16 read prompts slightly faster than any quantized level.

Is it possible to quantize a model in-house?

Yes. llama.cpp ships a llama-quantize tool that converts a 16-bit GGUF file into any of the levels above, optionally with an importance matrix. It mainly matters for fine-tuned models, since popular ones are published ready-made.

Is a quantized model safe to use for regulated work?

Quantization changes accuracy, not where the data goes, so it has no bearing on privacy or compliance by itself. What it does change is error rate on figures and reasoning, so any workflow where a wrong number has consequences should keep a human review step and be tested at the exact level that will run in production.

Should the context cache be quantized too?

It can be, and it saves memory on long contexts, but it is a separate setting with its own quality cost. Test it the same way: run your long-document cases with and without it and compare answers before relying on it.