How a model gets smaller, and how to check it’s still the same model

How do you fit a number into 4 bits?

The idea

Round it. If every price in a shop were rounded to the nearest 5 cents, there’d be fewer different prices, and none would be off by more than 2.5 cents. Quantisation does the same to weights: it sets out a grid of allowed values and moves each weight to the nearest point on it.

The grid comes from a scale. A block of weights shares one scale, kept at higher precision; each weight is stored as a small whole number, and whole number × scale gives the weight back, approximately. Getting the approximate value back is called dequantising.

The worked example

1. Four weights: 0.12, −0.53, 0.98 and 0.31. The target is 4-bit signed whole numbers from −7 to 7. (Four bits can hold −8 to 7; −7 to 7 keeps the grid symmetric.)
2. The scale stretches the largest magnitude to the top of the range: 0.98 ÷ 7 = 0.14.
3. Divide each weight by the scale and round: 0.86, −3.79, 7.00 and 2.21 become 1, −4, 7 and 2. Only those four small numbers and the one scale are stored.
4. Dequantise by multiplying back: 0.14, −0.56, 0.98 and 0.28. The errors are 0.02, 0.03, 0.00 and 0.03.

Why the errors stay small

Rounding to the nearest whole number is never off by more than a half, so no weight moves by more than half a step: scale ÷ 2, or 0.07 here. That bound is what makes a round-to-nearest quant checkable (see Dequantise and compare). The slider shows the trade: every bit removed halves the number of grid points and roughly doubles the step.

Where you’ll see it

llama.cpp’s Q8_0 and Q4_0 round to nearest (Q4_0 maps each block’s largest weight to −8 and uses the whole −8 to 7 range), and so does MLX’s default quantisation, with a scale and an offset for every 64 weights. These need only the weights, so anyone can repeat them. MLX conversions can also use calibrated methods, so an “MLX 4-bit” label alone doesn’t say which.

What it can’t do. Every weight gets the same treatment, however much it matters to the output. At 8 bits, and often at 4, that’s fine; at 3 bits and below, plain rounding loses far more quality than the calibrated methods further down.

Why not one scale for the whole tensor?

The idea

One unusually large weight sets the scale for everything that shares it. If a whole tensor shared one scale, a single outlier would stretch the grid and flatten most small weights to zero or one step. Splitting the tensor into small blocks, each with its own scale, keeps the damage inside one block.

Smaller blocks are more accurate, but every scale costs bits. llama.cpp’s Q4_0 stores blocks of 32 weights with one 16-bit scale, so it spends 4 + 16 ÷ 32 = 4.5 bits a weight; Q8_0 uses the same blocks with 8-bit values, 8.5 bits a weight. At 4.5 bits, a 70-billion-weight model takes about 39 GB.

What the animation shows

1. 32 made-up weights, one of them an outlier, rounded to 4 bits with one shared scale.
2. Two blocks of 16: the block without the outlier gets a finer grid of its own.
3. Four blocks of 8: only the outlier’s block stays coarse and the average error drops, but each weight now carries 6 bits.

K-quants

llama.cpp’s K-quants (Q4_K, Q5_K, Q6_K and the rest) go a level further: super-blocks of 256 weights split into sub-blocks, whose own scales are quantised to save space. Q4_K also stores a minimum for each sub-block and still comes to about 4.5 bits a weight. Instead of taking the largest weight, it searches for the scale with the smallest error, which can clip a few of the largest values. A suffix like Q4_K_M names a mix: the output layer and parts of the attention and feed-forward weights are kept at 6 bits (Q6_K).

What it can’t do. Blocks contain an outlier’s damage but don’t remove it: its block still gets a coarse grid. Methods that keep outliers at higher precision (SpQR) or rescale channels before rounding (AWQ) go further.

Why would 4 bits want uneven steps?

The idea

Whole numbers spread their values evenly. Floating-point numbers, like scientific notation, store an exponent and a few digits, so their values sit close together near zero and further apart as they grow. Trained weights cluster near zero with a few large ones, which suits that shape.

The 4-bit floating-point format, FP4 (called E2M1: a sign, two exponent bits and one mantissa bit), can hold 0, 0.5, 1, 1.5, 2, 3, 4 and 6, each positive or negative. FP8 comes in two layouts: E4M3, which reaches 448, and E5M2, which reaches 57,344.

Two 4-bit block formats

• MXFP4, from the Open Compute Project’s Microscaling specification: blocks of 32 FP4 values sharing one 8-bit scale that can only be a power of two. That’s 4 + 8 ÷ 32 = 4.25 bits a weight.
• NVFP4, from NVIDIA: blocks of 16 FP4 values, each block with an FP8 (E4M3) scale, plus one 32-bit scale for the whole tensor. Smaller blocks and finer scales cost a little more: about 4.5 bits a weight.

Rounding or calibrated?

Within each block the weights are rounded to nearest, but the format alone doesn’t say how a file was made. NVIDIA’s tools quantise both the weights and the activations (the numbers passed between layers) by default, and set the activations’ scales from sample text. A weight-only NVFP4 file made by rounding needs no sample text, and other tools may apply calibrated methods before rounding.

What it can’t do. A format sets the grid, not the method: a file labelled NVFP4 or MXFP4 doesn’t tell you on its own whether it was made by plain rounding.

Can one weight’s rounding error be made up by its neighbours?

The idea

Rounding a weight up or down changes the layer’s output a little. GPTQ’s insight is that the weights not yet rounded can be nudged to cancel some of that change. It works through a layer’s weights a column at a time: round a column, then adjust the columns still to come so the layer’s output on the sample text moves back towards the original.

Because each weight is rounded after earlier nudges, its final grid point can differ from the one nearest its original value. That’s deliberate: the layer as a whole stays closer to the original than plain rounding gets it. The animation is a real, tiny GPTQ run on made-up numbers.

Step by step

1. Run sample text through the model and record the inputs each layer sees. The paper used 128 passages of 2,048 tokens from C4, a public web-text dataset.
2. From those inputs, work out how the errors of different weights interact (second-order information about the layer).
3. Round the first column to the grid.
4. Adjust every column not yet rounded to cancel the error, then move to the next column. Real implementations do this 128 columns at a time on a GPU.

A relative

Intel’s AutoRound reaches for the same goal differently: it learns each weight’s rounding direction, and how much to clip, with about 200 steps of gradient descent on sample text.

What it can’t do. The corrections depend on the sample text and the order of the columns, so a GPTQ file can’t be rebuilt without them, and its weights break the half-step bound that checks plain rounding. The choice of sample text can also change the result.

Which weights deserve the finest grid?

The idea

A weight’s rounding error reaches the output multiplied by the input it meets, so a weight whose input is usually large matters more than one whose input is usually small. AWQ (activation-aware weight quantisation) starts from the observation that protecting about 1% of weights, those on the channels with the largest inputs, recovers much of the quality rounding loses.

It protects them without spending extra bits. Each input channel gets a scale that grows with how large its inputs run on sample text: the weights are multiplied by it before rounding and the inputs divided by it, with the division folded into the operation before. The maths is unchanged, but the important weights lose less to rounding. The animation shows a simplified version, with one channel scaled by 2.

Step by step

1. Run sample text and record the average size of each input channel.
2. Search for the scaling strength that keeps the layer’s output closest to the original.
3. Scale the weights, fold the inverse into the operation before, and round to nearest. No retraining is needed.

llama.cpp’s imatrix

An importance matrix (imatrix) uses the same insight differently. A tool runs sample text through the model and records how large each channel’s inputs are; the quantiser then weighs each rounding error by that importance when it chooses scales. The lowest-bit formats (IQ1, IQ2 and IQ3_XXS) refuse to quantise without one.

What it can’t do. AWQ’s scales are folded into neighbouring weights, so an AWQ file’s numbers differ from the original’s even before rounding; comparing them weight by weight means undoing those scales first. An imatrix quant can’t be rebuilt without its imatrix file and the same llama.cpp version.

What if the grid itself is learned?

The idea

Rounding gives each weight its own grid. Codebook methods round groups of weights together: they learn a small table of typical patterns, a codebook, and store for each group the number of its nearest entry. A table fitted to where the weights actually are wastes fewer values than a regular grid, which matters most at 2 to 3 bits a weight.

AQLM learns several codebooks and adds their entries together. QTIP, and the EXL3 format built on it, encodes weights along a trellis (see EXL3). llama.cpp’s 2-bit i-quants (IQ2_XXS and friends) use points of a fixed lattice (E8) instead of a learned table. All of them use sample text.

Training for low precision

Quantisation-aware training (QAT) goes further: the model is trained or fine-tuned while its weights are rounded on the way through, so it learns to tolerate the precision it will run at. Google released QAT versions of Gemma 3 this way, and Meta’s LLM-QAT paper did it for LLaMA models. A QAT model’s weights have legitimately moved away from the original’s.

What it can’t do. A codebook quant can’t be compared with the original weight by weight in any simple way, and QAT changes the weights on purpose. For both, the only check against the original is how the model behaves.

How can a weight stored in 2 bits have 65,536 possible values?

The idea

EXL3 is the format of ExLlamaV3, a library for running models locally on consumer GPUs, and several configurations on the Local board use it. It’s a streamlined version of QTIP, a 2024 method from Cornell, and it combines three ideas: rotate the weights so outliers are spread thin, store them as one long string of bits read through a sliding window, and feed rounding errors forward as GPTQ does.

The sliding window is the unusual part. Each weight is decoded from a 16-bit window of the string, but at 2 bits a weight the window then moves on by only 2 bits, so neighbouring weights share most of their bits. Each weight can still take any of 65,536 values while costing only 2 new bits. The catch is that the weights can’t be chosen one at a time, so a search chooses the whole string at once.

Step by step

1. Flip signs at random, then rotate each tensor on both sides with a Hadamard transform: a fixed pattern of sums and differences, applied in blocks of 128. Every weight is shared among many others, so outliers are spread thin and the weights take on a bell shape. The file keeps the signs and scales needed to undo it, and at run time the rotation is undone, either on each layer’s inputs and outputs or on the weights themselves.
2. Run sample text through the model and record what each layer sees: by default a bundled mix of 250 rows of 2,048 tokens.
3. Split each tensor into tiles of 16 × 16 weights, each stored as one string of bits. The Viterbi algorithm, a standard way to find the best path through a chain of linked choices, finds the string whose decoded values come closest to the rotated weights.
4. As the tiles are encoded in turn, adjust the weights still to come to make up for the errors so far, using the sample text: the same kind of update as GPTQ.

Where the values come from

Each window’s 16 bits go through a short fixed formula instead of a stored table. In the default codebook, the window is multiplied by a constant, the four bytes of the result are added up, and the total is centred and scaled. A total of four random-looking bytes is bell-shaped, just as four dice total 14 far more often than 4, so the 65,536 possible windows give a bell-shaped spread of values to match the rotated weights. A formula needs no table lookups, which keeps decoding fast on a GPU.

Reading the labels

A label like 4.0bpw gives the average bits a weight. Each tensor gets a whole number of bits a weight from 1 to 8, or 1.5, 2.5 or 3.5 with the default codebook, and the converter mixes them so the whole model comes to about the target. A Local board label such as “EXL3 4.0bpw experts, FP8 dense” means the experts (the blocks of weights a mixture-of-experts model picks between for each token) are EXL3 at that average, and the other layers are FP8.

What it can’t do. An EXL3 tensor can be decoded and rotated back for comparison, but with shared bits and errors fed forward, no simple bound says how far each weight may move. The converter’s default sample text ships with it and its random signs are seeded from each layer’s position, so a rebuild with the same version and settings starts from the same inputs; nothing published says it gives identical bytes, on the same GPU or another. In practice, the check is comparing outputs.

Does rebuilding it from the original give the same bytes?

The idea

A hash is a short fingerprint computed from a file’s bytes; SHA-256 is the usual one. Change a single bit anywhere and the fingerprint changes completely, so two files with the same SHA-256 are, for practical purposes, identical.

Round-to-nearest methods have no randomness in them: the same weights, tool, version and settings give the same output. So a round-to-nearest quant can be rebuilt from the original weights and the two hashes compared. A match means the download is exactly what the original weights produce, without having to trust whoever made it.

Step by step

1. Download the original weights at a fixed commit of the publisher’s repository. Hugging Face shows a SHA-256 for each large file, so each download can be checked against it.
2. Quantise them with the same tool, version and settings the quant’s maker used.
3. Hash the result and compare it with the download’s hash.

What it can’t do. A different tool version, or metadata written into the file, can change the bytes even when no weight differs; hashing tensor by tensor sidesteps the metadata. Calibrated methods can’t be rebuilt without their sample text. HQQ is a near miss: it needs no sample text, but it optimises each block’s zero point in a loop, and nothing published says the result is identical across GPUs and library versions.

Is every weight within half a step of the original?

The idea

When a quant can’t be rebuilt byte for byte, its numbers can still be compared. Dequantise each tensor and subtract the original, weight by weight. For round to nearest, the worked example’s bound must hold: no weight further than half a step from the original. A weight well outside means the quant wasn’t made from these weights by rounding.

There are edge cases. A format can clip a value at the edge of its range: Q4_0 can clip a value nearly as large as its block’s largest but of the opposite sign, and K-quants’ scale search can clip a few of the largest values. Floating-point formats have uneven steps, so their bound is half the local step.

How the other families look

• imatrix quants stay close to the bound, but can’t be rebuilt exactly without the imatrix file.
• GPTQ and AutoRound break it by design: moving weights off their nearest level is the point.
• AWQ needs its scales undone first, since they’re folded into neighbouring weights.
• Codebook, EXL3 and QAT quants have no simple weight-by-weight bound.

What it can’t do. The bound shows the numbers came from these weights by rounding. It can’t confirm methods that move weights on purpose, and it says nothing about whether the original weights can be trusted.

Does it answer like the original?

The idea

When the numbers can’t be compared, compare behaviour. At each step a model gives every possible next token a probability. KL divergence measures how far the quant’s probabilities are from the full-precision model’s on the same text: zero means identical, and larger means more different.

llama.cpp’s perplexity tool has a mode for this. It first saves the full-precision model’s probabilities over a test text, then runs the quant against them and reports the mean KL divergence and how often both models give the same token the highest probability (“Same top p”), among other statistics.

Step by step

1. Run the full-precision model over a test text and save its probabilities for every token.
2. Run the quant over the same text.
3. Average the KL divergence over every position, and count how often the top token agrees.

What it can’t do. This catches broken and badly degraded quants, but not subtle tampering. A backdoor that switches on only for a trigger phrase looks perfect on ordinary test text, because the text never contains the trigger, and such backdoors can survive safety training (Hubinger et al., 2024). The check also needs the full-precision model to run, which may not fit.

Can opening a model file run code?

The idea

Older PyTorch checkpoints (.bin and .pt files) are saved with Python’s pickle, a general format for saving Python objects. A pickle is a little program: loading it follows its instructions, which can call any function. Python’s own documentation warns: “The pickle module is not secure. Only unpickle data you trust.”

Safetensors is a deliberately simple alternative. The first 8 bytes give the length of a JSON header listing each tensor’s name, data type, shape and byte range; the raw bytes follow. There’s no code in it, so there’s nothing to run. A 2023 audit by Trail of Bits found no flaw leading to code execution, and the transformers library has saved safetensors by default since version 4.35.

What it makes possible

1. Safe loading: reading a header and copying bytes can’t execute anything.
2. Fast loading: the file can be memory-mapped, so only the parts actually used are read, and one tensor can be loaded without the rest.
3. Practical checking: each tensor’s bytes sit at a known place, ready to hash on their own or to dequantise and compare.

What it can’t do. Safetensors makes loading safe; it doesn’t make the weights trustworthy. A backdoored model saved as safetensors is still backdoored, so where the file came from still matters. PyTorch 2.6 and later also load .bin and .pt files more defensively by default (weights_only), but the pickle format itself still allows code.