Quantization: INT8, INT4, GPTQ and AWQ

Imagine you have a huge recipe book where every measurement is written to six decimal places — '2.718281 grams of salt.' In practice, nobody needs that precision; '2.7 grams' cooks the same dish and this course gets far thinner. Quantization does exactly this to a model's weights: it stores each number using fewer bits.

By default, LLM weights are stored in 16-bit floating point (FP16 or BF16). A 7-billion-parameter model in FP16 needs about 14 GB just to hold the weights (2 bytes x 7B). Quantizing to 8-bit integers (INT8) roughly halves that to ~7 GB; quantizing to 4-bit (INT4) roughly quarters it to ~3.5 GB. That's the whole point: a model that wouldn't fit on a consumer GPU in FP16 suddenly fits comfortably in INT4, and because less data has to be moved from GPU memory on every token, inference also gets faster (LLM inference is usually memory-bandwidth-bound, not compute-bound).

The trade-off is precision. Squeezing a wide range of values into 16 possible levels (4-bit) inevitably rounds them, and that rounding error can degrade output quality. The art of modern quantization is minimizing that damage. Two methods dominate for 4-bit:

GPTQ (Gradient/Post-Training Quantization) quantizes weights one layer at a time, using a small calibration dataset and second-order (Hessian) information to choose rounding that minimizes the error each layer introduces to its outputs. It's accurate and widely supported.

AWQ (Activation-aware Weight Quantization) starts from the observation that not all weights matter equally — a small fraction of 'salient' weights (identified by looking at activation magnitudes) carry most of the model's quality. AWQ protects those weights by scaling them before quantizing, so the important channels keep their precision while the rest are compressed hard. It's often slightly faster to apply than GPTQ and gives very competitive quality at 4-bit.

Rules of thumb: INT8 typically costs under 1% quality and is a safe default; INT4 (via GPTQ or AWQ) saves the most memory but should be validated on your actual task, since precise or reasoning-heavy tasks are the most sensitive to the extra rounding. Serving engines like vLLM accept a quantization= flag so you can load an already-quantized checkpoint directly.

# Loading a 4-bit AWQ-quantized model for serving with vLLM.
# The model weights are ~4x smaller than FP16, so a 7B model
# fits in ~4GB of VRAM instead of ~14GB.
from vllm import LLM, SamplingParams

llm = LLM(
    model="TheBloke/Mistral-7B-Instruct-v0.2-AWQ",
    quantization="awq",   # or "gptq" for a GPTQ checkpoint
    max_model_len=4096,
)

params = SamplingParams(temperature=0.7, max_tokens=256)
out = llm.generate(["Explain quantization in one sentence."], params)
print(out[0].outputs[0].text)

# --- Illustrating the memory math behind the choice ---
def weight_memory_gb(num_params_billions: float, bits: int) -> float:
    """Approximate VRAM (GB) just to hold the weights."""
    bytes_per_param = bits / 8
    return num_params_billions * 1e9 * bytes_per_param / 1e9

for bits in (16, 8, 4):
    print(f"7B model at {bits}-bit: ~{weight_memory_gb(7, bits):.1f} GB")
# 16-bit: ~14.0 GB | 8-bit: ~7.0 GB | 4-bit: ~3.5 GB