Benefits of Quantization for Cheaper GPU Inference

Your 70B model needs roughly 140 GB just to sit in memory, and the GPU bill follows the memory. Quantization for GPU inference means storing model weights in 8 or 4 bits instead of 16, and it is the cheapest lever you have on a self-hosted LLM. Below: what it buys you, how to serve a quantized model in vLLM, and when to skip it.

Five benefits of quantization for GPU inference

1. Weights shrink with the bit count. An 8-bit weight takes half the space of a 16-bit one, and a 4-bit weight a quarter. The vLLM FP8 guide puts the saving at 2x for FP8. For a 70B-class model:

| Precision | Weights | One 80 GB GPU? | | --- | --- | --- | | BF16 | ~140 GB | No | | FP8 | ~71 GB | On paper, with no room for KV cache | | INT4 (W4A16) | ~40 GB | Yes, ~30 GB left over |

These are back-of-envelope numbers at 90% GPU memory utilization. INT4 lands above a clean quarter because lm_head and the embeddings stay in 16 bits and scales are stored.

2. The freed memory becomes KV cache. Every running request keeps a KV cache, the stored attention keys and values for each token so far. Memory the weights stop using goes to that cache: larger batches or longer contexts. Compute the per-token cost from your model config:

from transformers import AutoConfig

cfg = AutoConfig.from_pretrained('<your-model-id>')
head_dim = getattr(cfg, 'head_dim', None) or cfg.hidden_size // cfg.num_attention_heads
elements = 2 * cfg.num_hidden_layers * cfg.num_key_value_heads * head_dim  # K and V
print('16-bit KV cache per token:', elements * 2 / 1024, 'KiB')
print('FP8 KV cache per token:   ', elements * 1 / 1024, 'KiB')

For a Llama 3 70B config (80 layers, 8 KV heads, head size 128) this prints 320 KiB in 16-bit, so the ~30 GB left by INT4 holds on the order of 90,000 tokens before overhead. You can quantize the cache too: --kv-cache-dtype fp8 halves the per-token cost. With plain fp8 vLLM runs no calibration and sets every scale to 1.0, so the KV cache docs recommend calibrating with LLM Compressor.

3. You can buy a cheaper GPU class, or fewer GPUs. FP8 weights and activations (W8A8) need NVIDIA compute capability 8.9 or higher: Ada, Hopper or Blackwell. Ampere cards get weight-only FP8 through Marlin kernels, which saves memory but not compute, and the INT4 path needs Ampere or newer. A model that wanted two big cards can often run on one, which also removes the tensor-parallel traffic between them.

4. Throughput and latency can improve. The vLLM docs cite up to 1.6x throughput for FP8 with minimal accuracy impact, and say INT4 helps keep latency low at low queries per second. Treat those as the docs' claims and benchmark your own prompts.

5. Trying it costs little. FP8_DYNAMIC uses plain round-to-nearest quantization, so it needs no calibration data and no retraining.

Quantization for GPU inference in vLLM: load a quantized model

vLLM recommends LLM Compressor for producing FP8, INT8 and INT4 checkpoints, and still loads existing AWQ and GPTQ ones. Install it in a separate environment from vLLM. Here is FP8 from the docs:

from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = '<your-model-id>'
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map='auto', dtype='auto')
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

recipe = QuantizationModifier(targets='Linear', scheme='FP8_DYNAMIC', ignore=['lm_head'])
oneshot(model=model, recipe=recipe)

SAVE_DIR = MODEL_ID.split('/')[1] + '-FP8-Dynamic'
model.save_pretrained(SAVE_DIR)
tokenizer.save_pretrained(SAVE_DIR)

For INT4, the INT4 W4A16 guide uses a GPTQ recipe with calibration data:

# Imports and the calibration dataset `ds` are set up as in the INT4 W4A16 guide
recipe = GPTQModifier(targets='Linear', scheme='W4A16', ignore=['lm_head'])
oneshot(model=model, dataset=ds, recipe=recipe, max_seq_length=2048, num_calibration_samples=512)
model.save_pretrained(SAVE_DIR, save_compressed=True)

The docs suggest starting at 512 samples of 2048 tokens and matching the calibration data to your real traffic. Then serve the saved directory. With no --quantization flag, vLLM reads the checkpoint's quantization_config:

# INT4 on a single 80 GB GPU
vllm serve ./<your-model>-W4A16-G128 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

# FP8 across two GPUs, with an FP8 KV cache
vllm serve ./<your-model>-FP8-Dynamic \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8

No time to produce a checkpoint? vllm serve <model-id> --quantization fp8_per_tensor quantizes at load time, per the current online quantization docs. Older releases spelled it differently, so check your version.

When quantization is not worth it

Quantization is lossy. Small models have less slack, and 4-bit schemes tend to regress before 8-bit ones. If an 8B model already fits your GPU in BF16, you save little and risk more. It is also pointless if the smaller footprint doesn't change the GPU count or class you can buy.

Before switching, run the same eval on both checkpoints. The docs warn that quantized models can be sensitive to the BOS token, so keep add_bos_token=True:

# Run once per checkpoint with identical settings
lm_eval \
  --model vllm \
  --model_args pretrained=<path-or-model-id>,add_bos_token=True \
  --tasks gsm8k --num_fewshot 5 --batch_size auto --limit 250

GSM8K is a math benchmark, so swap in tasks that resemble your workload and replay a few hundred real prompts. Ship it if the gap fits your tolerance. If INT4 fails, try FP8 or recalibrate with data closer to production.

Quantization can turn a 140 GB model into a 40 GB one, and for a team paying by the GPU-hour in production that is a budget line you can actually move, as long as your eval says the answers survived.