Module 3 · Inference engineering · Chapter 25 · Free
GPUs and memory: sizing a model to hardware
Three things a GPU gives you
When you read about a GPU for inference, three separate properties matter. Mixing them up is the most common source of bad sizing advice.
- Compute: how many arithmetic operations per second it can do, quoted separately for each number format. This sets how fast prompts are processed (Chapter 24).
- Memory capacity: how many gigabytes it holds. Often called VRAM. This decides what fits.
- Memory bandwidth: how many bytes per second can be read from that memory. This sets how fast tokens are generated at small batch.
Capacity is the first gate because it is binary. If the weights do not fit, nothing else matters. Speed is a matter of degree; fit is a yes or no.
- For good speed the model weights sit in GPU memory for the whole time the server is running. If they do not fit, the model cannot run efficiently on that GPU.
- The KV cache (Chapter 23) is stored in the same memory, and it grows with context length and the number of simultaneous requests. Whatever the weights leave over is what the cache gets.
- Every decode step streams the weights and cache across this path. Its bandwidth, quoted in terabytes per second on spec sheets, often limits how fast tokens appear.
- When a model is too big for one GPU, it can be split across several. The GPUs then exchange data over a fast link, so the quality of that link matters for multi-GPU speed.
- The host's memory is large but much slower to reach from the GPU. Spilling there keeps things from crashing, but speed usually drops sharply. Plan to fit in GPU memory.
The sizing formula
A first estimate has three parts. Everything is in bytes, then divided by a billion for gigabytes here.
1. Weights = parameters x bytes per parameter. A model with 8 billion parameters stored in a 2-byte format (such as FP16 or BF16) takes 16 GB. In a 1-byte format (8-bit) it takes 8 GB. In a 4-bit format it takes roughly 4 GB, plus a small amount for scaling data. Precision is the biggest lever on the weights.
2. KV cache = 2 x layers x KV heads x head dimension x tokens x batch x bytes per element. The 2 is for keys and values. Use KV heads, not attention heads: in grouped-query attention several attention heads share one set of keys and values, so the cache is smaller than the head count suggests. "Tokens" means context length held per request, prompt plus output, and "batch" means requests alive at once.
3. Activations and runtime overhead: temporary working memory, the inference engine's own buffers and some fragmentation. It is hard to calculate exactly, so the calculator uses a percentage of weights plus cache. A margin of 10 to 20 percent is a common starting assumption. Measure on your engine to refine it.
Two caveats. First, for a mixture-of-experts model, use the total parameter count for weights, since all experts generally must be resident (Chapter 23). Second, some architectures compress or limit the cache (for example with a sliding attention window), which makes this formula an overestimate for them. Check the model's configuration file for the real layer, head and dimension values.
VRAM calculator
Pick a preset or type your own values. Presets are illustrative shapes, not specifications of any named model: look up the real numbers in the model's published configuration. The GPU memory sizes in the table are editable and also illustrative.
VRAM calculator (illustrative presets)
| GPU memory each (GB) | GPUs | Combined | Fits? |
|---|
Show the formulas with your numbers
Reading a spec sheet
Look for three lines first: memory size, memory bandwidth and compute at the precision you plan to use. As a real example, NVIDIA's H100 page, at the time this chapter was written, lists 80 GB of memory at 3.35 TB/s for its SXM version and 94 GB at 3.9 TB/s for its NVL version. Check the current page, as products and listings change. Notice two things. Higher precision numbers for compute are often quoted "with sparsity", which gives a higher figure that only applies to models using that feature, so compare like with like and read the footnotes. And an interconnect speed is listed separately: it matters only when you use more than one GPU.
At a high level, NVIDIA's data center GPUs have moved through architecture generations. Its pages name Hopper (which includes the H100) and the newer Blackwell, whose page highlights support for 4-bit floating point (FP4) in its second-generation Transformer Engine. Newer generations generally bring more memory, more bandwidth and support for lower-precision formats, which together change what fits and how fast it runs. For exact numbers, always go to the current official product page.
Where models run: three tiers
- Data center: GPUs with large memory, high bandwidth and fast links to sibling GPUs, usually rented as instances with one to several GPUs. A big model is split across several. This is the setting for multi-user serving with high throughput.
- Desktop and workstation: consumer or professional GPUs with much less memory. Quantized small and mid-size models often fit for one user or a small team. Good for development, private prototypes and low-volume use.
- Mobile and edge: phones and laptops share one pool of memory between the model and everything else. Only small, heavily quantized models are practical. Privacy and offline use are the draw.
Other accelerators exist beyond NVIDIA GPUs, such as cloud-vendor AI chips. Their trade-offs are mainly in software maturity and availability, so check the engine support you need before committing.
Match the term to what it limits
Check your understanding
Recap
- A GPU gives compute, memory capacity and memory bandwidth. Capacity decides fit, bandwidth decides decode speed, compute decides prefill speed.
- Total memory is roughly weights + KV cache + overhead. Weights = parameters x bytes per parameter.
- Cache = 2 x layers x KV heads x head dimension x tokens x batch x bytes. Use KV heads for grouped-query attention.
- Lower precision shrinks weights directly. Mixture-of-experts models still need all experts' weights resident.
- Plan for peak: long contexts and many concurrent users are what exhaust memory.
Chapter quiz
Four questions. Score 70% or more to mark this chapter complete.