A 24 GB GPU does not give a local model 24 GB to use. The display stack, runtime, temporary buffers, graph captures, allocator fragmentation, model weights, and KV cache all compete for the same capacity. If your deployment plan begins and ends with the number printed on the box, it is not a plan yet.
For a conservative first pass, reserve 10% of advertised memory. That turns a 24 GB label into 21.6 GiB of planning capacity. It is not a universal law—your measured runtime may support a different number—but it forces the hidden margin into the worksheet before you buy hardware or promise a context window.
The other common mistake is treating quantized model weights as total inference memory. Exactly four bits per parameter gives a useful arithmetic floor:
Dense parameter count | Exact INT4 weight floor | What the floor excludes |
|---|---|---|
7B | 3.26 GiB | Checkpoint overhead, KV cache, runtime, headroom |
13B | 6.05 GiB | Checkpoint overhead, KV cache, runtime, headroom |
32B | 14.90 GiB | Checkpoint overhead, KV cache, runtime, headroom |
70B | 32.60 GiB | Checkpoint overhead, KV cache, runtime, headroom |
That table answers one narrow question: the dense weight equivalent at exactly four bits. It does not prove that a packaged checkpoint will load, that your requested context will fit, or that the result will be fast.
