How much VRAM do you need to run an LLM locally?

A language model running on your own computer needs memory for three things: its weights, the KV cache that holds the conversation so far, and a margin for the engine itself. The weights depend on the model's size and how heavily it is compressed (quantized). The KV cache depends on the model's architecture and on how much context you give it. That second part is the one most guides skip, and it is why a model that "fits" can still fail when you paste in a long document.

The calculator below uses the same formula we checked against 13 real model files (details in How we checked).

The formula in three lines

• Weights (bytes) = parameters × bits per weight ÷ 8. An 8-billion-parameter model at Q4_K_M (4.89 bits per weight, as measured by llama.cpp) needs about 5 GB.
• KV cache (bytes) = 2 × layers × KV heads × head dimension × bytes per value × tokens of context. The 2 is for keys and values. At the default 16-bit precision, each value takes 2 bytes.
• Margin: we add 1 GiB for compute buffers and the GPU driver. This is our assumption, not a measurement. The real figure depends on the engine, the batch size and the GPU.

The architecture numbers come from each model's config.json on Hugging Face (num_hidden_layers, num_key_value_heads, head_dim). The calculator has them built in for 12 popular open models; for any other model, choose "Other model" and copy them in.

How much memory popular models need

With an 8,192-token context, which is enough for a long chat or a few pages of text:

Three patterns stand out:

1. 8 GB cards are tight for 8B models. Qwen3 8B at Q4_K_M comes to about 7.3 GB with 8K of context, so it fits, but there is little room for a longer context or for anything else using the GPU.
2. 16 GB covers most mid-size models at Q4_K_M, including Qwen3 14B (11.5 GB) and OpenAI's gpt-oss-20b (15.1 GB). OpenAI's own model card says gpt-oss-20b runs "within 16GB of memory", which matches our estimate.
3. 70B models do not fit a single consumer card. Llama 3.3 70B needs about 43 GB just for its Q4_K_M weights. On a 24 GB or 32 GB GPU, part of the model has to run from system RAM, which is much slower.

Quantization: what Q4_K_M and Q8_0 mean

Quantization stores each weight with fewer bits. llama.cpp publishes the measured bits per weight for each format on Llama 3.1 8B:

The bits per weight are above the nominal 4, 5 or 8 because each block of weights also stores scaling factors, and the "K" mixes keep some layers at higher precision. llama.cpp notes that quantization "may introduce some accuracy loss", which it measures with perplexity and KL divergence. How much that matters depends on the model and the task, so if a model is borderline, it is worth trying the next format up before buying more memory.

Some models are published already quantized. OpenAI's gpt-oss models ship with their mixture-of-experts weights in MXFP4, so the calculator uses the real file size for them (13.8 GB for gpt-oss-20b, 65.4 GB for gpt-oss-120b) and ignores the quantization setting.

Context length is the hidden cost

The KV cache grows in a straight line with the context. For Llama 3.1 8B at 16-bit precision it takes exactly 1 GiB at 8,192 tokens, 4 GiB at 32,768 and 16 GiB at its full 131,072 tokens. At full context, the cache is three and a half times the size of the model's own Q4_K_M weights.

Engines pick a default for you. Ollama's documentation says it uses a 4K context below 24 GiB of VRAM, 32K between 24 and 48 GiB, and 256K above that, and recommends at least 64,000 tokens for agents, web search and coding tools. If you raise it, you can see whether the model still fits with ollama ps: the PROCESSOR column shows "100% GPU" when it does.

Two settings shrink the cache:

• Quantized KV cache. Ollama's OLLAMA_KV_CACHE_TYPE accepts f16 (the default), q8_0 (about half the memory, which Ollama says "usually has no noticeable impact" on quality) and q4_0 (about a quarter, with a loss that "may be more noticeable at higher context sizes"). It needs Flash Attention, which Ollama turns on automatically when the hardware supports it.
• A shorter context. If your prompts are short, 4K or 8K is enough, and the memory you save can go to a better quantization.

Why some new models need far less cache

Many recent models do not keep the whole context in every layer. Some layers use a sliding window and only look at the last few hundred or thousand tokens. Others use linear attention, whose memory does not grow with the context at all. Calculators that treat every layer as full attention overestimate these models badly:

These savings depend on the engine. llama.cpp keeps a reduced cache for sliding-window layers by default (its --swa-full option, which turns this off, defaults to false). If you use an engine that allocates the full cache for every layer, expect the larger figure.

Mixture-of-experts models: memory for all, speed of a few

Models such as Qwen3-Coder 30B-A3B or gpt-oss-20b are mixtures of experts. Only a few experts work on each token: 3.3 billion of Qwen3-Coder's 30.5 billion parameters, and 3.6 billion of gpt-oss-20b's 21 billion, according to their model cards. All the experts still have to be loaded, so memory follows the total size. Speed follows the active size, which is why these models feel much faster than a dense model of the same file size.

Which GPU: VRAM and bandwidth

NVIDIA's comparison table lists the memory of each GeForce card:

VRAM decides what fits. Memory bandwidth decides how fast it runs: to write each new token, the GPU reads every active weight once. That gives a hard ceiling of bandwidth ÷ weight size. For Qwen3 8B at Q4_K_M (5.0 GB), the ceiling is about 358 tokens per second on an RTX 5090 (1,792 GB/s) and about 89 on an RTX 5060 (448 GB/s). Real speeds are lower, because the GPU also reads the cache and does other work, but the ratio between cards holds. We have not measured real speeds ourselves: these are theoretical limits, not benchmarks.

On Apple Silicon Macs, the GPU uses the same unified memory as the rest of the system, so the relevant number is your Mac's total memory minus what macOS and your apps use. LM Studio recommends at least 16 GB on a Mac and says 8 GB machines should "stick to smaller models and modest context sizes". If you are choosing between apps, see Ollama vs LM Studio.

When the model does not fit

You have four options, roughly from best to worst:

1. Use a smaller quantization (Q4_K_M instead of Q6_K), or a smaller model from the same family.
2. Shorten the context or quantize the KV cache.
3. Split the model between GPU and CPU. Ollama does this automatically when a model does not fit in VRAM, and ollama ps shows the split. It works, but every layer on the CPU slows down generation.
4. Add a GPU. Ollama spreads a model across several GPUs when it does not fit on one, according to its FAQ.

For a step-by-step setup, see how to run an LLM locally.

How we checked

We compared the weights formula against the real size of 13 GGUF files that unsloth publishes on Hugging Face, from Qwen3 4B to Llama 3.3 70B, at Q4_K_M and Q8_0. Every estimate fell within 3% of the real file:

Gemma 3 and Mistral Small come out slightly high because their parameter count includes a vision encoder that ships in a separate file. The KV cache formula is exact arithmetic on the published architecture; the test suite behind this page checks it against known values (Llama 3.1 8B: 16 GiB at 131,072 tokens). What we have not measured is the runtime margin, which we set at 1 GiB, and real-world speed.