Best GPU for local LLMs: what fits in each VRAM tier

The best GPU for local LLMs is the one whose video memory (VRAM) fits the model you want to run, because a model that does not fit runs far slower or not at all. This guide does not rank cards and quotes no prices or benchmarks, which we cannot verify and which change weekly. Instead it answers the question that decides the purchase: which models fit in 8, 12, 16, 24 and 32 GB of VRAM, using our validated memory formulas, NVIDIA's published specifications and Ollama's hardware documentation, all checked on October 6, 2026.

How much VRAM do you need to run a local LLM?

A model needs memory for its weights, for the key-value cache that grows with the conversation, and for a margin for the engine. Our calculator, tested against real GGUF file sizes (within 3%), gives these totals. Each cell is the smallest VRAM tier that fits the model in 4-bit Q4_K_M quantization, at two context lengths. We treat 1 GB as 1,000,000,000 bytes, which is conservative.

Table: smallest VRAM tier that fits (our calculation, Q4_K_M)

Read it as a rule of thumb for each tier:

• 8 GB: 4B to 8B models at a short context. At 32K tokens even an 8B model needs about 10 GB.
• 12 GB: 8B and 12B models with room for long context, and a 14B model at a short one.
• 16 GB: a 14B model with a long context, and gpt-oss 20B, which stores its weights in native 4-bit form.
• 24 GB: 24B to 35B models, including the coding mixture-of-experts models, at moderate context.
• 32 GB: a 32B model with a 32K context, or a 24B model in 8-bit quantization.
• Above 32 GB: Llama 3.3 70B needs about 47 GB at a short context and gpt-oss 120B about 67 GB, which is where cloud APIs start to compete. See LLM API pricing.

Mixture-of-experts models such as Qwen3-Coder 30B-A3B load all their weights, so they need memory for the full 30 billion parameters even though only about 3 billion are active per token. The active count affects speed, not the memory you need. To test your own case, use the VRAM calculator.

How much VRAM does each NVIDIA card have?

NVIDIA's own comparison page lists these memory configurations (checked October 6, 2026):

The RTX 5090 has 32 GB, the 5080, 5070 Ti, 4080 and 4070 Ti SUPER have 16 GB, the 5070 and 4070 have 12 GB, and the 5060 Ti and 4060 Ti come in 16 GB or 8 GB versions, so check the exact variant before you buy. The 4090 has 24 GB.

Why memory bandwidth matters

When a model fits entirely in the GPU, each new token requires reading the model's weights once, so memory bandwidth sets a ceiling on speed: tokens per second can never exceed bandwidth divided by the bytes of weights. NVIDIA publishes the bandwidth for its 50 series. Applying that formula to Llama 3.1 8B in Q4_K_M, which has 4.91 GB of weights, gives these upper limits:

Table: theoretical speed ceilings for Llama 3.1 8B Q4_K_M (our calculation)

These are ceilings, not measurements: real speed is lower because of overhead and the first-token wait, and we have not benchmarked any card. They explain why two cards with the same VRAM can feel different, and why the 5060 Ti with 16 GB can run a bigger model than the 5070 but at a lower ceiling.

Which GPUs does Ollama support?

Ollama's documentation says it supports NVIDIA GPUs with compute capability 5.0 or higher and driver version 550 or newer; AMD GPUs through ROCm (version 7), including Radeon RX, Radeon PRO, Ryzen APU and Instinct families, on Linux and Windows; Apple devices through the Metal API (the Apple M series, on macOS Sonoma 14 or newer); and additional GPUs on Windows and Linux through Vulkan. Software support matters as much as VRAM: a card that Ollama or llama.cpp cannot use gives you nothing, so check the support list for your exact model. Our guide to running an LLM locally shows the install steps.

What if the model does not fit?

You have five options, in the order we would try them:

1. Use a smaller or more compressed model. Q3_K_M needs about 18% less memory for the weights than Q4_K_M (3.996 against 4.894 bits per weight in llama.cpp's table), at a quality cost that depends on the model.
2. Shorten the context. The key-value cache is a straight line in the number of tokens (see the context window guide).
3. Quantize the cache. Ollama's OLLAMA_KV_CACHE_TYPE accepts q8_0, which uses about half the memory of the default f16.
4. Accept offloading. Part of the model runs on the CPU from system RAM, which is slower. Run ollama ps and look for "100% GPU" to see whether a model fits completely.
5. Use an API instead. For a model above your VRAM tier, compare with the cost of an API.

Which VRAM tier should you aim for?

Without prices or benchmarks, the honest guidance is by the model you want: 8 GB for small assistants, 12 to 16 GB for the 8B to 14B class with long documents, 24 GB to try the 24B to 35B models that many people use for coding, 32 GB for a 32B model with room to spare. Whichever you choose, check the exact card variant, the memory bandwidth, whether your software supports it and the current retailer price, which this page does not give.