Local Models & Hardware

Why run a model on your own computer

A local model runs on your own hardware instead of a vendor's servers. Your prompts and files stay on your machine, there is no price per token, and it works without an internet connection once the model is downloaded. The trade-off is hardware: a model has to fit in memory, and the memory that matters most is the video memory (VRAM) of your graphics card, or the unified memory of an Apple Silicon Mac. The guides in this section take you from "will this model fit?" to a working setup, and they show their arithmetic so you can check it. Where we quote a number, it comes from the model's published files, NVIDIA's or Ollama's own documentation, or a calculation we explain.

Where to start

If you have never run a model locally, follow the guides in this order:

1. How much VRAM do you need to run an LLM? is the best first stop. It explains the formula (weights plus the key-value cache plus a margin) and has a calculator: about 5 GB of weights for an 8B model at Q4_K_M, 20 GB for a 32B model, and 1 GiB of cache per 8,192 tokens of context on Llama 3.1 8B.
2. How to run an LLM locally installs Ollama on Windows, macOS or Linux, downloads a model that fits your memory, starts a chat and shows how to check with ollama ps that the model really runs on the GPU.
3. Ollama commands is the cheat sheet you keep open afterwards: run, pull, ls, ps, stop and rm, where the models are stored and which environment variables change the behavior.
4. Best GPU for local LLMs is for the moment you decide to buy hardware. It maps VRAM tiers to the models they fit, explains why memory bandwidth sets a speed ceiling, and quotes no prices or benchmarks, because we have not measured them.
5. Ollama vs LM Studio compares the two most popular apps by license, features and API, so you can choose between a command line and a desktop app.

• Ollama not using your GPU? is the fix-it guide for the most common problem: how to read ollama ps, why a model splits between CPU and GPU, driver and AMD requirements, Docker, and the settings that force the CPU.
• Ollama vs llama.cpp explains how the two relate (Ollama lists llama.cpp as its backend), the defaults behind most speed claims (context length, parallel requests, GPU layers), the API and ports of each, and when vLLM is the better choice.
• NPU vs GPU explains what the NPU in a Copilot+ PC can run, why Ollama and LM Studio still use the GPU, and the vendor tools (AMD's NPU software for Ryzen, Intel OpenVINO, Windows ML) that do reach the NPU, with the limits they document.
• Transcribe audio to text locally sets up Whisper with faster-whisper and reports our own CPU test: a minute of English in about 11.5 seconds with no word errors, and Spanish dialogue at a 3.2% word error rate.
• Run an LLM on a Mac: Ollama, LM Studio and MLX shows how much unified memory each model size needs and how Ollama, LM Studio and MLX use Apple silicon.
• Open WebUI: How to Install It with Docker and Ollama sets up a private chat interface for local models with Docker, covers the admin account, hardening and the license.

The rule that decides everything

A model that fits entirely in the GPU runs fast. A model that does not fit is split between the GPU and the system memory, and every layer that runs on the processor slows generation down. So the question to answer before you download anything is how much memory your chosen model and context length need. Two settings change the answer more than people expect. The first is quantization, which stores the weights with fewer bits and usually decides whether a model fits at all; our guide on what quantization is explains Q4_K_M and Q8_0 with measured data. The second is the context length, because the cache grows in a straight line with it: what a context window is shows why a longer context costs memory you may not have.

What a local model is good for, and what it is not

A local model suits private documents, offline work, experiments, scripts that call a model thousands of times, and learning how models behave. It does not replace a frontier assistant for every task: the largest models need far more memory than any single consumer card offers, and a small model on your laptop will not match them. We do not rank local models by quality, because we have not run a hands-on test; we tell you what fits and how to measure the result on your own work. If you only need an occasional answer, a subscription is simpler, and our ChatGPT alternatives guide includes a local model as one of seven options. If you are building software, compare the running cost of your own hardware with the price per token of the main APIs.

Safety and privacy

Running locally keeps your data on your machine, but it does not make a model correct. Local models can invent facts just like hosted ones, so read why model hallucinations happen before you rely on an answer. Also remember that a local API port is reachable by other programs on the same computer: keep the default bind address unless you know why you are changing it.

What to read next

If you want a local model to take actions, such as searching files or calling tools, read what an agent is and what MCP is. If you want it to answer from your own documents, start with what RAG is.