Local models: a practitioner's field guide.

When local actually makes sense

Most practitioners are better served by a frontier API for the hard work and a local model for the privacy-sensitive slice. That's the honest version of this guide compressed into a sentence. The rest of it is the texture around that claim, because the cases where local genuinely earns its keep are real, and the cases where it doesn't are easy to talk yourself into anyway.

The cases where running locally is the right answer:

• Data residency and privacy. The prompt and the response never leave the machine. For draft work involving anything regulated (health, legal, internal financial, unreleased product details), this is the strongest argument and often the only one that matters.
• Offline capability. On a plane, on bad hotel wifi, in a basement with no signal, a local model keeps working. For travel and field work, this is genuinely valuable.
• No per-token cost. Once the hardware exists, the marginal cost of a generation is electricity. For batch jobs that would burn through API credits (summarizing thousands of documents, classifying a corpus, embedding a backlog), this changes the economics.
• No rate limits. You hit your hardware ceiling, not someone else's pricing tier.
• Learning. Understanding what a model is, what tokens look like coming out, what "context window" means when the machine you can see is the one running out of memory: there is no faster education than running an LLM locally.

The cases where it isn't:

• You need the best answer. Frontier closed models (Sonnet 4.5, GPT-5, Gemini 3 Pro) are still meaningfully better than the best open-weights model that fits on consumer hardware. The gap has narrowed, not closed.
• You need speed. A 70B model on an M3 Max generates around 8 tokens per second. The same prompt against a hosted API streams at 60 to 120 tokens per second. For interactive use, the cloud feels like a different product.
• You want breadth. Hosted services give you image, audio, video, tools, web search, and code execution behind one API. Local gets you text generation and (with effort) image input. Multimodal beyond that is possible but rough.
• You don't want to operate it. Local models are software you run. They need updates, disk space, occasional troubleshooting, and a passing understanding of what's happening when something goes wrong.

The runtimes, ranked by approachability

"Running a model locally" is shorthand for "using a runtime that loads a model file and exposes a way to talk to it." There are five worth knowing about. They're built on overlapping foundations (most of the ergonomic ones wrap llama.cpp), so picking one is a question of how much you want to manage versus how much control you want.

Two more runtimes that matter for specific situations:

A short Ollama session, end to end:

# pull and run a model
ollama run llama3.3

# pull without running, then list
ollama pull qwen3:30b
ollama list

# the server is already running on 11434
curl http://localhost:11434/api/tags

# stop a running model to free memory
ollama stop llama3.3

Quantization, in plain English

Every open-weights model is published as a pile of numbers (the weights). The original numbers are 16-bit or 32-bit floats. That's a lot of bytes per number and a lot of memory to load. Quantization is the trick of using fewer bits per weight, accepting a little quality loss for a lot of memory savings.

The naming convention you'll see most often comes from llama.cpp's GGUF format: Q4_K_M, Q5_K_S, Q8_0, and so on. The leading Q<n> is the bit width. The suffix is which "mixture" was used (which weights got more bits, which got fewer).

The practical translation:

For most practitioners, the rule is: start with Q4_K_M, move to Q5_K_M if quality feels thin, move to Q8_0 if you have memory to spare. Below Q4 the quality drop is real; above Q5 the memory cost rarely pays for itself.

One more piece of vocabulary. K-quants (the ones with _K_ in the name) are llama.cpp's smarter quantization that gives more bits to the weights that need them. Legacy quants like Q4_0 or Q4_1 still exist but are strictly worse than the K-quant of the same bit width. If you have a choice, take the K version.

Hardware reality

The single most important number for running local models is how much fast memory you have. Not disk, not CPU cores: the memory the GPU (or Apple's unified memory) can read at speed. That number sets the ceiling on what you can run.

Apple Silicon: the dark-horse story

Apple's M-series chips share memory between CPU and GPU. The whole 64 or 128 gigabytes is GPU memory if it needs to be. On every other platform, GPU memory is a separate, much smaller pool, and getting more of it is expensive. This is the architectural detail that quietly turned MacBooks into the most cost-effective local-LLM machines on the market.

An M3 Max 128GB MacBook Pro at around $4,500 will run a 70B model at Q4 entirely in memory and generate at about 8 to 10 tokens per second. The Nvidia rig that runs the same model at the same quality starts at two RTX 4090s ($3,200) plus a workstation to host them, and the unified-memory advantage stops mattering when you go bigger only if you need batch throughput, which most local users don't.

Nvidia: still the answer for the upper end

If you're going past 70B, or if you need the throughput of batched inference, Nvidia is still where the headroom is. CUDA is the most mature backend, every runtime supports it, and the 24GB and 48GB cards are the standard.

AMD and CPU-only

ROCm has gotten substantially better. As of mid-2025, llama.cpp on a Radeon RX 7900 XTX is competitive with a 4080 on most models, but the driver story is still bumpier than CUDA, and not every runtime supports it equally well. CPU-only inference works for models up to 8B if your RAM is fast (DDR5 helps a lot), but the tokens-per-second numbers are slow enough that it's better thought of as "batch tool" than "interactive."

Rough numbers as of May 2026, on Q4 quantizations, single-user, generating short replies:

These are rough. Actual numbers depend on the specific model, the exact quantization, your context length, and what else the machine is doing. Treat the table as orders of magnitude, not benchmarks.

The model landscape

Open-weights releases come constantly, and the leaderboard reshuffles every few months. As of May 2026, these are the families that actually matter for local use:

• Llama 3.3 (8B, 70B). Meta's open-weights workhorse. The 8B is the best small model for general chat; the 70B is the best general-purpose model that fits on serious consumer hardware. The 405B exists but is impractical to run locally for almost everyone. If you're not sure what to pull, pull Llama 3.3.
• Qwen 3 (multiple sizes up to 235B). Alibaba's family. Particularly strong at code and multilingual tasks; the 30B and 72B variants are excellent and frequently outperform Llama 3.3 70B on benchmarks. The MoE variants give better tokens-per-second at the cost of more memory. My current daily-driver for code completion locally.
• Mistral and Mixtral (Small 3, Large 2). The European answer. Mistral Small 3 (24B) is a quietly excellent generalist that punches above its weight class; Mixtral 8x22B and Large 2 are bigger but harder to fit. See the dedicated Mistral guide for more on the family.
• DeepSeek V3 / R1. Mixture-of-experts at huge scale (670B+ total parameters, ~37B active). Quality is excellent but local hosting requires serious hardware: 4× A100 territory, or M3/M4 Ultra with 192GB+ for a quantized fit. For most local users, this is "interesting on paper."
• Phi-4 (14B) from Microsoft. Small, focused, unusually capable for its size. The benchmark scores look implausible until you actually use it; it's not as good as Llama 3.3 70B but it's surprisingly close at a fraction of the resource cost. The dark-horse pick if you want something smarter than 8B but lighter than 30B.
• Gemma 3 (1B, 4B, 12B, 27B) from Google. The open-weights side of Google's lineup. Multimodal at 4B and above (image input). The 27B variant is competitive with Llama 3.3 70B on many tasks at much lower memory cost. The smaller ones are the right pick for embedding-class hardware (single-board computers, Pi 5, edge devices).

Two patterns to internalize. First, model size is not the whole story. A well-distilled 30B model will beat a poorly-trained 70B on the tasks it was trained for. Second, "better on benchmarks" doesn't always mean "better in your hands". Pull two candidates, run them on the actual prompts you care about, pick the one whose outputs you prefer.

Practical workflows

The workflows below are the ones that have justified the hardware for me, more than the abstract argument about privacy. If your work doesn't touch anything in this list, the case for local is weaker than you think.

The OpenAI-compatible server pattern

Every serious local runtime exposes an HTTP server that speaks the OpenAI Chat Completions API. This is, quietly, the most important fact in the local-LLM ecosystem. It means any tool that knows how to talk to OpenAI also knows how to talk to your laptop.

Ollama is the friendliest version. It runs on port 11434 the moment you install it; no flags, no config:

# point any OpenAI client at it
export OPENAI_API_BASE=http://localhost:11434/v1
export OPENAI_API_KEY=ollama  # any string; auth is not enforced

# then call it like normal
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.3",
    "messages": [{"role": "user", "content": "say hello"}]
  }'

llama.cpp's llama-server binary does the same, with more flags:

llama-server -m ./models/llama-3.3-70b-q4.gguf \
  --port 8080 \
  --ctx-size 8192 \
  --n-gpu-layers 99

LM Studio exposes a server toggle in its UI that does the same on a port you choose. vLLM ships vllm serve with the same surface for GPU rigs.

Why this matters: it makes local and cloud swappable. The same client code, the same prompt format, the same tool integrations. You can develop against a cheap hosted model for fast feedback, deploy against a local model for a privacy-sensitive customer, fall back to local when the network is down, A/B between them. The pattern that took years for cloud APIs to standardize on (one HTTP shape, swappable backends) is the default for local.

Limits and pitfalls

The places local will disappoint you, in roughly the order you'll meet them.

When to choose local, when to reach for cloud

The cleanest decision frame I've found, in order:

Pick the tool by the constraint that matters most: privacy, capability, cost, or convenience. One of those is doing the work in any given decision. — TWD

Concretely:

• Privacy is the constraint: local. Anything regulated, anything pre-disclosure, anything you'd rather not see in someone else's logs.
• Capability is the constraint: cloud. If the task is at the edge of what current models can do, run it on the model that has the best chance of finishing.
• Cost is the constraint: local for high-volume batch jobs (embeddings, classification, summarization at scale), cloud for low-volume hard problems where the per-call price doesn't matter.
• Convenience is the constraint: cloud, almost always. The hosted experience is friction-free in a way local has not yet matched and may never match.

The combination most practitioners settle into looks like this. A frontier hosted model (Claude, ChatGPT, Gemini, pick your favorite) for hard problems, novel work, anything multimodal, anything where the answer needs to be as good as possible. A local Ollama install for the privacy-sensitive slice, for offline use, and for the batch jobs that would otherwise burn API credit. The two coexist, and the ergonomics of the OpenAI-compatible server pattern mean swapping between them is a config change.

One closing observation

The argument for running models locally has gotten stronger in the last two years, not weaker. Open-weights releases have closed most of the quality gap on most everyday tasks. Apple Silicon turned a niche hobby into something a single laptop can do well. Tooling has converged on the OpenAI HTTP shape, which makes everything portable. None of that means local is the right answer for every workflow, or even for most of yours. It means local is now a real option for the workflows where it fits, instead of a curiosity. Use it where it fits, ignore it where it doesn't, and don't romanticize either side.

If any of this is out of date by the time you read it: the Ollama blog, the LM Studio release notes, and the Hugging Face open-llm-leaderboard are the three places I check. They lag the actual releases by a few days, but the signal-to-noise on each of them is high.