Ollama
One install, one pull, an OpenAI-compatible endpoint on :11434. The right default unless you have a reason not to.
GGUF · CPU + CUDA + Metal
Local inference, without the guesswork
Run a real language model on hardware you already own.
Open-weight models are good enough now that a single consumer GPU — sometimes just a laptop — will carry everyday work. The hard part is knowing what actually fits, which runtime to reach for, and where the speed goes.
Pick a runtime first
The runtime decides your quantisation formats, your hardware support, and how much of this you have to think about.
llama.cpp
What Ollama runs underneath. Reach for it directly when you want control over offload layers, KV cache type, or batch settings.
GGUF · everything
vLLM
Server-grade throughput with continuous batching and paged attention. Worth it once you are serving more than one person.
Safetensors · CUDA
LM Studio
A desktop app over the same GGUF ecosystem. Easiest way to try a few models before committing to a stack.
GGUF · desktop
Will it fit?
A workable estimate: weights ≈ parameters × bits-per-weight ÷ 8, then leave headroom for the KV cache and the context you actually intend to use.
| Model size | Q4 weights | Q8 weights | Comfortable on |
|---|---|---|---|
| 7–8B | ~4.5 GB | ~8 GB | 8 GB VRAM, or 16 GB unified memory |
| 13–14B | ~8 GB | ~14 GB | 12–16 GB VRAM |
| 30–34B | ~19 GB | ~34 GB | 24 GB VRAM |
| 70B | ~40 GB | ~70 GB | 2 × 24 GB, or 64 GB unified memory |
Quantisation below 4-bit saves memory but starts costing noticeably more quality. If a model only fits at Q2, the smaller model at Q4 is usually the better machine.
Sixty seconds to a local model
- 1
Install a runtime
On macOS or Linux,
curl -fsSL https://ollama.com/install.sh | shis the shortest path. - 2
Pull something small first
Start with an 8B model. Confirm the whole loop works before you spend an evening downloading 40 GB.
- 3
Watch where it runs
If the model spilled to CPU, tokens per second will fall off a cliff. Reduce context or step down a quantisation until it stays on the GPU.
- 4
Point your tools at it
Most clients accept an OpenAI-compatible base URL. Swap the endpoint and the API key stops leaving your machine.