Running Models Locally

Ollama: the quickest start

Install Ollama for your OS, then:

ollama pull llama3.1:8b          # or a DeepSeek / Qwen / Mistral tag from the Ollama library
ollama run llama3.1:8b           # interactive chat in the terminal
ollama list                      # downloaded models

Ollama runs a local server on port 11434 with an HTTP API:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "Explain Python decorators briefly."}],
  "stream": false
}'

Check the Ollama model library for current model names and sizes.

llama.cpp and GGUF

llama.cpp runs quantised models efficiently on CPUs, Apple Silicon and GPUs. Models are distributed as GGUF files with quantisation levels such as Q4_K_M (good balance) or Q8_0 (higher quality, larger). LM Studio provides a desktop interface over the same idea.

Hardware tips

• Apple Silicon Macs use unified memory, so a 32 GB Mac can run models that would need a large GPU on a PC.
• NVIDIA GPUs: VRAM is the limit; the whole model should fit in VRAM for good speed.
• CPU only: works for small models, slowly.
• Smaller, well-tuned models often beat larger ones on narrow tasks; test with your own prompts.

Quantisation trade-offs

Lower-bit quantisation reduces memory and increases speed, with some quality loss that is usually small at 4–5 bits for chat but more noticeable for maths, code and long reasoning. Compare quantisations on your own test set.

Privacy

Local models keep prompts on your machine, which is useful for sensitive documents. Downloaded models are code-adjacent artefacts: get them from official or reputable sources, and prefer the safetensors/GGUF formats over arbitrary pickle files.

Exercise

Install Ollama, run two different small models, ask both the same five questions from your own work, and record speed (tokens per second) and quality. Call one through its HTTP API from Python.

Check yourself

• What does Ollama provide on port 11434?
• What is GGUF?
• Why is VRAM the key constraint on NVIDIA GPUs?