The Open-Weight Model Landscape
Alongside hosted APIs from providers like Anthropic and OpenAI, many organisations release open-weight models: the trained weights are downloadable, so you can run them on your own hardware or a cloud server. Families you will see include DeepSeek, Llama, Qwen, Mistral and Gemma, with new releases appearing frequently.
Open weights vs hosted APIs
| Hosted API | Open-weight, self-hosted | |
|---|---|---|
| Setup | an API key | hardware, serving software, operations |
| Capability | typically the most capable frontier models | strong, improving; varies by size |
| Data control | data sent to the provider (check their terms) | data stays on your infrastructure |
| Cost | pay per token | pay for hardware/GPU time, regardless of usage |
| Customisation | prompting, some fine-tuning | full fine-tuning and modification |
| Operations | provider handles scaling and updates | you do |
Choose self-hosting for strict data residency, offline use, very high steady volume, research or deep customisation. Choose hosted APIs for the best capability, fastest development and spiky traffic.
Licences matter
"Open" ranges from permissive licences (Apache 2.0, MIT) to custom licences with usage restrictions (for example limits on very large companies or certain uses). Read the model card and licence before using a model commercially.
Model sizes and what they need
Models are described by parameter count (for example 7B, 70B). Rough memory needs for inference:
| Precision | Bytes per parameter | 7B model | 70B model |
|---|---|---|---|
| 16-bit | 2 | ~14 GB | ~140 GB |
| 8-bit | 1 | ~7 GB | ~70 GB |
| 4-bit | 0.5 | ~3.5–4 GB | ~35–40 GB |
Plus extra memory for the context (the KV cache). Mixture-of-experts models (some DeepSeek and Mixtral variants) have many total parameters but activate only some per token, which affects speed more than memory.
Model cards
Read them for: intended use, training data notes, context length, chat template, benchmark results (treat with healthy scepticism) and known limitations.
Check yourself
- Name two reasons to self-host a model.
- Roughly how much memory does a 7B model need at 4-bit precision?
- Why must you read a model's licence?
I tuoi progressi sono salvati solo in questo browser.
Invia per la revisione (10 punti)