Choosing and Evaluating Models

Benchmarks are a starting point

Public leaderboards measure general skills, but your task is specific. Benchmark contamination (test questions leaking into training data) and prompt-format differences also skew results. Use them to shortlist, then test yourself.

Build your own evaluation

1. Collect 30–100 real examples of your task with expected outputs or grading criteria.
2. Run each candidate model (and quantisation) with the same prompts.
3. Grade with code checks where possible (exact match, valid JSON, tests passing) and a rubric for open-ended answers.
4. Record quality, latency and cost per task.

Total cost of ownership

Compare honestly:

Hosted API:  requests/month × average tokens × price per token
Self-hosted: GPU hours × hourly price (including idle time)
             + engineering time to deploy, monitor and update
             + storage, bandwidth and redundancy

Self-hosting wins at high, steady volume or when data rules require it; hosted APIs often win for variable traffic and small teams.

Hybrid patterns

• Route simple, high-volume tasks to a small local model and hard tasks to a frontier API.
• Use an open model for sensitive data that must stay on-premises, and a hosted model elsewhere.
• Wrap all providers behind one internal interface so you can switch based on your evaluations.

Exercise

Compare one open model (two quantisations) against one hosted API model on 30 examples of a real task. Produce a table of accuracy, average latency and estimated monthly cost at your expected volume, and write a recommendation.

Check yourself

• Why are public benchmarks not enough?
• What costs does self-hosting add beyond GPU time?
• Describe one hybrid routing pattern.