DeepSeek & Open-Source LLMs

Serving Models in Production

35 min

Serving software

Tool Strength
vLLM high-throughput GPU serving with efficient batching and memory management
Text Generation Inference (TGI) production serving from Hugging Face
llama.cpp server / Ollama simple, CPU and small-GPU friendly
SGLang fast serving with structured generation features

Many of these expose an OpenAI-compatible HTTP API, so existing client code can point at your server by changing the base URL.

# vLLM example (GPU machine)
pip install vllm
vllm serve <hugging-face-model-id> --max-model-len 16384

Where to run GPUs

  • Cloud GPU instances (AWS, GCP, Azure) or GPU-focused clouds (RunPod, Lambda, Modal, and similar).
  • Serverless GPU platforms that start containers on demand (cold starts can take tens of seconds for large models).
  • Managed inference providers that host open models behind an API, combining open weights with hosted convenience.

Throughput and latency

  • Batching many requests together is what makes GPU serving economical; vLLM does this continuously.
  • Measure time-to-first-token and tokens per second under realistic concurrency.
  • Long contexts use more memory (KV cache), reducing how many requests fit at once; cap max-model-len to what you need.

Security and operations

  • Never expose a model server directly to the internet without authentication and rate limits; put your own API in front.
  • Pin model versions and record which version produced which outputs.
  • Monitor GPU memory, utilisation, queue length and error rates.
  • Budget for idle time: a GPU costs money whether or not anyone uses it.

Exercise

Serve a small open model with an OpenAI-compatible server (vLLM on a cloud GPU, or Ollama locally), put a Flask endpoint with authentication and rate limiting in front, and load-test it with 10 concurrent users, recording latency and throughput.

Check yourself

  • Why does batching matter for GPU serving costs?
  • What does "OpenAI-compatible endpoint" mean for your client code?
  • Why put your own API in front of a model server?

Your progress is saved in this browser only.

Submit for review (10 points)