Cost, Caching, Safety and Production

Understand the bill

You pay per input token and output token, at prices that differ by model. Cost per feature = (requests) × (average tokens in × input price + tokens out × output price). Log response.usage for every call so you know real numbers per feature and per user.

Prompt caching

Long, repeated prefixes (a big system prompt, documentation, few-shot examples) can be cached, making repeat reads much cheaper and faster.

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=16000,
    cache_control={"type": "ephemeral"},    # cache the reusable prefix automatically
    system=LONG_SYSTEM_PROMPT_WITH_DOCS,
    messages=[{"role": "user", "content": question}],
)
print(response.usage.cache_read_input_tokens)   # > 0 on cache hits

Caching is a prefix match: keep stable content first and put anything that changes (timestamps, user IDs, the question) after it. A datetime.now() inside the system prompt silently breaks the cache on every call.

Other cost levers, in order

1. Cache stable prefixes.
2. Trim inputs: send only the relevant documents, summarise long histories.
3. Set sensible max_tokens and ask for concise output where appropriate.
4. Lower effort for simple routes; keep higher effort where quality needs it.
5. Use Batches for non-urgent bulk work (half price).
6. Use a smaller model for simple, high-volume tasks, after testing that quality holds.

Measure cost per completed task: a cheap request that needs retries is not cheap.

Refusals and fallbacks

Models may decline some requests (stop_reason == "refusal", with details in stop_details). Handle it explicitly with a friendly message rather than showing empty output. Anthropic's API can also re-run a declined request on another model automatically:

response = client.beta.messages.create(
    model="claude-opus-5-5",
    max_tokens=16000,
    betas=["server-side-fallback-2026-07-01"],
    fallbacks="default",
    messages=[{"role": "user", "content": question}],
)
if response.stop_reason == "refusal":
    show("Sorry, I can’t help with that one.")

Safety and privacy

• Tell users when they are talking to AI, and do not present generated content as human-verified.
• Do not send secrets or unnecessary personal data to the model; check your provider's data retention terms and mention AI processing in your privacy policy.
• Moderate user-facing generation where your audience requires it, and keep humans in the loop for consequential decisions.
• Protect against prompt injection in tool-using features: least-privilege tools, confirmation for actions, untrusted content clearly marked as data.

Supporting multiple providers

Wrap model calls in your own small interface (generate(prompt, schema=None)), with one implementation per provider using its official SDK. Your app code then never depends on a single vendor, and you can compare providers on your own test set.

Production checklist

• [ ] API keys in environment variables; separate keys per environment
• [ ] Auth, per-user rate limits and input size limits on every AI endpoint
• [ ] Specific error handling; stop_reason checked on every response
• [ ] Streaming for long outputs; worker timeouts above the longest generation
• [ ] Usage logged per feature and per user; budget alerts at the provider
• [ ] Prompts and model IDs in configuration and versioned in Git
• [ ] A test set to check quality before changing models, prompts or effort

Final project

Build "Ask PandaDev": a streaming learning assistant in Flask that answers questions about your learning modules using two tools (search_modules, get_track_progress), extracts a structured "next steps" plan with Pydantic, caches its system prompt, logs usage per user, rate-limits requests and handles refusals gracefully. Measure cost per conversation on 20 test questions.

Check yourself

• Why does a timestamp in the system prompt break caching?
• In what order should you apply cost levers?
• Why wrap provider calls behind your own interface?