Evals. Cost caps. Fallback models. Grounding. Refusal criteria. Retry logic. The prompt is 10% of the work. This page is the other 90% — what we build around the model to make it survive real users on Monday morning.
| Task | First choice | Escalate to | Why |
|---|---|---|---|
| Classify a support ticket | gpt-4o-mini | gpt-4o (rare) | Cheap, fast, 95%+ accurate with a good prompt |
| Extract structured data from a PDF | gpt-4o-mini + Structured Outputs | gpt-4o on parse failure | Cheap does 90% of it |
| Answer from a knowledge base (RAG) | gpt-4o-mini | gpt-4o for multi-hop | Retrieval matters more than model size here |
| Write a customer-facing email | gpt-4o | — | Brand voice matters, don't cheap out |
| Reasoning / multi-step agent | o-series (or Claude Sonnet) | gpt-4o for cost | Depth of thought is what you're paying for |
| Embeddings | text-embedding-3-small | -large for high-precision RAG | Small is 5× cheaper, ~95% as good |
| Whisper transcription | whisper-1 | gpt-4o-transcribe for diarisation | Whisper is a bargain |
Every prompt change opens a PR that runs against our eval set. If accuracy drops on any category, CI fails. This is how we tell "the demo felt great" from "the app is actually better."
Hard per-user and per-endpoint token caps, cheap models for cheap tasks (routing 80% of calls to gpt-4o-mini or similar), aggressive prompt caching, and daily anomaly alerts. Predictable spend is a design decision, not a hope.
Prices per 1M tokens, input / output. Latency is p50 first-token for streaming completions of ~500 tokens. Numbers move — the shape stays the same.
Tell us your goals — we'll recommend the right technology honestly, even if it isn't this one.