Guides/Model selection
Model selection attributes
Product-facing AI is mostly choosing under constraints: user latency expectations, budget, how much text fits in one request, and whether you need tools or vision. Provider catalogs throw dozens of knobs at you — here's what actually maps to decisions.
Why these attributes matter
Every attribute below shows up on invoices, SLAs, or UX research. None of them alone tells you "the best model" — but together they rule out bad fits fast (e.g., huge context with no budget for output tokens, or ultra-low latency with complex reasoning).
Think in archetypes (fast edge class vs balanced flagship vs frontier reasoning) rather than chasing weekly leaderboard rankings — SKUs and prices change; the tradeoffs stay familiar.
Context window
The context window is how many tokens can participate in one forward pass: your prompt, retrieved documents, tool outputs, and the answer so far. If the combined stack doesn't fit, you truncate, summarize, or switch to workflows that chunk work across calls.
Long-context SKUs help when you truly need everything in one shot (large specs, depositions, multi-file repo context). For steady chat, practical limits are often workflow design — memory policies and retrieval — not raw token count alone.
Latency vs throughput
Latency is usually experienced as time to first token (TTFT) for streaming UIs — users notice stalls before they evaluate answer quality.
Throughput is sustained tokens per second once generation is underway. It matters for long completions, batch jobs, and cost-controlled pipelines where wall-clock time dominates.
Interactive assistants optimize TTFT; offline summarization may care more about total time at high throughput and stable batch pricing.
Cost: input, output, cache, and extras
Most catalogs quote price per million tokens separately for input and output. Output is often several times input — agent loops that repeatedly emit long completions add up fast.
Prompt caching can bill cheaper "cache read" vs full input when large prefixes repeat (system prompts, RAG corpora). Evaluate whether your workload repeats stable prefixes enough to benefit.
Line items like web search or hosted tools may be priced per call or per 1k operations — treat them as separate capacity planning from raw tokens.
Illustrative archetypes (not live prices)
The table below uses invented bands so you can read provider dashboards — numbers shift by vendor and date. Always verify current SKUs before forecasting budget.
| Archetype | Context | Latency (indicative) | Throughput | Input ($/M tok) | Output ($/M tok) | Typical fit |
|---|---|---|---|---|---|---|
| Edge / fast class | ~128K–200K | ~0.4–0.9s TTFT | ~80–120 tps | $0.15–$1 | $0.60–$4 | Low-latency chat, classification, routing |
| Balanced flagship | ~200K | ~0.6–1.2s TTFT | ~45–95 tps | $3–$8 | $15–$25 | General assistants, drafting, multi-step tools |
| Frontier reasoning | ~200K–1M | ~0.8–2s TTFT | ~40–95 tps | $5–$15 | $25–$75 | Hard analysis, long chains, vision + tools |
Capabilities and modalities
Check whether you need vision, structured outputs (JSON), tool/function calling, or particular languages. A cheaper text-only path fails if your product screenshot flows require multimodal understanding.
Capability tiers often correlate with price and latency — align SKU choice to the smallest modal surface that actually ships.
Trust, compliance, and data handling
Procurement asks about zero data retention (ZDR), training opt-outs, regions, and subprocessors — attributes that rarely appear on latency charts but gate deployment for regulated teams.
Treat compliance labels like any other requirement: if you need a specific regime, narrow the provider shortlist before optimizing tokens per dollar.
Scenario priorities
Rank what matters for this product moment — then match to an archetype. "High" means that column dominates SKU discussion for that row.
| Scenario | Prioritize latency | Prioritize cost | Prioritize reasoning | Prioritize context |
|---|---|---|---|---|
| Streaming chat assistant | High | Med | Med | Low |
| Batch summarization / ETL | Low | High | Med | High |
| Agent with tools (many steps) | Med | Med | High | High |
| Regulated workload (procurement) | Med | Med | Med | Med |
Latency vs capability (mental map)
Real APIs don't sit in neat quadrants — but teams argue less when they agree where they're trying to land on this map before debating vendor names.
Putting it together
- Start from workflow + SLA (streaming UX vs batch; fits-in-context vs must retrieve).
- Estimate input/output mix — agents and long answers skew cost toward output tokens.
- Apply compliance filters before optimizing price — otherwise you benchmark models you can't ship.
- Pick an archetype, run a short bake-off on real prompts, then revisit — catalogs churn; your scenarios shouldn't.