AI Expert

Guides/Caching & request shaping

Caching & request shaping for COGS

Pricing fights start after launch. The fastest margin wins usually combine hit-rate instrumentation, cache keys that respect privacy, and smaller prompts before swapping model tiers — see also pricing & unit economics and model selection.

Stack mental model

Layers from client session cache down to model invocationLatency & COGS shortcuts (hit closest first)Session / UI memo — identical follow-up tapsInstant replay without network — invalidate on task changeSemantic / embedding cache — near-duplicate queriesWatch staleness — TTL + content-version keysPrompt prefix / provider cache— repeated system + docsBiggest bill wins when doc bundlesrepeat across usersRequest shaping — batch, trimcontext, route to smaller modelCheaper than caching wrong answers —measure before stacking cachesMisses fall through to live generation —observability tags should label cache tier
Instrument hit rate by tier; a blind cache hides regressions until finance notices COGS.

Tactics & failure modes

TacticBest COGS / latency winsWhat breaks trust
Semantic cacheSupport bots, repeated policy questions, stable corporaWrong answer persistence — pair TTL with eval alerts
Prompt / prefix cacheShared system prompts + large doc prefixes across tenants (where allowed)Privacy boundaries — never cross-tenant bleed keys
Deduped fan-outBurst traffic to same answer — collapse in-flight requestsThundering herd on cold miss — add jitter and backoff
Batch & offline queuesSummaries, indexing, low-interactive workloadsUser expectation mismatch if UI promises realtime

Latency vs spend trajectory

Relative TTFT & token spend index by path (illustrative)

Indices are directional — measure your own P95 with vendor dashboards and tracing.

Session replay skips generation entirely — ensure UX communicates freshness when it matters.