AI Expert

Guides/Latency & streaming UX

Latency budget & streaming UX

Users judge stalls by time-to-first-token (TTFT) and cadence of streamed output — not only total latency. Pair backend optimizations with interaction design so perceived speed tracks real improvements.

Lifecycle timeline (conceptual)

Annotate each segment with your telemetry ranges — the diagram convinces stakeholders when backed by p50/p95 numbers.

Timeline from user send to first token and streamed completionSendQueue + promptassemblycache fill, routingTTFTperceived stallToken streamsustained tpsDoneUX focuses on shrinking TTFT + keeping stream smooth — not only total wall time.
Annotate with your real p50/p95 TTFT from logs — this diagram earns credibility when labeled.

UX tactics vs perceived lift (illustrative)

Validate with qualitative sessions + lightweight surveys — indices here are directional for roadmap prioritization conversations.

Relative lift on perceived responsiveness (survey proxy, illustrative)

Pair UX tactics with backend moves (smaller prompts, faster routing SKU) for end-to-end wins.

Playbook snippets

Streaming beats blocking endpoints for perceived responsiveness even when total time is similar — users tolerate waits better when partial text arrives early.

Skeleton layouts + staged disclosure anchor attention during unavoidable TTFT waits — combine with honest progress hints where backend exposes routing state.

Optimistic UI helps only when rollback paths are trustworthy — pair with undo flows for destructive actions.