AI Expert

Guides/Observability

Observability for shipped AI

Models feel like black boxes until you instrument them like any other service: traces that reconstruct a single user turn, metrics that fire before users rage-quit, and eval hooks that connect regressions to releases.

Incident loop

AI incidents stall when teams lack replay — negotiate observability when you negotiate SLAs, not after launch.

Observe triage reproduce fix deploy loop for AI featuresObservemetrics + tracesTriageseverity + blastReproducereplay requestFix + shipprompt/tool/modelDocumentAlertTie each step to owners —AI incidents stall without replay data.
Ask engineering for request IDs, trace IDs, and frozen prompts whenever CS escalates a bad answer.

Signals teams actually use (illustrative)

Replace indices with your stack's reality — the ranking matters more than absolute numbers.

Signal usefulness for triage (self-reported index)

Combine cheap automated signals with slower eval drift detectors — don't wait for user outrage only.

Trace anatomy

Demand consistent span names across services — otherwise dashboards compare apples to oranges across environments.

Example trace spans for one assistant turnRequest timeline (one user message)Auth + policyRetrievalLLMTool: CRMFormatCorrelate slow TTFT with retrieval vs model spans —don't blame the LLM by default.
Ask for span attributes (model SKU, chunk IDs, tool args digest) in PRDs so incidents become searchable.

Questions for kickoff

  • What identifiers tie analytics events to traces and to model provider invoices?
  • Which spans must exist on every request vs optional when tools fire?
  • How do we detect embedding or retrieval regressions independently from LLM upgrades?