Guides/Experimentation & rollout

Experimentation & rollout

Model and prompt changes behave like releases — you need progressive exposure, guardrail metrics, and a kill switch more than a single launch-day blog post. This guide frames the ladder PMs negotiate with Eng.

Rollout ladder

Names vary by company — align vocabulary with your feature-flag tool and incident response runbooks.

Skip stages only when blast radius is tiny — AI regressions show up in cohorts, not localhost.

Signals & exit criteria

Stage	Watch signals	Typical exit criteria
Shadow	Offline parity, golden-set diff vs prod baseline	No severe regressions on safety + quality suites
Canary	Latency p95, error budget, CS ticket themes	Stable week over week vs control — define numeric gates
A/B	North-star + guardrail metrics + human spot checks	Stat sig lift or strategic bet documented + exec sign-off

Learning vs risk

Indices are directional — replace with your governance thresholds before exec reviews.

Learning power vs exposure risk (schematic)

Shadow learns slowly but safely; A/B learns fastest once infra & eval mature.

Use indices to justify stage duration — not to skip monitoring at “green” dashboards.