When Production AI Breaks

Production AI incidents are rarely limited to “the model is down.” The API may be healthy while retrieval serves stale permissions. A prompt change may degrade one workflow. A provider may return slower but valid responses until queues overwhelm the system. Incident management needs to cover code, configuration, data, models, integrations, and customer impact.

The objective during an incident is to reduce harm and restore a safe service—not to win an argument about the root cause. Investigation becomes faster when roles, evidence, rollback choices, and communication paths are prepared in advance.

Stabilize the response before debugging deeply

Declare an incident when customer impact or credible risk crosses a defined threshold. Assign an incident commander to coordinate decisions, an operations lead to investigate and mitigate, and a communications owner to keep internal and external stakeholders current. On a small team, one person may hold two roles, but ownership should still be explicit.

Write the known facts, affected customers or workflows, start time, current severity, and next update time in one incident channel or record. Preserve evidence while reducing impact. Safe mitigations can include disabling a feature flag, routing to a fallback model, pausing ingestion, limiting traffic, isolating a tenant, or switching to a documented manual process.

If sensitive data may have crossed an authorization boundary, treat containment and evidence preservation as security work. Follow the organization's response plan and applicable notification process rather than improvising in a public status update.

Debug from the customer symptom inward

Start with a specific failed journey and a time window. Obtain its request, trace, pseudonymous tenant, job, document, and release identifiers. Compare it with a successful journey where possible.

Form a falsifiable hypothesis: “jobs accepted after release X are waiting because workers cannot refresh provider credentials.” Check the smallest set of signals that can disprove it. Randomly searching logs for “error” creates noise and can expose sensitive content.

Log analysis is most effective when events are structured and correlated. Look for the first abnormal boundary, not merely the last exception. A timeout at the API may originate in queue saturation, a retry storm, a slow vector query, or a provider rate limit. Check deploys, feature flags, prompt and model versions, schema migrations, permission updates, and external status changes on the same timeline.

Use production access carefully. Time-bound elevated access, record queries, redact outputs, and avoid copying customer data into chat or tickets. Prefer purpose-built diagnostic views over direct database access.

Choose rollback based on state

Kubernetes Deployments can roll back application revisions, but an AI release is often broader than a container. Maintain versioned records for image, chart, configuration, prompt, model, retrieval index, feature flags, and database migrations.

Rollback is safest when changes are backward-compatible and external side effects are idempotent. It may be unsafe after a destructive schema migration, a sent customer message, or a reprocessed corpus. In those cases, disable the affected path, restore from a verified backup, or apply a forward fix with explicit review.

Define rollback triggers and authority before deployment. Test the procedure, including how to verify recovery. “Pods are green” is insufficient; replay or exercise the customer journey that failed.

Communicate with useful certainty

An initial customer message should acknowledge impact, describe affected functionality in plain language, give available workarounds, and state when the next update will arrive. Say what is known and unknown. Avoid speculative causes and overly precise restoration times.

Update on the promised cadence even if investigation continues. After recovery, confirm what is working and whether customers need to take action. Security or privacy incidents require coordination with legal and security owners before detailed external statements.

Support needs the same source of truth, an approved response, affected-customer scope, and an escalation route. This prevents contradictory one-off explanations.

Keep that source of truth actionable with maintained documentation and runbooks.

Write a learning-focused postmortem

Build the timeline from telemetry, deploy records, tickets, and incident notes. Explain impact, detection, contributing conditions, response, and recovery. Separate human decisions made with available information from system conditions that made mistakes likely.

Action items should have owners, priorities, and due dates. Include improvements to prevention, detection, mitigation, and response—not only the code fix. Share an appropriately scoped summary with customers when useful and permitted.

Common failure modes

Waiting for certainty before declaring or communicating an incident.
Letting several people direct mitigation without one commander.
Searching unstructured logs broadly and exposing customer data.
Rolling back code without accounting for prompts, indexes, schemas, or side effects.
Promising a recovery time based on hope rather than evidence.
Writing a blame-focused postmortem with vague actions such as “be more careful.”
Closing the incident when infrastructure recovers but the customer workflow remains broken.

Implementation checklist

Define severities, declaration thresholds, roles, and escalation paths.
Maintain correlated, access-controlled telemetry and release records.
Pre-plan safe degradation, feature disablement, and manual fallbacks.
Version every behavior-changing artifact and test rollback paths.
Prepare internal, status-page, and direct-customer message templates.
Run incident exercises that include data, provider, and permission failures.
Verify recovery through the affected customer journey.
Complete a postmortem with owned, tracked actions.

Measurable signals

Track time to detect, declare, acknowledge, mitigate, and recover; time to first customer update; update-cadence adherence; percentage of incidents with a usable trace and release record; rollback success; repeat incidents; postmortem completion; and action-item closure. Review false alarms and incidents found by customers before internal monitoring.