AI incidents are difficult when the evidence is fragmented. Provider dashboards show one slice, application logs show another, and agent behavior may span several model calls. A reliable response process narrows the scope first, then moves from aggregate signals to representative request or trace evidence.
Key takeaways
- Triage by scope before changing routing or retry policy.
- Inspect representative requests and traces rather than relying only on aggregate charts.
- Record the recovery action and correlate it with the metrics that improved.
Step 1: define the user-visible symptom
Start with what is actually wrong: elevated errors, slow responses, missing output, tool failures, cost spikes or degraded quality. Different symptoms can share the same provider incident, but they need different evidence.
Record the affected time range and workloads before making changes. That baseline prevents the investigation from drifting as new data arrives.
Step 2: narrow provider, model and workload scope
Compare success rate and latency by provider and model, then by application or agent. If only one workload is affected, investigate its release and request shape before rerouting the entire platform.
If several workloads fail on the same provider at the same time, provider health becomes a stronger hypothesis.
Step 3: inspect request and trace evidence
Open representative failures and slow requests. Check model, provider, status, latency, token sizes and retry history. For agents, follow the whole run to identify whether the failure happened in a model call, a tool or orchestration logic.
A few concrete traces often explain more than another dashboard refresh.
Step 4: choose the smallest safe recovery
Possible actions include rerouting one workload, disabling a problematic fallback, rolling back a release, reducing retry count or increasing a timeout when the upstream behavior justifies it.
Avoid broad changes when the evidence points to a narrow problem. Every recovery action should be reversible and observable.
Step 5: close the loop
After recovery, confirm that the user-visible metric improves and that the change did not create a cost or latency regression elsewhere. Preserve the incident timeline, root cause, affected workloads and recovery action.
Then ask which missing signal delayed diagnosis. The best reliability improvement often comes from making the next incident easier to explain.
FAQ
Common questions
What should I check first during an LLM incident?
Define the affected time window and workload, then compare error and latency signals by provider and model. Narrow the scope before changing routing or retry policy.
CLYVEL
Put the operating model into practice.
Clyvel connects production AI traffic, cost, reliability and governance in one operations layer.
See Clyvel Incident OperationsSources and further reading
Clyvel Research uses primary technical and vendor references wherever a claim benefits from external context.
Read the research methodology