LLM Observability: What to Monitor in Production AI

A dashboard can show that latency increased. Observability should help identify whether the change came from one model, one provider, a larger prompt, a retry pattern or a tool call inside an agent.

That distinction is important because AI systems often return a valid HTTP response even when the experience is operationally poor. A request may succeed but be slow, expensive, routed unexpectedly or part of a multi-step workflow that failed elsewhere.

For every model request, capture the model, provider, latency, status, input tokens, output tokens and a workload identifier such as application, agent, environment or workspace.

Those dimensions let teams move from aggregate totals to operational questions. Instead of seeing that token use increased, you can identify which workload created the increase. Instead of seeing a global latency average, you can compare providers or models and inspect the requests behind the change.

A simple chatbot may generate one model call per user action. An agent can generate a chain of model calls, tool calls, retries and decisions. Request logs alone become insufficient because the interesting unit is now the workflow.

OpenTelemetry's Generative AI semantic conventions standardize telemetry for model operations, including requested model, token usage and operation duration, and can represent messages and tool calls when content capture is enabled. That direction is useful because it gives engineering teams a shared vocabulary rather than a new logging schema for every provider.

Cost is not only a finance metric. A latency regression that triggers retries can become a cost regression. A prompt update can increase context size. A routing change can move a high-volume workload to a more expensive model.

Request-level cost attribution lets teams correlate those changes. The useful question is rarely 'What was our bill?' It is 'What changed in production that caused the bill to move?'

A useful observability surface should let a team move from overview to evidence. Start with traffic, success rate, P95 latency, error rate and spend. Then drill into providers, models, applications or agents. Finally, inspect the request or trace that explains the anomaly.

That workflow reduces the gap between detecting a problem and understanding it. It also creates a shared operational view for engineering, platform and FinOps teams.