Generative AI adds model, token and workflow dimensions that normal HTTP telemetry does not explain on its own. OpenTelemetry's GenAI semantic conventions provide a useful shared vocabulary for model operations. The production goal is not to collect every possible attribute; it is to create consistent telemetry that lets a team move from a slow, expensive or failed workflow to the evidence behind it.
Key takeaways
- Keep a stable trace context across model calls, tool calls and retries when they belong to one logical operation.
- Capture model, provider, operation duration and token usage consistently before adding high-cardinality detail.
- Treat prompt and response content as sensitive data; observability does not require indiscriminate content capture.
Why GenAI needs additional semantic context
A normal span can tell you that an HTTP call took two seconds. It cannot automatically tell you which requested model handled the call, how many input or output tokens were used, whether the call was part of an agent trace or whether the same workflow later retried against another provider.
GenAI-aware attributes close that gap. They let teams correlate model operations with the broader application trace instead of building a separate observability island for every provider SDK.
Start with the minimum operational fields
For each model operation, preserve a trace or request identifier, operation name, provider or system, requested model, resolved model when available, status, duration and token usage when the provider reports it. Add workload identity such as application, agent or environment through your own stable resource or span attributes.
This is enough to answer a large class of production questions without storing full prompts. It supports provider comparison, model mix analysis, latency investigation, token trends and request-level cost attribution.
Agent traces need parent-child structure
An agent may plan, call a model, invoke a tool, retry and call another model. Those events should not be flattened into unrelated logs. Parent-child spans give the workflow a shape and make critical-path latency visible.
A trace should let an operator see which step dominated duration, which call failed and whether a retry or tool invocation created additional cost. That is far more useful than a dashboard that only reports aggregate request totals.
Be deliberate about content capture
Prompts, responses and tool arguments can contain customer data, credentials or other sensitive material. Capturing them by default can turn an observability system into a new data-governance problem.
Prefer metadata-first telemetry. If content capture is required for debugging or evaluation, define explicit retention, access and redaction rules and make the behavior visible to operators. Operational usefulness and data minimization can coexist.
Normalize first, optimize later
The largest long-term benefit of a shared telemetry model is comparability. When providers and models report into one normalized request or trace view, teams can investigate the same latency, error, token and cost dimensions regardless of upstream vendor.
Once that evidence is trustworthy, it can drive routing, reliability and FinOps decisions. Observability becomes the measurement layer beneath the rest of the AI control plane rather than a separate set of charts.
FAQ
Common questions
What should I capture for LLM observability with OpenTelemetry?
Start with trace context, operation, provider/system, model, status, duration and token usage when available. Add stable workload identity, then expand only where the extra detail supports a real operational question.
Should OpenTelemetry traces store full prompts and responses?
Not by default. Prompt and response content can be sensitive. Use metadata-first telemetry and only capture content under explicit access, retention and redaction rules.
CLYVEL
Put the operating model into practice.
Clyvel connects production AI traffic, cost, reliability and governance in one operations layer.
Explore Clyvel ObservabilitySources and further reading
Clyvel Research uses primary technical and vendor references wherever a claim benefits from external context.
Read the research methodology