LLM Observability in Production: Tracing Agent Call Chains with OpenTelemetry GenAI Semantic Conventions

Background: Why LLM Applications Need More Than API Logs

Traditional web service issues are usually pinpointed via HTTP status codes, API latency, slow database queries, and error stacks. But LLM applications have a much longer failure path: a single user request can pass through RAG retrieval, prompt assembly, model calls, function calls, external tools, retries, secondary summarization, content moderation, and the final response.

When users report “slow responses,” “sudden cost spikes,” “the Agent called the wrong tool,” or “RAG retrieved irrelevant material,” standard logs often only show a successful API return, without revealing what happened at each intermediate step.

This is the value of LLM Observability: not just logging a single chat.completions.create(), but weaving model calls, retrieval, tools, tokens, latency, anomalies, costs, and quality signals into an analyzable execution chain.

In 2026, OpenTelemetry further strengthened its semantic conventions for GenAI scenarios. The official article states that GenAI semantic conventions standardize the recording of model names, input/output token counts, and, when explicitly enabled, prompts, completions, tool calls, and tool results.

Core Problem: What to Observe in an Agent Call Chain

A production-grade Agent request should be broken down into at least the following categories of observability objects.

1. Request Entry Point

The entry layer should record the user request, tenant, session, business scenario, call source, request ID, and Trace ID. It doesn’t necessarily need to store the raw user input, but it must be able to link an LLM execution to a business request.

Key fields include:

• trace_id: Spans the entire call chain.
• session_id: Links multi-turn conversations.
• tenant_id: Essential for multi-tenant cost and issue diagnosis.
• user_intent: Optional, for subsequent quality analysis.
• feature_flag / prompt_version: To pinpoint issues introduced by prompt changes.

2. Retrieval and Context Construction

Common RAG system issues are often not about model capability but unstable retrieval context quality. Therefore, you should record:

• Queries before and after query rewriting.
• Retrieval source, index name, and filter conditions.
• Top-k document IDs, scores, and version numbers.
• Ranking changes before and after reranking.
• The chunk IDs that ultimately entered the prompt.

The OpenInference documentation explicitly states that one of its specification goals is to provide visibility into LLM calls and their surrounding application context, including vector store retrieval and external tool usage.

3. Model Calls

Model calls are the easiest to observe but also the easiest to observe incorrectly. Recording only the model name and latency is insufficient. At a minimum, you should record:

• provider / model: The model provider and specific model.
• operation name: e.g., chat, embedding, rerank, tool_call, agent_run.
• input_tokens / output_tokens: Core metrics for cost and performance analysis.
• streaming / non-streaming: Affects TTFT and user experience.
• finish_reason: To determine truncation, refusal, tool call, or normal stop.
• temperature / max_tokens: Important context for explaining output instability.

Datadog’s article on OpenTelemetry GenAI support also explains that the GenAI semantic conventions establish a unified schema for prompts, model responses, token usage, tool/agent calls, and provider metadata.

4. Tool Calls and External Dependencies

A significant portion of an Agent’s uncertainty comes from tool calls. Each tool call should be a separate Span, not buried within the model’s output text.

Tool Spans should include:

• tool name.
• tool type, e.g., function, datastore, extension.
• Parameter schema version.
• Parameter summary or redacted parameters.
• Execution latency.
• Return status.
• Error type.
• Whether retry or degradation was triggered.

If the tool result affects the final answer, also record the tool result ID or summary for subsequent auditing. Do not save complete sensitive results by default.

5. Output Quality and Post-Processing

LLM Observability should not stop at “whether it returned successfully.” For production applications, you also need to capture quality signals:

• Whether cited evidence was hit.
• Whether structured output validation passed.
• Whether security policies were triggered.
• Whether manual review was initiated.
• Whether the user liked, disliked, or followed up.
• Whether it entered an offline evaluation dataset.

LangSmith’s documentation also covers observability for individual traces, production-wide performance metrics, dashboards, alerts, feedback, and online evaluations.

Engineering Value of OpenTelemetry GenAI Semantic Conventions

The significance of OpenTelemetry GenAI semantic conventions is not “yet another set of field names,” but to move LLM application observability data from tool-specific formats to a unified standard.

Unified Fields, Reduced Migration Costs

If every framework defines its own model_name, llm_model, provider_model, token_input_count, integrating with data warehouses, alerting systems, cost systems, and audit systems becomes chaotic.

The GenAI semantic conventions unify common fields into a structure like:

• gen_ai.request.model
• gen_ai.response.model
• gen_ai.provider.name
• gen_ai.operation.name
• gen_ai.usage.input_tokens
• gen_ai.usage.output_tokens
• gen_ai.response.finish_reasons

MLflow’s documentation also clearly states that the GenAI semantic conventions provide a standard schema for describing AI and LLM telemetry and support mapping to gen_ai.* fields during OTLP export.

One Instrumentation, Multiple Consumers

An ideal architecture is: the application side generates traces using OpenTelemetry or compatible libraries, which are then processed by the OpenTelemetry Collector and routed to multiple backends.

Application / Agent Runtime
    -> OpenTelemetry SDK or GenAI Instrumentation
    -> OpenTelemetry Collector
    -> Redaction / Sampling / Enrichment / Routing
    -> Observability Backend / Data Lake / Evaluation System

The benefits of this architecture are:

• Application code is not tied to any single observability platform.
• Redaction, sampling, and routing can be centrally managed.
• The same trace can feed into APM, cost systems, evaluation systems, and audit systems.
• Platform migration does not require rewriting all instrumentation.

Datadog’s article also emphasizes that using the Collector allows applying processors like redaction, sampling, enrichment, and routing before data leaves the network.

Compatibility with OpenLLMetry, OpenInference, MLflow, Phoenix, etc.

The current LLM Observability ecosystem does not have just one standard. OpenLLMetry provides non-intrusive tracing based on OpenTelemetry and can export to existing observability stacks. OpenInference provides conventions and plugins for tracing AI applications and is natively supported by Phoenix.

Phoenix documentation notes that different instrumentation standards use different semantic conventions to describe LLM operations, so a Span Processor is needed for attribute name mapping, format conversion, and data preservation.

This means that when implementing in production, you shouldn’t just ask “which UI looks better,” but first ask:

• Is my data schema portable?
• Does it support OTLP?
• Can redaction be done at the Collector layer?
• Can it be exported to a data lake or evaluation system?
• Can it be correlated with existing APM traces?

Recommended Implementation Architecture

Layer 1: Minimal Application Instrumentation

Don’t start by writing all Spans manually. Prioritize using mature instrumentation:

Tool	Use Case
OpenLLMetry	Quickly add tracing for OpenAI, Anthropic, LangChain, LlamaIndex, etc.
OpenInference	Projects within the Phoenix/Arize ecosystem needing RAG and tool context recording
MLflow Tracing	Teams already using MLflow needing integration with experiments, evaluation, and prompt management
LangSmith	LangChain/LangGraph projects needing a unified platform for debugging, feedback, and online evaluation

However, automatic instrumentation cannot replace business Spans. It’s still recommended to manually supplement core business nodes:

from opentelemetry import trace

tracer = trace.get_tracer("agent-runtime")

with tracer.start_as_current_span("agent.plan") as span:
    span.set_attribute("app.prompt.version", "support-agent-v17")
    span.set_attribute("app.tenant.id", tenant_id)
    span.set_attribute("app.routing.policy", "fast-model-first")
    plan = build_plan(user_message)

Layer 2: Centralized Collector Governance

LLM traces may contain sensitive information. Don’t blindly send all prompts and tool results to a SaaS platform.

The Collector layer should at least perform:

• Redaction: Mask phone numbers, emails, ID numbers, addresses, keys, contract numbers.
• Sampling: Sample high-frequency successful requests; retain all anomalous requests.
• Enrichment: Add environment, version, business line, tenant tier.
• Routing: Route sensitive business traces to internal networks, general aggregated metrics to SaaS.
• Retention: Set different retention periods based on data sensitivity levels.

It’s recommended to make “whether to capture full Prompt/Completion” an explicit toggle, not a default behavior.

Layer 3: Metrics and Alerts

Don’t just build a pretty trace UI. Production alerts should at least cover the following metrics:

Metric	Description	Suggested Use
TTFT	Time to First Token for streaming responses	User experience alert
Total latency	Complete request duration	End-to-end SLA
Input tokens	Number of input tokens	Cost and context bloat management
Output tokens	Number of output tokens	Cost and truncation analysis
Tool call count	Number of tool calls per request	Agent loop and anomaly detection
Retry count	Number of model or tool retries	Stability analysis
Retrieval hit rate	Evidence hit rate from retrieval	RAG quality analysis
Validation failure rate	Proportion of structured output failures	Engineering reliability alert
Safety block rate	Rate of safety policy blocks	Risk monitoring

Layer 4: From Traces Back to Evaluation

The value of traces should not be limited to troubleshooting. High-value production traces should be fed back into:

• golden datasets.
• prompt regression tests.
• agent tool-use benchmarks.
• reranker training or tuning samples.
• safety rule regression test cases.

A 2026 survey on Agent traces and execution provenance also points out that looking only at final answer accuracy is insufficient to explain how the output was generated, how evidence supports claims, whether tool calls were reasonable, and where errors originated.

Common Pitfalls

Pitfall 1: Only recording model requests and responses

This only solves the most superficial problems. The real stability issues often lie in retrieval, tools, retries, prompt versions, and post-processing.

Pitfall 2: Saving full prompts as regular logs

Prompts may contain user privacy, business data, access credentials, and internal policies. Saving everything by default amplifies compliance risks.

Pitfall 3: Connecting to only one observability SaaS without retaining an open schema

Convenient in the short term, but can lead to data lock-in in the long run. It’s recommended to at least ensure OTLP export, standard field mapping, and batch export capabilities.

Pitfall 4: Only looking at average latency

LLM applications should focus more on percentiles, TTFT, tool call latency, retry latency, and latency differences across different model paths.

Pitfall 5: Not linking traces with prompt versions

When prompt versions, model versions, tool schema versions, and retrieval index versions are missing, it’s very difficult to reproduce issues in production.

Pre-Launch Checklist

Before going live, check each item:

1. Does every request have a unified trace_id?
2. Can you navigate from a business request to the LLM trace?
3. Are model, provider, tokens, and finish reason recorded?
4. Are retrieved document IDs and index versions recorded?
5. Is each tool call broken out into its own Span?
6. Are tool errors, retries, and degradations recorded?
7. Is there a toggle for Prompt/Completion capture?
8. Is redaction performed at the Collector or gateway layer?
9. Is there a distinction between sampling successful requests and retaining all anomalous requests?
10. Are there alerts for cost, latency, tool loops, and structured output failures?
11. Can production traces be fed back into offline evaluation sets?
12. Are access permissions and retention periods defined for trace data?

Conclusion

The core of LLM Observability is not “looking at a single model call,” but reconstructing the full context of an Agent’s decision: how the user request entered the system, what evidence was retrieved, which model version was called, what tools were executed, where tokens and latency were spent, and why the output was accepted or blocked.

The OpenTelemetry GenAI semantic conventions provide a more robust engineering path: describe GenAI workloads with open fields, centrally govern data with the Collector, and connect APM, cost, evaluation, and audit systems with standard traces.

For teams pushing RAG, Agents, tool calls, and multi-model routing into production, it’s recommended to incorporate observability design into the architecture early, rather than patching in logs after issues arise in production.

Key References

• OpenTelemetry GenAI Semantic Conventions Repository: https://github.com/open-telemetry/semantic-conventions-genai
• OpenTelemetry Blog: Inside the LLM Call — GenAI Observability with OpenTelemetry: https://opentelemetry.io/blog/2026/genai-observability/
• OpenLLMetry Documentation: https://www.traceloop.com/docs/openllmetry/introduction
• OpenInference Documentation: https://arize-ai.github.io/openinference/
• Phoenix Documentation — Translating Semantic Conventions: https://arize.com/docs/phoenix/tracing/concepts-tracing/translating-conventions
• MLflow: OpenTelemetry GenAI Semantic Conventions: https://mlflow.org/docs/latest/genai/tracing/opentelemetry/genai-semconv/
• LangSmith Observability Documentation: https://docs.langchain.com/langsmith/observability
• Datadog: LLM Observability Natively Supports OpenTelemetry GenAI Semantic Conventions: https://www.datadoghq.com/blog/llm-otel-semantic-convention/
• From Agent Traces to Trust: Evidence Tracing and Execution Provenance in LLM Agents: https://arxiv.org/abs/2606.04990