Agent Observability in Practice: From Trace to Eval in a Production Loop

Background: Agent Production Failures Are Rarely Single-Point Errors

When a traditional web service fails, the debugging path is usually straightforward: check the HTTP status code, endpoint latency, database slow queries, exception stack traces, and server metrics. Large language model applications, especially Agent systems, are far more complex.

A seemingly ordinary user request might involve:

• Multi-turn conversation history concatenation
• Prompt template rendering
• Retrieval, ranking, and re-ranking
• Multiple model calls
• Tool calls, failure retries, and permission checks
• Guardrail interception
• Structured output parsing
• User feedback and online evaluation

When the final answer is wrong, it’s not necessarily a model capability issue. It could be that the retrieval results are stale, a tool returned incomplete data, the system prompt was overwritten, or a retry logic led the Agent down the wrong branch.

Therefore, LLM production governance cannot rely solely on regular logs. A more sensible approach is to establish Agent Observability: use Trace to reconstruct the execution process, Eval to measure output quality, Dataset to solidify failure samples, and Alert to drive continuous improvement.

Core Principles: The Three-Layer Division of Trace, Span, and Eval

Trace: The Complete Execution Chain of a Business Request

In Agent scenarios, a Trace should not be equated to a single HTTP request. It’s better suited to represent a complete business workflow, such as:

• A single customer service Q&A session
• A code modification task
• A RAG retrieval Q&A session
• An automated toolchain execution
• A multi-Agent collaboration

The value of Trace isn’t about “logging a lot,” but about establishing parent-child relationships: which step triggered which step, what input each step used, how long it took, where it failed, and what evidence supports the final output.

A usable Trace metadata set typically includes:

trace_id: trace_8f7a...
workflow_name: customer_support_agent
group_id: conversation_123
user_segment: paid_user
release_version: 2026.06.29
prompt_version: support-v17
model: gpt-4.1
sampling_policy: error_or_high_value

Span: Breaking the Black-Box Process into Locatable Steps

A Span is an operation fragment within a Trace. Common Spans in Agent systems include:

Span Type	Description
llm.generation	Model call
retriever.search	Vector or keyword search
tool.call	External tool invocation
guardrail.check	Safety, format, permission, or compliance check
parser.validate	Structured output parsing
memory.read / memory.write	Long-term memory read/write
eval.score	Online or offline evaluation result

More Spans aren’t always better. A good rule of thumb is: if a step could independently fail, slow down, or affect quality, it should be a Span.

Eval: Turning “Looks Good” into Comparable Metrics

Trace solves “what happened,” while Eval solves “how good was the result.”

Eval can be divided into two categories:

1. Offline Evaluation: Comparing versions on a fixed dataset before release. Ideal for regression testing, model switching, Prompt changes, and retrieval strategy modifications.
2. Online Evaluation: Scoring real traffic samples after release. Suitable for monitoring safety, format, relevance, hallucinations, tool misuse, and user experience degradation.

Evaluators shouldn’t be limited to LLM-as-a-Judge. A more robust approach for production is to combine:

• Code Rules: JSON Schema, field ranges, permission boundaries
• Traditional Metrics: Exact match, string containment, BLEU/ROUGE, etc.
• RAG Metrics: Context Precision, Context Recall, Faithfulness
• Agent Metrics: Tool Call Accuracy, Tool Call F1, Agent Goal Accuracy
• Human Review: For high-risk samples and evaluator calibration
• Pairwise Comparison: For determining which version (old vs. new) is better

Engineering Implementation: From Instrumentation to Closed Loop

Step 1: Unify the Trace Schema

Don’t let each business team invent their own field names. Otherwise, you won’t be able to aggregate, query, or migrate across tools later.

Start by defining a minimal schema:

trace:
  id: string
  workflow_name: string
  group_id: string
  started_at: datetime
  ended_at: datetime
  status: success | error | timeout | blocked
  metadata:
    tenant_id: string
    user_segment: string
    release_version: string
    prompt_version: string
    model: string

span:
  id: string
  parent_id: string
  type: llm | retriever | tool | guardrail | parser | memory | eval
  name: string
  started_at: datetime
  ended_at: datetime
  status: success | error | timeout
  input_summary: string
  output_summary: string
  token_usage:
    input_tokens: integer
    output_tokens: integer
  cost_estimate: number
  error:
    code: string
    message: string

If you plan to integrate with OpenTelemetry in the future, design your fields to align closely with semantic conventions. Don’t stuff everything into a single extra_json field.

Step 2: Instrument Critical Paths First, Don’t Capture Everything Initially

The most common mistake in production is trying to record all Prompts, all outputs, and all intermediate states in the first version of Observability.

A safer sequence is:

1. First, record Trace ID, Span parent-child relationships, latency, status, model name, Prompt version, and tool name.
2. Then, record tokens, cost, retry count, and timeout reasons.
3. Next, save input/output summaries for error samples.
4. Finally, save the full context for low-sensitivity, authorized, and sampled requests.

This approach lets you establish a queryable trace chain first, without being overwhelmed by privacy, cost, and storage issues from the start.

Step 3: Solidify Failed Traces into Datasets

The key to improving Agent quality isn’t “fixing the Prompt on the fly when a problem is found online,” but turning the problem into a regression asset.

A typical flow is:

1. Online Trace detects a failure sample.
2. Tag it by error type, e.g., retrieval_miss, tool_wrong_args, schema_invalid.
3. Clean sensitive information.
4. Extract it as a Dataset sample.
5. Write the expected behavior or reference answer.
6. Automatically run offline evaluation before the next Prompt, model, RAG, or toolchain change.

Only then will production incidents become a source of systemic capability, not just one-off patches.

Step 4: Attach Evaluation Results Back to the Trace

Don’t make Eval a separate report. It’s better to make evaluation results a part of the Trace.

For example:

{
  "eval": {
    "trace_id": "trace_8f7a...",
    "evaluator": "rag_faithfulness_v3",
    "score": 0.42,
    "label": "fail",
    "reason": "answer contains claims not supported by retrieved context",
    "evaluator_version": "2026.06.29"
  }
}

This way, during debugging, you can start from a low-scoring sample, navigate to its Trace, and examine its retrieval results, tool parameters, model output, and version information.

Step 5: Control Sampling and Costs

Online evaluation can’t simply run on all traffic. Common sampling strategies include:

• 100% sampling for error requests
• High sampling for high-value customers
• Increased sampling during new version canary releases
• Low sampling for low-risk chit-chat
• Separate sampling for long-context requests
• Forced sampling for tool call failures or retries

Cost dashboards should at least break down costs by: model, business line, Prompt version, toolchain, and user segment. Looking only at total tokens is meaningless because you can’t pinpoint which version or scenario caused the cost anomaly.

Applicable Scenarios

RAG Q&A Systems

RAG systems most need to see if “the answer is supported by evidence.” The Trace should at least retain: the query, retrieved document IDs, re-ranking scores, context snippet summaries, the final answer, and Faithfulness or Groundedness scores.

Tool-Calling Agents

For tool-calling Agents, parameters, permissions, and side effects are most critical. The Trace must record the tool name, parameter summary, permission check result, return status, retry count, and whether any external writes occurred.

Multi-Agent Collaboration

Multi-Agent systems are prone to unclear responsibility. The Trace needs to distinguish Agent names, roles, handoff reasons, handoff inputs/outputs, and which Agent is responsible for the final decision.

High-Compliance Industries

Scenarios like finance, healthcare, and government can’t just pursue observability; they must also pursue auditability. You may not be able to store the full Prompt, but you must at least save the version, evidence, authorization, redacted input/output summaries, and evaluation conclusions.

Common Pitfalls

Pitfall 1: Equating Observability with a Logging Platform

A logging platform records events; Observability must reconstruct causality. Without Trace IDs, Span hierarchies, version information, and evaluation results, even a mountain of logs is just fragmented data.

Pitfall 2: Only Monitoring Latency and Error Rates

The most critical problem for LLM applications is often not a 500 error, but “consistently producing wrong answers.” Therefore, quality metrics must be included in monitoring, including relevance, factuality, format compliance, tool call accuracy, and safe refusal rates.

Pitfall 3: Saving All Prompts and Outputs

This introduces privacy, compliance, storage cost, and internal misuse risks. It’s better to save summaries, hashes, structured fields, or redacted samples based on data sensitivity levels.

Pitfall 4: Adding Evaluation Only After Going Live

Evaluation should start during the development phase. The value of online Trace is to supplement the real-world distribution, not to replace the offline regression set.

Pitfall 5: Not Versioning Evaluators

Prompts change, models change, and evaluators change. Without evaluator versioning, you can’t explain why the same batch of samples scored differently this week compared to last week.

Go-Live Checklist

Trace and Span

• Does every business request have a globally unique Trace ID?
• Can you trace back from the final answer to the model calls, tool calls, and retrieval results?
• Are the model name, Prompt version, release version, and environment recorded?
• Are errors, timeouts, blocks, and user cancellations distinguished?
• Is Trace ID propagation supported across services?

Data and Privacy

• Is high-sensitivity raw content capture disabled by default?
• Are input/output redaction rules configured?
• Is there a data retention period?
• Are Trace viewing permissions restricted?
• Can data be deleted by user or tenant?

Evaluation and Alerting

• Is there an offline regression dataset?
• Is there an online sampling evaluation strategy?
• Are rule-based evaluation, LLM Judge, and human review distinguished?
• Are evaluators themselves versioned?
• Can low-scoring Traces be automatically routed to an analysis queue?

Cost and Performance

• Are input tokens, output tokens, and total cost estimates recorded?
• Is cost broken down by business line, model, and Prompt version?
• Are long-context requests monitored separately?
• Is the latency impact of Trace collection on the main execution path limited?
• Are concurrency and budget limits set for evaluation tasks?

Summary

Once LLM applications are in production, the real challenge isn’t “can we call the model,” but “when the result gets worse, can we explain why, reproduce the scene, and verify the fix.”

A practical Agent Observability system should have four closed loops:

1. Trace Loop: Reconstruct the complete execution process.
2. Eval Loop: Turn quality issues into comparable metrics.
3. Dataset Loop: Solidify online failures into offline regressions.
4. Governance Loop: Control risk with redaction, permissions, sampling, and retention policies.

Only by connecting these capabilities can a team upgrade from “guessing based on logs” to “improving the system based on evidence.”

References

• OpenTelemetry GenAI Semantic Conventions
• OpenAI Agents SDK Tracing
• LangSmith Observability
• LangSmith Evaluation
• Arize Phoenix Tracing
• OpenInference
• Ragas Documentation
• From Agent Traces to Trust: Evidence Tracing and Execution Provenance in LLM Agents