Phase 7 · Months 6 to 8

Trace every step an agent takes

Why logs are not enough

A normal request might make one database query and one API call. An agent request can make twenty model calls and fifteen tool calls, retrieve context several times, and branch. In a flat log stream you cannot follow what caused what. Distributed tracing is a core skill for AI engineering.

Learn OpenTelemetry

Learn traces, spans, context propagation, metrics, logs, collectors, exporters, and sampling. Instrument model calls, retrieval, tool calls, approvals, and external APIs. Record the same attributes on every span: model, provider, prompt version, workflow version, tokens, cost, tool, tenant, outcome, and failure category. Follow the OpenTelemetry semantic conventions for generative AI where they fit.

Use an LLM tracing tool

Use Langfuse or a similar tool to inspect traces, prompt versions, token use, latency, and eval scores. Learn how the same data reaches a general monitoring tool such as New Relic. A trace should answer five questions: what the agent saw, what it decided, which tool it called, how long each step took, and what it cost.

Trace your coding agents too

Coding agents are now among the largest AI workloads in an engineering organisation. Claude Code, for example, can export usage and cost data over OpenTelemetry. Send your own agent usage to the same tools. Cost per engineer, per repository, and per task type is exactly what a platform team gets asked about.

Track the metrics that matter

Track task success, p50 and p95 latency, p50 and p95 cost, token use, tool error rate, retry rate, timeout rate, retrieval quality, and how often a person overrides the agent. Split each metric by workflow version and model version, so you can find which change caused a regression.

Build an agent operations dashboard

Instrument your product and the PR reviewer from end to end. Build dashboards for volume, success rate, latency, cost, model mix, tool failures, and the most common failure categories. Link each alert to its traces, and carry correlation IDs through queues and workers.

You are done when

When an agent misbehaves, you can go from an unusual number on a dashboard to one trace. From that trace you can reconstruct the context, model calls, and tool actions that produced the result.

Exit checklist