How to Trace LLM Calls With OpenTelemetry

A single request to an AI feature can involve a retrieval query, two model calls, three tool calls and a retry. When a user reports a slow or wrong answer, logs scattered across services rarely show what happened in what order. Distributed tracing does.

OpenTelemetry is the vendor-neutral standard for traces, and it now includes conventions for generative AI calls.

Step 1: Instrument the service

Install the OpenTelemetry SDK for your language and export traces to a collector:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-agent")

Auto-instrumentation for your web framework and HTTP client adds spans for incoming requests and outgoing calls with very little code.

Step 2: Wrap each model call in a span

with tracer.start_as_current_span("chat claude-sonnet-5") as span:
    span.set_attribute("gen_ai.request.model", "claude-sonnet-5")
    response = client.messages.create(...)
    span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)

The gen_ai.* attribute names come from OpenTelemetry's semantic conventions for generative AI. They're still marked as in development, so check the current names, but using them means tracing tools can recognize the data. Several open-source instrumentation libraries add these spans for common SDKs automatically.

Step 3: Add spans for tools and retrieval

Give tool calls and retrieval queries their own spans with the tool name, duration and result size. A trace then shows the whole path: request, retrieval, model call, tool call, second model call, response.

Step 4: Decide what content to record

Prompts and responses are useful for debugging and risky for privacy. Record them only where your data policy allows, preferably as span events that the collector can drop, and never by default in production.

Step 5: Send traces somewhere useful

Point the collector at a tracing backend such as Jaeger, Grafana Tempo or a commercial observability service. Add trace IDs to your application logs, so a support ticket can lead straight to the trace.

Things to watch

• Sampling. Tracing every request at high volume is expensive. Keep all errors and slow requests, and sample the rest.
• Streaming responses. End the span when the stream finishes, not when the first chunk arrives, or durations will look too short.
• Attribute size. Long prompt text in attributes can exceed backend limits. Truncate it or store content separately.

Start by tracing one route end to end, and use that trace on the next slow-request report.