ESENPTFR
Contact
Change language
ESENPTFR

Distributed Tracing

Spans, traces, OpenTelemetry, Jaeger, and how to follow requests across multiple services.

What is distributed tracing?

When a user request travels through 5, 10, or 20 services before returning a response, how do you know where the time was spent? Which service failed? Which call was the slowest?

Distributed tracing solves exactly this problem: it lets you follow the complete journey of a request across all the services involved, measuring timings and capturing context at every step.

Core concepts

Trace

A trace represents the complete journey of an end-to-end operation. For example, a “create order” trace includes everything from the moment the frontend sends the request until the user receives the confirmation.

Every trace has a unique identifier (traceId) that is propagated across all the services involved.

Span

A span represents a unit of work within a trace. Each service or component that takes part in the operation creates one or more spans.

A span contains:

  • Name: A description of the operation (e.g., “POST /api/orders”).
  • Timestamps: The start and end of the operation.
  • SpanId: The span’s unique identifier.
  • ParentSpanId: A reference to the parent span, forming a tree.
  • Tags/Attributes: Additional metadata (HTTP method, status code, etc.).
  • Events/Logs: Point-in-time events within the span.
  • Status: OK, ERROR, or UNSET.

Propagation context

For tracing to work across services, the context (traceId, spanId) must be propagated on every call. The most common mechanisms are:

  • HTTP headers: traceparent, tracestate (the W3C Trace Context standard).
  • Messaging headers: Attributes on Kafka, RabbitMQ, or SQS messages.
  • gRPC metadata: Automatic propagation on gRPC calls.

OpenTelemetry

OpenTelemetry (OTel) is the open standard for observability instrumentation. It unifies the collection of traces, metrics, and logs under a single API.

Why OpenTelemetry?

Before OTel, every vendor had its own instrumentation API. If you used Jaeger, you instrumented with the Jaeger API. If you switched to Zipkin, you had to re-instrument everything. OpenTelemetry solves this with a vendor-neutral API.

OTel components

  1. API: The interface for instrumenting code. It is stable and does not change.
  2. SDK: The implementation of the API, with configuration for exporters, samplers, etc.
  3. Exporters: Send data to specific backends (Jaeger, Zipkin, OTLP).
  4. Collector: An intermediate component that receives, processes, and exports telemetry.
  5. Auto-instrumentation: Libraries that instrument popular frameworks automatically.

The Collector

The OTel Collector is a key component in production architectures:

Services → OTel Collector → Backend (Jaeger, Tempo, etc.)

Benefits of using a Collector:

  • Decoupling: Services don’t need to know which backend to send data to.
  • Processing: It can filter, sample, and enrich traces before exporting them.
  • Buffering: It absorbs traffic spikes without losing data.
  • Multi-backend: It can send data to multiple destinations simultaneously.

Jaeger

Jaeger is an open-source distributed tracing platform, originally developed by Uber.

Key features

  • Trace visualization: A web interface to explore traces as waterfall diagrams.
  • Search: Find traces by service, operation, tags, duration, or traceId.
  • Comparison: Compare two traces to identify performance differences.
  • Dependency graphs: Visualize the relationships between services based on the collected traces.

Jaeger architecture

Jaeger can be deployed in several ways:

  • All-in-one: A single binary for development and testing.
  • Production: A separate Collector, storage (Elasticsearch, Cassandra), and Query service.
  • With OTel Collector: Using the Collector as the receiver instead of the Jaeger agent.

Alternatives to Jaeger

  • Zipkin: Another open-source tracing platform, simpler in scope.
  • Grafana Tempo: A cost-optimized trace backend, integrated with Grafana.
  • AWS X-Ray: A managed tracing service on AWS.
  • Datadog APM: A SaaS platform with built-in tracing.

Correlation across signals

The true power of tracing emerges when it is correlated with logs and metrics:

Trace → Logs

Include the traceId in every log line. When you find a slow or error-prone trace, you can search for all the associated logs by filtering on that traceId.

Trace → Metrics

Traces can generate metrics automatically:

  • Latency by service and operation.
  • Error rate by endpoint.
  • Distribution of span durations.

This is known as “metrics derived from traces,” and tools like Grafana Tempo support it natively.

Exemplars

Exemplars are links from a metric to a specific trace. When you see a latency spike on a dashboard, you can click and jump straight to the trace that caused it.

Sampling

In high-traffic systems, capturing 100% of traces can be prohibitively expensive. Sampling lets you capture only a representative subset.

Types of sampling

  • Head-based sampling: The decision to capture is made at the start of the trace. Simple, but it can miss interesting traces.
  • Tail-based sampling: The decision is made at the end of the trace, once the outcome is known. It allows you to capture 100% of errors and slow traces, but it requires buffering.
  • Rate limiting: Capture at most N traces per second.
  • Capture 100% of errors — always.
  • Capture 100% of slow traces (above a threshold).
  • Apply probabilistic sampling to the rest (e.g., 10% of successful traces).

Best practices

Meaningful instrumentation

Don’t instrument every function — instrument the operations that cross boundaries:

  • HTTP/gRPC calls between services.
  • Database queries.
  • Publishing/consuming messages.
  • Calls to external services.

Descriptive span names

Use names that identify the operation, not the implementation:

  • Good: POST /api/orders, OrderRepository.save
  • Bad: handler, function1, doStuff

Useful attributes

Add attributes that help diagnose problems:

  • http.method, http.status_code, http.url
  • db.system, db.statement (without sensitive data)
  • user.id, order.id (business identifiers)

Consistent propagation

Make sure the context is propagated across all communication channels — HTTP, messaging, cron jobs, asynchronous workers. An incomplete trace is worse than no trace at all.

Summary

Distributed tracing is the third pillar of observability and arguably the most valuable in microservice architectures. It lets you understand the complete flow of an operation, identify bottlenecks, and diagnose errors that span multiple services. With OpenTelemetry as the standard and tools like Jaeger, implementing tracing is more accessible than ever.