Traceability

Traceability in distributed systems: correlation IDs, distributed tracing, and structured logging to follow the journey of each operation across services.

Why traceability is essential

In a monolith, following the flow of an operation is relatively simple: everything happens in the same process, with a single log. In a distributed system, a single user action can traverse 5 or more services, each with its own logs, timings, and potential failures.

Without traceability, diagnosing a problem becomes like searching for a needle in a distributed haystack. Traceability makes it possible to reconstruct the complete path of an operation across all the services involved.

The three pillars of traceability

1. Correlation ID

A unique identifier that accompanies an operation from its origin to the last service that processes it. All logs, events, and metrics related to that operation share the same correlation ID.

2. Distributed Tracing

A system that records the timing and relationships between operations in each service, making it possible to visualize the complete flow as a tree of spans.

3. Structured logging

Logs in a structured format (JSON) that include the correlation ID and contextual metadata, making search and correlation easier.

Correlation ID in action

sequenceDiagram
    participant FE as Frontend
    participant GW as API Gateway
    participant BFF as BFF
    participant MS1 as ms-orders
    participant MS2 as ms-inventory
    participant EB as Event Bus

    FE->>GW: POST /orders (X-Correlation-ID: abc-123)
    GW->>BFF: Forward (X-Correlation-ID: abc-123)
    BFF->>MS1: Crear orden (X-Correlation-ID: abc-123)
    MS1->>EB: OrderPlaced (correlationId: abc-123)
    EB->>MS2: OrderPlaced (correlationId: abc-123)
    MS2->>MS2: Reservar stock (log: abc-123)

How the Correlation ID propagates

1. The frontend generates a unique UUID when starting the operation (or the API Gateway generates one if none is provided)
2. It is included in the X-Correlation-ID HTTP header
3. Each service extracts it from the incoming request and includes it in:
   • All of its logs
   • All outgoing HTTP calls
   • All events it publishes to the Event Bus
   • All metrics it records
4. Event consumers extract it from the event payload

Fundamental rule

Never generate a new correlation ID in an intermediate service. If a service receives a request without a correlation ID, it may generate one. But if one is already present, it must propagate it as is.

Distributed Tracing

Distributed tracing goes beyond the correlation ID. It records the temporal structure of the flow: which service called which, how long each operation took, and where an error occurred.

Key concepts

Trace: represents the complete flow of an operation, from origin to end. A trace contains multiple spans.

Span: represents an individual operation within a service. It has a start, an end, a name, and metadata.

Parent-Child: spans are organized into a tree. The BFF’s span is the parent of the span of the microservice it invokes.

Example of a trace

Trace: abc-123
├── [Gateway] POST /orders (5ms)
│   └── [BFF] createOrder (45ms)
│       ├── [ms-orders] createOrder (30ms)
│       │   └── [DB] INSERT INTO orders (8ms)
│       └── [ms-orders] publishEvent (5ms)
└── [ms-inventory] handleOrderPlaced (20ms)  ← asíncrono
    └── [DB] UPDATE stock (6ms)

This trace shows that:

• The total request took 50ms (Gateway → BFF → ms-orders)
• The database operation in ms-orders took 8ms
• The asynchronous processing in inventory took an additional 20ms

Tracing tools

The most common tools for distributed tracing are:

OpenTelemetry is becoming the de facto standard because it is vendor-neutral and allows switching tools without modifying the instrumentation code.

Structured logging

Why structured logs

Plain-text logs are hard to search and correlate:

2024-01-15 10:30:00 INFO Order created successfully for customer 789

Structured logs in JSON enable precise searches:

{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "INFO",
  "service": "ms-orders",
  "correlationId": "abc-123",
  "traceId": "trace-456",
  "spanId": "span-789",
  "message": "Order created successfully",
  "context": {
    "orderId": "order-456",
    "customerId": "cust-789",
    "totalAmount": 150.00
  }
}

Standard fields in every log

All services should include these fields in every log entry:

Log centralization

In a distributed system, the logs from each service should be sent to a centralized system where they can be searched and correlated:

graph LR
    MS1[ms-orders] --> AGG[Log Aggregator]
    MS2[ms-inventory] --> AGG
    MS3[ms-payments] --> AGG
    BFF[BFF] --> AGG
    AGG --> STORE[Elasticsearch / CloudWatch / Loki]
    STORE --> UI[Kibana / Grafana]

Traceability in asynchronous flows

Traceability in asynchronous flows is more complex because there is no direct HTTP connection between the publisher and the consumer of the event.

Propagation through events

The correlation ID and trace ID must be included in the event payload:

{
  "type": "OrderPlaced",
  "metadata": {
    "correlationId": "abc-123",
    "traceId": "trace-456",
    "parentSpanId": "span-789",
    "timestamp": "2024-01-15T10:30:00Z",
    "source": "ms-orders"
  },
  "payload": { ... }
}

When the consumer processes the event, it creates a new span that is a child of the original span, maintaining the traceability chain.

Challenge: latency between publishing and consumption

In asynchronous flows, time may pass between publishing and consumption. The trace may show a temporal gap that is normal and expected.

Diagnosing problems with traceability

Scenario: the user reports that their order was not confirmed

1. Obtain the order’s correlation ID (from the frontend or the BFF logs)
2. Search for all logs with that correlation ID
3. Visualize the distributed trace
4. Identify where the flow stopped:
   • Was the OrderPlaced event published?
   • Did the inventory service receive it?
   • Was there an error during processing?
   • Was the response event published?

Without traceability, this diagnosis would require reviewing each service’s logs manually, searching by timestamp and hoping to find something relevant.

Summary

Traceability is what makes a distributed system operable. The correlation ID connects all the operations of a flow, distributed tracing shows the temporal structure and dependencies, and structured logging enables efficient search and correlation. Without these three pillars, diagnosing problems in production becomes an almost impossible task.