Errors and resilience

Resilience patterns in action: circuit breaker, retry, timeout, and fallback applied to real flows between services.

Why resilience is part of the flow

In a distributed system, failures aren’t exceptions: they’re a normal part of operation. Services that go down, networks that fail, databases that get saturated. The question isn’t whether it will fail, but how the system behaves when it does.

Resilience patterns aren’t isolated theoretical concepts. They’re applied to the real flows between services so the system keeps working in a degraded mode instead of collapsing entirely.

The four main patterns

Timeout

Sets a maximum wait time for an operation. If there’s no response within that time, the request is cancelled and the error is handled.

Retry

Retries an operation that failed, on the assumption that the failure may be transient (a load spike, a momentary network error).

Circuit Breaker

Detects when a service is failing repeatedly and stops sending it requests for a while, allowing it to recover.

Fallback

Provides an alternative response when the primary operation fails.

Timeout in action

sequenceDiagram
    participant BFF as BFF
    participant MS as ms-catalog
    participant DB as Database

    BFF->>MS: GET /products (timeout: 3s)
    MS->>DB: SELECT * FROM products
    Note over DB: La DB está lenta (5s)
    
    BFF--xMS: Timeout después de 3s
    BFF->>BFF: Manejar timeout
    BFF->>BFF: Devolver respuesta de caché o error

How to configure timeouts

Every call between services must have an explicit timeout. Without one, a request can hang indefinitely, consuming resources.

Rules of thumb:

• The timeout should be greater than the service’s normal response time
• But less than the time the user is willing to wait
• Different operations can have different timeouts

What to do on a timeout

1. Log the timeout with context (service, operation, duration)
2. Return a degraded response if possible (cached data, default values)
3. Don’t assume the operation failed: it may have completed but the response was lost

Retry in action

sequenceDiagram
    participant BFF as BFF
    participant MS as ms-orders

    BFF->>MS: POST /orders
    MS-->>BFF: 503 Service Unavailable

    Note over BFF: Esperar 1s (intento 2)
    BFF->>MS: POST /orders
    MS-->>BFF: 503 Service Unavailable

    Note over BFF: Esperar 2s (intento 3)
    BFF->>MS: POST /orders
    MS-->>BFF: 201 Created

Retry strategies

Immediate retry: retry with no wait. Only useful for very transient errors.

Retry with fixed backoff: wait a fixed amount of time between retries (1s, 1s, 1s).

Retry with exponential backoff: double the wait time on each attempt (1s, 2s, 4s, 8s). This is the most recommended strategy because it reduces pressure on the failing service.

Retry with jitter: add a random component to the backoff to prevent multiple clients from retrying at the same time (thundering herd).

When to retry and when not to

Retry:

• 503 errors (Service Unavailable)
• 429 errors (Too Many Requests)
• Network timeouts
• Connection errors

Don’t retry:

• 400 errors (Bad Request) — the request is invalid
• 401/403 errors — authentication/authorization problem
• 404 errors — the resource doesn’t exist
• 409 errors (Conflict) — state conflict

Idempotency and retry

If a write operation is retried (create order, process payment), the receiving service must be idempotent. Otherwise, the retry can create duplicates.

Circuit Breaker in action

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: N fallos consecutivos
    Open --> HalfOpen: Después del timeout
    HalfOpen --> Closed: Request exitoso
    HalfOpen --> Open: Request falla

The three states

Closed: the circuit works normally. Requests pass through to the service. If failures accumulate, the circuit opens.

Open: the circuit is broken. Requests don’t reach the service; an error or fallback is returned immediately. This protects the failing service and prevents the caller from wasting resources waiting.

Half-Open: after a while, the circuit lets a test request through. If it succeeds, the circuit closes. If it fails, it opens again.

A concrete example

The BFF calls the catalog microservice. If 5 of the last 10 requests failed:

1. The circuit breaker opens
2. For 30 seconds, all requests to the catalog receive a fallback (cached data)
3. After 30 seconds, a test request is allowed through
4. If it works, the circuit closes and normal traffic resumes

Typical configuration

Fallback in action

When an operation fails (due to a timeout, an open circuit breaker, or an error), the fallback provides an alternative:

Degradation levels

Not all failures require the same response. You can define levels:

1. Minimal degradation: slightly stale data (5-minute cache)
2. Moderate degradation: reduced functionality (catalog without advanced filters)
3. Severe degradation: minimal functionality (read-only, no write operations)
4. Unavailability: maintenance page

Combining patterns

In practice, these patterns are combined:

BFF → [Timeout: 3s] → [Retry: 3 intentos con backoff] → [Circuit Breaker] → ms-catalog
                                                                    ↓ (si abierto)
                                                              [Fallback: caché]

1. The BFF makes a request with a 3-second timeout
2. If it times out, it retries up to 3 times with exponential backoff
3. If the retries fail, the circuit breaker opens
4. With the circuit open, subsequent requests receive the fallback directly

Observability of resilience

Each pattern should produce metrics and logs:

• Timeouts: how many, on which service, duration
• Retries: how many retries, success rate at each attempt
• Circuit Breaker: state transitions, time in each state
• Fallbacks: how many times it was triggered, what type of fallback

These metrics make it possible to detect problems before they turn into serious incidents.

Summary

Resilience isn’t a component you bolt on at the end. It’s a property of the system that’s designed into every flow. Timeout prevents infinite waits, retry handles transient failures, circuit breaker protects saturated services, and fallback maintains degraded functionality. Combined, they let the system keep working even when parts of it fail.