Retry Pattern

How to handle transient failures in distributed systems by retrying operations with backoff strategies to improve resilience.

What problem it solves

In distributed systems, calls between services can fail for transient reasons: momentary network issues, services restarting, temporary rate limiting, or load spikes. These failures are temporary, and the operation will likely succeed if retried.

Without Retry:
  Service A ──► Service B
                 ❌ Error 503 (temporary overload)
  
  Service A returns an error to the user
  (even though B recovers in 2 seconds)

The problem is that without retries, a transient failure lasting milliseconds turns into an error visible to the user, when simply waiting a moment and retrying would have solved it.

Common transient failures

Failure type	Cause	Typical duration
Network timeout	Momentary congestion	Milliseconds to seconds
HTTP 503	Service restarting or overloaded	Seconds
HTTP 429	Rate limit reached	Seconds to minutes
Connection refused	Service starting up	Seconds
DNS resolution failure	Expired DNS cache	Milliseconds

How it works

The Retry pattern intercepts failed calls and retries them automatically according to a configured strategy. The key lies in when to retry, how many times, and how long to wait between attempts.

With Retry:
  Service A ──► Service B
                 ❌ Attempt 1: Error 503
                 ⏳ Wait 1 second
                 ❌ Attempt 2: Error 503
                 ⏳ Wait 2 seconds
                 ✅ Attempt 3: OK 200
  
  The user receives a successful response

Wait strategies (backoff)

Strategy	Formula	Example (3 attempts)	Use
Fixed delay	Constant `delay`	1s, 1s, 1s	Simple cases
Exponential backoff	`delay × 2^attempt`	1s, 2s, 4s	Recommended standard
Exponential + jitter	`random(0, delay × 2^attempt)`	0.7s, 1.3s, 3.8s	Avoids thundering herd
Linear backoff	`delay × attempt`	1s, 2s, 3s	Predictable increment

Exponential backoff with jitter

The most recommended strategy is exponential backoff with jitter (random variation). Without jitter, when a service recovers, all clients retry at the same time, creating a load spike that can take it down again (the thundering herd problem).

Without jitter (thundering herd):
  Client 1: retry at 1s, 2s, 4s
  Client 2: retry at 1s, 2s, 4s    ← All at the same time
  Client 3: retry at 1s, 2s, 4s
  
  Service B receives 3 simultaneous requests in each interval

With jitter:
  Client 1: retry at 0.7s, 1.8s, 3.2s
  Client 2: retry at 1.2s, 2.3s, 4.5s  ← Spread out over time
  Client 3: retry at 0.9s, 1.5s, 3.9s
  
  Service B receives requests spread out over time

Configuration parameters

Parameter	Description	Typical value
Max retries	Maximum number of retries	3-5
Initial delay	Wait time before the first retry	100ms - 1s
Max delay	Maximum wait time between retries	30s - 60s
Backoff multiplier	Exponential multiplication factor	2
Retryable errors	Error codes that justify a retry	408, 429, 500, 502, 503, 504

What to retry and what not to

✅ Retry (transient failures):
  - HTTP 408 Request Timeout
  - HTTP 429 Too Many Requests
  - HTTP 500 Internal Server Error (with caution)
  - HTTP 502 Bad Gateway
  - HTTP 503 Service Unavailable
  - HTTP 504 Gateway Timeout
  - Connection timeout / Connection refused

❌ Do NOT retry (permanent failures):
  - HTTP 400 Bad Request (invalid data)
  - HTTP 401 Unauthorized (incorrect credentials)
  - HTTP 403 Forbidden (no permissions)
  - HTTP 404 Not Found (resource does not exist)
  - HTTP 409 Conflict (state conflict)
  - HTTP 422 Unprocessable Entity (validation failed)

Advantages

Resilience against transient failures: Most network errors resolve themselves
Transparent to the user: The retry happens without the user noticing
Simple implementation: It is one of the easiest resilience patterns to implement
Configurable: Parameters can be tuned according to context and criticality
Complements other patterns: Works well with Circuit Breaker and Timeout

Trade-offs / Disadvantages

Accumulated latency: Each retry adds to the total wait time
Additional load: Retries generate more traffic toward the service that is already failing
Non-idempotent operations: Retrying an operation that already executed can cause duplicates
False positives: Retrying a permanent error wastes resources
Thundering herd: Without jitter, synchronized retries can make the situation worse
Configuration complexity: Incorrect parameters can cause more problems than solutions

When to use

HTTP calls to external services or microservices
Network operations that can fail for transient reasons
Database queries that can fail due to exhausted connections
Publishing messages to brokers that may be temporarily unavailable
Any idempotent operation that fails intermittently

When to avoid

Non-idempotent operations where retrying can cause duplicated side effects
When the error is clearly permanent (400, 401, 403, 404)
Long-running operations where the total timeout would be unacceptable
When you already have an open Circuit Breaker (the retry should not try to go through an open circuit)
Systems where the additional latency from retries is unacceptable

Common technologies and implementations

Category	Options
Libraries	Resilience4j (Java), Polly (.NET), tenacity (Python), axios-retry (Node.js)
HTTP Clients	Native configuration in HttpClient (.NET), OkHttp (Java), Got (Node.js)
Cloud	AWS SDK (built-in retry), Azure SDK (retry policies), Google Cloud Client Libraries
Service Mesh	Istio, Linkerd, Envoy (retry at the infrastructure level)

Relationship with other patterns

Retry is rarely used on its own. It is part of a resilience chain:

Request
   │
   ▼
┌──────────┐
│ Timeout  │  Defines how long to wait per attempt
└────┬─────┘
     ▼
┌──────────┐
│  Retry   │  Retries if the attempt fails
└────┬─────┘
     ▼
┌──────────┐
│ Circuit  │  Cuts off if there are too many failures
│ Breaker  │
└────┬─────┘
     ▼
Downstream service

Timeout: Defines how long to wait on each individual attempt before considering it failed
Circuit Breaker: If retries fail consistently, the circuit breaker opens and prevents further attempts
Idempotency: Downstream services must be idempotent for retries to be safe
Bulkhead: Isolates resources so that retries to one service do not affect others

Next steps

Retry is the first line of defense against transient failures. To define how long to wait on each attempt, explore the Timeout pattern. To ensure that retries do not cause duplicated effects, review the Idempotency pattern.