ESENPTFR
Contact
Change language
ESENPTFR

Failure Simulator

Simulate production failures — service outages, timeouts, network errors — and practice resilience strategies.

Goal

In this lab you’ll simulate common failures in distributed systems and practice the resilience strategies that mitigate them. Each scenario presents a failure, asks you to identify the impact, and to propose a solution.

In production, failures don’t give you a heads-up. Here you can practice without consequences.

Scenario 1: Cascading timeout

The failure

Service A calls service B, which calls service C. Service C starts responding slowly (3 seconds instead of 200ms). It isn’t down — just slow.

[Service A] ──timeout 5s──► [Service B] ──timeout 5s──► [Service C: SLOW 3s]

What happens?

  1. C responds in 3s → B waits 3s + its own processing (500ms) = 3.5s
  2. A waits 3.5s for B → users see latency of 4s+
  3. A’s and B’s threads are blocked waiting → the connection pool is exhausted
  4. New requests to A start failing with timeouts → cascading failure

Exercise: Design the mitigation

Answer these questions:

  1. What timeout would you set on each service?

    • Hint: timeouts should decrease down the chain (A > B > C)

  2. Where would you place a circuit breaker?

    • Hint: at the point where the failure originates (B → C)

  3. What response would B return when the circuit breaker is open?

    • Hint: a degraded response, cache, or a fast error

  4. How would you detect this problem before it affects users?

    • Hint: p99 latency metrics, percentile-based alerts
Show solution 
[Service A]           [Service B]               [Service C]
  timeout: 2s           timeout: 1s               ---
                        circuit breaker: ON
                        fallback: local cache
                        bulkhead: separate pool
  • Decreasing timeouts: A=2s, B=1s. If C takes longer than 1s, B cuts off immediately.
  • Circuit breaker on B→C: After 5 consecutive failures, B stops calling C for 30 seconds.
  • Fallback: When the circuit breaker is open, B responds with cached data (even if it’s stale).
  • Bulkhead: B uses a separate connection pool for C, so C’s failures don’t exhaust the connections available to other services.

Scenario 2: Overloaded database

The failure

It’s Black Friday. Traffic has multiplied by 5x. The catalog service’s PostgreSQL database is at 95% CPU. Queries that normally take 50ms now take 2 seconds. The connection pool is full.

Observed symptoms

  • API latency: p50 = 500ms (normal: 100ms), p99 = 8s (normal: 300ms)
  • Connection pool: 100/100 connections in use
  • PostgreSQL CPU: 95%
  • 503 errors on the frontend: 15% of requests

Exercise: Immediate response

You’re on call. What do you do in the next 30 minutes?

  1. Immediate action (first 5 minutes):

    • What can you do without changing code?

  2. Short-term mitigation (next 25 minutes):

    • What configuration changes would help?

  3. Future prevention:

    • What architectural changes would avoid this problem?
Show solution

Immediate (5 min):

  • Enable read-only mode for non-critical operations
  • Increase the connection pool size if the server allows it
  • Enable the read replica if it exists but isn’t in use

Short-term (25 min):

  • Enable a Redis cache for the most frequent catalog queries
  • Reduce the timeout for slow queries to free up connections
  • Scale up the PostgreSQL instance vertically (more CPU/RAM)

Future prevention:

  • Implement a read cache (Redis) as a permanent layer
  • Add read replicas to distribute the read load
  • Implement rate limiting to protect the database
  • Configure auto-scaling based on CPU/connection metrics
  • Review and optimize the most expensive queries (EXPLAIN ANALYZE)

Scenario 3: Lost message in the queue

The failure

The orders service publishes an OrderCreated event to RabbitMQ. The payments service consumes that event and processes the payment. But one day you discover that 50 orders from the last 24 hours were never processed — the event was published but the payments service never received it.

Investigation

What could have happened? Analyze each possibility:

  1. The message never reached RabbitMQ

    • Does the publisher use publisher confirms?
    • What happens if RabbitMQ doesn’t acknowledge the publish?

  2. The message reached RabbitMQ but was lost there

    • Is the queue durable?
    • Are the messages persistent?
    • Did RabbitMQ restart and lose in-memory messages?

  3. The consumer received the message but failed to process it

    • Does the consumer ACK before or after processing?
    • Is there a dead letter queue for messages that fail?

  4. The consumer was down when the message arrived

    • Does the queue retain messages when there are no consumers?
    • Is there a TTL that expires unconsumed messages?

Exercise: Design a resilient messaging system

[Orders Service]
    │
    ▼ (publisher confirms)
[RabbitMQ]
    │ (durable queue, persistent messages)
    ▼
[Payments Service]
    │ (ACK after processing)
    │
    ├── success → ACK
    └── failure → NACK → Dead Letter Queue → alert

Messaging resilience checklist

Show checklist
  • Publisher confirms: The publisher waits for confirmation from RabbitMQ before considering the message sent
  • Durable queue: The queue survives RabbitMQ restarts
  • Persistent messages: Messages are written to disk, not just memory
  • Manual ACK: The consumer ACKs only after processing successfully
  • Dead letter queue: Messages that fail N times go to a separate queue for investigation
  • Idempotency: The consumer can process the same message twice without duplicate effects
  • Monitoring: Alerts when the queue grows more than normal or there are messages in the DLQ
  • Retry with backoff: Failed messages are retried with increasing delays

Scenario 4: Cache split brain

The failure

You have a Redis cluster with 3 nodes (1 master + 2 replicas). During a network partition, a replica is promoted to master. Now you have 2 masters accepting writes with different data.

Impact

  • Some users see the old price of a product ($100)
  • Other users see the new price ($80)
  • When the partition is resolved, which data wins?

Exercise

  1. How would you detect a split brain in production?
  2. What resolution strategy would you use? (last-write-wins, merge, manual)
  3. How would you prevent this scenario?
Show solution

Detection:

  • Monitor the number of masters in the cluster (alert if > 1)
  • Sentinel or Redis Cluster detect and resolve it automatically

Resolution:

  • Redis uses last-write-wins by default when reconnecting
  • The “losing” master’s data is discarded
  • For critical data (prices, inventory), the source of truth should be the database, not the cache

Prevention:

  • Configure min-replicas-to-write so the master rejects writes if it doesn’t have a quorum
  • Use Redis as a cache (not as the source of truth) — always be able to rebuild from the DB
  • Implement health checks that detect network partitions

Reflection

After completing the scenarios:

  1. How many of these failures have you seen in production?
  2. Does your current system have protections against these scenarios?
  3. Which failure seems most likely in your context?
  4. What concrete action can you take tomorrow to improve your system’s resilience?

Systems don’t fail in new ways — they fail in the same ways over and over. Knowing the failure patterns is the best preparation.