Operational Resilience

Chaos engineering, disaster recovery, backups, runbooks, and practices for keeping the system running in the face of failures.

What is operational resilience?

Operational resilience is a system’s ability to keep running (or recover quickly) in the face of failures, errors, and unexpected situations. It’s not about preventing failures — in distributed systems, failures are inevitable — but about designing the system to tolerate them.

Operational resilience covers:

• Failure prevention through robust design
• Rapid problem detection
• Automatic or assisted recovery
• Continuous learning from incidents

Chaos engineering

Chaos engineering is the practice of deliberately injecting failures into the system to verify that it can handle them. The premise is simple: if you don’t test for failures in a controlled environment, you’ll discover them in production.

Principles

1. Define the steady state: what does the system look like when it’s working well? (metrics, latency, error rate)
2. Formulate a hypothesis: “If service X fails, the system should degrade gracefully”
3. Inject the failure: simulate the failure condition
4. Observe the impact: did the system behave as we expected?
5. Learn and improve: document findings and fix weaknesses

Types of experiments

• Terminating instances: what happens if a pod dies suddenly?
• Injecting latency: what happens if a downstream service responds slowly?
• Simulating network errors: what happens if the connection between services is lost?
• Filling up the disk: what happens if storage runs out?
• Saturating CPU/memory: what happens under extreme resource pressure?

Tools

• Chaos Monkey (Netflix): randomly terminates instances in production
• Litmus Chaos: chaos engineering framework for Kubernetes
• Gremlin: commercial chaos engineering platform
• Chaos Mesh: open source tool for Kubernetes
• Toxiproxy: proxy for simulating adverse network conditions

Best practices

• Start in staging environments before production
• Begin with small, controlled experiments
• Always have a rollback plan
• Involve the team — chaos engineering is a team exercise
• Document every experiment and its results

Disaster recovery

Disaster recovery (DR) is the plan for recovering the system after a catastrophic failure: loss of a datacenter, massive data corruption, or a security attack.

Key metrics

• RTO (Recovery Time Objective): maximum acceptable time to restore service
• RPO (Recovery Point Objective): maximum amount of data that can be lost (measured in time)

Disaster recovery plan

1. Identify critical systems: which services are essential to the business?
2. Define RTO and RPO: for each critical system
3. Design the strategy: backups, replication, multi-region
4. Document the process: step by step for recovery
5. Test regularly: an untested plan is a plan that doesn’t work
6. Update the plan: every change in the architecture can affect DR

Backups

Backups are the last line of defense against data loss.

Types of backup

• Full backup: complete copy of all data
• Incremental backup: only the data that changed since the last backup
• Differential backup: the data that changed since the last full backup

3-2-1 rule

• 3 copies of the data
• 2 different storage types
• 1 offsite copy (in another geographic location)

Best practices

• Automate: manual backups get forgotten
• Verify: regularly test that backups can be restored
• Encrypt: backups contain sensitive data
• Retention: define clear policies for how long they’re kept
• Monitor: alert if a backup fails

Example backup policy

Primary database:
- Full backup: daily at 02:00 UTC
- Incremental: every 1 hour
- Retention: 30 days for daily, 7 days for incremental
- Storage: S3 with cross-region replication
- Verification: automatic weekly restore in test environment

Runbooks

A runbook is a document that describes step by step how to respond to an incident or perform an operation. It’s the difference between a chaotic response and an orderly one.

Structure of a runbook

1. Title and description: which incident or operation it covers
2. Symptoms: how the problem is detected (alerts, metrics, logs)
3. Impact: what is affected and how serious it is
4. Diagnosis: steps to confirm the root cause
5. Resolution: exact steps to resolve the problem
6. Verification: how to confirm the problem is resolved
7. Post-mortem: what to document after the incident

Example: runbook for high API latency

## High latency in API Gateway

### Symptoms
- Alert: p99 latency > 2s for 5 minutes
- Dashboard: latency chart in red

### Diagnosis
1. Check CPU/memory metrics of the pods
2. Review recent error logs
3. Check latency of downstream services
4. Check database connections

### Resolution
If CPU > 90%:
  → Scale horizontally: kubectl scale deployment api --replicas=X

If DB connections saturated:
  → Check connection pool settings
  → Restart pods if there are connection leaks

If downstream service slow:
  → Check the service's health
  → Activate circuit breaker if necessary

### Verification
- Confirm that p99 latency < 500ms
- Verify that error rate < 0.1%
- Monitor for 15 minutes

Incident management

Incident response process

1. Detection: automatic alerts or manual report
2. Triage: assess severity and impact
3. Communication: notify stakeholders
4. Investigation: diagnose the root cause
5. Mitigation: restore service as soon as possible
6. Resolution: apply the definitive fix
7. Post-mortem: document what happened, why, and how to prevent it

Severity levels

Blameless post-mortem

After every significant incident, conduct a blameless post-mortem:

• What happened: detailed timeline of the incident
• Why it happened: root cause analysis (5 whys)
• What we did well: what worked in the response
• What we can improve: concrete actions to prevent recurrence
• Action items: assigned tasks with owners and dates

Summary

Operational resilience is not optional in distributed systems. Chaos engineering validates that the system tolerates failures, disaster recovery prepares for the worst, backups protect the data, runbooks guide incident response, and incident management organizes recovery. The key is to practice, document, and continuously improve.