Alerting
Thresholds, SLOs, SLIs, on-call, alert fatigue, escalation and how to design an effective alerting system.
Why does alerting matter?
Having metrics and dashboards is useless if nobody looks at them when something breaks. Alerting is the mechanism that turns observability data into action: it notifies the right people when something requires attention, before users are affected.
But a poorly designed alerting system can be worse than having no alerts at all. If the team receives 50 alerts a day that require no action, they will eventually ignore all of them — including the critical ones.
SLIs, SLOs and SLAs
SLI (Service Level Indicator)
An SLI is a quantitative metric that measures one aspect of the service from the user’s perspective:
- Availability: Percentage of successful requests.
- Latency: Percentage of requests completed in under X ms.
- Throughput: Requests processed per second.
- Correctness: Percentage of responses with correct data.
SLO (Service Level Objective)
An SLO is an internal target for an SLI. It defines the level of service the team commits to maintaining:
- “99.9% of requests must complete in under 500ms.”
- “Monthly availability must be at least 99.95%.”
- “99% of payments must be processed in under 3 seconds.”
SLA (Service Level Agreement)
An SLA is a formal contract with consequences (usually financial) if it isn’t met. SLAs are typically less strict than internal SLOs to leave a margin of safety.
Error Budget
The error budget is the difference between 100% and the SLO. If your availability SLO is 99.9%, your monthly error budget is 0.1% — roughly 43 minutes of allowed downtime.
The error budget enables informed decisions:
- If there’s plenty of budget left: you can prioritize features and development speed.
- If the budget is running out: prioritize stability and reduce risk.
Alert design
Symptom-based vs cause-based alerts
- Symptom-based alerts (recommended): “The 5xx error rate exceeds 1%.” They detect user impact.
- Cause-based alerts: “CPU usage exceeds 90%.” These can generate false positives — high CPU doesn’t always mean trouble.
Prioritize symptom-based alerts. Use cause-based alerts only as a complement for diagnosis.
Static vs dynamic thresholds
Static thresholds: Fixed values defined manually.
- Advantage: Simple to understand and configure.
- Disadvantage: They don’t adapt to variable traffic patterns.
Dynamic thresholds: Calculated automatically based on historical patterns.
- Advantage: They adapt to normal variations (more traffic during business hours).
- Disadvantage: More complex to implement and can generate false positives during legitimate changes.
Severities
Define clear severity levels with associated actions:
| Severity | Meaning | Action | Example |
|---|---|---|---|
| Critical | Direct user impact | Immediate notification, on-call | Service down, errors > 5% |
| Warning | Potential degradation | Notification during business hours | Elevated latency, disk at 80% |
| Info | Situation to monitor | Log to team channel | Deploy completed, scaling event |
Evaluation windows
Don’t alert on instantaneous spikes. Use time windows to avoid false positives:
for: 5m— The condition must hold for 5 minutes before the alert fires.- Longer windows for noisy metrics, shorter ones for critical metrics.
Alert fatigue
What is it?
Alert fatigue occurs when the team receives so many alerts that it stops paying attention to them. It’s one of the most common and dangerous problems in operations.
Signs of alert fatigue
- The team silences alerts without investigating them.
- Alerts pile up without being addressed.
- Real incidents are detected through user complaints, not through alerts.
- The on-call team is exhausted and demotivated.
How to fight it
- Every alert must be actionable: If it doesn’t require human action, it’s not an alert — it’s a log or a metric.
- Periodic review: Review your alerts monthly. Remove the ones that didn’t lead to any useful action.
- Grouping: Group related alerts to avoid cascades of notifications.
- Deduplication: If the same alert fires 10 times in 5 minutes, send a single notification.
- Runbooks: Every alert must have an associated runbook that explains what to do.
On-call
Rotations
An effective on-call system includes:
- Weekly rotations: Distribute the load across the whole team.
- Escalation: If the primary on-call doesn’t respond within X minutes, escalate to the secondary.
- Compensation: On-call must be compensated — it’s work outside regular hours.
Escalation
Define a clear escalation chain:
- Level 1: On-call engineer of the team that owns the service.
- Level 2: Tech lead or senior engineer of the team.
- Level 3: Incident commander / management.
Escalation timings depend on severity:
- Critical: Escalate after 15 minutes without a response.
- Warning: Escalate after 1 hour without a response.
On-call tools
- PagerDuty: Leading platform for incident and on-call management.
- OpsGenie (Atlassian): Alternative with good Jira integration.
- Grafana OnCall: Open-source solution integrated with Grafana.
Runbooks
A runbook is a document that describes the steps to follow when a specific alert fires.
Structure of a runbook
- Description: What this alert means.
- Impact: What the user experiences.
- Diagnosis: Steps to investigate the root cause.
- Mitigation: Immediate actions to restore the service.
- Resolution: Steps to resolve the root cause.
- Contacts: Who to escalate to if it can’t be resolved.
Automating runbooks
When a runbook has repetitive, well-defined steps, consider automating them:
- Diagnostic scripts that collect information automatically.
- Automatic mitigation actions (service restart, scaling).
- Bots that run the first steps of the runbook and report the result.
Best practices
Start simple
Don’t try to alert on everything from day one. Start with the RED metrics for each critical service and expand gradually.
Test your alerts
Alerts are code — they should be tested:
- Verify that they fire when the condition is met.
- Verify that they don’t fire on normal variations.
- Simulate failures periodically to validate the full chain.
Post-mortems
After every significant incident, run a post-mortem:
- Did the alert fire in time?
- Was the runbook useful?
- What can be improved?
Use post-mortems to continuously improve the alerting system.
Summary
An effective alerting system is the bridge between observability and action. Base your alerts on SLOs, prioritize symptoms over causes, actively fight alert fatigue and make sure every alert has an associated runbook. The goal is not to have more alerts, but to have the right alerts that let the team keep the service healthy.