Reducing alert fatigue: a practical guide for on-call teams

Alert fatigue is usually described as a discipline problem — engineers get desensitised, start ignoring pages, and eventually miss a real one. That framing puts the fix on the individual. In practice, alert fatigue is almost always a design problem in how alerts are generated and routed, and the fix belongs in the system, not in asking people to pay closer attention to noise.

Start by pulling three months of paging history and classifying every alert as actionable or not. Actionable means a human did something in response beyond acknowledging it. In most systems we have seen, well under half of all pages meet that bar — the rest are duplicate signals from the same underlying failure, transient blips that self-resolved, or checks tuned so sensitively that they fire on normal variance.

The single highest-leverage fix is grouping. A bad deploy that spikes latency, error rate and queue depth simultaneously should generate one incident with one page, not three or four separate alerts competing for the same engineer’s attention at 3am. Grouping related signals into a single incident is what actually moves the "pages per week" number, more than tuning any individual threshold.

The second fix is replacing static thresholds with baselines. A fixed "alert if error rate exceeds 2%" rule is either too sensitive during a genuine traffic spike or too slow during a genuine quiet-period failure. A baseline that learns a service’s normal pattern by day of week and time of day catches real anomalies faster and false-positives far less often than any hand-tuned static number.

Third, audit your routing, not just your thresholds. Alerts that page the wrong team, or page a team for an issue they cannot act on without another team’s involvement, train people to ignore pages regardless of how well-tuned the underlying signal is. Every alert should route to whoever can actually do something about it, first time.

Finally, measure the right thing afterward. "Pages per on-call shift" is a better fatigue metric than "alerts fired," because it reflects what a human actually experiences. Teams that get this number down — through grouping, baselining and correct routing rather than by simply deleting alerts — see a corresponding drop in missed or delayed responses to the pages that remain.