SRE Alerting Best Practices: Avoiding Alert Fatigue

The fastest way to make monitoring useless is to page someone on the very first failed check. It feels safer -- you never miss anything -- but it trains whoever receives the alert to assume it's probably nothing, because most of the time, it is. Once that happens, the one alert that actually matters gets the same shrug as the hundred before it.

Why single-check alerting fails

A single failed HTTP request can mean the server is down -- or it can mean a router somewhere between the monitor and the server dropped one packet. A single poor performance score can mean the site regressed -- or it can mean a cold cache on that one run. Neither of these is distinguishable from a single data point. Alerting on it anyway is a coin flip dressed up as vigilance.

The fix: require a streak, not a blip

The rule that actually works, and the one we use for SREmonitor's own alerts, is simple: don't alert until a check has failed several times in a row. Three consecutive failures is a reasonable default -- enough to rule out a one-off network blip, not so many that a genuine outage runs half an hour before anyone hears about it. The moment the streak breaks, the alert should say so too -- a "recovered" notification closes the loop without anyone needing to go check manually.

This is a stateless rule: look at the last four data points, and check whether the most recent three all breached while the one before that didn't. No counters to maintain, no state to get out of sync -- just a pattern in the history you already have.

What a genuinely useful alert includes

An alert that just says "something is wrong" sends the recipient straight into investigation mode from zero. A better alert front-loads what you'd otherwise have to go looking for:

• What actually breached and by how much, against what threshold.
• A link straight to the failing run -- not just the dashboard, the specific check that triggered it.
• A screenshot, if the check produces one -- seeing what actually rendered is faster than guessing from a number.
• A short list of likely causes for that specific metric, so triage starts somewhere concrete instead of nowhere.

Recipients: default sensibly, let people override

The simplest setup that still works for a team is to default alerts to the account owner's own email -- so it works with zero configuration -- while allowing an override to a shared address for teams that want alerts to land in a distribution list instead of one person's inbox. Either way, the address is just a delivery destination; it shouldn't require creating a new login just to receive a notification.

A short checklist

1. Require a consecutive-failure streak before firing an alert -- three is a reasonable default.
2. Always send a matching "recovered" notification when the streak breaks.
3. Include the specific failing value, the threshold, and a direct link to that run.
4. Attach a screenshot where one exists, and a short list of likely causes.
5. Default the recipient sensibly, but make it easy to point somewhere else.

This is exactly how alerting works across uptime checks and performance checks in SREmonitor. For broader writing on SRE and platform engineering practices, see Platform Signals.