The Alert That Cried Wolf: Designing Monitoring People Don't Mute

The most dangerous state for a monitoring system is not broken. Broken gets fixed. The dangerous state is noisy, because noise gets muted, and muting is invisible. The channel is still there, the emails still send, the dashboard still updates, and the humans have quietly stopped believing any of it. Then the real failure arrives, formatted identically to the five hundred false alarms before it, and it gets the treatment the system has trained everyone to give: a glance, a sigh, a filter.

The failure mode is real, and it is a design problem before it is a discipline problem: human wiring meeting bad monitoring. Joint Commission Sentinel Event Alert 50 puts the share of hospital alarm signals requiring no clinical intervention between 85 and 99 percent, and the estimate usually cited for critical care sits near nine in ten. Alarm management became National Patient Safety Goal NPSG.06.01.01 the same year, because tuning out constant stimulus is what brains do.

The clinical literature also names the mechanism, and the mechanism transfers. Alarms are built deliberately high-sensitivity and low-specificity, because missing a real event is treated as worse than generating another false one. That asymmetry manufactures noise on purpose. The same asymmetry operates on anyone who has ever added a threshold alert because missing the event felt worse than sending a hundred false ones.

Security operations reports the same shape in a different building, and one of the reasons overlaps. The ACM Computing Surveys review of alert fatigue in security operations centres sorts the contributing pressures into four, and one of them is the one the device manufacturers hit from the other direction: detection tooling tuned to minimise false negatives produces excessive false positives. The other three are alert volume, alert velocity, and the drift of the threats themselves. Volume is a capacity problem and an organization can hire against it. Tuning is a signal-quality problem, and hiring reduces what it costs without fixing what causes it.

The canonical case is Target's 2013 breach, and it is worth stating carefully, because the popular retelling has the mechanism wrong. The version that circulates says the alerts were buried in routine noise. The Senate Commerce Committee's kill-chain analysis says something else: the FireEye system triggered urgent alerts with each installation of the exfiltration malware, contemporary reporting describes each as the most urgent on the tool's graded scale, and the security team neither reacted to them nor let the software delete the malware automatically. They carried top severity and were ignored anyway.

What the record does support is worse for anyone who builds alerting. The alerts carried maximum severity and almost no information, naming unfamiliar malware under a generic label. Severity was turned all the way up. Separately, the automatic eradication feature had been switched off, and one reading offered at the time attributes that to a team still building confidence in letting the software act on its own. That reading is an inference and should be held as one. If it holds, the thesis at the end of this article names the proximate mechanism in that case.

I have watched the small-scale version of this play out repeatedly, and it shaped every alerting decision in my connectivity validator. Those decisions generalize, so here they are as design principles rather than config options. The sections after them take three of the six apart.

Alert on persistence, not on blips. A single failed probe is weather. The validator requires a configurable number of consecutive failures before anyone hears about it, because transient flaps are constant in real networks, and a system that pages on every one is a system training its audience that pages mean nothing. The cost is honest and bounded: detection latency of threshold times interval. The benefit is that when the email arrives, it has already survived a filter, and the reader knows it.

Alert once per incident, then announce recovery. An outage that generates one email is information. An outage that generates one email every five minutes for a weekend is 576 copies of the same fact, and by Monday the sender is in a filter rule forever. Alert on the transition into failure, stay quiet during it, and send exactly one recovery notice, because the recovery message is what lets a human stop thinking about the incident, and a system that never says "resolved" leaves everyone carrying open loops or, worse, assuming silence means fixed.

Make the payload actionable or do not send it. The test for every alert: can the recipient do something with this, right now, that they would not have done otherwise? An actionable payload names what failed, where, since when, what the last observed detail was, and where the full log lives. "Check failed on host X" fails the test. If no action exists, the event is a log line or a weekly digest rather than an alert. Demoting it costs no information and protects the channel's credibility for the events that need it.

The delivery path must not share fate with the thing being monitored. The alert about the mail relay cannot travel through the mail relay, argued at length in the validator and dispatcher writeups. A monitoring system that goes silent precisely during failures has the same observable behavior as a healthy environment, and that ambiguity is the whole disease.

Triage what deserves to interrupt a human at all. This is the meta-decision above the others. Every alerting system drifts toward more alerts, because adding one is always locally reasonable and removing one requires someone to accept risk. The counterweight is a periodic, deliberately unkind review: for each alert type, when did this last fire, was action taken, and would anything bad have happened on a one-day delay? Alerts failing that review get demoted to logs. A short list of trusted alerts outperforms a long list of ignored ones by a margin that is hard to overstate, because the value of an alert is the response it reliably produces rather than the mere fact of its existence, and response is a budget that noise spends.

Suppress planned work, or the channel gets muted for it. Every principle above assumes the failure is unexpected. A patching window produces exactly the signals an emitter escalates: services down, ports refusing, mounts absent. All true, all correct, none of them worth a human's attention. The engineer's response is to mute the channel for the window, which is rational and is also the disease, because what they have created is a scheduled interval during which no failure of any kind is visible. Unrelated failures land inside that interval at the same rate they land outside it. The blind spot arrives on a schedule. It is a calendared outage of the monitoring system, booked for the hours when the infrastructure is being changed and is therefore most likely to break.

What alert-once actually costs

Alert-once is the only one of the six that forces the emitter to hold state, and state is where the cost lives. A stateless check cannot alert once, because it has no way to know whether it has already spoken. It has to remember three things across runs: whether an alert is currently open, how many consecutive failures it has seen, and how many consecutive successes. The connectivity validator keeps that in a file per target.

The thresholds in that state are two numbers, deliberately unequal: three consecutive failures to open an alert, ten consecutive successes to close it.

Symmetric thresholds are what most people write first, and they fail on exactly the case this exists to handle. A service that alternates up and down every few minutes satisfies both thresholds repeatedly and produces an alert and a recovery notice on every cycle, until the recovery channel is as noisy as the alert channel and muting one means muting both. Requiring sustained health to close an alert costs latency on the recovery notice, which is the cheap direction to be slow in.

Cooldown windows are the common alternative and they fail differently. A sixty-minute cooldown on a ninety-minute outage sends two alerts for one event.

Recovery stops being a special case once the state exists. The same comparison runs the other way, which is why an emitter that tracks state gets recovery notices for free while one that only tests a threshold has them bolted on afterward.

The expensive part is keeping the count correct under concurrency, and a monitoring script is scheduled precisely so that it runs while things are going wrong. A probe that hangs past its own interval means the scheduler starts a second copy, both copies read the same counters, and the one that finishes last overwrites the other's increments. The failure is silent and it biases toward undercounting, so the alert that should have fired on the third failure fires on the fifth, or not at all. Serializing the read, probe and write behind a lock is what separates a counter from a guess, and it is the same reason the mail dispatcher takes an exclusive lock before it touches its spool.

Two smaller things follow. State has to be written on every run, including the quiet ones, because an emitter that only records state when something happened cannot tell a first failure from a fortieth. And it cannot live anywhere that clears on reboot, because an emitter that forgets during a restart re-alerts for every incident already open, which is precisely when the channel can least afford it.

The payload test, applied

The difference between a payload that passes the actionability test and one that fails it is small enough on the page to be worth showing.

printf 'Check failed on host %s\n' "$target_name"

printf 'FAILING: %s\n' "$target_name"
printf 'Layer:   TCP connect to port %s\n' "$service_port"
printf 'Since:   %s (%d consecutive failures)\n' "$first_failure_time" "$failure_count"
printf 'Last OK: %s\n' "$last_success_time"
printf 'Detail:  %s\n' "$last_error_detail"
printf 'Log:     %s\n' "$log_path"

Both are one alert about one event. The first tells the reader that something they cannot identify happened at a time they cannot determine for a reason they cannot see. Every question it raises needs a login to answer, so what arrived is an assignment.

The second answers the questions the reader was going to ask anyway. The layer tells them whether this is DNS, routing, or an application. The duration tells them whether to act now or at the start of the shift. The last-success timestamp brackets the change window without anyone opening a terminal. The detail line frequently ends the investigation on its own.

The cost of the second is the emitter tracking a few variables it already has in scope. The cost of the first is a login at 3am, which is where the decision to stop bothering gets made.

When the delivery path stays correct and goes silent

The fourth principle usually gets stated as a network claim: do not route alerts about the mail relay through the mail relay. That framing is too narrow, and the dispatcher is where I found out how narrow.

It spools any message it cannot deliver and retries on a timer. A message that keeps failing eventually retires to a dead-letter directory, and a status command returns a non-zero code while any remain. That status code is the operator's only signal that something has stopped being deliverable.

The flush loop started with two failure classes. A connection-level failure means the relay is unreachable, which is one condition affecting every queued message, so the run stops rather than burning an attempt on all of them. A message-level rejection is specific to one message, so that message is charged an attempt and the run continues.

except smtplib.SMTPException:
    attempt_count += 1
except OSError:
    break

except OSError:
    break
except smtplib.SMTPException:
    attempt_count += 1

smtplib.SMTPException subclasses OSError. In the second form the broad handler shadows the specific one, so every message-level rejection is reclassified as a connection failure. The loop breaks instead of charging an attempt. The counter never rises. Nothing ever reaches the dead-letter directory. The status command keeps reporting "pending," and a permanently undeliverable message retries forever.

None of that is visible from outside. The tool's stated promise, that a message is delivered or on disk, holds exactly. The spool is intact, the process exits cleanly, the timer keeps firing. What is gone is the operator's only signal that a message has stopped being deliverable at all.

That block is the collision reduced to the two handlers that cause it; the shipped loop names specific connection-level exceptions ahead of both.

Fixing the order exposed a second question underneath it, and the original answer to that one was also wrong. My taxonomy had two classes in it. The relay offers three. SMTPAuthenticationError, SMTPHeloError and SMTPNotSupportedError all subclass SMTPException, so a stale credential, a rejected greeting, and a relay declining STARTTLS were all arriving in the message-rejection branch. Each of those is a property of the session, identical for every message in the spool, and each was being charged to one message at a time. A flush run against a relay with a wrong password therefore walked the entire queue, charged every message an attempt, and exited reporting a clean run. At the shipped ceiling of twenty attempts on a five-minute timer, one wrong password retires every alert on the host to dead letters in about ninety-five minutes. Naming those three alongside the connection-level exceptions fixes it, and costs one line. Finding it took building a queue, failing it deliberately, and reading what the spool looked like afterward.

A third shape was sitting one line above both of them. Each spooled record got read before the try block that guards delivery, so a record that failed to parse raised through the loop and ended the run. Every message behind it in sort order stayed spooled, and the next timer fired into the same record and died the same way. It exited 1, which is the code the tool documents for an ordinary non-empty spool. A spool that had stopped draining forever and a spool with three messages in it were the same observation from outside, on hosts that have no mail agent to carry the traceback anywhere.

So the principle reaches past network paths. The delivery path shared fate with an exception hierarchy rather than with the monitored system, and the observable behavior of that failure matches the observable behavior of health. Two lines in the wrong order, and no amount of testing the network path would have found it.

Making the window cheap enough not to mute

The sixth principle is an obligation on the emitter: make suppression cheap enough that nobody reaches for the mute button. Three properties decide whether it works.

Suppression expires on a timestamp rather than a toggle. A mute someone has to remember to undo is a mute that outlives the window, and an alerting system silenced indefinitely by a forgotten toggle is indistinguishable from a broken one.

Suppression holds the alert and lets the probe keep running. That order is easy to get backwards, and getting it backwards costs the thing the window was supposed to protect. If the emitter skips its probe entirely while suppressed, its counters sit at zero, and a service that broke during the change window starts counting from scratch when suppression lifts. The full persistence threshold then gets added on top of the maintenance window before anyone hears anything. Probing and counting continue; only the alert is held. Then a service still down when the window closes alerts on the very next run rather than three runs later.

And suppression is scoped to one emitter, or it is the mute button with extra steps. Silencing one check for twenty minutes is a maintenance window. Silencing the channel schedules the failure mode.

The suppressed runs still log. That is what makes the question afterward answerable: did this break during the change, or was it already broken when we started?

Two layers sit above these six, and a single tool cannot reach them, but leaving them unnamed would be its own kind of dishonesty. The first is severity. Surviving triage earns an alert a channel, and only some of those channels should wake anybody; mature systems route by consequence, a wake-someone-up path for the events that cannot wait and a quieter channel for the ones that hold until morning. The second is escalation: a page that goes unacknowledged has to travel somewhere next, or it becomes another silent failure wearing the costume of a sent message. Both belong to the alerting platform and the on-call rotation around it rather than to any single emitter, and the six principles here are what a well-behaved emitter owes that larger system rather than a replacement for it.

An alerting system's real output is trust: accumulated slowly by relevance, spent instantly by noise. Every design choice above protects that balance. The wolf story usually gets told as a lesson about the boy, but for the people who build the systems doing the crying, it is a lesson about how few false alarms a village needs before it stops coming, and about whose job it was to make the crying mean something.

Where this evidence comes from

• The Joint Commission. Sentinel Event Alert, Issue 50: Medical Device Alarm Safety in Hospitals. 8 April 2013. https://www.jointcommission.org/resources/sentinel-event/sentinel-event-alert-newsletters/
• The Joint Commission. National Patient Safety Goal NPSG.06.01.01, Improve the Safety of Clinical Alarm Systems. Issued June 2013; EP 1 effective 1 July 2014.
• Tariq, S., Baruwal Chhetri, M., Nepal, S., and Paris, C. Alert Fatigue in Security Operations Centres: Research Challenges and Opportunities. ACM Computing Surveys 57, 9, Article 224 (April 2025). https://doi.org/10.1145/3723158
• U.S. Senate Committee on Commerce, Science, and Transportation, Majority Staff Report. A "Kill Chain" Analysis of the 2013 Target Data Breach. 26 March 2014.
• Python Software Foundation. smtplib, SMTP Protocol Client. Python 3 standard library documentation. https://docs.python.org/3/library/smtplib.html

Sources verified 2026-09-11.