The alerting setup on this platform has a rule that suppresses a warning-level alert automatically while a related critical-level alert is already firing for the same underlying condition. On-call gets paged once for the root cause, not twice for the same failure wearing two severity labels. It sounds like a small thing to write down as a rule, but it’s the kind of small thing that decides whether an on-call rotation trusts its own pager.
The problem this solves
Alert thresholds tend to come in pairs: a warning at 80% and a critical at 95%, or a warning when a check fails once and a critical when it fails three times in a row. That’s a reasonable way to define thresholds, for the metrics-based kind specifically; the same pairing doesn’t map cleanly onto log-based or uptime-based alerts, which fail in their own ways. It’s a bad way to define notifications, because when the underlying condition gets bad enough to cross both thresholds, both alerts fire, and without anything correcting for it, on-call gets paged twice for one problem, staggered by however long it took the metric to cross from warning into critical.
Two pages for one incident doesn’t just cost a few extra seconds of triage. It teaches the person holding the pager that the alerting system doesn’t model causality: that it treats “disk is 82% full” and “disk is 97% full” as two unrelated facts about the world rather than two readings of the same trend. Enough of that, and the natural response is to stop trusting individual pages at face value, which is the actual mechanism behind alert fatigue: not too much volume in the abstract, but too much volume that turns out, on inspection, to have been redundant.
Inhibition, not just routing
The fix isn’t a routing rule (send X to Slack, Y to PagerDuty), it’s suppression at the alert level, so the redundant notification never fires in the first place. Prometheus Alertmanager, probably the most widely deployed tool for this exact job, calls this mechanism an inhibition rule, and its documented behavior is close to a direct description of the problem: an inhibition rule mutes notifications for alerts matching a set of target matchers when another alert exists (the source alert) that matches a separate set of source matchers, provided a specified set of labels have identical values in both alerts (Alertmanager configuration docs).
Concretely: a rule with source_matchers: [severity="critical"] and target_matchers: [severity="warning"], correlated by equal: [alertname, cluster, service], means a critical alert for a given alertname/cluster/service combination suppresses the warning alert for that exact same combination, not warnings in general, only the one that’s describing the same underlying condition the critical alert is already describing. That equal list is the part that keeps the rule honest. Without it, a single critical alert somewhere could suppress unrelated warnings across the whole environment, which trades one kind of alert fatigue (redundant pages) for a worse one (silently dropped signal that had nothing to do with the thing that inhibited it).
The Alertmanager docs are also specific about an edge case worth knowing: a missing label and a label with an empty value are treated as the same thing for matching purposes, and an alert doesn’t inhibit itself even if it happens to match both the source and target side of a rule. Both of those are the kind of detail that only matters once — right up until a rule silently doesn’t fire the way you expected because a label was absent rather than empty, and then it matters a lot.
Verifying the rule does what it claims
An inhibition rule is config, and config that’s wrong doesn’t announce itself: it either fails to suppress something it should have (the redundant page still fires) or suppresses something it shouldn’t (a genuinely independent alert goes silent because it happened to share a label). Both failure directions are invisible until the moment they matter, which is exactly the wrong time to discover a mismatch between what the rule was supposed to do and what it actually does once real alerts start hitting it.
The reasonable defense is verifying the rule against realistic alert payloads before it’s relied on in production, not just reading the YAML and trusting that the matchers and equal list say what you think they say. That can be as direct as constructing a couple of representative firing alerts — one on the source side, one on the target side, sharing the labels the rule’s equal list expects — and confirming the target actually gets suppressed and, separately, confirming a similarly-shaped alert that differs on one of those equal labels does not get suppressed. The second check matters as much as the first: a rule that suppresses everything it’s pointed at isn’t obviously broken until you specifically test the boundary where it’s supposed to stop applying.
What this pattern actually decides
There’s a design choice buried in “suppress the warning when the critical is firing” that’s worth being explicit about: it decides that severity levels for the same underlying condition are not independent signals, they’re a timeline of one signal. Warning and critical aren’t two different problems that happen to correlate: they’re the same problem observed at two different points on its way to getting worse. Once that’s true, notifying on both is redundant by construction, and the honest fix is structural (inhibition) rather than behavioral (asking on-call to mentally dedupe two pages that arrive four minutes apart).
That’s different from other noise-reduction techniques that look similar from a distance. Grouping combines multiple alerts into one notification because they’re related but not necessarily causally connected — several pods crash-looping in the same deployment, say. Silencing is a manual, time-boxed override for known maintenance. Inhibition is neither of those: it’s an automatic, ongoing rule that encodes “if this is already true, that other thing isn’t new information.” It’s the one of the three that requires you to understand the causal relationship between two alert conditions before you can write the rule: you can’t inhibit correctly without first being honest about which alerts are really just louder versions of each other.
Where this generalizes past one config file
The interesting part isn’t Alertmanager’s specific YAML shape: it’s that this same reasoning applies to any alerting system with cascading severity thresholds, whether the mechanism is called inhibition, dependency-based suppression, or something else entirely. The underlying question is always the same one: when a condition gets bad enough to cross two thresholds, does the alerting system know those two crossings describe the same event, or does it treat them as two separate facts that happen to be about the same metric?
Getting that wrong in either direction has a cost. Suppress too aggressively (an overly broad equal list, or matchers that are too permissive on the source side) and a genuinely independent problem goes quiet because it happened to share a label with something else that was already firing. Suppress too narrowly, or not at all, and every threshold crossing becomes its own page, which is how a rotation ends up with a pager that fires constantly and gets ignored by reflex. The engineering work here isn’t turning inhibition on; it’s being precise about the equal labels, because that precision is the entire difference between “on-call trusts every page” and “on-call has learned to triage the pager before triaging the incident.”
Walking through the disk-full case
It’s easier to feel why this matters with a concrete (if illustrative, not a real figure from this platform) scenario rather than an abstract one: a warning-level threshold and a critical-level threshold defined against disk utilization on the same filesystem, both carrying alertname="DiskSpaceLow" plus a host and mountpoint label.
Without inhibition, a disk that fills up slowly (logs that aren’t rotating, say) crosses the warning threshold first and pages. Some time later it crosses the critical threshold and pages again. On-call now has two separate notifications for one continuously worsening condition, and has to do the correlation manually: is this critical alert the same problem as that earlier warning, or a second, unrelated thing that happens to have fired around the same time? Under a real incident, with several other things also happening, that correlation isn’t always obvious at a glance, and getting it wrong costs time exactly when time is expensive.
With an inhibition rule (source_matchers: [alertname="DiskSpaceLow", severity="critical"], target_matchers: [alertname="DiskSpaceLow", severity="warning"], equal: [alertname, host, mountpoint]), the warning for that specific host and mountpoint goes quiet the moment the critical for that same host and mountpoint fires. On-call gets exactly one notification, and it’s the more urgent of the two, which is also the one that needs a response. The warning didn’t vanish from the system: Alertmanager still knows it’s active, and it’ll un-suppress automatically if the critical resolves while the warning condition somehow persists. It just stopped generating a redundant notification while a more severe one about the identical condition was already open.
Maintaining the rule as thresholds move
Inhibition rules aren’t a write-once artifact any more than the alert thresholds they depend on are. If someone later retunes the warning threshold to catch a slow leak earlier, or splits DiskSpaceLow into per-filesystem-type variants, the equal labels and matchers need re-checking against the new shape of the alerts: a rule that was correctly scoped against the old label set can silently stop matching, or start matching too broadly, once the alerts underneath it change shape. That’s not a reason to avoid inhibition; it’s a reason to treat the inhibition config as part of the same review whenever alert definitions themselves get revised, rather than a separate piece of config that gets set up once during initial rollout and then forgotten until something unexpectedly pages twice again.
Sources: