A Change in Log Rate Is the Alert

A lot of monitoring is wired to the wrong question. It watches for whether a particular error line appears in the logs, and pages when it does. That sounds reasonable and it’s usually useless, because the errors worth caring about are rarely binary. A single “queue full, retrying” is Tuesday. Fifty of them in a minute is an incident. Presence-based alerting can’t tell those apart — so it either screams at every harmless blip until people mute it, or it’s tuned so loose it never fires. The signal that actually carries meaning is the rate.

Presence is a coin toss between noise and silence

Alert on “does this line exist,” and you’re stuck picking a bad failure mode. Rare-but-normal events — a transient retry, a slow query, a reconnect — appear in healthy systems all the time. Page on every occurrence and you train everyone to ignore the pager, which means you’ve built an alarm people sleep through. Tune it to only fire on something dramatic and you miss the slow build. Neither setting works, because the underlying question is wrong. “Did it happen?” has almost no information in it. “Is it happening more than normal?” has all of it.

One error is a data point. A hundred errors in a minute is a story. Only one of those is worth waking someone up for, and presence-based alerting can’t tell which is which.

Rate turns background noise into a threshold

Watching frequency over a window changes the whole picture. Count how many times the event happens in, say, the last minute, and alert when that count crosses a threshold. Now the harmless trickle stays quiet — it never crosses the line — and the flood, the sudden burst that actually indicates something broke, trips the alert cleanly. You’ve stopped asking “did this ever happen” and started asking “did the rate of this just change,” which is the question a human would actually ask reading the logs over your shoulder.

The same idea catches the opposite failure, too: a line that normally appears steadily and then stops is often as meaningful as a spike. A heartbeat that goes quiet, a stream of successful transactions that flatlines. Rate monitoring sees the drop as readily as the surge, where presence monitoring is blind to “it used to be here and isn’t.”

Rate crossing a threshold is a good trigger for a first response

Once you’re measuring rate, the threshold-crossing becomes a natural place to hang an automated first response before you escalate to a human. The pattern I like: when the failure rate crosses the line, fire a remediation step, then run a check a few times with a short wait between attempts — and only if it still hasn’t recovered, page a person. Something recovered on its own, or by the automatic nudge? No 3am alert for a problem that’s already gone. Still broken after the automatic attempt? Now a human’s attention is warranted, and they arrive to a real problem instead of a blip that self-healed while they found their laptop.

That structure respects the scarcest resource in operations — human attention — by spending it only on the failures that are both sustained and resistant to the easy fix. It’s the rate-based cousin of only surfacing what needs a decision, and it keeps the pager meaningful.

Rate needs structured, countable events

None of this works if you can’t reliably count the thing. Rate monitoring assumes you can match “this specific event” cleanly and tally it, which is a strong argument for logging in a form you can machine-count rather than free-form prose — the case I make in structured logs beat clever sentences. If every occurrence of the event is phrased three different ways, your count is wrong and your threshold is meaningless. Consistent, matchable log events are what make “how often” answerable at all.

It also pairs with treating recurring noise as a signal in its own right. A specific line that shows up at a steady low rate isn’t random — it’s the system telling you about a condition that’s always mildly present, which is the mindset behind false positives being a pattern. Rate monitoring is how you notice when that steady background hum suddenly gets loud.

Watch the derivative, not just the value

The habit that generalizes past logs: for a lot of signals, the change is more informative than the level. A steady stream of retries is fine; a sudden acceleration of them is the story. So when I set up monitoring now, I try to ask not just “is this bad thing present” but “is this thing happening at a rate that’s different from normal,” because that difference is almost always where the incident actually lives. Presence tells you something exists. Rate tells you something changed — and change is what you actually want to be paged about. If you’ve replaced a noisy presence alert with a rate-based one and finally trusted your pager again, I’d like to hear how you tuned it.