izebra_projects/fan_speed
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Alarm on cooling that cannot keep up, not on load

Browse Source 
A rising temperature is what a busy machine looks like; the previous
trend alarm would have mailed continuously through a multi-hour zpool
migration. trend_verdict() now alarms only when the fans are already at
maximum and it is still climbing, or when temperature and fan speed are
both rising and the rise is not decelerating.

Tells a plateau from a runaway by comparing the first half of the window
against the second - a load step settles once the fans catch up, a
failing fan does not. Verified quiet against a live migration at load
average 18 with CPU at 68c.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Zeb Hering
2026-08-28 22:00:17 -07:00
parent 86f3b0c323
commit 80fc40a1ff
3 changed files with 93 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
README.md
View File

@@ -92,14 +92,26 @@ Email via `mail` to `ALERT_EMAIL`, which the host's postfix relays. Four conditi
| Alarm | Fires when |
|---|---|
| `disk_temp` | A drive is within `DISK_ALARM_OFFSET` (5c) of its own limit |
| `trend` | CPU or hottest disk has climbed `TREND_RISE_ALARM` (8c) across the window while still under every threshold |
| `trend` | Temperatures climbing **and the fans cannot keep up** — see below |
| `smart` | `smartctl -H` reports anything other than PASSED/OK |
| `cpu` / `gpu` | Threshold crossed; `cpu` also means fan control was handed back to Dell's profile |
| `disk_count` | Fewer drives answered than were present at startup |
**The trend alarm is the one worth having.** A dying fan or a blocked intake shows up as
a steady climb long before anything crosses a threshold — by the time an absolute alarm
fires you have already been running hot for hours.
**The trend alarm is the one worth having**, but a rising temperature is not by itself a
fault — it is what a busy machine looks like, and a multi-hour zpool migration will do it
all afternoon. What matters is whether the fans are answering. `trend_verdict()` returns
one of three things:
| Verdict | Meaning | Alarms |
|---|---|---|
| `no_headroom` | Climbing while the fans are already at `HIGH_FAN_SPEED` | yes — nothing left to give |
| `not_converging` | Climbing, fans ramping with it, and the rise is *not slowing down* | yes — cooling is losing ground |
| `quiet` | Anything else, including a large climb the fans absorbed and that has plateaued | no — that is just load |
It tells a plateau from a runaway by comparing the first half of the window against the
second: a normal load step decelerates once the fans catch up, a failing fan or a blocked
intake does not. A dying fan still gets caught long before an absolute threshold, without
mailing you every time the machine gets busy.
Every alarm is **rate limited per key** with a one hour cooldown, and sends a single
recovery notice when it clears. On a 10s loop an un-throttled alarm sends 360 emails an