izebra_projects/fan_speed
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Alarm on cooling that cannot keep up, not on load

Browse Source 
A rising temperature is what a busy machine looks like; the previous
trend alarm would have mailed continuously through a multi-hour zpool
migration. trend_verdict() now alarms only when the fans are already at
maximum and it is still climbing, or when temperature and fan speed are
both rising and the rise is not decelerating.

Tells a plateau from a runaway by comparing the first half of the window
against the second - a load step settles once the fans catch up, a
failing fan does not. Verified quiet against a live migration at load
average 18 with CPU at 68c.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Zeb Hering
2026-08-28 22:00:17 -07:00
parent 86f3b0c323
commit 80fc40a1ff
3 changed files with 93 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

48
monitor.sh
View File

@@ -258,9 +258,55 @@ record_sample() {
fi
}
# Is the cooling losing? A rising temperature on its own is not a fault - it is
# what a busy machine looks like, and a zpool migration will do it for hours.
# What matters is whether the fans are answering it:
#
# no_headroom temperature climbing while the fans are already flat out.
# Unambiguous: there is nothing left to give.
# not_converging temperature climbing, fans climbing with it, and the rise is
# not slowing down. Cooling is responding but losing ground.
# quiet anything else, including a big climb that the fans absorbed
# and that has since plateaued. That is just load.
#
# Compares the first half of the window against the second to tell a plateau
# from a runaway - a normal load step decelerates, a cooling failure does not.
#
# trend_verdict <temperature column> <fan column>
trend_verdict() {
local tcol=$1 fcol=$2 lines window oldest mid newest fan_old fan_new first second
lines=$(wc -l < "$TREND_FILE" 2>/dev/null || echo 0)
if [ "$lines" -lt "$TREND_SAMPLES" ]; then echo quiet; return; fi
window=$(tail -n "$TREND_SAMPLES" "$TREND_FILE")
oldest=$(printf '%s\n' "$window" | head -1 | cut -d, -f"$tcol")
mid=$(printf '%s\n' "$window" | sed -n "$(( (TREND_SAMPLES + 1) / 2 ))p" | cut -d, -f"$tcol")
newest=$(printf '%s\n' "$window" | tail -1 | cut -d, -f"$tcol")
fan_old=$(printf '%s\n' "$window" | head -1 | cut -d, -f"$fcol")
fan_new=$(printf '%s\n' "$window" | tail -1 | cut -d, -f"$fcol")
[ -n "$oldest" ] && [ -n "$mid" ] && [ -n "$newest" ] || { echo quiet; return; }
# Not climbing meaningfully - nothing to say either way.
[ $((newest - oldest)) -ge "$TREND_RISE_ALARM" ] || { echo quiet; return; }
# Climbing with the fans already flat out.
if [ "$fan_new" -ge "$HIGH_FAN_SPEED" ]; then echo no_headroom; return; fi
# Climbing, fans climbing too, and the second half of the window rose at least
# as fast as the first - it is not settling.
first=$((mid - oldest))
second=$((newest - mid))
if [ "$fan_new" -gt "$fan_old" ] && [ "$second" -ge "$first" ]; then
echo not_converging; return
fi
echo quiet
}
# rise_over_window <column> -> degrees climbed from the oldest sample in the
# window to the newest. 0 until there is a full window of history, so a restart
# cannot alarm on a partial series.
# cannot alarm on a partial series. Reported in alarm text; the decision to
# alarm belongs to trend_verdict.
# Columns: 2 cpu, 3 gpu, 4 disk, 5 fan.
rise_over_window() {
local column=$1 oldest newest lines