A symmetric deadband catches the fan on the way up and holds it there:
on iz-pve0 under load, 17 of 20 samples ran at 38% while the curve
wanted 34-36%, and nothing would bring it down until the request fell a
full 8 points. Damping was turning into a permanently louder machine.
A re-assert may now settle the speed downward to the current request.
Upward moves still require a full deadband crossing, so the drift this
was built to prevent cannot come back - covered by a test.
Verified under a live migration: settled 38 -> 36 on the first
re-assert, then one change in ten minutes while the request wandered
36-42, CPU stable at 66-69c and disks a degree cooler.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Span sets sensitivity: 22 points over 7c needs only 1.6c of drift to
cross a 5% deadband, over 11c it needs 2.5c. Replaying 180 recorded
samples, widening the ramp and raising the deadband took the same window
from 19 fan changes to 1.
Confirmed live under a zpool migration: 19 changes became 2, average fan
speed fell from 40% to 38%, and CPU temperature was unchanged at 64-68c.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A rising temperature is what a busy machine looks like; the previous
trend alarm would have mailed continuously through a multi-hour zpool
migration. trend_verdict() now alarms only when the fans are already at
maximum and it is still climbing, or when temperature and fan speed are
both rising and the rise is not decelerating.
Tells a plateau from a runaway by comparing the first half of the window
against the second - a load step settles once the fans catch up, a
failing fan does not. Verified quiet against a live migration at load
average 18 with CPU at 68c.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DISK_RAMP_HIGH_OFFSET below DISK_ALARM_OFFSET means the alarm fires
while there is still cooling left unused. iz-pve0 was tuned that way by
hand (full at limit-2, alarm at limit-5), which would have mailed 3c
before the fans were actually flat out, for every drive class.
check_ramp_ordering() now clamps at startup and logs it, rather than
exiting - a fan controller that refuses to start leaves the fans
wherever they were.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A ramp that starts inside the range drives normally occupy makes every
load spike move the fans, which cycles audibly. On iz-pve0 the rpool
mirror works at 53-61c against a ramp starting at 52c, holding the fans
at 7920 RPM (+27% over stock) to cool drives rated to 70c. Starting the
ramp at 62c holds stock RPM flat and the drives settle at 53-54c.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two problems found by A/B against Dell's own profile on an R720xd, 8
minute windows at matched CPU temperature:
The CPU ramp started at a fixed 45c, calibrated on a chassis idling at
48c. On one idling at 58c that sat 40% up the ramp doing nothing useful
and ran 6878 RPM against stock's 6240. The ramp is now derived from the
CPU sensor's own upper-non-critical threshold, and the fan floor is
per-host since PWM->RPM is chassis specific.
The 5 minute re-assert re-pushed the current request rather than the
held value, so the deadband leaked a few percent every interval and the
setpoint drifted 32 -> 36 -> 31. next_fan_speed now separates "has it
moved enough to adopt" from "is it time to re-push".
Also documents that running above stock is correct on hosts whose drives
need it: removing the drives' vote on iz-pve0 hit stock RPM exactly and
took two rear-bay SSDs from 54c to 61c in minutes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The setpoint was a continuous function of instantaneous temperature and
was pushed to the BMC every pass, so ~1% of fan per degree of CPU noise
meant the fans never settled. Measured on iz-pve0: stock iDRAC held one
speed for 5 minutes through 57-59c jitter while this script made 7
changes in 16 minutes off the same signal.
Adds a 5% deadband around the applied speed, with full-speed requests
never held back and a periodic re-assert in case the BMC forgets.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Testing on a second host showed the internal/external split cannot be
inferred: iz-pve1's MD1200 sits behind its own HBA while iz-pve0's
internal drives sit behind a SAS expander, so the previous
"pci-*-scsi-*" glob matched every drive on one host and none on the
other. DISK_EXCLUDE_PATTERN now names the shelf's HBA per host.
Also fixes parse_disk_limit, which used awk's for-in over split fields
and so returned 0 or 70 from "0/70" depending on iteration order.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fan speed now takes the loudest request from CPU/GPU and from each drive
interpolated against its own reported temperature limit, so bay position
and drive technology stop mattering. Reported limits are clamped by class
because WD Reds report the SCT critical limit (85) rather than an
operating maximum.
Adds a rolling temperature history and four alarms (absolute, rising
trend, SMART health, disk count), rate limited per key with a one hour
cooldown, delivered by mail through the host relay.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
