Commit Graph

5 Commits

Author SHA1 Message Date
Zeb Hering
86f3b0c323 Enforce that full cooling arrives no later than the disk alarm
DISK_RAMP_HIGH_OFFSET below DISK_ALARM_OFFSET means the alarm fires
while there is still cooling left unused. iz-pve0 was tuned that way by
hand (full at limit-2, alarm at limit-5), which would have mailed 3c
before the fans were actually flat out, for every drive class.

check_ramp_ordering() now clamps at startup and logs it, rather than
exiting - a fan controller that refuses to start leaves the fans
wherever they were.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 21:49:50 -07:00
Zeb Hering
1be394a0ac Calibrate the CPU ramp per host and fix the deadband leak
Two problems found by A/B against Dell's own profile on an R720xd, 8
minute windows at matched CPU temperature:

The CPU ramp started at a fixed 45c, calibrated on a chassis idling at
48c. On one idling at 58c that sat 40% up the ramp doing nothing useful
and ran 6878 RPM against stock's 6240. The ramp is now derived from the
CPU sensor's own upper-non-critical threshold, and the fan floor is
per-host since PWM->RPM is chassis specific.

The 5 minute re-assert re-pushed the current request rather than the
held value, so the deadband leaked a few percent every interval and the
setpoint drifted 32 -> 36 -> 31. next_fan_speed now separates "has it
moved enough to adopt" from "is it time to re-push".

Also documents that running above stock is correct on hosts whose drives
need it: removing the drives' vote on iz-pve0 hit stock RPM exactly and
took two rear-bay SSDs from 54c to 61c in minutes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 21:12:54 -07:00
Zeb Hering
56ae7542a2 Hold fan speed unless the change is worth making
The setpoint was a continuous function of instantaneous temperature and
was pushed to the BMC every pass, so ~1% of fan per degree of CPU noise
meant the fans never settled. Measured on iz-pve0: stock iDRAC held one
speed for 5 minutes through 57-59c jitter while this script made 7
changes in 16 minutes off the same signal.

Adds a 5% deadband around the applied speed, with full-speed requests
never held back and a periodic re-assert in case the BMC forgets.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 20:07:33 -07:00
Zeb Hering
64e6e03dba Exclude an external shelf by explicit config, not by path shape
Testing on a second host showed the internal/external split cannot be
inferred: iz-pve1's MD1200 sits behind its own HBA while iz-pve0's
internal drives sit behind a SAS expander, so the previous
"pci-*-scsi-*" glob matched every drive on one host and none on the
other. DISK_EXCLUDE_PATTERN now names the shelf's HBA per host.

Also fixes parse_disk_limit, which used awk's for-in over split fields
and so returned 0 or 70 from "0/70" depending on iteration order.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 18:43:20 -07:00
Zeb Hering
562d8e8d46 Add disk temperature monitoring, trend tracking and email alarms
Fan speed now takes the loudest request from CPU/GPU and from each drive
interpolated against its own reported temperature limit, so bay position
and drive technology stop mattering. Reported limits are clamped by class
because WD Reds report the SCT critical limit (85) rather than an
operating maximum.

Adds a rolling temperature history and four alarms (absolute, rising
trend, SMART health, disk count), rate limited per key with a one hour
cooldown, delivered by mail through the host relay.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 18:36:07 -07:00