Add disk temperature monitoring, trend tracking and email alarms
Fan speed now takes the loudest request from CPU/GPU and from each drive interpolated against its own reported temperature limit, so bay position and drive technology stop mattering. Reported limits are clamped by class because WD Reds report the SCT critical limit (85) rather than an operating maximum. Adds a rolling temperature history and four alarms (absolute, rising trend, SMART health, disk count), rate limited per key with a one hour cooldown, delivered by mail through the host relay. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Zeb Hering
163
README.md
Normal file
163
README.md
Normal file
@@ -0,0 +1,163 @@
|
||||
# fan_speed
|
||||
|
||||
Fan control for Dell PowerEdge servers over IPMI, driven by CPU, GPU **and disk**
|
||||
temperature, with trend tracking and email alarms.
|
||||
|
||||
Upstream ([tigerblue77/Dell_iDRAC_fan_controller_Docker](https://github.com/tigerblue77/Dell_iDRAC_fan_controller_Docker))
|
||||
sets fan speed from CPU and GPU only. On a chassis with two dozen drives that misses the
|
||||
thing most likely to be quietly cooking, so this fork adds disks to the curve, records a
|
||||
rolling temperature history, and mails when something looks wrong.
|
||||
|
||||
For the MD1200 disk shelf, see the separate
|
||||
[md1200-fan-control](https://git.izebra.net/izebra_projects/md1200-fan-control) repo —
|
||||
that enclosure has its own controller and its own serial protocol.
|
||||
|
||||
## How the fan speed is decided
|
||||
|
||||
Each heat source asks for a fan speed and the loudest request wins:
|
||||
|
||||
```
|
||||
speed = max( interpolate(hottest of CPU/GPU, 45 -> 75),
|
||||
per-drive interpolation against each drive's own limit )
|
||||
```
|
||||
|
||||
Sources are never compared as raw temperatures — an 85c CPU and a 45c disk are both
|
||||
"fine", and a single `max()` over the numbers would be meaningless.
|
||||
|
||||
### Why disks are measured against their own limits
|
||||
|
||||
Every drive reports its own maximum operating temperature (SATA: `Min/Max Temperature
|
||||
Limit`; SAS: `Drive Trip Temperature`). Each drive's ramp is derived from that number:
|
||||
|
||||
```
|
||||
ramp starts at limit - DISK_RAMP_LOW_OFFSET (default 18)
|
||||
full speed at limit - DISK_RAMP_HIGH_OFFSET (default 8)
|
||||
```
|
||||
|
||||
So a Samsung SSD rated to 70c ramps 52→62, and a Toshiba spinner rated to 60c ramps
|
||||
42→52. The 50c SSD asks for *less* airflow than the 45c HDD, which is correct — it is
|
||||
further from its own limit.
|
||||
|
||||
This matters because **temperature does not tell you what you think it does**. On these
|
||||
servers the rear-bay SSDs idle 10c hotter than the front-bay spinners: they sit in
|
||||
preheated exhaust air. A shared threshold would peg the fans for drives that are fine
|
||||
and ignore the ones that are not. Measuring each drive against its own envelope makes
|
||||
the curve independent of both drive technology and bay position.
|
||||
|
||||
**Reported limits are clamped by class.** They are not uniformly trustworthy — Samsung
|
||||
and Kioxia report a real operating maximum (70), Toshiba and Seagate report 60, and WD
|
||||
Reds report **85**, which is the SCT critical limit and not somewhere you want a drive
|
||||
living. `HDD_LIMIT_CAP` (60) and `SSD_LIMIT_CAP` (70) bound whatever the drive claims,
|
||||
and supply the value when a drive reports nothing.
|
||||
|
||||
Drive class comes from `/sys/block/<dev>/queue/rotational`. That is derived from the
|
||||
device's RPM flag, which a few SAS drives behind HBAs report incorrectly; if you hit one,
|
||||
SMART's `Rotation Rate` field is the fallback.
|
||||
|
||||
## Alarms
|
||||
|
||||
Email via `mail` to `ALERT_EMAIL`, which the host's postfix relays. Four conditions:
|
||||
|
||||
| Alarm | Fires when |
|
||||
|---|---|
|
||||
| `disk_temp` | A drive is within `DISK_ALARM_OFFSET` (5c) of its own limit |
|
||||
| `trend` | CPU or hottest disk has climbed `TREND_RISE_ALARM` (8c) across the window while still under every threshold |
|
||||
| `smart` | `smartctl -H` reports anything other than PASSED/OK |
|
||||
| `cpu` / `gpu` | Threshold crossed; `cpu` also means fan control was handed back to Dell's profile |
|
||||
| `disk_count` | Fewer drives answered than were present at startup |
|
||||
|
||||
**The trend alarm is the one worth having.** A dying fan or a blocked intake shows up as
|
||||
a steady climb long before anything crosses a threshold — by the time an absolute alarm
|
||||
fires you have already been running hot for hours.
|
||||
|
||||
Every alarm is **rate limited per key** with a one hour cooldown, and sends a single
|
||||
recovery notice when it clears. On a 10s loop an un-throttled alarm sends 360 emails an
|
||||
hour, at which point the alarm is the outage.
|
||||
|
||||
`smartctl -H` across two dozen drives runs hourly, not per loop. Absolute and trend
|
||||
checks stay on the fast loop.
|
||||
|
||||
## Trend history
|
||||
|
||||
Every pass appends to `log/temps.csv`:
|
||||
|
||||
```
|
||||
epoch,cpu,gpu,hottest_disk,fan_speed
|
||||
```
|
||||
|
||||
Trimmed to `TREND_SAMPLES` (90 = 15 minutes at a 10s interval). The trend check compares
|
||||
the newest sample to the oldest in that window, and reports zero until a full window has
|
||||
accumulated so a restart cannot alarm on a partial series. Flat file, `tail` to trim — no
|
||||
rrdtool, no database.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
fan_speed.sh # the service loop (default)
|
||||
fan_speed.sh once # a single pass, prints what it decided
|
||||
fan_speed.sh disks # every drive: temperature and the limit in use
|
||||
fan_speed.sh selftest # parsers, curve, trend detector, alert rate limiting
|
||||
```
|
||||
|
||||
`selftest` touches no hardware and sends no mail — it runs the pure logic against canned
|
||||
smartctl output and a synthetic history file. Run it after any edit.
|
||||
|
||||
## Install
|
||||
|
||||
```sh
|
||||
install -m 755 fan_speed.sh functions.sh monitor.sh /root/fan_speed/
|
||||
mkdir -p /root/fan_speed/log /root/fan_speed/state
|
||||
install -m 644 fan_speed.service /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now fan_speed.service
|
||||
```
|
||||
|
||||
## Tuning
|
||||
|
||||
| Variable | Default | |
|
||||
|---|---|---|
|
||||
| `CHECK_INTERVAL` | `10` | Seconds between passes |
|
||||
| `LOW_FAN_SPEED` / `HIGH_FAN_SPEED` | `18` / `50` | Percent |
|
||||
| `LOW_TEMPERATURE_THRESHOLD` | `45` | CPU/GPU ramp start |
|
||||
| `CPU_TEMPERATURE_THRESHOLD` | `90` | Above this, Dell's profile takes over |
|
||||
| `GPU_TEMPERATURE_THRESHOLD` | `75` | |
|
||||
| `DISK_RAMP_LOW_OFFSET` | `18` | Ramp starts this far below each drive's limit |
|
||||
| `DISK_RAMP_HIGH_OFFSET` | `8` | Full speed this far below it |
|
||||
| `DISK_ALARM_OFFSET` | `5` | Alarm this far below it |
|
||||
| `HDD_LIMIT_CAP` / `SSD_LIMIT_CAP` | `60` / `70` | Ceiling on what a drive may claim |
|
||||
| `TREND_SAMPLES` | `90` | Window length, in passes |
|
||||
| `TREND_RISE_ALARM` | `8` | Degrees of climb that alarms |
|
||||
| `ALERT_EMAIL` | `Servers@ntfy1.izebra.xyz` | |
|
||||
| `ALERT_COOLDOWN` | `3600` | Seconds between repeats of one alarm |
|
||||
| `SMART_CHECK_INTERVAL` | `3600` | Seconds between SMART sweeps |
|
||||
|
||||
`DISK_RAMP_LOW_OFFSET` is the knob to reach for first. At the default 18 an SSD rated to
|
||||
70c starts ramping at 52c; raise the offset to react earlier and louder, lower it to stay
|
||||
quiet longer. It is set where it is because the hottest drive on `iz-pve1` idles at 47c —
|
||||
close enough that a smaller offset would have the fans tracking normal daily drift, and
|
||||
you would lose the ability to tell "disks are warm" from "disks are fine".
|
||||
|
||||
Those defaults were sized against one chassis. Watch a day of `log/temps.csv` before
|
||||
trusting them anywhere else.
|
||||
|
||||
## Notes
|
||||
|
||||
- Kernel device names are not stable — a shelf rescan renamed `sdaa`–`sdai` to
|
||||
`sds`–`sdaa` mid-session. Everything resolves through `/dev/disk/by-path` on every
|
||||
pass; never persist an `sdX`.
|
||||
- `DISK_GLOB` matches internal drives only (`pci-*-scsi-*`). Drives behind a SAS expander
|
||||
are a separate enclosure with separate cooling and are deliberately excluded.
|
||||
- Drive temperature limits are read once at startup. They do not change, and `smartctl -x`
|
||||
is far heavier than the `-A` used on the fast loop.
|
||||
- `smartctl -n standby` throughout, so a sleeping drive is skipped rather than spun up
|
||||
just to be measured. That is also why `disk_count` alarms on "fewer drives answered"
|
||||
rather than on a device disappearing.
|
||||
|
||||
## Files
|
||||
|
||||
| File | |
|
||||
|---|---|
|
||||
| `fan_speed.sh` | Main loop, fan speed decision, alarm conditions, selftest |
|
||||
| `monitor.sh` | Disk reading, trend history, alarm delivery — all local code |
|
||||
| `functions.sh` | Vendored upstream: IPMI, iDRAC, interpolation |
|
||||
| `fan_speed.service` | systemd unit |
|
||||
Reference in New Issue
Block a user