1be394a0ac
Two problems found by A/B against Dell's own profile on an R720xd, 8 minute windows at matched CPU temperature: The CPU ramp started at a fixed 45c, calibrated on a chassis idling at 48c. On one idling at 58c that sat 40% up the ramp doing nothing useful and ran 6878 RPM against stock's 6240. The ramp is now derived from the CPU sensor's own upper-non-critical threshold, and the fan floor is per-host since PWM->RPM is chassis specific. The 5 minute re-assert re-pushed the current request rather than the held value, so the deadband leaked a few percent every interval and the setpoint drifted 32 -> 36 -> 31. next_fan_speed now separates "has it moved enough to adopt" from "is it time to re-push". Also documents that running above stock is correct on hosts whose drives need it: removing the drives' vote on iz-pve0 hit stock RPM exactly and took two rear-bay SSDs from 54c to 61c in minutes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
219 lines
11 KiB
Markdown
219 lines
11 KiB
Markdown
# fan_speed
|
||
|
||
Fan control for Dell PowerEdge servers over IPMI, driven by CPU, GPU **and disk**
|
||
temperature, with trend tracking and email alarms.
|
||
|
||
Upstream ([tigerblue77/Dell_iDRAC_fan_controller_Docker](https://github.com/tigerblue77/Dell_iDRAC_fan_controller_Docker))
|
||
sets fan speed from CPU and GPU only. On a chassis with two dozen drives that misses the
|
||
thing most likely to be quietly cooking, so this fork adds disks to the curve, records a
|
||
rolling temperature history, and mails when something looks wrong.
|
||
|
||
For the MD1200 disk shelf, see the separate
|
||
[md1200-fan-control](https://git.izebra.net/izebra_projects/md1200-fan-control) repo —
|
||
that enclosure has its own controller and its own serial protocol.
|
||
|
||
## How the fan speed is decided
|
||
|
||
Each heat source asks for a fan speed and the loudest request wins:
|
||
|
||
```
|
||
speed = max( interpolate(hottest of CPU/GPU, 45 -> 75),
|
||
per-drive interpolation against each drive's own limit )
|
||
```
|
||
|
||
Sources are never compared as raw temperatures — an 85c CPU and a 45c disk are both
|
||
"fine", and a single `max()` over the numbers would be meaningless.
|
||
|
||
The result is then **held unless it moves by `FAN_SPEED_DEADBAND` (5%)**. This matters
|
||
more than it sounds. The CPU curve is roughly 1% of fan per degree and idle CPU noise is
|
||
±2c, so without a deadband the setpoint changes on almost every pass and the fans never
|
||
settle — measurably worse than stock. Measured on iz-pve0: stock iDRAC held one speed for
|
||
5 minutes straight through 57→59c jitter, while this script made 7 distinct changes in 16
|
||
minutes off the same signal. A request for full speed is never held back, and the held
|
||
value is re-pushed every `FAN_REASSERT_INTERVAL` in case the BMC forgets it.
|
||
|
||
Note that iDRAC's own profile regulates to a target *RPM* (closed loop, rock steady),
|
||
while manual control sets a *PWM* percentage (open loop, so measured RPM still wanders a
|
||
percent or two at a fixed setpoint). That residual is inherent to manual control, not
|
||
something the deadband can remove.
|
||
|
||
### Why disks are measured against their own limits
|
||
|
||
Every drive reports its own maximum operating temperature (SATA: `Min/Max Temperature
|
||
Limit`; SAS: `Drive Trip Temperature`). Each drive's ramp is derived from that number:
|
||
|
||
```
|
||
ramp starts at limit - DISK_RAMP_LOW_OFFSET (default 18)
|
||
full speed at limit - DISK_RAMP_HIGH_OFFSET (default 8)
|
||
```
|
||
|
||
So a Samsung SSD rated to 70c ramps 52→62, and a Toshiba spinner rated to 60c ramps
|
||
42→52. The 50c SSD asks for *less* airflow than the 45c HDD, which is correct — it is
|
||
further from its own limit.
|
||
|
||
This matters because **temperature does not tell you what you think it does**. On these
|
||
servers the rear-bay SSDs idle 10c hotter than the front-bay spinners: they sit in
|
||
preheated exhaust air. A shared threshold would peg the fans for drives that are fine
|
||
and ignore the ones that are not. Measuring each drive against its own envelope makes
|
||
the curve independent of both drive technology and bay position.
|
||
|
||
**Reported limits are clamped by class.** They are not uniformly trustworthy — Samsung
|
||
and Kioxia report a real operating maximum (70), Toshiba and Seagate report 60, and WD
|
||
Reds report **85**, which is the SCT critical limit and not somewhere you want a drive
|
||
living. `HDD_LIMIT_CAP` (60) and `SSD_LIMIT_CAP` (70) bound whatever the drive claims,
|
||
and supply the value when a drive reports nothing.
|
||
|
||
Drive class comes from `/sys/block/<dev>/queue/rotational`. That is derived from the
|
||
device's RPM flag, which a few SAS drives behind HBAs report incorrectly; if you hit one,
|
||
SMART's `Rotation Rate` field is the fallback.
|
||
|
||
## Alarms
|
||
|
||
Email via `mail` to `ALERT_EMAIL`, which the host's postfix relays. Four conditions:
|
||
|
||
| Alarm | Fires when |
|
||
|---|---|
|
||
| `disk_temp` | A drive is within `DISK_ALARM_OFFSET` (5c) of its own limit |
|
||
| `trend` | CPU or hottest disk has climbed `TREND_RISE_ALARM` (8c) across the window while still under every threshold |
|
||
| `smart` | `smartctl -H` reports anything other than PASSED/OK |
|
||
| `cpu` / `gpu` | Threshold crossed; `cpu` also means fan control was handed back to Dell's profile |
|
||
| `disk_count` | Fewer drives answered than were present at startup |
|
||
|
||
**The trend alarm is the one worth having.** A dying fan or a blocked intake shows up as
|
||
a steady climb long before anything crosses a threshold — by the time an absolute alarm
|
||
fires you have already been running hot for hours.
|
||
|
||
Every alarm is **rate limited per key** with a one hour cooldown, and sends a single
|
||
recovery notice when it clears. On a 10s loop an un-throttled alarm sends 360 emails an
|
||
hour, at which point the alarm is the outage.
|
||
|
||
`smartctl -H` across two dozen drives runs hourly, not per loop. Absolute and trend
|
||
checks stay on the fast loop.
|
||
|
||
## Calibrating a host
|
||
|
||
**The defaults are not portable and the fans will be wrong on an uncalibrated host.**
|
||
Airflow, bay layout and the PWM→RPM relationship are all chassis specific. Calibration is
|
||
a one-off, takes about five minutes, and is what keeps this from being louder than stock:
|
||
|
||
1. Hand the fans back to Dell and let them settle, then record what stock actually does:
|
||
`ipmitool raw 0x30 0x30 0x01 0x01`, wait a minute, then read
|
||
`ipmitool sdr type fan` and `ipmitool sdr type temperature`.
|
||
2. Take manual control (`ipmitool raw 0x30 0x30 0x01 0x00`) and sweep PWM to find the
|
||
percentage that reproduces that RPM:
|
||
`ipmitool raw 0x30 0x30 0x02 0xff 0x1c` (0x1c = 28%), waiting ~25s per step.
|
||
3. Set `LOW_FAN_SPEED` to that percentage. This is the floor the host idles at.
|
||
4. Set `CPU_RAMP_LOW_OFFSET` so the CPU ramp *starts a few degrees above the host's
|
||
normal idle*, so idle sits on the floor rather than permanently part-way up a ramp.
|
||
|
||
Measured example — an R720xd (`iz-pve0`) whose CPUs idle at 60c and whose stock profile
|
||
holds 6240 RPM. Here 28% PWM == 6240 RPM, so `LOW_FAN_SPEED=28` and
|
||
`CPU_RAMP_LOW_OFFSET=15` (ramp starts at 62c). An R730xd (`iz-pve1`) idling at 48c is
|
||
fine on the 18/24 defaults. Put the values in the unit as `Environment=` lines.
|
||
|
||
### Expect to run above stock on some hosts, legitimately
|
||
|
||
Dell's profile does not look at drive temperature at all — only CPU, inlet and exhaust.
|
||
On `iz-pve0` two rear-bay SSDs sit in preheated exhaust air, and this was measured
|
||
directly: raising `DISK_RAMP_LOW_OFFSET` so those drives stopped voting dropped the fans
|
||
to 28% (exactly stock's 6240 RPM) and both SSDs climbed 54c → 61c within minutes.
|
||
Restoring their vote brought them back to 53-55c at 34% / 7080 RPM.
|
||
|
||
So on that host **~13% more RPM than stock is the price of keeping two SSDs 6-8c cooler**,
|
||
and it is the whole reason this script exists. If you would rather have stock noise and
|
||
hotter drives, raise `DISK_RAMP_LOW_OFFSET` until the drives stop asking — that is a
|
||
preference, not a bug. What is *not* acceptable is being louder than stock for no reason,
|
||
which is what an uncalibrated CPU curve does.
|
||
|
||
## Trend history
|
||
|
||
Every pass appends to `log/temps.csv`:
|
||
|
||
```
|
||
epoch,cpu,gpu,hottest_disk,fan_speed
|
||
```
|
||
|
||
Trimmed to `TREND_SAMPLES` (90 = 15 minutes at a 10s interval). The trend check compares
|
||
the newest sample to the oldest in that window, and reports zero until a full window has
|
||
accumulated so a restart cannot alarm on a partial series. Flat file, `tail` to trim — no
|
||
rrdtool, no database.
|
||
|
||
## Usage
|
||
|
||
```
|
||
fan_speed.sh # the service loop (default)
|
||
fan_speed.sh once # a single pass, prints what it decided
|
||
fan_speed.sh disks # every drive: temperature and the limit in use
|
||
fan_speed.sh selftest # parsers, curve, trend detector, alert rate limiting
|
||
```
|
||
|
||
`selftest` touches no hardware and sends no mail — it runs the pure logic against canned
|
||
smartctl output and a synthetic history file. Run it after any edit.
|
||
|
||
## Install
|
||
|
||
```sh
|
||
install -m 755 fan_speed.sh functions.sh monitor.sh /root/fan_speed/
|
||
mkdir -p /root/fan_speed/log /root/fan_speed/state
|
||
install -m 644 fan_speed.service /etc/systemd/system/
|
||
systemctl daemon-reload
|
||
systemctl enable --now fan_speed.service
|
||
```
|
||
|
||
## Tuning
|
||
|
||
| Variable | Default | |
|
||
|---|---|---|
|
||
| `CHECK_INTERVAL` | `10` | Seconds between passes |
|
||
| `LOW_FAN_SPEED` / `HIGH_FAN_SPEED` | `18` / `50` | Percent |
|
||
| `FAN_SPEED_DEADBAND` | `5` | Hold the current speed until the request moves this far |
|
||
| `FAN_REASSERT_INTERVAL` | `300` | Re-push the held speed this often regardless |
|
||
| `LOW_TEMPERATURE_THRESHOLD` | `45` | CPU/GPU ramp start |
|
||
| `CPU_TEMPERATURE_THRESHOLD` | `90` | Above this, Dell's profile takes over |
|
||
| `GPU_TEMPERATURE_THRESHOLD` | `75` | |
|
||
| `DISK_RAMP_LOW_OFFSET` | `18` | Ramp starts this far below each drive's limit |
|
||
| `DISK_RAMP_HIGH_OFFSET` | `8` | Full speed this far below it |
|
||
| `DISK_ALARM_OFFSET` | `5` | Alarm this far below it |
|
||
| `HDD_LIMIT_CAP` / `SSD_LIMIT_CAP` | `60` / `70` | Ceiling on what a drive may claim |
|
||
| `TREND_SAMPLES` | `90` | Window length, in passes |
|
||
| `TREND_RISE_ALARM` | `8` | Degrees of climb that alarms |
|
||
| `DISK_EXCLUDE_PATTERN` | *(empty)* | by-path substring to skip, e.g. an external shelf's HBA |
|
||
| `ALERT_EMAIL` | `Servers@ntfy1.izebra.xyz` | |
|
||
| `ALERT_COOLDOWN` | `3600` | Seconds between repeats of one alarm |
|
||
| `SMART_CHECK_INTERVAL` | `3600` | Seconds between SMART sweeps |
|
||
|
||
`DISK_RAMP_LOW_OFFSET` is the knob to reach for first. At the default 18 an SSD rated to
|
||
70c starts ramping at 52c; raise the offset to react earlier and louder, lower it to stay
|
||
quiet longer. It is set where it is because the hottest drive on `iz-pve1` idles at 47c —
|
||
close enough that a smaller offset would have the fans tracking normal daily drift, and
|
||
you would lose the ability to tell "disks are warm" from "disks are fine".
|
||
|
||
Those defaults were sized against one chassis. Watch a day of `log/temps.csv` before
|
||
trusting them anywhere else.
|
||
|
||
## Notes
|
||
|
||
- Kernel device names are not stable — a shelf rescan renamed `sdaa`–`sdai` to
|
||
`sds`–`sdaa` mid-session. Everything resolves through `/dev/disk/by-path` on every
|
||
pass; never persist an `sdX`.
|
||
- **An external shelf must be excluded by hand.** Set `DISK_EXCLUDE_PATTERN` to its HBA's
|
||
PCI address in the systemd unit. This cannot be inferred: `iz-pve1`'s MD1200 sits behind
|
||
its own HBA at `pci-0000:04:00.0`, while `iz-pve0`'s *internal* drives sit behind a SAS
|
||
expander at `pci-0000:02:00.0` — so "behind an expander" identifies an external
|
||
enclosure on one host and the internal backplane on the other. `iz-pve1` sets it,
|
||
`iz-pve0` has no shelf and leaves it empty.
|
||
- Drive temperature limits are read once at startup. They do not change, and `smartctl -x`
|
||
is far heavier than the `-A` used on the fast loop.
|
||
- `smartctl -n standby` throughout, so a sleeping drive is skipped rather than spun up
|
||
just to be measured. That is also why `disk_count` alarms on "fewer drives answered"
|
||
rather than on a device disappearing.
|
||
|
||
## Files
|
||
|
||
| File | |
|
||
|---|---|
|
||
| `fan_speed.sh` | Main loop, fan speed decision, alarm conditions, selftest |
|
||
| `monitor.sh` | Disk reading, trend history, alarm delivery — all local code |
|
||
| `functions.sh` | Vendored upstream: IPMI, iDRAC, interpolation |
|
||
| `fan_speed.service` | systemd unit |
|