86f3b0c323
DISK_RAMP_HIGH_OFFSET below DISK_ALARM_OFFSET means the alarm fires while there is still cooling left unused. iz-pve0 was tuned that way by hand (full at limit-2, alarm at limit-5), which would have mailed 3c before the fans were actually flat out, for every drive class. check_ramp_ordering() now clamps at startup and logs it, rather than exiting - a fan controller that refuses to start leaves the fans wherever they were. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
254 lines
13 KiB
Markdown
254 lines
13 KiB
Markdown
# fan_speed
|
||
|
||
Fan control for Dell PowerEdge servers over IPMI, driven by CPU, GPU **and disk**
|
||
temperature, with trend tracking and email alarms.
|
||
|
||
Upstream ([tigerblue77/Dell_iDRAC_fan_controller_Docker](https://github.com/tigerblue77/Dell_iDRAC_fan_controller_Docker))
|
||
sets fan speed from CPU and GPU only. On a chassis with two dozen drives that misses the
|
||
thing most likely to be quietly cooking, so this fork adds disks to the curve, records a
|
||
rolling temperature history, and mails when something looks wrong.
|
||
|
||
For the MD1200 disk shelf, see the separate
|
||
[md1200-fan-control](https://git.izebra.net/izebra_projects/md1200-fan-control) repo —
|
||
that enclosure has its own controller and its own serial protocol.
|
||
|
||
## How the fan speed is decided
|
||
|
||
Each heat source asks for a fan speed and the loudest request wins:
|
||
|
||
```
|
||
speed = max( interpolate(hottest of CPU/GPU, 45 -> 75),
|
||
per-drive interpolation against each drive's own limit )
|
||
```
|
||
|
||
Sources are never compared as raw temperatures — an 85c CPU and a 45c disk are both
|
||
"fine", and a single `max()` over the numbers would be meaningless.
|
||
|
||
The result is then **held unless it moves by `FAN_SPEED_DEADBAND` (5%)**. This matters
|
||
more than it sounds. The CPU curve is roughly 1% of fan per degree and idle CPU noise is
|
||
±2c, so without a deadband the setpoint changes on almost every pass and the fans never
|
||
settle — measurably worse than stock. Measured on iz-pve0: stock iDRAC held one speed for
|
||
5 minutes straight through 57→59c jitter, while this script made 7 distinct changes in 16
|
||
minutes off the same signal. A request for full speed is never held back, and the held
|
||
value is re-pushed every `FAN_REASSERT_INTERVAL` in case the BMC forgets it.
|
||
|
||
Note that iDRAC's own profile regulates to a target *RPM* (closed loop, rock steady),
|
||
while manual control sets a *PWM* percentage (open loop, so measured RPM still wanders a
|
||
percent or two at a fixed setpoint). That residual is inherent to manual control, not
|
||
something the deadband can remove.
|
||
|
||
### Why disks are measured against their own limits
|
||
|
||
Every drive reports its own maximum operating temperature (SATA: `Min/Max Temperature
|
||
Limit`; SAS: `Drive Trip Temperature`). Each drive's ramp is derived from that number:
|
||
|
||
```
|
||
ramp starts at limit - DISK_RAMP_LOW_OFFSET (default 18)
|
||
full speed at limit - DISK_RAMP_HIGH_OFFSET (default 8)
|
||
```
|
||
|
||
So a Samsung SSD rated to 70c ramps 52→62, and a Toshiba spinner rated to 60c ramps
|
||
42→52. The 50c SSD asks for *less* airflow than the 45c HDD, which is correct — it is
|
||
further from its own limit.
|
||
|
||
This matters because **temperature does not tell you what you think it does**. On these
|
||
servers the rear-bay SSDs idle 10c hotter than the front-bay spinners: they sit in
|
||
preheated exhaust air. A shared threshold would peg the fans for drives that are fine
|
||
and ignore the ones that are not. Measuring each drive against its own envelope makes
|
||
the curve independent of both drive technology and bay position.
|
||
|
||
A drive with a lower tolerance drives the fans even while a hotter drive does not.
|
||
With `iz-pve0`'s settings (floor 28%, ramp `limit-8` to `limit-5`):
|
||
|
||
| Drive | Limit | Ramp | Asks at 45c | at 50c | at 55c |
|
||
|---|---|---|---|---|---|
|
||
| Samsung 860 EVO | 70c | 62-65c | 28% | 28% | 28% |
|
||
| Toshiba / Seagate HDD | 60c | 52-55c | 28% | 28% | **50%** |
|
||
| a 55c-rated drive | 55c | 47-50c | 28% | **50%** | **50%** |
|
||
|
||
At 55c the SSD contributes nothing while the spinner asks for full speed, which is the
|
||
whole point: the same temperature means different things to different drives.
|
||
|
||
**Full cooling must arrive no later than the alarm.** `DISK_RAMP_HIGH_OFFSET` must be
|
||
`>=` `DISK_ALARM_OFFSET`, or the alarm mails you while there is still cooling left unused.
|
||
`check_ramp_ordering()` clamps this at startup and logs when it does — it clamps rather
|
||
than exits, because a fan controller that refuses to start leaves the fans wherever they
|
||
happened to be.
|
||
|
||
**Reported limits are clamped by class.** They are not uniformly trustworthy — Samsung
|
||
and Kioxia report a real operating maximum (70), Toshiba and Seagate report 60, and WD
|
||
Reds report **85**, which is the SCT critical limit and not somewhere you want a drive
|
||
living. `HDD_LIMIT_CAP` (60) and `SSD_LIMIT_CAP` (70) bound whatever the drive claims,
|
||
and supply the value when a drive reports nothing.
|
||
|
||
Drive class comes from `/sys/block/<dev>/queue/rotational`. That is derived from the
|
||
device's RPM flag, which a few SAS drives behind HBAs report incorrectly; if you hit one,
|
||
SMART's `Rotation Rate` field is the fallback.
|
||
|
||
## Alarms
|
||
|
||
Email via `mail` to `ALERT_EMAIL`, which the host's postfix relays. Four conditions:
|
||
|
||
| Alarm | Fires when |
|
||
|---|---|
|
||
| `disk_temp` | A drive is within `DISK_ALARM_OFFSET` (5c) of its own limit |
|
||
| `trend` | CPU or hottest disk has climbed `TREND_RISE_ALARM` (8c) across the window while still under every threshold |
|
||
| `smart` | `smartctl -H` reports anything other than PASSED/OK |
|
||
| `cpu` / `gpu` | Threshold crossed; `cpu` also means fan control was handed back to Dell's profile |
|
||
| `disk_count` | Fewer drives answered than were present at startup |
|
||
|
||
**The trend alarm is the one worth having.** A dying fan or a blocked intake shows up as
|
||
a steady climb long before anything crosses a threshold — by the time an absolute alarm
|
||
fires you have already been running hot for hours.
|
||
|
||
Every alarm is **rate limited per key** with a one hour cooldown, and sends a single
|
||
recovery notice when it clears. On a 10s loop an un-throttled alarm sends 360 emails an
|
||
hour, at which point the alarm is the outage.
|
||
|
||
`smartctl -H` across two dozen drives runs hourly, not per loop. Absolute and trend
|
||
checks stay on the fast loop.
|
||
|
||
## Calibrating a host
|
||
|
||
**The defaults are not portable and the fans will be wrong on an uncalibrated host.**
|
||
Airflow, bay layout and the PWM→RPM relationship are all chassis specific. Calibration is
|
||
a one-off, takes about five minutes, and is what keeps this from being louder than stock:
|
||
|
||
1. Hand the fans back to Dell and let them settle, then record what stock actually does:
|
||
`ipmitool raw 0x30 0x30 0x01 0x01`, wait a minute, then read
|
||
`ipmitool sdr type fan` and `ipmitool sdr type temperature`.
|
||
2. Take manual control (`ipmitool raw 0x30 0x30 0x01 0x00`) and sweep PWM to find the
|
||
percentage that reproduces that RPM:
|
||
`ipmitool raw 0x30 0x30 0x02 0xff 0x1c` (0x1c = 28%), waiting ~25s per step.
|
||
3. Set `LOW_FAN_SPEED` to that percentage. This is the floor the host idles at.
|
||
4. Set `CPU_RAMP_LOW_OFFSET` so the CPU ramp *starts a few degrees above the host's
|
||
normal idle*, so idle sits on the floor rather than permanently part-way up a ramp.
|
||
|
||
Measured example — an R720xd (`iz-pve0`) whose CPUs idle at 60c and whose stock profile
|
||
holds 6240 RPM. Here 28% PWM == 6240 RPM, so `LOW_FAN_SPEED=28` and
|
||
`CPU_RAMP_LOW_OFFSET=15` (ramp starts at 62c). An R730xd (`iz-pve1`) idling at 48c is
|
||
fine on the 18/24 defaults. Put the values in the unit as `Environment=` lines.
|
||
|
||
### Put the disk ramp above the drives' normal working temperature
|
||
|
||
The single most common cause of a noisy host is a disk ramp that starts *inside* the
|
||
range the drives normally live in. Then every load spike pushes the fan up, the fan drops
|
||
back, and it cycles audibly forever.
|
||
|
||
Worked example, `iz-pve0`: `sda`/`sdh` are the rpool mirror in rear bays — busy root SSDs
|
||
that idle at 53-55c and touch 60-61c under load, rated to 70c. With the default ramp
|
||
starting at 52c they were permanently part-way up it, dragging the fans to 7920 RPM
|
||
(+27% over stock) to cool drives that were never in trouble. Starting the ramp at 60c
|
||
still cycled, because load spikes crossed it. Starting it at **62c**, just above their
|
||
working peak, holds 6240 RPM flat — and the drives settle at 53-54c anyway.
|
||
|
||
The rule: find the drives' normal peak, put `DISK_RAMP_LOW_OFFSET` a couple of degrees
|
||
above it, and let the alarm (`DISK_ALARM_OFFSET`, 5c below the limit) catch anything real.
|
||
Cooling a healthy drive from 57c to 53c is not worth 27% more fan.
|
||
|
||
### Expect to run above stock on some hosts, legitimately
|
||
|
||
Dell's profile does not look at drive temperature at all — only CPU, inlet and exhaust.
|
||
On `iz-pve0` two rear-bay SSDs sit in preheated exhaust air, and this was measured
|
||
directly: raising `DISK_RAMP_LOW_OFFSET` so those drives stopped voting dropped the fans
|
||
to 28% (exactly stock's 6240 RPM) and both SSDs climbed 54c → 61c within minutes.
|
||
Restoring their vote brought them back to 53-55c at 34% / 7080 RPM.
|
||
|
||
So on that host **~13% more RPM than stock is the price of keeping two SSDs 6-8c cooler**,
|
||
and it is the whole reason this script exists. If you would rather have stock noise and
|
||
hotter drives, raise `DISK_RAMP_LOW_OFFSET` until the drives stop asking — that is a
|
||
preference, not a bug. What is *not* acceptable is being louder than stock for no reason,
|
||
which is what an uncalibrated CPU curve does.
|
||
|
||
## Trend history
|
||
|
||
Every pass appends to `log/temps.csv`:
|
||
|
||
```
|
||
epoch,cpu,gpu,hottest_disk,fan_speed
|
||
```
|
||
|
||
Trimmed to `TREND_SAMPLES` (90 = 15 minutes at a 10s interval). The trend check compares
|
||
the newest sample to the oldest in that window, and reports zero until a full window has
|
||
accumulated so a restart cannot alarm on a partial series. Flat file, `tail` to trim — no
|
||
rrdtool, no database.
|
||
|
||
## Usage
|
||
|
||
```
|
||
fan_speed.sh # the service loop (default)
|
||
fan_speed.sh once # a single pass, prints what it decided
|
||
fan_speed.sh disks # every drive: temperature and the limit in use
|
||
fan_speed.sh selftest # parsers, curve, trend detector, alert rate limiting
|
||
```
|
||
|
||
`selftest` touches no hardware and sends no mail — it runs the pure logic against canned
|
||
smartctl output and a synthetic history file. Run it after any edit.
|
||
|
||
## Install
|
||
|
||
```sh
|
||
install -m 755 fan_speed.sh functions.sh monitor.sh /root/fan_speed/
|
||
mkdir -p /root/fan_speed/log /root/fan_speed/state
|
||
install -m 644 fan_speed.service /etc/systemd/system/
|
||
systemctl daemon-reload
|
||
systemctl enable --now fan_speed.service
|
||
```
|
||
|
||
## Tuning
|
||
|
||
| Variable | Default | |
|
||
|---|---|---|
|
||
| `CHECK_INTERVAL` | `10` | Seconds between passes |
|
||
| `LOW_FAN_SPEED` / `HIGH_FAN_SPEED` | `18` / `50` | Percent |
|
||
| `FAN_SPEED_DEADBAND` | `5` | Hold the current speed until the request moves this far |
|
||
| `FAN_REASSERT_INTERVAL` | `300` | Re-push the held speed this often regardless |
|
||
| `LOW_TEMPERATURE_THRESHOLD` | `45` | CPU/GPU ramp start |
|
||
| `CPU_TEMPERATURE_THRESHOLD` | `90` | Above this, Dell's profile takes over |
|
||
| `GPU_TEMPERATURE_THRESHOLD` | `75` | |
|
||
| `DISK_RAMP_LOW_OFFSET` | `18` | Ramp starts this far below each drive's limit |
|
||
| `DISK_RAMP_HIGH_OFFSET` | `8` | Full speed this far below it |
|
||
| `DISK_ALARM_OFFSET` | `5` | Alarm this far below it |
|
||
| `HDD_LIMIT_CAP` / `SSD_LIMIT_CAP` | `60` / `70` | Ceiling on what a drive may claim |
|
||
| `TREND_SAMPLES` | `90` | Window length, in passes |
|
||
| `TREND_RISE_ALARM` | `8` | Degrees of climb that alarms |
|
||
| `DISK_EXCLUDE_PATTERN` | *(empty)* | by-path substring to skip, e.g. an external shelf's HBA |
|
||
| `ALERT_EMAIL` | `Servers@ntfy1.izebra.xyz` | |
|
||
| `ALERT_COOLDOWN` | `3600` | Seconds between repeats of one alarm |
|
||
| `SMART_CHECK_INTERVAL` | `3600` | Seconds between SMART sweeps |
|
||
|
||
`DISK_RAMP_LOW_OFFSET` is the knob to reach for first. At the default 18 an SSD rated to
|
||
70c starts ramping at 52c; raise the offset to react earlier and louder, lower it to stay
|
||
quiet longer. It is set where it is because the hottest drive on `iz-pve1` idles at 47c —
|
||
close enough that a smaller offset would have the fans tracking normal daily drift, and
|
||
you would lose the ability to tell "disks are warm" from "disks are fine".
|
||
|
||
Those defaults were sized against one chassis. Watch a day of `log/temps.csv` before
|
||
trusting them anywhere else.
|
||
|
||
## Notes
|
||
|
||
- Kernel device names are not stable — a shelf rescan renamed `sdaa`–`sdai` to
|
||
`sds`–`sdaa` mid-session. Everything resolves through `/dev/disk/by-path` on every
|
||
pass; never persist an `sdX`.
|
||
- **An external shelf must be excluded by hand.** Set `DISK_EXCLUDE_PATTERN` to its HBA's
|
||
PCI address in the systemd unit. This cannot be inferred: `iz-pve1`'s MD1200 sits behind
|
||
its own HBA at `pci-0000:04:00.0`, while `iz-pve0`'s *internal* drives sit behind a SAS
|
||
expander at `pci-0000:02:00.0` — so "behind an expander" identifies an external
|
||
enclosure on one host and the internal backplane on the other. `iz-pve1` sets it,
|
||
`iz-pve0` has no shelf and leaves it empty.
|
||
- Drive temperature limits are read once at startup. They do not change, and `smartctl -x`
|
||
is far heavier than the `-A` used on the fast loop.
|
||
- `smartctl -n standby` throughout, so a sleeping drive is skipped rather than spun up
|
||
just to be measured. That is also why `disk_count` alarms on "fewer drives answered"
|
||
rather than on a device disappearing.
|
||
|
||
## Files
|
||
|
||
| File | |
|
||
|---|---|
|
||
| `fan_speed.sh` | Main loop, fan speed decision, alarm conditions, selftest |
|
||
| `monitor.sh` | Disk reading, trend history, alarm delivery — all local code |
|
||
| `functions.sh` | Vendored upstream: IPMI, iDRAC, interpolation |
|
||
| `fan_speed.service` | systemd unit |
|