9961e972df
Retiring 8x2TB for 4x8TB + a 16TB moved the hottest drive to 40-41c, exactly the old 40c ramp start, and the fan flipped 10<->15% every 30 seconds. The ramp is now 44-56c, which puts the drives back on the floor with 19c of headroom to their 60c limit. Adds the deadband from fan_speed.sh so the shelf does not chase every degree once drives do climb into the ramp, with a downward-only periodic settle so damping cannot leave it permanently loud. selftest and disks no longer take the serial lock - they do not open the port, and refusing to run them while the service is up is unhelpful. The new tests caught a real bug: in `local a=$1 b=$a` bash expands the right-hand sides before the locals exist, so the held value was silently empty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
181 lines
7.9 KiB
Markdown
181 lines
7.9 KiB
Markdown
# md1200-fan-control
|
||
|
||
Temperature-aware fan control for a Dell PowerVault MD1200 disk shelf, driven over the
|
||
EMM's serial debug console.
|
||
|
||
Out of the box an MD1200 runs its fans at a fixed, very loud speed. The EMM firmware
|
||
accepts a `_shutup <percent>` command that overrides it — but it forgets the setting
|
||
after a few seconds, and it has no idea how hot your drives actually are. This repo
|
||
reads drive temperatures on the host and re-sends an appropriate fan speed on a
|
||
keepalive loop.
|
||
|
||
## Hardware you need
|
||
|
||
The MD1200 EMM exposes its debug console on the **top-left RJ45 port of the left EMM**
|
||
(the one labelled for service use, not the SAS ports). It is not Ethernet — it is RS-232
|
||
behind an RJ45 jack, so you need the Dell service cable:
|
||
|
||
**Dell Password Reset / Service Cable — part `MN657`, also sold as `CT109`.**
|
||
|
||
Same cable is listed for the MD1000 / MD1220 / MD3000 / MD3200 family, so search on any
|
||
of those. It terminates in a DB9 serial connector; if your host has no physical serial
|
||
port, add any USB-to-DB9 adapter and point the script at the resulting `/dev/ttyUSB*`.
|
||
On the machine this was written for, the cable lands on an onboard UART at `/dev/ttyS1`.
|
||
|
||
Serial settings: **38400 8-N-1**, no flow control.
|
||
|
||
> Note: Dell's own KB articles list 115200 for MD3xxx/ME3xxx *controllers*. That is a
|
||
> different port on different hardware. The MD1200 EMM debug console is 38400, which is
|
||
> what this repo uses and what was verified against the real shelf.
|
||
|
||
Once connected you get a prompt like:
|
||
|
||
```
|
||
BlueDress.106.000 >
|
||
```
|
||
|
||
## Debug commands
|
||
|
||
There is no `help` — the firmware answers `unknown_cmd` to `help`, `_help` and `?`. The
|
||
two commands this project relies on:
|
||
|
||
| Command | Effect |
|
||
|---|---|
|
||
| `_shutup <0-100>` | Set fan speed to that percentage. Forgotten if not re-sent. |
|
||
| `_temp_rd` | Print the enclosure temperature sensors. |
|
||
|
||
`_temp_rd` output looks like:
|
||
|
||
```
|
||
BP_1[2] = 30c <- drive backplane
|
||
BP_2[3] = 31c <- drive backplane
|
||
SIM0[0] = 36c <- enclosure management module
|
||
EXP0[4] = 59c <- SAS expander chip, runs hot by design
|
||
|
||
AVG = 39c
|
||
```
|
||
|
||
Commands are terminated with a bare `\r` (not `\n`).
|
||
|
||
## What this does
|
||
|
||
`das_fanctl.sh` runs as a systemd service and loops:
|
||
|
||
1. Read the temperature of every drive in the shelf with `smartctl`, take the hottest.
|
||
2. Interpolate a fan percentage between the low and high thresholds.
|
||
3. Send `_shutup <percent>`, and re-send it every 5s so the EMM doesn't revert.
|
||
4. Re-read temperatures every 30s.
|
||
|
||
**Drive temperature, not enclosure air.** The `BP_*` sensors read backplane intake air,
|
||
which sat at 30c while the drives themselves were 32–39c — it lags what you actually
|
||
care about. Drive temps come from `smartctl -n standby`, so a sleeping drive is skipped
|
||
rather than spun up just to be measured. If no drive answers (all in standby, smartctl
|
||
missing), it falls back to the hottest `BP_*` sensor with its own threshold pair.
|
||
|
||
`EXP0` is deliberately ignored. The SAS expander idles around 59c by design; including
|
||
it in the curve pegs the fans permanently.
|
||
|
||
## Usage
|
||
|
||
```
|
||
das_fanctl.sh # the service loop (default)
|
||
das_fanctl.sh once # one read + set, prints what it did
|
||
das_fanctl.sh disks # per-drive temperatures and the hottest
|
||
das_fanctl.sh temps # raw _temp_rd output from the shelf
|
||
das_fanctl.sh selftest # asserts the parser, the curve, and that drives are readable
|
||
```
|
||
|
||
## Install
|
||
|
||
The script sources `functions.sh` by absolute path, so it expects to live in
|
||
`/root/fan_speed/`:
|
||
|
||
```sh
|
||
install -m 755 das_fanctl.sh functions.sh /root/fan_speed/
|
||
mkdir -p /root/fan_speed/log
|
||
install -m 644 fan_speed_das.service /etc/systemd/system/
|
||
systemctl daemon-reload
|
||
systemctl enable --now fan_speed_das.service
|
||
```
|
||
|
||
Check it: `das_fanctl.sh selftest`, then `tail -f /root/fan_speed/log/das_fan.log`.
|
||
|
||
```
|
||
2026-08-24 23:26:03 || DAS disk:39c | Fan Speed:10
|
||
```
|
||
|
||
## Tuning
|
||
|
||
All knobs are at the top of `das_fanctl.sh`:
|
||
|
||
| Variable | Default | Meaning |
|
||
|---|---|---|
|
||
| `PORT` | `/dev/ttyS1` | Serial device the service cable lands on |
|
||
| `CHECK_INTERVAL` | `30` | Seconds between temperature readings |
|
||
| `KEEPALIVE_INTERVAL` | `5` | How often to re-send `_shutup` |
|
||
| `DISK_LOW_TEMPERATURE_THRESHOLD` | `44` | At/below this, run at `LOW_FAN_SPEED` |
|
||
| `DISK_HIGH_TEMPERATURE_THRESHOLD` | `56` | At/above this, run at `HIGH_FAN_SPEED` |
|
||
| `FAN_SPEED_DEADBAND` | `8` | Hold the current speed until the request moves this far |
|
||
| `SETTLE_INTERVAL` | `300` | Allow a downward settle this often |
|
||
| `LOW_FAN_SPEED` | `10` | Percent |
|
||
| `HIGH_FAN_SPEED` | `60` | Percent |
|
||
| `BP_*_THRESHOLD` | `30` / `45` | Fallback curve, backplane air sensors |
|
||
|
||
**Start the ramp above the drives' normal working range.** This is the single most
|
||
common way to make a shelf cycle audibly: if the ramp starts inside the range the drives
|
||
already occupy, ordinary drift crosses it and the fans flip every cycle. The curve is
|
||
~4%/degC, so one degree moves the fan noticeably.
|
||
|
||
Worked example: the original 40–50c was sized for 8×2TB drives idling at 32–39c. Retiring
|
||
those for 4×8TB + a 16TB moved the hottest drive to 40–41c — exactly the old ramp start —
|
||
and the fan flipped 10↔15% every 30 seconds (18 samples at 10, 22 at 15). Moving the ramp
|
||
to 44–56c put the drives back on the floor: 11 consecutive samples at 10%, and they still
|
||
have 19c of headroom to their 60c limit.
|
||
|
||
`FAN_SPEED_DEADBAND` damps what remains, so once drives do climb into the ramp it does not
|
||
chase every degree. A settle is permitted every `SETTLE_INTERVAL`, downward only —
|
||
a symmetric deadband would catch the fan on the way up and never let it back down.
|
||
|
||
`KEEPALIVE_INTERVAL` is inherited from the 5s cadence the original setup used, not
|
||
measured. How fast the EMM actually reverts is unknown; if the fans audibly surge
|
||
between keepalives, lower it.
|
||
|
||
## Notes
|
||
|
||
- Only one process may talk to the serial port. The script takes an exclusive `flock` on
|
||
`/run/das_fanctl.lock` and exits rather than interleaving garbage with another writer.
|
||
Detach `attachDAS.sh`'s `screen` session before starting the service.
|
||
- **`DISK_GLOB` must name the shelf's HBA on any host with more than one.** The default
|
||
`*-sas-exp*-lun-0` means "behind a SAS expander", which is not the same as "in the
|
||
shelf": a server's own backplane is frequently behind an expander too. On `iz-pve0` the
|
||
internal drives sit on `pci-0000:02:00.0` and the shelf on `pci-0000:04:00.0`, so the
|
||
default glob matched all 17 drives and would have set the shelf's fans from the
|
||
server's internal drive temperatures. Set
|
||
`Environment=DISK_GLOB=/dev/disk/by-path/*04:00.0*-lun-0` in the unit.
|
||
- **The server's own fan controller needs the mirror-image setting.** Whichever host the
|
||
shelf is plugged into must set `DISK_EXCLUDE_PATTERN` to that same HBA in
|
||
`fan_speed.service`, or the shelf's drives drive the *server* fans as well. Moving the
|
||
shelf between hosts means moving both settings.
|
||
- `PORT` is per host too. Check which UART actually answers rather than assuming:
|
||
`grep -v uart:unknown /proc/tty/driver/serial` lists the real ones, then send
|
||
`_temp_rd` to each and watch for the `BlueDress` prompt.
|
||
|
||
## Files
|
||
|
||
| File | |
|
||
|---|---|
|
||
| `das_fanctl.sh` | The fan controller |
|
||
| `fan_speed_das.service` | systemd unit for it |
|
||
| `functions.sh` | Only `calculate_interpolated_fan_speed` is used here — the rest is iDRAC/IPMI |
|
||
| `attachDAS.sh` | Drop into an interactive `screen` session on the shelf console |
|
||
| `get_das_temp.sh` | Superseded by `das_fanctl.sh temps` — it never read the reply back |
|
||
| `log_rotate.txt` | logrotate snippet for the log directory |
|
||
|
||
## Credit
|
||
|
||
`functions.sh` comes from the
|
||
[Dell iDRAC fan controller](https://github.com/tigerblue77/Dell_iDRAC_fan_controller_Docker)
|
||
project; this repo only uses its `calculate_interpolated_fan_speed`. Prior art on the MD1200 console specifically:
|
||
[tonybaltovski/md1200-reduce-fans-systemd](https://github.com/tonybaltovski/md1200-reduce-fans-systemd)
|
||
and [iamjoshgilman/md1200-fan-controller](https://github.com/iamjoshgilman/md1200-fan-controller).
|