From db69c08c73af20395890771ac0c3b2b7e397e950 Mon Sep 17 00:00:00 2001 From: Zeb Hering Date: Tue, 25 Aug 2026 10:10:43 -0700 Subject: [PATCH] Add temperature-aware MD1200 fan control das_fanctl.sh reads drive temps via smartctl (falling back to the EMM backplane sensors), interpolates a fan percentage, and re-sends _shutup on a 5s keepalive so the shelf does not revert to full speed. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 156 ++++++++++++++++++++++++++++++ attachDAS.sh | 3 + das_fanctl.sh | 127 +++++++++++++++++++++++++ fan_control.sh | 105 +++++++++++++++++++++ fan_speed.service | 8 ++ fan_speed.sh | 191 +++++++++++++++++++++++++++++++++++++ fan_speed_das.service | 12 +++ functions.sh | 215 ++++++++++++++++++++++++++++++++++++++++++ get_das_temp.sh | 10 ++ log_rotate.txt | 8 ++ 10 files changed, 835 insertions(+) create mode 100644 README.md create mode 100755 attachDAS.sh create mode 100755 das_fanctl.sh create mode 100755 fan_control.sh create mode 100644 fan_speed.service create mode 100755 fan_speed.sh create mode 100644 fan_speed_das.service create mode 100755 functions.sh create mode 100755 get_das_temp.sh create mode 100755 log_rotate.txt diff --git a/README.md b/README.md new file mode 100644 index 0000000..1a7417d --- /dev/null +++ b/README.md @@ -0,0 +1,156 @@ +# md1200-fan-control + +Temperature-aware fan control for a Dell PowerVault MD1200 disk shelf, driven over the +EMM's serial debug console. + +Out of the box an MD1200 runs its fans at a fixed, very loud speed. The EMM firmware +accepts a `_shutup ` command that overrides it — but it forgets the setting +after a few seconds, and it has no idea how hot your drives actually are. This repo +reads drive temperatures on the host and re-sends an appropriate fan speed on a +keepalive loop. + +## Hardware you need + +The MD1200 EMM exposes its debug console on the **top-left RJ45 port of the left EMM** +(the one labelled for service use, not the SAS ports). It is not Ethernet — it is RS-232 +behind an RJ45 jack, so you need the Dell service cable: + +**Dell Password Reset / Service Cable — part `MN657`, also sold as `CT109`.** + +Same cable is listed for the MD1000 / MD1220 / MD3000 / MD3200 family, so search on any +of those. It terminates in a DB9 serial connector; if your host has no physical serial +port, add any USB-to-DB9 adapter and point the script at the resulting `/dev/ttyUSB*`. +On the machine this was written for, the cable lands on an onboard UART at `/dev/ttyS1`. + +Serial settings: **38400 8-N-1**, no flow control. + +> Note: Dell's own KB articles list 115200 for MD3xxx/ME3xxx *controllers*. That is a +> different port on different hardware. The MD1200 EMM debug console is 38400, which is +> what this repo uses and what was verified against the real shelf. + +Once connected you get a prompt like: + +``` +BlueDress.106.000 > +``` + +## Debug commands + +There is no `help` — the firmware answers `unknown_cmd` to `help`, `_help` and `?`. The +two commands this project relies on: + +| Command | Effect | +|---|---| +| `_shutup <0-100>` | Set fan speed to that percentage. Forgotten if not re-sent. | +| `_temp_rd` | Print the enclosure temperature sensors. | + +`_temp_rd` output looks like: + +``` + BP_1[2] = 30c <- drive backplane + BP_2[3] = 31c <- drive backplane + SIM0[0] = 36c <- enclosure management module + EXP0[4] = 59c <- SAS expander chip, runs hot by design + + AVG = 39c +``` + +Commands are terminated with a bare `\r` (not `\n`). + +## What this does + +`das_fanctl.sh` runs as a systemd service and loops: + +1. Read the temperature of every drive in the shelf with `smartctl`, take the hottest. +2. Interpolate a fan percentage between the low and high thresholds. +3. Send `_shutup `, and re-send it every 5s so the EMM doesn't revert. +4. Re-read temperatures every 30s. + +**Drive temperature, not enclosure air.** The `BP_*` sensors read backplane intake air, +which sat at 30c while the drives themselves were 32–39c — it lags what you actually +care about. Drive temps come from `smartctl -n standby`, so a sleeping drive is skipped +rather than spun up just to be measured. If no drive answers (all in standby, smartctl +missing), it falls back to the hottest `BP_*` sensor with its own threshold pair. + +`EXP0` is deliberately ignored. The SAS expander idles around 59c by design; including +it in the curve pegs the fans permanently. + +## Usage + +``` +das_fanctl.sh # the service loop (default) +das_fanctl.sh once # one read + set, prints what it did +das_fanctl.sh disks # per-drive temperatures and the hottest +das_fanctl.sh temps # raw _temp_rd output from the shelf +das_fanctl.sh selftest # asserts the parser, the curve, and that drives are readable +``` + +## Install + +The script sources `functions.sh` by absolute path, so it expects to live in +`/root/fan_speed/`: + +```sh +install -m 755 das_fanctl.sh functions.sh /root/fan_speed/ +mkdir -p /root/fan_speed/log +install -m 644 fan_speed_das.service /etc/systemd/system/ +systemctl daemon-reload +systemctl enable --now fan_speed_das.service +``` + +Check it: `das_fanctl.sh selftest`, then `tail -f /root/fan_speed/log/das_fan.log`. + +``` +2026-08-24 23:26:03 || DAS disk:39c | Fan Speed:10 +``` + +## Tuning + +All knobs are at the top of `das_fanctl.sh`: + +| Variable | Default | Meaning | +|---|---|---| +| `PORT` | `/dev/ttyS1` | Serial device the service cable lands on | +| `CHECK_INTERVAL` | `30` | Seconds between temperature readings | +| `KEEPALIVE_INTERVAL` | `5` | How often to re-send `_shutup` | +| `DISK_LOW_TEMPERATURE_THRESHOLD` | `40` | At/below this, run at `LOW_FAN_SPEED` | +| `DISK_HIGH_TEMPERATURE_THRESHOLD` | `50` | At/above this, run at `HIGH_FAN_SPEED` | +| `LOW_FAN_SPEED` | `10` | Percent | +| `HIGH_FAN_SPEED` | `60` | Percent | +| `BP_*_THRESHOLD` | `30` / `45` | Fallback curve, backplane air sensors | + +The defaults were sized against an idle spread of 32–39c across nine mixed SATA/SAS +drives. Your airflow is not that airflow — watch a day of log before trusting them. + +`KEEPALIVE_INTERVAL` is inherited from the 5s cadence the original setup used, not +measured. How fast the EMM actually reverts is unknown; if the fans audibly surge +between keepalives, lower it. + +## Notes + +- Only one process may talk to the serial port. The script takes an exclusive `flock` on + `/run/das_fanctl.lock` and exits rather than interleaving garbage with another writer. + Detach `attachDAS.sh`'s `screen` session before starting the service. +- The disk glob is `/dev/disk/by-path/*-sas-exp*-lun-0` — anything behind a SAS expander, + i.e. the shelf. The host's own drives hang off `pci-…-scsi-*` and are not matched. A + second enclosure would be lumped in with the first. + +## Files + +| File | | +|---|---| +| `das_fanctl.sh` | The fan controller | +| `fan_speed_das.service` | systemd unit for it | +| `functions.sh` | Shared helpers, incl. `calculate_interpolated_fan_speed` | +| `attachDAS.sh` | Drop into an interactive `screen` session on the shelf console | +| `get_das_temp.sh` | Superseded by `das_fanctl.sh temps` — it never read the reply back | +| `fan_speed.sh` `fan_control.sh` `fan_speed.service` | Host-side iDRAC/IPMI fan control, unrelated to the shelf | +| `log_rotate.txt` | logrotate snippet for the log directory | + +## Credit + +`calculate_interpolated_fan_speed` and the host-side iDRAC scripts derive from the +[Dell iDRAC fan controller](https://github.com/tigerblue77/Dell_iDRAC_fan_controller_Docker) +project. Prior art on the MD1200 console specifically: +[tonybaltovski/md1200-reduce-fans-systemd](https://github.com/tonybaltovski/md1200-reduce-fans-systemd) +and [iamjoshgilman/md1200-fan-controller](https://github.com/iamjoshgilman/md1200-fan-controller). diff --git a/attachDAS.sh b/attachDAS.sh new file mode 100755 index 0000000..ec09376 --- /dev/null +++ b/attachDAS.sh @@ -0,0 +1,3 @@ +killall screen +stty -F /dev/ttyS1 38400 raw -echoe -echok -echoctl -echoke +screen -L /dev/ttyS1 38400 diff --git a/das_fanctl.sh b/das_fanctl.sh new file mode 100755 index 0000000..5443502 --- /dev/null +++ b/das_fanctl.sh @@ -0,0 +1,127 @@ +#!/bin/bash + +#Zeb H. +#Dell MD1200 DAS fan control - read temps, set fan speed to match. +#Drives the curve off drive temperature, falls back to the enclosure backplane sensors. +#Usage: das_fanctl.sh [loop|once|temps|disks|selftest] (default: loop) + +source /root/fan_speed/functions.sh + +PORT=/dev/ttyS1 +LOG=/root/fan_speed/log/das_fan.log +CHECK_INTERVAL=30 # seconds between temperature readings +KEEPALIVE_INTERVAL=5 # re-send _shutup this often, the DAS forgets it otherwise +LOW_FAN_SPEED=10 +HIGH_FAN_SPEED=60 + +# Disks in the enclosure - anything behind the DAS SAS expander. The server's own +# drives hang off pci-...-scsi-* instead, so this glob does not pick them up. +DISK_GLOB='/dev/disk/by-path/*-sas-exp*-lun-0' +DISK_LOW_TEMPERATURE_THRESHOLD=40 # at/below this -> LOW_FAN_SPEED (idle spread is 32-39c) +DISK_HIGH_TEMPERATURE_THRESHOLD=50 # at/above this -> HIGH_FAN_SPEED + +# Backplane air sensors, only used when no disk will answer (all spun down, smartctl gone) +BP_LOW_TEMPERATURE_THRESHOLD=30 +BP_HIGH_TEMPERATURE_THRESHOLD=45 + +SPEED=$LOW_FAN_SPEED + +# Only one talker on the serial line (detach screen/attachDAS.sh first) +exec 9>/run/das_fanctl.lock +flock -n 9 || { echo "another das_fanctl is running (or /dev/ttyS1 is busy)" >&2; exit 1; } + +stty -F $PORT 38400 raw -echo -echoe -echok -echoctl -echoke + +# Fire and forget - the DAS answers with a bare prompt, nothing worth reading +set_fan() { printf '_shutup %s\r' "$1" > $PORT; } + +# Send a command, return whatever the DAS says back +das_cmd() { + local out; out=$(mktemp) + timeout 3 cat $PORT > "$out" & + local pid=$! + sleep 0.2 + printf '%s\r' "$1" > $PORT + wait $pid + cat "$out"; rm -f "$out" +} + +# Hottest drive-backplane sensor from a _temp_rd readout on stdin. +# ponytail: BP_* only - EXP0 (SAS expander) idles ~59c and would peg the fans, +# SIM0 is the enclosure module. Add them here if drives ever run cool but the DAS cooks. +hottest_bp() { + awk -F'[=c]' '/BP_[0-9]+\[/ {gsub(/ /,"",$2); if ($2+0 > m) m = $2+0} END {print m+0}' +} + +# Hottest drive in the enclosure, 0 if nothing answered. +# -n standby so a sleeping drive is skipped rather than spun up just to be measured. +# SATA reports attribute 194, SAS reports "Current Drive Temperature". +hottest_disk() { + local link dev temp max=0 + for link in $DISK_GLOB; do + [ -e "$link" ] || continue + dev=$(readlink -f "$link") + temp=$(smartctl -n standby -A "$dev" 2>/dev/null | + awk '/Temperature_Celsius/ {print $10; exit} /Current Drive Temperature/ {print $4; exit}') + [ -n "$temp" ] && [ "$temp" -gt "$max" ] 2>/dev/null && max=$temp + done + echo $max +} + +# Read temps, recalculate SPEED, apply it +one_pass() { + local disk bp source temp low high + disk=$(hottest_disk) + if [ "$disk" -gt 0 ]; then + source=disk; temp=$disk + low=$DISK_LOW_TEMPERATURE_THRESHOLD; high=$DISK_HIGH_TEMPERATURE_THRESHOLD + else + bp=$(das_cmd _temp_rd | hottest_bp) + if [ "$bp" -le 0 ] 2>/dev/null; then + echo "$(date +'%Y-%m-%d %H:%M:%S') || no disk or backplane reading, holding fan at ${SPEED}%" >> $LOG + return 1 + fi + source=backplane; temp=$bp + low=$BP_LOW_TEMPERATURE_THRESHOLD; high=$BP_HIGH_TEMPERATURE_THRESHOLD + fi + + SPEED=$(calculate_interpolated_fan_speed "$temp" $low $high $LOW_FAN_SPEED $HIGH_FAN_SPEED) + set_fan "$SPEED" + echo "$(date +'%Y-%m-%d %H:%M:%S') || DAS ${source}:${temp}c | Fan Speed:$SPEED" >> $LOG + echo "DAS ${source}:${temp}c -> fan ${SPEED}%" +} + +selftest() { + local canned=' BP_1[2] = 30c + BP_2[3] = 44c + SIM0[0] = 36c + EXP0[4] = 59c + AVG = 39c' + [ "$(printf '%s' "$canned" | hottest_bp)" = 44 ] || { echo "FAIL: bp parser"; exit 1; } + [ "$(printf '%s' 'garbage' | hottest_bp)" = 0 ] || { echo "FAIL: bp parser on junk"; exit 1; } + [ "$(calculate_interpolated_fan_speed 40 40 50 10 60)" = 10 ] || { echo "FAIL: low clamp"; exit 1; } + [ "$(calculate_interpolated_fan_speed 55 40 50 10 60)" = 60 ] || { echo "FAIL: high clamp"; exit 1; } + [ "$(calculate_interpolated_fan_speed 45 40 50 10 60)" = 35 ] || { echo "FAIL: midpoint"; exit 1; } + [ "$(hottest_disk)" -gt 0 ] || { echo "FAIL: no disk temperatures readable"; exit 1; } + echo "selftest OK" +} + +case "${1:-loop}" in + temps) das_cmd _temp_rd ;; + disks) for l in $DISK_GLOB; do + d=$(readlink -f "$l") + printf '%-12s %s\n' "$d" "$(smartctl -n standby -A "$d" 2>/dev/null | + awk '/Temperature_Celsius/ {print $10"c"; exit} /Current Drive Temperature/ {print $4"c"; exit}')" + done; echo "hottest: $(hottest_disk)c" ;; + once) one_pass ;; + selftest) selftest ;; + loop) while true; do + one_pass + # keepalive between readings - matches the old 5s timer cadence + for _ in $(seq $((CHECK_INTERVAL / KEEPALIVE_INTERVAL))); do + sleep $KEEPALIVE_INTERVAL + set_fan "$SPEED" + done + done ;; + *) echo "usage: $0 [loop|once|temps|disks|selftest]" >&2; exit 1 ;; +esac diff --git a/fan_control.sh b/fan_control.sh new file mode 100755 index 0000000..94aaa9f --- /dev/null +++ b/fan_control.sh @@ -0,0 +1,105 @@ +#!/bin/bash +# +# https://github.com/brezlord/iDRAC7_fan_control +# Modified by Zeb Hering +# A simple script to control fan speeds on Dell generation 12 PowerEdge servers. +# If the inlet temperature is above $TEMP_THRESHOLD C enable iDRAC dynamic control and exit program. +# If inlet temp is below $TEMP_THRESHOLD C set fan control to manual and set fan speed to predetermined value. +# The tower servers T320, T420 & T620 inlet temperature sensor is after the HDDs so temperature will +# be higher than the ambient temperature. + +# Variables +IDRAC_IP="10.1.1.251" +IDRAC_USER="root" +IDRAC_PASSWORD="*^b#!efRhE8qULqUCa4U" +# Fan speed in % +SPEED0="0x00" +SPEED5="0x05" +SPEED10="0x0a" +SPEED15="0x0f" +SPEED20="0x14" +SPEED25="0x19" +SPEED30="0x1e" +SPEED35="0x23" +TEMP_THRESHOLD="65" # iDRAC dynamic control enable threshold +TEMP_SENSOR="0Eh" # CPU 1 Temp +TEMP_SENSOR="0Fh" # CPU 2 Temp +SENSOR_NAME="Sensor" + +if [ "$TEMP_SENSOR" == "04h" ] +then + SENSOR_NAME="Inlet" +elif [ "$TEMP_SENSOR" == "01h" ] +then + SENSOR_NAME="Exhaust" +elif [ "$TEMP_SENSOR" == "0Eh" ] +then + SENSOR_NAME="CPU1" +elif [ "$TEMP_SENSOR" == "0Fh" ] +then + SENSOR_NAME="CPU2" +fi + +# Get system date & time. +DATE=$(date +%d-%m-%Y\ %H:%M:%S) +echo "Date $DATE" + +# Get temperature from iDRAC for CPU1 and CPU2. +T_CPU1=$(ipmitool -I lanplus -H $IDRAC_IP -U $IDRAC_USER -P $IDRAC_PASSWORD sdr type temperature | grep "0Eh" | cut -d"|" -f5 | cut -d" " -f2) +T_CPU2=$(ipmitool -I lanplus -H $IDRAC_IP -U $IDRAC_USER -P $IDRAC_PASSWORD sdr type temperature | grep "0Fh" | cut -d"|" -f5 | cut -d" " -f2) + +echo "--> iDRAC IP Address: $IDRAC_IP" +echo "--> Current CPU1 Temp: $T_CPU1" +echo "--> Current CPU2 Temp: $T_CPU2" + +# Compare the temperatures of CPU1 and CPU2 and set $T to the hotter one. +if [ $(echo "$T_CPU1 > $T_CPU2" | bc) -eq 1 ]; then + T=$T_CPU1 + SENSOR_NAME="CPU1" +else + T=$T_CPU2 + SENSOR_NAME="CPU2" +fi + +echo "--> Hottest Sensor: $SENSOR_NAME - Temp: $T" + +# If ambient temperature is above $TEMP_THRESHOLD C enable dynamic control and exit, if below set manual control. +if [[ $T > $TEMP_THRESHOLD ]] +then + echo "--> Temperature is above Dynamic Threshold - $TEMP_THRESHOLD C" + echo "--> Enabled dynamic fan control" + ipmitool -I lanplus -H $IDRAC_IP -U $IDRAC_USER -P $IDRAC_PASSWORD raw 0x30 0x30 0x01 0x01 + exit 1 +else + echo "--> Temperature is below Dynamic Threshold - $TEMP_THRESHOLD C" + echo "--> Disabled dynamic fan control" + ipmitool -I lanplus -H $IDRAC_IP -U $IDRAC_USER -P $IDRAC_PASSWORD raw 0x30 0x30 0x01 0x00 +fi + +# Set fan speed dependant on ambient temperature if inlet temperaturte is below $TEMP_THRESHOLD C. +# If inlet temperature between 0 and 30deg C then set fans to 10%. +if [ "$T" -ge 0 ] && [ "$T" -le 30 ] +then + echo "--> Setting fan speed to 10%" + ipmitool -I lanplus -H $IDRAC_IP -U $IDRAC_USER -P $IDRAC_PASSWORD raw 0x30 0x30 0x02 0xff $SPEED10 + +# If inlet temperature between 35 and 40deg C then set fans to 15% +elif [ "$T" -ge 35 ] && [ "$T" -le 40 ] +then + echo "--> Setting fan speed to 15%" + ipmitool -I lanplus -H $IDRAC_IP -U $IDRAC_USER -P $IDRAC_PASSWORD raw 0x30 0x30 0x02 0xff $SPEED15 + +# If inlet temperature between 40 and 50deg C then set fans to 20% +elif [ "$T" -ge 40 ] && [ "$T" -le 50 ] +then + echo "--> Setting fan speed to 20%" + ipmitool -I lanplus -H $IDRAC_IP -U $IDRAC_USER -P $IDRAC_PASSWORD raw 0x30 0x30 0x02 0xff $SPEED20 + +# If inlet temperature between 55 and $TEMP_THRESHOLD C then set fans to 25% +elif [ "$T" -ge 51 ] && [ "$T" -le $TEMP_THRESHOLD ] +then + echo "--> Setting fan speed to 25%" + ipmitool -I lanplus -H $IDRAC_IP -U $IDRAC_USER -P $IDRAC_PASSWORD raw 0x30 0x30 0x02 0xff $SPEED25 +fi + + diff --git a/fan_speed.service b/fan_speed.service new file mode 100644 index 0000000..fd3b715 --- /dev/null +++ b/fan_speed.service @@ -0,0 +1,8 @@ +[Unit] +Description=Fan_Speed service + +[Service] +ExecStart=/bin/bash -c /root/fan_speed/fan_speed.sh + +[Install] +WantedBy=multi-user.target diff --git a/fan_speed.sh b/fan_speed.sh new file mode 100755 index 0000000..37a0991 --- /dev/null +++ b/fan_speed.sh @@ -0,0 +1,191 @@ +#!/bin/bash + +source /root/fan_speed/functions.sh + +# Trap the signals for container exit and run graceful_exit function +trap 'graceful_exit' SIGINT SIGQUIT SIGTERM + +# Variable defintions +# LOW_FAN_SPEED = Lowest fan speed +# HIGH_FAN_SPEED = max fan speed. (0-100) +# IDRAC_HOST = local or IP of iDRAC +# IDRAC_USERNAME = username of iDRAC +# IDRAC_PASSWORD = password if iDRAC +# CHECK_INTERVAL = time to wait in seconds before performing another temp check (if doing GPU might want to keep this low like 5 seconds) +# CPU_TEMPERATURE_THRESHOLD = max CPU temp before disabling this automation and setting idrac back to auto +# GPU_TEMPERATURE_THRESHOLD = max GPU TEMP before running a kill on the container running LLM +# TABLE_HEADER_PRINT_INTERVAL = Number of loops before we output temps + + +IDRAC_HOST=local +IDRAC_USERNAME=root +IDRAC_PASSWORD=calvin +CHECK_INTERVAL=10 #takes about 8 seconds to query all data +LOW_TEMPERATURE_THRESHOLD=45 #only decimal numbers +CPU_TEMPERATURE_THRESHOLD=90 #only decimal numbers +GPU_TEMPERATURE_THRESHOLD=75 #only decimal numbers +LOW_FAN_SPEED=18 #only decimal numbers +HIGH_FAN_SPEED=50 #only decimal numbers +TABLE_HEADER_PRINT_INTERVAL=2 + + +####################################### +# Check if the iDRAC host is set to 'local' or not then set the IDRAC_LOGIN_STRING accordingly +if [[ $IDRAC_HOST == "local" ]]; then + # Check that the Docker host IPMI device (the iDRAC) has been exposed to the Docker container + if [ ! -e "/dev/ipmi0" ] && [ ! -e "/dev/ipmi/0" ] && [ ! -e "/dev/ipmidev/0" ]; then + echo "/!\ Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0, check that you added the device to your Docker container or stop using local mode. Exiting." >&2 + exit 1 + fi + IDRAC_LOGIN_STRING='open' +else + echo "iDRAC/IPMI username: $IDRAC_USERNAME" + echo "iDRAC/IPMI password: $IDRAC_PASSWORD" + IDRAC_LOGIN_STRING="lanplus -H $IDRAC_HOST -U $IDRAC_USERNAME -P $IDRAC_PASSWORD" +fi +####################################### + +####################################### +# This script ONLY runs on Dell Servers +get_Dell_server_model +if [[ ! $SERVER_MANUFACTURER == "DELL" ]]; then + echo "/!\ Your server isn't a Dell product. Exiting." >&2 + exit 1 +fi +####################################### + +####################################### +#****Probably don't need this any more**** +# Prepare, format and define initial variables +# Check if LOW_FAN_SPEED variable is in hexadecimal format. If not, convert it to hexadecimal +if [[ $LOW_FAN_SPEED == 0x* ]]; then + readonly DECIMAL_LOW_FAN_SPEED=$(printf '%d' $LOW_FAN_SPEED) + readonly HEXADECIMAL_LOW_FAN_SPEED=$LOW_FAN_SPEED +else + readonly DECIMAL_LOW_FAN_SPEED=$LOW_FAN_SPEED + readonly HEXADECIMAL_LOW_FAN_SPEED=$(convert_decimal_value_to_hexadecimal $LOW_FAN_SPEED) +fi +####################################### + +####################################### +# This returns the lowest temp between CPU and GPU highest thresholds +# Used to calcuate fan speed interpolation +# I picked the lowest temp as the fan speeds will ramp up faster to account for the lower temp +if [ $CPU_TEMPERATURE_THRESHOLD -le $GPU_TEMPERATURE_THRESHOLD ]; then + HIGH_TEMPERATURE_THRESHOLD=$CPU_TEMPERATURE_THRESHOLD +else + HIGH_TEMPERATURE_THRESHOLD=$GPU_TEMPERATURE_THRESHOLD +fi +####################################### + +####################################### +# Determine where to get CPU temps from iDRAC +# If server model is Gen 14 (*40) or newer +if [[ $SERVER_MODEL =~ .*[RT][[:space:]]?[0-9][4-9]0.* ]]; then + DELL_POWEREDGE_GEN_14_OR_NEWER=true + CPU1_TEMPERATURE_INDEX=2 + CPU2_TEMPERATURE_INDEX=4 +else + DELL_POWEREDGE_GEN_14_OR_NEWER=false + CPU1_TEMPERATURE_INDEX=1 + CPU2_TEMPERATURE_INDEX=2 +fi +####################################### + +####################################### +# Check if sensors are present on the server +IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT=true +IS_CPU2_TEMPERATURE_SENSOR_PRESENT=true +retrieve_cpu_temperatures $IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT $IS_CPU2_TEMPERATURE_SENSOR_PRESENT +if [ -z "$EXHAUST_TEMPERATURE" ]; then + echo "No exhaust temperature sensor detected." + IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT=false +fi +if [ -z "$CPU2_TEMPERATURE" ]; then + echo "No CPU2 temperature sensor detected." + IS_CPU2_TEMPERATURE_SENSOR_PRESENT=false +fi +####################################### + +####################################### +# Print out config information before starting +#echo "Server Manufacturer: $SERVER_MANUFACTURER" +#echo "Server Model: $SERVER_MODEL" +#echo "iDRAC Host: $IDRAC_HOST" +#echo "Check Interval: $CHECK_INTERVAL" +#echo "Low Temp Threshold: $LOW_TEMPERATURE_THRESHOLD" +#echo "CPU High Temp threshold: $CPU_TEMPERATURE_THRESHOLD" +#echo "GPU High Temp Threshold: $GPU_TEMPERATURE_THRESHOLD" +#echo "Low Fan Speed: $LOW_FAN_SPEED" +#echo "High Fan Speed: $HIGH_FAN_SPEED" +####################################### + +####################################### +# Loop (this is where the magic happens) +while true; do + # Sleep for the specified interval before taking another reading + sleep $CHECK_INTERVAL & + SLEEP_PROCESS_PID=$! + + #Get Temps + retrieve_cpu_temperatures $IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT $IS_CPU2_TEMPERATURE_SENSOR_PRESENT + retrieve_gpu_temperature + + i=0 + # Get highest CPU Temp and check if any of them are over heating + HIGHEST_CPU_TEMPERATURE=$CPU1_TEMPERATURE + if $IS_CPU2_TEMPERATURE_SENSOR_PRESENT; then + if [ $CPU2_TEMPERATURE -gt $CPU1_TEMPERATURE ]; then + HIGHEST_CPU_TEMPERATURE=$CPU2_TEMPERATURE + fi + fi + + #echo $HIGHEST_CPU_TEMPERATURE + + + # Define Function to check for if CPU temps are greater than the threshold + function CPU_OVERHEAT() { [ $HIGHEST_CPU_TEMPERATURE -gt $CPU_TEMPERATURE_THRESHOLD ]; } + + # Define function to check if GPU temperature are above the threshold + function GPU_OVERHEAT() { [ $GPU_TEMPERATURE -gt $GPU_TEMPERATURE_THRESHOLD ]; } + + # Check if CPU is overheating then apply Dell default dynamic fan control profile if true + # or set CPU fan based on interpolated temps + if CPU_OVERHEAT; then + apply_Dell_fan_control_profile + if ! $IS_DELL_FAN_CONTROL_PROFILE_APPLIED; then + IS_DELL_FAN_CONTROL_PROFILE_APPLIED=true + COMMENT="CPU temperature is too high, Dell default dynamic fan control profile applied for safety" + fi + else + #this combines CPU and GPU temps and outputs which one is highest + HIGHEST_TEMPERATURE=$HIGHEST_CPU_TEMPERATURE + if [ $GPU_TEMPERATURE -gt $HIGHEST_CPU_TEMPERATURE ]; then + HIGHEST_TEMPERATURE=$GPU_TEMPERATURE + fi + if [ "$HIGHEST_TEMPERATURE" -gt "$LOW_TEMPERATURE_THRESHOLD" ]; then + DECIMAL_CURRENT_FAN_SPEED=$(calculate_interpolated_fan_speed $HIGHEST_TEMPERATURE $LOW_TEMPERATURE_THRESHOLD $HIGH_TEMPERATURE_THRESHOLD $LOW_FAN_SPEED $HIGH_FAN_SPEED) + else + DECIMAL_CURRENT_FAN_SPEED=$DECIMAL_LOW_FAN_SPEED + fi + apply_user_fan_control "$DECIMAL_CURRENT_FAN_SPEED" + COMMENT=$(echo "CPU1:$CPU1_TEMPERATURE | CPU2:$CPU2_TEMPERATURE | GPU:$GPU_TEMPERATURE | Inlet:$INLET_TEMPERATURE | Exhaust:$EXHAUST_TEMPERATURE | Fan Speed:$DECIMAL_CURRENT_FAN_SPEED") + fi + + if GPU_OVERHEAT; then + COMMENT="GPU temperature is too high, killing CT(s) to prevent GPU damage" + # This will eventually have a function to kill the CT(s) that use the Tesla P4 + fi + + # Print temperatures, active fan control profile and comment if any change happened during last time interval + if [ $i -eq $TABLE_HEADER_PRINT_INTERVAL ]; then + #echo " ------- Temperatures -------" + #$echo " Date & time Inlet CPU 1 CPU 2 Exhaust Active fan speed profile Third-party PCIe card Dell default cooling response Comment" + i=0 + fi + #printf "%19s %3d°C %3d°C %3s°C %5s°C %40s %51s %s\n" "$(date +"%d-%m-%Y %T")" $INLET_TEMPERATURE $CPU1_TEMPERATURE "$CPU2_TEMPERATURE" "$EXHAUST_TEMPERATURE" "$CURRENT_FAN_CONTROL_PROFILE" "$THIRD_PARTY_PCIE_CARD_DELL_DEFAULT_COOLING_RESPONSE_STATUS" "$COMMENT" + z_date=$(date +"%Y-%m-%d %H:%M:%S") + echo "$z_date || $COMMENT" >> /root/fan_speed/log/fan_speed.log + ((i++)) + wait $SLEEP_PROCESS_PID +done diff --git a/fan_speed_das.service b/fan_speed_das.service new file mode 100644 index 0000000..1d312dd --- /dev/null +++ b/fan_speed_das.service @@ -0,0 +1,12 @@ +[Unit] +Description=Fan Speed Script - izebra (Dell MD1200 DAS) + +[Service] +User=root +WorkingDirectory=/root/fan_speed/ +ExecStart=/root/fan_speed/das_fanctl.sh +Restart=always +RestartSec=10 + +[Install] +WantedBy=multi-user.target diff --git a/functions.sh b/functions.sh new file mode 100755 index 0000000..fc366e4 --- /dev/null +++ b/functions.sh @@ -0,0 +1,215 @@ +# Define global functions +# This function applies Dell's default dynamic fan control profile +function apply_Dell_fan_control_profile() { + # Use ipmitool to send the raw command to set fan control to Dell default + ipmitool -I $IDRAC_LOGIN_STRING raw 0x30 0x30 0x01 0x01 > /dev/null + CURRENT_FAN_CONTROL_PROFILE="Dell default dynamic fan control profile" +} + +#Get the TEMP of the TESLA P4 installed in this servers +#Should probably set a variable so this function can check temps of any GPU using nvidia-smi +retrieve_gpu_temperature() { + if [ -x /usr/bin/nvidia-smi ]; then + # Get the index of the Tesla P4 GPU using nvidia-smi + index=$(nvidia-smi --query-gpu=name,index --format=csv | grep "Tesla P4" | cut -d ',' -f 2) + else + index="" + fi + + if [ "$index" != "" ]; then + # Get the temperature of the Tesla P4 GPU using nvidia-smi and the index obtained above + GPU_TEMPERATURE=$(nvidia-smi --query-gpu=temperature.gpu --format=csv | head -$(( $(($index + 2)) )) | tail -1) + else + GPU_TEMPERATURE="0" + fi +} + +# Apply user-defined fan control settings +# +# This function applies user-defined fan control settings based on the fan speed. +# It handles both decimal and hexadecimal fan speed inputs, converting between them as needed. +# The function then applies the fan control and updates the current fan control profile. +# +# Parameters: +# $1 (LOCAL_FAN_SPEED): The desired fan speed. Can be in decimal (0-100) or hexadecimal (0x00-0x64) format. +# +# Global variables used: +# CURRENT_FAN_CONTROL_PROFILE: Updated with the current fan control profile description. +# +# Returns: +# None. +function apply_user_fan_control() { + local LOCAL_FAN_SPEED=$1 + + if [[ $LOCAL_FAN_SPEED == 0x* ]]; then + local LOCAL_DECIMAL_FAN_SPEED + LOCAL_DECIMAL_FAN_SPEED=$(printf '%d' "$LOCAL_FAN_SPEED") + local LOCAL_HEXADECIMAL_FAN_SPEED=$LOCAL_FAN_SPEED + else + local LOCAL_DECIMAL_FAN_SPEED=$LOCAL_FAN_SPEED + local LOCAL_HEXADECIMAL_FAN_SPEED + LOCAL_HEXADECIMAL_FAN_SPEED=$(convert_decimal_value_to_hexadecimal "$LOCAL_FAN_SPEED") + fi + + apply_fan_control_to_specified_value "$LOCAL_HEXADECIMAL_FAN_SPEED" + CURRENT_FAN_CONTROL_PROFILE="User static fan control profile ($LOCAL_DECIMAL_FAN_SPEED%)" +} + +# Apply fan control to a specified value +# +# This function sets the fan speed to a user-specified value using ipmitool. +# It first checks if the input value is in hexadecimal format, and converts it +# if necessary. Then it sends raw commands to iDRAC to set the fan control. +# +# Parameters: +# $1 (VALUE): The desired fan speed value. Can be in decimal or hexadecimal format. +# If in decimal, it will be converted to hexadecimal. +# +# Returns: +# None +# +# Note: +# This function uses the global variable $IDRAC_LOGIN_STRING for iDRAC login. +function apply_fan_control_to_specified_value() { + local VALUE=$1 + + # Check if the input value is a hexadecimal number, if not, convert it to hexadecimal + if [[ $VALUE != 0x* ]]; then + VALUE=$(convert_decimal_value_to_hexadecimal "$VALUE") + fi + + # Use ipmitool to send the raw command to set fan control to user-specified value + ipmitool -I $IDRAC_LOGIN_STRING raw 0x30 0x30 0x01 0x00 > /dev/null + ipmitool -I $IDRAC_LOGIN_STRING raw 0x30 0x30 0x02 0xff "$VALUE" > /dev/null +} + +# Calculate the interpolated fan speed based on CPU temperature +# +# This function calculates the interpolated fan speed based on the current CPU temperature +# and predefined thresholds. It uses linear interpolation to adjust the fan speed +# within a specified range when the CPU temperature exceeds a certain threshold. +# +# Parameters: +# $1 (HIGHEST_TEMPERATURE): The current highest CPU/GPU temperature (in Celsius) +# $2 (TEMPERATURE_THRESHOLD_FOR_FAN_SPEED_INTERPOLATION): The lower temperature threshold for fan speed interpolation (in Celsius) +# $3 (TEMPERATURE_THRESHOLD): The upper temperature threshold for fan speed interpolation (in Celsius) +# $4 (LOCAL_DECIMAL_FAN_SPEED): The base fan speed (as a decimal percentage, 0-100) +# $5 (LOCAL_DECIMAL_HIGH_FAN_SPEED): The maximum fan speed (as a decimal percentage, 0-100) +# +# Returns: +# The calculated interpolated fan speed as a decimal percentage (0-100) +# If the temperature is below or equal to the lower threshold, returns the base fan speed +# If the temperature is above or equal to the upper threshold, returns the maximum fan speed +# +# Usage: +# calculate_interpolated_fan_speed +# +# Example: +# calculate_interpolated_fan_speed 70 60 80 30 100 +function calculate_interpolated_fan_speed() { + local HIGHEST_CPU_TEMPERATURE=$1 + local CPU_TEMPERATURE_THRESHOLD_FOR_FAN_SPEED_INTERPOLATION=$2 + local CPU_TEMPERATURE_THRESHOLD=$3 + local LOCAL_DECIMAL_FAN_SPEED=$4 + local LOCAL_DECIMAL_HIGH_FAN_SPEED=$5 + + # If temperature is below or equal to the lower threshold, return the base fan speed + if [ "$HIGHEST_CPU_TEMPERATURE" -le "$CPU_TEMPERATURE_THRESHOLD_FOR_FAN_SPEED_INTERPOLATION" ]; then + echo "$LOCAL_DECIMAL_FAN_SPEED" + return + fi + + # If temperature is above or equal to the upper threshold, return the max fan speed + if [ "$HIGHEST_CPU_TEMPERATURE" -ge "$CPU_TEMPERATURE_THRESHOLD" ]; then + echo "$LOCAL_DECIMAL_HIGH_FAN_SPEED" + return + fi + + # F1 - lower fan speed + # F2 - higher fan speed + # T_CPU - highest temperature of both CPUs (if only one exists that will be CPU1 temp value) + # T1 - lower temperature threshold + # T2 - higher temperature threshold + # Fan speed = F1 + ( ( F2 - F1 ) * ( T_CPU - T1 ) / ( T2 - T1 ) ) + + local TEMPERATURE_INTERPOLATION_ACTIVATION_RANGE=$((CPU_TEMPERATURE_THRESHOLD - CPU_TEMPERATURE_THRESHOLD_FOR_FAN_SPEED_INTERPOLATION)) + local FAN_VALUE_TO_ADD=0 + + if [ $TEMPERATURE_INTERPOLATION_ACTIVATION_RANGE -gt $FAN_VALUE_TO_ADD ]; then + local TEMPERATURE_ABOVE_THRESHOLD_FOR_FAN_SPEED_INTERPOLATION=$((HIGHEST_CPU_TEMPERATURE - CPU_TEMPERATURE_THRESHOLD_FOR_FAN_SPEED_INTERPOLATION)) + local FAN_WINDOW=$((LOCAL_DECIMAL_HIGH_FAN_SPEED - LOCAL_DECIMAL_FAN_SPEED)) + FAN_VALUE_TO_ADD=$((FAN_WINDOW * TEMPERATURE_ABOVE_THRESHOLD_FOR_FAN_SPEED_INTERPOLATION / TEMPERATURE_INTERPOLATION_ACTIVATION_RANGE)) + fi + + local DECIMAL_CURRENT_FAN_SPEED=$((LOCAL_DECIMAL_FAN_SPEED + FAN_VALUE_TO_ADD)) + echo $DECIMAL_CURRENT_FAN_SPEED +} + + +# Convert first parameter given ($DECIMAL_NUMBER) to hexadecimal +# Usage : convert_decimal_value_to_hexadecimal $DECIMAL_NUMBER +# Returns : hexadecimal value of DECIMAL_NUMBER +function convert_decimal_value_to_hexadecimal () { + local DECIMAL_NUMBER=$1 + local HEXADECIMAL_NUMBER=$(printf '0x%02x' $DECIMAL_NUMBER) + echo $HEXADECIMAL_NUMBER +} + +# Retrieve temperature sensors data using ipmitool +# Usage : retrieve_temperatures $IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT $IS_CPU2_TEMPERATURE_SENSOR_PRESENT +function retrieve_cpu_temperatures() { + if (( $# != 2 )); then + printf "Illegal number of parameters.\nUsage: retrieve_temperatures \$IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT \$IS_CPU2_TEMPERATURE_SENSOR_PRESENT" >&2 + return 1 + fi + local IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT=$1 + local IS_CPU2_TEMPERATURE_SENSOR_PRESENT=$2 + + local DATA=$(ipmitool -I $IDRAC_LOGIN_STRING sdr type temperature | grep degrees) + + # Parse CPU data + local CPU_DATA=$(echo "$DATA" | grep "3\." | grep -Po '\d{2}') + CPU1_TEMPERATURE=$(echo $CPU_DATA | awk "{print \$$CPU1_TEMPERATURE_INDEX;}") + if $IS_CPU2_TEMPERATURE_SENSOR_PRESENT; then + CPU2_TEMPERATURE=$(echo $CPU_DATA | awk "{print \$$CPU2_TEMPERATURE_INDEX;}") + else + CPU2_TEMPERATURE="-" + fi + + # Parse inlet temperature data + INLET_TEMPERATURE=$(echo "$DATA" | grep Inlet | grep -Po '\d{2}' | tail -1) + + # If exhaust temperature sensor is present, parse its temperature data + if $IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT; then + EXHAUST_TEMPERATURE=$(echo "$DATA" | grep Exhaust | grep -Po '\d{2}' | tail -1) + else + EXHAUST_TEMPERATURE="-" + fi +} + +# Prepare traps in case of container exit +function graceful_exit() { + apply_Dell_fan_control_profile + + echo "/!\ WARNING /!\ Container stopped, Dell default dynamic fan control profile applied for safety." + exit 0 +} + +# Helps debugging when people are posting their output +function get_Dell_server_model() { + IPMI_FRU_content=$(ipmitool -I $IDRAC_LOGIN_STRING fru 2>/dev/null) # FRU stands for "Field Replaceable Unit" + # TODO - Add check if connection was established. There are a chance user type wrong login and pass. In my case it returns "Error: Unable to establish IPMI v2 / RMCP+ session" + + SERVER_MANUFACTURER=$(echo "$IPMI_FRU_content" | grep "Product Manufacturer" | awk -F ': ' '{print $2}') + SERVER_MODEL=$(echo "$IPMI_FRU_content" | grep "Product Name" | awk -F ': ' '{print $2}') + + # Check if SERVER_MANUFACTURER is empty, if yes, assign value based on "Board Mfg" + if [ -z "$SERVER_MANUFACTURER" ]; then + SERVER_MANUFACTURER=$(echo "$IPMI_FRU_content" | tr -s ' ' | grep "Board Mfg :" | awk -F ': ' '{print $2}') + fi + + # Check if SERVER_MODEL is empty, if yes, assign value based on "Board Product" + if [ -z "$SERVER_MODEL" ]; then + SERVER_MODEL=$(echo "$IPMI_FRU_content" | tr -s ' ' | grep "Board Product :" | awk -F ': ' '{print $2}') + fi +} \ No newline at end of file diff --git a/get_das_temp.sh b/get_das_temp.sh new file mode 100755 index 0000000..7b5db60 --- /dev/null +++ b/get_das_temp.sh @@ -0,0 +1,10 @@ +#/bin/bash + +#Zeb H. +#Dell MD1200 Temperature Reading Script. + +#set TTY Flags +stty -F /dev/ttyS1 38400 raw -echoe -echok -echoctl -echoke + +#Print temperature on screen +echo -e -n '_temp_rd\r' > /dev/ttyS1 diff --git a/log_rotate.txt b/log_rotate.txt new file mode 100755 index 0000000..076ae21 --- /dev/null +++ b/log_rotate.txt @@ -0,0 +1,8 @@ +Create a file in the /etc/logrotate.d/ folder with the following text + +/root/fan_speed/log/fan_speed.log{ + rotate 30 + hourly + missingok + notifempty +}