Add disk temperature monitoring, trend tracking and email alarms
Fan speed now takes the loudest request from CPU/GPU and from each drive interpolated against its own reported temperature limit, so bay position and drive technology stop mattering. Reported limits are clamped by class because WD Reds report the SCT critical limit (85) rather than an operating maximum. Adds a rolling temperature history and four alarms (absolute, rising trend, SMART health, disk count), rate limited per key with a one hour cooldown, delivered by mail through the host relay. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Zeb Hering
163
README.md
Normal file
163
README.md
Normal file
@@ -0,0 +1,163 @@
|
|||||||
|
# fan_speed
|
||||||
|
|
||||||
|
Fan control for Dell PowerEdge servers over IPMI, driven by CPU, GPU **and disk**
|
||||||
|
temperature, with trend tracking and email alarms.
|
||||||
|
|
||||||
|
Upstream ([tigerblue77/Dell_iDRAC_fan_controller_Docker](https://github.com/tigerblue77/Dell_iDRAC_fan_controller_Docker))
|
||||||
|
sets fan speed from CPU and GPU only. On a chassis with two dozen drives that misses the
|
||||||
|
thing most likely to be quietly cooking, so this fork adds disks to the curve, records a
|
||||||
|
rolling temperature history, and mails when something looks wrong.
|
||||||
|
|
||||||
|
For the MD1200 disk shelf, see the separate
|
||||||
|
[md1200-fan-control](https://git.izebra.net/izebra_projects/md1200-fan-control) repo —
|
||||||
|
that enclosure has its own controller and its own serial protocol.
|
||||||
|
|
||||||
|
## How the fan speed is decided
|
||||||
|
|
||||||
|
Each heat source asks for a fan speed and the loudest request wins:
|
||||||
|
|
||||||
|
```
|
||||||
|
speed = max( interpolate(hottest of CPU/GPU, 45 -> 75),
|
||||||
|
per-drive interpolation against each drive's own limit )
|
||||||
|
```
|
||||||
|
|
||||||
|
Sources are never compared as raw temperatures — an 85c CPU and a 45c disk are both
|
||||||
|
"fine", and a single `max()` over the numbers would be meaningless.
|
||||||
|
|
||||||
|
### Why disks are measured against their own limits
|
||||||
|
|
||||||
|
Every drive reports its own maximum operating temperature (SATA: `Min/Max Temperature
|
||||||
|
Limit`; SAS: `Drive Trip Temperature`). Each drive's ramp is derived from that number:
|
||||||
|
|
||||||
|
```
|
||||||
|
ramp starts at limit - DISK_RAMP_LOW_OFFSET (default 18)
|
||||||
|
full speed at limit - DISK_RAMP_HIGH_OFFSET (default 8)
|
||||||
|
```
|
||||||
|
|
||||||
|
So a Samsung SSD rated to 70c ramps 52→62, and a Toshiba spinner rated to 60c ramps
|
||||||
|
42→52. The 50c SSD asks for *less* airflow than the 45c HDD, which is correct — it is
|
||||||
|
further from its own limit.
|
||||||
|
|
||||||
|
This matters because **temperature does not tell you what you think it does**. On these
|
||||||
|
servers the rear-bay SSDs idle 10c hotter than the front-bay spinners: they sit in
|
||||||
|
preheated exhaust air. A shared threshold would peg the fans for drives that are fine
|
||||||
|
and ignore the ones that are not. Measuring each drive against its own envelope makes
|
||||||
|
the curve independent of both drive technology and bay position.
|
||||||
|
|
||||||
|
**Reported limits are clamped by class.** They are not uniformly trustworthy — Samsung
|
||||||
|
and Kioxia report a real operating maximum (70), Toshiba and Seagate report 60, and WD
|
||||||
|
Reds report **85**, which is the SCT critical limit and not somewhere you want a drive
|
||||||
|
living. `HDD_LIMIT_CAP` (60) and `SSD_LIMIT_CAP` (70) bound whatever the drive claims,
|
||||||
|
and supply the value when a drive reports nothing.
|
||||||
|
|
||||||
|
Drive class comes from `/sys/block/<dev>/queue/rotational`. That is derived from the
|
||||||
|
device's RPM flag, which a few SAS drives behind HBAs report incorrectly; if you hit one,
|
||||||
|
SMART's `Rotation Rate` field is the fallback.
|
||||||
|
|
||||||
|
## Alarms
|
||||||
|
|
||||||
|
Email via `mail` to `ALERT_EMAIL`, which the host's postfix relays. Four conditions:
|
||||||
|
|
||||||
|
| Alarm | Fires when |
|
||||||
|
|---|---|
|
||||||
|
| `disk_temp` | A drive is within `DISK_ALARM_OFFSET` (5c) of its own limit |
|
||||||
|
| `trend` | CPU or hottest disk has climbed `TREND_RISE_ALARM` (8c) across the window while still under every threshold |
|
||||||
|
| `smart` | `smartctl -H` reports anything other than PASSED/OK |
|
||||||
|
| `cpu` / `gpu` | Threshold crossed; `cpu` also means fan control was handed back to Dell's profile |
|
||||||
|
| `disk_count` | Fewer drives answered than were present at startup |
|
||||||
|
|
||||||
|
**The trend alarm is the one worth having.** A dying fan or a blocked intake shows up as
|
||||||
|
a steady climb long before anything crosses a threshold — by the time an absolute alarm
|
||||||
|
fires you have already been running hot for hours.
|
||||||
|
|
||||||
|
Every alarm is **rate limited per key** with a one hour cooldown, and sends a single
|
||||||
|
recovery notice when it clears. On a 10s loop an un-throttled alarm sends 360 emails an
|
||||||
|
hour, at which point the alarm is the outage.
|
||||||
|
|
||||||
|
`smartctl -H` across two dozen drives runs hourly, not per loop. Absolute and trend
|
||||||
|
checks stay on the fast loop.
|
||||||
|
|
||||||
|
## Trend history
|
||||||
|
|
||||||
|
Every pass appends to `log/temps.csv`:
|
||||||
|
|
||||||
|
```
|
||||||
|
epoch,cpu,gpu,hottest_disk,fan_speed
|
||||||
|
```
|
||||||
|
|
||||||
|
Trimmed to `TREND_SAMPLES` (90 = 15 minutes at a 10s interval). The trend check compares
|
||||||
|
the newest sample to the oldest in that window, and reports zero until a full window has
|
||||||
|
accumulated so a restart cannot alarm on a partial series. Flat file, `tail` to trim — no
|
||||||
|
rrdtool, no database.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```
|
||||||
|
fan_speed.sh # the service loop (default)
|
||||||
|
fan_speed.sh once # a single pass, prints what it decided
|
||||||
|
fan_speed.sh disks # every drive: temperature and the limit in use
|
||||||
|
fan_speed.sh selftest # parsers, curve, trend detector, alert rate limiting
|
||||||
|
```
|
||||||
|
|
||||||
|
`selftest` touches no hardware and sends no mail — it runs the pure logic against canned
|
||||||
|
smartctl output and a synthetic history file. Run it after any edit.
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```sh
|
||||||
|
install -m 755 fan_speed.sh functions.sh monitor.sh /root/fan_speed/
|
||||||
|
mkdir -p /root/fan_speed/log /root/fan_speed/state
|
||||||
|
install -m 644 fan_speed.service /etc/systemd/system/
|
||||||
|
systemctl daemon-reload
|
||||||
|
systemctl enable --now fan_speed.service
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tuning
|
||||||
|
|
||||||
|
| Variable | Default | |
|
||||||
|
|---|---|---|
|
||||||
|
| `CHECK_INTERVAL` | `10` | Seconds between passes |
|
||||||
|
| `LOW_FAN_SPEED` / `HIGH_FAN_SPEED` | `18` / `50` | Percent |
|
||||||
|
| `LOW_TEMPERATURE_THRESHOLD` | `45` | CPU/GPU ramp start |
|
||||||
|
| `CPU_TEMPERATURE_THRESHOLD` | `90` | Above this, Dell's profile takes over |
|
||||||
|
| `GPU_TEMPERATURE_THRESHOLD` | `75` | |
|
||||||
|
| `DISK_RAMP_LOW_OFFSET` | `18` | Ramp starts this far below each drive's limit |
|
||||||
|
| `DISK_RAMP_HIGH_OFFSET` | `8` | Full speed this far below it |
|
||||||
|
| `DISK_ALARM_OFFSET` | `5` | Alarm this far below it |
|
||||||
|
| `HDD_LIMIT_CAP` / `SSD_LIMIT_CAP` | `60` / `70` | Ceiling on what a drive may claim |
|
||||||
|
| `TREND_SAMPLES` | `90` | Window length, in passes |
|
||||||
|
| `TREND_RISE_ALARM` | `8` | Degrees of climb that alarms |
|
||||||
|
| `ALERT_EMAIL` | `Servers@ntfy1.izebra.xyz` | |
|
||||||
|
| `ALERT_COOLDOWN` | `3600` | Seconds between repeats of one alarm |
|
||||||
|
| `SMART_CHECK_INTERVAL` | `3600` | Seconds between SMART sweeps |
|
||||||
|
|
||||||
|
`DISK_RAMP_LOW_OFFSET` is the knob to reach for first. At the default 18 an SSD rated to
|
||||||
|
70c starts ramping at 52c; raise the offset to react earlier and louder, lower it to stay
|
||||||
|
quiet longer. It is set where it is because the hottest drive on `iz-pve1` idles at 47c —
|
||||||
|
close enough that a smaller offset would have the fans tracking normal daily drift, and
|
||||||
|
you would lose the ability to tell "disks are warm" from "disks are fine".
|
||||||
|
|
||||||
|
Those defaults were sized against one chassis. Watch a day of `log/temps.csv` before
|
||||||
|
trusting them anywhere else.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Kernel device names are not stable — a shelf rescan renamed `sdaa`–`sdai` to
|
||||||
|
`sds`–`sdaa` mid-session. Everything resolves through `/dev/disk/by-path` on every
|
||||||
|
pass; never persist an `sdX`.
|
||||||
|
- `DISK_GLOB` matches internal drives only (`pci-*-scsi-*`). Drives behind a SAS expander
|
||||||
|
are a separate enclosure with separate cooling and are deliberately excluded.
|
||||||
|
- Drive temperature limits are read once at startup. They do not change, and `smartctl -x`
|
||||||
|
is far heavier than the `-A` used on the fast loop.
|
||||||
|
- `smartctl -n standby` throughout, so a sleeping drive is skipped rather than spun up
|
||||||
|
just to be measured. That is also why `disk_count` alarms on "fewer drives answered"
|
||||||
|
rather than on a device disappearing.
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
| File | |
|
||||||
|
|---|---|
|
||||||
|
| `fan_speed.sh` | Main loop, fan speed decision, alarm conditions, selftest |
|
||||||
|
| `monitor.sh` | Disk reading, trend history, alarm delivery — all local code |
|
||||||
|
| `functions.sh` | Vendored upstream: IPMI, iDRAC, interpolation |
|
||||||
|
| `fan_speed.service` | systemd unit |
|
||||||
8
fan_speed.service
Normal file
8
fan_speed.service
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Fan_Speed service
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
ExecStart=/bin/bash -c /root/fan_speed/fan_speed.sh
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
314
fan_speed.sh
Executable file
314
fan_speed.sh
Executable file
@@ -0,0 +1,314 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
#
|
||||||
|
# Dell PowerEdge fan control driven by CPU, GPU and disk temperature, with
|
||||||
|
# trend tracking and email alarms.
|
||||||
|
#
|
||||||
|
# Usage: fan_speed.sh [run|once|disks|selftest] (default: run)
|
||||||
|
|
||||||
|
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||||
|
source "$SCRIPT_DIR/functions.sh"
|
||||||
|
source "$SCRIPT_DIR/monitor.sh"
|
||||||
|
|
||||||
|
# Trap the signals for container exit and run graceful_exit function
|
||||||
|
trap 'graceful_exit' SIGINT SIGQUIT SIGTERM
|
||||||
|
|
||||||
|
# Variable defintions
|
||||||
|
# LOW_FAN_SPEED = Lowest fan speed
|
||||||
|
# HIGH_FAN_SPEED = max fan speed. (0-100)
|
||||||
|
# IDRAC_HOST = local or IP of iDRAC
|
||||||
|
# IDRAC_USERNAME = username of iDRAC
|
||||||
|
# IDRAC_PASSWORD = password if iDRAC
|
||||||
|
# CHECK_INTERVAL = time to wait in seconds before performing another temp check (if doing GPU might want to keep this low like 5 seconds)
|
||||||
|
# CPU_TEMPERATURE_THRESHOLD = max CPU temp before disabling this automation and setting idrac back to auto
|
||||||
|
# GPU_TEMPERATURE_THRESHOLD = max GPU TEMP before running a kill on the container running LLM
|
||||||
|
# TABLE_HEADER_PRINT_INTERVAL = Number of loops before we output temps
|
||||||
|
|
||||||
|
IDRAC_HOST=local
|
||||||
|
IDRAC_USERNAME=root
|
||||||
|
IDRAC_PASSWORD=${IDRAC_PASSWORD:-calvin}
|
||||||
|
CHECK_INTERVAL=10 #takes about 8 seconds to query all data
|
||||||
|
LOW_TEMPERATURE_THRESHOLD=45 #only decimal numbers
|
||||||
|
CPU_TEMPERATURE_THRESHOLD=90 #only decimal numbers
|
||||||
|
GPU_TEMPERATURE_THRESHOLD=75 #only decimal numbers
|
||||||
|
LOW_FAN_SPEED=18 #only decimal numbers
|
||||||
|
HIGH_FAN_SPEED=50 #only decimal numbers
|
||||||
|
TABLE_HEADER_PRINT_INTERVAL=2
|
||||||
|
|
||||||
|
# Disk thresholds, alarm routing and trend window all live in monitor.sh.
|
||||||
|
|
||||||
|
#######################################
|
||||||
|
# Self test - pure logic only, no IPMI and no drives touched.
|
||||||
|
selftest() {
|
||||||
|
local tmp; tmp=$(mktemp -d)
|
||||||
|
export STATE_DIR="$tmp/state" TREND_FILE="$tmp/temps.csv" LOG_FILE="$tmp/log" DRY_RUN=1
|
||||||
|
local fail=0
|
||||||
|
check() { # check <description> <expected> <actual>
|
||||||
|
if [ "$2" = "$3" ]; then echo " ok $1"; else echo " FAIL $1: expected '$2' got '$3'"; fail=1; fi
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "temperature parsing:"
|
||||||
|
check "SATA Temperature_Celsius" 35 \
|
||||||
|
"$(printf '194 Temperature_Celsius 0x0022 115 099 000 Old_age Always - 35\n' | parse_disk_temperature)"
|
||||||
|
check "Seagate Airflow_Temperature_Cel" 41 \
|
||||||
|
"$(printf '190 Airflow_Temperature_Cel 0x0032 059 051 000 Old_age Always - 41\n' | parse_disk_temperature)"
|
||||||
|
check "Intel Temperature_Internal" 46 \
|
||||||
|
"$(printf '194 Temperature_Internal 0x0022 100 100 000 Old_age Always - 46\n' | parse_disk_temperature)"
|
||||||
|
check "SAS Current Drive Temperature" 46 \
|
||||||
|
"$(printf 'Current Drive Temperature: 46 C\n' | parse_disk_temperature)"
|
||||||
|
check "Temperature_Difference ignored" "" \
|
||||||
|
"$(printf '190 Temperature_Difference_from_100 0x0022 059 051 000 Old_age Always - 41\n' | parse_disk_temperature)"
|
||||||
|
check "no temperature reported" "" "$(printf 'Device Model: whatever\n' | parse_disk_temperature)"
|
||||||
|
|
||||||
|
echo "limit parsing:"
|
||||||
|
check "SATA Min/Max limit" 70 "$(printf 'Min/Max Temperature Limit: 0/70 Celsius\n' | parse_disk_limit)"
|
||||||
|
check "SATA negative min" 85 "$(printf 'Min/Max Temperature Limit: -41/85 Celsius\n' | parse_disk_limit)"
|
||||||
|
check "SAS trip temperature" 60 "$(printf 'Drive Trip Temperature: 60 C\n' | parse_disk_limit)"
|
||||||
|
|
||||||
|
echo "fan curve (limit 70 -> ramp 52..62, limit 60 -> ramp 42..52):"
|
||||||
|
check "SSD at 47c is below its ramp" 18 "$(calculate_interpolated_fan_speed 47 52 62 18 50)"
|
||||||
|
check "SSD at 57c is mid ramp" 34 "$(calculate_interpolated_fan_speed 57 52 62 18 50)"
|
||||||
|
check "SSD at 62c is flat out" 50 "$(calculate_interpolated_fan_speed 62 52 62 18 50)"
|
||||||
|
check "HDD at 47c is mid ramp" 34 "$(calculate_interpolated_fan_speed 47 42 52 18 50)"
|
||||||
|
|
||||||
|
echo "trend detection:"
|
||||||
|
TREND_SAMPLES=5
|
||||||
|
: > "$TREND_FILE"
|
||||||
|
for t in 40 40 41 40 40; do echo "0,50,0,$t,20" >> "$TREND_FILE"; done
|
||||||
|
check "flat series does not alarm" 0 "$(rise_over_window 4)"
|
||||||
|
: > "$TREND_FILE"
|
||||||
|
for t in 38 41 44 47 50; do echo "0,50,0,$t,20" >> "$TREND_FILE"; done
|
||||||
|
check "rising series reports climb" 12 "$(rise_over_window 4)"
|
||||||
|
: > "$TREND_FILE"
|
||||||
|
echo "0,50,0,38,20" >> "$TREND_FILE"
|
||||||
|
check "partial window stays silent" 0 "$(rise_over_window 4)"
|
||||||
|
|
||||||
|
echo "alert rate limiting:"
|
||||||
|
local out
|
||||||
|
out=$( { raise_alert testkey "first" "body"; raise_alert testkey "second" "body"; } | grep -c '^MAIL\[' )
|
||||||
|
check "second alert inside cooldown suppressed" 1 "$out"
|
||||||
|
out=$(clear_alert testkey "recovered" | grep -c '^MAIL\[')
|
||||||
|
check "recovery notice sent once" 1 "$out"
|
||||||
|
out=$(clear_alert testkey "recovered" | grep -c '^MAIL\[')
|
||||||
|
check "recovery not repeated" 0 "$out"
|
||||||
|
|
||||||
|
rm -rf "$tmp"
|
||||||
|
[ $fail -eq 0 ] && echo "selftest OK" || { echo "selftest FAILED"; return 1; }
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ "$1" = "selftest" ]; then selftest; exit $?; fi
|
||||||
|
#######################################
|
||||||
|
|
||||||
|
#######################################
|
||||||
|
# Check if the iDRAC host is set to 'local' or not then set the IDRAC_LOGIN_STRING accordingly
|
||||||
|
if [[ $IDRAC_HOST == "local" ]]; then
|
||||||
|
# Check that the Docker host IPMI device (the iDRAC) has been exposed to the Docker container
|
||||||
|
if [ ! -e "/dev/ipmi0" ] && [ ! -e "/dev/ipmi/0" ] && [ ! -e "/dev/ipmidev/0" ]; then
|
||||||
|
echo "/!\ Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0, check that you added the device to your Docker container or stop using local mode. Exiting." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
IDRAC_LOGIN_STRING='open'
|
||||||
|
else
|
||||||
|
echo "iDRAC/IPMI username: $IDRAC_USERNAME"
|
||||||
|
IDRAC_LOGIN_STRING="lanplus -H $IDRAC_HOST -U $IDRAC_USERNAME -P $IDRAC_PASSWORD"
|
||||||
|
fi
|
||||||
|
#######################################
|
||||||
|
|
||||||
|
#######################################
|
||||||
|
# This script ONLY runs on Dell Servers
|
||||||
|
get_Dell_server_model
|
||||||
|
if [[ ! $SERVER_MANUFACTURER == "DELL" ]]; then
|
||||||
|
echo "/!\ Your server isn't a Dell product. Exiting." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
#######################################
|
||||||
|
|
||||||
|
#######################################
|
||||||
|
# Prepare, format and define initial variables
|
||||||
|
# Check if LOW_FAN_SPEED variable is in hexadecimal format. If not, convert it to hexadecimal
|
||||||
|
if [[ $LOW_FAN_SPEED == 0x* ]]; then
|
||||||
|
readonly DECIMAL_LOW_FAN_SPEED=$(printf '%d' $LOW_FAN_SPEED)
|
||||||
|
readonly HEXADECIMAL_LOW_FAN_SPEED=$LOW_FAN_SPEED
|
||||||
|
else
|
||||||
|
readonly DECIMAL_LOW_FAN_SPEED=$LOW_FAN_SPEED
|
||||||
|
readonly HEXADECIMAL_LOW_FAN_SPEED=$(convert_decimal_value_to_hexadecimal $LOW_FAN_SPEED)
|
||||||
|
fi
|
||||||
|
#######################################
|
||||||
|
|
||||||
|
#######################################
|
||||||
|
# This returns the lowest temp between CPU and GPU highest thresholds
|
||||||
|
# Used to calcuate fan speed interpolation
|
||||||
|
# I picked the lowest temp as the fan speeds will ramp up faster to account for the lower temp
|
||||||
|
if [ $CPU_TEMPERATURE_THRESHOLD -le $GPU_TEMPERATURE_THRESHOLD ]; then
|
||||||
|
HIGH_TEMPERATURE_THRESHOLD=$CPU_TEMPERATURE_THRESHOLD
|
||||||
|
else
|
||||||
|
HIGH_TEMPERATURE_THRESHOLD=$GPU_TEMPERATURE_THRESHOLD
|
||||||
|
fi
|
||||||
|
#######################################
|
||||||
|
|
||||||
|
#######################################
|
||||||
|
# Determine where to get CPU temps from iDRAC
|
||||||
|
# If server model is Gen 14 (*40) or newer
|
||||||
|
if [[ $SERVER_MODEL =~ .*[RT][[:space:]]?[0-9][4-9]0.* ]]; then
|
||||||
|
DELL_POWEREDGE_GEN_14_OR_NEWER=true
|
||||||
|
CPU1_TEMPERATURE_INDEX=2
|
||||||
|
CPU2_TEMPERATURE_INDEX=4
|
||||||
|
else
|
||||||
|
DELL_POWEREDGE_GEN_14_OR_NEWER=false
|
||||||
|
CPU1_TEMPERATURE_INDEX=1
|
||||||
|
CPU2_TEMPERATURE_INDEX=2
|
||||||
|
fi
|
||||||
|
#######################################
|
||||||
|
|
||||||
|
#######################################
|
||||||
|
# Check if sensors are present on the server
|
||||||
|
IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT=true
|
||||||
|
IS_CPU2_TEMPERATURE_SENSOR_PRESENT=true
|
||||||
|
retrieve_cpu_temperatures $IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT $IS_CPU2_TEMPERATURE_SENSOR_PRESENT
|
||||||
|
if [ -z "$EXHAUST_TEMPERATURE" ]; then
|
||||||
|
echo "No exhaust temperature sensor detected."
|
||||||
|
IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT=false
|
||||||
|
fi
|
||||||
|
if [ -z "$CPU2_TEMPERATURE" ]; then
|
||||||
|
echo "No CPU2 temperature sensor detected."
|
||||||
|
IS_CPU2_TEMPERATURE_SENSOR_PRESENT=false
|
||||||
|
fi
|
||||||
|
#######################################
|
||||||
|
|
||||||
|
#######################################
|
||||||
|
# Read every drive's own temperature limit once - they do not change, and
|
||||||
|
# smartctl -x is far heavier than the -A used on the fast loop.
|
||||||
|
cache_disk_limits
|
||||||
|
echo "Monitoring $DISK_COUNT_EXPECTED disk(s)."
|
||||||
|
LAST_SMART_CHECK=0
|
||||||
|
#######################################
|
||||||
|
|
||||||
|
#######################################
|
||||||
|
# One measurement pass: read everything, decide a fan speed, raise alarms.
|
||||||
|
one_pass() {
|
||||||
|
retrieve_cpu_temperatures $IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT $IS_CPU2_TEMPERATURE_SENSOR_PRESENT
|
||||||
|
retrieve_gpu_temperature
|
||||||
|
retrieve_disk_temperatures
|
||||||
|
|
||||||
|
# Get highest CPU Temp and check if any of them are over heating
|
||||||
|
HIGHEST_CPU_TEMPERATURE=$CPU1_TEMPERATURE
|
||||||
|
if $IS_CPU2_TEMPERATURE_SENSOR_PRESENT; then
|
||||||
|
if [ "$CPU2_TEMPERATURE" -gt "$CPU1_TEMPERATURE" ]; then
|
||||||
|
HIGHEST_CPU_TEMPERATURE=$CPU2_TEMPERATURE
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$HIGHEST_CPU_TEMPERATURE" -gt "$CPU_TEMPERATURE_THRESHOLD" ]; then
|
||||||
|
# CPU is overheating - hand cooling back to Dell and say so out loud.
|
||||||
|
apply_Dell_fan_control_profile
|
||||||
|
IS_DELL_FAN_CONTROL_PROFILE_APPLIED=true
|
||||||
|
COMMENT="CPU temperature is too high, Dell default dynamic fan control profile applied for safety"
|
||||||
|
raise_alert cpu "Fan control disengaged on $(hostname)" \
|
||||||
|
"CPU at ${HIGHEST_CPU_TEMPERATURE}c exceeds the ${CPU_TEMPERATURE_THRESHOLD}c threshold. Dell's default profile has been restored."
|
||||||
|
record_sample "$HIGHEST_CPU_TEMPERATURE" "$GPU_TEMPERATURE" "$HOTTEST_DISK_TEMPERATURE" 0
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
IS_DELL_FAN_CONTROL_PROFILE_APPLIED=false
|
||||||
|
clear_alert cpu "Fan control restored on $(hostname)"
|
||||||
|
|
||||||
|
# CPU and GPU share one curve; disks are interpolated against their own limits
|
||||||
|
# in retrieve_disk_temperatures. Each source asks for a speed, loudest wins -
|
||||||
|
# temperatures from different classes of hardware are not comparable directly.
|
||||||
|
HIGHEST_TEMPERATURE=$HIGHEST_CPU_TEMPERATURE
|
||||||
|
if [ "$GPU_TEMPERATURE" -gt "$HIGHEST_CPU_TEMPERATURE" ]; then
|
||||||
|
HIGHEST_TEMPERATURE=$GPU_TEMPERATURE
|
||||||
|
fi
|
||||||
|
if [ "$HIGHEST_TEMPERATURE" -gt "$LOW_TEMPERATURE_THRESHOLD" ]; then
|
||||||
|
CPU_GPU_FAN_SPEED=$(calculate_interpolated_fan_speed "$HIGHEST_TEMPERATURE" \
|
||||||
|
$LOW_TEMPERATURE_THRESHOLD $HIGH_TEMPERATURE_THRESHOLD $LOW_FAN_SPEED $HIGH_FAN_SPEED)
|
||||||
|
else
|
||||||
|
CPU_GPU_FAN_SPEED=$DECIMAL_LOW_FAN_SPEED
|
||||||
|
fi
|
||||||
|
|
||||||
|
DECIMAL_CURRENT_FAN_SPEED=$CPU_GPU_FAN_SPEED
|
||||||
|
FAN_SPEED_DRIVER="cpu/gpu"
|
||||||
|
if [ "$DISK_FAN_SPEED" -gt "$DECIMAL_CURRENT_FAN_SPEED" ]; then
|
||||||
|
DECIMAL_CURRENT_FAN_SPEED=$DISK_FAN_SPEED
|
||||||
|
FAN_SPEED_DRIVER="disk"
|
||||||
|
fi
|
||||||
|
|
||||||
|
apply_user_fan_control "$DECIMAL_CURRENT_FAN_SPEED"
|
||||||
|
|
||||||
|
COMMENT="CPU1:$CPU1_TEMPERATURE | CPU2:$CPU2_TEMPERATURE | GPU:$GPU_TEMPERATURE | Inlet:$INLET_TEMPERATURE | Exhaust:$EXHAUST_TEMPERATURE | Disk:$HOTTEST_DISK_TEMPERATURE/$HOTTEST_DISK_LIMIT($(basename $HOTTEST_DISK_DEVICE)) | Fan Speed:$DECIMAL_CURRENT_FAN_SPEED($FAN_SPEED_DRIVER)"
|
||||||
|
|
||||||
|
record_sample "$HIGHEST_CPU_TEMPERATURE" "$GPU_TEMPERATURE" "$HOTTEST_DISK_TEMPERATURE" "$DECIMAL_CURRENT_FAN_SPEED"
|
||||||
|
check_alarms
|
||||||
|
}
|
||||||
|
|
||||||
|
check_alarms() {
|
||||||
|
# A drive within DISK_ALARM_OFFSET of its own limit: fans are already flat out
|
||||||
|
# for it and it is still climbing.
|
||||||
|
if [ "$HOTTEST_DISK_TEMPERATURE" -gt 0 ] &&
|
||||||
|
[ "$HOTTEST_DISK_TEMPERATURE" -ge $((HOTTEST_DISK_LIMIT - DISK_ALARM_OFFSET)) ]; then
|
||||||
|
raise_alert disk_temp "Disk over temperature on $(hostname)" \
|
||||||
|
"$HOTTEST_DISK_DEVICE at ${HOTTEST_DISK_TEMPERATURE}c, its own limit is ${HOTTEST_DISK_LIMIT}c. Fans at ${DECIMAL_CURRENT_FAN_SPEED}%."
|
||||||
|
else
|
||||||
|
clear_alert disk_temp "Disk temperature normal on $(hostname)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$GPU_TEMPERATURE" -gt "$GPU_TEMPERATURE_THRESHOLD" ]; then
|
||||||
|
raise_alert gpu "GPU over temperature on $(hostname)" \
|
||||||
|
"GPU at ${GPU_TEMPERATURE}c exceeds the ${GPU_TEMPERATURE_THRESHOLD}c threshold."
|
||||||
|
else
|
||||||
|
clear_alert gpu "GPU temperature normal on $(hostname)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Climbing steadily while still under every threshold - a dying fan or a
|
||||||
|
# blocked intake looks exactly like this long before anything crosses a limit.
|
||||||
|
local cpu_rise disk_rise
|
||||||
|
cpu_rise=$(rise_over_window 2)
|
||||||
|
disk_rise=$(rise_over_window 4)
|
||||||
|
if [ "$cpu_rise" -ge "$TREND_RISE_ALARM" ] || [ "$disk_rise" -ge "$TREND_RISE_ALARM" ]; then
|
||||||
|
raise_alert trend "Temperature climbing on $(hostname)" \
|
||||||
|
"Over the last $((TREND_SAMPLES * CHECK_INTERVAL / 60)) minutes: CPU +${cpu_rise}c, hottest disk +${disk_rise}c. Nothing has crossed a threshold yet. Check airflow and fans."
|
||||||
|
else
|
||||||
|
clear_alert trend "Temperature stabilised on $(hostname)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# A drive that stops answering is either asleep or gone. Worth knowing which.
|
||||||
|
if [ "$DISKS_READ" -lt "$DISK_COUNT_EXPECTED" ]; then
|
||||||
|
raise_alert disk_count "Disk missing on $(hostname)" \
|
||||||
|
"Read $DISKS_READ of $DISK_COUNT_EXPECTED disks. A drive is in standby, has dropped off the bus, or has failed."
|
||||||
|
else
|
||||||
|
clear_alert disk_count "All disks reporting on $(hostname)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# SMART is expensive across two dozen drives - hourly, not every loop.
|
||||||
|
local now; now=$(date +%s)
|
||||||
|
if [ $((now - LAST_SMART_CHECK)) -ge "$SMART_CHECK_INTERVAL" ]; then
|
||||||
|
LAST_SMART_CHECK=$now
|
||||||
|
check_disk_health
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
#######################################
|
||||||
|
|
||||||
|
case "${1:-run}" in
|
||||||
|
once)
|
||||||
|
one_pass
|
||||||
|
echo "$COMMENT"
|
||||||
|
log_line "$COMMENT"
|
||||||
|
;;
|
||||||
|
disks)
|
||||||
|
for dev in $(disk_devices); do
|
||||||
|
printf '%-12s %-4s limit=%sc\n' "$dev" "$(disk_temperature "$dev")c" "$(disk_limit "$dev")"
|
||||||
|
done
|
||||||
|
;;
|
||||||
|
run)
|
||||||
|
while true; do
|
||||||
|
sleep $CHECK_INTERVAL &
|
||||||
|
SLEEP_PROCESS_PID=$!
|
||||||
|
one_pass
|
||||||
|
log_line "$COMMENT"
|
||||||
|
wait $SLEEP_PROCESS_PID
|
||||||
|
done
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "usage: $0 [run|once|disks|selftest]" >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
215
functions.sh
Executable file
215
functions.sh
Executable file
@@ -0,0 +1,215 @@
|
|||||||
|
# Define global functions
|
||||||
|
# This function applies Dell's default dynamic fan control profile
|
||||||
|
function apply_Dell_fan_control_profile() {
|
||||||
|
# Use ipmitool to send the raw command to set fan control to Dell default
|
||||||
|
ipmitool -I $IDRAC_LOGIN_STRING raw 0x30 0x30 0x01 0x01 > /dev/null
|
||||||
|
CURRENT_FAN_CONTROL_PROFILE="Dell default dynamic fan control profile"
|
||||||
|
}
|
||||||
|
|
||||||
|
#Get the TEMP of the TESLA P4 installed in this servers
|
||||||
|
#Should probably set a variable so this function can check temps of any GPU using nvidia-smi
|
||||||
|
retrieve_gpu_temperature() {
|
||||||
|
if [ -x /usr/bin/nvidia-smi ]; then
|
||||||
|
# Get the index of the Tesla P4 GPU using nvidia-smi
|
||||||
|
index=$(nvidia-smi --query-gpu=name,index --format=csv | grep "Tesla P4" | cut -d ',' -f 2)
|
||||||
|
else
|
||||||
|
index=""
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$index" != "" ]; then
|
||||||
|
# Get the temperature of the Tesla P4 GPU using nvidia-smi and the index obtained above
|
||||||
|
GPU_TEMPERATURE=$(nvidia-smi --query-gpu=temperature.gpu --format=csv | head -$(( $(($index + 2)) )) | tail -1)
|
||||||
|
else
|
||||||
|
GPU_TEMPERATURE="0"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Apply user-defined fan control settings
|
||||||
|
#
|
||||||
|
# This function applies user-defined fan control settings based on the fan speed.
|
||||||
|
# It handles both decimal and hexadecimal fan speed inputs, converting between them as needed.
|
||||||
|
# The function then applies the fan control and updates the current fan control profile.
|
||||||
|
#
|
||||||
|
# Parameters:
|
||||||
|
# $1 (LOCAL_FAN_SPEED): The desired fan speed. Can be in decimal (0-100) or hexadecimal (0x00-0x64) format.
|
||||||
|
#
|
||||||
|
# Global variables used:
|
||||||
|
# CURRENT_FAN_CONTROL_PROFILE: Updated with the current fan control profile description.
|
||||||
|
#
|
||||||
|
# Returns:
|
||||||
|
# None.
|
||||||
|
function apply_user_fan_control() {
|
||||||
|
local LOCAL_FAN_SPEED=$1
|
||||||
|
|
||||||
|
if [[ $LOCAL_FAN_SPEED == 0x* ]]; then
|
||||||
|
local LOCAL_DECIMAL_FAN_SPEED
|
||||||
|
LOCAL_DECIMAL_FAN_SPEED=$(printf '%d' "$LOCAL_FAN_SPEED")
|
||||||
|
local LOCAL_HEXADECIMAL_FAN_SPEED=$LOCAL_FAN_SPEED
|
||||||
|
else
|
||||||
|
local LOCAL_DECIMAL_FAN_SPEED=$LOCAL_FAN_SPEED
|
||||||
|
local LOCAL_HEXADECIMAL_FAN_SPEED
|
||||||
|
LOCAL_HEXADECIMAL_FAN_SPEED=$(convert_decimal_value_to_hexadecimal "$LOCAL_FAN_SPEED")
|
||||||
|
fi
|
||||||
|
|
||||||
|
apply_fan_control_to_specified_value "$LOCAL_HEXADECIMAL_FAN_SPEED"
|
||||||
|
CURRENT_FAN_CONTROL_PROFILE="User static fan control profile ($LOCAL_DECIMAL_FAN_SPEED%)"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Apply fan control to a specified value
|
||||||
|
#
|
||||||
|
# This function sets the fan speed to a user-specified value using ipmitool.
|
||||||
|
# It first checks if the input value is in hexadecimal format, and converts it
|
||||||
|
# if necessary. Then it sends raw commands to iDRAC to set the fan control.
|
||||||
|
#
|
||||||
|
# Parameters:
|
||||||
|
# $1 (VALUE): The desired fan speed value. Can be in decimal or hexadecimal format.
|
||||||
|
# If in decimal, it will be converted to hexadecimal.
|
||||||
|
#
|
||||||
|
# Returns:
|
||||||
|
# None
|
||||||
|
#
|
||||||
|
# Note:
|
||||||
|
# This function uses the global variable $IDRAC_LOGIN_STRING for iDRAC login.
|
||||||
|
function apply_fan_control_to_specified_value() {
|
||||||
|
local VALUE=$1
|
||||||
|
|
||||||
|
# Check if the input value is a hexadecimal number, if not, convert it to hexadecimal
|
||||||
|
if [[ $VALUE != 0x* ]]; then
|
||||||
|
VALUE=$(convert_decimal_value_to_hexadecimal "$VALUE")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Use ipmitool to send the raw command to set fan control to user-specified value
|
||||||
|
ipmitool -I $IDRAC_LOGIN_STRING raw 0x30 0x30 0x01 0x00 > /dev/null
|
||||||
|
ipmitool -I $IDRAC_LOGIN_STRING raw 0x30 0x30 0x02 0xff "$VALUE" > /dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
# Calculate the interpolated fan speed based on CPU temperature
|
||||||
|
#
|
||||||
|
# This function calculates the interpolated fan speed based on the current CPU temperature
|
||||||
|
# and predefined thresholds. It uses linear interpolation to adjust the fan speed
|
||||||
|
# within a specified range when the CPU temperature exceeds a certain threshold.
|
||||||
|
#
|
||||||
|
# Parameters:
|
||||||
|
# $1 (HIGHEST_TEMPERATURE): The current highest CPU/GPU temperature (in Celsius)
|
||||||
|
# $2 (TEMPERATURE_THRESHOLD_FOR_FAN_SPEED_INTERPOLATION): The lower temperature threshold for fan speed interpolation (in Celsius)
|
||||||
|
# $3 (TEMPERATURE_THRESHOLD): The upper temperature threshold for fan speed interpolation (in Celsius)
|
||||||
|
# $4 (LOCAL_DECIMAL_FAN_SPEED): The base fan speed (as a decimal percentage, 0-100)
|
||||||
|
# $5 (LOCAL_DECIMAL_HIGH_FAN_SPEED): The maximum fan speed (as a decimal percentage, 0-100)
|
||||||
|
#
|
||||||
|
# Returns:
|
||||||
|
# The calculated interpolated fan speed as a decimal percentage (0-100)
|
||||||
|
# If the temperature is below or equal to the lower threshold, returns the base fan speed
|
||||||
|
# If the temperature is above or equal to the upper threshold, returns the maximum fan speed
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# calculate_interpolated_fan_speed <highest_cpu_temp> <lower_threshold> <upper_threshold> <base_fan_speed> <max_fan_speed>
|
||||||
|
#
|
||||||
|
# Example:
|
||||||
|
# calculate_interpolated_fan_speed 70 60 80 30 100
|
||||||
|
function calculate_interpolated_fan_speed() {
|
||||||
|
local HIGHEST_CPU_TEMPERATURE=$1
|
||||||
|
local CPU_TEMPERATURE_THRESHOLD_FOR_FAN_SPEED_INTERPOLATION=$2
|
||||||
|
local CPU_TEMPERATURE_THRESHOLD=$3
|
||||||
|
local LOCAL_DECIMAL_FAN_SPEED=$4
|
||||||
|
local LOCAL_DECIMAL_HIGH_FAN_SPEED=$5
|
||||||
|
|
||||||
|
# If temperature is below or equal to the lower threshold, return the base fan speed
|
||||||
|
if [ "$HIGHEST_CPU_TEMPERATURE" -le "$CPU_TEMPERATURE_THRESHOLD_FOR_FAN_SPEED_INTERPOLATION" ]; then
|
||||||
|
echo "$LOCAL_DECIMAL_FAN_SPEED"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# If temperature is above or equal to the upper threshold, return the max fan speed
|
||||||
|
if [ "$HIGHEST_CPU_TEMPERATURE" -ge "$CPU_TEMPERATURE_THRESHOLD" ]; then
|
||||||
|
echo "$LOCAL_DECIMAL_HIGH_FAN_SPEED"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# F1 - lower fan speed
|
||||||
|
# F2 - higher fan speed
|
||||||
|
# T_CPU - highest temperature of both CPUs (if only one exists that will be CPU1 temp value)
|
||||||
|
# T1 - lower temperature threshold
|
||||||
|
# T2 - higher temperature threshold
|
||||||
|
# Fan speed = F1 + ( ( F2 - F1 ) * ( T_CPU - T1 ) / ( T2 - T1 ) )
|
||||||
|
|
||||||
|
local TEMPERATURE_INTERPOLATION_ACTIVATION_RANGE=$((CPU_TEMPERATURE_THRESHOLD - CPU_TEMPERATURE_THRESHOLD_FOR_FAN_SPEED_INTERPOLATION))
|
||||||
|
local FAN_VALUE_TO_ADD=0
|
||||||
|
|
||||||
|
if [ $TEMPERATURE_INTERPOLATION_ACTIVATION_RANGE -gt $FAN_VALUE_TO_ADD ]; then
|
||||||
|
local TEMPERATURE_ABOVE_THRESHOLD_FOR_FAN_SPEED_INTERPOLATION=$((HIGHEST_CPU_TEMPERATURE - CPU_TEMPERATURE_THRESHOLD_FOR_FAN_SPEED_INTERPOLATION))
|
||||||
|
local FAN_WINDOW=$((LOCAL_DECIMAL_HIGH_FAN_SPEED - LOCAL_DECIMAL_FAN_SPEED))
|
||||||
|
FAN_VALUE_TO_ADD=$((FAN_WINDOW * TEMPERATURE_ABOVE_THRESHOLD_FOR_FAN_SPEED_INTERPOLATION / TEMPERATURE_INTERPOLATION_ACTIVATION_RANGE))
|
||||||
|
fi
|
||||||
|
|
||||||
|
local DECIMAL_CURRENT_FAN_SPEED=$((LOCAL_DECIMAL_FAN_SPEED + FAN_VALUE_TO_ADD))
|
||||||
|
echo $DECIMAL_CURRENT_FAN_SPEED
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# Convert first parameter given ($DECIMAL_NUMBER) to hexadecimal
|
||||||
|
# Usage : convert_decimal_value_to_hexadecimal $DECIMAL_NUMBER
|
||||||
|
# Returns : hexadecimal value of DECIMAL_NUMBER
|
||||||
|
function convert_decimal_value_to_hexadecimal () {
|
||||||
|
local DECIMAL_NUMBER=$1
|
||||||
|
local HEXADECIMAL_NUMBER=$(printf '0x%02x' $DECIMAL_NUMBER)
|
||||||
|
echo $HEXADECIMAL_NUMBER
|
||||||
|
}
|
||||||
|
|
||||||
|
# Retrieve temperature sensors data using ipmitool
|
||||||
|
# Usage : retrieve_temperatures $IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT $IS_CPU2_TEMPERATURE_SENSOR_PRESENT
|
||||||
|
function retrieve_cpu_temperatures() {
|
||||||
|
if (( $# != 2 )); then
|
||||||
|
printf "Illegal number of parameters.\nUsage: retrieve_temperatures \$IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT \$IS_CPU2_TEMPERATURE_SENSOR_PRESENT" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
local IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT=$1
|
||||||
|
local IS_CPU2_TEMPERATURE_SENSOR_PRESENT=$2
|
||||||
|
|
||||||
|
local DATA=$(ipmitool -I $IDRAC_LOGIN_STRING sdr type temperature | grep degrees)
|
||||||
|
|
||||||
|
# Parse CPU data
|
||||||
|
local CPU_DATA=$(echo "$DATA" | grep "3\." | grep -Po '\d{2}')
|
||||||
|
CPU1_TEMPERATURE=$(echo $CPU_DATA | awk "{print \$$CPU1_TEMPERATURE_INDEX;}")
|
||||||
|
if $IS_CPU2_TEMPERATURE_SENSOR_PRESENT; then
|
||||||
|
CPU2_TEMPERATURE=$(echo $CPU_DATA | awk "{print \$$CPU2_TEMPERATURE_INDEX;}")
|
||||||
|
else
|
||||||
|
CPU2_TEMPERATURE="-"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Parse inlet temperature data
|
||||||
|
INLET_TEMPERATURE=$(echo "$DATA" | grep Inlet | grep -Po '\d{2}' | tail -1)
|
||||||
|
|
||||||
|
# If exhaust temperature sensor is present, parse its temperature data
|
||||||
|
if $IS_EXHAUST_TEMPERATURE_SENSOR_PRESENT; then
|
||||||
|
EXHAUST_TEMPERATURE=$(echo "$DATA" | grep Exhaust | grep -Po '\d{2}' | tail -1)
|
||||||
|
else
|
||||||
|
EXHAUST_TEMPERATURE="-"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Prepare traps in case of container exit
|
||||||
|
function graceful_exit() {
|
||||||
|
apply_Dell_fan_control_profile
|
||||||
|
|
||||||
|
echo "/!\ WARNING /!\ Container stopped, Dell default dynamic fan control profile applied for safety."
|
||||||
|
exit 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# Helps debugging when people are posting their output
|
||||||
|
function get_Dell_server_model() {
|
||||||
|
IPMI_FRU_content=$(ipmitool -I $IDRAC_LOGIN_STRING fru 2>/dev/null) # FRU stands for "Field Replaceable Unit"
|
||||||
|
# TODO - Add check if connection was established. There are a chance user type wrong login and pass. In my case it returns "Error: Unable to establish IPMI v2 / RMCP+ session"
|
||||||
|
|
||||||
|
SERVER_MANUFACTURER=$(echo "$IPMI_FRU_content" | grep "Product Manufacturer" | awk -F ': ' '{print $2}')
|
||||||
|
SERVER_MODEL=$(echo "$IPMI_FRU_content" | grep "Product Name" | awk -F ': ' '{print $2}')
|
||||||
|
|
||||||
|
# Check if SERVER_MANUFACTURER is empty, if yes, assign value based on "Board Mfg"
|
||||||
|
if [ -z "$SERVER_MANUFACTURER" ]; then
|
||||||
|
SERVER_MANUFACTURER=$(echo "$IPMI_FRU_content" | tr -s ' ' | grep "Board Mfg :" | awk -F ': ' '{print $2}')
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if SERVER_MODEL is empty, if yes, assign value based on "Board Product"
|
||||||
|
if [ -z "$SERVER_MODEL" ]; then
|
||||||
|
SERVER_MODEL=$(echo "$IPMI_FRU_content" | tr -s ' ' | grep "Board Product :" | awk -F ': ' '{print $2}')
|
||||||
|
fi
|
||||||
|
}
|
||||||
220
monitor.sh
Normal file
220
monitor.sh
Normal file
@@ -0,0 +1,220 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
#
|
||||||
|
# Disk temperature monitoring, trend tracking and email alarms.
|
||||||
|
#
|
||||||
|
# Sourced by fan_speed.sh alongside functions.sh. functions.sh is vendored from
|
||||||
|
# upstream (tigerblue77/Dell_iDRAC_fan_controller_Docker); everything in here is
|
||||||
|
# local, so keep the two files separate to keep that boundary readable.
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- configuration
|
||||||
|
|
||||||
|
# Internal drives only. The DAS shelf enumerates behind a SAS expander
|
||||||
|
# (*-sas-exp*) and has its own controller - it must not drive server fans.
|
||||||
|
DISK_GLOB=${DISK_GLOB:-/dev/disk/by-path/pci-*-scsi-*}
|
||||||
|
|
||||||
|
# Each drive's fan ramp is derived from its own maximum operating temperature:
|
||||||
|
# ramp starts at limit - DISK_RAMP_LOW_OFFSET
|
||||||
|
# full speed at limit - DISK_RAMP_HIGH_OFFSET
|
||||||
|
# Larger low offset = react earlier = louder. See README before changing.
|
||||||
|
DISK_RAMP_LOW_OFFSET=${DISK_RAMP_LOW_OFFSET:-18}
|
||||||
|
DISK_RAMP_HIGH_OFFSET=${DISK_RAMP_HIGH_OFFSET:-8}
|
||||||
|
|
||||||
|
# A drive this close to its own limit raises an alarm - the fans are already
|
||||||
|
# flat out for it and it is still climbing.
|
||||||
|
DISK_ALARM_OFFSET=${DISK_ALARM_OFFSET:-5}
|
||||||
|
|
||||||
|
# Drives report their limit but do not agree on what it means: Samsung and
|
||||||
|
# Kioxia report a true operating maximum (70), Toshiba and Seagate report 60,
|
||||||
|
# WD Reds report 85 - the SCT critical limit, not an operating maximum.
|
||||||
|
# Clamp by class so one optimistic drive cannot quietly disable cooling.
|
||||||
|
HDD_LIMIT_CAP=${HDD_LIMIT_CAP:-60}
|
||||||
|
SSD_LIMIT_CAP=${SSD_LIMIT_CAP:-70}
|
||||||
|
|
||||||
|
ALERT_EMAIL=${ALERT_EMAIL:-Servers@ntfy1.izebra.xyz}
|
||||||
|
ALERT_COOLDOWN=${ALERT_COOLDOWN:-3600}
|
||||||
|
STATE_DIR=${STATE_DIR:-/root/fan_speed/state}
|
||||||
|
LOG_FILE=${LOG_FILE:-/root/fan_speed/log/fan_speed.log}
|
||||||
|
|
||||||
|
TREND_FILE=${TREND_FILE:-/root/fan_speed/log/temps.csv}
|
||||||
|
TREND_SAMPLES=${TREND_SAMPLES:-90} # 90 x 10s CHECK_INTERVAL = 15 minutes
|
||||||
|
TREND_RISE_ALARM=${TREND_RISE_ALARM:-8} # degrees of climb across that window
|
||||||
|
|
||||||
|
SMART_CHECK_INTERVAL=${SMART_CHECK_INTERVAL:-3600}
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ disk temps
|
||||||
|
|
||||||
|
log_line() {
|
||||||
|
echo "$(date +'%Y-%m-%d %H:%M:%S') || $*" >> "$LOG_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Block devices behind DISK_GLOB, deduplicated. Resolved fresh every call:
|
||||||
|
# kernel names are not stable, a shelf rescan renamed sdaa-sdai to sds-sdaa.
|
||||||
|
disk_devices() {
|
||||||
|
local link dev
|
||||||
|
for link in $DISK_GLOB; do
|
||||||
|
case "$link" in *-part*) continue ;; esac
|
||||||
|
[ -e "$link" ] || continue
|
||||||
|
dev=$(readlink -f "$link")
|
||||||
|
[ -b "$dev" ] && echo "$dev"
|
||||||
|
done | sort -u
|
||||||
|
}
|
||||||
|
|
||||||
|
# smartctl -A output on stdin -> temperature in celsius, empty if it did not say.
|
||||||
|
# Drives disagree on the attribute name: 194 Temperature_Celsius (most),
|
||||||
|
# 190 Airflow_Temperature_Cel (Seagate), 194 Temperature_Internal (Intel),
|
||||||
|
# and SCSI/SAS drives report a "Current Drive Temperature" line instead.
|
||||||
|
# Temperature_Difference_from_100 is a delta, not a reading - never match it.
|
||||||
|
parse_disk_temperature() {
|
||||||
|
awk '
|
||||||
|
/Current Drive Temperature/ { print $4; exit }
|
||||||
|
/^ *[0-9]+ +[A-Za-z_]*Temperature[A-Za-z_]*/ && !/Difference/ { print $10; exit }'
|
||||||
|
}
|
||||||
|
|
||||||
|
# smartctl -x output on stdin -> the drive's own maximum operating temperature.
|
||||||
|
parse_disk_limit() {
|
||||||
|
awk '
|
||||||
|
/Min\/Max Temperature Limit/ { split($0, a, /[ \/]+/); for (i in a) if (a[i] ~ /^[0-9]+$/) v = a[i]; print v; exit }
|
||||||
|
/Drive Trip Temperature/ { print $4; exit }'
|
||||||
|
}
|
||||||
|
|
||||||
|
# -n standby: skip a sleeping drive rather than spinning it up to measure it.
|
||||||
|
disk_temperature() {
|
||||||
|
smartctl -n standby -A "$1" 2>/dev/null | parse_disk_temperature
|
||||||
|
}
|
||||||
|
|
||||||
|
# The drive's own limit, clamped by class (see HDD_LIMIT_CAP above).
|
||||||
|
disk_limit() {
|
||||||
|
local dev=$1 limit rotational cap
|
||||||
|
limit=$(smartctl -x "$dev" 2>/dev/null | parse_disk_limit)
|
||||||
|
rotational=$(cat "/sys/block/$(basename "$dev")/queue/rotational" 2>/dev/null)
|
||||||
|
if [ "$rotational" = 1 ]; then cap=$HDD_LIMIT_CAP; else cap=$SSD_LIMIT_CAP; fi
|
||||||
|
if [ -z "$limit" ] || ! [ "$limit" -le "$cap" ] 2>/dev/null; then limit=$cap; fi
|
||||||
|
echo "$limit"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Limits do not change, and smartctl -x is far heavier than -A. Read them once.
|
||||||
|
declare -A DISK_LIMIT
|
||||||
|
cache_disk_limits() {
|
||||||
|
local dev
|
||||||
|
DISK_LIMIT=()
|
||||||
|
for dev in $(disk_devices); do
|
||||||
|
DISK_LIMIT[$dev]=$(disk_limit "$dev")
|
||||||
|
done
|
||||||
|
DISK_COUNT_EXPECTED=${#DISK_LIMIT[@]}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Reads every drive. Sets:
|
||||||
|
# HOTTEST_DISK_TEMPERATURE / _DEVICE / _LIMIT, DISK_FAN_SPEED, DISKS_READ
|
||||||
|
# Each drive is interpolated against its OWN limit and the highest resulting fan
|
||||||
|
# speed wins - so a 50c SSD rated to 70 asks for less than a 45c disk rated to 60.
|
||||||
|
# That makes the curve independent of drive technology and of bay position.
|
||||||
|
retrieve_disk_temperatures() {
|
||||||
|
local dev temp limit speed
|
||||||
|
HOTTEST_DISK_TEMPERATURE=0
|
||||||
|
HOTTEST_DISK_DEVICE="-"
|
||||||
|
HOTTEST_DISK_LIMIT=0
|
||||||
|
DISK_FAN_SPEED=$LOW_FAN_SPEED
|
||||||
|
DISKS_READ=0
|
||||||
|
|
||||||
|
for dev in $(disk_devices); do
|
||||||
|
temp=$(disk_temperature "$dev")
|
||||||
|
[ -n "$temp" ] || continue
|
||||||
|
DISKS_READ=$((DISKS_READ + 1))
|
||||||
|
|
||||||
|
limit=${DISK_LIMIT[$dev]}
|
||||||
|
[ -n "$limit" ] || limit=$(disk_limit "$dev")
|
||||||
|
|
||||||
|
speed=$(calculate_interpolated_fan_speed "$temp" \
|
||||||
|
$((limit - DISK_RAMP_LOW_OFFSET)) $((limit - DISK_RAMP_HIGH_OFFSET)) \
|
||||||
|
"$LOW_FAN_SPEED" "$HIGH_FAN_SPEED")
|
||||||
|
[ "$speed" -gt "$DISK_FAN_SPEED" ] && DISK_FAN_SPEED=$speed
|
||||||
|
|
||||||
|
if [ "$temp" -gt "$HOTTEST_DISK_TEMPERATURE" ]; then
|
||||||
|
HOTTEST_DISK_TEMPERATURE=$temp
|
||||||
|
HOTTEST_DISK_DEVICE=$dev
|
||||||
|
HOTTEST_DISK_LIMIT=$limit
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------- alarms
|
||||||
|
|
||||||
|
notify() {
|
||||||
|
local subject=$1 body=$2
|
||||||
|
if [ -n "$DRY_RUN" ]; then
|
||||||
|
echo "MAIL[$ALERT_EMAIL] $subject"
|
||||||
|
else
|
||||||
|
printf '%s\n' "$body" | mail -s "$subject" "$ALERT_EMAIL"
|
||||||
|
fi
|
||||||
|
log_line "ALERT $subject"
|
||||||
|
}
|
||||||
|
|
||||||
|
# raise_alert <key> <subject> <body>
|
||||||
|
# Rate limited per key. Without this a 10s loop sends 360 mails an hour and the
|
||||||
|
# alarm becomes the outage.
|
||||||
|
raise_alert() {
|
||||||
|
local key=$1 subject=$2 body=$3
|
||||||
|
local stamp="$STATE_DIR/$key.alert" now last
|
||||||
|
mkdir -p "$STATE_DIR"
|
||||||
|
now=$(date +%s)
|
||||||
|
if [ -f "$stamp" ]; then
|
||||||
|
last=$(cat "$stamp")
|
||||||
|
[ $((now - last)) -lt "$ALERT_COOLDOWN" ] && return 0
|
||||||
|
fi
|
||||||
|
echo "$now" > "$stamp"
|
||||||
|
notify "$subject" "$body"
|
||||||
|
}
|
||||||
|
|
||||||
|
# clear_alert <key> <subject> - one recovery notice, only if the alarm was up.
|
||||||
|
clear_alert() {
|
||||||
|
local key=$1 subject=$2
|
||||||
|
local stamp="$STATE_DIR/$key.alert"
|
||||||
|
[ -f "$stamp" ] || return 0
|
||||||
|
rm -f "$stamp"
|
||||||
|
notify "$subject" "Recovered at $(date '+%Y-%m-%d %H:%M:%S')."
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------- trend
|
||||||
|
|
||||||
|
# record_sample <cpu> <gpu> <disk> <fan>
|
||||||
|
record_sample() {
|
||||||
|
mkdir -p "$(dirname "$TREND_FILE")"
|
||||||
|
echo "$(date +%s),$1,$2,$3,$4" >> "$TREND_FILE"
|
||||||
|
if [ "$(wc -l < "$TREND_FILE")" -gt $((TREND_SAMPLES * 2)) ]; then
|
||||||
|
tail -n "$TREND_SAMPLES" "$TREND_FILE" > "$TREND_FILE.tmp" && mv "$TREND_FILE.tmp" "$TREND_FILE"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# rise_over_window <column> -> degrees climbed from the oldest sample in the
|
||||||
|
# window to the newest. 0 until there is a full window of history, so a restart
|
||||||
|
# cannot alarm on a partial series.
|
||||||
|
# Columns: 2 cpu, 3 gpu, 4 disk, 5 fan.
|
||||||
|
rise_over_window() {
|
||||||
|
local column=$1 oldest newest lines
|
||||||
|
lines=$(wc -l < "$TREND_FILE" 2>/dev/null || echo 0)
|
||||||
|
if [ "$lines" -lt "$TREND_SAMPLES" ]; then echo 0; return; fi
|
||||||
|
oldest=$(tail -n "$TREND_SAMPLES" "$TREND_FILE" | head -1 | cut -d, -f"$column")
|
||||||
|
newest=$(tail -n 1 "$TREND_FILE" | cut -d, -f"$column")
|
||||||
|
[ -n "$oldest" ] && [ -n "$newest" ] || { echo 0; return; }
|
||||||
|
echo $((newest - oldest))
|
||||||
|
}
|
||||||
|
|
||||||
|
# --------------------------------------------------------------- SMART health
|
||||||
|
|
||||||
|
# Expensive across two dozen drives - the caller runs this hourly, not per loop.
|
||||||
|
check_disk_health() {
|
||||||
|
local dev status failed=""
|
||||||
|
for dev in $(disk_devices); do
|
||||||
|
status=$(smartctl -H "$dev" 2>/dev/null | grep -iE "overall-health|SMART Health Status")
|
||||||
|
[ -n "$status" ] || continue
|
||||||
|
case "$status" in
|
||||||
|
*PASSED*|*OK*) ;;
|
||||||
|
*) failed+="$dev: $status"$'\n' ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
if [ -n "$failed" ]; then
|
||||||
|
raise_alert smart "SMART failure on $(hostname)" "$failed"
|
||||||
|
else
|
||||||
|
clear_alert smart "SMART recovered on $(hostname)"
|
||||||
|
fi
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user