Shell Reliability Library: The Plumbing Every Script Rewrote

Why this exists, and why it did not exist first

The connectivity validator and the drift reporter are both published, and if you read them side by side you can see the problem this library exists to name: they carry near-identical blocks of the same plumbing, logging, locking, temporary directory handling, and dependency checks that are implicit in one and explicit in the other. Copy-paste drift between them is not hypothetical; the two log_message implementations differ, which is exactly the kind of trivial inconsistency that costs twenty minutes during an incident when you are grepping two logs and your timestamps do not line up.

The order of operations matters here, and it is the whole design philosophy: the library was extracted from working scripts, not designed in advance of them. A reliability framework written before its consumers is abstraction theater, full of options nobody asked for and guesses about needs that never materialize. This library contains exactly the functions two real tools demonstrated a shared need for, and nothing else. When a third tool needs a sixth function, the sixth function gets written then.

I will also flag what this library does not claim: nothing here is self-healing, and I have deliberately avoided that word. Nothing in these hundred-odd lines heals anything. What they do is narrower and more honest: they make scripts fail cleanly, loudly, and without leaving debris. Failing well is a legitimately hard engineering property and it does not need to borrow a grander name.

What is in it

Five functions, prefixed reliability_ so a sourcing script's own names never collide:

1. reliability_require_commands: dependency preflight. Every missing tool reported at once, at startup, instead of one at a time at whatever line happens to need it, forty seconds into a run that has already mutated state.
2. reliability_acquire_lock: single-instance enforcement via flock, with the lock released by an EXIT trap no matter how the script dies.
3. reliability_create_temporary_directory: mktemp with registered cleanup, so temporary trees cannot outlive the process even on failure paths nobody tested.
4. reliability_log_message: one timestamp format, one destination convention, optional syslog tee through logger, so every tool's log greps the same way.
5. reliability_run_with_retry: bounded attempts, exponential backoff with jitter, for operations where retrying is the correct response to failure. Which, in a monitoring context, is a smaller set than it first appears, and that is worth its own section.

The library

#!/usr/bin/env bash
#
# reliability_library.sh
# Shared reliability plumbing for operational scripts. Source this file;
# do not execute it. All public names are prefixed reliability_.
#
# Sourcing contract: call reliability_initialize <tool_name> <log_file>
# before anything else. EXIT trap registration happens there, once.
#

reliability_tool_name="unnamed_tool"
reliability_log_file="/dev/null"
reliability_cleanup_paths=()
reliability_lock_descriptor=""

reliability_initialize() {
    reliability_tool_name="$1"
    reliability_log_file="$2"
    trap reliability_run_cleanup EXIT
}

reliability_log_message() {
    local message_text="$1"
    printf '%s %s %s\n' "$(date --iso-8601=seconds)" "$reliability_tool_name" "$message_text" \
        >> "$reliability_log_file"
    if command -v logger > /dev/null 2>&1; then
        logger --tag "$reliability_tool_name" -- "$message_text"
    fi
}

reliability_require_commands() {
    local missing_command_list=()
    local candidate_command
    for candidate_command in "$@"; do
        if ! command -v "$candidate_command" > /dev/null 2>&1; then
            missing_command_list+=("$candidate_command")
        fi
    done
    if [ "${#missing_command_list[@]}" -gt 0 ]; then
        reliability_log_message "ERROR: missing required commands: ${missing_command_list[*]}"
        printf 'Missing required commands: %s\n' "${missing_command_list[*]}" >&2
        exit 2
    fi
}

reliability_acquire_lock() {
    local lock_file_path="$1"
    # Let the shell assign a free descriptor instead of hardcoding one and
    # reaching for eval. A fixed number can collide with a descriptor the
    # sourcing script is already using. Requires bash 4.1 or newer.
    exec {reliability_lock_descriptor}>"${lock_file_path}"
    if ! flock -n "$reliability_lock_descriptor"; then
        reliability_log_message "Previous run still holds ${lock_file_path}, exiting."
        exit 0
    fi
}

reliability_create_temporary_directory() {
    local -n destination_variable_reference="$1"
    local created_directory
    created_directory=$(mktemp -d) || return 1
    reliability_cleanup_paths+=("$created_directory")
    # shellcheck disable=SC2034  # nameref output parameter: this assignment writes to the caller's variable
    destination_variable_reference="$created_directory"
}

reliability_run_cleanup() {
    local cleanup_path
    for cleanup_path in "${reliability_cleanup_paths[@]:-}"; do
        [ -n "$cleanup_path" ] && rm -rf "$cleanup_path"
    done
}

reliability_run_with_retry() {
    local max_attempts="$1"
    local base_delay_seconds="$2"
    shift 2
    local attempt_number=1
    local final_exit_status=0
    local backoff_delay_seconds
    while true; do
        # Capture the command's own exit status. Using `if "$@"; then...` here
        # is a trap: an if with a false condition and no else yields $? = 0, so
        # the retry would report success after exhausting all attempts.
        "$@" && return 0
        final_exit_status=$?
        if [ "$attempt_number" -ge "$max_attempts" ]; then
            reliability_log_message "Retry exhausted after ${max_attempts} attempts: $*"
            return "$final_exit_status"
        fi
        backoff_delay_seconds=$((base_delay_seconds * (2 ** (attempt_number - 1)) + RANDOM % 3))
        reliability_log_message "Attempt ${attempt_number} failed, retrying in ${backoff_delay_seconds}s: $*"
        sleep "$backoff_delay_seconds"
        attempt_number=$((attempt_number + 1))
    done
}

The refactor, before and after

The validator's opening ceremony went from this:

exec 200>"$lock_file"
if ! flock -n 200; then
    log_message "Previous run still holds the lock, exiting."
    exit 0
fi
mkdir -p "$state_directory" || { log_message "ERROR: cannot create ${state_directory}"; exit 2; }

to this, and gained dependency preflight it previously lacked entirely:

source /usr/local/lib/reliability_library.sh
reliability_initialize "connectivity_validator" "$log_file"
reliability_require_commands ping nc curl showmount smbclient mail flock
reliability_acquire_lock "$lock_file"

That preflight line is the quiet win of the whole refactor. The original validator would have discovered a missing smbclient at check seven of seven, logged a confusing SMB failure, and incremented a failure counter toward a false alert. Now a host missing a dependency refuses to start and says why, which converts a future 2 a.m. mystery into a deploy-time error message.

The paragraph this project actually earns: retry versus threshold

The design tension that makes a retry function dangerous in a monitoring toolbox is this. The validator's consecutive-failure threshold is itself a retry mechanism: cron is the loop, the counter is the backoff, and the threshold is the sensitivity dial. If individual checks were also wrapped in reliability_run_with_retry, every check would silently absorb transient failures before the counter ever saw them, and the threshold would stop measuring what it claims to measure. Three consecutive failed runs would quietly mean nine consecutive failed probes, and nobody reading the config would know it.

So the rule I settled on, and the one thing I would ask anyone sourcing this library to internalize: retry where the operation is the goal, count where the operation is the measurement. Sending the drift report email is a goal, so it gets three attempts with backoff, because a transient SMTP hiccup should not eat a report. Probing port 8080 is a measurement, so it gets exactly one attempt per run, because absorbing its failures is falsifying the data. Same function, opposite correctness, and the difference is entirely about what the caller is for.

A bug worth admitting to

The first version of the temporary directory function returned its path on stdout, for capture with command substitution. Syntax checks passed. The runtime gate did not: command substitution runs the function in a subshell, the cleanup array append happened in that subshell and evaporated with it, and the EXIT trap in the parent cleaned up nothing. The fix is a nameref assignment into the caller's variable, which keeps the registration in the parent shell. I am leaving this paragraph in the writeup because it is the honest version of the story every "battle-tested utility library" post skips: the library exists to prevent leaked temp directories and its first draft leaked temp directories. Verification gates catch what review does not.

Known limitations

Namerefs require bash 4.3 or later, which is everywhere that matters in 2026 but worth stating. Bash only, and the dispatcher project being in Python means this library's logging conventions get a second implementation there; keeping the two log formats aligned is a discipline, not an enforcement. The eval in reliability_acquire_lock is confined to a numeric descriptor but is still an eval, documented accordingly. No test suite yet beyond bash -n and the two consumers, which is honestly thin, and a bats test file is the first roadmap item for exactly that reason.