Break/Fix Taught Me Safer Automation

I came to automation from the other direction. Most people learn to build systems and then learn what breaks. I spent two decades in operations and support learning, in exhaustive detail, what breaks, and I am now building the toolchain fluency, Ansible, CI pipelines, infrastructure as code, on top of that foundation. It is a strange order to do things in, and it has one large advantage: by the time I write a script, I have already met its failure modes personally. Every guardrail below is something break/fix work made reflexive long before I knew the DevOps vocabulary for it.

Dry run is the default, execution is the flag

Support teaches you that the command as typed and the command as intended diverge more often than anyone admits, and the gap is widest under pressure. So the scripts I write invert the usual convention: running with no arguments shows you what would happen. Making changes requires an explicit flag.

if [ "$execute_changes" != "true" ]; then
    log_message "DRY RUN: would remove ${stale_snapshot_count} snapshots older than ${retention_days} days"
    exit 0
fi

The cost is one extra flag to type. The benefit is that the failure mode of "ran it by accident" becomes "read a report by accident." I have never once regretted this tradeoff.

The inversion has a second benefit that only shows up later. Because the dry run is the default, it is the path that gets exercised constantly, which means it stays correct. Tools that bolt on a dry-run flag afterward tend to have one that drifts, reporting an intention the execution path no longer matches, and the drift is invisible until someone trusts the report. Making the safe path the common path is what keeps it honest.

Bound the blast radius before you trust the logic

The instinct from incident work: do not ask "is my logic correct," ask "how much damage does this do when my logic is wrong." Because it will be, eventually, against some input nobody imagined.

Concretely, that means hard caps written into the script itself. A cleanup job that computes it should delete forty thousand files should refuse to run, because forty thousand is outside the envelope of anything the author contemplated and a human should look, rather than because forty thousand is necessarily wrong. The check costs three lines. The alternative once cost someone their weekend, and in support you are frequently the person that weekend happens to.

The number itself matters less than people expect. A cap set roughly, at some multiple of the largest run you have ever seen, catches the catastrophic case, which is the one where a path variable came back empty or a date calculation inverted and the candidate set became everything. That failure produces a number that is obviously absurd rather than one slightly larger than normal, and any cap in the right order of magnitude stops it. Waiting until you can set the threshold precisely is how the check never gets written.

Timeouts on everything that touches a network

An operation without a timeout is an operation that can hang forever, and a script wedged in a hung state is worse than a failed one: it holds locks, blocks schedules, and lies to your monitoring by neither succeeding nor failing. Anyone who has watched a process sit in uninterruptible sleep against a dead NFS server does not need this argued twice. timeout(1) wraps almost anything. Soft mounts, --max-time, -w flags: every network-touching tool has a bound, and the discipline is using it every time, not when you remember.

The reason this one is reflexive for support people specifically is that the hung script is the ticket you receive, not the ticket you file. A job that failed produces an alert and a person who knows what broke. A job that hung produces a queue that stopped moving, a schedule with a hole in it, and several hours of somebody working out which of forty jobs is the one that never came back. The timeout exists to protect the ability to diagnose everything downstream of it, well before it does anything for the script itself.

One instance at a time, on purpose

Scheduled scripts overlap. The five-minute cron job eventually takes six minutes, and now two copies are mutating the same state. flock makes this a solved problem in one line, and the support-side lesson is that this failure mode is invisible until the day it is catastrophic, because ninety-nine overlapping runs are harmless and the hundredth corrupts something.

exec 200>/var/lock/snapshot_cleanup.lock
flock -n 200 || { log_message "Previous run still active, exiting."; exit 0; }

Two decisions are embedded in those two lines and both are deliberate. The lock is taken without waiting, so a run that finds the previous one still going exits immediately rather than queuing behind it, because a backlog of queued runs against a job that is getting slower is how a small problem becomes a stampede. And the exit status is zero, because a skipped run is a normal outcome rather than a failure, and reporting it as a failure trains everyone to ignore the alert. Both choices are about what the surrounding system concludes, not about what the script does.

Log the decision, not just the action

Incident logs full of actions and empty of reasons are the bane of every postmortem I have ever contributed to. So the standard I hold my own scripts to: log what was evaluated and why the branch was taken, not merely what was done. "Removed snapshot X" is an action. "Snapshot X aged 47 days against retention of 30, removing" is a decision, and six months later it is the difference between auditing behavior and archaeology.

The version of this that pays off most is logging the decisions that resulted in no action. A run that examined four hundred candidates and removed none should say so, and say why, because the silent successful run is indistinguishable from the run that found nothing because its query was broken. That is the same silent-success problem that shows up everywhere unattended work is done, and it is exactly what designing the failure behavior first is meant to catch before it ships.

The honest frame

I want to be precise about what I am claiming here. The guardrails are earned expertise; the surrounding toolchain is work in progress. I am actively building fluency in Ansible, Terraform, and CI/CD, and plenty of that ecosystem is still new to me. What operations experience contributes is not the tools. It is the threat model: a durable, slightly paranoid sense of how automation fails in production, learned from being the person who answered when it did.

None of these guardrails are theoretical here. The locking, the structured logging, the bounded retry, and the cleanup-on-exit handling all live as tested functions in a small reliability library that the rest of my tooling sources, and its own test suite includes a regression case for a retry that reported success after every attempt had failed. That defect is what this whole essay is about, found in my own code rather than someone else's, which is the only way anybody actually learns it.