Idempotency Before the Word

A question worth asking about every script you write sounds almost too simple to matter: what happens if it runs twice?

Not because you plan to run it twice. Because the world does. Cron fires a job whose previous run has not finished. A script dies halfway and gets rerun. A network hiccup makes a caller retry. A colleague, unsure whether you already ran the thing, runs the thing. In real operations, "this will only ever execute exactly once" is a promise nobody can keep, so the scripts that survive are the ones that never needed it. A script that produces the same correct end state whether it runs once or five times has a property with a formal name, idempotency, and I want to sell you the property before the vocabulary, because the property is the part that saves your weekend.

Actions and destinations

Watch it in miniature, in two commands that look like siblings:

mkdir /var/lib/my_tool
mkdir -p /var/lib/my_tool

Run the first twice and the second run fails: the directory exists. Run the second twice and both runs succeed, and the end state is identical. mkdir describes an action, "create this." mkdir -p describes a destination, "this should exist," and a destination you have already reached is not an error. That one conceptual flip, from actions to destinations, is the entire subject.

Now the version that bites. A setup script appends a line to a config file:

echo "max_connections=200" >> server.conf

Run it twice and the file holds the line twice, and depending on the software, duplicate directives are ignored, or the last one wins, or parsing breaks. The action succeeded both times; the destination was only correct after the first. The idempotent form checks the destination first:

grep -q '^max_connections=200$' server.conf \
    || echo "max_connections=200" >> server.conf

Read as a sentence: ensure this line is present. Same shape everywhere. rm fails on a missing file; rm -f ensures absence. Creating a user account errors if the account exists; ensuring the account exists does not. The idempotent versions are usually one flag or one guard clause away, and the habit is asking, for every state-changing line, "what does this do to a world where it already ran?"

That guard-clause form is worth reading closely, because it is easy to write a version that only looks idempotent. The anchored pattern in the example matches the whole line, so a file containing max_connections=2000 does not satisfy it and the correct line still gets added. An unanchored grep -q max_connections would match that wrong value, conclude the destination was reached, and leave the file misconfigured while reporting success. The check has to be as specific as the thing it is asserting, and a check that is too loose converts a missing line into a silent misconfiguration, which is worse than the duplicate it was written to prevent.

Why this makes retries safe

Two payoffs make this more than tidiness.

The first is that idempotent operations make retries safe, and retries are the backbone of every reliability mechanism worth having. An SMTP dispatcher can honestly promise at-least-once delivery, meaning a rare duplicate email instead of a lost alert, only because a duplicate is harmless. The same retry logic wrapped around a non-idempotent action, charge the card, append the line, would convert every transient failure into a small disaster. Retry and idempotency are a package deal: the first is only safe in the presence of the second.

This is also the reason the two get built in the wrong order so often. Retry is the visible feature, the thing that makes a flaky job stop paging people, so it goes in first and appears to work. The idempotency it silently depends on is nobody's ticket. The result is a job that is fine for months and then, during the one outage where the retry actually engages several times, does its work several times. The retry did not cause that. It revealed it.

Why this makes failures resumable

The second payoff is that idempotent scripts are resumable. When a ten-step setup script dies at step six, the non-idempotent version demands archaeology, figuring out exactly which steps completed so you can hand-run the remainder. The idempotent version has one recovery procedure: run it again. Steps one through five notice their destinations are already reached and glide through; step six resumes the work. That single property converts "partial failure" from an incident into an inconvenience.

The value of that shows up most in who can perform the recovery. Archaeology requires the person who wrote the script, or at least someone willing to read all ten steps at three in the morning and reason about which ones left traces. Run it again requires nobody in particular. A recovery procedure that fits in four words is a recovery procedure that survives staff turnover, vacation, and the specific engineer being unreachable, which is most of what makes automation trustworthy rather than merely clever.

Where the idea leads

And here is where the concept opens a door, because if you follow "describe destinations, not actions" to its conclusion, you stop writing scripts full of guarded commands and start writing descriptions of the end state, handing the how to an engine that checks the world and converges it. That is precisely what configuration management and infrastructure-as-code tools are. An Ansible task does not say "append this line," it says "this line should be present in this file," and running the play against an already-correct host changes nothing, reports nothing changed, and that nothing is the entire point.

I am actively building fluency in that toolchain, and the honest framing is that the tools are new to me while this idea is not: two decades of watching automation get rerun at the wrong moment teaches you run-twice safety in your bones, and it turns out the industry built a whole discipline on top of it.

If you internalize one transferable concept before touching any of those tools, make it this one, and the way to internalize it is not to read about it. Take a script you already own, run it twice against the same input, and compare the two end states. Whatever differs between them is the part that was never idempotent, and you will find it in about a minute.