Design for the Failure First

Every system has two behaviors: what it does when everything works, and what it does when something breaks. The first one is easy to design, easy to demonstrate, and almost never the one that matters. The happy path is the version you build first because it is the version you can watch succeed, and watching it succeed is exactly what convinces you that you are done. You are not done. A system's real character is its failure behavior, because in anything that runs long enough, failure is routine. Networks drop, disks fill, a dependency goes down, a job fires twice, a command finishes halfway and dies. Failure is not an exception to handle later. It is the first thing to design.

The effort is usually spent backwards

Most of the design attention goes to the path that needs it least. The happy path will, for the most part, take care of itself, because it is the case everything was built to do. The paths that will actually cost you a night are the ones that get a shrug and a "we will handle errors later," and later is the incident. Reversing that is most of what separates a script that survives production from one that merely demonstrated well once. You design the failure behavior deliberately, up front, and let the happy path be the easy part it always was.

The questions you ask before it ships

The discipline is a small, fixed set of questions, asked of anything that will run unattended and matter:

• How does this fail? Name the modes out loud. The network is unreachable, the disk is full, a permission is wrong, a dependency is down, the input is malformed. You cannot design for failures you have not named.
• What does half-done look like? This is the one people skip and the one that gets them.
• Who finds out, and how? A failure nobody hears about is the most expensive kind.
• Can it safely run again? If the answer is not a confident yes, you do not yet have a recoverable system.
• What does it leave behind when it dies? Partial files, held locks, a half-written state.

None of these are error-handling syntax. They are design questions, and their answers change the shape of the thing before a single error is caught. Asking them is cheap. Discovering their answers during an outage is not.

Partial failure is the one that gets you

Total success is fine and total failure is usually loud, obvious, and safe, because a job that clearly did nothing leaves you a clean place to start again. The failure that costs you is the one in the middle: the run that copied half the records, wrote half the archive, updated the database but not the index, and then stopped. A system that cannot tell you whether it half-ran is worse than one that failed cleanly, because now you can neither trust its output nor safely repeat it. The two design answers are atomicity, where the operation either fully happened or did not happen at all, and idempotency, where running it again is safe no matter how the last run ended. Build one of them in and the half-done question stops being the thing you dread.

Atomicity sounds like it needs a database, and at the level most unattended jobs operate it needs one filesystem property instead. A rename within a filesystem is atomic: the destination name either refers to the old file or the new one, never to something half-written. So the pattern that makes a file-producing job safe is to write to a temporary path in the same directory, flush it, and rename it into place as the last action. A job killed at any point before the rename leaves a stray temporary file and an intact previous version. A job killed after it has already finished. The failure window shrinks from the whole duration of the write to the instant of a rename, which is as close to eliminated as this class of problem gets.

Two details decide whether that pattern actually holds. The temporary file has to live in the same directory as the destination, because a rename across filesystems is a copy plus a delete and is not atomic. And the durability guarantee needs the data flushed before the rename, or a crash can leave you with a renamed file whose contents never reached disk, which is the worst of both outcomes: a file that looks complete and is not. Get those two right and the pattern is genuinely reliable. Get either wrong and it provides confidence without protection, which is worse than no pattern at all.

A silent failure is worse than a loud one

The failure you can see is a good day. The one that matters is the job that ran at three in the morning, reported success, and left you to discover a week later that the backups have been empty the whole time. Designing for failure means designing so the failure has a voice: the job exits non-zero when it should, it records what it was doing at the moment it stopped, and it raises the alarm in a way you will actually notice rather than one more line in a channel that has cried wolf so often nobody reads it. An unattended system's only account of itself is the output you designed for it, and if you did not design that account, the system goes mute at precisely the moment you need it to speak.

The exit status deserves specific attention, because it is the part most often broken by accident in shell. A pipeline reports the status of its last command, so a check that ends in a formatting step will report success even when the command that produced the data failed outright. A function whose result is captured in a command substitution runs in a subshell, so its failure does not stop the caller. A loop that logs an error and keeps going will exit zero having done nothing useful. Every one of those is a job that reports success after failing, which is the same defect as the empty backup wearing different clothes. Whatever else you check before shipping an unattended job, check what it returns when the thing it depends on is broken, because that is the value everything downstream will trust.

Testing the happy path proves almost nothing

Failure design gets skipped because the happy-path test passes, and a green test feels like proof. But the happy path was never in question. The test that tells you something is the one that pulls the network mid-run, fills the disk, kills the process halfway, and then checks that the system did something sane with the wreckage. If you have never watched your own system fail on purpose, you do not know how it fails. You are guessing, and the first honest test of your guess will be the incident itself.

The objection to this is always that fault injection is expensive, and at the level most of these jobs operate it is not. A dependency you do not control can be replaced by a stub on the path that returns an error. A full disk can be a small loopback filesystem or a quota. An unreachable host can be an address that routes nowhere. A process killed halfway is one signal at a checkpoint you chose.

None of that requires a chaos engineering platform, and all of it can live in the same test suite as everything else, which is what turns "we should test failures" from an intention into a set of cases that run every time. The reason to write them as tests rather than perform them once by hand is that the failure paths are exactly the code nobody exercises again, so they are exactly the code that rots without anyone noticing.

Not everything earns this

Failure-first design has an opposite failure mode, which is applying it to everything. Most things do not need it. A command you run once and watch, a script you will happily rerun the second it errors, an interactive tool with a person in the loop: wrapping those in retries and traps and alerting is wasted effort, and it is its own junior mistake, the one where every trivial script grows the machinery of a load-bearing service it was never going to be.

The judgment is knowing which systems earn the discipline. The ones that run unattended, on a schedule, where nobody is watching and a failure is either expensive or invisible, earn all of it, up to and including the layered failover thinking a real availability path demands. The ones a human babysits earn almost none. Spending the failure budget where it pays, and not everywhere, is the actual skill.

The move underneath all of it is to design backward from the moment it breaks. Before you ship, picture the thing failing at the worst possible time, unattended, and ask what you will wish it had done. Told you plainly. Left nothing corrupted behind it. Been safe to run again.

Those three are worth treating as a checklist with a signature rather than as advice, because each one has a specific way of being faked. A job tells you plainly only if you have checked what it returns on the failure path, not just on the success path. It leaves nothing corrupted only if the write pattern makes partial output impossible, not merely unlikely. It is safe to run again only if someone has actually run it twice against the same input and compared. Those three verifications take an afternoon on a job that will run unattended for years.