What a Bounded Blast Radius Still Lets Through

Blast radius shows up in change templates, runbook headers, automation reviews and postmortem sections. In my own archive I found the phrase repeatedly, across documents that never once defined it.

What that vagueness buys is a false sense of coverage. When a term stays undefined, any control that addresses some piece of it feels like it addresses the whole. Picture a junior engineer told to "check the blast radius" before a change ships. They count the targets, see a small number, and approve it. Nobody asked whether the count was even right, how fast it would run, or whether it could be undone. A review that asks "have you bounded the blast radius" accepts an answer that covers one of four questions.

In Microsoft's Azure Well-Architected Framework, blast radius refers to scope: how much is affected. That's the conventional sense of the term. For the narrower operational question I actually care about, whether an automated job stays recoverable, scope alone isn't enough. I use four questions of my own here: extent, rate, reversibility, detectability. This is my review model for automation failure, rather than an established definition of blast radius: four questions that expose how much damage a bounded target set can still cause, how quickly it can happen, whether it can be undone, and how soon you'd know. They're distinct questions, but they aren't mathematically independent. Rate affects the detection window. Quarantine touches both reversibility and rate. A deliberate canary changes exposure and detection together. A job can look well bounded on one question while still ending your week.

This piece also doesn't measure two things the broader term often includes. It says nothing about business severity: ten affected cache entries and ten affected billing records aren't the same risk, even at identical extent. It also says nothing about downstream propagation: what a job directly touches versus what breaks two systems away from it. Severity belongs in a review. It isn't one of these four questions.

Every experimental or calculated number in the demonstrations below is generated by the reproduction script or by an explicit calculation shown in the piece. The full set is also collected in one script. Nothing here has to be taken on the strength of the prose alone: Reproductions for this piece. It checks its own dependencies before running anything and tells you plainly if your system is missing one, instead of failing halfway through.

The demonstrations below don't all carry the same kind of weight, and I've tried to label the difference instead of letting every number read as equally empirical. Some show deterministic system behavior: shell expansion, set -u, and glob behavior with dotfiles. Those results should be consistent under the same shell settings. Some measure one controlled environment, such as the rename-against-delete timing, in a single run. Some calculate a result from assumptions I state before the numbers, as with the canary formula. One simulates a sampling model: seeded, reproducible on a given Python version, but not guaranteed across every version. Read each accordingly.

The cap that failed the case it was written for

A cap, in this context, is the simplest safety check there is. Count how many things the job is about to touch, and refuse to proceed if that count is higher than some fixed number. That list of things about to be touched, the files a delete script found, the rows a migration selected, is the candidate set. Start with the control everyone reaches for, because it deserves its reputation and it also has a specific hole. The advice, which I have given myself in writing, runs like this. Put a hard cap in the script, set it at some multiple of the largest run you have ever seen. Let the job refuse to proceed past it. The reasoning is that the catastrophic failure produces a number so obviously absurd that any cap in roughly the right order of magnitude stops it. This is the case where a path variable came back empty and the candidate set became everything.

Here's the mechanism itself, executed rather than narrated. The output below is real, shortened only by dropping the mktemp-generated sandbox prefix each path actually runs under; running the reproduction script yourself shows the unshortened version. The collapsed path used in the count below is the actual computed result of the collapse, concatenated onto a sandboxed stand-in for the real root. That's mechanical derivation, rather than a same-named directory built in parallel and asserted to match. The exact paths below aren't the point. The important part is the last line: the same expansion result feeds both, so the collapsed path is derived, with no coincidence involved in the match:

DATA_DIR correctly set,   ${DATA_DIR}/current_run resolves to: /srv/app/data/current_run
DATA_DIR unset (empty),   ${DATA_DIR}/current_run resolves to: /current_run   (a path directly beneath the filesystem root)
-- this script never touches the real /; the collapsed path below is that same result, concatenated onto a sandbox root --
collapsed expansion: /current_run   ->   sandbox representation: /stand_in_for_real_root/current_run

One unset variable and the job's path resolves to /current_run, directly beneath the filesystem root, while looking exactly like a job addressing a subdirectory of the intended application path. Bash's default behavior explains why. Reference a variable that was never set, and bash substitutes an empty string. It doesn't raise an error. So ${DATA_DIR}/current_run with DATA_DIR unset becomes /current_run. That's a real, valid path. It's just a different one than anyone intended. set -u is the standard fix, and it does catch this specific case:

-- the same unset case, under set -u, in a fresh subshell --
bash: line 1: DATA_DIR: unbound variable
(aborted: set -u caught it, exactly as intended)

It doesn't catch everything that produces the same collapse. An empty-but-assigned variable passes right through. It can arise in CI systems, environment loaders, and sourced config files:

-- set -u does NOT catch an empty-but-assigned variable --
resolves to: /current_run

For that case you need an explicit check. ${DATA_DIR:?must be set} aborts on either unset or empty.

I built a case to see whether the "obviously absurd number" reasoning holds once the collapse is shallow instead of deep. A normal target set of five thousand files. A separately sized collapsed-path target set of seven thousand five hundred. Both checked against a cap of ten thousand:

normal run       ($DATA_DIR/current_run)        candidate count: 5000   cap of 10000 -> PROCEED
collapsed path   (derived sandbox path above)   candidate count: 7500   cap of 10000 -> PROCEED

The cap passed both. Both numbers are mine, chosen when I built the test tree. Here's the actual finding, stated plainly: a cap cannot distinguish two candidate sets that both land under the threshold. The collapsed path's count is a property of whatever happens to live at that literal location, never a fixed property of the failure itself. It could be five files. It could be five hundred thousand.

Calibrate the conclusion carefully, because the cap earns its keep often, though rarely for the reason usually given. Whether a collapsed path produces an absurd count depends on traversal depth and on the shape of the tree it accidentally addressed. A deep recursive walk of a production root can generate millions of candidates and trip a sane cap quickly. This is the specific case the advice was originally describing, and a cap is well suited to catching it. A shallow walk, or a collapse that lands on a sparse part of the tree, generates a number that looks ordinary. Whether the cap catches that case depends on what happens to exist at the unintended path. Keep the cap. It's a cheap guard against an oversized candidate set. Just don't treat it as sufficient: treat it as one guard among several, with the other three questions below covering the failure modes a count alone can't see.

Extent: count the targets, and count them the way the job will

Extent is the question closest to the conventional scope meaning of blast radius. Enumerating it accurately is harder than it looks, since the enumeration has to match the mechanism doing the work.

Four files in a directory, two of them dotfiles, counted two ways. No ls sits in the pipeline. The find count uses null delimiters rather than newline-delimited output piped through wc -l, so a filename containing a literal newline can't be miscounted:

total entries via find (null-safe count): 4
glob d/* matches (array expansion, no ls): 2

In this test, a shell glob without dotglob set sees two of the four entries. dotglob is the option that tells the shell to include dotfiles in a wildcard match. Missing it cuts both ways: the job can leave files behind that a reviewer never saw counted, or a review can count a target set the job was never going to touch. Picture a cleanup script written and tested with find, then handed off and rewritten with a glob for convenience. The review saw one target list. The job runs a different one. A cleanup written with d/* leaves the dotfiles behind and reports success, so the disk stays full and the operator believes the job ran. A cleanup rehearsed with find and executed with a glob touches a different set than the one that got reviewed. Enumerate with the same mechanism you will execute with, or the count you showed your reviewer describes a job nobody is going to run.

This is where dry-run output earns its cost. A dry run that prints a count is weak evidence, since a count can be right for the wrong set. One that prints the actual resolved target list, first and last entries included, is different. It gives a reviewer something concrete to check, reading /current_run/... where /srv/app/data/current_run/... belonged, rather than a bare number to trust. Counts hide that. Paths reveal it.

Rate: how much is already running before anyone can react

Rate is a question I often see left off a review, and it determines whether a human has any role in the outcome. Concurrency, the word this section leans on, just means how many things run at the same time instead of one after another.

Forty targets, a quarter second of work each, measured against a four-hundred-millisecond observation deadline, at four levels of concurrency. This is an observation point, with no interrupt involved. Exact counts at that boundary will shift a little with scheduler timing. The durable lesson in this harness is the relationship between concurrency and how much stays queued; these specific digits aren't:

P    completed   in_progress   queued
1    1           1             38
4    4           4             32
10   10          10            20
40   40          0             0

In this run, P=10 produced ten completed, ten in progress, and twenty still queued, having never started. The exact split can move a little with scheduler timing on a different run. Here, "in progress" means started but not yet complete. The experiment doesn't model whether the underlying operation can be interrupted once it starts; that's a property of the operation itself, and queued work and already-started work can have very different stop and recovery behavior in a real job. If the job stopped accepting new work at that observation deadline, only the twenty still-queued targets would remain unstarted. The ten completed and ten in-progress targets are a different case. In a real destructive job, the completed targets would already be changed; the harness itself only sleeps and records timestamps, it doesn't change anything. Whether the in-progress ones can still be stopped depends on whether that specific operation is interruptible, something this experiment doesn't measure. Treating those last two states as one "untouched" category would hide the distinction that matters: a target already running carries a different recovery cost than one still queued, even when this experiment can't tell you what that cost is. In this run, full parallelism completed all forty simulated tasks by the observation deadline. At that point, there's no queued work left for a later stop to prevent. That trade is easy to make when someone adds -P to a slow script, and it can go unrecorded as a risk decision.

Rate control can be simple: cap the parallelism, pause between batches, or process a first batch and wait for confirmation. Each keeps a meaningful number of targets "queued" rather than "in progress" at any moment. Manual pauses don't scale well once a rollout spans thousands of targets, where staged rollout can't wait on a human between batches. The honest version is an automated stop condition tied to telemetry: an error-rate threshold, a latency spike, or a health-check regression, instead of a person watching a terminal. The rate limit also has to account for telemetry latency. At a rollout rate of ten targets a second, a sixty-second telemetry delay exposes six hundred targets to whatever the job does before the first alert can fire. Rate control buys time for detection to act; sizing one without the other means sizing against a made-up number.

Reversibility: what does undoing cost?

Reversibility measures what undoing costs, varying separately from how many things you touched. A review asking only about extent misses this.

Five thousand files, one filesystem, and a round trip rather than a one-way move. The hash strings below don't need reading character by character, only whether the "before" and "after" ones match. Each time line pairs a command with the figure it produced; the script prints those figures on their own lines and sends them to standard error, so a run redirected with > alone drops them:

environment: Linux 6.18.44-fc-v33, mv (GNU coreutils) 9.4, filesystem: ext4
source device id:      65024
quarantine device id:  65024
manifest before (content-hashed, beyond names/sizes): 7a23df5b90d566bdf9f522375f820471a2d51319ef572a9061ca877f90e29f43
time mv "$REV_SOURCE" "$REV_QUARANTINE"       # 0.001 s
time mv "$REV_QUARANTINE" "$REV_SOURCE"       # 0.001 s
manifest after:  7a23df5b90d566bdf9f522375f820471a2d51319ef572a9061ca877f90e29f43   match: yes
time rm -rf -- "$REV_SOURCE"                  # 0.032 s, no undo for this one

The manifest, each file's content hashed and the list sorted before the aggregate hash, matches exactly before and after. The script exits with an error if it doesn't, instead of only printing the mismatch and moving on. A hash is a short fingerprint of a file's actual bytes. Change one byte, and the fingerprint changes. Two matching fingerprints are strong evidence the underlying bytes match; a size-and-name check would only tell you the file looks right, since two files of the same size could otherwise differ. A same-filesystem directory rename changes the namespace entry without visiting each child; recursive deletion has to process the directory contents. That mechanism explains the timing gap above; one measurement never establishes a general performance law. The mechanism depends on staying on one filesystem. The script checks that directly. It compares the source device for the original directory against the quarantine directory. That's stronger than printing a filesystem type and asking you to trust that both locations share it. If they didn't match, the script would say so and stop, rather than let the timing claim below silently stop applying without anyone noticing.

Filesystem reversibility isn't the whole claim. A rename restores the bytes and the name. It says nothing about a process expecting the original path gone, something already recreated, or an external system reacting before you renamed back. One specific version of that gap: a process already writing to a file when the directory moves keeps writing, uninterrupted. Linux associates an open file descriptor with the underlying file itself, not with the pathname used to open it. A quarantine rename doesn't pause anything already in flight. The writer keeps streaming into the same file under its new location until something stops the process itself. "Trivially reversible" is accurate about the operation, incomplete about the system around it.

The "same filesystem" qualifier matters too. A mount boundary is where a path crosses into a different mounted filesystem, which doesn't always mean a different physical disk or volume, since Linux can mount the same filesystem at more than one point and still refuse a rename across them. Across filesystems, GNU mv falls back from rename to copy-and-remove. None of the speed advantage remains, and the disk pressure you wanted to avoid comes right back. A same-filesystem .quarantine directory, or copy-on-write snapshotting where available, is what makes rename-first pay off outside the demo. A practical alternative for a file-oriented destructive job is to stop deleting and start moving. Move into a dated quarantine instead, one a scheduled job clears after a retention window. The irreversible step then happens on a schedule, separated from the moment of the original mistake. The implementation itself is small, a directory and a cron entry, but quarantine isn't free: it consumes storage and creates a retention policy that has to be operated, on top of the process-expectation caveat above.

Detectability: the arithmetic behind a canary, and where it breaks

Detectability asks whether the damage surfaces while intervention still helps. Canary deployment, running a change against a small subset of targets first and watching for trouble before touching the rest, is the standard answer. It rests on an assumption worth naming first: that faults land on targets independently and at random. Real faults can cluster instead, by version, region, tenant shape, hardware class. Treat the numbers below as a model under a stated assumption, distinct from a description of how your fleet actually fails.

A canary catches a fault when at least one sampled target exhibits it. Under independent, random placement, for a fault present in a given proportion of targets, detection probability is one minus the chance every sample missed. "Independent" is the load-bearing word there. It means one target's fate says nothing about another's; one server failing doesn't make its neighbor any more or less likely to fail. Here's the plain version of the formula: find the odds every sample dodges the fault. Subtract that from certainty. What's left is the odds at least one sample caught it:

fault_rate  canary=1  canary=3  canary=5  canary=10
       1%   1.0%      3.0%      4.9%      9.6%
       5%   5.0%      14.3%     22.6%     40.1%
      10%   10.0%     27.1%     41.0%     65.1%
      25%   25.0%     57.8%     76.3%     94.4%
      50%   50.0%     87.5%     96.9%     99.9%

Read the five-percent row as model output rather than an observed rate. It covers only one link in the chain: it assumes the canary exercises the failing behavior and a real fault, if present, is perfectly detected. Either assumption failing lowers these numbers further. A five-target canary against a fault affecting one target in twenty detects it in 22.6 percent of trials under this model, a 77.4 percent miss rate. In those missed trials, the canary comes back green. If the rollout continues past that green result, the fault reaches the full fleet carrying a clean bill of health the arithmetic never supported, before the fault clustering addressed in the next section. Ten targets reach 40.1 percent, a 59.9 percent miss rate, under the same assumptions.

When direction matters more than sample size

A fault confined to one configuration differs from one scattered at random. It calls for a different sampling strategy: stratified sampling, deliberately drawing from every known group instead of drawing randomly from the whole fleet. This section's model differs from the canary section above it: that one assumed independent placement, this one draws without replacement from a fixed, finite fleet of two hundred, the model random.sample actually implements. The simulation uses four configurations, with the fault confined to the twenty machines running configuration D. It runs two hundred thousand trials per sample size, seeded so this matches every rerun on the same Python version; random.sample's algorithm isn't guaranteed stable across versions the way random() itself is:

fleet of 200, fault confined to config D (20 targets, 10% of fleet)
  random sample of 1   -> detection  10.0%
  random sample of 4   -> detection  34.8%
  random sample of 5   -> detection  41.3%
  random sample of 10  -> detection  66.0%
  random sample of 20  -> detection  89.1%
  stratified 1 per config (4 targets) -> detection 100.0%

The 100 percent figure is exact by construction. It's a fact about the setup more than a statistical finding: the fault is confined to one configuration, the stratified draw takes one sample from every configuration, and detection is guaranteed because one machine is sampled from every configuration, including D. The useful comparison is the one beside it. Random sampling needs twenty targets, five times the sample size, to reach 89.1 percent. That's still short of the four-target deliberate sample's guaranteed detection. That ratio is the real argument for keeping a current description of how your fleet varies. A fleet inventory isn't just documentation. It's part of the detection mechanism, because it tells you which axis your sample needs to cover.

The catch is that stratification only helps when your buckets line up with the axis a fault actually varies along. Think of it like checking one smoke detector per floor of a building instead of ten detectors crammed into the same room. That only works if the fire could start on any floor. If every real fire starts in the same room, ten detectors in that room beat one per floor. Sample by region against a fault tracking package version and your four targets may share one version, buying almost nothing over random selection. The practical rule isn't "always stratify," it's "know which axis a fault follows, and stratify along it when you can." Covering two axes at once requires additional targets and covers the common case where you guessed one right.

Structural limits bound what the job can reach

Every control so far is procedural. It lives inside the job, depending on the author remembering to add it and the code path actually executing on the day. What follows here sits outside that set of four questions entirely, a different kind of control rather than a fifth one to weigh alongside them. A structural limit lives outside the job, enforced by something with no interest in the job's intentions. Every control above this point lives in code someone wrote and could have written wrong. This one lives in the kernel instead. The kernel is the core of the operating system. Programs rely on it to perform privileged operations involving files, processes, memory, and other system resources. No earlier control in this piece uses a kernel-enforced authorization boundary as its primary protection.

Same directory tree, half owned by an unprivileged uid and half owned by root at mode 700, walked by the same command under two identities. The boundary enforcing this is ordinary Unix DAC, discretionary access control. Those are the ordinary owner, group, and other permission rules represented by the mode bits ls -l shows on every file, enforced by the kernel on every access. A production system might layer more on top. Capabilities are finer-grained slices of root's power, grantable one at a time instead of all-or-nothing. SELinux and AppArmor can impose additional mandatory policy on top of that. When a matching profile is loaded and enforcing, that policy can constrain even root. ACLs are per-file permission lists beyond the basic owner/group/everyone model. This demo isolates the base Linux discretionary-access-control mechanism. The long temp-directory path below is just where this run happened to build its test files; what matters is which half stays reachable; ignore the exact characters in the path:

-- as root, ordinary DAC permissions do not stop the job --
/tmp/revisualized_blast_radius_permission_boundary.xhWR9s/owned/scratch.txt
/tmp/revisualized_blast_radius_permission_boundary.xhWR9s/protected/critical.conf

-- as uid 65534, the kernel refuses the out-of-scope half --
/tmp/revisualized_blast_radius_permission_boundary.xhWR9s/owned/scratch.txt
find: '.../protected': Permission denied

Then the destructive operation itself, attempted under the restricted identity against both halves:

-- attempting deletion under the restricted identity against both halves --
rm: cannot remove '.../protected/critical.conf': Permission denied
owned file still present: no
protected file still present: yes

Compare that to the cap from the first section. A cap is a line of code, and its protection depends on the implementation remaining present, reachable, and correctly enforced. It's editable, bypassable, or removable by a maintainer who found it firing during testing. It's a decision that looks reasonable at the time and leaves no trace in review. The permission boundary doesn't depend on the job checking its own path. Once the process runs under the restricted identity, the ordinary DAC enforcement demonstrated here applies regardless of what the job's code does or doesn't verify. What it guarantees is narrower than "defeats every bug": it sets a hard upper bound on what the failure can affect. A collapsed path or wrong glob that stays inside the permitted scope still does whatever damage it was going to do. The boundary doesn't reach inside its own perimeter. What it does reach is everything outside it, unconditionally.

This reframes scope from something a script declares into something a script is granted. Local uid switching, the smallest complete example, is one implementation. Namespaces wall off what a process can even see: its own view of processes, networking, or the filesystem. Read-only filesystems restrict what it can modify, a different property than what it can see. RBAC, role-based access control, assigns permissions through defined roles rather than directly to individual accounts, constraining what an account is authorized to do. Cgroups bound how much CPU, memory, or I/O it can consume, a separate axis from what it can reach and a related but distinct form of containment. The principle generalizes across scales: a job's effective authority can be limited to the target it's currently addressing, rather than one identity holding unrestricted access to everything, even though the exact implementation doesn't generalize the same way.

The cost is real: someone has to create and maintain those identities as systems change, ongoing work competing with shipping. Running automation as root removes that independent boundary. What remains depends on the job enforcing itself correctly. One limit specific to this demo: the script itself starts as root. It only drops to the restricted identity when setpriv runs, so anything executing before that line still holds full authority. Production automation should drop privilege before the process starts instead, a systemd security directive or a container's execution policy.

A review card, meant to be used

Four questions, each with a factual answer in place of an opinion, meant to sit beside a change review instead of summarizing this piece.

BLAST-RADIUS REVIEW

EXTENT
  What can this job touch, counted by the mechanism that will execute?
  Can I see the first and last resolved paths, beyond a bare count?

RATE
  How many targets are already running before a stop can take effect?
  What's queued versus what's already in progress at that point?

REVERSIBILITY
  What exactly undoes this operation, and does it stay on one filesystem?
  Does recovery depend on a backup nobody has tested?

DETECTABILITY
  Which failure surfaces first, and which target would expose it?
  Is the sample deliberate, or random against a fault that has direction?

A review that covers extent and detectability, a scope question and a detection question, can still pass while rate and reversibility go unspecified. Those are the two questions that say how much room you have to intervene once a bad outcome starts, and whether intervention can actually undo it. That gap is the actual argument of this piece. The useful question was never whether a change has a blast radius. Every change does. The useful question is whether you know what it can touch, how fast it can happen, what stays reversible, and how soon the failure becomes visible.

Where this evidence comes from

Sources verified 2026-09-17 (the four below; GNU Bash, Findutils, and Coreutils below them were added in earlier revisions and are carried forward unchecked this pass):

• Microsoft Azure Well-Architected Framework, failure mode analysis, for the conventional scope-based definition of blast radius cited above. https://learn.microsoft.com/en-us/azure/well-architected/reliability/failure-mode-analysis
• rename(2), Linux manual page, for open file descriptors on the old pathname surviving a rename, and for EXDEV being scoped to the same mounted filesystem rather than the same device in every mount topology. https://man7.org/linux/man-pages/man2/rename.2.html
• Python random module documentation, for random.sample's sampling without replacement and its algorithms being subject to change across Python versions. https://docs.python.org/3/library/random.html
• setpriv(1), Linux manual page, for dropping privilege when running a job under a restricted identity, and for UID changes not by themselves altering the inheritable and bounding capability sets. https://man7.org/linux/man-pages/man1/setpriv.1.html
• GNU Bash manual, shell parameter expansion and the set -u option for unbound variables. https://www.gnu.org/software/bash/manual/bash.html
• GNU Findutils manual, find predicates and traversal depth control. https://www.gnu.org/software/findutils/manual/html_mono/find.html
• GNU Coreutils manual, mv and rename semantics within and across filesystems. https://www.gnu.org/software/coreutils/manual/coreutils.html
• Reproduction script for the experiments and calculations in this piece. https://github.com/revisualize/blast-radius-reproductions