Automation October 17, 2026 16 min read

When the Automation Platform Is What Failed

Picture a team automating a recovery procedure. The action is reviewed for blast radius, given a rollback path, wired to a verification step, and shipped. Each new procedure gets the same treatment, until forty procedures run through one platform: a scheduler, an executor, a policy check, and a credential that can reach every target on the list.

Then the platform stops. The services it manages are still healthy. What failed is the thing that decides, authorizes, and executes.

Each procedure had an answer for its own failure, and every one of them inherits the platform's. The platform needs an answer to a different question: if the automation platform itself fails, who detects it, what fails safe, and how does the organization continue?

Some platforms answer most of that by construction, with leader election, durable queues, idempotent actions, and reconciliation after restart; for them this page is a checklist. A smaller platform is a scheduler, one service account, and a directory of scripts that grew one procedure at a time. This page is for that platform.

Three ways the platform fails

Flaviu Cristian's 1991 classification of server failures names several classes, among them omission, timing, response, and crash. For a small automation platform, three situations drawn from them need separate handling.

It stops running. A crash in Cristian's sense: after a first omission, the server produces nothing until it restarts. The scheduler is down, the executor cannot start, or the queue has stopped draining. A stopped nightly reconciliation produces no error, only drift found later.

It runs and decides wrongly. A response failure: the platform responds with the wrong value or state transition. The policy check is answering from stale inventory, from an approval it should have treated as expired, or from a target list that grew past its intended scope. Actions are taken, they report success, and each one carries an authorization nobody would have granted. For recovery automation it can do more harm than a clean stop, because the record it leaves looks like normal operation.

Its state is unknown. This is an operational condition that several of Cristian's classes can produce. The platform is unresponsive, and from outside, a crashed platform and a slow one look identical; waiting alone cannot tell you which one you have. If it was slow, an action may still complete. Acting by hand means first deciding whether an action is still in flight, and if the only record of in-flight work lives inside the down platform, nobody can answer that. It is the half-run problem at platform scale. Waiting is the safe move, made under pressure to act. NIST SP 800-53 control SI-17, in its discussion, lists doing nothing among the steps that may follow alerting operators, a documented basis for waiting.

A liveness check notices silence but cannot tell a dead platform from a slow one. A platform that decides wrongly passes liveness checks, because it is alive.

Detecting a platform that cannot report on itself

A platform cannot be the sole witness to its own health. Every check in this section follows one rule: the evidence has to come from outside the failure domain it is meant to detect. If the platform raises or delivers its own alerts, the failure that matters most is the one that silences them.

For a platform that stops, the detector watches for expected events that never arrived. A watcher on a separate host, with a separate credential, holds a list of expected jobs from a source the scheduler cannot modify, such as protected job definitions under version control, and alerts when a job's events stop arriving on time. One reading the scheduler's own list misses the job it lost.

For a platform that decides wrongly, the detector checks invariants the policy check was supposed to enforce. Three invariants cover the expired approval and an oversized batch: no action without a matching approval record, no approval used outside its validity window, and no more actions per window than the declared ceiling. A count limit passes a wrong list that stays under it, the case What a Bounded Blast Radius Still Lets Through builds, so stale inventory needs a scope check against the target's own state before authorization and target evidence afterward. Give every request one identifier that travels through approval, execution, the target's own log, and verification. Then compare what the targets say happened with what the platform says it did, and investigate a lasting difference.

For a platform whose state is unknown, detection becomes a recording problem. Each action gets one record, kept as appended events: opened, decided, dispatched, an executor-observed or indeterminate outcome, verified or not performed, closed. A refusal goes from decided to closed. Each event is made durable before the next step depends on it; for a file, that means fsync(2) after every append, plus a sync of the containing directory when the file is created or renamed. A timeout or a lost connection after dispatch records an indeterminate outcome. Dispatched is written before the request goes out, so a process failure after that point leaves arrival unknown. Either way, only target evidence, from outside the platform, moves an action to verified or not performed, and missing evidence leaves it indeterminate; an executor that returned has only reported. A local record survives a crash; for loss of the host, the next step waits for an acknowledged, durable write off the host.

The same events sharpen the watcher. No dispatch by the deadline for an eligible job narrows the search to scheduling or dispatch, a dispatch with no outcome to execution, and an unverified outcome to verification or its evidence. Each detector also needs a named owner and an escalation path outside the platform. The watcher can fail too, so give it its own heartbeat, checked from outside the platform and the watcher. Test the chain once: withhold a harmless canary job's events and confirm the alert reaches a person.

Failing safe has a direction

Failing safe needs a decision, because the two safe directions conflict.

Fail closed means the platform stops acting when it cannot establish authority. An expired approval, an incomplete evidence bundle, a policy check that cannot be reached: each becomes a refusal. Saltzer and Schroeder's 1975 design principles name the basis fail-safe defaults, crediting E. Glaser: base access on permission, because a mistake in that direction is quickly visible and one in the other can go unnoticed.

Fail open means the platform keeps acting when it cannot establish the authority or evidence an action requires.

For recovery automation, when a change needs current authority or current evidence, losing either should stop the change. That rule has a price: the remediation the platform would have run never runs, and the outage continues until a person acts. The right failure state still depends on what the function controls; NIST SC-24 (Fail in Known State) leaves that choice to the organization.

Read-only functions are harder: evidence collection, inventory, and status reporting. Failing those closed loses evidence at the moment it is most useful. So split the decision by function, and let the platform lose the ability to act before it loses the ability to observe. Decision and mutation fail closed. Collection and observation degrade, and label their own output as degraded so a consumer knows what it is holding.

The split needs one more rule: degraded output may inform a person, and it may never satisfy the evidence requirement for a mutation. A half-blind collector can still inform a responder; its gap is what a mutation's evidence check exists to catch.

Recording needs its own decision. NIST AU-5 leaves the response to an audit-logging failure to the organization, listing options from overwriting the oldest records to shutting the system down. Narrowed to mutation, the strictest option reads: if the action record cannot accept a write, the executor stops. That makes the recorder a dependency of every remediation, so it has to be the simplest and most available component the platform has.

Separation of duties across the platform

A directory-of-scripts platform can run everything as one service account. That account schedules, evaluates policy, executes, and writes the log entry saying the action was approved and verified. When the platform decides wrongly, every record that could show the error was written by the identity that made it.

Saltzer and Schroeder call the remedy separation of privilege: with two keys held by different parties, no single accident or breach of trust opens the lock. NIST control AC-5 applies separation of duties to people and roles. Extending it across the platform goes beyond the control. Four of the roles below can be machine identities; approval belongs to a person or an independently controlled authorization system. Few platforms need all five at once; the point is removing combinations of authority:

Role	May do	Must be unable to do alone	Failure it contains
Scheduler	Start a declared job at a declared time	Change what the job does or which targets it reaches	A runaway schedule can repeat a job and cannot widen it
Policy check	Read the request, the approval store, and the evidence; return allow or refuse with a reason	Execute anything, or edit an approval	A compromised policy check cannot run the action itself
Approver	Approve or reject one declared action, in scope, with a validity window	Change the action's implementation or the action record	An approval cannot be stretched to cover a different action
Executor	Perform exactly the declared action against an allowlisted target	Run a free-form command, grant itself approval, or edit the action record	A compromised or confused executor is bounded by its allowlist
Recorder	Append events to the action record	Rewrite or delete an existing entry	The executor cannot erase the record of what happened

Separation limits what each role can do; the outside invariants catch a wrong decision. The approver's row holds only if the approval names exactly what it covers: the request identifier, a digest of the declared action, the target scope, and a not-before and not-after time. Both the policy check and the executor compare the action in hand against that binding, and the approval must still be valid when the executor starts the change.

NIST AU-9 asks that audit information be protected from unauthorized access, modification, and deletion; the recorder is one way to meet that. A separate identity protects the record from the actor. A separate host moves it outside that host's failure domain.

On one host, the first version needs no new tooling: give each machine identity its own Unix account and its own credentials, since four accounts sharing one key or one broad sudo rule still act as one. Limit the executor, through sudo rules, to allowlisted commands with fixed or validated arguments. Keep the action record in a file only the recorder account can write, on a filesystem that supports the append-only attribute, set with chattr +a; only root or a process holding CAP_LINUX_IMMUTABLE can set or clear it (chattr(1)). Permissions keep the executor out, and the attribute limits the recorder to appends. It is containment rather than an audit system: descriptors opened before it was set keep their access, a privileged process can clear it, and rotation takes a privileged step. With per-append fsync(2), the record survives a crash; loss of the host needs the off-host write.

A denial path, worked

To know what fails safe, make the platform refuse something on purpose and read what it does; otherwise a denial path first runs during an incident.

A constructed example. A request to restart one lab service instance carries an approval valid for thirty minutes, and it is evaluated forty-two minutes after approval. A correct platform leaves this record:

request_id: REQ-10472
request:    restart service instance lab-svc-02
approval:   APR-0412 for REQ-10472, valid 09:10 to 09:40 UTC, approval store clock
evaluated:  09:52 UTC, policy check clock
skew bound: 2 minutes, both on lab NTP
decision:   refused
reason:     approval expired 12 minutes before evaluation
action:     none taken
recorded:   action record, before the response was returned
returned:   requester notified with the reason

Five things make that refusal trustworthy. Its decision names the condition that failed. No part of the action ran before the check. Recording happened before the response went out, so a crash between the two still leaves a record. That record went to a store the executor cannot edit. Resubmitting it unchanged produces the same refusal.

Expiry compares a time written on one machine with a clock on another. Name the time source both clocks follow, record which clock wrote each timestamp, and state the skew the check tolerates. Outside that bound, refuse and name clock drift as the reason.

Next, make the approval store unreachable and send a request with a valid approval. A platform that fails closed refuses with a reason that names the unreachable store. A platform that hangs without a bounded decision is worse than one that refuses, because a person can act on a refusal and a hang is silence. A platform that proceeds solely because it cannot check current authority has shown that this path fails open whenever the approval store is down.

A third test follows from the recorder decision: make the action-record store refuse writes and send a valid request. The executor should stop before the action, and say why.

Run all three on disposable systems.

The manual path decays

How the organization continues has two halves. The first is working while the platform is down, which needs an execution path independent of the failed platform. For a small platform that path is a person following a runbook. Automating a procedure removes the reason anyone performs it by hand, so the manual path goes unexercised, the people who knew it move on, and its document drifts as the system changes. Eventually the automation can become the only working knowledge of the procedure, held as code that assumes a running platform.

Keep the manual procedure as a real runbook a person can follow: preconditions, an expected outcome for each step, an exit condition, and an escalation path. A stub is worse than a stated gap, because it reads as coverage.

Exercise it. A manual path nobody has walked since the automation shipped is a hypothesis. Walking it on a schedule, and after material changes, surfaces the drift before an incident does.

When the platform comes back

Coming back without damage is the second half. Restarting is the easy part. The hard part is keeping work issued before the failure from landing among the new work.

First: what happens to the work the scheduler missed? Scheduler defaults leave it unanswered. Cron's special handling, in both cronie's and Debian's cron(8), covers local clock changes under three hours, for fixed-time jobs. For downtime, anacron(8) exists because cron assumes the machine runs continuously; it runs a day-scale job that missed its period. A calendar-based systemd timer with Persistent=true records when it last fired and, on activation, triggers the service once if at least one run was missed (systemd.timer(5)). Forty missed hourly firings produce one catch-up activation; without the option, they produce none. Timers catching up at the same boot also start together, and RandomizedDelaySec= can spread them out.

A reconciliation that converges on desired state wants one catch-up run. A job that processes each interval's data separately needs every interval, and none of these mechanisms provides that on its own. Catch-up policy belongs to the job's owner, written next to the schedule, and any job whose policy is "whatever the scheduler does" is still undecided.

Second: is anything from before the failure still moving? A lock is weak evidence either way. On one host, a flock(2) lock belongs to the open file description, so a background child that inherited the descriptor keeps the lock after the script that took it has exited. The exec 200> form in Break/Fix Taught Me Safer Automation is sound until such a child starts. Across hosts, a lease that expired shows only that its holder stopped renewing. Burrows's 2006 lock-service paper describes the remaining hazard: a holder sends a request, fails, loses the lock to another client, and its old request arrives afterward. His remedy is a sequencer: the lock's name, mode, and generation number, which the target checks before acting. Its number is now often called a fencing token.

A small platform can borrow the idea. The lock issues a fencing number that increases and is never reused. After restart, no change to a target resumes until that target confirms it has durably recorded the new number; an unreachable target holds back only its changes. A target-side wrapper sits on every path that can change the target, records the newest number durably, refuses any lower one, and checks the number at the same serialization point that authorizes and performs the change. A request issued before the failure that arrives after that confirmation is refused; one that lands before it is an unknown outcome, which the record handles. A recovering platform will be asked to run work it may already have run, so retry an unknown outcome only where idempotency makes repeating it harmless; otherwise reconcile from target state read after confirmation.

Signals that the platform is drifting

A few measures show the platform degrading before it fails. Decisions should complete within a stated time or fail closed within it. Unauthorized, expired, and out-of-scope executions should number zero, counted outside the platform. Actions that ran should close on target evidence. Evidence collection should report its own completeness, which may be the earliest of these to move. Rising approval latency gives people a reason to route around the platform.

What to do first

If none of this exists, order beats coverage.

For the three automations with the highest consequence, write down who detects a failure, what fails safe, how the work continues, and what happens to runs it missed. If those four answers cannot be written, that is the finding.

Then run the denial tests: an expired approval, an unreachable approval store, and an action-record store that refuses writes. Confirm the platform refuses and says why, and that each refusal it can record lands somewhere it cannot edit.

Then walk one manual procedure end to end against a disposable environment and record each place the document and the system disagree. That defect list tells you whether the rest of this page is a program you need.

References

Sources verified 2026-10-14. Manual pages are cited from a community-run mirror that serves the current package; the version named is the one read on the verification date, and behavior can change between releases.