Infrastructure 10 min read

Why Change Management Is Deliberately Slow

Somewhere right now, a meeting is spending an hour on one paragraph of a change plan. Around the table, or the call, are people who have read the same procedure three times, asking questions like what happens if step four fails, who verifies the state before step five, and what exactly is the back-out if the second half goes sideways. To the modern sensibility this scene is the definition of legacy dysfunction. Deployment frequency is the celebrated metric of the age. Elite teams ship many times a day. An hour per paragraph reads as a pathology rather than a process.

Sometimes it is exactly that. And sometimes that meeting is the single most rational activity in the entire organization, and the difference between the two is not culture or maturity. It is weight.

Rollback is a story you have to be able to tell

Every change process, fast or slow, is secretly organized around one question: if this goes wrong, how do we get back?

In the weightless world, that question has a one-word answer. The desired state lives in a repository; the running system is a projection of it. Deploy, observe, and if the observation displeases you, revert. The cost of being wrong is minutes, so the correct strategy is to be wrong often and cheaply, and everything about modern delivery culture, the small batches, the automated gates, the ship-constantly cadence, follows from that low price.

Now change the physics. A firmware update on a storage controller. A schema migration on a database under load. A modification to a replication relationship protecting years of state. An upgrade to the firewall pair that everything transits. For these, the previous state is not sitting in a repository waiting to be reprojected, because the previous state includes the data, and the data has been mutating since the moment the change landed. Rollback, where it exists, is a procedure: multi-step, manual, time-boxed, and rehearsed, with its own failure modes. Where it does not exist, the only path is forward through the failure, on a bridge call, at whatever hour it is.

There is a sharper version of this worth stating, because it is the distinction most change plans get wrong. Reversibility and recoverability are different properties, and a plan that conflates them will be confident at exactly the wrong moment. A change is reversible if you can put the system back the way it was. A change is recoverable if you can reach a good state by some path, which may be forward. Most weighted changes are recoverable and not reversible, and the honest change plan says which one it is holding. A back-out step that reads "restore from backup" is a recovery claim wearing the costume of a reversal, and the difference becomes visible only when someone asks how long the restore takes and whether anyone has run one at that size recently.

When reversal is cheap, moving fast is correct. When reversal is expensive or impossible, moving deliberately is correct. The change window, the maintenance freeze, the pre-checks and post-checks, the hour spent on one paragraph: these are the price of consequence, and organizations that operate weighted systems pay it because the alternative is paying much more.

What the slow ritual is actually doing

In a well-run change window on a weighted system, every piece of apparent ceremony turns out to be doing engineering work.

The pre-check is establishing ground truth. Before touching anything, the engineer verifies the system's actual state, not its assumed state: health, redundancy, capacity, replication currency, the condition of every safety net the plan depends on. This is where the difference between the documented environment and the real one gets discovered, and it is discovered before the change instead of during it. It is also the part of the ritual that scripts best, which is why a declarative pre-flight manifest that reports every unmet assumption rather than stopping at the first is worth more than a checklist someone reads aloud.

The privileged interactive session, the thing the ideal most wants to abolish, is there because judgment is part of the procedure. The engineer running the change is watching it happen, reading output as it scrolls, holding at checkpoints, comparing observed behavior against expected at every step. The session is slow precisely because a human is in a position to stop. Automation executes a plan; it does not notice that step three produced something subtly unlike what step three usually produces. On systems where the cost of not-noticing is measured in data, keeping a human in the loop is a control, not a failure to automate.

The post-verification is the rollback decision in disguise. Reading the logs the system actually wrote, confirming services resumed, checking that replication caught up: this is the evidence-gathering that answers the only question that matters, are we in the good state or do we invoke the back-out while the window is still open. And the window itself is blast-radius containment: a bounded period where the risk is scheduled, the experts are awake and assembled, the users are warned, and the back-out has room to run. It is the weighted world's equivalent of a canary deployment. Same instinct, different physics.

The comparison is worth pushing on, because the window has one property the canary does not, and it cuts against the slow process. A canary is reversible mid-flight; you halt the rollout and the blast radius stops growing. A change window is a commitment to a duration, and once the change is underway the clock is the constraint that everything else negotiates with. This is why weighted change plans obsess over time estimates that look excessive from outside. The estimate functions as an input to the decision about whether the back-out still fits before the window closes, rather than as a schedule, and an estimate that was optimistic by ninety minutes is how a recoverable change becomes an outage. Anatomy of a Failover is that arithmetic worked through on a specific procedure, where each layer's recovery timer stacks on the one beneath it and the total is what the window has to accommodate.

None of this precludes tooling, and the best-run weighted environments script their pre-checks, their verification, and their evidence collection aggressively. The ritual subordinates automation to judgment, instead of the other way around. That translation, automating the evidence while leaving the decision human, is the third of the five practices in Modernizing What You Cannot Redeploy.

The ownership that appears on no diagram

Ownership is the second force that makes enterprise change slow, and it appears on no architecture diagram.

The ideal quietly assumes a unified team with access to everything it manages. One platform group, one pipeline, and whoever owns the code can restart, rebuild, or redeploy the thing it describes. Large enterprises are not built that way. Access is deliberately fragmented: the storage team holds the arrays, the network team holds the fabric, a UNIX team holds one population of servers and a Windows team another, security holds the appliances, and some boxes belong to no one but a vendor. Each boundary exists for a reason, least privilege, separation of duties, audit, contract, and each one is also a wall a change has to climb.

So the simplest imaginable operation, restart this one appliance, is a ticket. The ticket routes to a queue, and the queue belongs to a team, and if the team does not have access to the resource, the ticket does not get done. It gets closed, or bounced, or rerouted, and a task that takes ninety seconds of hands-on-keyboard takes four days of hands-on-process. Anyone who has worked inside a large organization recognizes this instantly, and anyone designing a modernization plan for one had better account for it, because those access boundaries have their own gravity. A pipeline that assumes it can touch what it manages will not survive contact with an enterprise that has decided, for defensible reasons, that nothing gets to touch everything.

The fragmentation has a second-order effect that is easy to miss and expensive to discover. Every boundary a change crosses is a point where context is transcribed rather than transferred. The storage engineer's precise statement of what they need becomes a ticket summary, which becomes a queue item, which becomes a different engineer's interpretation of an unfamiliar request. Nothing is lost dramatically. It degrades, one handoff at a time, and the change that executes is subtly not the change that was designed. The slow ritual's insistence on a written plan reviewed by everyone who will touch it is a direct countermeasure to exactly that decay, which is why the meeting has people in it who will not run a single command.

This is also why the bridge call is a permanent institution rather than a transitional embarrassment. When an incident spans the boundaries, and the interesting ones always do, the only mechanism that can cross every wall at once is people, assembled live, each bringing the access and the context their team holds. The most automated organizations on earth still convene them, at scale, in their worst hours. That is what operating consequential systems under fragmented ownership looks like, and it is universal.

Calibration, not doctrine

None of this is a defense of slowness as a virtue. Slowness is a cost, always. Change processes accrete ceremony that has outlived its reason; approval steps survive because removing them requires a meeting nobody schedules; and plenty of change advisory boards are reviewing stateless services that could ship forty times a day without anyone noticing. That is real dysfunction, and calling it out is fair.

It is worth being just as fair to the fast world, which dissolved change management into the pipeline rather than abolishing it. The canary release, the feature flag, the progressive rollout to one percent of traffic before a hundred, the automated gate that halts a deploy the moment error rates move: these are change controls, every one of them, expressed in a medium cheap enough that they run without a meeting. The instinct is the same one the change window serves. Establish ground truth, expose a small blast radius first, watch closely, and keep a fast path back. What differs is the weight of the medium, which is the entire subject of this essay.

The discipline is calibration. The question to ask of any change process is whether its speed matches the consequence of being wrong, rather than whether it is fast or slow in the abstract: what is the blast radius, what is the rollback story, who has access, and what does the worst credible failure cost. Stateless and reversible earns fast. Stateful and irreversible earns slow. Run everything through the heavy process and you waste years; run everything through the light one and you gamble data.

That calibration is testable, which is what keeps it from being a slogan. Take any step in your current change process and ask which of the four questions it answers. A step that answers none of them is ceremony and can go. A step that answers one badly, an approval from someone with no view of the blast radius, is worse than ceremony, because it manufactures confidence without adding information. What survives that audit is the process you actually needed, and the audit is worth running on the fast path too, where the risk is the opposite one: a gate that fires automatically and gets acknowledged automatically has stopped answering anything at all.

Infrastructure Change Management