The Ideal and the Actual

In October 2025, the most sophisticated infrastructure automation on the planet deleted its own DNS records and could not put them back.

The outage in Amazon's us-east-1 region started, by AWS's own account, with a latent race condition in the automated system that manages DNS for DynamoDB's regional endpoint. This was not neglected legacy tooling. It was the platonic ideal of 2026 operations: fully automated, redundant across availability zones, health checked, running the same playbook it had run successfully countless times. Two components of the automation stepped on each other in a sequence nobody had hit before, and the result was an empty DNS record for one of the most depended-upon endpoints on the internet. The automation could not repair the record it had emptied. Human engineers restored it by hand.

Then came the part that matters more. Fixing the record did not fix the region. The failure had cascaded into the internal systems that manage leases for EC2 capacity, and that state could not simply be redeclared into existence. It had to recover, slowly and grudgingly, through hours of congestive backlog. Full restoration took the better part of a day.

The incident demonstrates something narrower than automation being bad. Underneath the most weightless-looking layer of the modern stack there was state, state has weight, and weighted things do not spring back up because a pipeline says so. If that holds at AWS, it is worth asking how true it is everywhere else.

The story we tell ourselves

The modern infrastructure narrative is a genuinely good story, and it is worth telling honestly before poking at it.

It begins with pets. For decades, servers were named, hand-built, individually cared for, and irreplaceable. Every shop had its Frodo the mail server, lovingly patched, deeply understood by exactly one person, and catastrophic to lose. Virtualization loosened the hardware coupling but not the mindset. The VM was still a pet. It just lived in a nicer barn.

The real break came when engineers at web-scale companies inverted the premise. The phrase "cattle, not pets" traces to Bill Baker at Microsoft, describing scale-out versus scale-up, and was pushed into the cloud mainstream by Randy Bias around 2012: stop building servers you cannot afford to lose, and start building fleets where any individual member can die without ceremony. Design for failure. Let the herd route around the dead. Container platforms industrialized the idea, and the unit of concern stopped being the server at all. It became the workload.

From there the progression is familiar. DevOps merged the people who build with the people who run. Infrastructure as code turned environments into text files. GitOps made the repository the source of truth, so that the running world is continuously reconciled against a declared one. AIOps points at the horizon: telemetry-fed models that detect the anomaly, diagnose it, and remediate it before a human is paged. In parallel, the same philosophy transformed the layers around compute. Identity moved from on-premises directories and static group policy to cloud identity providers that re-evaluate every access attempt in context. The hard network perimeter dissolved into Zero Trust, where nothing is trusted for being inside. Endpoints stopped needing a corporate network at all, managed instead over the open internet. Monitoring became observability, streams of metrics, logs, and traces feeding platforms that baseline normal and flag drift.

That is the 2026 ideal: declarative, ephemeral, identity-centric, self-healing. Infrastructure as software, all the way down.

Who actually lives there

For a large and growing class of companies, the ideal stopped being aspiration and became Tuesday, an unremarkable one at that.

A company founded in the last decade, architected on a hyperscaler from day one, storing its state in managed databases and object storage, carrying no regulatory retention chain and no acquired legacy estate, can live the cattle model almost completely and correctly. Some of these organizations elastically run workloads at scales that sound fictional, spinning up enormous fleets of instances to absorb demand and destroying them without a second thought, triaging the failures to harden the design. For them, a server that cannot be safely destroyed genuinely is a design flaw. Nothing in this essay argues otherwise, and any version of this argument that sneers at that world should be discarded.

It is also worth noting what the cattle metaphor's own authors actually said, since the canonical framing itself stated explicitly that some assets are not disposable, that those assets require pet treatment, and that the goal is minimizing the pets rather than pretending they are extinct. It never claimed every system could be livestock.

This essay is about what lives inside that admitted exception, and the exception is no rounding error: it is most of the physical world.

Weight

Data has gravity. The phrase is usually credited to Dave McCrory, and it describes something every storage practitioner knows in their hands: past a certain mass, data stops being something you move and becomes something things move toward. A multi-petabyte store cannot be casually migrated, rebuilt, or discarded. There is nowhere for that mass to go. No pipeline conjures a second copy of ten petabytes into existence this afternoon, and no budget pays to keep one idling just in case.

But gravity does not stop at the data. Gravity implies mass, and mass runs all the way down the stack. The storage nodes have weight. The switches that fabric them together have weight. The power distribution units have weight. The uninterruptible power supplies have weight, and so does the generator behind them. The firewall appliance, the VPN concentrator to a distant data center, the KVM that is the path of last resort into everything else: each is a physical object with a procurement history, a firmware level, a maintenance contract, and a blast radius. Every one of them pulls on everything near it.

Weightless engineering rules do not apply to weighted systems; that single fact is the whole argument, and almost every confusion in the modernization debate comes from ignoring it.

The distinction gets subtle, because well-built weighted systems contain cattle-like behavior inside them. In a properly designed scale-out storage cluster, a node failure is a non-event. The system routes around the dead member, rebuilds protection from the surviving nodes, and re-streams data without a human touching anything. At the component level, that is the cattle model, executed as well as anyone executes it. But zoom out one level and the picture inverts. The cluster as a whole is a pet, arguably the most pet-like object in the building, because it holds irreplaceable state and there is no second herd. Component-cattle living inside a system-pet. Both descriptions are true at once, and knowing which level you are operating on is the difference between a routine Tuesday and a resume-generating event.

There is, in fact, a way to build a second herd, and it is the exception that proves the rule. Enterprise storage platforms can replicate a dataset to an entirely separate cluster, usually in another building or another region, so that a catastrophe at one site is survivable at the other. This is the real answer to data gravity, and every property of it is governed by weight. The replica is asynchronous, which means it is always some interval behind the source, and that interval is a number the business chooses in advance and then lives with: the recovery point, the quantity of recent work you accept losing if the primary vanishes between replication cycles.

Seeding the relationship in the first place is a campaign, not a command: pushing the initial baseline of a large dataset across a wide-area link is measured in days, sometimes weeks, throttled by bandwidth that costs real money and by the plain fact that hundreds of terabytes do not cross a link faster than the link allows. Failover itself, the act of promoting the replica and running the business from it, is a rehearsed procedure with its own runbook and its own verification steps, and it carries a return trip that is usually harder than the trip out, because failing back to the repaired primary means reconciling everything that changed while you were running on the copy.

None of this resembles a revert. All of it is the weighted world's version of redundancy: real, essential, and slow, priced in physics rather than in commands. Backup obeys the same law one layer over, where the honest question is never whether a copy exists but how long a restore of real size actually takes, since a backup you cannot restore inside the window the business can tolerate is a receipt, not a recovery.

What weight does to the rules

Once you take weight seriously, practices the modern narrative writes off as cultural lag start looking like what they actually are: correct engineering responses to physics and consequence.

First, the dependency chain. A cluster's internal redundancy is irrelevant if the PDU feeding it fails, or the UPS does not carry the load, or the pair of switches uplinking it lose both paths at once. In a cloud region, that whole substrate is abstracted into someone else's problem, and the abstraction is precisely what lets the layer above feel weightless. On-premises, it is your problem, and it is why infrastructure engineers obsess over things the application world has never heard of. The weight goes all the way down, and so does the responsibility.

Second, rollback. In the weightless world, rollback is a revert: the repository holds the previous truth, and reconciliation restores it. In the weighted world, rollback is a procedure, when it exists at all. A firmware update to a storage controller, a schema migration on a loaded database, a change to a replication relationship: none of these reverse with a command. Some do not reverse, period. The October outage made the point at hyperscale. Even after humans fixed the root cause, the dependent state could not be redeclared; it had to be nursed back through hours of recovery instead. When your rollback story is a multi-hour manual procedure or does not exist, slow and deliberate change is the only rational posture. The change window, the pre-checks, the post-verification, the meeting that spends an hour on one paragraph of a change plan: these are calibrated to blast radius, and the calibration is correct.

Third, the bridge call never dies. Every enterprise knows the ritual. Something is failing, or a deployment has broken an interaction nobody modeled, and suddenly a call exists with a storage engineer, a network engineer, a virtualization admin, an application owner, and a vendor on it. The modern narrative implies this is a symptom of immaturity that automation retires. The evidence says otherwise. The most automated infrastructure organization on earth spent an October night doing exactly this, at maximum scale, and its postmortem is the artifact of that call. Weighted systems interact in ways no model fully captures, and when they do, human experts converge on a live incident. It is a permanent feature of operating consequential systems, not a transitional practice.

Fourth, and less discussed, weight is organizational as well as physical. The ideal quietly assumes a unified platform team with API-level access to everything it manages. One pipeline, one source of truth, and whoever owns the code can redeploy the thing. Large enterprises do not work that way. Ownership is fragmented across storage, network, UNIX, virtualization, and security teams, each holding access to its own layer and locked out of the others, with vendor-managed appliances sitting outside everyone's reach. In that world, "just reboot it" is a ticket, routed across team boundaries, that can die in the queue because the receiving team does not have access to the resource. Access boundaries are gravity too. Any modernization plan that ignores them is describing a company that does not exist.

And underneath all of it, a quieter fact: the practitioners who run the weighted layer largely do not touch the weightless toolchain at all. Even inside development, the population most exposed to infrastructure as code, container tooling has gone nearly universal while IaC tooling has stayed a minority practice. Step outside development into the storage and network teams who operate the physical estate, and that exposure thins further. The pattern holds across the industry: most of networking keeps following the old-world model of reliability through box, power, and path redundancy, because the newer tooling solves a problem this layer does not have rather than because it is any kind of secret. The day-to-day texture of the weighted layer in 2026 is still privileged sessions, configuration review, flat-file logs, and packet captures.

Both worlds are true at once

So which picture of 2026 is real? The elastic fleet spinning up a hundred thousand instances before lunch, or the change window at two in the morning with an engineer reading logs by hand?

Both, simultaneously, and often inside the same company.

This is the part the discourse keeps getting wrong, because both camps want a verdict and the truth is a map. The weightless world is real, enormous, and growing, and its methods are correct for its physics. The weighted world is also real, also enormous, holds most of the data that matters, and its methods are correct for its physics. Treating the difference as a maturity gap, as if the storage team were simply a few sprints behind the platform team, misreads the situation completely. They are on different terrain, not different points on one road.

The failure modes are symmetric. Apply weightless rules to a weighted system and you get casual changes to things with irreversible failure modes, which is how organizations lose data. Apply weighted caution to a genuinely weightless system and you get change advisory boards for stateless microservices, which is how organizations lose years.

Neither doctrine, mastered in isolation, is the skill that matters in 2026. What matters is the judgment to tell, on sight, which world a given system lives in, and the discipline to apply the rules that belong to it.

The backbone and the seam

One more thing the October incident showed, almost in passing. When the automation failed, the engineers who fixed it dropped down to first principles: what is DNS actually returning, what state is this subsystem actually in, what does the traffic actually look like. The diagnostic backbone, DNS resolution, SNMP, packet capture, syslog, reading the log that was actually written rather than the dashboard that summarizes it, stayed load-bearing through the failure. It is the floor everyone lands on when the abstractions give out, stable because it is correct rather than because it is old.

What is genuinely transforming is the layer above it: identity, orchestration, validation, the management plane. That layer rewards learning, and the honest posture for a weighted-world practitioner is to build there directly, bringing version control, drift detection, and scripted verification to systems that will never be cattle, without pretending the substrate stopped mattering. Modernizing What You Cannot Redeploy works through that translation practice by practice, including where each translated practice quietly fails.

Both the ideal and the actual are real, and they describe different terrain rather than two stages of one maturity curve. The mistake still worth avoiding is applying one map to the other's ground.