Anatomy of a Failover

Every storage vendor says failover is seamless, and every storage engineer has watched a room full of people stare at a frozen application during one. Both are telling a version of the truth. "Seamless" is really a claim about four separate layers, each of which recovers on its own schedule, using its own mechanism, with its own definition of done. When a failover feels bad, it is almost never because failover failed. It is because one layer's clock ran long, and nobody in the room knew which clock they were watching. So let's walk down the stack and watch one node die properly.

Layer one: the address has to move

In most scale-out and clustered NAS designs, clients connect to a virtual IP, and failover begins with that address coming up on a surviving node. The new owner broadcasts a gratuitous ARP so switches and neighbors update their MAC tables immediately instead of waiting for cache entries to age out. This part is genuinely fast, typically well under a second, and it is the part vendors are picturing when they say seamless.

If gratuitous ARP is filtered or ignored somewhere in the path, though, clients keep sending traffic to a MAC address that no longer answers, and everything above this layer stalls for reasons no storage-side log will ever show you. Failover problems that vary by client subnet usually live here.

The filtering is rarely malicious and usually a security control doing its job. Dynamic ARP inspection drops gratuitous ARP that does not match a DHCP snooping binding, which a statically addressed cluster interface will not have. Port security and various flavors of ARP spoofing protection do the same thing for the same reason: a gratuitous ARP announcing that an address has moved to a new MAC is, structurally, indistinguishable from an ARP spoofing attack. The control cannot tell your failover from an attacker, which means the fix is an explicit exception rather than a debugging session, and it means the network team owns half of this failure even though every symptom appears on storage.

Layer two: TCP finds out the hard way

Here is the detail that explains most of the perceived pause: the surviving node does not have the failed node's TCP connections. Those sockets died with the node. The client's TCP stack, however, does not know that. From its perspective the peer just went quiet, so it does what TCP does: retransmits with exponential backoff, patiently, sometimes for minutes, waiting for a host that will never answer.

How many minutes is worth knowing precisely, because the default is longer than most people assume. A Linux client governs this with tcp_retries2, which defaults to 15 retransmission attempts, and because the intervals back off exponentially, exhausting that budget takes on the order of fifteen minutes rather than the seconds people picture. Nothing is broken during that interval. The client is behaving exactly as specified, waiting for a peer that is never going to answer, and no amount of tuning on the storage side shortens it.

What breaks the wait is a reset. If the client sends a segment to the takeover address and the new owner has no matching connection, the new owner responds RST, the client's connection dies immediately, and recovery can start. Clustered SMB implementations built on CTDB make this explicit with a trick called a tickle ACK, where the takeover node deliberately provokes the client into sending traffic so it can reset the stale connection right away instead of letting the client back off in silence.

The general rule: a fast, honest RST is a gift. Silence is the expensive failure mode. If your failover tests show a long dead period before any recovery begins, look at how quickly stale connections get reset before you look at anything else. This is also why a firewall between clients and the cluster deserves scrutiny during failover testing, since a stateful device that silently drops the segment carrying the RST converts the fast path back into the fifteen-minute one.

Layer three: the protocol reassembles its state

The connection is dead and the client has noticed. Now it reconnects and has a conversation whose difficulty depends entirely on how much state the protocol was carrying.

NFSv3 travels light. The protocol itself is stateless, so a reconnecting client mostly just resumes issuing operations, which is why v3 failovers can feel almost invisible. The exception is locking, which v3 outsources to the NLM and statd sidecar services, and that machinery is the roughest part of the v3 experience: the new server enters a grace window for lock reclaim, and stale lock state after ungraceful failovers is a classic source of the mount-works-but-the-application-hangs ticket.

NFSv4 brought state into the protocol and, with it, a formal recovery process. Clients hold leases, 90 seconds by default on a Linux server, and after a server restart or failover the new instance runs a grace period, which defaults to the lease time. During grace, the server's job is to let existing clients reclaim the opens and locks they held while refusing to hand out anything new, and it enforces that by answering new open and lock requests with NFS4ERR_GRACE. This is the pause people misdiagnose most: during those 90 seconds the system is deliberately protecting your locks from being stolen mid-failover rather than failing, which is the difference between an annoying pause and corrupted data.

The relationship between those two timers is the part worth internalizing, because it is where well-intentioned tuning does damage. Grace defaults to the lease time for a reason: a client whose lease has not yet expired still believes it holds its locks, so a grace period shorter than the lease can end while that client is still entitled to reclaim, at which point the server may hand its lock to somebody else. Shortening grace to make failovers look better is therefore a correctness change disguised as a performance change. NFSv4.1 improved the exit ramp honestly, with RECLAIM_COMPLETE, letting each client declare it has finished reclaiming so a server that has heard from everyone can end grace early instead of running out the clock. That is the safe way to get the same result: end grace when reclaim is provably done, rather than when a shorter timer says so.

SMB carries the most state of all: sessions, tree connects, open handles, leases. Plain SMB has the bluntest failover story, where the TCP connection dies and every open handle dies with it, and whether the user notices depends on whether the application handles a reopen gracefully. Office applications mostly do. Databases and hypervisors emphatically do not, which is exactly why SMB3 grew a coordinated set of resiliency features.

Those features come in three parts, and they solve three different problems. Durable handles let a client reattach to an open file after a disconnect, within a bounded window. Persistent handles, granted on shares marked continuously available, go further: the cluster preserves the handle state itself across the node failure, and while the client's reconnect window is open, commonly on the order of 60 seconds, competing opens are blocked so the returning client can reclaim what it held. The Witness protocol fixes the discovery problem, letting a client register with another cluster node that will notify it the moment its resource moves, so the client migrates on a push notification instead of discovering the failure by TCP timeout.

Witness is the one that changes the arithmetic rather than merely softening it. Every other mechanism on this list makes recovery cheaper once the client has noticed the failure. Witness attacks the noticing, which is the expensive part, and it is why an SMB3 deployment with witness can produce failover behavior that genuinely looks like the datasheet while the same cluster without it does not.

One distinction worth keeping sharp, because marketing material blurs it constantly: durable handles are a file-open resiliency feature, and continuous availability is a cluster-level guarantee built on top of persistent handles plus witness. Having the first does not mean you have the second.

Layer four: the clocks stack, they do not merge

Now assemble the whole picture from the client's chair. The pause an application experiences is roughly: time for the address to move, plus time for the client to learn its connection is dead, plus time for protocol state recovery, plus however the application's own timeouts interact with all of the above. Those are different timers owned by different subsystems, and this is the part I most wish more people understood: tuning one does not shorten another. You can drop the NFS grace period to 15 seconds and still eat a two-minute failover because stale TCP connections are timing out in silence. You can deploy witness and persistent handles perfectly and still take an outage because the application above the redirector gives up after 30 seconds and the reconnect window needed 45.

That stacking is also why a maintenance window for a controller upgrade is budgeted the way it is, and why an estimate that was optimistic by ninety minutes is how a recoverable change becomes an outage. The window has to hold the sum of these clocks, not the largest one, and then leave room for the back-out on top.

NFS mount options deserve a specific mention because they sit right in the middle of this stack. A hard mount retries forever, which sounds scary and is correct for anything that matters, because the alternative is soft mounts returning I/O errors to applications mid-failover, and an application that received EIO at the wrong moment is how a clean failover turns into a data integrity investigation. Hard, with reasonable timeo and retrans values, and interruptible so humans can escape, remains the boring right answer for data you care about. That choice is a small, concrete instance of the general principle that the failure path deserves designing before the happy path does, since the entire argument for hard mounts is about what happens when things are already going wrong.

What to actually do with this

Three things. First, test failover with live I/O and a stopwatch, because the only number that matters is the pause measured from the client, and it will not match the number in the datasheet. A scripted data-path exerciser that times each phase separately and records every run to a history file is worth considerably more here than a person watching a progress indicator, because the phase that got slower between two firmware levels is invisible without a baseline to compare against.

Second, know your clocks: the grace period, the handle reconnect window, the TCP behavior on address takeover, and your application's own timeout, written down side by side. The failover budget is the longest of them, and every application timeout must exceed it with margin.

Third, when a failover goes badly, diagnose by layer in order, address, transport, protocol, application, because each one fails with a distinct signature, and the fix for one layer applied to another is how these incidents stretch into their third hour. The signatures are specific enough to be useful: a failure that varies by client subnet is layer one, a long dead interval before any recovery activity is layer two, an error that names grace or reclaim is layer three, and a recovery that completed cleanly on the wire while the application stayed unhappy is layer four.

Sources and further reading

RFC 8881 (NFSv4.1, state recovery and grace, chapters 8 and 9), RFC 7530 (NFSv4.0), the linux-nfs.org lock recovery notes, nfsd(8) for lease-time and grace-time tunables, tcp(7) for tcp_retries2, and Microsoft's MS-SMB2 and MS-SWN protocol documents for durable/persistent handles and Witness.