Incident Response November 14, 2026 18 min read

How a Review Earns Its Causal Conclusion

The draft summary of the review reads, in full: "Cause: certificate renewal on 2026-11-02." Nobody in the room disputes it: the renewal happened, the failures followed it, and taking the renewed servers out of service stopped them. That line also points at the one part of the incident that recurs on a schedule, and the job that ran the renewal still had five servers left to do.

A one-line cause belongs in a postmortem summary. It earns the place when the review can show the mechanism, has tested the explanations that compete with it, and has written down the evidence that would reopen it. Say Only What the Evidence Supports in the Postmortem makes the case for stating a cause plainly once the evidence is in. This piece covers the work that comes before that plain sentence. The incident and every example below are constructed.

The incident

An internal order-status API runs on eight servers behind a load balancer that spreads requests evenly. Its certificate renews every 60 days through a job that updates one batch of servers per nightly window. At 2026-11-02 21:40 UTC, batch one replaced the certificate on servers 1 to 3. Batch two, servers 4 to 8, was scheduled for the same time the next night.

In the warehouse, 140 handheld scanners call the API. Their app, built on OpenSSL, carries its own trust store, which holds the organization's root certificate and nothing else. Office browsers, three internal services, and a synthetic probe also call the API. Those services and the probe share a trust bundle that holds the root and the issuing intermediate.

All times from here are UTC on 2026-11-03 unless another date is given. When the morning shift started at 06:00, the scanners went back to work, and from 06:02 their app logged failed calls. Support logged the first contact at 06:15: "scanners flaky, retry usually works." Throughout, the probe stayed green. On-call was engaged at 06:40. At 06:58 the load balancer log showed that every failed scanner request had gone to server 1, 2, or 3. At 07:05 the incident lead decided to drain those three servers, the drain took effect at 07:20, and scanner errors stopped at 07:22. By 08:30 the three servers were back on the previous certificate and configuration and back in the pool. Batch two was paused.

At close, the ticket carried a labeled hypothesis, the way Writing the Incident Ticket Someone Else Will Read recommends: failures correlated with the renewal, mechanism unknown. Nine days later, on 2026-11-12, the review met.

Nine findings a cause line absorbs

"Cause: certificate renewal" compresses several findings into one phrase, each needing its own evidence:

Finding	The question it answers	In this incident
Observed condition	What was measured, and where	Scanner calls failing from 06:02; all failures routed to servers 1 to 3
Trigger	What started this occurrence	Renewal batch one, 2026-11-02 21:40
Proximate mechanism	How the condition produced the failure	Established by the review, below
Contributing condition	What made the failure possible or larger	The scanner trust store holds the root alone
Enabling condition	Which design or process let it reach production	A configuration template change on 2026-10-28
Detection gap	Why it took this long to notice	Scanner failure logs fed no alert and the probe passed; the first signal anyone saw was a support contact at 06:15
Recovery gap	Why restoring service took as long as it did	The drain that stopped the errors waited 11 minutes for a second approver; the drained servers returned to the pool at 08:30, after the errors had stopped
Systemic pattern	Whether the same shape recurs across incidents	Checked against earlier reviews; one incident establishes no pattern
Residual uncertainty	What the review could still not explain	Why office browsers worked against the same servers

That draft summary named the trigger and stopped. A trigger is a useful finding, since it bounds the search and says what to pause, but most action items come from the other eight rows.

A contributing condition can be a sound design choice. A trust store that holds only the root is reasonable for a fleet of devices, and it had caused no failure before 21:40. Recording it as a condition the change ran into keeps the finding about the system and keeps the action items on the configuration that changed.

Trigger, first symptom, and mechanism

Three timestamps matter, and they sit far apart. On 2026-10-28 the configuration template changed. Batch one applied it at 2026-11-02 21:40. Scanners first failed at 06:02 the next morning, 8 hours and 22 minutes after the trigger, because the scanners were idle overnight.

A review that dates the incident from its first symptom can search the wrong night. Here the renewal was easy to find; elsewhere the gap runs to days.

A mechanism is a separate finding from either timestamp. A mechanism statement says what the change altered and how that alteration produced the observed failure, in enough detail that someone could reproduce it. "The renewal broke the scanners" names an event and an outcome; at the start of the review, several candidate mechanisms could fill the space between them.

List the explanations before testing any

Review work began with a list: every explanation the team had heard during the incident, along with what each one predicts. Predictions came first, before the review opened a log, because evidence read after a favorite explanation tends to get read in its favor. Every explanation tied to the renewal predicts failures that start after batch one and occur only against servers 1 to 3, the only servers it touched, so draining those servers should stop them.

Explanation	What it predicts
A. Servers 1 to 3 send an incomplete chain	Verify error 20 on clients that lack the intermediate; clients holding it succeed; failures only on servers 1 to 3
B. The new certificate on servers 1 to 3 chains to a root the scanners lack	Verify error 20 on the scanners and on the probe; a complete chain served; the new certificate fails to verify against the scanners' root
C. Scanner clocks run behind the new certificate's start time	A "not yet valid" error, only against servers 1 to 3, only from scanners whose clocks are wrong
D. Warehouse wireless drops connections	Timeouts and resets; failures that follow access points rather than servers; retries succeed
E. Servers 1 to 3 changed protocol or cipher settings the scanners cannot negotiate	A handshake failure before any certificate check, only on servers 1 to 3

Each row came from someone on the call; written with predictions, the list becomes a set of checks.

Weigh evidence by what it rules out

In the table of evidence, "expected" means the explanation's prediction includes the observation, "silent" means the prediction says nothing about it either way, and "contradicts" means the prediction says something else.

Evidence	A	B	C	D	E
1. Failures began after batch one	Expected	Expected	Expected	Silent	Expected
2. About 3 in 8 scanner calls failed, matching 3 of 8 servers	Expected	Expected	Silent	Silent	Expected
3. Scanner log: verify error 20, `unable to get local issuer certificate`	Expected	Expected	Contradicts	Contradicts	Contradicts
4. Servers 1 to 3 send one certificate; servers 4 to 8 send two	Expected	Contradicts	Silent	Silent	Silent
5. New server certificate verifies against the scanners' root through the existing intermediate	Expected	Contradicts	Silent	Silent	Silent
6. The probe, holding the intermediate, passed against every server	Expected	Contradicts	Silent	Silent	Silent
7. Draining servers 1 to 3 stopped the failures within two minutes	Expected	Expected	Expected	Contradicts	Expected

What convinced the room during the incident sits in rows 1, 2, and 7. Failures started after the renewal, they hit about 3 in 8 calls, and they stopped when the renewed servers left the pool. A, B, and E predict that every scanner call to servers 1 to 3 fails, so with even balancing they expect all three; C expects rows 1 and 7. Together these rows show where failures happened and say little about why. Persuasive evidence and discriminating evidence are often different rows. Row 7 did one more job: draining backend servers leaves a wireless fault untouched, so the drain ruled out D.

Row 3 rules out the most explanations for its cost: three of the five. A client that rejects a certificate for a clock problem reports certificate is not yet valid (OpenSSL error 9) or certificate has expired (error 10). Error 20 means the client could not find the certificate that issued the one it was handed. That string contradicts C, D, and E as explanations for these failures: a connection dropped before the certificate arrives never reaches certificate checking, and neither does a failed protocol or cipher negotiation. OpenSSL may stop at the first verification error, so a clock fault behind the chain failure can go unreported.

On its own, error 20 leaves A and B standing, because both predict it. Separating them takes what on-call saved at 07:10, before the drain: each server's certificate count and server 1's leaf. On-call's two commands, then the review's verification:

for host in api-1 api-2 api-3 api-4 api-5 api-6 api-7 api-8; do printf '%s: ' "${host}"; echo | openssl s_client -connect "${host}.orders.internal.example:443" -servername orders.internal.example -showcerts 2>/dev/null | grep -c 'BEGIN CERTIFICATE'; done;
echo | openssl s_client -connect api-1.orders.internal.example:443 -servername orders.internal.example 2>/dev/null | openssl x509 -out served_leaf.pem;
openssl verify -CAfile scanner_root.pem -no-CApath -no-CAstore -untrusted intermediate.pem served_leaf.pem;

For servers 1 to 3 the loop printed 1; for servers 4 to 8 it printed 2. A 0 would have meant that no certificate came back; the hidden errors would not say why. With the scanner root as the only trusted certificate, verification printed served_leaf.pem: OK: given the existing intermediate, the certificate the server sent builds a path to the scanners' own root. Verification checks signatures, which matching issuer names cannot. B expects a complete chain, a path to some other root, and a failing probe; rows 4, 5, and 6 show the opposite of each. Of the five explanations tested, A is the only one with no contradiction left.

A contradiction rules an explanation out only as far as its prediction was precise and the evidence direct. Row 3 contradicts C through the error type, which is strong but indirect evidence about clocks; the drain contradicts D by direct intervention. The rejected-explanations table below records which was which, so a later reader can weigh them differently.

Reproduce before accepting

An explanation that survives every check still needs its mechanism tested: someone produces the failure on purpose. So the review built a small lab: a throwaway root and intermediate, a server certificate (the leaf) issued by that intermediate, and the web server configuration rendered from the 2026-10-28 template. That configuration pointed the certificate setting at the leaf-only file instead of the full-chain file.

The lab client matches the trust profile of the clients that failed, not of the ones that stayed healthy. A client that already held the root and the intermediate would have passed: useful as a control, as the results below show, but unable to reproduce the failure. Against that server, a client trusting only the lab root, the same shape as the scanner trust store, failed the way the scanners had. By default openssl s_client keeps going after a verify failure and reports the last error it hit; with -verify_return_error it stops at the first failure, the way an application using default verification does:

Certificate chain
 0 s:CN = localhost
   i:CN = Lab Issuing CA
Verify return code: 20 (unable to get local issuer certificate)

With the certificate setting pointed back at the full chain, the same client reported two certificates and verify return code 0. A client whose bundle held the intermediate succeeded against the leaf-only server too, with return code 0. That third result explains row 6: the probe could not see this failure while it carried the certificate the servers had stopped sending.

The lab showed the mechanism produces exactly this error on a client with the scanners' trust profile. Row 4's chain counts and the rendered configuration tie it to the production servers. Reproduction alone proves the mechanism can cause the failure; with every other tested explanation contradicted, the review accepted that it did.

The conclusion record

The review recorded its conclusion, each line with what it rests on:

Conclusion accepted among five tested explanations, 2026-11-12.
Reviewers: API service owner, platform on-call lead, scanner app
owner. Evidence: INC-7342 checksum manifest (scanner logs and trust
bundle, 07:10 certificate counts and server 1's leaf, the existing
intermediate used to verify it, load balancer
logs, renewal job logs, template, rendered configuration, pipeline
records, clock readings, probe results, lab commands, output, and
OpenSSL versions); template change CHG-4410.

Scope:      order-status API servers 1 to 3, 2026-11-02 21:40 UTC to
            2026-11-03 07:22 UTC; warehouse scanner app.
Mechanism:  counts saved at 07:10 show servers 1 to 3 sending one
            certificate each. Server 1's saved certificate is the leaf;
            the configuration rendered from the CHG-4410 template, applied
            to all three, serves the leaf alone. The scanner app trusts
            only the root, so it could not build a path to it and
            rejected the connection (OpenSSL verify error 20).
            Reproduced in a lab with the rendered configuration and a
            client trusting only a lab root; the scanner app stayed
            outside the lab.
Trigger:    renewal batch one, 2026-11-02 21:40 UTC, which applied the
            CHG-4410 template to servers 1 to 3.
Enabling:   the CHG-4410 template points the certificate setting at the
            leaf-only file. No deployment check verified the served
            chain against the scanner trust profile (pipeline records,
            reviewed 2026-11-10).
Rejected:   other root (new certificate verifies to the scanner root
            through the existing intermediate; incomplete chain served;
            probe passed); scanner clocks (error 20 where a clock fault gives
            9 or 10; a concurrent clock fault stays possible, and a
            2026-11-12 sample of 12 scanners, 4 that failed, read
            within 2 s but cannot show incident-time clocks);
            wireless (failures stopped when backend servers were
            drained); protocol settings (handshakes reached
            certificate verification).
Reopens if: two scanners, or a test client holding the scanner bundle,
            report error 20 against this API while the chain captured
            from each backend that served those calls verifies
            against the scanner bundle. Reassess from the capture.
Open:       why office browsers succeeded against servers 1 to 3; not
            investigated. The scanner mechanism holds regardless of why
            the browsers succeeded.

Now the summary line can say it plainly: an incomplete certificate chain on three renewed servers stopped the warehouse scanners, which trust only the root, from verifying the API. It says where the fault lived and why the scanners failed while the probe stayed green.

Its "Reopens if" line names, in advance, the observation that would put the conclusion back in question and what to capture when it appears. The capture has to come from each backend that handled the failing calls; a healthy server's chain says nothing about the one that failed.

What the record keeps about rejected explanations

Each rejected explanation keeps its evidence, how direct that evidence was, and what would bring it back.

Explanation	Ruled out by	How direct	Comes back if
B. Other root	Verifies to the scanners' root; incomplete chain served; probe passed	Direct, by verification	A captured production chain verifies only to a root the scanners lack
C. Scanner clocks	Error 20 instead of 9 or 10; 12 clocks within 2 s at the review	Error type is indirect; the sample shows review-time clocks, nine days late	A scanner logs error 9 or 10 against a complete chain
D. Wireless	Failures stopped when backend servers were drained	Direct, by intervention	Failures appear as timeouts or follow an access point
E. Protocol settings	Handshakes reached verification	Inference from the error type	A handshake failure alert appears in the scanner log

The "How direct" column records the team's judgment, so a later reader can disagree; E, for instance, fell to a sound inference.

No synchronization logs were checked, so the rejection of clock skew rests on the error type, not on the late sample.

Detection and recovery gaps pay off on the next incident

Mechanism findings guard against one failure. Detection and recovery findings keep paying off when the next incident has nothing to do with certificates.

The probe passed throughout because its trust bundle held more than the scanners' did. Three action items follow: a second probe that uses the scanner trust store; a deployment check that connects to each server, captures the chain it actually serves, and verifies that chain against every client trust profile the service supports, starting with the scanners' root-only bundle; and an alert on the scanner app's own failure logs, which recorded the problem at 06:02, thirteen minutes before anyone called. The review's verify command supplies the intermediate itself, so it would pass a server that omits it; the deployment check takes the intermediate only from what the server sends.

Draining was decided at 07:05 and took effect at 07:20. Eleven of those fifteen minutes went to waiting for a second approver, because the load balancer runbook had no pre-approved drain for a certificate fault. The fix is a runbook entry for the drain with approval granted in advance and a verification step.

Batch two had the nearest deadline: the review confirmed the 2026-10-28 template would have broken servers 4 to 8 the same way. A corrected template passed the new chain check before batch two ran on 2026-11-13. Nothing in the draft summary pointed at the template.

When the evidence stops short

Some reviews cannot show the mechanism: the failure will not reproduce, or the logs that would split the last two explanations rotated out. In that case the record carries a working explanation instead of a conclusion: the explanation the review currently prefers, the alternatives still standing, the check that would separate them, who owns that check, and the date it is due. When two explanations survive every check the review can run, the record keeps both as open alternatives, or records them together when the evidence shows both contributed. Either way, among comparably urgent and safe fixes, those covering both explanations go first; with open alternatives, the check that separates them still runs.

The summary line changes to match. "Working explanation: incomplete chain on renewed servers; other-root explanation still open, check owned by the platform team, due 2026-11-20" gives a later reader the state of the review in one line. When the check comes back, a dated addition records the result. The working explanation becomes a conclusion only if the evidence shows its mechanism and rules out every other explanation still standing, the same standard the incident above had to meet; a pair recorded together needs each member's mechanism shown. If the result contradicts an explanation, the addition withdraws only that explanation, leaving any partner in a pair open, and names the check that did; the alternatives still standing go back on the list, and none becomes the conclusion until its own mechanism is shown and the rest are ruled out. If none are left, the record says the cause is open, reopens the list of explanations, and names the next check, its owner, and its date. If the result rules out some alternatives but leaves others, the record drops those it ruled out and names the next check, its owner, and its date. If it separates nothing, the working explanation stays, with the next check, its owner, and its date.

What the method leaves open

This method carries at least five limits, and a review that uses it should know them.

It has no test for when the list of explanations is complete. A review can accept the only uncontradicted explanation and still miss one nobody raised. That is why the record says "among five tested explanations" and carries a reopen line. Before closing, a review can ask what else would produce the same evidence, and add anything new to the list with its predictions; that lowers the risk without removing it.

It has no formal method for weighing evidence that shares one origin, only a caution. Two rows read from the same log, or two logs written by the same faulty component, carry less weight than two rows from unrelated sources, and separate systems can still share an upstream dependency, such as one host's clock. Rows 1, 2, 3, and 7 here all lean on the scanner failure log: row 1 for when failures started, row 3 for the error, row 7 for when they stopped, and row 2 for the failure count it sets against the load balancer's request counts. A defect in that log could weaken several of those rows at once, so together they count for less than four independent observations. Row 4 comes from the certificate counts on-call saved at 07:10, and row 5 from the review's verification of server 1's saved leaf. That separate path is part of why rows 4 and 5 carried the decision between A and B.

It gives no way to weigh two explanations that both contributed. The record can name both and say each was needed, but nothing here measures how much of the failure each one produced.

It has no scoring method for partial matches or evidence of uneven quality. Real evidence rarely sorts as cleanly as the table above. The one rule here, that a contradiction counts as far as the prediction was precise and the evidence direct, is a judgment, and the "How direct" column is where the record writes that judgment down.

It assumes the review can reach the evidence. When the mechanism sits in a vendor's system or another team's that will not share logs, the list can stall at two explanations, and the working explanation, with the check that would separate them, its owner, and its date, is the honest record.

References

Sources verified 2026-11-09 UTC.