Certificate Expiry Watcher

The problem

On 2021-04-06 at 12:00 UTC, a wildcard certificate for one of Epic Games' internal DNS zones expired. Epic's own write-up says it was installed across hundreds of production services. Logins failed across several of its games and its store, and although the certificate was reissued and began rolling out 37 minutes later, full recovery took until 17:35 UTC. The detail that matters most here: that internal zone sat outside the company's certificate monitoring, so the monitoring had no chance to warn anyone.

That is the case this tool is for. A certificate carries its expiry date inside it, readable by anyone who connects, months before it matters. Public-facing certificates increasingly renew themselves through ACME, and where that fits it is the right answer. The long tail is everything else: services on a private CA, management interfaces, an internal wiki on port 8443, the mail relay's implicit-TLS port, and wildcard certificates copied onto more services than anyone has written down. For those, expiry stays a calendar problem, and someone has to be assigned to read the calendar.

What it cannot see

Two other well-documented certificate failures mark the edge of this tool's reach, and they belong at the top of the page.

On 2018-12-06, Ericsson reported that an expired certificate in two software versions of its SGSN-MME, a mobile core network product, had disrupted service for operators in multiple countries. That certificate lived inside the product's software. A probe that connects to TLS endpoints never meets it.

The U.S. House Committee on Oversight and Government Reform's 2018 report on the Equifax breach found that the device inspecting encrypted traffic on the breached network had an expired certificate from 2016-01-31 until 2017-07-29. The committee calls that nineteen months; the two dates are 545 days apart, just under eighteen. Either way, the device inspected nothing in that time, and once the certificate was renewed, the team detected suspicious traffic. The same report counts more than 300 expired certificates across the company. A certificate copied onto an inspection appliance expires on its own schedule, whatever the service it was copied from presents.

So the scope is stated plainly: this watcher checks the certificate a running service presents over TLS. Certificates in files, keystores, application bundles, and appliances that never serve them need an inventory, which is a different tool.

Design decisions

Verification on, and each failure gets a name. Each endpoint is checked with full verification against the system trust store, or against a PEM bundle given with --ca-bundle, which a private CA outside the system store needs. The bundle replaces the system store for that run, so a host list that mixes public and private endpoints needs both sets of roots in it, or two runs. When verification fails, the OpenSSL error code decides the status: EXPIRED, NOT_YET_VALID, HOSTNAME_MISMATCH, or UNTRUSTED for everything else. The first version matched words in Python's error text instead, which would break without warning the day the wording changed. OpenSSL defines the code as a number, which stays fixed across releases.

The bundle decides what it can see. Give --ca-bundle the trust anchors your most limited clients actually hold, which is usually a set of roots, and leave out any intermediate a server fails to send. A server that sends its leaf without the intermediate that issued it breaks every client with no other copy of that intermediate, but a bundle holding it completes the chain on the watcher's side, and the watcher reports OK through the whole failure. How a Review Earns Its Causal Conclusion walks through a constructed incident with exactly that shape, where the synthetic probe stayed green while clients failed. The test suite runs one such server twice: root only gives UNTRUSTED with OpenSSL's "unable to get local issuer certificate", and root plus intermediate gives OK. If the clients being modelled really do hold that intermediate, it belongs in the bundle, and the choice belongs in writing.

A failed verification still reports a date, when it can. Python returns no certificate details from a connection that failed verification, so the first version printed EXPIRED with no date, which is the one fact the person reading the finding wants first. The current version opens a second connection without verification, to the same socket address the first one reached (interface scope included, for a link-local IPv6 peer), only to read the certificate presented there, and pulls the two validity dates out of its DER encoding using the layout in RFC 5280, section 4.1. Those dates are reported as presented and never used to decide trust. They can still belong to a different certificate: behind a load balancer the same address can reach another backend, and the expired certificate can be an intermediate while the leaf has not expired. So when the leaf's dates leave the status unexplained, the row prints them and says the failing certificate is elsewhere in the chain or behind another backend, and when the second connection fails, it says the dates were unreadable and why.

Unreachable is a finding too. A host that cannot be probed is a certificate nobody can vouch for, and a report that leaves it out turns "unknown" into implied fine, the same lie the preflight checker refuses to tell about clocks it cannot verify. Refused connections, timeouts, and failed name lookups are UNREACHABLE. A port that answers in plain text, which is what a STARTTLS service does, is TLS_FAILED. Both count toward exit 1.

One row per endpoint, one exit code. A completed run prints a row for every endpoint with its status, days remaining, and expiry date in UTC, then a summary line with the nearest expiry. A report that only appears when something is wrong gives no evidence the watcher ran at all, which is why the nightly wrapper can also log each run it sees end, clean ones included; the log helps only if something checks that it keeps growing. Exit 0 means every endpoint verified outside the warning window (30 days by default); 1 means findings; 2 means the run itself cannot be trusted, such as a malformed host list or an unreadable CA bundle, and in that case nothing is checked and no row is printed.

The host list is checked before anything connects. A port outside 1 to 65535, an IPv6 address without brackets, a numeric name other than four plain decimal numbers, a name with no DNS form or longer than the 253 characters DNS allows, two endpoints on one line, or an endpoint listed twice stops the run with the line number. The numeric rule is there because the resolver reads 010.0.0.1 as octal and connects to 8.0.0.1, and it judges each name in the DNS form the resolver receives, since fullwidth digits such as ０１０.０.０.１ become that same octal address on the way. That rule is written into the script itself, since Python's address parser read 010.0.0.1 as 10.0.0.1 up to Python 3.9.4. A control character anywhere on a line, a Unicode line separator included, stops the run too, since it would otherwise stop the run halfway or print a broken row. In 2001:db8::1:443, nobody can tell whether 443 is a port or the end of the address, so the script refuses to guess.

The core of the script

How one endpoint is checked, exactly as it appears in the repository. Alongside it in the same file sit the DER reader, the host list parser, the report, and certificate_name, which puts the name in its DNS form and drops a final root dot before verification: example.net. and example.net are one name to DNS, IDNA turns the fullwidth and ideographic full stops into that dot, and certificates normally name hosts without it.

def check_endpoint(host_name, port_number, timeout_seconds, verified_context, now):
    """Return (status, days_remaining, expiry_date_text, detail_text)."""
    connected = False
    peer_address = None
    try:
        with socket.create_connection((host_name, port_number),
                                      timeout=timeout_seconds) as tcp_connection:
            connected = True
            peer_address = (tcp_connection.family, tcp_connection.getpeername())
            with verified_context.wrap_socket(
                    tcp_connection, server_hostname=certificate_name(host_name)) as tls_connection:
                certificate = tls_connection.getpeercert()
    except ssl.SSLCertVerificationError as verification_error:
        # As near as this code gets to the moment OpenSSL judged the chain;
        # the second connection below can take a while.
        verification_clock = datetime.now(timezone.utc)
        verify_code = getattr(verification_error, "verify_code", None)
        verify_message = getattr(verification_error, "verify_message", None) or str(verification_error)
        if verify_code == OPENSSL_CERT_HAS_EXPIRED:
            status = "EXPIRED"
        elif verify_code == OPENSSL_CERT_NOT_YET_VALID:
            status = "NOT_YET_VALID"
        elif verify_code == OPENSSL_HOSTNAME_MISMATCH:
            status = "HOSTNAME_MISMATCH"
        elif verify_code == OPENSSL_IP_ADDRESS_MISMATCH:
            status = "HOSTNAME_MISMATCH"
        else:
            status = "UNTRUSTED"
        not_before, not_after, unreadable_reason = fetch_presented_validity(
            host_name, peer_address, timeout_seconds)
        if not_after is None:
            return status, None, "", f"{verify_message}; presented dates unreadable ({unreadable_reason})"
        detail_text = verify_message
        # The status comes from the whole chain; the dates are the leaf's.
        # When the leaf's own dates leave the status unexplained, print the
        # dates rather than a judgement about the leaf, and say where the
        # failing certificate must be instead.
        # The leaf is judged at the failed handshake, not at the run's clock
        # (earlier) or after the second connection (later): a leaf that
        # expired after the handshake failed played no part in it.
        leaf_explains_status = ((status == "EXPIRED" and not_after <= verification_clock)
                                or (status == "NOT_YET_VALID" and not_before > verification_clock))
        if status == "EXPIRED" and leaf_explains_status and not_after > now:
            # The leaf was still valid at the run's clock reading and had
            # expired by the handshake; its days come from the handshake, so
            # an EXPIRED row the leaf explains never shows days left.
            now = verification_clock
        if status in ("EXPIRED", "NOT_YET_VALID") and not leaf_explains_status:
            failing_kind = "expired" if status == "EXPIRED" else "not-yet-valid"
            detail_text = (f"{verify_message}; the presented leaf runs "
                           f"{not_before.date().isoformat()} to {not_after.date().isoformat()}, "
                           f"so the {failing_kind} certificate is elsewhere in the chain or "
                           f"behind another backend")
        elif status == "NOT_YET_VALID":
            detail_text = f"{verify_message}; valid from {not_before.date().isoformat()}"
        return status, days_until(not_after, now), not_after.date().isoformat(), detail_text
    except ssl.SSLError as tls_error:
        # The TCP connection opened and TLS itself failed, for example a
        # plain text service on that port, or no protocol version in common. A reset
        # during the handshake surfaces as OSError and lands below instead.
        detail_text = str(tls_error)
        if getattr(tls_error, "reason", None) in PROTOCOL_VERSION_REASONS:
            detail_text = (f"{detail_text}; the endpoint offers only TLS versions this "
                           f"host's OpenSSL refuses (often TLS 1.0 or 1.1)")
        return "TLS_FAILED", None, "", detail_text
    except OSError as connection_error:
        # Refused, unresolvable, unreachable, or timed out. When the TCP
        # connection was accepted first, say so: an open port that never
        # answers the handshake points somewhere else than a closed one.
        detail_text = str(connection_error) or type(connection_error).__name__
        if connected and isinstance(connection_error, socket.timeout):
            detail_text = (f"connected, then timed out after {timeout_seconds:g} s "
                           f"waiting for a TLS reply")
        elif connected:
            detail_text = f"connected, then {detail_text}"
        return "UNREACHABLE", None, "", detail_text
    except (ValueError, OverflowError) as socket_value_error:
        # The host list and --timeout are checked before anything connects,
        # so this is not expected; should the socket layer still refuse a
        # value, this endpoint gets a row and the run goes on.
        return "UNREACHABLE", None, "", f"the socket layer refused the endpoint: {socket_value_error}"

    not_after = datetime.fromtimestamp(
        ssl.cert_time_to_seconds(certificate["notAfter"]), tz=timezone.utc)
    return "OK", days_until(not_after, now), not_after.date().isoformat(), "verified"

Everything else, from the full script and the nightly wrapper to the test fixtures and the tests, is in the repository.

Running it nightly

# /etc/certificate_watcher/hosts.txt: the long tail that ACME leaves uncovered
#   dashboard01.example.net:8443       # internal dashboard, private CA
#   wiki.internal.example.net:8443
#   mailrelay.example.net:465          # implicit TLS; 25 and 587 need STARTTLS
#   [2001:db8::20]:9443

17 6 * * * bash /opt/certificate-expiry-watcher/examples/certificate_watch_nightly.sh /etc/certificate_watcher/hosts.txt --ca-bundle /etc/certificate_watcher/internal_ca.pem

The wrapper runs the watcher once and, on a non-zero exit, pipes the output that decided the exit code, with the exit status appended, into the SMTP alert dispatcher, which delivers it or spools it for retry: the rows for a finding, or for exit 2 the watcher's own error message, since a malformed host list prints no rows. If the dispatcher is missing, fails, or runs past its own two-minute limit, the wrapper writes the report to standard error, where cron mails it to the crontab owner on a host with cron mail. On a host with neither, the report reaches nobody, so the alert path needs testing before anyone relies on it. With CERTIFICATE_WATCH_LOG_FILE set, each run the wrapper sees end, clean ones too, is appended to a log when it ends; the log shows the watching happened only to someone, or something, that checks it keeps growing.

By default the wrapper also stops the watcher with timeout after ten minutes, and kills it ten seconds later if it is still running. --timeout bounds each connection attempt and the whole TLS handshake, which a separate measurement confirmed by feeding a handshake one byte every 0.2 seconds: it stopped at the limit. DNS lookups are bounded only by the resolver, though, and nothing inside the watcher bounds a long host list of slow endpoints. Without the cap, a stuck run would send nothing, and silence reads like health. A capped run exits 124, or 137 if it outlives those ten seconds, and mails the rows it printed before the cut, so the endpoint it was checking and any after it get no row; the wrapper runs Python unbuffered because under cron, output still sitting in a buffer when timeout stops the process is lost. Alert routing stays outside the watcher itself, which makes this the dispatcher's second consumer. Running it needs no root.

Evidence

Endpoint tests run the real script against live listeners on 127.0.0.1, one per case. Most serve certificates with fixed dates: valid until 2126, expired in 2021, valid only from 2100, issued for another name, issued by a CA outside the trust store, a leaf sent without its intermediate, and leaves within their own dates under an expired and under a not-yet-valid intermediate. Others answer in plain text, never answer, reset the connection, or offer only TLS 1.0 and 1.1, and a closed port and an unresolvable name cover the rest. Fixed dates keep each fixture's meaning until 2100, and the script that made them is in the repository. RFC 5280 encodes dates through 2049 as UTCTime and later ones as GeneralizedTime, so the fixtures carry both: the 2020, 2021, and 2026 dates in one, the dates from 2100 on in the other, and the tests read each.

As a check on the tests, test/mutation_controls.py in the repository first runs the untouched suite and stops unless it passes, then puts 81 deliberate defects into a scratch copy one at a time: a status mapped wrongly, an exclusive warning boundary, an ignored timeout, the hostname left out of verification, an unreachable endpoint not counted, a wrapper that exits 0 on findings, a wrapper that loses its rows when the cap stops it, and others. A named test fails for every one, and a workflow reruns the harness on every push. The harness covers the repository's code only; the lab scripts were rehearsed against stand-ins first. Two defects first slipped through. The test environment set PYTHONUNBUFFERED, which cron never does, so the wrapper tests now remove it. And a test meant to prove the content policy's key-material exemption passed with the exemption removed, because its planted word never stood alone; the test now plants it the way base64 bounds it. All 85 tests pass on Python 3.9 through 3.14 against OpenSSL 1.1.1, 3.0, and 3.5, with 3.9.0 and 3.9.4 among them. macOS and Windows are untested.

Outside the test suite, the first run was on my WSL2 workstation in October 2026, with Python 3.14 and OpenSSL 3.5. Three public sites verified against their real CA chains, with 31 to 79 days left. Public test hosts that serve known-bad certificates came back EXPIRED with its 2015 date, HOSTNAME_MISMATCH, and UNTRUSTED twice, a self-signed leaf and a self-signed root. A loopback port that accepted the connection and never answered came back UNREACHABLE after the ten-second timeout, which is why the detail now says when a connection was accepted first. That run used the workstation's own timeout, and the wrapper's run limit test passed there too. A later run printed UNTRUSTED for both self-signed hosts with no dates: one host reset even a plain connection, and the watcher had kept the status and dropped dates it could not fetch, as designed. Now the row says so, with the reason.

None of that covers a private CA, a load balancer that presents different certificates from different backends, or scheduled runs, so in November 2026 two virtual machines on my lab host ran them. A private CA's root alone verified the complete chain, and a leaf served without its intermediate came back UNTRUSTED. Four runs through haproxy alternated EXPIRED and OK at one address, each EXPIRED row naming the other backend. A link-local IPv6 peer kept its scope. In a two-hour soak on a one-minute timer, 121 runs reported OK until the warning crossing and EXPIRES_SOON after, one alert per finding, and the timer resumed by itself after a reboot.

Rehearsing that lab run against stand-in servers turned up one mistake. An endpoint given as an IP address its certificate leaves out came back UNTRUSTED, because OpenSSL reports a wrong address with its own code, 64, where a wrong name gets 62. Both now map to HOSTNAME_MISMATCH, and a test that fails on the old mapping keeps it that way.

Known limitations

Live TLS endpoints only, as the opening says. Implicit TLS only: STARTTLS services on ports such as 25, 587, 143, and 389 report TLS_FAILED, so watch their implicit-TLS ports (465, 993, 636) where they exist. The leaf's own dates are the ones reported; an intermediate that expires first fails verification and is caught, and the row says the expired certificate is elsewhere in the chain without naming which one. One address per endpoint is checked, the first that accepts a connection, so DNS round robin or a load balancer can hide a backend with an older certificate. An endpoint that offers only TLS 1.0 or 1.1, which older management interfaces still do, reports TLS_FAILED under OpenSSL 3, with a plain note when OpenSSL reports a protocol-version failure, and its certificate goes unread; Python 3.9 built against OpenSSL 1.1.1 still negotiates both and checks it like any other. One threshold for every endpoint, endpoints checked one at a time, and no revocation check.