Joseph Tracy. Infrastructure Engineer, Storage and Automation.

Operations 17 min read

Every Local Timestamp in an Incident Record Needs an Offset

A failover record in an incident ticket says it completed at 01:30 on Sunday 2026-11-01. The host is set to America/Los_Angeles, where the clocks read 01:30 twice that night, an hour apart. The ticket has no offset, so nothing in it says which pass of the clock the engineer saw.

An instant is a point in time, the same everywhere. An offset is the difference between a local clock reading and UTC at that instant, such as -07:00. A zone name, such as America/Los_Angeles, selects the rules that determine the valid offsets and transitions for each date. RFC 3339, the Internet profile of ISO 8601, makes the offset mandatory: its date-time grammar ends in a time-offset, either Z or a signed hours and minutes value.

RFC 5424 syslog derives its timestamp from RFC 3339 with a narrower syntax. A complete local date and time with an offset identifies an instant, as do UTC values and epoch seconds. A local timestamp without one identifies a reading on some clock, and the reader has to supply which clock. Keep source text as written, record each event's instant or mark it unresolved, and record the offset with local event times even where stored zone metadata could rebuild it, so the producer's mapping travels with the event.

Local time is the right thing to show a person and to schedule a region's working day. The trouble starts when a local reading with no offset becomes the record of when something happened. Away from transitions, in one known zone, it still maps to one instant if the zone assumption travels with it. It fails on the nights a zone changes its offset, when records from more than one zone share a timeline without each record's zone, and when rules change faster than hosts are updated.

Every output block below is printed by timestamp_reproductions.py in the timestamp-reproductions repository. It uses only standard-library code, reading zone data from the system or the tzdata package. It runs without root, changes no system configuration, fails if an asserted measurement changes, and lists any check the host could not run. The quoted runs come from one Ubuntu 26.04 workstation.

The hour that happens twice

2026-11-01 01:30:00 local  fold=0 -> 2026-11-01 01:30:00 -0700 PDT  (UTC 08:30:00Z)
2026-11-01 01:30:00 local  fold=1 -> 2026-11-01 01:30:00 -0800 PST  (UTC 09:30:00Z)
separation between the two readings: 1:00:00
compared as same-zone values, equal? True

date (uutils coreutils 0.10.0) -d '2026-11-01 01:30' -> 2026-11-01T01:30:00-0700 PDT   (exit 0)
gnudate (GNU coreutils 9.7) -d '2026-11-01 01:30' -> 2026-11-01T01:30:00-0700 PDT   (exit 0)

Two instants, an hour apart, with the same label. Python carries the choice in a fold attribute, added by PEP 495 because a wall clock reading alone cannot say which pass it means. Python also ignores fold when both values share a zone object, so the two readings compare as equal. Both date implementations on the workstation pick the first pass without a warning.

The ambiguity is as long as the transition, so on Lord Howe Island it lasts 30 minutes:

Australia/Lord_Howe 2026-04-05 01:45 local: fold=0 +1100, fold=1 +1030, separation 0:30:00

A timestamp from the repeated period resolves to one instant by itself only if it carries an offset or an equivalent marker, or was written as an instant. Otherwise it points at two candidates. A sequence number, a collector's receipt time, or a neighbouring system's log may support one of them, and the ticket should label that choice as a reconstruction. A correlation window has the same limit. One narrower than the transition can miss the right pairing. One wide enough to include both candidates leaves the match to another key.

The hour that never happens

On 2026-03-08 the same zone skipped from 01:59:59 to 03:00:00, so local 02:30 never occurred:

2026-03-08 01:59:59 local -> UTC 09:59:59Z
2026-03-08 03:00:00 local -> UTC 10:00:00Z   (one second later)
2026-03-08 02:30:00 local -> 02:30:00 -0800 PST  (UTC 10:30:00Z)   no error raised
that instant read back as local time -> 03:30:00 -0700 PDT

date (uutils coreutils 0.10.0) -d '2026-03-08 02:30' -> 2026-03-08T03:30:00-0700 PDT   (exit 0)
gnudate (GNU coreutils 9.7) -d '2026-03-08 02:30' -> gnudate: invalid date '2026-03-08 02:30'   (exit 1)

Python built a time that never happened, without error; converting it applied the PEP 495 gap rule, giving an instant that reads back as 03:30, an hour from the record. On the workstation, date is uutils 0.10.0, a Rust reimplementation of coreutils, and printed the same 03:30. GNU coreutils 9.7, installed beside it as gnudate, refused the string and exited 1, as GNU coreutils 9.4 did as date on an Ubuntu 24.04 host. Ubuntu's 25.10 release notes record the switch to rust-coreutils, with the GNU versions kept alongside. So the command named date changed behaviour with the distribution release.

A refusal can be detected; a silent conversion lands an hour away in the timeline. These results cover the tested implementations only. Test any parser you rely on with a time inside a gap, and require it to reject that time or report the conversion.

Duration arithmetic across a transition

start  2026-11-01 00:30 PDT -0700  = 07:30:00Z
end    2026-11-01 03:30 PST -0800  = 11:30:00Z
end - start, both aware, same zone object : 3:00:00
end - start, after converting to UTC      : 4:00:00
understated by                            : 1:00:00

The Python Library Reference states the rule: when two aware datetimes share the same tzinfo, subtraction ignores it. The three hours is that field subtraction, here the difference between the two wall clock readings. The four hours is the elapsed time between the two recorded instants, and measures the outage only if those records mark its start and end and their clocks were right. Other runtimes define their own arithmetic, so test the type you use. The spring case fails the other way:

01:30 -> 03:30 local on 2026-03-08, same zone object : 2:00:00
same interval in UTC                                 : 1:00:00
overstated by                                        : 1:00:00

outage spanning fall back        reported 180 min   actual 240 min   error -25%
outage spanning spring forward   reported 120 min   actual  60 min   error +100%

Against a service target, the fall back understatement can hide a real breach. The spring forward overstatement can report a breach that never happened. The percentages belong to these intervals only.

Sorting by string against sorting by instant

Four synthetic records, each a complete RFC 3339 timestamp from a different system:

-- sorted as strings --
   2026-11-01T01:05:00-08:00    failover completed
   2026-11-01T01:20:00-07:00    cache server evicted
   2026-11-01T08:45:00+00:00    api gateway returns 503
   2026-11-01T09:02:00+01:00    operator acknowledged

-- sorted by the instant each one names --
   2026-11-01T09:02:00+01:00    08:02:00Z   operator acknowledged
   2026-11-01T01:20:00-07:00    08:20:00Z   cache server evicted
   2026-11-01T08:45:00+00:00    08:45:00Z   api gateway returns 503
   2026-11-01T01:05:00-08:00    09:05:00Z   failover completed

sort with LC_ALL=C produces the same string order: a failover, a cache eviction, gateway errors, then an operator noticing. Instant order swaps the ends. The acknowledgement came first, and the failover completed 63 minutes later, 20 minutes after the gateway started returning errors. The first two records come from Los Angeles hosts either side of the fall back; the one reading 01:05 happened 45 minutes after the one reading 01:20.

RFC 3339 section 5.1 says string sorting gives time order when every timestamp uses the same time zone, expressed using the same string, with the same number of fractional digits. These records break the first condition. The precision condition catches logs that are already all UTC:

-- two UTC records, one written without fractional seconds --
   as strings : 2026-11-01T08:30:00.500Z  then  2026-11-01T08:30:00Z
   as instants: 2026-11-01T08:30:00Z  then  2026-11-01T08:30:00.500Z
   as epoch seconds     : 1793521800  and  1793521800   (tie, order lost)
   as epoch milliseconds: 1793521800000  and  1793521800500

A period sorts before the letter Z, so the later record lands first. Epoch seconds tie on that pair and milliseconds keep it apart; a source that records microseconds needs an integer microsecond key. The four incident records carry whole seconds, so epoch seconds order them:

2026-11-01T09:02:00+01:00   ->  1793520120   operator acknowledged
2026-11-01T01:20:00-07:00   ->  1793521200   cache server evicted
2026-11-01T08:45:00+00:00   ->  1793522700   api gateway returns 503
2026-11-01T01:05:00-08:00   ->  1793523900   failover completed

Byte-by-byte text order matches time order on keys like these: unsigned decimal values at one precision, zero-padded to one width. Keep the original string beside the key, because it carries the offset the source wrote.

Normalized instants still order events only by the clocks that recorded them. RFC 3164 warns that syslog delivers messages in no guaranteed order, and clock error, buffering, and replay can each put an effect before its cause. Sorted, the example suggests the eviction caused the 503s; timestamps cannot show that. A causal claim needs its own evidence, such as a source sequence number, an explicit dependency, or a verified request trace.

What the common formats leave out

One instant, 2026-11-01T08:30:00Z, in formats incident timelines are often built from:

RFC 3164 syslog  Nov  1 01:30:00                                 no year, no zone
Apache %t        [01/Nov/2026:01:30:00 -0700]                    offset present, one-second resolution
ISO 8601 local   2026-11-01T01:30:00                             no offset; 2 candidate instants in this zone
Python asctime   2026-11-01 01:30:00,000                         %(asctime)s field: local, no offset
RFC 3339 UTC     2026-11-01T08:30:00Z                            instant identified
epoch seconds    1793521800                                      instant identified, unreadable
RFC 9557         2026-11-01T01:30:00-07:00[America/Los_Angeles]  instant plus zone name

RFC 3164, which RFC 5424 obsoletes, describes the BSD syslog timestamp as local time with no year and no zone. Apache's default %t carries an offset at one-second resolution, and mod_log_config offers finer tokens. By default, Python's logging fills %(asctime)s using time.localtime() with no offset, as its documentation states. An application whose format includes that field and keeps the default conversion writes the ISO 8601 local problem into its own logs.

The last row uses RFC 9557 (April 2024), which updates RFC 3339 in two ways. It allows a bracketed zone name after a complete timestamp, and it changes what Z claims: the time in UTC is known and the offset to local time is unknown. A UTC timestamp field identifies the instant and is silent about where its source was; other fields in the record may say.

Offsets, zone names, and fixed-offset settings

2026-01-15 12:00 America/Los_Angeles -> offset -0800
2026-07-15 12:00 America/Los_Angeles -> offset -0700

An offset records the local-to-UTC difference at a timestamp and carries no transition rules. A recurring window stored as a fixed offset shifts against local working hours after each seasonal transition. One stored with a zone name stays tied to them; its UTC instants move when the rules change.

Two settings label a July instant with -0800:

2026-07-15 12:00 Etc/GMT+8 -> offset -0800 -08   (POSIX sign convention: +8 means eight hours behind UTC)
2026-07-15 12:00 TZ=PST8 (C library) -> offset -0800 PST   (POSIX TZ string with no daylight rule)

The first follows the POSIX sign convention that the tz database's etcetera file explains; the second still prints PST. A timestamp written with -0800 under either setting identifies the right instant, while its July local label is an hour behind Los Angeles civil time. That is a configuration error only if the host was meant to follow Los Angeles rules. In that case, any local value it wrote without an offset is an hour off too.

The rules your host is carrying

Zone rules change by government decision, and the tz database records them in numbered releases. Release 2026c (2026-07-08) recorded that Alberta stopped changing clocks; 2026d (2026-09-11) did the same for the Northwest Territories; 2026e (2026-09-29) recorded that Manitoba moves to permanent -05 effective 2026-10-31, 32 days later. On 2026-10-09 the workstation reported release 2026c, and its distribution's update and security archives offered nothing newer:

tz database release on this host: 2026c (/usr/share/zoneinfo/tzdata.zi)

America/Edmonton   01:30 fold=0 -0600  fold=1 -0600  hour repeats: no   2026-11-02 -0600 CST
America/Inuvik     01:30 fold=0 -0600  fold=1 -0700  hour repeats: yes  2026-11-02 -0700 MST
America/Winnipeg   01:30 fold=0 -0500  fold=1 -0600  hour repeats: yes  2026-11-02 -0600 CST

The script names the data source each zone loads from, requires all of them to share one, and reads that source's release label. Under 2026c, Inuvik and Winnipeg still repeat an hour; under 2026e, Winnipeg stays at -05. A process resolving America/Winnipeg with 2026c rules will render local times an hour behind official time from 02:00 on 2026-11-01 until it loads updated data. Measured against the rules in force, a system that captures UTC and renders local time later shows a wrong label; one that captures local time and converts it with stale rules writes a wrong instant. The 2026e notes say the database models the Manitoba change at 02:00 on 2026-11-01 instead of its legal date, as a temporary measure.

Wall clock against monotonic clock

time       implementation=clock_gettime(CLOCK_REALTIME)        adjustable=True  monotonic=False
monotonic  implementation=clock_gettime(CLOCK_MONOTONIC)       adjustable=False monotonic=True

time.time() reads the realtime clock, which can be set to jump. POSIX says the monotonic clock cannot be set through clock_settime() and counts from an unspecified point. Python's adjustable=False rules out jumps only; NTP may still adjust its rate. Python's and the kernel's documentation describe CLOCK_MONOTONIC_RAW as free of that rate correction, which leaves the hardware's own drift in place. On Linux, CLOCK_MONOTONIC, which time.monotonic() reads, stops during suspend; CLOCK_BOOTTIME counts through it, and Python reads it with time.clock_gettime(time.CLOCK_BOOTTIME). A monotonic duration holds only between two readings of the same clock, so record at least the host, the boot, the clock ID, and the units, and keep both readings.

Salvaging the records you already have

Each archived local timestamp ends in one of three states. Direct: its original text carries a complete date and time, plus either an offset or metadata recorded with it giving the producing process's zone setting and rule release, under which that local time occurred exactly once. Clock accuracy is a separate question. Reconstructed: the mapping relies on current rules or on independent evidence, stored beside the result and labelled inferred. Unresolved: keep a candidate set or a bounded interval instead of assigning an offset. Keep the original string in all three.

A practical retrofit is recording each host's zone setting and tz release in inventory on a schedule, with the date observed. For processes that used the host's setting, it bounds later reconstructions; a change between two observations stays unknown. On a systemd host, timedatectl show --property=Timezone --value prints the zone name; where shipped, the first line of /usr/share/zoneinfo/tzdata.zi names the release.

The recording contract

Consider a pipeline that stores UTC instants, never converts local input, and renders local time only in its interface. It needs no zone name on each event. It still needs layer 4 for timing claims, because a UTC timestamp from an unsynchronized host carries no bound on its error. That evidence can sit on each event or in host synchronization records it references. Every other field earns its place only when a claim depends on it. Each costs storage and an owner, so the contract comes in layers:

1 instant         2026-11-01T08:30:00Z                         resolved events, at the source's precision
1 instant_status  direct                                       direct, reconstructed, or unresolved
2 source_text     [01/Nov/2026:01:30:00 -0700]                 local-time evidence: kept verbatim
2 source_zone     America/Los_Angeles                          local-time evidence: producer's setting
2 source_release  unknown                                      local-time evidence: producer's tz release
3 rendered_local  2026-11-01T01:30:00-07:00                    rendering provenance: derived
3 rendered_with   zoneinfo, Python 3.14.4, tz 2026c            rendering provenance: what produced it
4 timeQuality     tzKnown="1" isSynced="1" syncAccuracy="4000" timing claims (synthetic values)
RFC 9557 form     2026-11-01T01:30:00-07:00[America/Los_Angeles]

Layer 1 applies to every event. It holds a status, direct, reconstructed, or unresolved, and for a resolved event the instant, in UTC or with its offset, at the source's precision. Padding a whole-second source with .000 can be read as a resolution it never had. An unresolved event has no honest instant; keep its source text and its candidates or a bounded interval. Give event, emission, and collector receipt times separate fields, each named for what it measures.

Layer 2 applies when local time is itself evidence. For a machine source, keep the source text verbatim and the zone setting the producing process used, which a TZ variable can make differ from the host's. Record its tz release when known, or unknown when not established, never the renderer's; it identifies the rules the producer may have used. A person's statement that they acted at 01:30 is different evidence: record which clock they read, such as a phone or a wall clock, and whether its zone was observed or inferred.

Layer 3 applies when a rendered local time will be quoted later. Record at least the library, runtime version, and tz release that produced it. When exact reproduction matters, keep a copy of the zone file itself; a digest alone only detects a mismatch.

Layer 4 applies to timing claims and borrows the RFC 5424 section 7.1 parameter names and meanings as local fields. tzKnown says the originator knows its time zone, and is 1 even when it emits UTC; it records no zone name. isSynced says it is synchronized to a reliable external source. syncAccuracy is the originator's maximum expected clock error between synchronizations, in microseconds, allowed only when isSynced is 1 and the originator knows that figure. The block's values are synthetic, and the fields record what the originator asserts; verifying it is separate work.

The RFC 9557 form puts the instant and a zone name in one string. It holds no source text, and its zone name proves nothing about the source's setting. Its offset and local time came from whatever rules produced the string, so layers 2 and 3 apply when their triggers hold. Its offset and zone name can disagree when producer and consumer carry different rule releases. RFC 9557 calls such a string inconsistent. A ! marks the zone suffix critical. Then the application must act on the conflict, for example by rejecting the string or resolving it with other information; without the mark, it may. The policy here goes further: preserve both values, flag the disagreement, and label any resolution as inferred.

Fields left unrecorded can sometimes be rebuilt from deployment records, and the result is an inference. Say Only What the Evidence Supports asks every impact claim to say whether it was observed, estimated, or unknown. A timestamp is a claim of the same kind, and these layers show which assumptions it rests on.

At read time, parse raw timestamps from more than one system before ordering them, or normalize them to one sortable key. Add a stable secondary key so the output is deterministic. A tie at the recorded precision stays a tie: the secondary key orders the lines and leaves the events unordered. A finished timeline hides how it was sorted, so record beside it the sort key, the parser and its version, the normalization policy, and the tie-break rule.

References

Sources verified 2026-10-09.

Incident Records Evidence