Systemd Evidence Collector

A service has failed and someone opens a terminal. They run systemctl status, scroll, run journalctl -u with a window they pick by feel, read what comes back, and form an opinion. Then the shell closes.

Two days later a review asks what the unit reported at the time. The commands are still available and the system has moved on, so re-running them answers a different question than the one being asked. The opinion formed on day one is now the only record of day one, and it arrives in the review as a conclusion with no evidence attached to it.

This is two tools. collect_systemd_evidence.sh writes the evidence down once. analyze_evidence_bundle.py reads a bundle and produces a review report.

What this sits next to

sos report already collects far more than this, and that broader collection model fits vendor support cases. It runs as root, sweeps system-wide state, and produces an archive measured in megabytes. This collects one unit, runs as whoever invokes it, and produces a directory a person reads in a few minutes.

systemctl status and journalctl are what this runs, and their output goes to a terminal and ends there. systemd-analyze verify is static and never sees runtime state; this captures both and keeps them beside each other.

So the contribution lies past the commands, in the record of which ones ran, what each exited with, and what the output was at that moment.

What the collector will not do

A safety boundary that comes after the features is decoration, so this one comes first.

Stated precisely, this is service-state read-only. No systemd or service state changes: no sudo, and no start, stop, restart, reload, reset, enable, disable, mask, unmask, or kill. It reads unit properties, unit status, a bounded journal slice, process information, and optionally one loopback health URL and one path, where it collects metadata only and never reads file contents.

Calling it read-only without that qualifier would overstate it, in a project about keeping claims the size of their evidence. It creates a directory, writes files under umask 077, and sets modes, all inside the output directory you name. A health probe issues one HTTP GET, and the receiving application decides for itself whether a GET has side effects.

I built the boundary in rather than documenting it, because a diagnostic tool that can also remediate gets used to remediate. The moment a person can pass a flag that restarts the unit, the bundle stops being evidence of the failed state and starts being evidence of the state after someone intervened.

The file that matters most

A bundle holds the outputs you would expect: systemctl-show.txt, systemctl-status.txt, journal.txt, unit-effective.txt, process.txt, and the health probe result where one was requested. systemctl-show.txt includes Restart, the restart policy the manager actually holds, beside NRestarts and RestartUSec. It also holds three files that describe the collection itself.

File	What it answers
command-outcomes.tsv	Which commands ran, and what each one exited with
manifest.tsv	Every retained evidence artifact and collection record present before finalization, with size and digest. It excludes itself and its separate digest anchor.
collection.json	Machine-readable record of the collection parameters

command-outcomes.tsv is the one I would keep if I could keep only one. A bundle where journalctl exited non-zero because the account lacked group membership, and a bundle where journalctl succeeded and the window was quiet, contain the same empty journal.txt. Without the outcome record, a reader treats both as evidence of a quiet journal, and one of them is evidence of nothing at all.

A manifest answers the same class of question. A bundle edited between collection and review is a different artifact from the one collected, and the analyzer recomputes every digest rather than assuming.

Coverage extends to the collection records as well as the evidence files. That was a defect until this revision: command-outcomes.tsv and collection.json sat outside the manifest, so the layer that tells a reader how to interpret the evidence could be rewritten without a single digest changing. Validation now also rejects a manifest that omits a captured artifact, lists one twice, or disagrees with the outcome record about a command's exit code.

That catches accidental change: a truncated copy, a file edited in place, a transfer that lost bytes, a bundle assembled from two runs. Deliberate alteration is a separate question, and the answer there is weaker. The digests come from the script that wrote the files, they sit in the same directory, and nothing signs them, so anyone who can edit an artifact can recompute the manifest to match.

So the collector prints the manifest digest on stdout and writes it to manifest.sha256. If the bundle has to survive a hostile reader, that digest has to leave the host at collection time and land somewhere the host controls nothing. Calling the bundle tamper-evident without that step would be the same kind of overstatement this tool exists to avoid.

What the report refuses to say

Each run emits analysis.md, analysis.json, and validation.txt. Its markdown report opens by stating that it is evidence-linked and observational with bounded review guidance: no cause is named, and no state change is authorized. Severities classify attention rather than ranking causes.

Findings are named for the observation rather than the interpretation. OBS-UNIT-FAILED records that the unit reported a failed state, points at systemctl-show.txt and systemctl-status.txt as the evidence behind it, and references the playbook section a person should work through. It proposes no fix and ranks nothing by likelihood, because a ranked list is a conclusion wearing a table.

This is the same discipline as the claim-language work, applied one layer earlier. Claim language governs what a person writes after they have decided. This governs what the tooling hands them before they decide.

Exit zero, with diagnostics attached

--include-unit-verify adds systemd-analyze verify output to the bundle. The verifier reports an ignored directive on stderr and still exits 0, so a unit containing Restart=on-failuer verifies at exit 0 while the manager runs it with no restart policy. Both the exit code and the diagnostic line count land in one row of command-outcomes.tsv, beside the output itself.

Two analyzer observations read that evidence. OBS-UNIT-VERIFY-DIAGNOSTICS fires when the verifier printed anything, whatever its exit status. OBS-RESTART-DECLARED-DIFFERS fires when the captured unit text, drop-ins included, resolves to a different restart policy from the one the manager reports. An absent assignment and an empty one both resolve to Restart=no. Both stay observations. A mismatch says the declared policy may not be the one in effect, and a person decides what that means.

The verifier behavior itself is the subject of The Directive That Was Ignored.

Synthetic scenarios and the test loop

A --fixture mode generates a deterministic bundle for one of six scenarios: healthy, failed_config, permission_failure, dependency_failure, incomplete_bundle, and ignored_directive. No systemd or service state is touched. Every test fixture and every committed example in the repository is produced this way.

That mode also closes the loop the test suite depends on. test/run_all_tests.sh regenerates the analyzer fixtures from the collector before running the analyzer tests, so a change to the bundle format that breaks the analyzer fails the suite. A fixture set committed once and never regenerated goes stale without any signal, and a suite passing against stale fixtures reports the same green as a suite that works.

Fifty-nine analyzer tests cover the six scenarios, the observation rules, and a long list of deliberately broken bundles. Collector tests assert exit codes, expected files, file modes, the Restart property in the live collection, and a verifier result that keeps its diagnostics at exit 0.

They also test the claims this page makes. A test fails if the source contains sudo or a state-changing systemctl verb, if a live run labels its own output as a fixture, if manifest.sha256 disagrees with the manifest, or if a run that lost artifacts still reports itself as complete. A safety boundary asserted only in prose is a boundary until someone edits the script.

Broken-bundle cases cover a manifest missing a captured artifact, a duplicate row, a header-only manifest, a malformed row, an unlisted file added to the directory, an edited collection.json or outcome record, a duplicated or unrecognized outcome row, a manifest exit code that disagrees with the outcomes, and a rewritten anchor. Each must come back invalid, and a malformed row must fail cleanly rather than raise. One test walks every fixture and fails if a report cites an evidence file that bundle never captured, which is how I found two reports doing exactly that.

Another group tests the failure paths rather than the successful ones. A stubbed sed forces redaction to fail, and the run must record an integrity error, retain nothing, and report itself as partial. A pre-existing output directory must keep its own mode. Journal boundaries must carry an explicit UTC suffix, and the health probe must refuse proxies.

One group matters more than the rest. It marks an artifact as skipped in the outcome record, writes a plausible signal into the file anyway, and requires the report to derive nothing from it. A second mutates the outcome record itself and keeps the manifest and its anchor consistent with the mutation, so the bundle is internally coherent and structurally invalid at once; the report must then carry no observation about the service at all. An observation drawn from evidence the bundle never vouched for would be the project failing at its own premise, so both halves of the contract have to hold before any content is read: the outcome record calls the artifact captured, and its manifest row verified.

Calling an artifact captured is itself a claim, so a third group tests what that word is allowed to mean. An outcome row recorded as captured with no exit status must invalidate the bundle and produce no observation about the service, because a row that claims a command ran while recording nothing about how it ended falls short of evidence that it ran. Collection metadata is checked field by field against the types and vocabularies the analyzer reads, with each case rehashed so its bytes still verify, which is the case that separates a record that is authentic from one that is usable. A last test reads every observation the fixtures can produce and fails if its reference points at a review guide section that has since been renamed or removed, because a rename that silently breaks every link is exactly the failure a reader would find first.

What has actually been run

Check	Status
Fixture suite, six scenarios	Passing
Analyzer suite, 59 tests including the broken bundles	Passing
Collector contract and safety tests	Passing
ShellCheck at the severity CI uses	Clean
Verifier behavior on systemd 255	Reproduced directly
Drop-in handling by the verifier on systemd 255	Reproduced directly
Live unprivileged collection on systemd 259	Passing
Live collection as root on systemd 259	Passing
Live collection with a restricted journal	Not recorded
Distributions other than Ubuntu	Not recorded

Unrecorded rows stay unrecorded rather than assumed.

test/live_smoke.sh covers what fixtures leave open. It requires systemd as PID 1, collects from a unit you name, verifies every digest, runs the analyzer, and changes no service state. The suite leaves it out, because the suite runs where systemd is absent.

Before publishing I ran it against a unit name the host has no fragment for, unprivileged and then as root. That case matters more than the staged ones. The collector captured what the manager said, recorded LoadState=not-found as evidence rather than as an error, marked unit-verify.txt unavailable because no fragment existed to verify, reported the run as partial, and still produced a bundle whose every manifest row verified. Reading it, the analyzer raised nothing about the service, because a bundle that documents its own gaps has earned no conclusion. A tool that invented a diagnosis from that evidence would be the failure this project exists to avoid.

No Python package dependencies

Python 3.9 or later, standard library only. Bash 4. No requirements.txt sits in the repository.

That is the precise claim. Collector runs still need system commands, and the README lists them by mode. Live collection needs systemctl, journalctl, and ps; fixture mode needs none of them. A health probe needs curl, path metadata needs namei and stat, verification needs systemd-analyze, and ordinary userland tools are needed throughout. A missing one is an environment error with its own exit code, so a stripped appliance produces a clear message rather than an artifact that reads like evidence about the unit.

Keeping Python dependency-free is a deliberate constraint rather than a small footprint for its own sake. The host where a service has failed is frequently the host where installing a package is a change request, and a diagnostic tool that has to be installed before it can diagnose arrives after the window it was meant to capture.

Known limits

The collector reads what an unprivileged account can read. A unit whose journal is restricted, or whose fragment sits somewhere the account cannot reach, produces a bundle with those rows failed in command-outcomes.tsv. The behavior is correct and the report labels it, but a bundle collected without the right group membership answers fewer questions than one collected with it.

Redaction is narrow and best-effort, well short of a secrets scanner. Assignments whose name contains one of a dozen credential words are blanked, plus user:password pairs inside URLs, and it blanks to the end of the line.

Blanking the whole line is the interesting part. An earlier rule stopped the value at whitespace, so Environment="TOKEN=abc 123" came out as Environment="TOKEN=[REDACTED] 123". That is the worst possible outcome: the output announces that a secret was handled while half of it sits in the file. A reviewer found it, and the fix was to fail toward over-redaction.

What still escapes: a secret under an unmatched name, an embedded certificate, a customer identifier, a token inside a JSON body. --path never opens the file it describes, but unit-effective.txt carries whatever the unit file holds. Read a bundle before it leaves a host.

One field escapes by design. The URL passed to --health-url is written into collection.json verbatim, as inputs.health_target, so the hostname, port, and path travel with the bundle. Redaction leaves it alone, because the filter blanks credential-shaped assignments inside captured command output rather than metadata the collector recorded about its own inputs. The reason to keep it is that a reader judging a health artifact has to know what was probed; a bundle that shows exit 22 against an unnamed endpoint answers very little. Against that, an internal endpoint name becomes part of the evidence. A reviewer raised this, and I kept the field and wrote the tradeoff into the limits rather than dropping it without saying so. An operator who cannot share the endpoint can edit the field and regenerate manifest.tsv and manifest.sha256, and the bundle still verifies.

Privileges decide how much journal there is to collect, and journal visibility depends on privileges, groups, ACLs, distribution defaults, and whether the host keeps a persistent journal at all. A restricted caller still sees journalctl exit 0. Recording the effective UID and a coarse access value lets the analyzer state in the report when a short slice may follow from the caller's context rather than from a quiet unit. The collector cannot prove a slice is complete, and the report says so instead of implying otherwise.

A bundle is also a window rather than a snapshot. Its commands run in sequence, so a process can exit between systemctl show reporting its PID and ps being asked about it. The journal query carries the lower expression you passed and a pinned upper boundary in UTC, and both are recorded.

The analyzer parses systemctl show key-value output. A property that systemd renames or reformats in a future release stops being parsed, and the report then shows fewer state facts rather than wrong ones. I would rather it degrade that direction.