All notable changes to `validateBackup` are documented here. Format loosely based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [Unreleased]

Stages of `Documentation/ImplementationPlan.md`, in progress. **This entry is extended in place as each stage completes, rather than versioned per stage** — the whole subproject stays uncommitted until coding is complete (see `Documentation/ImplementationPlan.md`'s "Version control checkpoint"), so minting a new version number per stage would produce phantom versions that never existed independently in SVN history, the same reasoning the project applies to `VERSION.md` bumps elsewhere. The first real version number and date are assigned when this is actually added to SVN.

### Fixed

- **`niceness` was silently never applied to any backup-side command** (found auditing resource guards ahead of production deployment against an 18M+ file dataset, 2026-09-13). `activeSideWrapper` — the only place `nice`/`ulimit -v` were ever added to a command — was called exclusively from the `defined $host && $host ne ''` branch of `buildListingCommand`/`buildChecksumCommand`, i.e. only for the active side's commands run over ssh. The backup side's own `find`/`stat`, digest, and `sort` commands ran completely un-niced regardless of what `niceness` was set to. Split the single `niceness` key into **`activeNiceness`** and **`backupNiceness`**, independently tunable (rodolico: the backup host is typically far more quiescent than the active one and may want little or no nicing even when the active side needs it aggressively). Added `backupSideWrapper` (nice only — no `ulimit -v`; the backup host doesn't carry the active host's must-never-OOM constraint, and `find`/`stat`/the digest commands already stream in bounded batches regardless of file count, so the wrapper's only real job on that side is CPU/IO priority) and wired it into all three backup-side command builders (`buildListingCommand`, `buildChecksumCommand`, `buildSortCommand`). Verified by hand: active-side commands still get both `ulimit -v` and `activeNiceness`'s `nice`; backup-side commands now get `backupNiceness`'s `nice` and nothing else; `sort` gets both `backupNiceness`'s `nice` and `-S <sortMemoryLimit>` when set.
  - While auditing this, also changed `sortMemoryLimit`'s **default from `''` (unbounded) to `500M`** — it was previously described as "not safety-critical, a courtesy cap only," which undersold it: `sort` is the one command in the whole pipeline that genuinely buffers (`find`/`stat`/digest all stream in bounded batches regardless of file count), and confirmed on real hardware to reach 1.3GB RSS sorting a mere few-hundred-MB listing with no `-S` set (`Documentation/TESTING.md` Part 3.4). `500M` bounds `sort`'s in-memory chunk size independent of file count (an 18M-file listing just does more merge passes, not more peak memory) — reasonable for a generally-quiescent backup host; `''` remains available to opt back into `sort`'s own uncontrolled default. Updated `validateBackup.datastructure`'s generated default config, `cleanupConfig`, `USAGE.md`, `ProblemDefinition.md`, and `Documentation/TESTING.md`'s Part 3.4 note accordingly.

- **`--dry-run` never showed the `sort` command it would run, despite `ProblemDefinition.md` promising it prints "the exact commands that would run"** (found manually running `Documentation/TESTING.md`'s Part 5.4, 2026-09-13). `sortListing` built its command inline and ran it immediately — unlike `buildListingCommand`/`buildChecksumCommand`, there was no separate builder `--dry-run` could call. Extracted `buildSortCommand` (mirrors `sortListing`'s command construction exactly) and `datasetTempDirPath` (the temp-directory naming `datasetTempDir` already used, pulled out so `--dry-run` can build the same paths without `datasetTempDir`'s `mkdir` side effect — `--dry-run` creates nothing on disk), and wired both into the `--dry-run` preview alongside the existing listing/checksum commands.

### Added

- **Stage 2 (script scaffolding + config).** `validateBackup` script skeleton with the standard header (license, purpose, debug-level legend, revision history) and `replicate/replicate`'s structural pattern: `$programDefinition`, config load/validate via `ZFS_Utils::loadOrCreateConfig`/`checkConfigKeys`, `Getopt::Long` CLI parsing, a `sessionType` named-profile overlay (new machinery — no precedent elsewhere in this project; applied before CLI overrides, so CLI still wins), and the full config-default precedence chain (CLI > sessionType profile > config file > built-in default). Also: logging init, a PID-based lock file guarding against overlapping runs (a live PID blocks and exits 0 without emailing; a dead one is reclaimed), pre-flight environment checks (temp/findings directories created and writable with enough free space, required commands on PATH, active host reachable via ssh — all read-only), and a scaffolding main flow that resolves and reports the configured active/backup dataset name mapping (`--dry-run` prints it without reading any file data). `validateBackup.datastructure` provides the ~39-key commented default configuration.
  - Dataset/snapshot resolution, file comparison, digesting, scrub handling, and full reporting/exit-code semantics are **not yet implemented** — see `Documentation/ImplementationPlan.md` Stages 3–6.
  - Found and fixed during this stage's own functional smoke test (not deferred to Stage 7): the `--dry-run` CLI flag was initially declared as `dryrun|n` (copied from `replicate`'s naming without checking it against this spec, which names the flag `--dry-run` everywhere) — corrected to `dry-run|n`. Also: `--debug 8` and `--debug 6`'s dump-and-die paths ran after the lock file was acquired but `die`d before reaching `cleanup()`, leaving a stale lock that would have blocked the next real run — both now release the lock explicitly before dying.

- **Stage 3 (resolution + scrub/autoscrub).** Phase 1: `expandDatasetPairs` turns each configured dataset into concrete active/backup pairs, expanding into child datasets where `recursive` is set, filtering on `excludeDataset`, and reporting backup-side datasets with no active counterpart as orphans (`findOrphanBackupDatasets`). `findSharedSnapshot` intersects each pair's snapshots on the short name and picks the newest by the ZFS `creation` property — deliberately not by parsing the snapshot name, so an undated hand-made snapshot still orders correctly and both sides agree after a `send`/`recv`. `resolveSnapshotRoot` derives `<mountpoint>/.zfs/snapshot/<snap>` and handles zvols, `mountpoint=none`, `mountpoint=legacy` (via the mount table, never mounting anything), and `mounted=no`, each as a recorded skip; `snapshotRootUsable` confirms traversability with a cheap `test -d -r -x` before any long scan. Snapshot lists are fetched one query per configured entry per side and bucketed per dataset, so a recursive tree costs two round trips rather than two per child.
  - Phase 2: `checkScrubFreshness` classifies every involved pool as `ok`/`stale`/`never`/`unknown` against `scrubMaxAge`, and `runScrubPhase` starts scrubs only when *all* of autoscrub-enabled, not-`--no-scrub`, not-`--dry-run`, pool-genuinely-stale, and not-already-scrubbing hold. `waitForScrubs` polls to completion or `scrubWaitMax`, on which it logs, skips validation, and exits non-zero. **A pool whose scrub state cannot be determined is classified `unknown` and never auto-scrubbed** — starting a multi-hour pool operation because a regex missed or a host blipped is the wrong failure mode. `zpool scrub` is asynchronous, so scrubbing both sides concurrently needs no forking.
  - `--dry-run` now prints the real plan (dataset mapping, resolved snapshot and roots per pair, scrub ages, and what it would scrub) and reads no file data. `buildResolutionReport` renders the `=== VALIDATION SUMMARY ===` block; Stage 6 replaces it with the full per-file version.
  - Only pools belonging to *resolved* pairs are checked or scrubbed, deliberately narrowing the one mutating operation's scope to what the run actually depends on. Skips that are permanent properties of a dataset (a zvol, a `mountpoint=none` container) are reported but kept out of the error list, so a pool containing a zvol does not make every run exit non-zero forever.
  - Debug levels added: `1` dumps the expanded pairs, `2` the pairs after snapshot/root resolution plus orphans, `3` the per-pool scrub states.
  - The active-side resource wrapper (`activeMemoryLimitKB`, `niceness`) is **not** applied yet: it guards the bulk listing/digest commands, which land in Stage 4, and wrapping fixed-size metadata queries in it would add shell-composition risk for no benefit. Stage 4 applies it in the command builder.
- **Stage 4 (listing + merge-join core).** Phase 3: `buildListingCommand` emits one `<relative path>\t<size>\t<mtime>` record per regular file from a snapshot root, with the path **first** so a byte-wise sort of the whole line *is* a sort by path, and a literal tab built in Perl so correctness does not depend on whether BSD `stat(1)` interprets a `\t` escape. `find -s` is deliberately not used — it orders per-directory, which is a different order from a byte-wise sort of whole paths. An **exit-status sentinel written to stderr** is what distinguishes a complete listing from one truncated by a dropped connection; both are otherwise well-formed, correctly sorted text. `runRedirectedCommand` is used instead of `ZFS_Utils::runCmd` for the bulk commands, because `runCmd` discards all output on a non-zero exit and `find` exits 1 on a single permission denial. `sortListing` sorts with `env LC_ALL=C TMPDIR=<tempDir> sort` (LC_ALL=C is mandatory — locale collation would disagree with the join's byte comparison) and removes the raw file immediately to bound peak disk. `find | sort` as one pipeline is avoided: FreeBSD `/bin/sh` has no `pipefail`.
  - Phase 4: `mergeJoinListings` streams both sorted listings in **constant memory** regardless of file count, classifying each path via `classifyPair` as `missing`/`extra`/`sizeDiff`/`mtimeDiff`/matched. `readListingRecord` flags a record malformed when its trailing two fields are not both integers — which is exactly what a filename containing a tab or newline produces — and those are counted and surfaced as a report caveat. `inMtimeWindow` applies the `newer`/`older` age band **in the join, not in `find`**: filtering during listing generation would drop a file from only one side whenever its two mtimes straddle the boundary, turning a genuine `mtimeDiff` into a false `missing`. `recordFinding` streams per-finding detail to a TSV in `findingsDir` (opened lazily, so a clean run leaves no file) while keeping only `maxReportFindingsPerDataset` in memory for the report.
  - The active-side resource guards deferred from Stage 3 are now applied via `activeSideWrapper`: `ulimit -v <activeMemoryLimitKB>` joined with `&&` rather than `;` — if the ceiling cannot be installed we deliberately do *not* run the unbounded command on a host with a 5-year uptime requirement — plus `nice` when `niceness` is set.
  - New config key `maxFindingsPerType` (default 100000) stops writing a systemic finding type to the findings file past that count, so a mis-mapped dataset cannot produce a ten-million-line file. Counters stay exact regardless.
  - `parallelListings` is marked NOT YET IMPLEMENTED in the datastructure rather than half-built: forking the two listings is a pure wall-clock optimization whose failure mode (a lost child exit status) overlaps the truncated-listing trap, which is the single most dangerous failure in the design. It waits until the sentinel handling is proven on real hardware.
  - Report now carries real per-dataset counters, plus caveat lines for a partial listing, malformed records, and an active age filter (in which case the `extra` count is explicitly *not* a complete orphan reconciliation).
  - **Tested** (Stage 7 tests for this stage, run now rather than deferred): new `testLibrary/test_mergeJoinListings.pl`, 50 assertions, all passing — covering record parsing and malformed detection, join pairing in both directions, empty listings, size/mtime comparison, `mtimeSlack`, the age band and its boundaries, and malformed-record accounting. It encodes the ordering trap as a regression guard: the same asymmetric scenario yields exactly one `missing` and no `extra` when correctly sorted, but invents a phantom `extra` for a file present on *both* sides when fed `find -s` order. Also exercised end to end against real file trees through the actual `find`/`stat`/`sort`/join pipeline, confirming correct counts and findings including the `Documentation.md` vs `Documentation/index.html` prefix collision and a path containing a space, plus the partial-listing and temp-cleanup paths.

- **Stage 5 (sampling + batched digests).** `sampleDecision`/`selectChecksumSample` decide, in the same join pass Stage 4 already runs, which compared files join the digest sample: `randFile` `0` disables digesting, `1` digests everything, `N>1` on average one in N, with `maxSampleFileSize` exclusion (counted via `sampleSkippedTooLarge`, not silent). The `randFile: 0`/`rand(0)==rand(1)` trap is guarded explicitly — `sampleDecision` returns before ever calling `rand()` when `$n` is falsy — and `cleanupConfig` now rejects a non-integer `randFile` at config time rather than at runtime. `randSeed`, when set, seeds Perl's `rand()` once for the whole run (not per dataset), so the same snapshot reproduces the same sample everywhere.
  - Sample paths are written NUL-delimited (`openSampleFile`) and digested in **one batched command per side** (`buildChecksumCommand` + `xargs -0`), never one round trip per file — the active side's command also goes through Stage 4's `ulimit -v`/`nice` wrapper, and neither side's `ssh` invocation ever carries `-n` (which would silently redirect stdin from `/dev/null` and digest nothing). `digestCommand` supports `sha256`/`md5` (BSD's `-r` reversed-format convention) and `cksum` (its own 3-field format) by name, via a small format table (`digestFormatFor`); an unrecognized command falls back to the `sha256`/`md5` shape.
  - `readChecksumOutput`/`compareChecksums` correlate the two sides **strictly by path**, never by line position — the trap being that a single unreadable file would otherwise misattribute every digest after it into a wall of false `checksumDiff`. A path present on one side only becomes `checksumUnavailable`, distinct from an actual mismatch.
  - A digest collection failure (e.g. the sentinel is missing) is recorded as a report caveat rather than failing the whole dataset — the already-completed size/mtime comparison remains valid on its own; `digestSampledFiles` is a deliberate no-op when nothing was sampled.
  - New config keys `checksumArgsPerCall` (default 500) and `skipSampleOnMismatch` (default 0 — a file already flagged `sizeDiff`/`mtimeDiff` stays eligible for digesting by default, since it is the single most informative file available to digest).
  - Report gained a `sampled/validated/checksumDiff/checksumUnavailable` line per dataset (only shown when `randFile` is nonzero) and shows the exclusion count when `maxSampleFileSize` applies.
  - Kept on Sonnet per the plan, with hand-verification in place of early automated testing (Stage 7 for this stage runs at the end, per the plan's sequencing decision): confirmed `int(rand(0))` is always `0` and that `sampleDecision(0)`/`sampleDecision(1)` return the documented values; confirmed a 1-in-1000 sample rate empirically over 200,000 trials (195 selected); confirmed path-keyed correlation gets the right answer even when the two sides' digest output is in completely different order, with a genuinely missing file correctly attributed as `checksumUnavailable` rather than a false `checksumDiff`; confirmed the built commands never contain `ssh -n`, that the active-side command alone carries the resource wrapper, and that `cksum` omits the `-r` flag. Also ran end to end against a real file tree: a file with **identical size and mtime but different content** (the entire reason this stage exists — the size/mtime comparison alone calls it a match) was correctly caught as `checksumDiff`, and a `chmod 000` file unreadable on the backup side was correctly reported as `checksumUnavailable` with the digest run's own caveat, not a false mismatch.

- **Stage 6 (reporting + exit codes).** `determineExitCode` classifies a completed run as `0` (clean), `1` (findings present), or `3` (one or more datasets could not be fully validated), computed from the same `$pairs`/`$orphans`/`$scrubStates` the report itself renders — so the report's new `Result:` headline and the actual exit code are derived from one classification and can never disagree. `2` (config/usage error) is set via `$config->{exitCodeOverride}` at each of the three `fatalError()` sites that can fire before any dataset has real results (unknown `sessionType`, invalid `randFile`, and a pre-flight failure), plus the earliest config-load bail-out, which now `exit 2`s explicitly instead of a bare `die`.
  - **A pool that comes back from a scrub with actual data errors forces exit 1 regardless of anything else in the run** — `zpool status`'s `healthy` flag (computed since Stage 3 but never actually consulted until now) is a more serious signal than any per-file finding, and skipping it would have been a real gap: `ProblemDefinition.md`'s Phase 2 always said scrub outcome belongs in the report. The pool status lines now also show repaired bytes and a `*** DATA ERRORS DETECTED ***` flag.
  - New `summarizeFindingsByDirectory`: tallies findings by containing directory as they're recorded (one pass, bounded by distinct-directory count, not finding count) and reports the top 10 whenever a dataset has findings — the "a mis-mapped dataset clusters findings under one directory" diagnostic the spec calls for.
  - `cleanup()` now attaches the findings file to the emailed report via `sendReport`'s `$extraArtifacts` (never copied to a report drive, matching that parameter's existing contract from a prior `sendReport` change).
  - A judgment call worth flagging: a pre-flight failure (unreachable `activeHost`, unwritable `tempDir`, a missing required binary) is classified as exit `2`, not `3`, on the reasoning that it prevents the run from starting at all rather than being "a dataset that failed" — no dataset has been attempted yet at that point. This is a reasonable reading of the spec rather than something the spec states explicitly.
  - **Tested end to end** against six scenarios covering every exit code and the two new report features: a clean run (exit `0`, no findings file created — confirms the lazy-open stays lazy), three findings clustered in one directory (exit `1`, `top directories by finding count` correctly attributes all three to it), a missing backup-side dataset (exit `3`, `Result: INCOMPLETE`), an unknown `sessionType` (exit `2`), a pool reporting data errors with otherwise-identical files (exit `1` forced, `*** DATA ERRORS DETECTED ***` shown), and an unreadable `zpool status` (scrub state `unknown`, exit `3`).

- **Stage 7 (unit tests).** `testLibrary/test_selectChecksumSample.pl` (22 assertions), `test_readChecksumOutput.pl` (26), `test_findSharedSnapshot.pl` (18), and `test_resolveSnapshotRoot.pl` (16) — 82 new assertions, plus the 50 already delivered early in Stage 4 (`test_mergeJoinListings.pl`, per the explicit exception recorded in `Documentation/ImplementationPlan.md`). **132 total, all passing.** All follow the project's extract-and-stub convention: the function under test is copied verbatim into the test file (with a comment saying so and a note to update both copies together), since this script has unguarded top-level code and cannot be `require`d.
  - `test_selectChecksumSample.pl`: confirms the `randFile` 0/1 guard directly (`int(rand(0))` really does behave like `rand(1)` in this Perl, and the guard runs before `rand()` is ever called), a statistical sampling-rate check, `maxSampleFileSize`'s boundary (`>`, not `>=`), and that the NUL-delimited sample file's content exactly matches what was selected.
  - `test_readChecksumOutput.pl`: parsing for all three supported `digestCommand` shapes (`sha256`/`md5`'s `-r` format, `cksum`'s 3-field format), `digestFormatFor`'s basename extraction and fallback, and — the important part — `compareChecksums` correlating correctly by path when the two sides' digest output is in completely unrelated order, with an unreadable file on either side producing exactly one `checksumUnavailable` attributed to the right path rather than a positional cascade.
  - `test_findSharedSnapshot.pl`: creation-time ordering beating alphabetical name ordering, `excludeSnap` (including the "all shared candidates excluded" case, distinguished from "nothing shared at all"), an undated snapshot (`creation` undef) being usable as the sole candidate but never beating a dated one, and the same-second tie-break by listing index.
  - `test_resolveSnapshotRoot.pl`: every dataset state (absent, zvol, `mountpoint=none`, `mounted=no`, `mountpoint=legacy` both resolvable and not, the normal case), the `expected`-skip flag's value in each case, and a dataset name that is a string-prefix of another mounted dataset's name *not* cross-matching in `findLegacyMountPoint`'s mount-table regex. **Adds the project's first `ssh` PATH shim in `testLibrary/`** (a fake `ssh` that strips `-o` options and the hostname and runs the remaining command locally) — every prior test only ever exercised the local-host branch of `sshCommand()`; this one exercises the remote-host branch for real, against a "remote" `mount -t zfs` and `findLegacyMountPoint`.
  - Not added in this stage, deliberately: dedicated tests for `determineExitCode`, `expandDatasetPairs`/orphan detection, and `computeMtimeWindow` — `inMtimeWindow` (the function that actually consumes `computeMtimeWindow`'s output) is already covered in `test_mergeJoinListings.pl`, and the other two were exercised thoroughly via the end-to-end scenario testing already recorded in the Stage 3 and Stage 6 entries above. Follows the plan's own priority order rather than attempting exhaustive coverage in one stage.

  - Found by Stage 3's smoke test (against stubbed `zfs`/`zpool`/`ssh`): a real defect in Stage 1's `ZFS_Utils::listSnapshots` — see `../CHANGELOG.md` [1.8.0] "Fixed". Every snapshot record came back with `dataset`/`snap` undefined, which made every pair report "no shared snapshot". Worth noting as a concrete instance of the batched-testing tradeoff: Stage 1 had deliberately left those two functions for Stage 7, so nothing caught it until a caller existed.

- **Stage 8 (documentation).** `README.md` (purpose, file inventory, requirements, basic usage, a cron example built on the `monthly`/`quarterly` session-type profiles, and a callout on `autoscrub`'s active-server-safety implications — meets `CommonProjectStandards/CodingStandards.md` §12's minimums directly, rather than the sibling subprojects' much thinner two-sentence READMEs). `USAGE.md` (every CLI option and all 43 keys in `validateBackup.datastructure`, plus `noScrub`/`logFile`/`lockFile`, which are legitimate config-file keys not present in the sample defaults — cross-checked mechanically against the datastructure file and the real `Getopt::Long` option list, which caught two real documentation gaps: `dryrun` and `noScrub` are genuine config-file/`sessionType`-settable keys distinct from their `--dry-run`/`--no-scrub` CLI spellings, and neither had been given its own row). `Documentation/validateBackup.md` (architecture, a function inventory grouped by phase, the key data-structure shapes, and a condensed pointer to the correctness constraints already detailed in `ProblemDefinition.md`). `Documentation/TESTING.md` (a manual test plan for real FreeBSD hardware, written before any test automation per this project's convention — 12 parts covering setup through cleanup, with Part 3 specifically targeting the BSD command-behavior assumptions that could not be verified from the Linux development machine, since every prior stage's testing used shims that cannot validate real `find`/`stat`/`sort`/`sha256` behavior).
  - No real hostnames anywhere in any of the four documents (checked mechanically); dollar amounts and cron times use placeholder values only.
  - Debugged `Documentation/TESTING.md` against real FreeBSD hardware (Parts 0, 3, 4, 5, 6, 11), restructured Part 4, and corrected several doc-only issues found along the way — including a significant one in Step 0.2 (the "deliberately introduce" corruption/missing/orphan edits were made to the backup's live filesystem *after* its `daily-test1` snapshot had already been taken, so none of them were ever visible to `validateBackup`, which only reads inside `.zfs/snapshot/`; fixed by re-snapshotting after the edits) and a config-key clarification (`newer`/`older`'s `0` is an active, maximally restrictive value, not a disable sentinel like `randFile`'s `0` — now documented in `ProblemDefinition.md` and `USAGE.md`). No `validateBackup` code changed by these. See project memory for specifics.