# validateBackup — Problem Definition ## Summary Validate that a backup server holds a faithful copy of an active server's data, by comparing files inside a ZFS snapshot that exists on both machines. One engine, two schedules, distinguished only by how much of the data gets digested: - **Sampled mode** (default, intended monthly from cron). Every file is checked for existence, size, and mtime; a configurable random sample is digested on both sides and compared. - **Full mode** (`--full`, intended quarterly). The same pipeline with `randFile: 1`, digesting every file. Not a separate implementation — one parameter value. Both use only the FreeBSD base system. The tool is **read-only with one deliberate exception**: when `autoscrub` is enabled and a pool's last scrub is older than `scrubMaxAge`, it starts a scrub and waits for completion before validating. It never writes to, mounts, unmounts, or destroys a dataset or snapshot, and never modifies a ZFS property. Builds on `ZFS_Utils.pm` in the root of this project. Library changes are acceptable so long as they remain backwards compatible with the existing projects that use it (`sneakernet/`, `replicate/`). Note that the reusable surface is smaller than it first appears: `ZFS_Utils.pm` supplies `logMsg`, `loadOrCreateConfig`, `checkConfigKeys`, `sendReport`, `humanReadable`, `humanDuration`, and `fatalError`, but has **no** ssh wrapper, **no** `shellQuote`, **no** dataset or snapshot listing, **no** `zfs get` wrapper, **no** dry-run mechanism, **no** temp-directory handling, and nothing for digests or file comparison. `replicate/replicate` — not `sneakernet` — is the structural template for the script. Some intermediate data may be very large and is stored in a configurable temporary directory on the backup server. ## Active-server safety (a hard constraint, not a preference) The active server is designed for 5+ years of uninterrupted uptime, is in use by end users while validation runs, and **must never be put at risk of memory exhaustion** — recovering from that is not an acceptable outcome. This constrains the design rather than merely informing it: - **Nothing that scales with dataset size may be held in memory on the active side.** The active host runs only `find`, batched `stat`, and a digest command under `xargs`. Each streams; none accumulates a file list. Peak resident size is a few megabytes whether the dataset holds ten thousand files or fifty million. - **All bulk state lives on the backup server**, as files in `tempDir`: the two listings, and `sort`'s spill files. `sort` is additionally capped with an explicit `-S` so it spills to disk rather than growing, and the pre-flight free-space check applies to it. - **Any external command invoked on the active side is wrapped in an explicit address-space limit**, so a tool that misbehaves is killed by the kernel and the dataset is reported as failed, instead of the run becoming the active server's problem. - The validation is read-only I/O against a snapshot, so it competes with production for disk bandwidth but not for correctness. `activeNiceness`/`backupNiceness` and an operator-chosen day/time window are the levers for that; memory is handled structurally, above, rather than by scheduling. ## Topology Runs on the **backup server**, as root. Backup-side commands run locally; active-side commands run via `ssh ''`, always with `BatchMode=yes` (a cron run must never block on a password prompt) and keepalives (a multi-hour `find` must survive an idle-timeout killer). Both servers are FreeBSD; only base-system BSD `find`, `stat`, `sha256`, `sort`, and `xargs` are used. The vocabulary throughout is **active** and **backup** — deliberately not `source`/`target`, which in `sneakernet` denote air-gap roles rather than hosts, and which read as a write destination in a read-only tool. ## Datasets Datasets are named on the active side and mapped to the backup side by rewriting `activePrefix` to `backupPrefix`, with an optional per-dataset override. `recursive` expands a dataset to its children on the active side. `excludeDataset` removes children from that expansion. Datasets present under `backupPrefix` with no active counterpart are reported as orphans. `type=volume` (zvol) datasets are skipped — they contain no files. Datasets with `mountpoint=none`, `mountpoint=legacy` and not mounted, or `mounted=no` are skipped with the reason recorded; the tool will not mount anything to reach them. ## Algorithm **Phase 1 — resolve.** For each dataset pair: list snapshots on both sides, intersect on the **short name** (the part after `@`, since dataset names differ by design), drop names matching `excludeSnap`, and pick the newest. Ordering is by the ZFS `creation` property in epoch seconds (`zfs list -H -p -o name,creation -rt snapshot -r`), **not** by parsing the snapshot name — `creation` is preserved across `zfs send`/`recv`, so both sides agree, and it is never absent, whereas name parsing yields nothing for a hand-made snapshot like `@premigration`. Resolution is **per dataset**, not once per parent tree: a child created after the last replication may have no shared snapshot at all. Snapshot roots are `/.zfs/snapshot//`, verified traversable with a cheap `test -d -r -x` before any long scan. **Phase 2 — scrub check.** For each distinct pool on both sides, parse `zpool status` for the last completed scrub. Flag any pool older than `scrubMaxAge` days. If `autoscrub` is enabled, start a scrub on each flagged pool (both sides in parallel if both are stale), poll until complete or `scrubWaitMax` elapses, then continue to Phase 3. On timeout: log, skip validation, exit non-zero. Scrub outcome (errors found, bytes repaired) appears in the report. **Phase 3 — list.** For each side, `cd` into the snapshot root and emit one tab-delimited `\t\t` record per regular file. Paths are made relative by the `cd` plus `find .`, so both sides produce directly comparable `./sub/file` strings. Both listings are then sorted **on the backup server** with `env LC_ALL=C sort` — one collation implementation, one locale, and `sort` spills to disk where an in-memory Perl sort of ten million records would not fit. **Phase 4 — compare.** Merge-join the two sorted listings, classifying each path as `missing` (active only), `extra` (backup only), `sizeDiff`, `mtimeDiff`, or matched, and accumulating `fileCount` and `totalFileSize` from the active side. Memory is constant in dataset size. During the join, select the digest sample and write the chosen paths to a NUL-delimited file. **Phase 5 — digest.** One batched command per side: `cd && xargs -0 -n sha256 -r`, reading the sample paths from stdin — piped **into ssh's stdin** for the active side, so no scp and no remote temp file. Correlate the two outputs **by path**, never by line position. Equal digests increment `filesValidated`; differing digests are `checksumDiff`; a path missing from one side's output is `checksumUnavailable` (unreadable file), not a mismatch. **Phase 6 — report.** Per-dataset and run totals, then delivery via `ZFS_Utils::sendReport`. ## Full mode (`--full`) `--full` sets `randFile: 1` and nothing else changes: phases 1–6 run exactly as in sampled mode, every file is digested on both sides, and the merge-join compares the results. There is no second code path to write, test, or keep correct, and memory on both hosts stays constant in dataset size. The practical difference is time, not structure — a full pass reads every byte off both pools. That is why it is a quarterly `sessionType` profile rather than the default, and why `digestCommand` matters more here than in sampled mode. ## Decision: full mode is `randFile: 1`, not rsync `rsync --dry-run --checksum` between the two snapshot directories is a correct and appealing full comparison, and was considered as the implementation of full mode. **It was rejected.** The decision and its reasoning are recorded here because the option is attractive enough that it will be proposed again: - **It puts a tree-scaled data structure in RAM on the host that cannot tolerate it.** rsync 3.x uses incremental recursion by default and so does *not* hold the whole file list at once — but that mode is disabled by several ordinary options (`--delete-before`, `--delete-after`, `--prune-empty-dirs`, `--delay-updates`, and historically `-H`), at which point it reverts to a full in-memory list of roughly 100 bytes per file. A `--delete` that resolves the wrong way, on a fifty-million-file dataset, is a gigabyte-scale allocation on a server with a five-year uptime requirement. The base-tools pipeline has no such cliff — there is no configuration of it that accumulates. - **It saves no code.** The usual argument for rsync is that it avoids writing a second full-tree comparison engine. But full mode needs no engine: it is `randFile: 1` on the pipeline sampled mode already requires. rsync saves nothing and adds a dependency outside FreeBSD base. - **It saves no I/O.** With `--checksum` each side reads its own local copy and only digests cross the network — exactly what the batched digest phase already does. Reading every byte off both pools dominates a full pass and is identical either way. What is genuinely given up is hash throughput — rsync 3.2+ can use `--checksum-choice=xxh64`, several times faster than SHA-256, which does become the bottleneck once a pool streams faster than roughly 400 MB/s. Most of that is recoverable inside base by setting `digestCommand`, since the threat model does not require a cryptographic hash. Also given up: comparison of symlinks, hardlinks, ownership, and permissions, none of which are in scope (only regular files are compared). rsync therefore remains available as an opt-in cross-check (`--use-rsync`, **default off**), for an operator who wants a second independent opinion on a specific dataset. When enabled it is invoked snapshot-to-snapshot, never live-to-live, with `--delete-during` stated explicitly rather than relying on bare `--delete`, and wrapped in an address-space limit on the active side per "Active-server safety". Its presence and version are checked in pre-flight so a missing or too-old binary is a clean config error, not a mid-run failure. **Why snapshot-to-snapshot, not live plus a `zfs diff` subtraction.** For the record, since the alternative was considered in detail: comparing `active-live` to `backup-live` and discarding the differences that `zfs diff ` attributes to post-snapshot churn is *logically sound* — anything flagged that `zfs diff` does not explain is real divergence, and the reverse set is benign (a file touched then reverted to identical content). It is rejected because the snapshot is already readable as a directory tree, making the subtraction unnecessary, and the live comparison adds avoidable failure modes: a target that moves for the hours the comparison runs (forcing `zfs diff` to run *after* it so its window is a superset of the churn), absolute-vs-relative path normalization between two tools' output conventions, `zfs diff` being per-dataset while `rsync -a` crosses datasets, directory `M` noise, rename mapping (`R old -> new` versus a transfer plus a delete), and unresolvable deletions that `zfs diff` emits as object-id notation like `/tank/data/<0x1f4a2>`, which can never match a path and would surface as phantom findings. ## Churn context and incremental scoping `zfs diff` has no cross-server capability, so it cannot compare the two machines. It is still worth running **locally on the active side** for two things: 1. **Report context (v1).** One `zfs diff ` per dataset reports how far the validated snapshot lags live — "validated `snapN`, 3 days and 41,000 changed paths behind current." This is real operational signal about whether the validation result is even meaningful, and it is a txg-level delta rather than a tree walk, so it is cheap. 2. **Incremental sample scoping (follow-on phase).** Recording the last successfully validated snapshot per dataset lets `zfs diff ` yield exactly the paths that changed between them. Digesting *those* rather than a blind 1-in-N sample gives far better coverage per byte read, because it targets the data that actually moved — which is where replication faults live. Deferred because it needs durable per-dataset state (a status file, as `sneakernet` keeps), which is a design question of its own; `--scope-changed` is reserved for it. ## Parameters Precedence: CLI over `sessionType` profile over config file over built-in default. **Selection and mapping:** `activeHost`, `activePrefix`, `backupPrefix`, `datasets` (with per-dataset backup override), `recursive`, `excludeDataset`, `excludeSnap`, `snapshot` (pin one name; must exist on both sides). `excludeSnap` is a regex matched against **snapshot names**, not dataset names. Its purpose is to keep automatic snapshots (`hourly`, `frequent`) out of the shared-snapshot candidate pool so validation anchors on a daily, weekly, or monthly snapshot. Dataset filtering is `excludeDataset`, which is a separate parameter. **Age band:** `newer` and `older`, in days, combinable to select a band. Applied to the **active** file's mtime for files present on the active side, and to the backup's for backup-only files. Applied **in the join, not in `find`** — filtering during listing generation would drop a file from only one side's listing when its two mtimes straddle the boundary, turning a genuine `mtimeDiff` into a false `missing`, which is precisely the defect the tool exists to find. An opt-in `filterDuringListing` exists for speed, default off, with that consequence documented. **Sampling:** `randFile` — `0` disables digesting, `1` digests every eligible file, `N>1` digests on average one file in N. `randSeed` makes a run reproducible against the same snapshot. Optional `maxSampleFileSize` excludes very large files from sampling, with the exclusion counted rather than silent. **Digest:** `digestCommand` selects the hash, default `sha256`. **The threat model is bit rot and replication error, not adversarial tampering**, so a cryptographically strong hash is not required and the faster base-system `md5` or `cksum` are legitimate choices on a large pool where hashing, not disk, is the bottleneck. The default stays conservative; the option exists so an operator can trade it deliberately. **Scrub:** `scrubMaxAge`, `autoscrub` (**default 0**), `scrubWaitMax`, `scrubPollInterval`. **Operational:** `tempDir`, `minTempFreeBytes`, `keepTemp`, `findingsDir`, `maxReportFindingsPerDataset`, `mtimeSlack` (default 0 — `zfs recv` preserves mtime exactly, so any difference is real), `parallelListings`, plus the standard `verbosity`, `debug`, `logFile`, and `report` block. Command paths (`findCommand`, `statCommand`, `digestCommand`, `rsyncCommand`) are overridable so an unusual non-interactive `PATH` cannot break a run. **Modes:** `--dry-run` resolves datasets, snapshots, roots, and scrub ages, prints the exact commands that would run, and reads no file data. `--no-scrub` performs the validation but refuses to start a scrub. `--full` is shorthand for `randFile: 1`. `--use-rsync` (default off) enables the optional rsync cross-check, with `rsyncOptions` and `checksumChoice` (passed as `--checksum-choice`; empty by default, since it needs rsync 3.2+ on both ends). `--scope-changed` is reserved for incremental scoping and is not implemented in v1. **Resource guards:** `activeMemoryLimitKB` sets the address-space ceiling applied to commands run on the active side (default deliberately generous but finite — a killed command is a reported failure, an OOM on the active server is not recoverable), `sortMemoryLimit` (default `500M`) becomes `sort -S` on the backup side (the one command in the pipeline that genuinely buffers — `find`/`stat`/the digest commands all stream in bounded batches regardless of file count — so this defaults to a real, non-empty bound rather than a courtesy cap; set to `''` to fall back to `sort`'s own uncontrolled default, not recommended at large file counts), and `activeNiceness`/`backupNiceness` independently apply `nice` to each side's own scanning/digesting/sorting commands so a validation pass yields to production I/O — independent because the two hosts typically have very different quiescence, and an operator tuning one aggressively should not be forced to tune the other the same way. A **PID lock file** prevents overlapping runs; a second invocation finding a live lock logs and exits 0 without emailing. Overlap is not expected at a monthly cadence, but an `autoscrub` run can last many hours and the lock makes that safe. **`sessionType`** allows a named block of pre-set parameters in the config file, selected as `--sessionType `, with any CLI option overriding the profile for that run only. Because the two modes run on different schedules, each is expected to live in its own profile — a `monthly` profile carrying the sampling parameters and a `quarterly` profile setting `full`. That keeps cron entries down to `validateBackup --sessionType quarterly`. ## Report and exit codes Counters always: `fileCount`, `totalFileSize`, `filesCompared`, `filesMatched`, `filesSampled`, `filesValidated`, and a count per error type. A clean dataset prints counters only. A dataset with errors additionally prints the first `maxReportFindingsPerDataset` findings plus a pointer to the findings file, and a "top directories by finding count" line — when a dataset mapping is wrong, findings cluster under one directory and that line diagnoses it instantly instead of burying it under a million undifferentiated `missing` lines. **"Findings" means errors only**; matched files are counted and discarded, which is what keeps memory flat. The report must never claim a clean result from an incomplete scan. An explicit caveat line is emitted whenever a listing was partial, a record was malformed, or an age filter was active — in the last case the `extra` count is age-filtered and is **not** a complete reconciliation of backup orphans. Exit codes: `0` clean, `1` findings present, `2` config/usage error, `3` one or more datasets skipped or failed. Separating 1 from 3 lets cron distinguish "the backup is wrong" from "I could not check" — different problems for different people. ## Failure handling No per-dataset failure is fatal. Each records a status (`ok`/`partial`/`skipped`/`failed`) with a stage and reason, and the run continues: ssh unreachable, dataset absent on one side, no shared snapshot, snapshot root not traversable, `find` permission denials (dataset marked `partial`, listing still used), unreadable sampled files, insufficient temp space. Repeated ssh failures trip a circuit breaker rather than timing out once per dataset for an hour. A hard `fatalError` is reserved for config-level impossibilities, and always routed through `cleanup` so the report still goes out. ## Traps the implementation must respect These are correctness traps that produce **plausible but wrong** results, which is worse than a crash: - **`find -s` is not `sort` order.** `find -s` sorts per-directory; `sort` compares whole path strings. With directory `a/` beside file `a.txt`, they disagree (`.` = 0x2E vs `/` = 0x2F), and a merge-join fed `find -s` output emits bogus `missing`/`extra` pairs wherever a directory name prefixes a sibling file name — `Documentation/` beside `Documentation.md` is enough. Canonical order is `LC_ALL=C sort` of the whole record line, with path first so a plain byte sort *is* a path sort. - **Collation must match on both ends.** `LC_ALL=C` on the `sort`, byte comparison in the join, and no `use locale` / no `:encoding` layers anywhere in the script. Any one of those silently changes the comparison. - **A truncated listing is indistinguishable from a valid one.** An ssh drop mid-scan yields a well-formed, correctly sorted, incomplete file, and the join would then report the remaining active files as nothing and the remaining backup files as `extra`. Every bulk command appends an **exit-status sentinel to stderr**; a missing sentinel fails the dataset. - **`ZFS_Utils::runCmd` discards output on non-zero exit.** `find` exits 1 on a single permission denial and `xargs` exits 123 if any child failed, so `runCmd` would throw away an entire ten-million-line listing over one unreadable file. Use it only for small metadata queries; bulk commands use `system()` with redirection. - **`rand` makes `0` and `1` both mean "everything".** Perl treats `rand(0)` as `rand(1)`, so `int(rand($n)) == 0` is always true for `$n` of 0 or 1 — a config of `randFile: 0` meaning "don't digest" would digest all ten million files. Guard `0` explicitly and validate `randFile` as a non-negative integer at config time. - **`newer`/`older` do *not* follow `randFile`'s "`0` disables it" convention — do not assume they do.** Their disable value is the empty string `''`; `0` is a real, active, maximally restrictive window boundary ("newer/older than right now"), computed the same as any other integer (`computeMtimeWindow`'s check is `ne ''`, not truthiness). Confirmed manually (2026-09-13): setting `newer: 0` does not disable the filter. - **Never correlate digests by position.** `sha256 -q` plus positional matching misattributes every digest after the first unreadable file, producing a wall of false `checksumDiff`. Use `sha256 -r` and match on the path it prints. - **Do not pass `ssh -n`** when piping the sample list to a remote `xargs` — it redirects stdin from `/dev/null` and silently digests nothing. - **Do not pipe `find | sort`.** FreeBSD `/bin/sh` has no `pipefail`, so `find`'s failure becomes invisible. - Filenames containing tab or newline break a line-based format. Records whose trailing two fields are not both integers are counted as malformed and reported. Both sides fragment identically, so the join still pairs them — odd-looking paths, not a flood of false findings. ## Deliverables - `validateBackup` — the script (no extension, per project convention). - `validateBackup.datastructure` — commented default configuration. - `README.md` — brief overview: what the script does and why it was built. - `USAGE.md` — all parameters and configuration options. - `CHANGELOG.md`. - `Documentation/validateBackup.md` — reference documentation. - `Documentation/TESTING.md` — manual test plan (written before any automation, per the convention in `replicate/TESTING_PLAN_PROMPT.md`). - `ZFS_Utils.pm` additions: `shellQuote`, `sshCommand`, `listSnapshots`, `getDatasetProperties` — placed in `@EXPORT_OK`, **not** `@EXPORT`, because `sneakernet` already defines its own `shellQuote` and adding it to the default export would trigger a redefinition warning there. ## To verify on test hardware No test server is available to the AI author, so testing is performed by building a non-destructive test suite which the maintainer syncs to the testing server and runs, reporting results back. These BSD behaviours are assumptions until confirmed there: `stat -f '%N\t%z\t%m'` emits a bare epoch and preserves a literal tab; `sort` honours `TMPDIR` and spills correctly on a large file; `sha256 -r` output format; `.zfs/snapshot/` is traversable with `snapdir=hidden` (so the tool need not change `snapdir`); `zfs list -p -o creation` is epoch seconds and matches across a `send`/`recv` pair; the real `getconf ARG_MAX`; and that `find -x . -type f -exec stat ... {} +` over a tree with an unreadable subdirectory exits 1, writes stderr, and still lists every readable file. **The active-side memory claim must be measured, not assumed** — it is the constraint the design rests on. On the largest available dataset, watch peak RSS of the active-side `find`/`stat`/`xargs`/digest processes (`ps -o rss,command` sampled during a run, or `time -l`) and confirm it stays flat as file count grows rather than tracking it. If `find` or `stat` turns out to accumulate on a very wide directory, that is worth knowing before this runs against production. Also measure digest throughput for `sha256` versus `md5` versus `cksum` on real data, to decide whether `digestCommand` should default to something faster than `sha256` for the quarterly full pass. Only if `--use-rsync` is ever enabled: whether `rsync` is installed on both hosts and at what version (`--checksum-choice` needs 3.2+ on both ends), that `rsync -n -i --checksum` against two read-only snapshot directories reports differences without attempting a write, and its actual peak RSS on the largest dataset with `--delete-during`. ## Future work **Incremental sample scoping** (`--scope-changed`), described above — it needs durable per-dataset state recording the last successfully validated snapshot, which is a design question of its own. **NUL-safe listing** via `vis(1)`/`unvis(1)`, for filenames containing tab or newline. The line-based format cannot represent them; v1 detects and counts such records instead. **Comparing metadata beyond size and mtime** — symlink targets, ownership, permissions, and hardlink structure. Only regular files' existence, size, mtime, and content are in scope for v1. This is the one real capability rsync would have brought for free, and is the most likely reason to revisit it.