# validateBackup — Reference Documentation

This document describes how `validateBackup` is built: its architecture, its function inventory, the internal data structures a maintainer would need to extend it, and the correctness constraints that shaped its design. For what it does and why, start with [ProblemDefinition.md](ProblemDefinition.md) — this document assumes that context and does not repeat its rationale. For every parameter, see [../USAGE.md](../USAGE.md).

## Architecture

The run proceeds through six phases plus a scrub phase, always in this order. Every phase is per-dataset-pair except the scrub phase, which operates on the distinct pools involved across all pairs.

1. **Resolve** (`expandDatasetPairs`, `resolveSnapshotsForPairs`) — turn the configured `datasets` block into concrete active/backup dataset pairs, then resolve each pair's shared snapshot and both snapshot roots.
2. **Scrub** (`runScrubPhase`) — check every involved pool's scrub freshness; start and wait for a scrub only when `autoscrub` is enabled and a pool is genuinely stale.
3. **List** (`generateListing`, `sortListing`) — for each side, emit a sorted `<path>\t<size>\t<mtime>` listing of every regular file under the snapshot root.
4. **Compare** (`mergeJoinListings`, `classifyPair`) — merge-join the two sorted listings in constant memory, classifying every path.
5. **Digest** (`digestSampledFiles`, `collectChecksums`, `compareChecksums`) — batch-digest the sampled paths on both sides and correlate by path.
6. **Report** (`buildResolutionReport`, `determineExitCode`) — render the summary and compute the exit code from the same classification.

Every phase records failures as a status on the affected pair (`skipped`/`failed`/`partial`) rather than dying — see "Failure handling" below. The only exception is a config-level impossibility (bad config, unreachable pre-flight), which exits before any dataset is attempted.

## Function inventory

### Startup and configuration

| Function | Responsibility |
|---|---|
| `initializeLogging` | Apply `verbosity`/console/TTY/log-file settings to `ZFS_Utils` package globals; truncate the log for a fresh run. |
| `applySessionType` | Shallow-overlay a named `sessionType` profile onto `$config`, before CLI overrides are applied. |
| `cleanupConfig` | Apply CLI overrides, fill in every default, validate `randFile`, and push the `datasets` block's global recurse/exclude defaults down into each entry. |
| `parseCommandLineOptions` | `Getopt::Long` wrapper; handles `--help`/`--version`/`--scope-changed` directly. |
| `acquireLock` / `releaseLock` | PID-based lock file — a live PID blocks a run silently; a dead one is reclaimed. |
| `commandExists` | `command -v` check for a required binary. |
| `getFreeDiskBytes` | Parses `df -k` for available bytes on a path. |
| `hostReachable` | A bare `ssh ... true` round trip. |
| `validateRuntimeEnvironment` | Pre-flight: temp/findings directories, required commands, active-host reachability. All read-only. |

### Phase 1 — resolution

| Function | Responsibility |
|---|---|
| `activeDatasetPath` / `mapDatasetName` | Pure string mapping: `activePrefix`/`backupPrefix` plus a per-dataset `backup` override. |
| `describeConfiguredDatasets` | Config-only preview of the active→backup mapping, used by the scaffolding log line and (historically) `--dry-run`'s early output. |
| `expandDatasetPairs` | Fetches `getDatasetProperties` per configured entry per side (recursive where enabled), applies `excludeDataset`, and builds one pair per active-side dataset. Zvols are ruled out here, at the point `type` is already known, with `expected => 1` so they don't count as a real resolution failure. |
| `mapChildDataset` | Maps one recursively-discovered active dataset to its backup counterpart by swapping the parent prefix. |
| `findOrphanBackupDatasets` | Backup-side datasets with no active counterpart, for recursively-expanded entries only. |
| `bucketSnapshotsByDataset` | Splits a flat `listSnapshots` result back out per dataset. |
| `findSharedSnapshot` | Intersects two datasets' snapshots on short name, applies `excludeSnap`, and picks the newest by the ZFS `creation` property (never by parsing the snapshot name). |
| `resolveSnapshotsForPairs` / `resolveOnePairSnapshot` | Batches the snapshot-list queries (one per configured entry per side, not one per child dataset), then resolves each pending pair's snapshot and both roots. |
| `findLegacyMountPoint` | Parses `mount -t zfs` for a `mountpoint=legacy` dataset's real path. Never mounts anything. |
| `resolveSnapshotRoot` | Derives `<mountpoint>/.zfs/snapshot/<snap>`, handling zvol / `mountpoint=none` / `mountpoint=legacy` / `mounted=no`, each as a skip with a reason and an `expected` flag. |
| `snapshotRootUsable` | Cheap `test -d -r -x` before committing to a long scan. |

### Phase 2 — scrub

| Function | Responsibility |
|---|---|
| `parseScrubDate` | Parses a `zpool status` scan-line ctime into epoch seconds (tolerates local-vs-UTC skew — see the function's own comment). |
| `getScrubState` | One `zpool status` call → freshness-relevant fields, including `healthy` (data-error state) and `repaired` bytes. |
| `poolsFromPairs` | Distinct pools across only the **resolved** pairs — deliberately narrows the one mutating operation's scope to what the run actually depends on. |
| `checkScrubFreshness` | Classifies each pool `ok`/`stale`/`never`/`unknown`. `unknown` is never eligible for `autoscrub`. |
| `startScrub` / `waitForScrubs` | Starts a scrub (asynchronous — no forking needed) and polls to completion or `scrubWaitMax`. |
| `runScrubPhase` | Orchestrates the above; enforces every `autoscrub` guard (enabled, not `--no-scrub`, not `--dry-run`, pool genuinely stale, not already running). |

### Phases 3–4 — listing and compare

| Function | Responsibility |
|---|---|
| `activeSideWrapper` | Wraps a command destined for the active host in `ulimit -v`/`nice`, failing closed (`&&`, not `;`) if the limit can't be installed. |
| `buildListingCommand` | The `find -x . -type f -exec stat ...` command plus the exit-status sentinel. |
| `runRedirectedCommand` / `readExitSentinel` | Runs a command with stdout/stderr to files (never `ZFS_Utils::runCmd`, which discards output on non-zero exit) and reads back the sentinel. |
| `generateListing` | Classifies a listing attempt `ok`/`partial`/`failed` from the sentinel and stderr. |
| `sortListing` | `env LC_ALL=C sort`, `TMPDIR` pointed at the dataset's own temp dir, raw file removed on success. |
| `readListingRecord` | Parses one sorted-listing line; flags a record malformed when its trailing two fields aren't both integers. |
| `mergeJoinListings` | The constant-memory streaming join — see "Correctness constraints" below. |
| `classifyPair` | Per-path classification: `missing`/`extra`/`sizeDiff`/`mtimeDiff`/matched, applying the mtime window and (for `both`) triggering `selectChecksumSample`. |
| `datasetTempDir` | Per-pair working directory under `tempDir`. |
| `compareDatasetPair` | Orchestrates one pair end to end: temp dir, both listings, both sorts, the join, then the digest phase. |

### Phase 5 — digest

| Function | Responsibility |
|---|---|
| `sampleDecision` | The `randFile` 0/1/N decision, with the `rand(0)==rand(1)` guard evaluated before `rand()` is ever called. |
| `openSampleFile` / `closeSampleFile` | Lazy NUL-delimited sample file per pair. |
| `selectChecksumSample` | Applies `maxSampleFileSize` then `sampleDecision` for one compared file. |
| `digestFormatFor` | Looks up the output-parsing shape for `sha256`/`md5` (BSD `-r` convention) or `cksum` (its own 3-field format), by basename. |
| `buildChecksumCommand` | The batched `xargs -0 ... digestCommand` command plus sentinel; never adds `ssh -n`. |
| `runCommandWithIO` | Like `runRedirectedCommand`, with stdin also redirected — the sample file, piped through ssh's own stdin forwarding for the active side. |
| `readChecksumOutput` | Parses one side's digest output into a path-keyed hash. |
| `collectChecksums` | Runs and classifies one side's digest pass. |
| `compareChecksums` | Correlates the two sides **strictly by path** — never by line position. |
| `digestSampledFiles` | Orchestrates the digest phase for one pair; a digest failure is a caveat, not a dataset failure. |

### Phase 6 — reporting

| Function | Responsibility |
|---|---|
| `newCounters` | The per-dataset counter set. |
| `computeMtimeWindow` / `inMtimeWindow` | The `newer`/`older` age band, computed once for the whole run. |
| `openFindingsFile` / `closeFindingsFile` | Lazy per-run findings TSV. |
| `recordFinding` | Counts a finding, tallies it by containing directory, and streams it to the findings file (capped by `maxFindingsPerType`) while keeping the first few in memory for the report. |
| `countActiveFile` | Adds one active-side file to the volume counters. |
| `removeRunTempDir` | Cleans up the whole run's temp tree (per-dataset dirs are already removed as each pair finishes). |
| `summarizeFindingsByDirectory` | Top-10 directories by finding count, from the tally `recordFinding` maintains. |
| `determineExitCode` | Classifies the completed run `0`/`1`/`3` from `$pairs`/`$orphans`/`$scrubStates`/`$proceed` — see `USAGE.md`'s exit code table. |
| `buildDatasetSummaryLines` / `buildResolutionReport` | Render the per-dataset and whole-run report body, including every mandatory caveat line. |
| `cleanup` | Single exit path: releases the lock, sends the report (with the findings file attached), and exits with `$config->{exitCodeOverride}` if set. |

## Key data structures

**A dataset pair** (`$pairs` elements), accumulated across phases:

```
{
   datasetKey, activeDataset, backupDataset, activeParent, backupParent, recursive,
   activeMeta, backupMeta, backupTree,        # from getDatasetProperties
   status,       # 'pending' -> 'resolved' -> 'ok'/'partial'/'failed', or 'skipped'/'failed' earlier
   stage, skipReason, expected,               # set on any non-'resolved'/'ok' status
   snapshot, snapshotCandidates, activeRoot, backupRoot,   # set once resolved
   tempDir, listingNotes, counters, reportFindings, dirCounts, durationSeconds,   # set once compared
}
```

**Counters** (`newCounters`): `fileCount`, `totalFileSize`, `filesCompared`, `filesMatched`, `backupOnlyCount`, `filesSampled`, `filesValidated`, `sampleSkippedTooLarge`, `listingWarnings`, `listingMalformed`, and `errorCounts => { missing, extra, sizeDiff, mtimeDiff, checksumDiff, checksumUnavailable }`.

**A scrub state** (`getScrubState`): `pool`, `host`, `side`, `available`, `inProgress`, `neverScrubbed`, `lastScrubEpoch`, `lastScrubText`, `repaired`, `healthy`, `freshness`, and (when applicable) `ageDays`/`note`/`scrubStarted`/`scrubCompleted`.

## Correctness constraints

These are load-bearing and documented at length in the code itself (`mergeJoinListings`, `buildListingCommand`, `sortListing`) and in [ProblemDefinition.md](ProblemDefinition.md)'s "Traps the implementation must respect". Summarized:

- Listings must be sorted with `env LC_ALL=C sort` over the whole record line (path first). `find -s`'s per-directory ordering is a *different* order and silently fabricates `missing`/`extra` pairs wherever a directory name prefixes a sibling file name.
- A listing's completeness is only knowable from its exit-status sentinel — a connection dropped mid-scan produces a well-formed, correctly sorted, *truncated* file, indistinguishable from a complete one by any other means.
- `ZFS_Utils::runCmd` is never used for a bulk command (listing, sort, digest) — it discards all output on a non-zero exit, and `find` exits `1` on a single permission denial.
- `randFile => 0` must disable sampling before `rand()` is ever called, because Perl's `rand(0)` behaves like `rand(1)`.
- Digest correlation is strictly by path, never by line position, so one unreadable file cannot misattribute every digest after it.
- No `use locale` or `:encoding`/`use open` layer appears anywhere in the script — the merge-join's byte comparison must agree exactly with the external `sort`'s `LC_ALL=C` ordering.

## Exit codes and the report

`determineExitCode` and `buildResolutionReport` are computed from the same run state, so the report's `Result:` headline and the process exit code can never disagree — see [../USAGE.md](../USAGE.md)'s exit code table for the full classification.

## Testing

Unit tests live in `../testLibrary/test_*.pl` (`mergeJoinListings`, `selectChecksumSample`, `readChecksumOutput`/`compareChecksums`, `findSharedSnapshot`, `resolveSnapshotRoot`), following the project's extract-and-stub convention. The manual hardware test plan is [TESTING.md](TESTING.md).