# validateBackup — Usage Reference

Every command-line option and configuration key, in one place. See [README.md](README.md) for basic invocation and [Documentation/ProblemDefinition.md](Documentation/ProblemDefinition.md) for the design rationale behind each choice.

## Configuration file

On first run, `validateBackup` generates `validateBackup.conf.yaml` next to the script from `validateBackup.datastructure` (a commented Perl hashref of defaults). Edit the generated YAML file, not the datastructure file, for day-to-day configuration — the datastructure file is only consulted again if the YAML file is deleted.

Precedence, highest first: **command-line option** > **`sessionType` profile** (if `--sessionType NAME` is given) > **config file value** > **built-in default**.

## Command-line options

| Option | Effect |
|---|---|
| `--dry-run`, `-n` | Resolve datasets, snapshots, and snapshot roots; check scrub freshness; print the exact commands that would run. Reads no file data at all. |
| `--no-scrub` | Validate normally, but never start a scrub even if `autoscrub` is enabled in the config. |
| `--full` | Shorthand for `randFile => 1` — digest every eligible file instead of a random sample. Reads every byte on both pools; intended for an occasional (e.g. quarterly) exhaustive pass, not every run. |
| `--use-rsync` | Also run the optional `rsync --checksum` cross-check (see `useRsync` below). Default off. |
| `--scope-changed` | Reserved for a future incremental-scoping release. Not implemented — using this flag exits immediately with an error (exit code `2`). |
| `--sessionType NAME` | Apply the named profile from the config file's `sessionType` block before applying any other command-line option. See "Session types" below. |
| `--verbosity LEVEL`, `-v LEVEL` | Logging verbosity, `0`–`5`. Default `1`. See "Verbosity levels" below. |
| `--debug LEVEL`, `-d LEVEL` | Dumps internal state and exits at a specific point in the run. See "Debug levels" below. Not for normal use. |
| `--help`, `-h` | Print usage and exit. |
| `--version` | Print the script's version and exit. |

## Configuration keys

### Selection and mapping

| Key | Default | Meaning |
|---|---|---|
| `activeHost` | *(required)* | Hostname of the active (source-of-truth) server, reached over ssh. |
| `activePrefix` | `''` | Prepended to each key under `datasets` to form the active-side dataset path. |
| `backupPrefix` | `''` | Prepended to each key under `datasets` to form the backup-side dataset path, unless that dataset sets its own `backup` override. |
| `datasets` | *(required)* | HASHREF. See "The `datasets` block" below. |
| `excludeSnap` | `''` | Regex matched against **snapshot names** (not dataset names) — excludes matching snapshots from the shared-snapshot candidate pool, e.g. `'^(hourly\|frequent)'` to anchor validation on a daily/weekly/monthly snapshot. |
| `snapshot` | `''` | If set, pin validation to this exact snapshot name instead of resolving the newest shared one. Must exist on both sides or the dataset is skipped. |

### The `datasets` block

Keys are **relative dataset paths** — combined with `activePrefix`/`backupPrefix` to form the real active- and backup-side paths. Two scalar keys act as global defaults for every dataset entry, overridable per entry:

```yaml
datasets:
  recursive: 0          # global default: expand into child datasets (1) or not (0)
  excludeDataset: ''    # global default: regex excluding dataset names from recursive expansion
  files/share:           # becomes <activePrefix>/files/share on the active side
    recursive: 1          # overrides the global default for this entry only
    backup: archive/old/files_share   # optional: overrides <backupPrefix>/files/share entirely
```

A dataset entry with no `backup` key maps to `<backupPrefix>/<key>` (or `<key>` alone if `backupPrefix` is empty). Datasets that exist on the backup side with no active-side counterpart are reported as orphans (not acted on).

### Age band

| Key | Default | Meaning |
|---|---|---|
| `newer` | `''` | If set to any value **including `0`** (an integer number of days), only compare files with active-side mtime newer than *(now − N days)* — `0` means "newer than right now," an active and maximally restrictive window, **not** disabled. Only the empty string `''` disables the filter; unlike `randFile`, `0` is not a disable sentinel here. |
| `older` | `''` | Same rule as `newer`: any set value including `0` is active (`0` means "older than right now"). Combinable with `newer` to select a band; `''` disables. |
| `filterDuringListing` | `0` | Opt-in speed optimization. **Leave this off unless you understand the tradeoff**: filtering during the file listing (rather than during comparison) can turn a genuine `mtimeDiff` into a false `missing`, because the two sides' listings would then be filtered independently against possibly-different mtimes for the same file. See `Documentation/ProblemDefinition.md`'s Parameters section. |

### Sampling and digesting

| Key | Default | Meaning |
|---|---|---|
| `randFile` | `1000` | `0` disables digesting entirely. `1` digests every eligible file (equivalent to `--full`). `N>1` digests on average one file in N. |
| `randSeed` | `''` | If set, seeds Perl's `rand()` once for the whole run, making the sample reproducible against the same snapshot. |
| `maxSampleFileSize` | `0` | `0` = no limit. Otherwise, files larger than this many bytes are excluded from sampling (counted separately, not silently dropped). |
| `digestCommand` | `sha256` | `sha256`, `md5`, or `cksum` — all present in the FreeBSD base system. The threat model here is bit rot and replication error, not tampering, so a faster non-cryptographic digest (`cksum`) is a legitimate choice on a very large pool where digesting, not disk I/O, is the bottleneck. |
| `checksumArgsPerCall` | `500` | Number of paths passed to each `xargs`-batched invocation of `digestCommand`. |
| `skipSampleOnMismatch` | `0` | If set, a file already flagged `sizeDiff`/`mtimeDiff` is excluded from digesting. Default off, since that file is the single most informative one available to digest. |

### Scrub and autoscrub

| Key | Default | Meaning |
|---|---|---|
| `scrubMaxAge` | `30` | Days; a pool whose last completed scrub is older than this is flagged `stale`. |
| `autoscrub` | `0` | **Default off — read `Documentation/ProblemDefinition.md`'s "Active-server safety" before enabling.** If set, starts a scrub on a `stale` or `never`-scrubbed pool and waits for it to complete before validating. This is the only operation in the tool that mutates system state. A pool whose scrub state cannot be determined (`unknown`) is never auto-scrubbed regardless of this setting. |
| `scrubWaitMax` | `28800` (8h) | Seconds to wait for a started scrub before giving up and skipping validation for the run. |
| `scrubPollInterval` | `300` (5m) | Seconds between `zpool status` polls while waiting on a scrub. |

### Modes

These back the `--dry-run`/`--no-scrub` command-line flags, but are also legitimate config-file (or `sessionType` profile) keys in their own right — e.g. a profile that always runs dry — using the internal (non-hyphenated) key name shown here, distinct from the CLI flag's own spelling.

| Key | Default | Meaning |
|---|---|---|
| `dryrun` | `0` | Same effect as `--dry-run`. Defaults **off**, unlike `replicate`'s equivalent: `validateBackup` is read-only by design (`autoscrub` is the one exception), so a cron entry should actually validate. |
| `noScrub` | `0` | Same effect as `--no-scrub`. Not present in the sample `validateBackup.datastructure` (it is primarily a CLI convenience), but honored if set in the config file or a `sessionType` profile. |

### Operational

| Key | Default | Meaning |
|---|---|---|
| `tempDir` | `/var/tmp/validateBackup` | Intermediate listings and sort-spill files (backup server only — nothing tree-scaled is ever written on the active side). |
| `minTempFreeBytes` | `5000000000` (5GB) | Skip a dataset rather than risk running out of temp space mid-listing. |
| `keepTemp` | `0` | If set, don't delete a dataset's temp files after it finishes — a debugging aid. |
| `findingsDir` | `<scriptDirectory>/findings` | Full per-finding detail lives here, one file per run; the report itself only carries the first `maxReportFindingsPerDataset` per dataset. |
| `maxReportFindingsPerDataset` | `20` | Findings shown inline in the report before it points at the findings file instead. |
| `maxFindingsPerType` | `100000` | Stop **writing** a given finding type to the findings file past this many **for the whole run** — a mis-mapped dataset can otherwise produce a ten-million-line file. Counters stay exact regardless. |
| `mtimeSlack` | `0` | Seconds of mtime difference to tolerate before flagging `mtimeDiff`. `0` is correct for a `zfs send`/`recv`-based backup, since mtime is preserved exactly. |
| `parallelListings` | `0` | **Not yet implemented.** Reserved for forking the active- and backup-side listings to run concurrently; deferred until the sentinel/truncation handling (see `Documentation/validateBackup.md`) is proven on real hardware, since its failure mode would overlap that trap. |
| `verbosity` | `1` | `0` none, `1` normal, up to `5` very verbose. |
| `debug` | `0` | See "Debug levels" below. |
| `displayLogsOnConsole` | `1` | Also print log messages to the console. |
| `displayLogsOnTTY` | `''` | If set to a TTY device (`/dev/` prefix optional), also write log messages there. |
| `logFile` | `<script path>.log` | Not present in the sample config (computed by default); set explicitly to override. Truncated at the start of each run. |
| `lockFile` | `<script path>.lock` | Not present in the sample config (computed by default); set explicitly to override. PID-based — a live PID blocks a second run (exit `0`, no report sent); a stale one is reclaimed. |
| `report` | *(required — see below)* | HASHREF: `from`, `email`, `subject`, and optionally `targetDrive` (a folder or labeled drive to also copy the report to — see `ZFS_Utils.md`'s `mountDriveByLabel`/`sendReport` for the full shape). |
| `findCommand` / `statCommand` / `rsyncCommand` | `find` / `stat` / `rsync` | Overridable in case an unusual non-interactive `PATH` doesn't have these under their normal names. |

### Resource guards (active-server safety)

| Key | Default | Meaning |
|---|---|---|
| `activeMemoryLimitKB` | `2097152` (2GB) | Address-space ceiling (`ulimit -v`) applied to every command run on the active side. Deliberately generous but finite: a command killed by this limit is a reported dataset failure, which is recoverable — an out-of-memory condition on the active server (built for 5+ years of uninterrupted uptime) is not. |
| `sortMemoryLimit` | `500M` | Passed as `sort -S` on the backup side, bounding `sort`'s in-memory chunk size regardless of file count — it is the one command in the whole pipeline that genuinely buffers, unlike `find`/`stat`/the digest commands, which stream in bounded batches. `500M` is a reasonable general default; raise it if the backup host has RAM to spare and you want fewer merge passes, lower it if the backup host is memory-constrained. Set to `''` to fall back to `sort`'s own uncontrolled default instead — not recommended at large file counts (confirmed on real hardware to reach 1.3GB RSS sorting a mere few-hundred-MB listing with no `-S` set). |
| `activeNiceness` | `0` | `0` = don't `nice`. `1`–`20` applies `nice` to the active side's `find`/`stat`/digest commands (run over ssh) so a validation pass yields to production disk I/O. |
| `backupNiceness` | `0` | Same, independently, for the backup side's own `find`/`stat`/digest/`sort` commands. Kept separate from `activeNiceness` because the backup host is typically far more quiescent than the active one and may need little or no nicing even when the active side needs it aggressively. |

### Optional rsync cross-check

**Default off, and not the primary validation engine** — see `Documentation/ProblemDefinition.md`'s "Decision: full mode is `randFile: 1`, not rsync" for why. Available as a second, independent opinion on a specific dataset if desired.

| Key | Default | Meaning |
|---|---|---|
| `useRsync` | `0` | Enable the cross-check (also settable via `--use-rsync`). |
| `rsyncOptions` | `''` | Extra options appended to the rsync invocation. |
| `checksumChoice` | `''` | Passed as `--checksum-choice` to rsync when set. Needs rsync 3.2+ on both ends. |

### Session types

`sessionType` is a HASHREF of named profiles, each a **flat, shallow overlay** onto the top-level config — a key present in a profile replaces the corresponding top-level key wholesale; it does not deep-merge nested structures. Select one with `--sessionType NAME`; any other command-line option given in the same invocation still overrides the profile.

```yaml
sessionType:
  monthly:
    randFile: 1000
  quarterly:
    randFile: 1        # equivalent to --full
```

## Verbosity levels

`0` none, `1` normal (the cron-appropriate default), up to `5` very verbose (per-command logging). Higher levels increase log volume substantially — reserve above `1` for troubleshooting a specific run, not for routine unattended use.

## Debug levels

`--debug N` dumps a specific piece of internal state via `Data::Dumper` and exits immediately — a development/troubleshooting aid, not a normal operating mode. Not compatible with a real validation run (the process exits before finishing).

| Level | Dumps |
|---|---|
| `1` | The expanded dataset pairs, before snapshot resolution. |
| `2` | The dataset pairs after snapshot and mount-root resolution, plus detected orphan backup datasets. |
| `3` | The per-pool scrub freshness states. |
| `4` | The dataset pairs after comparison (final counters and captured findings). |
| `6` | The accumulated `$errors` array, immediately before the final report is sent. |
| `8` | The pre-flight environment-check results. |
| `9` | The fully-resolved configuration, after CLI/`sessionType`/defaults are applied. |

## Exit codes

| Code | Meaning |
|---|---|
| `0` | Clean — every dataset validated with no findings, no pool issues, nothing skipped. |
| `1` | Findings present — at least one per-file discrepancy, an orphan backup dataset, a stale/never-scrubbed pool with `autoscrub` disabled, or (forced regardless of anything else) a pool that came back from a scrub with actual data errors. |
| `2` | Config or usage error — invalid configuration, an unknown `--sessionType`, an invalid `randFile`, or a pre-flight failure (unreachable `activeHost`, unwritable `tempDir`, a missing required command). The run never reached per-dataset processing. |
| `3` | Incomplete — one or more datasets could not be fully validated (a resolution failure that is not an expected/permanent property of the dataset, a partial listing, a pool with `unknown` scrub state, or a scrub-wait timeout that skipped validation for the run). |

Exit codes `1` and `3` are deliberately distinct so cron/monitoring can tell "the backup is wrong" apart from "the check itself couldn't complete" — different people act on those.