```markdown # sneakernet - Sneakernet replication script Perl script to perform sneakernet replication of ZFS datasets between two servers using an external transport drive. The script is designed for FreeBSD systems and integrates with `ZFS_Utils.pm` for shared helpers (mounting by GPT label, GELI handling, logging, etc.). Version: 1.10.11 License: Simplified BSD (FreeBSD) - see header in `sneakernet` script for full terms. --- **New in v1.10.x:** fixed a silent full-send degradation (a filesystem whose recorded resume snapshot expired before the next run would quietly get a full re-seed instead of an incremental) via a base-selection fallback, a long-TTL status-file anchor, and the `source.targetSnapshotList`/`source.fullSendPolicy` config keys - see "Base-Selection Fallback" below. Also fixed `sendTargetState` silently writing an empty `target_state.txt` on failure, and three bugs affecting error visibility for cleanup/one-shot scripts (a failed script copy is now reported; `runCleanupScripts` no longer returns/duplicates the shared error list; a failed cleanup script no longer triggers an extra "uninitialized value" warning). A `--servername`/`-s` option was added to override hostname-based source/target role detection, for testing both roles from one machine. `transport.encryptionKey` is no longer written to logs/reports in plaintext. **Folder-based transport** - omitting `transport.label` to use `transport.mountPoint` as a plain local directory instead of a labeled physical drive - is now actually reachable (it was dead code until v1.10.3; see "Folder-Based Transport" below) and pairs with `--servername` for exercising both source and target roles from one machine with no physical drive at all. **Folder-based report drive** (v1.10.4, `ZFS_Utils.pm` v1.7.0) extends the same idea to `target.report.targetDrive`: leaving `label` empty with `mountPoint` set lets `sendTargetState` write `target_state.txt` into a plain folder, so it - and the report drive generally - can be tested without a physical labeled drive either. **New in v1.6.0:** sneakernet can now verify each transport file on the source immediately after it is written, catching corruption or a bad/missing IV before the drive is marked ready for the target. The behavior is controlled by `transport.verifyStream` (`off`/`header`/`full`). See "Transport File Verification" below. **New in v1.5.0:** sneakernet now supports a "maintenance mode" flag. If a maintenance flag file is present (as configured), sneakernet will exit before replication begins. This allows administrators to temporarily block replication for maintenance or troubleshooting by creating a flag file on the target server, which may be designed to boot, run replication and shut down. --- ## Summary / Purpose `sneakernet` automates ZFS snapshot export/import using a removable transport disk. On the source server it creates zfs send streams (optionally encrypted) and writes them to files on the transport disk, along with a serial.txt file containing a timestamp to track when the data was created. On the target server it reads those files (optionally decrypts) and pipes them into `zfs receive` to update the target datasets. After successful import, the serial.txt file is removed to mark the transport drive as processed. The script also supports using GELI to protect disks on the target and can build combined GELI keys from a remote binary key and a local hex key (via helpers in `ZFS_Utils.pm`). The script includes integrated month-based cleanup script scheduling functionality (v1.4.0), allowing automated maintenance tasks to be selectively transferred and executed on the target server based on the current month. It also supports one-shot cleanup scripts for ad-hoc maintenance tasks that should only run once. ## Usage Run from the command line. A YAML config file is expected next to the script named `$scriptname.conf.yaml` (the script will create or update it if needed). Basic options: - `--dryrun`, `-n` : run without making destructive changes (no writes) - `--verbosity LEVEL`, `-v LEVEL` : set logging verbosity level (0-5, default: 0) - `--debug LEVEL`, `-d LEVEL` : set debug level (integer) - `--servername NAME`, `-s NAME` : override hostname-based source/target role detection (`determineServerRole`) with `NAME` instead of the real `hostname -s`. Intended for testing both roles from a single machine. - `--help`, `-h` : print help and exit - `--version`, `-V` : print script version and exit Example: ```bash perl sneakernet --dryrun --verbosity 2 # or perl sneakernet --dryrun -v 2 ``` ### Size tracking and validation As of version 1.4.1, sneakernet pre-calculates the size of each dataset replication stream and compares it to historical sizes store in source.historyFile. If the size exceeds the dataset.maxDelta for any dataset, the process is aborted with an error message. This guards against data corruption such as ransomware. The actual calculations are performed by calculating a runningAverage of the source.runningAverageCount most recent entries for the dataset in the source.historyFile file, then calculating if the estimated size is greater than the runningAverage * dataset.maxDelta Additionally, if the size of the replication exceeds the size of the transport media, the process is aborted with an error message. ### Configuration Validation The script automatically validates the configuration file each time it is loaded. If the configuration file doesn't exist, it will be created from the default structure defined in `sneakernet.datastructure`. The validation process: - Loads the configuration file and compares it against the default structure - Reports any missing configuration keys - Reports any required keys that exist but have no value (empty strings, undefined, etc.) - Exits with an error if validation fails Required keys that must have values include: - `datasets` - at least one dataset must be configured - `target.hostname` - target server hostname must be specified - `target.poolname` - target pool name must be specified - `source.hostname` - source server hostname must be specified - `source.poolname` - source pool name must be specified `transport.label` is **conditionally** required rather than unconditionally: either `transport.label` or `transport.mountPoint` must be set. This is checked explicitly right after the rest of the required-key validation (not via the same mechanism, which only supports unconditional requirements), and - unlike the actual mount step - is **not** skipped in `--dryrun`, so a misconfiguration is caught even during a dry run. See "Folder-Based Transport" below. ### Folder-Based Transport **As of v1.10.3.** When `transport.label` is left empty, sneakernet uses `transport.mountPoint` directly as a local directory instead of mounting a labeled physical drive: the directory is created if it doesn't already exist, and no mount/unmount is performed. This is intended for testing - most usefully paired with `--servername`/`-s` to run both the source and target roles from a single machine with no physical transport drive at all, by pointing both roles' configs at the same folder. ```yaml transport: label: '' # empty - activates folder-based transport mountPoint: /tmp/sneakernet-test # used directly as a plain directory ``` ```bash # exercise the source role perl sneakernet --dryrun -s my-source-hostname # exercise the target role against the same folder perl sneakernet --dryrun -s my-target-hostname ``` Notes: - Exactly one of `transport.label` or `transport.mountPoint` must be set - leaving both empty fails validation immediately with `Invalid config file: transport.label or transport.mountPoint must be set`, before any command-line options are even parsed. - When `transport.label` **is** set, it always takes priority and `mountPoint` is treated as where sneakernet mounts that physical drive (its normal, non-folder-based meaning) - folder-based transport only activates when `transport.label` is empty. - This was dead code from when it was first written until v1.10.3: `transport.label` was unconditionally required by config validation, so leaving it empty made the script die before ever reaching the folder-based logic, regardless of `transport.mountPoint`. ### Transport File Verification As of version 1.6.0, sneakernet can verify each transport file on the **source** server immediately after it is written, before `serial.txt` marks the drive as ready for the target. This guards against a silently corrupt write or a missing/garbled IV file that would otherwise only be discovered when the target tries (and fails) to import the data. The behavior is controlled by `transport.verifyStream`, which accepts a string or the matching integer: | Value | Int | Behavior | |---|---|---| | `off` | `0` | No verification (legacy behavior). | | `header` | `1` | **Default value written into a freshly generated config.** Validate the companion `.IV` file (present, hex-only, exactly 16 bytes) when encryption is enabled, then decrypt the leading portion of the file and confirm the ZFS send-stream magic (`DMU_BACKUP_MAGIC`) is present. Fast - reads only a small prefix. | | `full` | `2` | Everything `header` does, plus stream the entire decrypted file through `zfs receive -nF` against the source dataset the stream came from. The `-n` makes this a dry run that changes nothing; it validates the complete stream structure. Reads the whole file, so it is slower. | Notes: - Verification runs only on the source and only for files actually written (it is skipped in `--dryrun`). - A **failed** verification is fatal: sneakernet logs the failure and aborts via `fatalError` before `serial.txt` is created, so a bad drive is never handed to the target. - The header check finds the magic regardless of byte offset and sender endianness, so it works for both full and incremental streams. - The `full` check receives into the **source** dataset (with `-n`, so no changes are made) because the source's snapshot lineage matches the stream exactly, allowing incremental streams to validate cleanly. - When transport encryption is disabled (`transport.encryptionKey` empty), there is no IV to validate and the file is read directly; the magic/stream checks still apply. - If `verifyStream` is missing/empty (e.g. a config that predates this feature) or unrecognized, sneakernet falls back to `full` so the strongest check still runs. Freshly generated configs are written with `header`; set the key explicitly to choose. **Full/base sends are a special case (as of v1.10.6).** A filesystem's first-ever send (no incremental starting point) is a "full/base" stream - `zfs send pool/fs@snap` with no `-I`. The `full` check's receive-into-the-source-dataset trick can **never** validate one of these: its receive target always already has snapshots (it's the source's own live copy), and ZFS refuses to receive a full stream into any dataset with existing snapshots, even as a dry run (`cannot receive new filesystem stream: destination has snapshots` - the same error `target.allowFullOverwrite` handles on the target, but destroying anything here would mean destroying the *source's* live production dataset, which sneakernet will never do). This is a permanent, structural limitation, not an occasional edge case - it happens for every first-time send. sneakernet detects this automatically (`sendIsIncremental()`, which reads the already-built `zfs send` command text) and handles it one of two ways: - **Default** (`transport.verifyFullSendDataset` unset): fall back to a header-only check for that one file. Logged clearly as an expected, permanent limitation, not an error - the header check still confirms the encryption key/IV are correct and the file is a valid, undamaged ZFS stream. - **Opt-in, stronger** (`transport.verifyFullSendDataset` set to a dataset path on the **source's own pool**, e.g. `storage/.sneakernet_verify`): full/base sends get genuinely checksum-verified, same as incrementals, by dry-run receiving into a **never-created scratch child** of that dataset instead of the dataset's own live copy - since the scratch child never has snapshots, the "destination has snapshots" failure cannot occur. The scratch parent is created automatically if it doesn't exist yet (`mountpoint=none`/`canmount=off`, so it never mounts anywhere on the live system). Because the receive is always `-n`, nothing is ever actually written and no cleanup step is needed. ### Full Overwrite of an Existing Target When a dataset is dropped from the source status file (for example, it was temporarily disabled for a run and then re-enabled), the source produces a **full** `zfs send` stream for it. On the target, though, the dataset usually still exists with its old snapshots - and ZFS refuses to apply a full (base-less) stream onto a dataset that already has snapshots, even with `zfs receive -F`: ``` cannot receive new filesystem stream: destination has snapshots (eg. backup/ds@stale) must destroy them to overwrite it ``` `target.allowFullOverwrite` controls how sneakernet handles this on the target: - `0` (default) - log a clear, actionable error and skip that dataset, leaving it untouched. The operator can then destroy the target manually (`zfs destroy -r `) and re-run, or arrange an incremental. - `1` - destroy the existing target (`zfs destroy -r`) and retry the receive once. Safety notes: - The destroy/retry path triggers **only** when a receive fails with the recognizable "destination exists / has snapshots / must destroy" error. Any other receive failure (corrupt stream, decryption error, etc.) never causes a destroy, so a bad stream cannot wipe a good backup. - `zfs destroy -r` is **irreversible** and is not covered by the `stateFile`/`resetSnapshots.pl` rollback (that only restores snapshots created after a checkpoint, not a destroyed dataset). Enable `allowFullOverwrite` only when an automatic full re-sync of the affected dataset is acceptable. ### Stream Compression sneakernet can compress dataset streams with `xz` to reduce the space used on the transport drive. Compression is configured under `transport.compression` and applied on the source **before** encryption - encrypted output is effectively random and will not compress, so the order must be compress-then-encrypt: ``` source: zfs send … | xz | openssl enc … > dataset.xz target: cat dataset.xz | openssl enc -d … | xz -dc | zfs receive -F … ``` Compressed files are written with a `.xz` suffix (the IV sidecar becomes `dataset.xz.IV`). The **target detects the `.xz` suffix per-file and decompresses automatically** - it does not read `transport.compression`. This means a single transport drive may freely mix compressed and uncompressed files, and the target needs no matching configuration. Configuration (`transport.compression`): | Key | Default | Meaning | |---|---|---| | `method` | `off` | `off` disables compression; `xz` enables it. | | `level` | `6` | xz level 0-9. `6` balances ratio and CPU; higher levels cost much more CPU/RAM for little gain on largely-incompressible data. | | `threads` | `0` | xz worker threads: `0` = all cores (`-T0`), positive `N` = that many, `half` = half the detected cores (FreeBSD `hw.ncpu` / Linux `nproc`). | | `nice` | `0` | If greater than 0, run the compressor under `nice -n ` to yield CPU to other work. | Notes: - Requires the `xz` binary on **both** hosts (base system on FreeBSD; standard on Linux). - The default `threads: 0` / `nice: 0` suit a **dedicated** backup source and air-gapped target (the intended deployment). On a shared/production source, prefer `nice` over capping `threads` - the scheduler then lets compression use spare cycles while yielding instantly to other work. Decompression on the target is much lighter than compression. - Compression composes with verification: `verifyTransportFile` decompresses (after decrypting) before checking the ZFS magic / running `zfs receive -nF`. Running with `verifyStream` >= `header` is recommended when compression is on, since `/bin/sh` on FreeBSD has no `pipefail` and a mid-pipe `xz` failure would otherwise be caught only by verification. - The replication size estimate (`zfs send -Pv`) is the **uncompressed** size, so `checkSizeAgainstTransport` is conservative - it may refuse a stream that would actually fit once compressed, but it will never overfill the drive. ## Dependencies ### System Requirements - FreeBSD or Linux system with ZFS support - FreeBSD system utilities: `geli`, `zfs`, `zpool`, `geom`, `gpart`, `mount`, `/usr/sbin/sendmail` - `openssl` (transport encryption) and `xz` (only when `transport.compression.method` is `xz`) - Perl 5.10 or higher ### Perl Modules **Core Modules (included with Perl):** - `strict`, `warnings` - basic pragma modules - `FindBin` - locate directory of original script - `Getopt::Long` - command-line option parsing - `File::Basename` - file path manipulation - `Data::Dumper` - data structure serialization (for debugging) - `Exporter` - module import/export (used by ZFS_Utils.pm) - `POSIX` - POSIX standard functions (used by ZFS_Utils.pm for date formatting) - `File::Path` - directory creation (used by ZFS_Utils.pm) **Required CPAN Modules:** - `YAML::Tiny` or `YAML` - YAML configuration file parsing (at least one required) **Optional CPAN Modules:** - `JSON::XS` or `JSON::PP` or `JSON` - JSON configuration support (fallback if YAML not available) **Shared Module:** - `ZFS_Utils.pm` (included in repository) - provides `loadConfig`, `mountDriveByLabel`, `logMsg`, and other utility functions ### Installation on Various Operating Systems Below are example commands to install the required Perl modules and system dependencies for sneakernet on common operating systems. Adjust as needed for your environment. **FreeBSD:** ```bash pkg install p5-YAML-Tiny # Optional JSON support: pkg install p5-JSON-XS ``` **Debian/Ubuntu:** ```bash apt install libyaml-tiny-perl libyaml-libyaml-perl # Optional JSON support: apt install libjson-xs-perl ``` **RedHat/CentOS/Fedora:** ```bash dnf install perl-YAML-Tiny perl-YAML-LibYAML # or for older systems: yum install perl-YAML-Tiny perl-YAML-LibYAML # Optional JSON support: dnf install perl-JSON-XS # or: yum install perl-JSON-XS ``` **Using CPAN (any system):** ```bash cpan YAML::Tiny # Optional JSON support: cpan JSON::XS ``` **Note:** Core Perl modules (`strict`, `warnings`, `FindBin`, `File::Basename`, `Data::Dumper`, `Getopt::Long`, `Exporter`, `POSIX`, `File::Path`) are included with standard Perl installations and do not require separate installation. ## Configuration Files ### sneakernet.datastructure The `sneakernet.datastructure` file contains the default configuration structure as a Perl hash reference. This file is used to automatically generate the YAML configuration file (`sneakernet.conf.yaml`) when it doesn't exist. **Purpose:** - Separates default configuration data from script code - Makes it easier to modify default settings without editing the main script - Provides a template for the YAML configuration file **Format:** The file is a Perl data structure that returns a hash reference when evaluated with `do()`. It contains all the default configuration keys, values, and structure that will be written to the YAML file. **Location:** Must be located in the same directory as the `sneakernet` script, named `sneakernet.datastructure`. **Variable Interpolation:** The configuration can include special variables (like ``) that are automatically interpolated by the `interpolateConfig()` function (from ZFS_Utils module) before the YAML file is created. Any string with a substring containing will have that tested against a hash of values and replaced with the value of $values->{varname}, thus `/sneakernet.log` will be replaced with the value of `$values->{scriptDirectory}`. Following are are available in the system. * => Directory of the sneakernet script * => Full path to the sneakernet script, including scriptname **Example Structure:** ```perl { 'dryrun' => 0, 'verbosity' => 1, 'logFile' => '/sneakernet.log', 'source' => { 'hostname' => 'source-server', 'poolname' => 'pool', }, 'target' => { 'hostname' => 'target-server', 'poolname' => 'backup', }, 'transport' => { 'label' => 'sneakernet', 'fstype' => 'ufs', 'mountPoint' => '/mnt/sneakernet', 'datasetDir' => 'datasets', 'serialFile' => 'serial.txt', }, 'datasets' => { 'dataset1' => { 'source' => 'pool', 'target' => 'backup', 'dataset' => 'dataset1', }, }, } ``` ### Configuration File (sneakernet.conf.yaml) The YAML configuration file that is automatically created from `sneakernet.datastructure` if it doesn't exist. This is the file you should edit to customize the behavior of the script for your environment. **Note** If you would prefer not to manually edit a YAML file, we have created a utility, configFileEditor, which assists you in editing a configuration file of this type. It is available via svn export http://svn.dailydata.net/svn/perlutils/trunk. It can also assist in merging changes if the datastructure file updates in future versions. ### Configuration File Structure When you run `sneakernet`, it attempts to `loadConfig($scriptFullPath.conf.yaml)`. If the file doesn't exist, the script: 1. Loads the default configuration from `sneakernet.datastructure` (a Perl data structure file) 2. Interpolates variables (like script paths) into the configuration using `interpolateConfig()` 3. Creates the YAML configuration file using `makeConfig()` from `ZFS_Utils` 4. Validates the configuration against required keys This separation of code and configuration data makes it easier to customize and maintain the default configuration without modifying the script itself. ### Configuration Keys The following documents the important keys, types and defaults. Top-level keys - `dryrun` (bool) - default: `0` - if true, actions that change state are not executed. - `verbosity` (int) - default: `1` - controls logging verbosity. - `debug` (int) - default: `0` - enables debug breakpoints (see “Debug Levels”). - `statusFile` (string) - path to status file used by source server to track last replicated snapshots. On each run the previous file is backed up to `statusFile.YYYY-MM-DD_HH.MM.SS` (see `statusFileBackups`), and entries for datasets not processed this run (for example, ones temporarily disabled in config) are carried over so they are not re-sent as full streams when re-enabled. - `statusFileBackups` (int) - default: `5` - how many timestamped backups of the status file to retain. Each run renames the prior status file to `statusFile.YYYY-MM-DD_HH.MM.SS` and then prunes the oldest beyond this count. `0` keeps none; a negative value keeps all (no pruning). - `logFile` (string) - path to runtime log file. - `displayLogsOnConsole` (bool) - default: `1` - if true, log messages are printed to console (STDOUT) in addition to the log file. - `displayLogsOnTTY` (string) - default: `''` - if set to a TTY device path (e.g., `/dev/ttyv1`), log messages are also written to that TTY device. Useful for headless systems to display logs on physical console. `source` (hash) - `hostname` (string) - hostname of the source server (used to detect running role) - `poolname` (string) - zpool name on the source (default: `pool`) - `runningAverageCount` (int) - Number of historical values to average for size increase estimates - `historyFile` (string) - name of file to store historical replication data - `targetSnapshotList` (string) - default: `''` - path to an operator-supplied `zfs list -rt snap` inventory of the *target* (for example, a copy of `target.stateFileName`/`target_state.txt` brought back from the target). When set and the file exists and is readable, it is used **in place of** `statusFile` to determine what the target already has, giving `makeReplicateCommands` an exact/confirmed base match instead of only ever being able to infer one from source-side snapshot dates. This is the supported recovery path when a recorded resume point has expired or was pruned - see "Base-Selection Fallback" below. Leave empty (the default) to always use `statusFile`. - `fullSendPolicy` (string) - default: `'warn'` - one of `allow`\|`warn`\|`skip`\|`abort`. Governs what happens when a filesystem *had* a recorded resume point but no base at all (exact or inferred) can be found for it, so only a full send would satisfy it. `allow`: send it, minimal logging. `warn`: send it, but log loudly and include it in the emailed report's `OVERVIEW`. `skip`: exclude just that filesystem from this run rather than sending an unrequested full stream - other filesystems and datasets still replicate normally. `abort`: fail the entire run. A filesystem with **no** recorded resume point at all (a genuine first send, e.g. a newly added dataset) is never subject to this policy under any setting. - `cleanUpScriptsDir` (string) - default: `''` - absolute path on source server where cleanup scripts are located. If configured, scripts from this directory will be encrypted and copied to the transport drive at `target.cleanUpScriptsDir` for execution on the target server after replication. - `oneShotCleanup` (string) - default: `''` - optional directory for one-time cleanup scripts. Scripts in this directory are copied to the transport drive and then deleted from the source after successful copy. Useful for ad-hoc maintenance tasks that should only run once. - `cleanupScriptSchedule` (hash) - default: `{}` - optional month-based scheduling for cleanup scripts. Maps script basenames to arrays of month numbers (1-12) when each script should be transferred and executed. Scripts without schedule entries will not be copied. See "Cleanup Script Scheduling" section for details. - `report` (hash) - `email` (string) - email to send the report to - `subject` (string) - optional subject - `targetDrive` (hash) - `fstype` (string) - filesystem type of report drive (ufs/msdos) - `checkInterval` (int) - polling interval (seconds) - `waitTimeout` (int) - default: `300` - how long to wait for the report drive to appear (seconds) - `label` (string) - GPT label of the report drive. **As of v1.7.0:** leave empty (with `mountPoint` set) to use folder-based mode instead of a physical labeled drive - see `mountPoint` below and `mountDriveByLabel()` in `ZFS_Utils.md`. - `mountPoint` (string) - where to mount the labeled drive (default: `/mnt/label`). If `label` is empty, this is instead used directly as a local folder (created if missing, no mount/unmount) - useful for testing without a physical report drive. `target` (hash) - `hostname` (string) - hostname of the target server - `poolname` (string) - zpool name on the target (default: `backup`) - `maintenanceMode` (hash) - configuration for maintenance mode flag(s): - `flags` (hash) - keys specify where to look (e.g., 'local', 'transport', etc.), values are the filenames to check. If any flag file is found, sneakernet will exit before replication. Example: `flags: { local: '/tmp/maintenance.flag', transport: 'flags/maint.flag' }`. In the latter case, will look for transport mount path/flags/maint.flag. - `checkPeriod` (int, optional) - seconds to wait between checks (default: `15`) - `timeout` (int, optional) - maximum seconds to wait for maintenance mode to clear (default: `300`) - `stateFileName` (string) - Not to be confused with `stateFile`, this is the name of a file containing a complete `zfs list -rt snap` inventory of the target pool, written to `target.report.targetDrive` (if defined) by `sendTargetState()` on every target run, prefixed with a self-describing header (creation time, hostname, sneakernet version, SVN revision). If replication on the source ever reports it cannot find a common snapshot to resume from, copy this file to the source and point `source.targetSnapshotList` at it - this is the supported recovery path, not a manual `statusFile` replacement. The same file is also attached to the target's emailed report, if configured, via `sendReport`'s `$extraArtifacts` parameter, so it can reach the source faster than waiting for the report drive to physically travel back. As of 1.10.1 the inventory is read into memory and written by perl rather than redirected inside the command string, and `sendTargetState` verifies it actually got snapshot lines before writing: if the `zfs list` fails (most often because the pool is not imported when `cleanup` runs) no file is written at all, any previous copy on the report drive is left intact, and the reason appears in the report's OVERVIEW. Before 1.10.1 that case silently produced a **0-byte** file while logging success. **As of v1.10.4:** `target.report.targetDrive` no longer requires a physical labeled drive - if `label` is empty and `mountPoint` is set, `target_state.txt` is written directly into `mountPoint` as a plain folder (see `target.report.targetDrive.mountPoint` above and `mountDriveByLabel()`'s folder-based fallback in `ZFS_Utils.md`, v1.7.0). - `stateFile` (string) - base path for timestamped state files that capture snapshot inventory before updates (default: `$scriptDirectory/states/targetState`). Used with `createStateFile()` to enable rollback capability via `resetSnapshots.pl`. - `cleanUpScriptsDir` (string) - default: `''` - relative path on transport drive containing encrypted Perl scripts to run after successful target update (e.g., 'cleanup_scripts'). Scripts are decrypted using transport encryption settings and executed via `eval()`. Useful for automated cleanup tasks, notifications, or custom post-processing. - `shutdownAfterReplication` (bool) - default: `0` - if true, attempt to shutdown after completion - `allowFullOverwrite` (bool) - default: `0` - controls what happens when a **full** stream cannot be received because the target dataset already has snapshots (see "Full Overwrite of an Existing Target"). When `0`, sneakernet logs an actionable error and leaves the dataset untouched. When `1`, it destroys the existing target (`zfs destroy -r`) and retries the receive once. **Destructive and irreversible** - the destroy is not recoverable via `resetSnapshots.pl`. - `geli` (hash) - when present, instructs the script to decrypt/mount GELI-protected pool(s): - `secureKey` (hash) - `label` (string) - GPT label of the key disk (default: `replica`) - `fstype` (string) - filesystem of key disk (default: `ufs`) - `checkInterval` (int) - polling interval for key disk - `waitTimeout` (int) - how long to wait for the key disk - `keyfile` (string) - filename of the remote binary key on the key disk (default: `geli.key`) - `localKey` (string) - 64-hex-character 256-bit key string or path to file containing hex - `target` (string) - path where the combined keyfile should be written (e.g. `/media/geli.key`) - `poolname` (string) - pool name to import on target - `diskList` (array) - optional list of device names to try (e.g. `['da0','da1']`) - `report` (hash) - if configured, sends a report after replication (see "Logging & Reports") - `email` (string) - email to send the report to - `subject` (string) - optional subject (auto-generated if empty) - `targetDrive` (hash) - this is also where `sendTargetState()` writes `target_state.txt` (see `stateFileName` above), not just the emailed/copied report text - `fstype` (string) - default: `msdos` - filesystem type of report drive - `label` (string) - GPT or msdos label of the report drive. **As of v1.7.0:** leave empty (with `mountPoint` set) to use folder-based mode instead of a physical labeled drive. - `mountPoint` (string) - where to mount the labeled drive (default: `/mnt/label`). If `label` is empty, this is instead used directly as a local folder (created if missing, no mount/unmount) - the only way to test `sendTargetState()`'s actual file-write/header behavior without a real labeled physical drive, since it is otherwise skipped entirely when no drive is configured. `transport` (hash) - `label` (string) - GPT label of the transport drive (default: `sneakernet`) - `fstype` (string) - filesystem type for mounting (default: `ufs`) - `mountPoint` (string) - target mount point (default in sample: `/mnt/sneakernet`) - `pathSubstitution` (string) - separator used when converting dataset paths to filenames on transport media (default: `.`). Example: `pool/data/child` becomes `pool.data.child` when set to `.`. - `datasetDir` (string) - subdirectory on transport disk for storing dataset files (default: `datasets`). Source creates this directory and writes dataset files to it; target reads from this path. - `serialFile` (string) - name of the serial file on transport disk to track processing status (default: `serial.txt`). Source writes timestamp to this file; target checks for it and removes it after successful import. - `timeout` (int) - how long to wait for the transport device to appear (seconds) - `checkInterval` (int) - polling interval when waiting for the device (seconds) - `encryptionKey` (string) - hex key used by `openssl enc -aes-256-cbc` for transport encryption. Generate with: `openssl rand 32 | xxd -p | tr -d '\n'` (or, if `xxd` isn't available: `openssl rand 32 | perl -0777 -ne 'print unpack("H*", $_)'`). IVs are automatically generated randomly for each encrypted file. - `verifyStream` (string|int) - generated default: `header` - controls whether each transport file is verified on the source immediately after it is written (see "Transport File Verification"). Accepts a string (`off`, `header`, `full`) or the matching integer (`0`, `1`, `2`). A failed verification aborts replication. If the key is missing/empty (for example a config file that predates this feature) or holds an unrecognized value (logged), sneakernet falls back to `full` - the strongest check - so it fails safe toward maximum verification rather than silently weakening or disabling it. - `verifyFullSendDataset` (string) - default: `''` (empty) - only relevant when `verifyStream` is `full`. A full/base send (a filesystem's first-ever send) can never pass the normal `full` check (see "Transport File Verification"); left empty, it falls back to a header-only check instead. Set to a dataset path on the **source's own pool** (e.g. `storage/.sneakernet_verify`) to instead genuinely verify full/base sends too, via a never-created scratch child of this dataset (created automatically if missing). - `compression` (hash) - optional `xz` compression of dataset streams, applied before encryption (see "Stream Compression"). Disabled when the key is absent or `method` is `off`. Sub-keys: - `method` (string) - default: `off` - `off` to disable, `xz` to enable. - `level` (int) - default: `6` - xz compression level (0-9). - `threads` (int|string) - default: `0` - xz worker threads: `0` = all cores (`-T0`), a positive integer = that many, `half` = half the detected cores. - `nice` (int) - default: `0` - if greater than 0, run the compressor under `nice -n `. `datasets` (hash) - Keys are logical dataset names (user-defined blocks). Each dataset object contains: - `source` - parent or root dataset on the source (string) - `target` - parent or root dataset on the target (string) - `dataset` - dataset name. Must not contain `transport.pathSubstitution` or the mapping between dataset paths and transport filenames becomes ambiguous. - `maxDelta` - The maximum amount of change expected between runs (float). 2.5 would allow a 250% increase in size Validation behavior: - If a configured dataset name contains `transport.pathSubstitution` (for example `disk.0` when `pathSubstitution` is `.`), sneakernet logs an error, skips that dataset, and continues processing the remaining datasets. Example minimal YAML snippet (derived from the script defaults): ```yaml dryrun: 0 logFile: /path/to/sneakernet.log source: hostname: source-host poolname: pool target: hostname: target-host poolname: backup geli: secureKey: label: replica keyfile: geli.key localKey: e98c66...bc9c target: /media/geli.key transport: label: sneakernet mountPoint: /mnt/sneakernet datasets: files_share: source: pool target: backup dataset: files_share ``` ## Debug Levels The `debug` setting is a simple breakpoint mechanism intended for troubleshooting. It causes the script to exit at specific checkpoints, optionally dumping state. If `debug` is not set or is `0`, no debug breakpoints are triggered. **Active breakpoints in this version:** - `0` - Normal execution (no debug breakpoints). - `1-4` - Undefined - `5` - Exit after size estimation and transport free-space validation, before any replication commands are executed (source-side only). - `6` - Exit in target mode after mountGeliForTarget done - `7-9` - Undefined - `10` - Dump full configuration via `Data::Dumper` and exit immediately after config load/validation (before CLI overrides and logging initialization). These are deliberate early exits for inspection only. Use `verbosity` for runtime logging detail. ## Cleanup Script Scheduling Starting with version 1.4.0, sneakernet includes integrated month-based scheduling for cleanup scripts, eliminating the need for the separate `setCleanupScripts` utility. This feature allows you to configure which maintenance scripts should be transferred and executed based on the current month. ### How It Works 1. **Month-Based Filtering**: On the source server, sneakernet examines each script in `source.cleanUpScriptsDir` 2. **Schedule Matching**: Compares the script basename against the `source.cleanupScriptSchedule` configuration 3. **Selective Transfer**: Only scripts scheduled for the current month (1-12) are encrypted and copied to transport 4. **Target Execution**: The target server receives and executes only the scheduled scripts ### Configuration Add a `cleanupScriptSchedule` section to your `source` configuration block in `sneakernet.conf.yaml`: ```yaml source: hostname: source-server poolname: pool cleanUpScriptsDir: /path/to/cleanup/scripts cleanupScriptSchedule: cleanSnaps: # Every month - 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 - 11 - 12 runSmart: # Semi-annually (January, July) - 1 - 7 scrubZFS: # Quarterly (offset from trimZFS) - 2 - 5 - 8 - 11 trimZFS: # Quarterly - 3 - 6 - 9 - 12 ``` **Important:** Scripts without schedule entries will NOT be copied. Only explicitly scheduled scripts are transferred. ### Configuration Format - **Keys**: Script basenames (filenames without path) - **Values**: YAML block lists of month numbers (1 = January, 12 = December), one `- N` per line as shown above. Do not use inline `[1, 2, 3]` flow style — it only parses correctly when `YAML::XS` is installed. `ZFS_Utils::loadConfig` falls back to `YAML::Tiny` when `YAML::XS` is absent, and `YAML::Tiny` silently mis-parses flow-style sequences as a plain string, which makes `sneakernet` fail with a `strict refs` error. - **Omitted scripts**: Scripts not listed in the schedule are never copied ### Example Schedules **Monthly maintenance:** ```yaml cleanupScriptSchedule: cleanSnaps: - 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 - 11 - 12 ``` **Quarterly tasks (staggered):** ```yaml cleanupScriptSchedule: scrubZFS: # Q1, Q2, Q3, Q4 first month - 1 - 4 - 7 - 10 trimZFS: # Q1, Q2, Q3, Q4 second month - 2 - 5 - 8 - 11 checkups: # Q1, Q2, Q3, Q4 third month - 3 - 6 - 9 - 12 ``` **Semi-annual tasks:** ```yaml cleanupScriptSchedule: runSmart: # January and July - 1 - 7 deepScrub: # June and December - 6 - 12 ``` ### Logging With verbosity level 2 or higher, sneakernet will log: - Current month - Which scripts match the current month's schedule - Which scripts are being skipped (not scheduled for this month) - Total count of scripts before and after filtering Example log output (verbosity >= 2): ``` filterCleanupScriptsByMonth: Current month is 1 filterCleanupScriptsByMonth: Including zpoolStats (scheduled for month 1) filterCleanupScriptsByMonth: Including runSmart (scheduled for month 1) filterCleanupScriptsByMonth: Skipping scrubZFS (not scheduled for month 1, runs in months: 2,5,8,11) filterCleanupScriptsByMonth: Filtered 5 scripts down to 2 for current month copyCleanupScripts: After filtering, 2 script(s) scheduled for this month ``` ### Available Sample Scripts The `../cleanupScripts` directory contains ready-to-use maintenance scripts: - **zpoolStats**: Collects ZFS pool statistics (typically run monthly) - **transportStats**: Gathers transport disk statistics (typically run monthly) - **runSmart**: Performs sequential SMART tests on all detected drives - **scrubZFS**: Manages ZFS pool scrubbing operations - **trimZFS**: Performs TRIM operations on ZFS pools for SSD optimization - **cleanSnaps**: Cleans up old snapshots based on information embedded in snapshot names See [cleanupScripts/README.md](../cleanupScripts/README.md) for details on each script. ### Cleanup Script Requirements Scripts should be: 1. **Perl scripts**: They will be executed via `eval` on the target 2. **Self-contained**: Include all necessary logic and error handling 3. **Idempotent**: Safe to run multiple times 4. **Return results**: Can return status via `return` statement Example script structure: ```perl #!/usr/bin/env perl use strict; use warnings; # Initialize results my @results; # Perform maintenance task push @results, "Starting maintenance task..."; # Check disk space my $df_output = `df -h /backup`; push @results, "Disk usage:", $df_output; # Return results (will be logged by sneakernet) return join("\n", @results); ``` ### Migration from setCleanupScripts If you were previously using the `setCleanupScripts` utility: 1. Copy your script schedule from `setCleanupScripts.yaml` to `sneakernet.conf.yaml` 2. Add the schedule under `source.cleanupScriptSchedule` 3. Remove any cron jobs running `setCleanupScripts` 4. Sneakernet will now handle scheduling automatically Example migration: **Old setCleanupScripts.yaml:** ```yaml sourcePath: ../cleanupScripts destinationPath: ./cleanupScripts scripts: runSmart: [1, 7] scrubZFS: [2, 5, 8, 11] trimZFS: [3, 6, 9, 12] ``` **New sneakernet.conf.yaml:** ```yaml source: cleanUpScriptsDir: /path/to/cleanupScripts # Was sourcePath cleanupScriptSchedule: # New in v1.4.0 runSmart: [1, 7] scrubZFS: [2, 5, 8, 11] trimZFS: [3, 6, 9, 12] ``` ## Functions (script-level / documented) The following functions are defined inside `sneakernet` and are documented here. Many helpers used by the script are provided by `ZFS_Utils.pm` (imported at the top of the script). ### Initialization Functions The main program initialization has been refactored into dedicated subroutines for improved clarity and maintainability (v1.3.1): - `loadOrCreateConfig($programDefinition)` - Loads or creates the YAML configuration file. - Behavior: attempts to load existing config file using `loadConfig()`. If file doesn't exist, loads default configuration structure from `sneakernet.datastructure`, interpolates variables using `interpolateConfig()`, creates YAML file using `makeConfig()`, then loads the newly created file. - Arguments: `$programDefinition` is hashref containing script paths and metadata (`scriptDirectory`, `scriptFullPath`, `configFileName`). - Returns: configuration HASHREF. - Dies on: datastructure file errors, config creation failure, validation failure. - Use case: centralizes config loading/creation logic, making main program flow more readable. - `parseCommandLineOptions()` - Parses command-line arguments and handles help/version requests. - Behavior: uses `GetOptions()` to parse `--dryrun`, `--verbosity LEVEL`, `--debug LEVEL`, `--servername NAME`, `--version`, and `--help` flags. `--verbosity` accepts an integer argument (0-5). Displays help text and exits for `--help`. Displays version and exits for `--version`. Dies with error message if invalid options provided. - Arguments: none (reads from `@ARGV`). - Returns: HASHREF of parsed options (`{dryrun => 0|1, verbosity => N, debug => N, servername => STRING}`) - only keys the caller actually passed are present. - Side effects: may exit program with status 0 (for help/version) or status 2 (for invalid options). - Use case: separates CLI parsing logic from main program flow. - `initializeLogging($config, $programDefinition)` - Initializes the ZFS_Utils logging subsystem with configuration values. - Behavior: sets `$ZFS_Utils::verboseLoggingLevel`, `$ZFS_Utils::displayLogsOnConsole`, and `$ZFS_Utils::displayLogsOnTTY` from config. Sets default values for `statusFile` and `logFile` if not already configured. Deletes old log file if it exists (one run per log file). - Arguments: `$config` is configuration hashref, `$programDefinition` contains script paths. - Returns: nothing (void). Modifies package globals in ZFS_Utils namespace. - Side effects: deletes existing log file at `$ZFS_Utils::logFileName`. - Use case: centralizes logging setup, ensuring consistent initialization. - `determineServerRole($config)` - Identifies whether this server is running as source, target, or unknown. - Behavior: executes `hostname -s` to get short hostname, compares against `$config->{source}->{hostname}` and `$config->{target}->{hostname}`. - Arguments: `$config` is configuration hashref with source and target hostname definitions. - Returns: string - `'source'`, `'target'`, or `'unknown'`. - Use case: determines which replication workflow to execute, enables single script for both source and target servers. ### Core Replication Functions - `getStatusFile($filename)` - Returns: ARRAYREF of status lines (snapshot names). Reads `$filename` if present; returns empty arrayref and logs an informational message if file missing or unreadable. - `writeStatusFile($filename, $statusList, $config)` - Writes the provided ARRAYREF of status lines to `$filename`. - Behavior: if a file already exists, renames it to `$filename.YYYY-MM-DD_HH.MM.SS` (a timestamped backup) and calls `pruneStatusBackups()` to retain only `config.statusFileBackups` of them. Then writes the new lines (one per line). Dies on failure to back up or write. - `pruneStatusBackups($filename, $config)` - Removes old timestamped status-file backups (`$filename.`), keeping the most recent `config.statusFileBackups` (default 5). `0` keeps none; a negative value disables pruning (keeps all). Because the timestamp format sorts lexically in chronological order, the newest are identified by a reverse string sort. Logs, but does not die, if a backup cannot be removed. - `mergeStatusLists($oldList, $newList)` - Merges the freshly built status list with the previous one so datasets not processed this run are not lost. - Behavior: keeps every entry from `$newList` (these win for filesystems sent this run), then appends entries from `$oldList` only for filesystems not present in `$newList` at all. Keys on the filesystem portion (text before the first `@`); keeps up to **2** entries per filesystem (not just 1, as of v1.10.0), since `makeReplicateCommands` may emit a second "long-TTL anchor" line alongside the real resume point (see "Base-Selection Fallback" above). - Returns: ARRAYREF of merged status lines (new entries first, carried-over entries after). - Use case: a dataset temporarily disabled (commented out / removed from `datasets`) for one run keeps its last-sent snapshot in the status file, avoiding a full re-send when it is re-enabled. - `dirnameToFileName($string, $delimiter='/', $substitution='.')` - Utility to turn dataset-like strings into filename-safe strings. Example: `pool/fs/sub` -> `pool.fs.sub`. - `buildEncryptionPipeline($config, $outputPath)` - Helper function to build OpenSSL encryption pipeline for transport encryption. - Behavior: generates a random IV using `randomBytes(IVLENGTH)`, writes IV to companion `.IV` file at `${outputPath}.IV`, returns OpenSSL encryption command string suitable for piping. If no encryption key is configured, returns empty string. - `randomBytes()` reads `/dev/urandom`; if that is unavailable it falls back to Perl's `rand()`, which is **not** cryptographically secure and produces a predictable IV. **As of v1.10.2** this fallback logs a loud warning (`randomBytes: WARNING: ... falling back to rand() - generated IV is NOT cryptographically random`) instead of happening silently. - Arguments: `$config` is configuration hashref containing `transport.encryptionKey`, `$outputPath` is the full path where encrypted data will be written (IV file uses same path with `.IV` suffix). - Returns: string containing OpenSSL encryption pipeline (e.g., `| openssl enc -aes-256-cbc -K -iv `) or empty string if encryption disabled. - Use case: eliminates code duplication by centralizing encryption logic used by `doSourceReplication()` and `copyCleanupScripts()`. - `buildDecryptionPipeline($config, $inputPath)` - Helper function to build OpenSSL decryption pipeline for transport decryption. - Behavior: reads IV from companion `.IV` file at `${inputPath}.IV`, returns OpenSSL decryption command string suitable for piping. If no encryption key is configured or IV file doesn't exist, returns empty string. - Arguments: `$config` is configuration hashref containing `transport.encryptionKey`, `$inputPath` is the full path of encrypted data file (IV file expected at same path with `.IV` suffix). - Returns: string containing OpenSSL decryption pipeline (e.g., `| openssl enc -d -aes-256-cbc -K -iv `) or empty string if decryption not needed. - Use case: eliminates code duplication by centralizing decryption logic used by `processTransportFiles()` and `executeCleanupScript()`. - `buildCompressionPipeline($config)` - Builds the compression segment of the source pipeline, inserted before encryption (see "Stream Compression"). - Behavior: reads `transport.compression`; returns an `xz` pipeline fragment (e.g. ` | xz -zc -6 -T0`, optionally prefixed with `nice -n `) when `method` is `xz`, or an empty string when compression is absent/`off`/misconfigured. - Arguments: `$config` is the configuration hashref. - Returns: pipeline fragment string, or empty string. Companion helpers `resolveCompressionThreads()` (maps `0`/`N`/`half` to a `-T` value) and `cpuCount()` (cross-platform core count) support it. - `shellQuote($string)` - Utility to single-quote a string for safe interpolation into a `/bin/sh` command line (embedded single quotes are escaped). Used when building the verification pipelines so file and dataset names containing spaces or shell metacharacters are handled safely. - `verifyModeFromConfig($config)` - Normalizes `transport.verifyStream` into a mode constant. - Behavior: accepts a string (`off`/`header`/`full`, case-insensitive) or an integer (`0`/`1`/`2`). Missing/empty values fall back to `full` (the strongest check, so a pre-existing config without the key is still fully verified); an unrecognized value is logged and also falls back to `full`. - Returns: one of the internal constants `VERIFY_OFF` (0), `VERIFY_HEADER` (1), or `VERIFY_FULL` (2). - `verifyTransportFile($config, $encFile, $receiveTarget, $mode)` - Verifies a single transport file just written by `doSourceReplication` (see "Transport File Verification"). - Behavior: when encryption is enabled, reads and validates the companion IV file (`$encFile.IV` must be exactly 16 bytes of hex). For `VERIFY_HEADER`, decrypts a leading prefix (via `openssl`, or reads directly when encryption is off) and searches for the ZFS send-stream magic. For `VERIFY_FULL`, additionally pipes the entire decrypted stream through `zfs receive -nF $receiveTarget` (a dry run). - Arguments: `$config` is the configuration hashref; `$encFile` is the full path to the written file; `$receiveTarget` is the ZFS dataset used as the dry-run receive target (the source dataset the stream came from, or a scratch dataset for a full/base send - see `sendIsIncremental()`); `$mode` is `VERIFY_HEADER` or `VERIFY_FULL`. - Returns: `1` if the file passes the requested checks, `0` otherwise (with details logged). Callers treat `0` as fatal. - **Limitation:** only works when `$receiveTarget` already has the stream's "from" snapshot (an incremental send) or is a scratch dataset guaranteed to have no children. It can never work for a full/base send received into the source's own copy of that dataset, since that copy always already has snapshots. - `sendIsIncremental($command)` (as of v1.10.6) - Detects whether a generated `zfs send` command (as returned by `ZFS_Utils::makeReplicateCommands`) is an incremental send or a full/base send, by checking for a standalone `-I` token. `makeReplicateCommands` only ever produces four fixed command shapes (plain/recursive, each full or incremental), so this is a complete and exact test. - Arguments: `$command` - a `zfs send ...` command string. - Returns: true if incremental, false if full/base. - `ensureScratchVerifyDataset($config, $dataset)` (as of v1.10.6) - Ensures the scratch dataset named by `transport.verifyFullSendDataset` exists, creating it (`zfs create -p -o mountpoint=none -o canmount=off`) if not. Called at most once per source run regardless of how many full/base sends occur. - Arguments: `$config` is the configuration hashref (used for `fatalError`'s cleanup path); `$dataset` is the scratch dataset's full path. - Returns: nothing; calls `fatalError` (does not return) if the dataset doesn't exist and can't be created. - `doSourceReplication($config, $statusList)` - Performs replication on the source server. - Behavior: if `source.targetSnapshotList` is configured and the file exists and is readable, it is used in place of `$statusList` for the remainder of this call (logged at verbosity 1 with the file's line count and modification time). Before listing snapshots for each dataset, confirms the source dataset actually exists (`ZFS_Utils::datasetExists()`) - a destroyed-and-not-recreated dataset is skipped with a message pushed onto the error summary rather than silently producing nothing to send (see "Destroyed Source Dataset" below). Enumerates source snapshots, builds zfs send commands with `makeReplicateCommands` (from `ZFS_Utils`) by passing the dataset name, source parent path, target parent path, `source.fullSendPolicy`, and a warnings-collector arrayref. `makeReplicateCommands` intelligently filters snapshots by matching the full parent+dataset path to avoid false matches with similarly-named datasets in different locations, then determines whether to use recursive or per-filesystem sends and incremental or full sends based on snapshot availability - including a fallback base-selection pass when a recorded resume point no longer exists on the source (see "Base-Selection Fallback" below). Every fallback/no-base decision returned in the warnings arrayref is logged and, except for `kind => 'confirmed'`/`'no_base_allowed'`, pushed onto the run's error summary so it appears in the emailed report's `OVERVIEW`. The commands are optionally piped through `xz` (if `transport.compression` is enabled) and then `openssl enc` (if transport encryption key is set), and writes send streams to files on the transport mount point (with a `.xz` suffix when compressed). Honors `$config->{dryrun}`. - `checkSizeAgainstTransport($sizeBytes, $mountPoint)` - Checks if the estimated replication stream size (in bytes) will fit on the transport drive mounted at the given mount point. Uses `df` to determine available space and logs the result. Returns true if there is enough space, false otherwise. Used in `doSourceReplication` to prevent overfilling the transport drive. - `estimateReplicationStreamSize($command, $cmd_label)` - Estimates the size in bytes of a ZFS send stream for a given zfs send command by parsing the output of `zfs send -Pv ...`. Used to pre-calculate the size of each replication stream before running the actual send. - `validateSizeEstimateHistory($historyFile, $runningAverageCount, $maxDelta, $cmdLabel, $sizeEstimate)` - Validates a new ZFS send stream size estimate against historical averages for a dataset/command. Appends the new entry to the history file and returns 1 if within the allowed delta, 0 otherwise. Used to detect unexpected size increases before replication. ### Base-Selection Fallback (v1.10.0 / ZFS_Utils.pm v1.5.0) Previously, if a filesystem's recorded resume point (the last snapshot named in `statusFile`) no longer existed on the source - for example because retention pruned it before the next run happened - `makeReplicateCommands` had no fallback and silently generated a full (non-incremental) send instead. For a large, long-lived dataset this can mean an unexpected multi-terabyte re-seed where only a small incremental delta had actually changed, invisible below verbosity 4. `makeReplicateCommands` now tries, in order: 1. **Exact match** (unchanged): the recorded snapshot still exists on the source. 2. **Confirmed fallback**: the recorded snapshot is gone, but a newer source snapshot dated at or before the recorded resume point is also present in the recorded target snapshot set (the status file, or - far more reliably - `source.targetSnapshotList`). This is a verified match. 3. **Inferred fallback**: no candidate is confirmed present on the target, but the newest source snapshot dated at or before the recorded resume point is used anyway, on the assumption that the prior send used `-I` (which ships every intermediate snapshot) and both sides age out on identical name-encoded retention rules. This is an *assumption the source cannot verify* - if wrong, `zfs receive` on the target will refuse the stream (non-destructive; costs one wasted drive trip). Every inferred substitution is logged and included in the emailed report. 4. **No base found**: governed by `source.fullSendPolicy` (see above). To make step 2 (confirmed) the normal case instead of the exception, `makeReplicateCommands` also records a second "long-TTL anchor" line per filesystem in the status file - the newest snapshot in the same send whose retention suffix is roughly 3 months or longer - alongside the real resume point. Because it was part of a confirmed send, a future run can match it exactly even after the primary (always shortest-retention) resume point has expired. `mergeStatusLists` keeps up to 2 entries per filesystem to preserve this anchor across runs (previously 1). The most reliable option remains `source.targetSnapshotList`: pointing it at a real inventory from the target (see `target.stateFileName`) turns every fallback into a confirmed match, with no assumption about target state at all. ### Destroyed Source Dataset (v1.10.8) This is a different, earlier failure mode than the base-selection fallback above: that fallback handles a *snapshot* going missing from an otherwise-intact source dataset. This section covers the *entire dataset* being destroyed on the source (and not recreated) while it remains listed under `datasets:` in the config. Before v1.10.8, `zfs list -rt snap` failing because the dataset no longer exists was indistinguishable from the dataset genuinely having zero snapshots right now - both cases produced an empty snapshot list, `makeReplicateCommands` generated no commands, and `doSourceReplication` logged only a generic `"Nothing to do for $dataset"` line at verbosity ≥ 1, with nothing pushed to the run's error summary. A destroyed source dataset would therefore silently stop replicating forever - its target copy simply stops receiving updates - with no signal in a default (verbosity 0) run's report. `doSourceReplication` now checks `ZFS_Utils::datasetExists()` for each configured dataset's source path *before* attempting to list its snapshots. If the dataset does not exist, that dataset is skipped for this run and a message naming it is pushed onto the error summary, so it appears in the emailed report's `OVERVIEW` on every run until resolved - either by recreating the dataset (if the destruction was accidental) or by removing its entry from `datasets:` (if it is no longer needed). This check is independent of `source.fullSendPolicy`, which governs a different situation: a source dataset that *does* exist but has no snapshot in common with what the target last recorded. ### Replication Stream Size Pre-Check (v1.4.1) Starting with version 1.4.1, sneakernet performs a global pre-check for all datasets before any replication begins. The process is structured in three distinct phases: 1. **Command Generation:** - For every dataset, all required ZFS send commands are generated and collected. No replication is started at this stage. 2. **Size Estimation and Validation:** - For each generated command, the script estimates the ZFS send stream size using `estimateReplicationStreamSize`. - Each estimate is validated against historical averages using `validateSizeEstimateHistory` to detect anomalies or unexpected increases (e.g., due to ransomware or corruption). - The total estimated size for all commands across all datasets is summed. - The script uses `checkSizeAgainstTransport` to ensure the total estimated size will fit on the transport drive. - If any estimate fails validation or the total size exceeds available space, **no replication is performed** and the process aborts with a clear error message. 3. **Replication Execution:** - Only if all checks pass, the script proceeds to run all previously generated commands, writing replication streams to the transport drive. This three-phase approach ensures that all size and space checks are completed before any data transfer begins, preventing partial replication and providing early warning if a dataset's replication size increases unexpectedly. This design improves reliability and helps guard against data corruption or unexpected storage usage. - Returns: `$newStatus` ARRAYREF of updated status lines. - `cleanup($config, $message, $errors)` - Performs final cleanup and reporting actions. - Behavior: builds a provenance line (hostname, role, sneakernet version, `getWorkingCopyRevision`) and prepends it to the report - **as of v1.10.0**, this appears on every report so version drift between source and target is visible without having to think to check. Logs disk usage and zpool list, attempts to unmount the transport drive, sends report via `sendReport`, and optionally shuts down the machine if configured. Honors `dryrun`. - **As of v1.10.7:** the provenance line also includes total elapsed run time (e.g. `Host: hostname (source) sneakernet v1.10.7 svn revision: 160 elapsed: 1h 2m 5s`), formatted by `ZFS_Utils::humanDuration()` from `$config->{scriptStartTime}` (recorded once, right after the configuration is confirmed valid, in "main program starts here"). Shows `elapsed: unknown` if `scriptStartTime` was never set, which should not happen in a normal run. - **As of v1.4.3:** When running in target mode, if a report drive is configured, sneakernet will create a file on the report drive listing all current snapshots on the target. This file (named per `config.target.stateFileName`) is useful for diagnostics, manual recovery, or syncing with the source server. - **As of v1.10.0:** that same file is also passed to `sendReport` as an extra artifact, so it is attached to the emailed report (if `target.report.email` is configured) in addition to being written to the report drive - see "Base-Selection Fallback" above. - **As of v1.10.1:** in target mode, `sendTargetState($config, $errors)` is called **before** the OVERVIEW section is assembled from `$errors`, specifically so that a failed target-state write (e.g. `zfs list` failing because the pool isn't imported) appears in the report instead of only in the log. Assembling the OVERVIEW first, as earlier versions did, would silently drop that failure from the report. - **As of v1.10.4:** the report drive `sendTargetState` writes to no longer has to be a real physical drive - `target.report.targetDrive.label` can be left empty with `mountPoint` set to use a plain local folder instead (see `stateFileName` above and `mountDriveByLabel()`'s folder-based fallback in `ZFS_Utils.md`, v1.7.0). - `captureBaselineSnapshots($config)` - Helper function for Phase 1 of target replication: captures baseline snapshot state. - Behavior: queries ZFS for all existing snapshots on each configured dataset, uses `getLatestSnapshots()` to identify the most recent snapshot per filesystem, stores results in a hashref. - Arguments: `$config` is configuration hashref containing dataset definitions. - Returns: hashref mapping filesystem names to their latest snapshot names (e.g., `{'pool/dataset' => 'snapshot@2026-01-17'}`). - Use case: provides baseline for before/after comparison reports, separated from processing logic for clearer code structure. - `processTransportFiles($config, $originalSnaps)` - Helper function for Phase 2 of target replication: processes and imports dataset files from transport. - Behavior: reads all regular files from transport directory (excluding `.IV` files), converts filenames to dataset paths via `fullDatasetName()`, verifies target datasets exist (creates if missing), builds the decryption prefix with `buildDecryptionPipeline()`, and delegates the receive (and full-overwrite handling) to `receiveTransportStream()`. - Arguments: `$config` is configuration hashref, `$originalSnaps` is hashref from `captureBaselineSnapshots()` (used to track newly created datasets). - Returns: nothing (void). Honors `$config->{dryrun}`. Logs detailed progress at various verbosity levels. - Use case: separates file processing logic from state capture and reporting for better maintainability. - `receiveTransportStream($config, $cmdPrefix, $targetDataset)` - Runs `$cmdPrefix | zfs receive -F $targetDataset` and handles the case where a full stream cannot be applied to an existing dataset (see "Full Overwrite of an Existing Target"). - Behavior: captures stderr (via a local `$ZFS_Utils::merge_stderr`) so the failure can be classified. On success returns immediately. If the receive fails specifically with a "destination exists / has snapshots / must destroy" error and `target.allowFullOverwrite` is true, it runs `zfs destroy -r $targetDataset` and retries the receive once; if `allowFullOverwrite` is false it returns an actionable error without destroying anything. Any other failure is returned as-is and never triggers a destroy. - Arguments: `$config` (reads `target.allowFullOverwrite`), `$cmdPrefix` (the `cat file | [decrypt]` portion), `$targetDataset` (full target path). - Returns: `($ok, $errorText)` - `$ok` is 1 on success, 0 on failure with `$errorText` describing why. - `generateComparisonReport($config, $originalSnaps)` - Helper function for Phase 3 of target replication: generates before/after comparison report. - Behavior: re-queries ZFS for current snapshots on each dataset, uses `snapShotReport()` to generate comparison showing CONSISTENT/ADDED/UNCHANGED flags and snapshot counts, logs results and appends to report message. - Arguments: `$config` is configuration hashref, `$originalSnaps` is baseline snapshot state from `captureBaselineSnapshots()`. - Returns: string containing formatted report text showing changes for each filesystem. - Use case: separates reporting logic from processing for clearer code structure. - `updateTarget($config)` - Performs replication on the target server by orchestrating three helper functions. - Behavior: calls `captureBaselineSnapshots()` to capture baseline state, calls `processTransportFiles()` to import datasets from transport, calls `generateComparisonReport()` to generate before/after comparison. - Arguments: `$config` is configuration hashref. - Returns: nothing (void). Honors `$config->{dryrun}`. - Note: refactored in v1.2.9 from monolithic 88-line function into 3 focused helpers following Single Responsibility Principle. - `fullDatasetName($datasetList, $myName, $process='target')` - Resolves dataset names by looking them up in the configuration and prepending the appropriate parent path. - Behavior: handles both direct matches (e.g., "dataset1") and hierarchical children (e.g., "dataset1/child/grandchild") by progressively walking the path to find the longest matching prefix in config. - Arguments: `$datasetList` is the datasets hashref from config, `$myName` is the name to resolve (may contain slashes), `$process` selects 'source' or 'target' parent path. - Returns: full dataset path (string) or empty string if no match found. - Example: `fullDatasetName({dataset1 => {target => 'backup'}}, 'dataset1/child', 'target')` returns `'backup/dataset1/child'`. - `createStateFile($filename, $dataset, $message)` - Creates a timestamped state file capturing current snapshot inventory for rollback capability. - Behavior: appends timestamp to `$filename` (format: YYYY-MM-DD_HH.MM.SS), creates parent directory if needed, writes header with rollback instructions and user message (automatically commented), queries ZFS for all current snapshots on `$dataset` (including children), writes complete snapshot list to file. This file can be used with `../testLibrary/resetSnapshots.pl` to roll back any snapshots created after this point. - Arguments: `$filename` is base path (defaults to `$scriptDirectory/states/targetState`), `$dataset` is ZFS dataset to inventory, `$message` is optional description. - Returns nothing (void). Logs creation at verbosity >= 1. - `prepareTargetDirectory($targetDir, $config)` - Helper function to prepare target directory on transport drive for cleanup scripts. - Behavior: checks if target directory exists, creates it if missing (honors dryrun), logs creation at verbosity >= 2. - Arguments: `$targetDir` is full path to target directory on transport, `$config` is configuration hashref. - Returns: nothing (void). - Use case: separates directory setup logic from main script copying workflow. - `removeObsoleteScripts($sourceFiles, $targetDir, $config)` - Helper function to remove obsolete scripts from transport drive that no longer exist in source. - Behavior: reads all files from target directory, compares against source file list, removes files that exist on transport but not in source directory. - Arguments: `$sourceFiles` is arrayref of filenames from source directory, `$targetDir` is full path to target directory on transport, `$config` is configuration hashref. - Returns: nothing (void). Honors `$config->{dryrun}`. Logs removal operations. - Use case: keeps transport drive clean by removing outdated cleanup scripts automatically. - `filterCleanupScriptsByMonth($files, $config)` - Filters cleanup scripts based on current month and configured schedule. - Behavior: extracts current month (1-12), checks each file's basename against `$config->{source}->{cleanupScriptSchedule}`, includes only scripts scheduled for current month. Scripts without schedule entries are excluded. Returns all files unchanged if no schedule configured. - Arguments: `$files` is ARRAYREF of full file paths to filter, `$config` is configuration hashref with optional `source.cleanupScriptSchedule` mapping basenames to month arrays. - Returns: ARRAYREF of filtered file paths that should run this month. - Use case: enables month-based scheduling of maintenance tasks without external utilities. Integrated in v1.4.0 from setCleanupScripts functionality. - Example schedule: `{runSmart => [1,7], scrubZFS => [2,5,8,11], cleanSnaps => [1..12]}` - Logs filtering details at verbosity >= 2. - `copyCleanupScripts($config)` - Orchestrates copying and encryption of cleanup scripts from source server to transport drive. - Behavior: validates source directory exists, calls `filterCleanupScriptsByMonth()` to select scripts for current month (v1.4.0+), adds any one-shot scripts from `source.oneShotCleanup` directory (v1.4.0+), calls `prepareTargetDirectory()` to create target directory, calls `removeObsoleteScripts()` to clean up old files, calls `encryptAndCopyScript()` for each script. One-shot scripts are deleted from source after successful copy. - Arguments: `$config` is the configuration hashref. - Returns nothing (void). Respects `$config->{dryrun}` mode. - Use case: enables source server to deploy cleanup scripts to transport drive for execution on target server after replication. One-shot scripts allow for ad-hoc maintenance tasks that should only run once. - Month-based filtering eliminates need for separate setCleanupScripts utility (v1.4.0+) - Called automatically during source replication workflow if `source.cleanUpScriptsDir` is configured. - Note: refactored in v1.2.9 from monolithic 143-line function into 3 focused helpers following Extract Method pattern. Enhanced in v1.4.0 with integrated scheduling and one-shot script support. - `encryptAndCopyScript($config, $scriptFile, $targetDir, $isOneShot)` - Helper function to encrypt and copy a single script from source to transport. - Behavior: reads source file content, encrypts using `buildEncryptionPipeline()` if encryption enabled, writes encrypted content to target directory. If `$isOneShot` is true, deletes source file after successful copy. - Arguments: `$config` is configuration hashref, `$scriptFile` is full path to source script, `$targetDir` is target directory path on transport, `$isOneShot` is boolean flag indicating if script should be deleted after copy. - Returns: nothing (void). Honors `$config->{dryrun}`. Logs detailed progress. - **As of v1.10.2:** a copy/encrypt failure is also pushed onto the shared `$errors` (not just logged), so it reaches the emailed report's OVERVIEW instead of being visible only in the log. This matters most for a one-shot script (config rotation, code upgrade) - previously, a one-shot that silently failed to reach the transport drive looked identical to a clean run. - Use case: encapsulates single-file encryption and copy logic. One-shot scripts are automatically deleted enabling single-use maintenance tasks. - `executeCleanupScript($config, $scriptFile)` - Helper function to execute a single cleanup script from transport drive. - Behavior: reads encrypted script file, decrypts using `buildDecryptionPipeline()` if encryption enabled, executes decrypted content as Perl code using `eval()`, captures and logs output or errors. The `eval` compiles in `runCleanupScripts`'s (and ultimately the whole script's) lexical scope, so a cleanup script can see `$config`, `$programDefinition`, and every `ZFS_Utils`-exported function with no `use` needed. - Arguments: `$config` is configuration hashref, `$scriptFile` is the full path to the encrypted script file on the transport drive. - Returns: `($result, $errors)` where `$result` is the script output and `$errors` is a string of error messages (empty string if no errors). - Scripts can return either just `$result` or `($result, $errorString)` for two-value return. - Honors `$config->{dryrun}`. Logs execution details and any errors. - Use case: encapsulates single-script execution logic with proper error handling. - `runCleanupScripts($config)` - Orchestrates execution of post-replication cleanup scripts from the transport drive on target server. - Behavior: validates scripts directory exists on transport, reads all files (excluding `.IV` files), calls `executeCleanupScript()` for each script file. - Arguments: `$config` is the configuration hashref. - Returns: arrayref of error strings collected from **this run's** cleanup scripts only (empty arrayref if no errors). - **Bug fixed in v1.10.2:** this function previously had no trailing `return` on its normal path (yielding `undef`) and returned the shared `$errors` global itself on its early-return paths. The caller then spliced that return value back into `$errors`, which - on the early-return paths - spliced the array into itself, duplicating every pre-existing error in the report. It now always returns a fresh, local arrayref; errors are still pushed onto the shared `$errors` directly as they occur (so `cleanup()` sees them), with no splice needed at the call site. - Respects `$config->{dryrun}` mode. - Use case: automated cleanup tasks, email notifications, custom post-processing after replication completes. - Security note: scripts are executed with same privileges as the sneakernet script, so ensure scripts are trusted and properly secured during transport. - Note: refactored in v1.2.9 from monolithic 104-line function into focused orchestrator + helper following cyclomatic complexity reduction. - `updateTarget($config)` - Reads files from the transport disk and feeds them into `zfs receive` to update target datasets. - Behavior: detects file->dataset mapping via the filename (uses `dirnameToFileName` reversal), optionally decrypts with `openssl enc -d` if encryption was used, and calls `zfs receive -F` for each file. ## Main flow summary 1. Load YAML config using `ZFS_Utils::loadConfig`. If the config file doesn't exist, create it from the default structure in `sneakernet.datastructure` using `ZFS_Utils::makeConfig()` and `ZFS_Utils::interpolateConfig()`. Validate configuration against required keys automatically. 2. Parse CLI flags (dryrun, verbosity level, help, version). CLI flags override config when present. 3. Determine whether the running host matches `source.hostname` or `target.hostname` and set `runningAs` accordingly. 4. Mount the transport drive (fatal error if not found) using `ZFS_Utils::mountDriveByLabel`. 5. If running as source: - Check if serial.txt already exists on transport drive; if so, exit with warning (prevents duplicate writes). - Clean transport directory (non-recursive), produce send streams (optionally `xz`-compressed, then encrypted) and write to files on transport drive (compressed files get a `.xz` suffix). - If `transport.verifyStream` is `header` or `full`, verify each written file (and its IV) via `verifyTransportFile()`; abort replication on failure. - Merge the new status with the previous one (`mergeStatusLists()`) to preserve entries for datasets not processed this run, then write the status file - backing up the prior copy with a timestamp and pruning old backups per `statusFileBackups`. - If `source.cleanUpScriptsDir` is configured, copy and encrypt cleanup scripts to transport drive via `copyCleanupScripts()`. - Write serial.txt with current timestamp to mark when data was created. 6. If running as target: - Check for serial.txt on transport drive; if missing, exit with warning (data may have been processed already). - If `target.geli` present, attempt to decrypt/mount GELI disks (via `ZFS_Utils::mountGeli`). - Update target datasets by reading files from transport, decrypting and (for `.xz` files) decompressing, and running `zfs receive -F` (via `receiveTransportStream()`). If a full stream cannot overwrite an existing dataset that has snapshots, behavior depends on `target.allowFullOverwrite` (destroy-and-retry vs. log an error). - Remove serial.txt after successful import to mark completion. - If `target.cleanUpScriptsDir` is configured, execute cleanup scripts from transport drive via `runCleanupScripts()`. 7. Run `cleanup()` to unmount, send reports, and optionally shutdown. ## Logging & Reports - The script uses `ZFS_Utils::logMsg` throughout. Default log path is set from `$config->{logFile}` and the module exposes `$logFileName` and `$displayLogsOnConsole` for customizing behavior at runtime. - Reports can be saved to a drive (via `mountDriveByLabel`) and/or emailed via `/usr/sbin/sendmail` using `ZFS_Utils::sendReport`. ## Security notes - GELI combined keys are created by XOR'ing a remote binary key and a local 256-bit hex key - the resulting key is written with mode `0600`. Keep these files and the key disk physically secure. - Transport encryption uses `openssl enc -aes-256-cbc`; manage the encryption key material carefully. ## Common operational tasks For step-by-step procedures on rotating the transport drive encryption key or pushing a code upgrade to an air-gapped target, see [Operations.md](Operations.md). ## Example quick-run checklist 1. Edit `sneakernet.conf.yaml` (create from defaults if necessary) and confirm `transport.mountPoint` and `label`. 2. On source: run `perl sneakernet --dryrun --verbosity 2` to validate the planned commands. 3. On source: run without `--dryrun` to execute replication. 4. Physically move the drive to the target, insert it, and run the script on the target host. ## Troubleshooting - If the transport drive does not mount, check that the GPT label matches `transport.label` and that the filesystem type matches `transport.fstype`. - If GELI attach fails, verify the keyfile exists on the secure key disk and that the local key hex string is correct (exactly 64 hex characters) and that the combined keyfile is created at the configured `target` path. - Use `--dryrun` and `--verbose` to inspect command strings before running them. - For key rotation or a remote code upgrade, see [Operations.md](Operations.md) rather than improvising a one-off cleanup script. --- ## Version History For detailed version history and changelog, see [CHANGELOG.md](../CHANGELOG.md). --- Document last updated: 2026-08-02