Rev 36 | Blame | Compare with Previous | Last modification | View Log | Download | RSS feed
```markdown
# sneakernet — Sneakernet replication script
Perl script to perform sneakernet replication of ZFS datasets between two servers using
an external transport drive. The script is designed for FreeBSD systems and integrates
with `ZFS_Utils.pm` for shared helpers (mounting by GPT label, GELI handling, logging, etc.).
Version: 1.0
License: Simplified BSD (FreeBSD) — see header in `sneakernet` script for full terms.
---
## Summary / Purpose
`sneakernet` automates ZFS snapshot export/import using a removable transport disk. On the
source server it creates zfs send streams (optionally encrypted) and writes them to files on
the transport disk. On the target server it reads those files (optionally decrypts) and
pipes them into `zfs receive` to update the target datasets. The script also supports
using GELI to protect disks on the target and can build combined GELI keys from a
remote binary key and a local hex key (via helpers in `ZFS_Utils.pm`).
## Usage
Run from the command line. A YAML config file is expected next to the script named
`$scriptname.conf.yaml` (the script will create or update it if needed).
Basic options:
- `--dryrun`, `-n` : run without making destructive changes (no writes)
- `--verbose`, `-v` : increase logging verbosity (can be stacked)
- `--help`, `-h` : print help and exit
- `--version`, `-V` : print script version and exit
Example:
```bash
perl sneakernet --dryrun --verbose
```
## Dependencies
- FreeBSD system utilities: `geli`, `zfs`, `zpool`, `geom`, `gpart`, `mount`, `/usr/sbin/sendmail`
- Perl core modules: `strict`, `warnings`, `FindBin`, `Getopt::Long`, `File::Basename`, `Data::Dumper`
- Shared module in repository: `ZFS_Utils.pm` (provides `loadConfig`, `mountDriveByLabel`, `logMsg`, etc.)
- Optional CPAN modules: YAML::XS or YAML::Tiny (used by `ZFS_Utils::loadConfig`)
## Configuration (YAML)
The script ships with an in-code default `$config` hash that is used as the basis for the
YAML configuration. The following documents the important keys, types and defaults. When you
run `sneakernet` it attempts to `loadConfig($scriptFullPath.conf.yaml, $config)` and will
create the file from the defaults if it does not exist.
Top-level keys
- `dryrun` (bool) — default: `0` — if true, actions that change state are not executed.
- `verbosity` (int) — default: `1` — controls logging verbosity.
- `status_file` (string) — path to status file used to track last replicated snapshots.
- `log_file` (string) — path to runtime log file.
`source` (hash)
- `hostname` (string) — hostname of the source server (used to detect running role)
- `poolname` (string) — zpool name on the source (default: `pool`)
- `report` (hash)
- `email` (string) — email to send the report to
- `subject` (string) — optional subject
- `targetDrive` (hash)
- `fstype` (string) — filesystem type of report drive (ufs/msdos)
- `check_interval` (int) — polling interval (seconds)
- `label` (string) — GPT label of the report drive
- `mount_point` (string) — optional mount point override
`target` (hash)
- `hostname` (string) — hostname of the target server
- `poolname` (string) — zpool name on the target (default: `backup`)
- `shutdown_after_replication` (bool) — default: `0` — if true, attempt to shutdown after completion
- `geli` (hash) — when present, instructs the script to decrypt/mount GELI-protected pool(s):
- `secureKey` (hash)
- `label` (string) — GPT label of the key disk (default: `replica`)
- `fstype` (string) — filesystem of key disk (default: `ufs`)
- `check_interval` (int) — polling interval for key disk
- `wait_timeout` (int) — how long to wait for the key disk
- `keyfile` (string) — filename of the remote binary key on the key disk (default: `geli.key`)
- `localKey` (string) — 64-hex-character 256-bit key string or path to file containing hex
- `target` (string) — path where the combined keyfile should be written (e.g. `/media/geli.key`)
- `poolname` (string) — pool name to import on target
- `diskList` (array) — optional list of device names to try (e.g. `['da0','da1']`)
`transport` (hash)
- `label` (string) — GPT label of the transport drive (default: `sneakernet`)
- `fstype` (string) — filesystem type for mounting (default: `ufs`)
- `mount_point` (string) — target mount point (default in sample: `/mnt/sneakernet`)
- `timeout` (int) — how long to wait for the transport device to appear (seconds)
- `check_interval` (int) — polling interval when waiting for the device (seconds)
- `encryption` (hash)
- `key` (string) — hex key used by `openssl enc -aes-256-cbc` for transport encryption
- `IV` (string) — hex IV used by encryption (defaults to zeros in sample)
`datasets` (hash)
- Keys are logical dataset names (user-defined blocks). Each dataset object contains:
- `source` — parent or root dataset on the source (string)
- `target` — parent or root dataset on the target (string)
- `dataset` — dataset name
Example minimal YAML snippet (derived from the script defaults):
```yaml
dryrun: 0
log_file: /path/to/sneakernet.log
source:
hostname: source-host
poolname: pool
target:
hostname: target-host
poolname: backup
geli:
secureKey:
label: replica
keyfile: geli.key
localKey: e98c66...bc9c
target: /media/geli.key
transport:
label: sneakernet
mount_point: /mnt/sneakernet
datasets:
files_share:
source: pool
target: backup
dataset: files_share
```
## Functions (script-level / documented)
The following functions are defined inside `sneakernet` and are documented here. Many
helpers used by the script are provided by `ZFS_Utils.pm` (imported at the top of the script).
- `getStatusFile($filename)`
- Returns: ARRAYREF of status lines (snapshot names). Reads `$filename` if present; returns
empty arrayref and logs an informational message if file missing or unreadable.
- `writeStatusFile($filename, $statusList)`
- Writes the provided ARRAYREF of status lines to `$filename`.
- Behavior: backs up an existing file to `$filename.bak` before writing. Dies on failure.
- `dirnameToFileName($string, $delimiter='/', $substitution='.')`
- Utility to turn dataset-like strings into filename-safe strings. Example: `pool/fs/sub` -> `pool.fs.sub`.
- `doSourceReplication($config, $statusList)`
- Performs replication on the source server.
- Behavior: enumerates source snapshots, builds zfs send commands with `makeReplicateCommands` (from `ZFS_Utils`),
optionally pipes through `openssl enc` (if transport encryption key is set), and writes send streams to files on the
transport mount point. Honors `$config->{dryrun}`.
- Returns: `$newStatus` ARRAYREF of updated status lines.
- `cleanup($config, $message)`
- Performs final cleanup and reporting actions.
- Behavior: logs disk usage and zpool list, attempts to unmount the transport drive, sends report via `sendReport`, and
optionally shuts down the machine if configured. Honors `dryrun`.
- `updateTarget($config)`
- Reads files from the transport disk and feeds them into `zfs receive` to update target datasets.
- Behavior: detects file->dataset mapping via the filename (uses `dirnameToFileName` reversal), optionally decrypts
with `openssl enc -d` if encryption was used, and calls `zfs receive -F` for each file.
## Main flow summary
1. Load YAML config using `ZFS_Utils::loadConfig` with the default config provided in the script.
2. Parse CLI flags (dryrun, verbose, help, version). CLI flags override config when present.
3. Determine whether the running host matches `source.hostname` or `target.hostname` and set `runningAs` accordingly.
4. Mount the transport drive (fatal error if not found) using `ZFS_Utils::mountDriveByLabel`.
5. If running as source:
- Clean transport directory (non-recursive), produce send streams and write to files on transport drive.
- Update status file with newest snapshots.
6. If running as target:
- If `target.geli` present, attempt to decrypt/mount GELI disks (via `ZFS_Utils::mountGeli`).
- Update target datasets by reading files from transport and running `zfs receive`.
7. Run `cleanup()` to unmount, send reports, and optionally shutdown.
## Logging & Reports
- The script uses `ZFS_Utils::logMsg` throughout. Default log path is set from `$config->{log_file}` and
the module exposes `$logFileName` and `$displayLogsOnConsole` for customizing behavior at runtime.
- Reports can be saved to a drive (via `mountDriveByLabel`) and/or emailed via `/usr/sbin/sendmail` using
`ZFS_Utils::sendReport`.
## Security notes
- GELI combined keys are created by XOR'ing a remote binary key and a local 256-bit hex key — the resulting
key is written with mode `0600`. Keep these files and the key disk physically secure.
- Transport encryption uses `openssl enc -aes-256-cbc`; manage the encryption key material carefully.
## Example quick-run checklist
1. Edit `sneakernet.conf.yaml` (create from defaults if necessary) and confirm `transport.mount_point` and `label`.
2. On source: run `perl sneakernet --dryrun --verbose` to validate the planned commands.
3. On source: run without `--dryrun` to execute replication.
4. Physically move the drive to the target, insert it, and run the script on the target host.
## Troubleshooting
- If the transport drive does not mount, check that the GPT label matches `transport.label` and that the filesystem
type matches `transport.fstype`.
- If GELI attach fails, verify the keyfile exists on the secure key disk and that the local key hex string is correct
(exactly 64 hex characters) and that the combined keyfile is created at the configured `target` path.
- Use `--dryrun` and `--verbose` to inspect command strings before running them.
---
Document last updated: 2025-12-15