# sneakernet Test Plan — All Changes Since SVN r150

**For:** a technician comfortable with ZFS/FreeBSD, not necessarily a programmer. **Where:** the test host, using dataset `storage/testing` as scratch space. Nothing here touches production data as long as you follow the dataset names given below exactly.

This is the single, definitive sneakernet test plan, covering all sneakernet changes since SVN r150 plus the pre-existing `maxDelta` size-guard feature. It consolidates what used to be split across two overlapping documents (a short developer checklist and this step-by-step plan) into one.

**How to use this document:** each Part is a numbered sequence of Steps. Every Step has an exact command to run and an **Expected result** to compare against. Write **PASS** or **FAIL** next to each Step as you go. If a Step fails, stop that Part, write down the actual output, and move on to the next Part — don't try to debug it yourself unless the fix is obvious.

**Total estimated time: approximately 6.75 to 7.5 hours (add ~20 minutes more if doing every optional step: Part 8 Step 8.5 and Part 13 Step 13.3).** Parts are mostly independent, so this can be split across several sessions — each Part starts with its own reset. Part 0 (setup) must be done first and only once.

---

## Before you start

- Confirm `storage` is an existing pool on the test host with room to spare (a few GB is plenty — test files are small).
- Confirm there is **no production sneakernet already running via cron on this machine**. Run `crontab -l` (as the user that would run it, often root) and look for any line mentioning `sneakernet`. If one exists, this plan still won't touch it (everything below uses its own install directory and its own datasets), but be aware it may run in the background while you test — that's fine, just don't let its transport drive/config get confused with your test setup.
- You will need `zfs`, `openssl`, `xz`, and ideally `svn`/`svnversion` on the `PATH`. Check with `which zfs openssl xz svnversion` — if `svnversion` is missing, a few log lines will show `(svn not available)` instead of a revision number; note it and move on, it is not a failure.
- **This tests the current *working copy* of the code, which includes uncommitted changes not yet in SVN.** Do not `svn checkout` a fresh copy for this — that would only get you revision 157, missing everything tested here. Get the exact working-copy directory you were given (the same one this document lives in) copied onto the test host — e.g. `tar` it up and `scp`/copy it over, or work directly on the test host if that's where it already lives.

---

## Part 0 — One-time setup

**Estimated time: 45 minutes**

### Step 0.1 — Install a private copy of the code

Do **not** test against any production install. Copy the working copy (see "Before you start") to its own directory:

```sh
mkdir -p /storage/testing
zfs create storage/testing
zfs create storage/testing/code
```

Copy the entire `zfs_utils` working-copy tree (the one containing this file) into `/storage/testing/code/`, so you end up with:

```
/storage/testing/code/ZFS_Utils.pm
/storage/testing/code/sneakernet/sneakernet
/storage/testing/code/sneakernet/sneakernet.datastructure
/storage/testing/code/sneakernet/cleanupScripts/...
/storage/testing/code/utilities/...
```

**Expected result:** `ls /storage/testing/code/sneakernet/sneakernet` shows the file, and `perl -c /storage/testing/code/sneakernet/sneakernet` prints `... syntax OK`.

### Step 0.2 — Confirm the version you're testing

```sh
grep 'our $VERSION' /storage/testing/code/ZFS_Utils.pm /storage/testing/code/sneakernet/sneakernet
```

**Expected result:**
```
ZFS_Utils.pm:our $VERSION = '1.7.0';
sneakernet:our $VERSION = '1.10.4';
```
If your numbers are higher, that's fine (more fixes landed) — just note the actual numbers on your report. If they're *lower*, you have an incomplete copy — stop and re-copy.

### Step 0.3 — Create the test dataset tree

```sh
zfs create storage/testing/src
zfs create storage/testing/src/ds1
zfs create storage/testing/src/ds2
zfs create storage/testing/dst
zfs create storage/testing/transport
zfs create storage/testing/report
zfs create storage/testing/oneshot
```

**Expected result:** `zfs list -r storage/testing` shows all seven datasets, each mounted under `/storage/testing/...`.

### Step 0.4 — Generate a test transport encryption key

```sh
openssl rand -hex 32 > /storage/testing/testkey.txt
cat /storage/testing/testkey.txt
```

**Expected result:** a 64-character line of hex digits (0-9, a-f only). Keep this file — you'll need the value inside it for the config below and for one later verification step.

### Step 0.5 — Create the test config file

Create `/storage/testing/code/sneakernet/sneakernet.conf.yaml` with **exactly** this content, except replace `PASTE_YOUR_KEY_HERE` with the 64-character value from Step 0.4:

```yaml
---
datasets:
  ds1:
    dataset: ds1
    source: storage/testing/src
    target: storage/testing/dst
    maxDelta: 0.9
  ds2:
    dataset: ds2
    source: storage/testing/src
    target: storage/testing/dst
    maxDelta: 0.9
debug: '0'
displayLogsOnConsole: '1'
displayLogsOnTTY: ''
dryrun: '0'
verbosity: '4'
logFile: /storage/testing/code/sneakernet/sneakernet.log
statusFile: /storage/testing/code/sneakernet/sneakernet_target.status
statusFileBackups: 5
source:
  hostname: test-source
  poolname: storage/testing/src
  runningAverageCount: 6
  historyFile: /storage/testing/code/sneakernet/history.tsv
  cleanUpScriptsDir: /storage/testing/code/sneakernet/cleanupScripts
  oneShotCleanup: /storage/testing/oneshot
  targetSnapshotList: ''
  fullSendPolicy: warn
  cleanupScriptSchedule:
    helloWorld:
    - 1
    - 2
    - 3
    - 4
    - 5
    - 6
    - 7
    - 8
    - 9
    - 10
    - 11
    - 12
  report:
    email: ''
    subject: ''
    targetDrive:
      label: ''
      mountPoint: ''
target:
  hostname: test-target
  poolname: storage/testing/dst
  stateFileName: target_state.txt
  stateFile: /storage/testing/code/sneakernet/states/targetState
  cleanUpScriptsDir: cleanUpScripts
  allowFullOverwrite: 0
  shutdownAfterReplication: 0
  report:
    email: ''
    subject: ''
    targetDrive:
      label: ''
      mountPoint: /storage/testing/report
transport:
  label: ''
  mountPoint: /storage/testing/transport
  pathSubstitution: '.'
  datasetDir: datasets
  serialFile: serial.txt
  encryptionKey: PASTE_YOUR_KEY_HERE
  verifyStream: header
  compression:
    method: 'off'
    level: 6
    threads: 0
    nice: 0
```

Notice `transport.label` and `target.report.targetDrive.label` are **both empty**, and `transport.mountPoint` / `target.report.targetDrive.mountPoint` point at the datasets you just created instead of a physical drive. This is the folder-based mode Parts 2 and 3 test directly.

`displayLogsOnConsole: '1'` is required here, not optional — nearly every Step below runs sneakernet through `tee /tmp/partN.log` and then greps that file for a log line. `loadConfig` only fills in `sneakernet.datastructure`'s defaults (which include `displayLogsOnConsole: 1`) when it creates a brand-new config file; since you're hand-creating this one, an omitted key simply stays unset, and sneakernet's own log lines will only reach `sneakernet.log`, never your terminal or `tee`'s capture — every later grep against a `/tmp/partN.log` file would then find nothing, with no indication why.

**Expected result:** `perl /storage/testing/code/sneakernet/sneakernet --version` prints `sneakernet v1.10.4` (or higher) with no errors.

### Step 0.6 — Smoke test

```sh
cd /storage/testing/code/sneakernet
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
zfs snapshot storage/testing/src/ds2@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source
perl sneakernet -s test-target
zfs list -rt all storage/testing/dst
```

**Expected result:** the last command shows `storage/testing/dst/ds1` and `storage/testing/dst/ds2`, each with one snapshot whose name matches what you just created on the source. If this works, your environment is ready — proceed to Part 1. If it doesn't, stop here; nothing past this point will work either.

---

## Part 1 — Basic sanity

**Estimated time: 5 minutes**

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 1.1 — Version and help

```sh
perl sneakernet --version
perl sneakernet --help
```

**Expected result:** version string printed and exits 0; help text lists `--dryrun`, `--verbosity`, `--debug`, `--servername`, `--version`, `--help` (six options).

---

## Part 2 — Folder-based transport + running both roles on one machine

**This directly tests a fix made in this round of changes** — the feature existed in the code but was unreachable until now.

**Estimated time: 15 minutes**

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 2.1 — Confirm the config as shipped

```sh
grep -A2 '^transport:' sneakernet.conf.yaml | head -3
```

**Expected result:** shows `label: ''` and `mountPoint: /storage/testing/transport`.

### Step 2.2 — Run source, then target, using `--servername`

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 4 2>&1 | tee /tmp/part2.log
```

**Expected result:** log contains a line like `Using folder-based transport at /storage/testing/transport`. **Does not** contain "Configuration file is empty or invalid" or "transport.label or transport.mountPoint must be set".

```sh
ls /storage/testing/transport/
```

**Expected result:** a `datasets/` directory and a `serial.txt` file are present — created directly as plain files, no drive was mounted.

```sh
perl sneakernet -s test-target -v 4 2>&1 | tee -a /tmp/part2.log
zfs list -rt snap storage/testing/dst/ds1
```

**Expected result:** target run completes without error; the new snapshot appears on `storage/testing/dst/ds1`.

### Step 2.3 — Confirm the failure mode still works correctly

Temporarily blank out `mountPoint` too, to prove the "both empty" case still fails safely:

Edit `sneakernet.conf.yaml` and change `mountPoint: /storage/testing/transport` (under `transport:`) to `mountPoint: ''`, then:

```sh
perl sneakernet -s test-source; echo "exit code: $?"
```

**Expected result:** immediately fails with `Invalid config file: transport.label or transport.mountPoint must be set`, non-zero exit code. **Now restore** `mountPoint: /storage/testing/transport` before continuing.

---

## Part 3 — Folder-based report drive and `target_state.txt`

**This also directly tests a fix made in this round of changes**, needed specifically so this plan could test the report drive at all without a physical labeled disk.

**Estimated time: 15 minutes**

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 3.1 — Confirm the report-drive config

```sh
grep -A3 'targetDrive:' sneakernet.conf.yaml | grep -A2 'label'
```

**Expected result:** under `target:`, `label: ''` and `mountPoint: /storage/testing/report`.

### Step 3.2 — Run a full source→target trip and inspect the report drive

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 2
perl sneakernet -s test-target -v 2 2>&1 | tee /tmp/part3.log
ls /storage/testing/report/
cat /storage/testing/report/target_state.txt
```

**Expected result:**
- The log contains `sendTargetState: Created target state file at /storage/testing/report/target_state.txt` with a snapshot count in parentheses.
- `target_state.txt` exists and is **not empty** — this is the exact bug that was fixed (a previous version of this code could silently produce a 0-byte file here while claiming success).
- The file's first 5 lines are a `#`-prefixed header showing a creation timestamp, hostname, `sneakernet v...`, and an SVN revision (or `unavailable` if `svnversion` isn't installed).
- After the header, plain lines like `storage/testing/dst/ds1@2026-...`.

### Step 3.3 — Confirm it fails safely, not silently

Temporarily break the pool name so the inventory read fails:

Edit `sneakernet.conf.yaml`, change `target.poolname` from `storage/testing/dst` to `storage/testing/doesnotexist`, then:

```sh
rm -f serial.txt; date +%s > /storage/testing/transport/serial.txt   # fake a pending target run
perl sneakernet -s test-target -v 1 2>&1 | tee /tmp/part3b.log
grep -i "could not read the snapshot inventory" /tmp/part3b.log
cat /tmp/part3b.log | grep -A3 "OVERVIEW"
```

**Expected result:** the log explicitly says it could not read the inventory and asks "Is the pool imported?"; this same message appears inside the `=== OVERVIEW ===` block of the report text, not just the log. **Restore** `target.poolname: storage/testing/dst` afterward.

---

## Part 4 — Transport-file verification (`transport.verifyStream`)

**Estimated time: 35 minutes**

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 4.1 — `header` mode (the default)

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 3 2>&1 | tee /tmp/part4a.log
grep "verifyTransportFile" /tmp/part4a.log
perl sneakernet -s test-target
```

**Expected result:** a line mentioning the header check passing (look for "header check OK" or similar wording in the `verifyTransportFile` lines); target run completes normally.

### Step 4.2 — `full` mode on an incremental send

Edit `sneakernet.conf.yaml`, change `verifyStream: header` to `verifyStream: full`.

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 3 2>&1 | tee /tmp/part4b.log
grep "verifyTransportFile" /tmp/part4b.log
perl sneakernet -s test-target
```

**Expected result:** log shows both a header check AND a full-stream check (a dry-run `zfs receive -nF` against the source dataset itself). Target run still completes normally. Leave `verifyStream: full` set for Steps 4.3 and 4.4 below; restore it to `header` at the end of 4.4.

### Step 4.3 — `full` mode on a brand-new dataset's first (full/base) send

This is the case that actually broke in production (`sneakernet/problem/sneakernet.log`, dataset `iscsi/mc-014`): a filesystem with **no prior recorded snapshot** always sends as a full/base stream (no `-I`), and `verifyStream: full`'s dry-run receive can never succeed against the source's own copy of that dataset — it always already has snapshots. By default (no `transport.verifyFullSendDataset` configured) sneakernet now detects this and falls back to a header-only check for that one file instead of aborting.

```sh
zfs create storage/testing/src/ds3
zfs snapshot storage/testing/src/ds3@$(date +%Y-%m-%d_%H.%M.%S)
```

Edit `sneakernet.conf.yaml` and add a new `ds3:` block under `datasets:` (copy the `ds1:` block and rename it to `ds3`, keeping the same `source`/`target`/`maxDelta`).

```sh
perl sneakernet -s test-source -v 3 2>&1 | tee /tmp/part4c.log
grep -i "full/base send" /tmp/part4c.log
perl sneakernet -s test-target
zfs list -rt all storage/testing/dst/ds3
```

**Expected result:** the log has a line explaining `ds3` is a full/base send and that it is falling back to a header-only check (it must **not** say anything about a scratch dataset here). The run completes normally — **no FATAL ERROR**, `serial.txt` is created. The target run receives `ds3` successfully and `zfs list -rt all storage/testing/dst/ds3` shows its snapshot.

### Step 4.4 — `full` mode on a first send, with `transport.verifyFullSendDataset` set

`transport.verifyFullSendDataset` is an opt-in that gets full/base sends genuinely checksum-verified (not just header-checked) by dry-run receiving them into a scratch dataset instead of their own live source copy. **This mechanism has not yet been confirmed against a real ZFS pool** (only tested against fake `zfs`/`openssl` stand-ins) — this step is that confirmation. Pay close attention to the actual output and write it down even if it doesn't match "Expected result" exactly; that information is more valuable here than a clean PASS.

```sh
zfs create storage/testing/src/ds4
zfs snapshot storage/testing/src/ds4@$(date +%Y-%m-%d_%H.%M.%S)
```

Edit `sneakernet.conf.yaml`: add a `ds4:` block under `datasets:` (same pattern as `ds3` above), and add `verifyFullSendDataset: storage/testing/verify` under `transport:` (this dataset does not exist yet — sneakernet should create it).

```sh
perl sneakernet -s test-source -v 3 2>&1 | tee /tmp/part4d.log
grep -i "scratch dataset" /tmp/part4d.log
zfs list -r storage/testing/verify
perl sneakernet -s test-target
zfs list -rt all storage/testing/dst/ds4
```

**Expected result:**
- The log has a line saying `ds4` is being verified against a scratch dataset (mentioning `storage/testing/verify`), not a header-only fallback line.
- `zfs list -r storage/testing/verify` shows the **parent dataset only, with no children** — the dry-run (`-n`) should never actually create the scratch child it verifies against. **If you see a leftover child dataset here (e.g. `storage/testing/verify/ds4`), that is important and unexpected — write down exactly what you see and do not delete it until this has been reported.**
- No FATAL ERROR; `serial.txt` is created; the target run receives `ds4` successfully.

**Afterward:** remove the `verifyFullSendDataset` line and restore `verifyStream: header` in `sneakernet.conf.yaml`. Also remove the `ds3:` and `ds4:` blocks you added under `datasets:` — otherwise every later Part's source run keeps sending them too (they'd still be configured), which throws off exact-count checks like Part 5's "one line for `ds1` and one for `ds2`". Then destroy their datasets so the known population returns to just `ds1`/`ds2`:

```sh
zfs destroy -r storage/testing/src/ds3 storage/testing/src/ds4 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
```

---

## Part 5 — Status carry-over for a disabled dataset

**Estimated time: 15 minutes**

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 5.1 — Baseline with both datasets

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
zfs snapshot storage/testing/src/ds2@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source; perl sneakernet -s test-target
cat sneakernet_target.status
```

**Expected result:** the status file has one line for `ds1` and one for `ds2`.

### Step 5.2 — Temporarily disable ds2

Edit `sneakernet.conf.yaml` and delete (or comment out) the entire `ds2:` block under `datasets:`.

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 3 2>&1 | tee /tmp/part5.log
grep -i "Carried over" /tmp/part5.log
cat sneakernet_target.status
perl sneakernet -s test-target
```

**Expected result:** log says "Carried over 1 status entry..."; the `ds2` line is **still present** in the status file even though `ds2` wasn't processed this run.

### Step 5.3 — Re-enable ds2 and confirm it resumes incrementally, not from scratch

Restore the `ds2:` block in `sneakernet.conf.yaml`.

```sh
zfs snapshot storage/testing/src/ds2@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 4 2>&1 | tee /tmp/part5b.log
grep "zfs send" /tmp/part5b.log | grep ds2
```

**Expected result:** the printed `ds2` command includes `-I` or `-i` (an incremental), **not** a bare `zfs send storage/testing/src/ds2@...` with no base — proving its old resume point survived being skipped.

---

## Part 6 — Timestamped status backups and pruning (`statusFileBackups`)

**Estimated time: 15 minutes**

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 6.1 — Confirm pruning keeps only the newest N

Edit `sneakernet.conf.yaml`, set `statusFileBackups: 2`.

```sh
for i in 1 2 3 4; do
  zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
  rm -f serial.txt; rm -rf /storage/testing/transport/*
  perl sneakernet -s test-source; perl sneakernet -s test-target
  sleep 1
done
ls -1 sneakernet_target.status.* | wc -l
ls -1 sneakernet_target.status.*
```

**Expected result:** exactly **2** timestamped backup files remain (the two most recent).

### Step 6.2 — Edge values

Set `statusFileBackups: 0`, repeat one send/receive cycle, then check:

```sh
ls -1 sneakernet_target.status.* 2>/dev/null | wc -l
```

**Expected result:** `0` — none kept.

Set `statusFileBackups: -1`, repeat two more cycles, then check:

```sh
ls -1 sneakernet_target.status.* | wc -l
```

**Expected result:** backups keep accumulating (no pruning). **Restore** `statusFileBackups: 5` when done.

---

## Part 7 — Full-overwrite of an existing target (`target.allowFullOverwrite`)

**Estimated time: 25 minutes**

This is the exact ZFS behavior at the heart of the original incident this whole project started from: a full stream cannot land on a target dataset that already has snapshots.

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 7.1 — Force a full send situation

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source; perl sneakernet -s test-target
# target now has ds1 with one snapshot
rm -f sneakernet_target.status sneakernet_target.status.*   # next send will be a FULL send
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
```

### Step 7.2a — `allowFullOverwrite: 0` (the default) — must refuse cleanly

```sh
grep allowFullOverwrite sneakernet.conf.yaml       # confirm it says 0
BEFORE=$(zfs list -H -t snap -o name storage/testing/dst/ds1 | wc -l)
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 3
perl sneakernet -s test-target -v 3 2>&1 | tee /tmp/part7a.log
grep -i "cannot overwrite\|allowFullOverwrite" /tmp/part7a.log
AFTER=$(zfs list -H -t snap -o name storage/testing/dst/ds1 | wc -l)
echo "before=$BEFORE after=$AFTER"
```

**Expected result:** an actionable error is logged mentioning `allowFullOverwrite`; `before` and `after` snapshot counts are **equal** — nothing was destroyed.

### Step 7.2b — `allowFullOverwrite: 1` — destroy-and-retry succeeds

Edit `sneakernet.conf.yaml`, set `allowFullOverwrite: 1`.

Step 7.2a's source run already recorded its attempted full send as "sent" in the status file, even though the receive failed - the source can't know in real time whether an async/physical-media transport actually lands on the target, so it doesn't wait for confirmation. That means, as far as the source is concerned, `ds1`/`ds2` are already up to date, and without forcing a full send again there is nothing left to retry. Force it again, same as Step 7.1:

```sh
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 3
perl sneakernet -s test-target -v 3 2>&1 | tee /tmp/part7b.log
grep -i "destroying existing\|Retrying receive" /tmp/part7b.log
zfs list -rt snap storage/testing/dst/ds1
```

**Expected result:** log shows "destroying existing... to receive a full stream" followed by "Retrying receive..."; the receive succeeds and only the newest snapshot(s) are present.

### Step 7.2c — Safety check: an unrelated failure must NOT destroy anything

This proves the destroy-and-retry path only fires for the specific "already has snapshots" error, not any random failure.

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source; perl sneakernet -s test-target
# target has ds1 with a snapshot again
KEY=$(cat /storage/testing/testkey.txt)
IVHEX=$(command -v xxd >/dev/null 2>&1 && openssl rand 16 | xxd -p | tr -d '\n' || openssl rand 16 | perl -0777 -ne 'print unpack("H*", $_)')
head -c 200000 /dev/urandom | openssl enc -aes-256-cbc -K "$KEY" -iv "$IVHEX" > /storage/testing/transport/datasets/storage.testing.dst.ds1
printf '%s' "$IVHEX" > /storage/testing/transport/datasets/storage.testing.dst.ds1.IV
date +%s > /storage/testing/transport/serial.txt
BEFORE=$(zfs list -H -t snap -o name storage/testing/dst/ds1 | wc -l)
perl sneakernet -s test-target -v 3 2>&1 | tee /tmp/part7c.log
grep -qi "destroying existing" /tmp/part7c.log && echo "FAIL: destroyed something it shouldn't have" || echo "PASS: no destroy"
AFTER=$(zfs list -H -t snap -o name storage/testing/dst/ds1 | wc -l)
echo "before=$BEFORE after=$AFTER"
```

**Expected result:** the "PASS: no destroy" line prints; `before` equals `after`; the log shows an import/receive error (garbage stream), not a snapshot-exists error. **Restore** `allowFullOverwrite: 0` when done.

---

## Part 8 — XZ stream compression (`transport.compression`)

**Estimated time: 35 minutes (add ~10 more if doing the optional Step 8.5)**

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 8.1 — Enable compression and confirm round-trip

Edit `sneakernet.conf.yaml`, change `method: 'off'` (under `transport.compression`) to `method: 'xz'`.

```sh
yes "compressible line for testing xz output shrinkage abcdefgh" | head -50000 > /storage/testing/src/ds1/big.txt
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 3 2>&1 | tee /tmp/part8.log
grep "xz" /tmp/part8.log
ls -l /storage/testing/transport/datasets/
perl sneakernet -s test-target -v 3
diff /storage/testing/src/ds1/big.txt /storage/testing/dst/ds1/big.txt && echo "PASS: contents match"
```

**Expected result:** log shows `xz` appearing in the pipeline before `openssl`; files in `datasets/` end in `.xz` (and `.xz.IV`); "PASS: contents match" prints (the target transparently decompresses).

### Step 8.2 — Verification still works under compression

With `verifyStream: header` still set (from Part 4's restore) and compression still on:

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 3 2>&1 | grep "verifyTransportFile"
perl sneakernet -s test-target
```

**Expected result:** header check still passes for the `.xz` file (it decrypts and decompresses before checking for the ZFS magic bytes).

### Step 8.3 — Compression actually shrinks the output

A rough before/after byte-size comparison, reusing the highly-compressible `big.txt` written in Step 8.1.

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/*
```

Edit `sneakernet.conf.yaml`, confirm `method: 'off'` (under `transport.compression`).

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source
UNCOMPRESSED=$(du -sk /storage/testing/transport/datasets | cut -f1)
perl sneakernet -s test-target
```

Edit `sneakernet.conf.yaml`, change `method: 'off'` to `method: 'xz'`.

```sh
rm -f serial.txt; rm -rf /storage/testing/transport/*
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source
COMPRESSED=$(du -sk /storage/testing/transport/datasets | cut -f1)
perl sneakernet -s test-target
echo "uncompressed=${UNCOMPRESSED}K compressed=${COMPRESSED}K"
```

**Expected result:** `compressed` is meaningfully smaller than `uncompressed` — `big.txt`'s content is a single repeated line, so the reduction should be large and obvious, not marginal.

### Step 8.4 — `threads` / `nice` are honored in the pipeline

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
```

Edit `sneakernet.conf.yaml` (compression still `method: 'xz'` from Step 8.3): set `threads: half` and `nice: 10`.

```sh
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 3 2>&1 | tee /tmp/part8d.log
grep -E "nice -n 10 xz -zc -6 -T[0-9]+" /tmp/part8d.log
perl sneakernet -s test-target
```

**Expected result:** a line showing the compression pipeline as `nice -n 10 xz -zc -6 -T<N>`, where `<N>` is roughly half the machine's CPU core count. **Restore** `threads: 0` and `nice: 0` afterward.

### Step 8.5 (Optional) — Mixed drive: one compressed file, one plain file, same trip

Confirms the target detects compression **per file** (by its `.xz` suffix), not from a config setting — so a drive can carry a mix without any special handling. This step hand-crafts the plain file, which is more fiddly than the others; skip it if short on time, it's not required to consider Part 8 passed.

```sh
zfs destroy -r storage/testing/dst/ds1 storage/testing/dst/ds2 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/*
```

Edit `sneakernet.conf.yaml`, confirm `method: 'xz'` is still set.

```sh
# ds1 goes through sneakernet normally and comes out compressed (.xz)
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source
ls /storage/testing/transport/datasets/          # expect ds1's file to end in .xz

# hand-add a PLAIN (uncompressed, still encrypted) file for ds2 onto the same drive
zfs snapshot storage/testing/src/ds2@$(date +%Y-%m-%d_%H.%M.%S)
SNAP=$(zfs list -H -o name -t snap storage/testing/src/ds2 | tail -1)
KEY=$(cat /storage/testing/testkey.txt)
IVHEX=$(openssl rand -hex 16)
zfs send "$SNAP" | openssl enc -aes-256-cbc -K "$KEY" -iv "$IVHEX" > /storage/testing/transport/datasets/ds2
printf '%s' "$IVHEX" > /storage/testing/transport/datasets/ds2.IV

perl sneakernet -s test-target -v 3 2>&1 | tee /tmp/part8e.log
zfs list -rt all storage/testing/dst/ds1 storage/testing/dst/ds2
```

**Expected result:** both `ds1` (from the `.xz` file) and `ds2` (from the hand-added plain file) receive successfully in the same target run — the target decompresses only the `.xz` file and passes the plain one straight through, with no config flag telling it which is which.

**Restore** `method: 'off'` before continuing to Part 9.

---

## Part 9 — Base-selection fallback when a resume point expires

**This is the single most important thing to test — it is the fix for the original production incident** (a 27.6T "full re-seed" of a dataset where only ~50G had actually changed, because the old code had no fallback when its recorded resume snapshot had been deleted).

**Estimated time: 50 minutes**

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 9.1 (Part 9a) — Inferred fallback, the common case

```sh
zfs snapshot storage/testing/src/ds1@S1_$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source; perl sneakernet -s test-target
# target now has S1
zfs snapshot storage/testing/src/ds1@S2_$(date +%Y-%m-%d_%H.%M.%S)
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source; perl sneakernet -s test-target
# target now has S1 and S2; status file records S2 as the resume point
zfs snapshot storage/testing/src/ds1@S3_$(date +%Y-%m-%d_%H.%M.%S)
S2NAME=$(zfs list -H -o name -t snap storage/testing/src/ds1 | grep S2_ | sed 's/.*@//')
zfs destroy storage/testing/src/ds1@$S2NAME    # simulate retention deleting the recorded resume point
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 4 2>&1 | tee /tmp/part9a.log
grep -E "no longer exists on the source|inferred" /tmp/part9a.log
grep "^zfs send" /tmp/part9a.log
perl sneakernet -s test-target
zfs list -rt snap storage/testing/dst/ds1
```

**Expected result:** log explicitly says the recorded resume point no longer exists on the source, and that it inferred a substitute base; the printed `zfs send` command is an **incremental** (`-I` or `-i`) from S1, **not** a bare full send. After the target run, `storage/testing/dst/ds1` shows S1, S2, and S3.

### Step 9.2 (Part 9b) — Confirmed fallback via a real target inventory

Same idea, but this time verify the base against an actual target listing rather than inferring it:

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/*
zfs snapshot storage/testing/src/ds1@S1_$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source; perl sneakernet -s test-target
zfs snapshot storage/testing/src/ds1@S2_$(date +%Y-%m-%d_%H.%M.%S)
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source; perl sneakernet -s test-target
zfs snapshot storage/testing/src/ds1@S3_$(date +%Y-%m-%d_%H.%M.%S)
S2NAME=$(zfs list -H -o name -t snap storage/testing/src/ds1 | grep S2_ | sed 's/.*@//')
zfs destroy storage/testing/src/ds1@$S2NAME
zfs list -H -o name -t snap storage/testing/dst/ds1 > /tmp/target_inventory.txt
```

Edit `sneakernet.conf.yaml`, set `source.targetSnapshotList: /tmp/target_inventory.txt`.

```sh
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 4 2>&1 | tee /tmp/part9b.log
grep -E "operator-supplied target snapshot list|confirmed" /tmp/part9b.log
perl sneakernet -s test-target
```

**Expected result:** log shows the supplied inventory file being used, and the base resolves via an exact/confirmed match rather than a guess. **Restore** `targetSnapshotList: ''` afterward.

### Step 9.3 (Part 9c) — No base at all (a recreated dataset)

Simulates the unfixable case from the real incident, where a dataset was destroyed and recreated and shares no history with the target at all:

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/*
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source; perl sneakernet -s test-target
# target now has ds1
zfs destroy -r storage/testing/src/ds1
zfs create storage/testing/src/ds1              # recreated - shares nothing with target's copy
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 4 2>&1 | tee /tmp/part9c.log
grep -i "no common base" /tmp/part9c.log
```

**Expected result:** log clearly states no common base snapshot could be found for `ds1`. **Leave this state as-is** — Part 10 continues directly from here.

### Step 9.4 — Source dataset destroyed and NOT recreated

A different, earlier failure mode than Step 9.3: there, `ds1` was destroyed and recreated, so it still existed (just with no shared history). Here, `ds1` is destroyed and never recreated at all. Before v1.10.8 this was silent — indistinguishable from "nothing new to send" — since `zfs list -rt snap` failing and `zfs list -rt snap` succeeding-but-empty produced the identical empty result.

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/*
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source; perl sneakernet -s test-target
# target now has ds1
zfs destroy -r storage/testing/src/ds1
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 2 2>&1 | tee /tmp/part9d.log
grep -i "does not exist - it may have been destroyed" /tmp/part9d.log
```

**Expected result:** the log names `ds1`'s full source dataset path and states it does not exist and may have been destroyed, suggesting either recreating it or removing it from `datasets:` in the config. No `FATAL ERROR` — the run completes normally, simply skipping `ds1` (any other configured dataset still processes normally). This message is pushed onto the run's internal error summary (not just logged), which is what makes it appear in a real emailed/report-drive report even when nothing else about the run looks unusual.

**Afterward:** recreate `ds1` so later Parts have it available again:

```sh
zfs create storage/testing/src/ds1
```

This step's destroy-without-recreate is a different state than Step 9.3 left. If you're continuing straight to Part 10 (which depends on Step 9.3's specific "recreated, no common base" state), re-run Step 9.3 first rather than relying on this step's ending state.

---

## Part 10 — `source.fullSendPolicy`

**Continues directly from the end of Part 9, Step 9.3 — do not reset.** This is the one Part in this plan without a "known dataset population" reset block at the top: it deliberately needs `ds1`'s no-common-base state left behind by Part 9 Step 9.3, so resetting here would defeat the point.

**Estimated time: 25 minutes**

### Step 10.1 — `allow`: full send proceeds quietly

Edit `sneakernet.conf.yaml`, set `fullSendPolicy: allow`.

```sh
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 4 2>&1 | tee /tmp/part10a.log
grep -c "^zfs send" /tmp/part10a.log
```

**Expected result:** a full-send command for `ds1` is generated (it will fail on the target side, same as Part 7 — that's expected and fine here, we're only checking the source's behavior).

### Step 10.2 — `warn` (the default): full send proceeds, loudly

Edit `sneakernet.conf.yaml`, set `fullSendPolicy: warn`.

```sh
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 1 2>&1 | tee /tmp/part10b.log
grep -i "sending a FULL stream" /tmp/part10b.log
```

**Expected result:** the warning appears even at low verbosity, and would be in the emailed report's OVERVIEW section (visible right at the top of the printed message).

### Step 10.3 — `skip`: excluded, but other datasets still work

Edit `sneakernet.conf.yaml`, set `fullSendPolicy: skip`.

```sh
zfs snapshot storage/testing/src/ds2@$(date +%Y-%m-%d_%H.%M.%S)   # ds2 has no problem
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 4 2>&1 | tee /tmp/part10c.log
grep -i "EXCLUDED" /tmp/part10c.log
grep "^zfs send" /tmp/part10c.log        # should show ds2's command, NOT ds1's
perl sneakernet -s test-target
zfs list -rt snap storage/testing/dst/ds2   # received normally
grep ds1 sneakernet_target.status            # ds1's OLD resume point is unchanged
```

**Expected result:** "EXCLUDED from this run" for `ds1`; only `ds2`'s send command appears; `ds2` replicates successfully; `ds1`'s status entry is untouched (nothing was falsely recorded as sent).

### Step 10.4 — `abort`: the whole run fails closed

Edit `sneakernet.conf.yaml`, set `fullSendPolicy: abort`.

```sh
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 1; echo "exit code: $?"
ls /storage/testing/transport/datasets/ 2>/dev/null
```

**Expected result:** non-zero exit code, `FATAL ERROR` in the output, and **nothing** written to the transport drive's `datasets/` folder — the whole run refused rather than sending anything partial. **Restore** `fullSendPolicy: warn` when done, and clean up: `zfs destroy -r storage/testing/src/ds1; zfs create storage/testing/src/ds1`.

---

## Part 11 — `source.targetSnapshotList` (already partially covered in Part 9b)

**Estimated time: 15 minutes**

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 11.1 — Confirm it produces an exact match with zero fallback warnings

```sh
zfs snapshot storage/testing/src/ds1@S1_$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source; perl sneakernet -s test-target
zfs snapshot storage/testing/src/ds1@S2_$(date +%Y-%m-%d_%H.%M.%S)
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source; perl sneakernet -s test-target
zfs snapshot storage/testing/src/ds1@S3_$(date +%Y-%m-%d_%H.%M.%S)
S2NAME=$(zfs list -H -o name -t snap storage/testing/src/ds1 | grep S2_ | sed 's/.*@//')
zfs destroy storage/testing/src/ds1@$S2NAME
zfs list -H -o name -t snap storage/testing/dst/ds1 > /tmp/target_inventory2.txt
```

Edit `sneakernet.conf.yaml`, set `source.targetSnapshotList: /tmp/target_inventory2.txt`.

```sh
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 4 2>&1 | tee /tmp/part11.log
grep -i "Using operator-supplied target snapshot list" /tmp/part11.log
grep -iE "confirmed|inferred" /tmp/part11.log
```

**Expected result:** log confirms the supplied file is in use (with a line/mtime count); no "inferred" warning appears at all — resolution is a clean exact match. **Restore** `targetSnapshotList: ''` afterward.

---

## Part 12 — Durable status-file anchor (long-TTL retention)

**Estimated time: 20 minutes**

This is what's supposed to make Parts 9b/11's manual recovery unnecessary in normal operation — a long-lived snapshot recorded automatically as a backup resume point.

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 12.1 — Confirm the anchor gets recorded

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)--3m
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 4 2>&1 | tee /tmp/part12a.log
perl sneakernet -s test-target
grep -c "^storage/testing/dst/ds1@" sneakernet_target.status
cat sneakernet_target.status | grep ds1
```

**Expected result:** **two** lines for `ds1` in the status file — the newest-sent snapshot, and the `--3m` snapshot recorded separately as a long-retention anchor.

### Step 12.2 — No duplicate line when the newest snapshot is itself long-retention

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/*
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)--1y
perl sneakernet -s test-source; perl sneakernet -s test-target
grep -c "^storage/testing/dst/ds1@" sneakernet_target.status
```

**Expected result:** exactly **one** line — no duplicate when the anchor and the primary resume point are the same snapshot.

### Step 12.3 — The anchor survives a gap and avoids the fallback guesswork entirely

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/*
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)--3m
perl sneakernet -s test-source; perl sneakernet -s test-target
zfs snapshot storage/testing/src/ds1@S2_$(date +%Y-%m-%d_%H.%M.%S)
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source; perl sneakernet -s test-target
zfs snapshot storage/testing/src/ds1@S3_$(date +%Y-%m-%d_%H.%M.%S)
S2NAME=$(zfs list -H -o name -t snap storage/testing/src/ds1 | grep S2_ | sed 's/.*@//')
zfs destroy storage/testing/src/ds1@$S2NAME
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 4 2>&1 | tee /tmp/part12c.log
grep -i "confirmed" /tmp/part12c.log
perl sneakernet -s test-target
```

**Expected result:** resolution is `confirmed`, not `inferred` — the anchor recorded automatically two runs earlier is what makes this possible, with no manual file editing needed this time (contrast with Part 9a, which needed the fallback to *guess*).

---

## Part 13 — Report provenance line and confirming secrets don't leak into logs

**Estimated time: 20 minutes (add ~10 more if doing the optional Step 13.3)**

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 13.1 — Provenance line

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 1 2>&1 | tee /tmp/part13a.log
grep -E "^Host: .* sneakernet v[0-9.]+ +svn revision:" /tmp/part13a.log
svnversion /storage/testing/code
```

**Expected result:** the report opens with a line like `Host: <hostname> (source) sneakernet v1.10.4 svn revision: <something>`, and the revision shown matches what `svnversion` reports directly against the working-copy root (`/storage/testing/code`, the parent of `sneakernet/` — `ZFS_Utils.pm` lives one level up, so a revision check scoped only to `sneakernet/` could miss drift in it). If `svnversion` isn't installed, the log line should say `(svn not available)` instead — not a failure, just note it.

### Step 13.2 — Confirm the transport key is never written to the log in plaintext

This is important — a previous version of this code wrote the encryption key to the log on every run, and the log gets emailed/copied to the report drive verbatim.

```sh
KEY=$(cat /storage/testing/testkey.txt)
rm -f sneakernet.log
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 4
grep -c "$KEY" sneakernet.log
```

**Expected result:** `0` — the key string does not appear anywhere in the log. You should instead see `-K [REDACTED]` wherever the encryption command would have shown the real key. Spot check:

```sh
grep "REDACTED" sneakernet.log | head -3
```

**Expected result:** at least one line showing `-K [REDACTED]` in place of the real key.

### Step 13.3 (Optional) — `target_state.txt` reaches the operator by email, not just the report drive

Requires a real mail relay reachable from the test host and an address you can check. Skip this step entirely if that's not available — Part 3 already confirmed `target_state.txt`'s content and header on the report drive; this step only adds the email-delivery path on top of that.

Edit `sneakernet.conf.yaml`, set `target.report.email` to a real address you can check (and `target.report.subject` to something recognizable, e.g. `TESTING Part 13.3`).

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/*
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 2
perl sneakernet -s test-target -v 2 2>&1 | tee /tmp/part13c.log
grep -i "Sending report via e-mail" /tmp/part13c.log
```

**Expected result:** the log confirms an email send attempt; the message that arrives in your inbox has a `target_state.txt` attachment, and its body opens with the same provenance line format confirmed in Step 13.1 (for the **target** host this time). **Restore** `target.report.email: ''` afterward.

---

## Part 14 — Cleanup script mechanism

**Estimated time: 15 minutes**

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 14.1 — Scheduled cleanup script (`helloWorld`)

The test config already schedules `helloWorld` for every month (Step 0.5).

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 3 2>&1 | tee /tmp/part14a.log
ls /storage/testing/transport/cleanUpScripts/
perl sneakernet -s test-target -v 3 2>&1 | tee /tmp/part14b.log
grep -A3 "helloWorld" /tmp/part14b.log
```

**Expected result:** an encrypted copy of `helloWorld` (plus its `.IV` file) appears on the transport drive; the target run's log shows its output — literally the words "hello" and (at verbosity ≥ 2) "world", plus a deliberate test error message that should appear under a `=== CLEANUP SCRIPT ERRORS ===` heading somewhere in the run's report text (this is expected — `helloWorld` intentionally includes an error message as part of its own self-test).

### Step 14.2 — One-shot cleanup script

```sh
cp /storage/testing/code/sneakernet/cleanupScripts/helloWorld /storage/testing/oneshot/
ls /storage/testing/oneshot/
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 2 2>&1 | tee /tmp/part14c.log
ls /storage/testing/oneshot/          # should now be EMPTY
grep -i "one-shot" /tmp/part14c.log
perl sneakernet -s test-target
```

**Expected result:** the copy of `helloWorld` disappears from `/storage/testing/oneshot/` after the source run (deleted after successful copy — a one-shot only ships once), and the log mentions removing it as a one-shot.

---

## Part 15 — Config-rotation one-shot script (`updateConfigKeys`)

**Estimated time: 20 minutes**

This exercises the fixed version of the generic config-editing one-shot — the mechanism intended for future key rotation and other config changes pushed to the air-gapped target.

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 15.1 — Prepare a customized copy

```sh
cp /storage/testing/code/sneakernet/cleanupScripts/updateConfigKeys /storage/testing/oneshot/rotateTest
```

Edit `/storage/testing/oneshot/rotateTest` and find the `my @updates = (` block (around line 107). Replace its contents with just:
```perl
my @updates = (
    'debug=7',
);
```
Save the file.

### Step 15.2 — Run it and confirm the target's config actually changed

```sh
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 2
ls /storage/testing/oneshot/                        # should be empty - shipped and removed
perl sneakernet -s test-target -v 2 2>&1 | tee /tmp/part15.log
grep -A5 "rotateTest" /tmp/part15.log
grep "^debug:" sneakernet.conf.yaml
ls sneakernet.conf.yaml.bak.*
```

**Expected result:**
- The script's own report shows `Updated: debug` with old value `0` and new value `7`.
- `sneakernet.conf.yaml` on disk now actually shows `debug: '7'`.
- A timestamped backup file `sneakernet.conf.yaml.bak.<timestamp>` exists next to it, containing the **old** value (`debug: '0'`).

**Set `debug` back to `0`** in `sneakernet.conf.yaml` by hand before continuing (or restore from the backup file).

### Step 15.3 — Confirm it refuses to run against a broken config

This proves the "abort if the loaded config is empty/invalid" safety check works.

```sh
cp sneakernet.conf.yaml /tmp/conf_backup_part15.yaml
echo "not valid yaml: [[[" >> sneakernet.conf.yaml
cp /storage/testing/code/sneakernet/cleanupScripts/updateConfigKeys /storage/testing/oneshot/rotateTest2
```
(no need to edit `rotateTest2`'s `@updates` this time — it will never get that far)
```sh
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 2 2>&1 | tee /tmp/part15c.log
```

This step is tricky because a badly broken config may prevent sneakernet from even starting. **If the source run itself fails to start** (likely, since the main config is now invalid), that is an acceptable outcome for this step — restore the good config immediately:

```sh
cp /tmp/conf_backup_part15.yaml sneakernet.conf.yaml
rm -f /storage/testing/oneshot/rotateTest2
```

**Expected result:** either the run refuses to start at all (acceptable), or if it does run the one-shot, the one-shot's own output reports an error about failing to load the config as a non-empty hash, and `sneakernet.conf.yaml` is confirmed **unchanged** (still broken, not truncated to something worse) afterward. The key thing to verify either way: nothing got silently corrupted.

---

## Part 16 — Standalone transport-file checker (`utilities/checkSneakernetFile`)

**Estimated time: 15 minutes**

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 16.1 — Good file

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source
F=$(ls /storage/testing/transport/datasets/ | grep -v '\.IV$' | head -1)
KEY=$(cat /storage/testing/testkey.txt)
perl /storage/testing/code/utilities/checkSneakernetFile \
    --file "/storage/testing/transport/datasets/$F" --key "$KEY" \
    --target storage/testing/src/ds1 --debug
echo "exit code: $?"
```

**Expected result:** `HEADER CHECK: OK` and `FULL-STREAM CHECK: OK`, exit code 0.

### Step 16.2 — Bad key, corrupted file, missing sidecar

```sh
perl /storage/testing/code/utilities/checkSneakernetFile \
    --file "/storage/testing/transport/datasets/$F" --key $(printf '0%.0s' {1..64})
echo "wrong key exit: $?"

cp "/storage/testing/transport/datasets/$F" /tmp/corrupt.bin
cp "/storage/testing/transport/datasets/$F.IV" /tmp/corrupt.bin.IV
dd if=/dev/urandom of=/tmp/corrupt.bin bs=1 count=512 seek=4096 conv=notrunc 2>/dev/null
perl /storage/testing/code/utilities/checkSneakernetFile --file /tmp/corrupt.bin --key "$KEY"
echo "corrupted exit: $?"

cp "/storage/testing/transport/datasets/$F" /tmp/noiv.bin
perl /storage/testing/code/utilities/checkSneakernetFile --file /tmp/noiv.bin --key "$KEY"
echo "no-IV exit: $?"
```

**Expected result:** three different non-zero exit codes (wrong key and corrupted file both fail the header or full-stream check; missing `.IV` sidecar fails immediately) — none of them exit 0.

> **Note:** `checkSneakernetFile` has the same underlying ZFS constraint as sneakernet's own `full` verification mode (see Part 4, Steps 4.3/4.4): if you manually point `--target` at a full/base file's own source dataset, `FULL-STREAM CHECK` will fail with "destination has snapshots" — that is expected, not a bug in the checker. Step 16.1 above avoids this by using `ds1`, which already has prior snapshots on the source, so its send is incremental. If you want to check a genuinely full/base file this way, point `--target` at an empty/scratch dataset instead.

```sh
perl sneakernet -s test-target     # consume the pending trip before moving on
```

---

## Part 17 (Optional / best-effort) — Upgrade builder utility

**Estimated time: 15 minutes. Skip entirely if the test host has no network route to the SVN server — that's expected and not a failure of anything being tested here.**

**Reset to a known dataset population:** this Part doesn't touch any datasets, but for consistency with the rest of this plan:

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 17.1 — Attempt a build

```sh
mkdir -p /tmp/upgradetest
perl /storage/testing/code/utilities/buildUpgradeOneShot.pl \
    --output /tmp/upgradetest/upgrade.pl
```

**Expected result (if network access exists):** prints a checked-out revision number, a payload size (expect somewhere around 150-250 KB), and writes `/tmp/upgradetest/upgrade.pl`. If it fails with a network/connection error, write down "N/A - no network access" and move on — this is not a defect in the code you're testing, and this feature isn't scheduled for real use until later anyway.

### Step 17.2 — If Step 17.1 succeeded, sanity-check the generated file

```sh
perl -c /tmp/upgradetest/upgrade.pl
```

**Expected result:** `... syntax OK`. Do **not** run this generated script against your test install or anything else — it's designed to overwrite a real sneakernet installation, and that's out of scope for this test plan.

---

## Part 18 — `maxDelta` behavior

**Estimated time: 25 minutes**

Covers three things: the abort message's wording, that normal/expected growth does NOT abort, and that leaving `maxDelta` unset entirely disables the check (and its history tracking) rather than just being permissive — this last case is exactly the workaround production used on the source host when `maxDelta` was "breaking for no reason," so it's worth confirming precisely.

**Reset to a known dataset population:**

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
zfs destroy -r storage/testing/dst/ds2 2>/dev/null
zfs destroy -r storage/testing/dst/ds3 2>/dev/null
zfs destroy -r storage/testing/dst/ds4 2>/dev/null
zfs destroy -r storage/testing/verify 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt
rm -rf /storage/testing/transport/* /storage/testing/report/* /storage/testing/oneshot/*
```

### Step 18.1 — Trigger the abort deliberately, check message wording

```sh
rm -f history.tsv
```

Edit `sneakernet.conf.yaml`, change `ds1`'s `maxDelta: 0.9` to `maxDelta: 0.01` (extremely tight — almost anything will exceed it once there's a history to compare against).

```sh
# first run establishes a baseline average - should succeed
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 2
perl sneakernet -s test-target
# second run: add a lot more data so the new estimate is much bigger than the average
yes "padding data to grow the estimated stream size for this test" | head -200000 > /storage/testing/src/ds1/pad.txt
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 2 2>&1 | tee /tmp/part18.log; echo "exit: $?"
grep -i "exceeds its running average" /tmp/part18.log
```

**Expected result:** if it aborts, the message names the dataset, states it exceeds its running average of the last N runs by more than `maxDelta` (shown as a percentage), and suggests checking for a full-send degradation — not a confusing older message that reads like a byte count was compared against a tiny fraction. (If it does *not* abort because your test data wasn't different enough in size, that's fine — just note that and move on; the important thing being tested is the wording, not whether it's easy to trigger with tiny test files.)

### Step 18.2 — Normal growth within bounds does NOT abort

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt history.tsv
rm -rf /storage/testing/transport/* /storage/testing/src/ds1/pad.txt
rm -f /storage/testing/src/ds1/pad2.txt
```

Edit `sneakernet.conf.yaml`, restore `ds1`'s `maxDelta` to `0.9` (90% - the same value used elsewhere in this test config).

```sh
# first run establishes a baseline average
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 2
perl sneakernet -s test-target
grep ds1 history.tsv
# second run: a SMALL amount of new data - well under a 90% increase
yes "small amount of new data" | head -50 > /storage/testing/src/ds1/pad2.txt
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 2 2>&1 | tee /tmp/part18b.log; echo "exit: $?"
perl sneakernet -s test-target
grep ds1 history.tsv
```

**Expected result:** the second run completes normally (no FATAL ERROR, no "exceeds its running average" message), `serial.txt` is created, and `grep ds1 history.tsv` shows **two** lines after the second run (one per run) instead of one.

### Step 18.3 — `maxDelta` unset disables the check (and history tracking) entirely

This is the actual production workaround (`maxDelta` removed from the source host's config after it kept aborting for reasons that weren't a real full-send degradation) - confirm precisely what that buys: not just "permissive," but that no history is even recorded.

```sh
zfs destroy -r storage/testing/dst/ds1 2>/dev/null
rm -f sneakernet_target.status sneakernet_target.status.* serial.txt history.tsv
rm -rf /storage/testing/transport/* /storage/testing/src/ds1/pad2.txt
```

Edit `sneakernet.conf.yaml` and **delete the entire `maxDelta:` line** from `ds1`'s block (not just set it to a large number - this is testing the "key absent" code path specifically).

```sh
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
perl sneakernet -s test-source -v 2
perl sneakernet -s test-target
grep -c ds1 history.tsv
# a large jump in size - would trigger Step 18.1's abort easily if maxDelta were still checked
yes "padding data to grow the estimated stream size for this test" | head -200000 > /storage/testing/src/ds1/pad.txt
zfs snapshot storage/testing/src/ds1@$(date +%Y-%m-%d_%H.%M.%S)
rm -f serial.txt; rm -rf /storage/testing/transport/*
perl sneakernet -s test-source -v 2 2>&1 | tee /tmp/part18c.log; echo "exit: $?"
perl sneakernet -s test-target
grep -c ds1 history.tsv
```

**Expected result:** both runs complete normally regardless of the large size jump (no FATAL ERROR), **and** `grep -c ds1 history.tsv` reports `0` after both runs — proving `maxDelta` unset doesn't just raise the bar, it disables size-history tracking for that dataset entirely.

**Afterward:** restore `ds1`'s `maxDelta: 0.9` in `sneakernet.conf.yaml` for later parts, and remove the `pad.txt`/`pad2.txt` files from `storage/testing/src/ds1`.

---

## Part 19 — Cleanup

**Estimated time: 10 minutes**

```sh
zfs destroy -r storage/testing
rm -rf /storage/testing
```

**Expected result:** `zfs list -r storage/testing` reports the dataset no longer exists. Confirm no stray files were left outside `/storage/testing` (check `/tmp/part*.log`, `/tmp/target_inventory*.txt`, `/tmp/upgradetest`, `/tmp/corrupt.bin*`, `/tmp/noiv.bin`, `/tmp/conf_backup_part15.yaml` — safe to delete all of these).

---

## Pass/Fail summary sheet

Copy this table into your report and fill in PASS/FAIL/N-A for each row.

| Part | What it tests | Result |
|---|---|---|
| 0 | Environment setup | |
| 1 | `--version` / `--help` | |
| 2 | Folder-based transport + `--servername` | |
| 3 | Folder-based report drive + `target_state.txt` | |
| 4 | Transport verification (header/full) | |
| 5 | Status carry-over for disabled dataset | |
| 6 | Timestamped status backups + pruning | |
| 7 | Full-overwrite of existing target (3 sub-tests) | |
| 8 | XZ stream compression (round-trip, shrinkage, threads/nice, optional mixed drive) | |
| 9 | Base-selection fallback + destroyed source dataset (4 sub-tests) | |
| 10 | `fullSendPolicy` (4 sub-tests) | |
| 11 | `targetSnapshotList` | |
| 12 | Long-TTL anchor (3 sub-tests) | |
| 13 | Report provenance (+ svnversion cross-check) + no plaintext key in logs (optional: email attachment) | |
| 14 | Cleanup script mechanism (scheduled + one-shot) | |
| 15 | Config-rotation one-shot script | |
| 16 | `checkSneakernetFile` utility | |
| 17 | Upgrade builder (optional) | |
| 18 | `maxDelta` behavior (3 sub-tests: wording, within-bounds pass, unset disables tracking) | |
| 19 | Cleanup | |

## Explicitly not tested here

Per instructions, these are excluded — they cannot be exercised on this system, and are considered stable ("rock solid") already:

- GELI disk decryption (`mountGeli`, `decryptAndMountGeli`, `utilities/decryptGeli.pl`)
- Physical zpool import/mount on the target
- `utilities/makeGeliKey`, `utilities/makeEncryptedZPool` (GELI setup tooling)

## If something fails

Write down: the Part/Step number, the exact command you ran, and the full output. Do not attempt to fix the code yourself. Everything in this plan is against a scratch dataset tree (`storage/testing`) — nothing here can damage production data as long as the dataset names above were followed exactly.