# Operations — common tasks

Task-oriented procedures for things an operator actually does with a running sneakernet deployment, as opposed to [sneakernet.md](sneakernet.md) (reference documentation: usage, config keys, functions) or [TESTING.md](TESTING.md)/[TESTING_automation.md](TESTING_automation.md) (the test plan and its automation). Both mechanisms below ride the same `source.oneShotCleanup` delivery channel: a script placed there bypasses the normal monthly `cleanupScriptSchedule` filter, is copied to the transport drive on the *next* source run regardless of month, and is removed from `oneShotCleanup` once that copy succeeds — so it ships exactly once. See [allowKeyRotation.md](allowKeyRotation.md) for the full design rationale and the pre-existing bugs this design deliberately routes around.

Both procedures are exercised by the automated test harness (`TESTING.md` Parts 15 and 17, run via `sneakernet/testing/`), but as of this writing neither has been used for a real rotation/upgrade against production. In most cases, the target machine is air-gapped and visited only occasionally — see the key-rotation project's schedule in `allowKeyRotation.md`. Before trusting either mechanism with real work against a host you can't easily recover, ship `sneakernet/cleanupScripts/helloWorld` through `oneShotCleanup` first and confirm it actually ran (its result appears in the next report) — that proves the delivery mechanism itself, cheaply, before using it for something that matters.

## Rotating the transport drive encryption key

Uses `sneakernet/cleanupScripts/updateConfigKeys` (a generic dot-notation config editor — YAMLPatch-based, backs up before writing, aborts rather than risk corrupting the file) pointed at a single key.

1. Generate a new key: `openssl rand -hex 32` (64 lowercase hex characters, same format as the existing `transport.encryptionKey`).
2. Copy `sneakernet/cleanupScripts/updateConfigKeys` to a new name and edit its `@updates` array down to one entry: `'transport.encryptionKey=<new key>'`.
3. Place that copy into the directory configured as `source.oneShotCleanup` in the **source** host's `sneakernet.conf.yaml`.
4. **Do not touch the source's own `transport.encryptionKey` yet.** The source must keep sending with the *old* key so this one-shot itself is still deliverable and decryptable normally.
5. Let the next scheduled replication cycle run as normal. The one-shot rides the transport drive (still old-key-encrypted), and executes on the **target** during that run's cleanup phase, rewriting only the target's on-disk config with the new key.
6. Confirm success before going further: check the target's report / `target_state.txt` for the one-shot's result, and that a fresh `sneakernet.conf.yaml.bak.<timestamp>` backup exists alongside a config now showing the new key.
7. Only once step 6 is confirmed, update the **source's** own `sneakernet.conf.yaml` to the same new key.
8. The following cycle uses the new key on both sides.

Keep the old key recorded offline until step 6 is confirmed. A mismatched key doesn't destroy anything — `zfs receive` simply refuses the stream — so recovery just means a tech with the old key in hand.

## Remote code upgrade (getting new code onto the air-gapped target)

Uses `utilities/buildUpgradeOneShot.pl` to package a real SVN checkout into a single self-extracting Perl one-shot.

1. From a machine with SVN and Perl access (the workstation is fine):
   ```sh
   perl utilities/buildUpgradeOneShot.pl \
     --output <source's oneShotCleanup dir>/upgradeToR<rev> \
     --rev <revision>          # defaults to HEAD
     # --repo and --install-dir also have production-matching defaults; override only if a
     # host's install path or repo URL ever differs from /usr/local/opt/zfs_utils
   ```
   This does a real `svn checkout` (not `export` — `export` loses `.svn`, which would make `getWorkingCopyRevision()`/the report's provenance line report `unknown` on the target afterward), packages it as `tar+xz`, and embeds the payload plus its SHA-256 in the generated script.
2. Place the generated file into the source host's `source.oneShotCleanup` directory.
3. Next replication cycle: the one-shot ships to the target and runs during that run's cleanup phase. It verifies the checksum (aborts on any mismatch), extracts to a temp directory, confirms `ZFS_Utils.pm` and `sneakernet/sneakernet` are present in the payload, archives the *current* install to `<installDir>.pre-upgrade.<timestamp>.tar.xz` right next to it, then copies the new tree over the install — explicitly preserving `sneakernet.conf.yaml`, `sneakernet_target.status*`, `*.log`, `history.tsv`, and `states/` (never overwritten).
4. Nothing needs restarting. sneakernet is invoked fresh each run, so the *next* invocation picks up the new code automatically — the run in progress when the swap happens keeps executing on what Perl already compiled into memory for that run.
5. Confirm: `sneakernet --version` on the target, and check the next report's provenance line shows a real revision number. `unknown` there means something went wrong with the checkout step (see the `export`-vs-`checkout` note above), not the upgrade mechanism itself.
6. If something needs to be undone, the pre-upgrade archive sits right beside the install directory for a tech on-site to restore by hand — there is no remote-access safety net for an air-gapped target, which is why the archive-before-swap step exists.

## If both are ever needed together

Ship them on separate drives, at least one cycle apart. Landing an upgrade and a key rotation together makes a failure much harder to attribute to one or the other, and the rotation's own choreography assumes the upgrade already landed and was confirmed first (`allowKeyRotation.md`).