# Log redaction, remote upgrade, and transport key rotation

> Accepted plan, 2026-07-30. Supersedes the original problem statement, which is preserved as an
> appendix at the bottom. Implementation pending.

## Context

This document originally asked for three things: redact keys from logs, allow the transport key to be
rotated on the air-gapped target, and allow remote code upgrades to that target. The redaction item
was prompted by finding that `transport.encryptionKey` is written to the log in plaintext and
`sendReport` mails the log verbatim — so the key has almost certainly been emailed on every successful
run. The upgrade item also resolves a problem left over from the previous project: the air-gapped
target host is stuck at **r143** with no deployment path.

**This is deliberately a long-term, low-urgency project.** Continued use of the current transport key
is an accepted risk until the new code can be tested properly. Nothing here should be rushed onto
either host.

The parts are not fully independent: redaction must reach the **target** before the key is rotated, or
the new key is written to the target host's log and mailed on its first run and the rotation accomplishes
nothing. Getting redaction onto the target requires the upgrade mechanism. That order already matches
the intended schedule.

### Schedule

| When | What | Code state |
|---|---|---|
| **Sat 2026-08-01** | Normal monthly run. Nothing new deployed. | source r157, target r143 |
| **Aug (≈1 month)** | Build and fully test Parts 1-4 off-production. | — |
| **≈Sep** | Ship the upgrade one-shot, **tech on standby** in case it fails. | target → current |
| **≈Oct** | Key rotation, on its own drive. | both → new key |

Full deployment being 30+ days out means the maintenance network window to the air-gapped target host will be long closed.
The safety net for the upgrade is a **tech on standby**, not remote access — which is why the
installer's verify-before-swap and archive-the-old-install behaviour carries the weight it does.

## Corrections to the original assumptions

**1. The payload is ~175KB, not 14MB.** Measured from a real checkout of r157: 2.4MB raw, **175KB
`tar.xz`**. Embedding it in the script is therefore entirely practical and needs no change to the
transport mechanism. The 14MB figure is probably the live install including logs, `states/` (never
pruned) and status backups.

**2. `svn export` does *not* carry revision information — use `svn checkout`.** Worth flagging because
it defeats the exact feature wanted. Measured:

| | raw | tar.xz | `svnversion` reports |
|---|---|---|---|
| `svn export` | 1016K | 161 KB | `Unversioned directory` → `getWorkingCopyRevision` returns `'unknown'` |
| `svn checkout` | 2.4M | **175 KB** | **`157`** |

`.svn` costs **14 KB compressed** — xz deduplicates its largely-redundant text. So ship a checkout:
the provenance line keeps working on the target with no code change, and a future `svn update` becomes
possible if the target host is ever networked. The instinct that the additional files are minimal was right; it
just needs to be a checkout rather than an export to get the revision.

**3. `cleanupScripts/updateConfigKeys` already does most of the rotation work** — dot notation,
creating keys that don't exist, `DELETE`, and a timestamped backup. It needs fixing rather than
writing (Part 3).

**4. A cleanup script is `eval`'d as a string inside sneakernet's own process and lexical scope**
([sneakernet:1732](sneakernet#L1732)). It sees `$config`, `$programDefinition`, `$scriptFile` and every
exported `ZFS_Utils` function with no `use` needed. Two consequences: it must **not** declare a lexical
named `$config` (that shadows sneakernet's), and because it is a string eval there is **no `__DATA__`
filehandle** — an embedded payload must be a normal string literal.

## Part 1 — Log redaction

The scope is narrower than it first looked. Every structured leak (`Dumper` of a config or sub-config)
is better handled by moving its verbosity guard out of reach than by pattern-matching, so the redactor
only has to deal with the command-line form.

### New `redactSecrets($string)` in `ZFS_Utils.pm`, exported

One anchored rule: `-K <hex>` → `-K [REDACTED]`, as produced by
[buildEncryptionPipeline](sneakernet#L453), `buildDecryptionPipeline` and `verifyTransportFile`. Keep
the implementation trivially extensible (a list of patterns) but do not add an unanchored "any 64 hex
chars" rule — that would mangle legitimate output such as checksums and snapshot GUIDs.

**Do not redact IVs.** They are written in plaintext to the `.IV` sidecar beside the ciphertext by
design, so masking them in logs buys nothing and adds noise.

### Call sites

| Where | Why |
|---|---|
| `logMsg` ([ZFS_Utils.pm:296](../ZFS_Utils.pm#L296)), applied to `$msg` on entry | Single chokepoint for all three sinks — log file, console, TTY. One call covers every logging path in sneakernet and replicate. |
| `sendReport` ([ZFS_Utils.pm:~1610](../ZFS_Utils.pm#L1610)), applied to the slurped log **and** `$message` | The log is read verbatim into the email and onto the report drive. Also catches cleanup-script output, which reaches the report **without** passing through `logMsg`. |

**Do not add a parameter to `logMsg`.** Its doc comment claims a `$logCaller` second argument that does
not exist — position 2 is actually `$filename`.

The highest-risk existing path is [ZFS_Utils.pm:216 and :220](../ZFS_Utils.pm#L216) — two **unguarded**
`runCmd` error logs that emit the full failing command at any verbosity. A failed run is exactly the
run that gets emailed. Redacting inside `logMsg` covers them automatically.

### Verbosity-guard changes instead of redaction

- **`Dumper($geliConfig)`** at [ZFS_Utils.pm:878](../ZFS_Utils.pm#L878) (≥ 4) and
  [:953](../ZFS_Utils.pm#L953) (≥ 3) — the only `localKey` exposure. Raise both to **≥ 6**. Documented
  max verbosity is 5, so this is unreachable except by deliberately passing `-v 6` for deep debugging.
  No redaction rule needed for `localKey`.
- **`updateConfigKeys`** lines 119 and 184 currently `Dumper` the whole config at *any* nonzero
  verbosity. **Comment them out** — they were development aids. Confirmed it is a *sample* script: the
  README calls it "**Sample script**", nothing in `sneakernet`, `replicate` or `ZFS_Utils.pm`
  references it, and it is absent from the source host's `cleanupScriptSchedule`, so the whitelist filter means
  it never runs in production today. The Part 3 rotation script will be a copy of it and simply won't
  carry those lines.

**Explicitly accepted, not fixed:** the `die Dumper($config)` hatches (`sneakernet:2098` at
`debug == 10`, `replicate:443` at `debug == 9`). These are development-only, never set in production,
and running the script requires root — which already implies read access to the config file. Recorded
so nobody later mistakes them for an oversight.

Also accepted: the key is visible in `ps` while a pipeline runs, since it is an argv element to
`openssl`. The machine is physically monitored until after the zpool is decrypted, so reading it
mid-decryption is not a realistic threat. Fixing it would mean abandoning `-K <hex>` for a passphrase
form, changing the KDF and the on-disk ciphertext format, with both hosts having to change together.

### Historical logs: deliberately not purged

`sneakernet.log` on both hosts, and every emailed/report-drive copy, contain the current key.
**Decision: leave them.** Redaction lands before rotation, so logs written from that point on are
clean, and the October rotation retires the key those old logs contain. By the end of the schedule
nothing live is exposed, and chasing down historical copies (mail spools, report drives in rotation)
would be a lot of effort to retire a key that expires anyway.

The residual, already accepted: between now and rotation, runs keep logging and mailing the current
key.

## Part 2 — Remote upgrade via a self-extracting one-shot

### Builder on the source: `utilities/buildUpgradeOneShot.pl`

1. `svn checkout -q -r <rev|HEAD> http://svn.dailydata.net/svn/zfs_utils/trunk <tmp>` — takes a
   revision argument as originally asked. A checkout, not an export, so `svnversion` works on the
   target (see correction 2).
2. `tar cJf` it, base64 it, compute a SHA-256.
3. Emit a one-shot into `source.oneShotCleanup` containing the base64 as a **single string literal**,
   the checksum, and the installer logic.

`source.oneShotCleanup` now exists and is under version control. Files placed there bypass the monthly
schedule filter and are copied on **every** source run, then unlinked from the source after a
successful copy — so anything left there ships whether intended or not. Keep it empty between
deliberate uses.

### The generated installer, on the target

- Verify the SHA-256 **before** extracting; abort on mismatch.
- Extract to a temp directory and confirm expected files are present (`ZFS_Utils.pm`,
  `sneakernet/sneakernet`) before touching the install.
- Archive the current install to a timestamped tarball, then swap files in.
- **Preserve, never overwrite:** `sneakernet.conf.yaml`, `sneakernet_target.status*`, `*.log`,
  `states/`. Use an explicit include list, not a blanket copy.
- Return `($result, $errorString)` per the [convention](cleanupScripts/README.md).

Replacing the running code on disk is safe: Perl has already compiled `ZFS_Utils.pm` and `sneakernet`
into memory, so the current run continues on the old code and the **next** run picks up the new. The
script persists on the transport drive after running — `serial.txt` is the run-once guard, and
`removeObsoleteScripts` removes it on the source's next run.

This is also how the target host gets off r143, including the `sendTargetState` fix from r157.

## Part 3 — Key rotation one-shot

Copy `cleanupScripts/updateConfigKeys` into `oneShotCleanup` with these fixes, each addressing a real
failure mode:

| Fix | Why it matters |
|---|---|
| Use `$programDefinition->{configFileName}` (visible in the eval scope), not the relative `'sneakernet.conf.yaml'` at line 82 | There is no `chdir` before the eval, so the relative path resolves against cron's cwd. The current script would operate on the wrong path or fail. |
| Never assign to `$config->{transport}->{encryptionKey}` in memory; use a differently-named lexical | `buildDecryptionPipeline` reads the **in-memory** key. Mutating it mid-loop would break decryption of every cleanup script processed after this one. |
| `loadConfig` from disk; **abort if it returns `{}`** | `loadConfig` returns an empty hashref on *any* failure. Writing that back would erase the config on an air-gapped host. |
| Do not write out the ambient `$config` | sneakernet injects `runningAs`, `servername`, and overwrites `transport.mountPoint` at runtime; dumping it would persist run-specific junk into the YAML. |
| Backup, then write to a temp file, verify it re-parses and retains required keys, then `rename` over the original | `makeConfig` is a bare truncating overwrite — no backup, no temp-and-rename, no permission preservation. |
| Comment out the `Dumper($config)` calls | Part 1. |

YAML round-trip loss is a non-issue: both live configs already have zero comments and are already
alphabetised and string-quoted, which is what the YAML writer produces. The target host appears to be on
`YAML::Tiny` (inferable from its config's quoting and indent style) — confirm it is installed there.

### Choreography for the rotation (≈Oct, after the upgrade is confirmed)

The run-internal ordering is safe — verified: `updateTarget` receives everything with the old key,
*then* `runCleanupScripts` runs, *then* the one-shot rewrites only the on-disk config. The in-memory
key is untouched, so any cleanup script processed later in the same loop still decrypts.

1. Upgrade already landed and confirmed. The target host reports the current revision and redaction is active.
2. Generate the new key. **The source keeps using the old key** for the rotation drive.
3. Rotation one-shot rides the next drive, encrypted with the **old** key.
4. Target receives with the old key, then the one-shot writes the new key to its config on disk.
5. Confirm from the target's report or `target_state.txt`, **then** switch the source to the new key.
6. The following run uses the new key on both sides.

Keep the old key recorded offline until step 5 is confirmed. A mismatch halts replication but destroys
nothing — `zfs receive` simply refuses the stream. Recovery means a tech with the old key in hand. Ship
the upgrade and the rotation on **separate drives, a month apart** as scheduled; an upgrade and a key
change landing together would make any failure much harder to attribute.

## Part 4 — Pre-existing bugs (in scope)

Small, self-contained, and the first two directly affect the reliability of Parts 2-3.

- **`encryptAndCopyScript` logs copy failures but never pushes them onto `$errors`**
  ([sneakernet:~1600](sneakernet#L1600)), so a cleanup script that fails to reach the transport drive
  is invisible in the emailed report. A silently-missing upgrade or rotation one-shot would look like a
  clean run.
- **`runCleanupScripts` has no trailing `return`**, so it yields `undef` on the normal path. On its
  early-return paths it returns the global `$errors` arrayref, which
  [sneakernet:2222](sneakernet#L2222) then splices into itself, duplicating every prior error in the
  report.
- **`logMsg("... output:\n$result")`** at [sneakernet:1685](sneakernet#L1685) is unguarded and
  `$result` is `undef` whenever a cleanup script dies, so every failure also emits an "uninitialized
  value" warning.
- **`randomBytes`** ([sneakernet:422](sneakernet#L422)) silently falls back to `rand()` when
  `/dev/urandom` cannot be opened, producing predictable IVs. Emit a **warning** and carry on — not a
  fatal error.

## Verification

- **Unit test `redactSecrets`** in `testLibrary/`: the three openssl command shapes, plus negative
  cases (a 64-hex checksum, `secureKey`'s `keyfile` path, and an IV must all pass through unchanged).
- **Prove the leak is closed:** full dry run on the source, then `grep -c <key>` against both
  `sneakernet.log` and a generated report — both must be 0, against the current baseline of one
  occurrence per dataset command.
- **Prove the one-shot path works on the target host before trusting it with real work:** ship
  `cleanupScripts/helloWorld` as a one-shot first; the README documents exactly this workflow. Do not
  let the upgrade installer be the first thing that mechanism ever runs.
- **Test the installer locally:** build it, run its extract-and-verify logic against a scratch
  directory, and confirm the checksum check fails correctly on a deliberately corrupted blob.
- **After the upgrade run:** `sneakernet --version` on the target host, and confirm the report's provenance line
  shows a real revision number rather than `unknown` — that is the check that catches a
  checkout-vs-export mistake.
- **After rotation:** confirm the next drive decrypts and that the new key appears in neither host's
  log.

## Aside: how exposed is the 2026-08-01 run, with r157 untested on the source?

Less than it feels. The r156→r157 diff, checked specifically for source-reachable behaviour:

- **`ZFS_Utils.pm` r157 is comment-and-version only.** Filtering the diff for any non-comment,
  non-`$VERSION` line returns nothing. Behaviourally identical to r156.
- **The `sendTargetState` rewrite — the bulk of the change — is unreachable on the source.** Its only
  call site is inside `if ( $config->{runningAs} eq 'target' )` ([sneakernet:1987](sneakernet#L1987)).
- **The reworded `maxDelta` abort is also unreachable.** The source host's config defines `maxDelta` nowhere, so
  `validateSizeEstimateHistory` returns 1 on its first line and the `fatalError` never fires.
- The `cleanup()` reordering only moves a block that is itself target-gated.

So the untested surface that run actually exercises is r156's base-selection change — which has 40
passing unit tests against the real captured incident data, a passing dry run, and a live run. The r157
delta on top is effectively inert on the source.

Worth confirming beforehand, from the previous project's step 5:

```sh
wc -l sneakernet_target.status                    # expect ~66-70, not 4986
grep -c -- '--1y\|--3m' sneakernet_target.status  # expect >=1 per replicated fs
grep -i '^dryrun' sneakernet.conf.yaml            # must be '0' for the cron run
```

---

## Appendix — original problem statement

Preserved verbatim; this file was not under version control, so overwriting it would have lost the
original. Delete this section if you don't want it.

> Need an automated way to invoke key rotation and remote upgrades. This could be done with a one shot
> cleanup script. I'm assuming the following, but give recommendations. Also one security upgrade
>
> # Security Upgrade
> Set logger to redact all keys from all logs. I think the only one is the transport.encryptionKey entry
>
> # Allow transport key to be rotated
> I think the simplest way is to have a one shot that will simply modify the configuration file on the
> remote server. It should be simple: load the file in place, replace the key, create a backup of the
> old config, then write the config file back out. Making it generic would allow other keys to be
> upgraded, such as when sending an upgrade (next block) and wanting to put default (or modified)
> values which may not exist.
>
> # Allow remote upgrades
> sneakernet can be upgraded via subversion on source machine but target machine is air gapped, so has
> no Internet connection. process that would grab a fresh copy of the subversion repository (or a
> version that is requested on the CLI), tar it, then put it in a place that a one-shot script could
> use would be good. The script would just take the tar file and overwrite the existing install,
> probably after making an archive of the existing one.
>
> Not sure how we would actually transport the tar file. I don't think the cleanup scripts considered
> that a script may need a separate data file. The current zfs_utils is 14M. One of two options comes
> to mind:
>
> 1. Modify the one-shot script loader to allow it to define a separate file to send on the transport
>    drive. That is a code change, but hopefully not a huge one. In this case, the tar could be gzipped
>    (or xz'd) to reduce size
> 2. I think there is a way to append data to the end of a Perl script. In this case, the tar file
>    would probably need to be mime encoded? Sounds like a lot of data.