Subversion Repositories zfs_utils

Rev

Rev 36 | Show entire file | Ignore whitespace | Details | Blame | Last modification | View Log | RSS feed

Rev 36 Rev 48
Line 1... Line 1...
1 
# sneakernet
 1 
```markdown
- 
 
 2 
# sneakernet — Sneakernet replication script
2 
 
 3 
 
3 
Lightweight perl script to perform zfs replication over a transport media like a disk drive. Data may be encrypted in transit using symentric key. Function (send/receive) determined by server name (hostname) which is matched in config file entries under 'source' and 'target'.
 4 
Perl script to perform sneakernet replication of ZFS datasets between two servers using
- 
 
 5 
an external transport drive. The script is designed for FreeBSD systems and integrates
- 
 
 6 
with `ZFS_Utils.pm` for shared helpers (mounting by GPT label, GELI handling, logging, etc.).
4 
 
 7 
 
5 
## Usage
 - 
 
6 
 
 - 
 
7 
CLI examples
 - 
 
8 
 
 - 
 
9 
```
 - 
 
10 
# run
 - 
 
11 
sneakernet
 8 
Version: 1.0
12 
```
 - 
 
13 
 
 - 
 
14 
## $config structure
 - 
 
15 
 
 - 
 
16 
sneakernet uses a top-level $config object stored as YAML (e.g., sneakernet.config.yaml) to control runtime behavior. If this file does not exist, a sample is created on first run. **You must edit this config with correct values**. Failure to do so could result in lost data.
 9 
License: Simplified BSD (FreeBSD) — see header in `sneakernet` script for full terms.
17 
 
 10 
 
18 
Example YAML:
 - 
 
19 
 
 - 
 
20 
```yaml
 - 
 
21 
---
 11 
---
22 
datasets:
 - 
 
23 
  dataset1:
 - 
 
24 
    filename: dataset1
 - 
 
25 
    source: pool/dataset1
 - 
 
26 
    target: backup/dataset1
 - 
 
27 
  files_share:
 - 
 
28 
    filename: files_share
 - 
 
29 
    source: pool/files_share
 - 
 
30 
    target: backup/files_share
 - 
 
31 
source_server:
 - 
 
32 
  hostname: nas
 - 
 
33 
  poolname: pool
 - 
 
34 
  status_file: ./sneakernet.status
 - 
 
35 
  report:
 - 
 
36 
    email: tech@example.org
 - 
 
37 
    subject: 'AG Transport Report'
 - 
 
38 
    targetDrive:
 - 
 
39 
      fstype: ''
 - 
 
40 
      label: ''
 - 
 
41 
      mount_point: ''
 - 
 
42 
target_server:
 - 
 
43 
  hostname: airgap
 - 
 
44 
  poolname: backup
 - 
 
45 
  geli:
 - 
 
46 
    diskList:
 - 
 
47 
      - da0
 - 
 
48 
      - da1
 - 
 
49 
    keydiskname: replica
 - 
 
50 
    keyfile: geli.key
 - 
 
51 
    localKey: e98c660cccdae1226550484d62caa2b72f60632ae0c607528aba1ac9e7bfbc9c
 - 
 
52 
    target: /media/geli.key
 - 
 
53 
    poolname: backup
 - 
 
54 
  report:
 - 
 
55 
    email: ''
 - 
 
56 
    subject: ''
 - 
 
57 
    targetDrive:
 - 
 
58 
      fstype: msdos
 - 
 
59 
      label: sneakernet_report
 - 
 
60 
      mount_point: /mnt/sneakernet_report
 - 
 
61 
transport:
 - 
 
62 
  disk_label: sneakernet
 - 
 
63 
  mount_point: /mnt/sneakernet
 - 
 
64 
  timeout: 600
 - 
 
65 
  encryption:
 - 
 
66 
    IV: '00000000000000000000000000000000'
 - 
 
67 
    key: ''
 - 
 
68 
```
 - 
 
69 
 
 - 
 
70 
Field reference
 - 
 
71 
 
 - 
 
72 
- datasets (hash, required): List of one or more definitions which are transferred
 - 
 
73 
  - filename (string, optional): filename to be used when storing on disk. If not entered, the dataset name is used
 - 
 
74 
  - source (string, required): full zfs path (no leading slash) on the source machine
 - 
 
75 
  - target (string, required): full zfs path (no leading slash) on target machine
 - 
 
76 
- source_server (hash, required): information used when processing on source server
 - 
 
77 
  - hostname (string, required): output of hostname command on source server
 - 
 
78 
  - poolname (string, required): name of the root pool on source server (not used)
 - 
 
79 
  - status_file (string, required): full path to a status file to be created listing last snapshot have already been processed for each dataset/child dataset
 - 
 
80 
  - report (hash, optional): contains parameters for sending report
 - 
 
81 
    - email (hash, optional): information on how to send report via e-mail
 - 
 
82 
      - address (string, optional): single e-mail address to send report to
 - 
 
83 
      - subject (string, optional); Subject line of e-mail
 - 
 
84 
    - targetDrive (hash, optional): information on saving the file on a local disk
 - 
 
85 
      - fstype (string, optional): use either **msdos** or **ufs** depending on how your disk is formatted. Default is ufs, which is not easily read on Windows or MacOS machines.
 - 
 
86 
      - label (string, optional): msdos label of partition to be mounted for the report
 - 
 
87 
      - mount_point (string, optional): directory to mount the partition on. Calculated as /mnt/label if not set
 - 
 
88 
- target_server (hash, required): information used when processing target server
 - 
 
89 
  - hostname (string, required): output of hostname command on target server
 - 
 
90 
  - poolname (string, required): name of the root pool on target server (not used)
 - 
 
91 
  - geli (hash, optional): information to decrypt geli encrypted drives and mount zpool
 - 
 
92 
    - disk_list (array, optional): list of geli encrypted disks which make up the pool. Searched for if not listed (drive names change on BSD)
 - 
 
93 
    - keydiskname (string, required): label on ufs file system containing key file
 - 
 
94 
    - keyfile (string, optional): name of the file on disk
 - 
 
95 
    - localKey (hex string, optional): 64 hexadecimal characters (256 bit) which is xor'd with keyfile to calculate target
 - 
 
96 
    - target (string, required): name of file on disk which is the key to unlock the geli disks. If keyfile and local key exist, this is created by xoring them, but may be found some other way.
 - 
 
97 
  - report (hash, optional): contains parameters for sending report
 - 
 
98 
    - email (hash, optional): information on how to send report via e-mail
 - 
 
99 
      - address (string, optional): single e-mail address to send report to
 - 
 
100 
      - subject (string, optional); Subject line of e-mail
 - 
 
101 
    - targetDrive (hash, optional): information on saving the file on a local disk
 - 
 
102 
      - fstype (string, optional): use either **msdos** or **ufs** depending on how your disk is formatted. Default is ufs, which is not easily read on Windows or MacOS machines.
 - 
 
103 
      - label (string, optional): msdos label of partition to be mounted for the report
 - 
 
104 
      - mount_point (string, optional): directory to mount the partition on. Calculated as /mnt/label if not set
 - 
 
105 
- transport (hash, required): hash with information on transport media.
 - 
 
106 
  - disk_label (string, required): label on transport disk. Assumed to be ufs disk with label
 - 
 
107 
  - mount_point (string, optional): mount point of disk with disk_label label. Calculated as /mnt/disk_label if not defined
 - 
 
108 
  - timeout (integer, optional): number of seconds to wait for transport disk to appear. Defaults to 600s (10 min).
 - 
 
109 
  - encryption (hash, optional): information used to encrypt/decrypt data on transport disk. This is a symetric key, so must be the same on both source and target machines
 - 
 
110 
    - IV (hex string, optional): Initial vector for encryption. Defaults to all zeros if not set
 - 
 
111 
    - key (hex string, optional): 64 hexadecimal characters (256bit) used to encrypt files in transit (while on transport disk). If not set, files are unencrypted
 - 
 
112 
-
- 
 
113 
 
 - 
 
114 
Notes on YAML
 - 
 
115 
 
 - 
 
116 
- Use a UTF-8 encoded file (config.yaml) and preserve numeric types (no quotes) for numeric fields.
 - 
 
117 
- Tooling reads YAML and validates types; comments are allowed but not preserved in runtime exports.
 - 
 
118 
 
 - 
 
119 
## Overview
 - 
 
120 
 
 - 
 
121 
The script acts differently based on whether a call to *hostname* returns the source or target servers names.
- 
 
122 
 
 - 
 
123 
The script requires the library (perl module) ZFS_Utils.pm to be in the parent document of the script. Most functions are defined there.  See ZFS_Utils.md for documentation.
 - 
 
124 
 
 - 
 
125 
**Note:** the variable \$DEBUG, if set to non-zero, will perform all functions except actual writes. In source mode, it will not create the files on the transport disk, in target mode it will not actually perform the receive to the datasets. However, the "what I would do" shows up in the logs.
 - 
 
126 
 
 - 
 
127 
#### Common startup
 - 
 
128 
 
 - 
 
129 
- Load config fileinto hash \$config.
 - 
 
130
- 
 
131 
  - If config file does not exist, it will create it and exit with a message
 - 
 
132
- 
 
133 
  - If config file does not contain source or target server names, it assumes it is not edited and exits with a message
 - 
 
134 
 
 - 
 
135 
- Mount transport disk
 - 
 
136
- 
 
137 
  - wait *timeout* seconds for disk label to show up in /dev/gpt, displaying prompt every 14 seconds. Dies if it can not find it
 - 
 
138
- 
 
139 
  - mount transport disk on *mount_point*
 - 
 
140 
 
 - 
 
141 
- Get results of *hostname -s* to determine if we are in source mode or target. If the name matches neither, fails with an error message
 - 
 
142 
 
 - 
 
143 
#### Source Mode
 - 
 
144 
 
 - 
 
145 
- read status file
 - 
 
146 
 
 - 
 
147 
- Perform replication, placing files on transport disk.
 - 
 
148
- 
 
149 
  - The files are named with the dataset, with all slashes replaced with periods, so datasets with embedded period will fail when in Target Mode
 - 
 
150
- 
 
151 
  - if encryption is defined, pass send stream through openssl (*openssl enc*)
 - 
 
152 
 
 - 
 
153 
- Writes the status file to disk. Status file will contain one line for each dataset sent, containing the last snapshot sent
 - 
 
154 
 
 - 
 
155 
#### Target Mode
 - 
 
156 
 
 - 
 
157 
- If geli encryption defined
 - 
 
158
- 
 
159 
  - Find geli key
 - 
 
160
- 
 
161 
  - Find geli disks, either defined or by searching for all disks which are not
 - 
 
162
- 
 
163 
    - mounted
 - 
 
164
- 
 
165 
    - partitioned
 - 
 
166
- 
 
167 
    - a member of an existing zpool
 - 
 
168
- 
 
169 
  - decrypt all geli disks
 - 
 
170
- 
 
171 
  - Import the zpool
 - 
 
172 
 
 - 
 
173 
- for each file in the root ofthe transport disk
 - 
 
174
- 
 
175 
  - receive into correct dataset
 - 
 
176
- 
 
177 
    - if filename has periods, calculate dataset as a path in the zpool
 - 
 
178
- 
 
179 
    - If encryption is defined, pass the file through openssl in decrypt mode (*openssl enc -d*)
 - 
 
180 
 
 - 
 
181 
#### Common Cleanup
 - 
 
182 
 
 - 
 
183 
- Umount transport disk and remove its *mount_path*
 - 
 
184 
 
 - 
 
185 
- Send report if defined
 - 
 
186 
 
 - 
 
187 
## Other Notes
 - 
 
188 
 
 - 
 
189 
Some areas of the script require further explanation. This section covers that.
 - 
 
190 
 
 - 
 
191 
#### Transport Encryption
 - 
 
192 
 
 - 
 
193 
Transport encryption is optionally set by placing a hex string representation of a 256bit key in transport/encryption. This key can be generated by hand, or you can use openssl, xxd and tr with the command:
 - 
 
194 
 
 - 
 
195 
`openssl rand 32 | xxd -p | tr -d '\n'`
 - 
 
196 
 
 - 
 
197 
If this key exists, the contents of the data on the transport script are encrypted with it, and can not be viewed without using the same key to decrypt (symetric encryption, or one time key pad)
 - 
 
198 
 
 - 
 
199 
You can further boost security by setting the IV parameter to something other than 0's.
 - 
 
200 
 
 - 
 
201 
#### Status file
 - 
 
202 
 
 - 
 
203 
Since the source server can not contact the target to see what is on there, the status file on the source server keeps track of what the last snapshot sent was. If it does not exist, the system assumes we need a full copy, which can require a very large media to do the transfer (size of files plus size of all deltas).
 - 
 
204 
 
 12 
 
205 
If your target server already has some snapshots on it, you can find the newest snapshot on it for each dataset, then make the file yourself.
 13 
## Summary / Purpose
206 
 
 14 
 
- 
 
 15 
`sneakernet` automates ZFS snapshot export/import using a removable transport disk. On the
207 
A sample from my test machine shows two datasets on the zpool *storage* name ComputerFiles and android. android has a child dataset which was out of sync, so that is show also.
 16 
source server it creates zfs send streams (optionally encrypted) and writes them to files on
- 
 
 17 
the transport disk. On the target server it reads those files (optionally decrypts) and
- 
 
 18 
pipes them into `zfs receive` to update the target datasets. The script also supports
- 
 
 19 
using GELI to protect disks on the target and can build combined GELI keys from a
- 
 
 20 
remote binary key and a local hex key (via helpers in `ZFS_Utils.pm`).
208 
 
 21 
 
209 
Running the command `zfs list -rt snap -H -o name storage` shows the following.
 - 
 
210 
 
 - 
 
211 
```
 - 
 
212 
storage@daily_2025-12-07_16.09.00--7d
 - 
 
213 
storage/ComputerFiles@daily_2025-12-07_16.09.00--7d
 - 
 
214 
storage/ComputerFiles@daily_2025-12-10_02.10.00--7d
 - 
 
215 
storage/android@daily_2025-12-07_16.09.00--7d
 - 
 
216 
storage/android@daily_2025-12-10_16.54.00--7d
 - 
 
217 
storage/android/old@daily_2025-12-07_16.09.00--7d
 - 
 
218 
storage/android/old@daily_2025-12-10_16.54.00--7d
 - 
 
219 
storage/android/old@daily_2025-12-10_16.58.00--7d
 - 
 
220 
```
 - 
 
221 
 
 - 
 
222 
From that, I can grab only the latest snapshots from each and create a status file containing
 - 
 
223 
 
 - 
 
224 
```
 - 
 
225 
storage/ComputerFiles@daily_2025-12-07_16.09.00--7d
 - 
 
226 
storage/android@daily_2025-12-07_16.09.00--7d
 - 
 
227 
storage/android/old@daily_2025-12-07_16.09.00--7d
 - 
 
228 
```
 - 
 
229 
 
 - 
 
230 
#### geli Encryption
 - 
 
231 
 
 - 
 
232 
This system was designed for an air gap server holding some sensitive information, in an insecure location. As such, we encrypt the disks that make up the zpool using geli, BSD's full disk encryption system. If your system does not use geli encryption, just leave the geli section blank or don't even include it.
 22 
## Usage
233 
 
 - 
 
234 
###### Finding Disks
 - 
 
235 
 
 - 
 
236 
Since BSD has a habit of changing disk drive names when a drive is added or removed from the system (on reboot), it is likely not best to rely on the drive names. However, in some cases it is fine. Listing the drives as an array will limit the drives which are decrypted, resulting in a slightly faster decryptions.
 - 
 
237 
 
 - 
 
238 
However, if the drives are not listed, a search will be made of all drives which are not mounted, have no slices (partitions), and are not a member of any other zpool. It then attempts to decrypt those. If it finds a drive or two which are not geli disks, is simply moves on to the next one.
 - 
 
239 
 
 - 
 
240 
###### Encryption Key
 - 
 
241 
 
 - 
 
242 
To make it easier to run without user interaction, we use a key file for the encryption. This file is passed to geli to decrypt the disks and make them available.
 - 
 
243 
 
 - 
 
244 
To make it a little safer, we randomly generated two keys of 256bit (32 bytes), then ran the binary XOR function on them to create a third key. When you do this, having two keys allow you to calculate the third one by simply XOR'ing them together.
 - 
 
245 
 
 - 
 
246 
The script in utilities named *makeGeliKey* can do all this for you. It will generate three keys, arbitrarily naming them combined_geli.key, local_geli_1.key and local_geli_2.key.
 - 
 
247 
 
 - 
 
248 
The idea is that you use combined_geli.key to encrypt your disks, then you put local_geli_1.key on a thumb drive and the hex representation of local_geli_2.key into the configuration file.
 - 
 
249 
 
 - 
 
250 
When you have indicated you have a geli setup, you plug the thumb drive in, the system finds it, XOR's it with the hex representation in the config file, generating the combined_geli.key file. After all disks have been decrypted, the system them shreds (erases as per DOD specs) the combined_geli.key. That means even if you lose your thumb drive and it is found by someone else, they can not do anything with it without access to the server, and you can change the local geli keys and generate new ones.
 - 
 
251 
 
 - 
 
252 
If you do not need this additional layer of security, you can simply the combined_geli.key file on your thumb drive and leave the keyfile and localKey empty. In that case, the decryption will assume the file is on the thumb drive and use it.
 - 
 
253 
 
 - 
 
254 
#### Disk Labels
 - 
 
255 
 
 23 
 
- 
 
 24 
Run from the command line. A YAML config file is expected next to the script named
256 
The transport disk, the optional security key USB drive, and the optional report USB are all identified by labels. When the system decides it needs a disk, it will look for the label in /dev/gpt or /dev/msdos (depending on what it thinks it needs), then mount by label.
 25 
`$scriptname.conf.yaml` (the script will create or update it if needed).
257 
 
 26 
 
258 
I chose to use UFS for both the keyfile and the transport disk. Security for the former (most people will assume it is a bad disk and reformat) and file system robustness for the latter. May give me time to create another key pair to invalidate the one lost.
 27 
Basic options:
259 
 
 28 
 
260 
For the report USB, I used FAT32 so the disk could be plugged into a Windows or MacOS machine and read easily.
 29 
- `--dryrun`, `-n`  : run without making destructive changes (no writes)
- 
 
 30 
- `--verbose`, `-v` : increase logging verbosity (can be stacked)
- 
 
 31 
- `--help`, `-h`    : print help and exit
- 
 
 32 
- `--version`, `-V` : print script version and exit
261 
 
 33 
 
262 
To create the disk structure for the transport and key file media, identify it, then run the following, changing *da9* to the disk name for your device. There are 5 locations for it (don't forget da9p1, where the da9 needs to be changed)
 - 
 
- 
 
 34 
Example:
263 
 
 35 
 
264 
```bash
 36 
```bash
265 
# create the GPT schema
 - 
 
266 
gpart create -s GPT da9
 - 
 
267 
# show what is there (nothing yet)
 - 
 
268 
gpart list /dev/da9
 - 
 
269 
# add a new slice of type UFS, and give it a label, sneakernet
 - 
 
270 
# use all of the available disk space for this (ie, only one slice)
 - 
 
271 
gpart add -t freebsd-ufs -l sneakernet da9
 - 
 
272 
# show what is up
 - 
 
273 
gpart list /dev/da9
 - 
 
274 
# format the partition with UFS
 - 
 
275 
newfs /dev/da9p1
 - 
 
276 
# it should now show up as an alias in /dev/gpt
 - 
 
277 
ls /dev/gpt
 - 
 
278 
# make a place to mount it
 - 
 
279 
mkdir /mnt/sneakernet
 - 
 
280 
# mount it
 - 
 
281 
mount /dev/gpt/sneakernet /mnt/sneakernet
 37 
perl sneakernet --dryrun --verbose
282 
# unmount it
 - 
 
283 
umount /mnt/sneakernet
 - 
 
284 
# clean up
 - 
 
285 
rmdir /mnt/sneakernet
 - 
 
286 
```
 38 
```
287 
 
 39 
 
288 
For the report disk, formatted with FAT32, use the following. Again, change all *da9* to the actual device you are using. There are five locations.
 - 
 
- 
 
 40 
## Dependencies
289 
 
 41 
 
- 
 
 42 
- FreeBSD system utilities: `geli`, `zfs`, `zpool`, `geom`, `gpart`, `mount`, `/usr/sbin/sendmail`
- 
 
 43 
- Perl core modules: `strict`, `warnings`, `FindBin`, `Getopt::Long`, `File::Basename`, `Data::Dumper`
- 
 
 44 
- Shared module in repository: `ZFS_Utils.pm` (provides `loadConfig`, `mountDriveByLabel`, `logMsg`, etc.)
- 
 
 45 
- Optional CPAN modules: YAML::XS or YAML::Tiny (used by `ZFS_Utils::loadConfig`)
290 
```bash
 46 
 
291 
# remove any existing schema
 47 
## Configuration (YAML)
- 
 
 48 
 
- 
 
 49 
The script ships with an in-code default `$config` hash that is used as the basis for the
- 
 
 50 
YAML configuration. The following documents the important keys, types and defaults. When you
- 
 
 51 
run `sneakernet` it attempts to `loadConfig($scriptFullPath.conf.yaml, $config)` and will
292 
gpart destroy -F /dev/da9
 52 
create the file from the defaults if it does not exist.
- 
 
 53 
 
293 
# create an mbr schema
 54 
Top-level keys
- 
 
 55 
 
- 
 
 56 
- `dryrun` (bool) — default: `0` — if true, actions that change state are not executed.
- 
 
 57 
- `verbosity` (int) — default: `1` — controls logging verbosity.
- 
 
 58 
- `status_file` (string) — path to status file used to track last replicated snapshots.
294 
gpart create -s mbr /dev/da9
 59 
- `log_file` (string) — path to runtime log file.
- 
 
 60 
 
- 
 
 61 
`source` (hash)
- 
 
 62 
 
295 
# add a fat32 file system labeled REPORT using all the space
 63 
- `hostname` (string) — hostname of the source server (used to detect running role)
296 
gpart add -t fat32 -l REPORT /dev/da9
 64 
- `poolname` (string) — zpool name on the source (default: `pool`)
- 
 
 65 
- `report` (hash)
297 
# format it, using msdos and the label.
 66 
  - `email` (string) — email to send the report to
298 
newfs_msdos -L REPORT -F 32 /dev/da9s1
 67 
  - `subject` (string) — optional subject
299 
# show what you have
 68 
  - `targetDrive` (hash)
- 
 
 69 
    - `fstype` (string) — filesystem type of report drive (ufs/msdos)
300 
gpart show /dev/da9
 70 
    - `check_interval` (int) — polling interval (seconds)
301 
# make a place to mount it
 71 
    - `label` (string) — GPT label of the report drive
302 
mkdir /mnt/REPORT
 72 
    - `mount_point` (string) — optional mount point override
- 
 
 73 
 
303 
# mount it
 74 
`target` (hash)
- 
 
 75 
 
304 
mount -t msdosfs /dev/msdosfs/REPORT /mnt/REPORT
 76 
- `hostname` (string) — hostname of the target server
- 
 
 77 
- `poolname` (string) — zpool name on the target (default: `backup`)
- 
 
 78 
- `shutdown_after_replication` (bool) — default: `0` — if true, attempt to shutdown after completion
- 
 
 79 
- `geli` (hash) — when present, instructs the script to decrypt/mount GELI-protected pool(s):
305 
# ummount it
 80 
  - `secureKey` (hash)
- 
 
 81 
    - `label` (string) — GPT label of the key disk (default: `replica`)
- 
 
 82 
    - `fstype` (string) — filesystem of key disk (default: `ufs`)
- 
 
 83 
    - `check_interval` (int) — polling interval for key disk
- 
 
 84 
    - `wait_timeout` (int) — how long to wait for the key disk
- 
 
 85 
    - `keyfile` (string) — filename of the remote binary key on the key disk (default: `geli.key`)
- 
 
 86 
  - `localKey` (string) — 64-hex-character 256-bit key string or path to file containing hex
- 
 
 87 
  - `target` (string) — path where the combined keyfile should be written (e.g. `/media/geli.key`)
- 
 
 88 
  - `poolname` (string) — pool name to import on target
- 
 
 89 
  - `diskList` (array) — optional list of device names to try (e.g. `['da0','da1']`)
- 
 
 90 
 
306 
umount /mnt/REPORT
 91 
`transport` (hash)
- 
 
 92 
 
- 
 
 93 
- `label` (string) — GPT label of the transport drive (default: `sneakernet`)
- 
 
 94 
- `fstype` (string) — filesystem type for mounting (default: `ufs`)
- 
 
 95 
- `mount_point` (string) — target mount point (default in sample: `/mnt/sneakernet`)
- 
 
 96 
- `timeout` (int) — how long to wait for the transport device to appear (seconds)
- 
 
 97 
- `check_interval` (int) — polling interval when waiting for the device (seconds)
307 
# clean up
 98 
- `encryption` (hash)
- 
 
 99 
  - `key` (string) — hex key used by `openssl enc -aes-256-cbc` for transport encryption
- 
 
 100 
  - `IV` (string) — hex IV used by encryption (defaults to zeros in sample)
- 
 
 101 
 
308 
rmdir /mnt/REPORT/
 102 
`datasets` (hash)
309 
```
 103 
 
- 
 
 104 
- Keys are logical dataset names (user-defined blocks). Each dataset object contains:
- 
 
 105 
  - `source` — parent or root dataset on the source (string)
- 
 
 106 
  - `target` — parent or root dataset on the target (string)
- 
 
 107 
  - `dataset` — dataset name
310 
 
 108 
 
311 
## Functions
 109 
Example minimal YAML snippet (derived from the script defaults):
312 
 
 110 
 
313 
- getStatusFile
 111 
```yaml
314
112 
dryrun: 0
315 
  - Reads the status file
 113 
log_file: /path/to/sneakernet.log
316
114 
source:
317 
  - Parameters: file name
 115 
  hostname: source-host
- 
 
 116 
  poolname: pool
318
117 
target:
319 
  - Returns: contents of status file as an array, one entry per line
 118 
  hostname: target-host
320 
 
 - 
 
321 
- writeStatusFile
 119 
  poolname: backup
322
120 
  geli:
- 
 
 121 
    secureKey:
- 
 
 122 
      label: replica
- 
 
 123 
      keyfile: geli.key
- 
 
 124 
    localKey: e98c66...bc9c
323 
  - writes updated status file. Old status file saved with .bak suffix (only one backup)
 125 
    target: /media/geli.key
324
126 
transport:
- 
 
 127 
  label: sneakernet
325 
  - Paramters: file name, array ref
 128 
  mount_point: /mnt/sneakernet
- 
 
 129 
datasets:
326
130 
  files_share:
327 
  - Returns: nothing
 131 
    source: pool
- 
 
 132 
    target: backup
- 
 
 133 
    dataset: files_share
- 
 
 134 
```
328 
 
 135 
 
329 
- replaceSlashWithDot
 136 
## Functions (script-level / documented)
330
137 
 
- 
 
 138 
The following functions are defined inside `sneakernet` and are documented here. Many
331 
  - Replaces all slashes (used in pathnames) with periods so the path can be used as a file name. Removes first element
 139 
helpers used by the script are provided by `ZFS_Utils.pm` (imported at the top of the script).
332
140 
 
333 
  - Parameters: string
 141 
- `getStatusFile($filename)`
- 
 
 142 
  - Returns: ARRAYREF of status lines (snapshot names). Reads `$filename` if present; returns
- 
 
 143 
    empty arrayref and logs an informational message if file missing or unreadable.
334
144 
 
- 
 
 145 
- `writeStatusFile($filename, $statusList)`
335 
  - Returns: string with all slashes removed and first entry removed
 146 
  - Writes the provided ARRAYREF of status lines to `$filename`.
- 
 
 147 
  - Behavior: backs up an existing file to `$filename.bak` before writing. Dies on failure.
336 
 
 148 
 
337 
- doSourceReplication
 149 
- `dirnameToFileName($string, $delimiter='/', $substitution='.')`
- 
 
 150 
  - Utility to turn dataset-like strings into filename-safe strings. Example: `pool/fs/sub` -> `pool.fs.sub`.
338
151 
 
- 
 
 152 
- `doSourceReplication($config, $statusList)`
- 
 
 153 
  - Performs replication on the source server.
- 
 
 154 
  - Behavior: enumerates source snapshots, builds zfs send commands with `makeReplicateCommands` (from `ZFS_Utils`),
339 
  - This is the meat of the script when in source mode; it finds all snapshots, calculates the new ones, creates and executes the commands to write them
 155 
    optionally pipes through `openssl enc` (if transport encryption key is set), and writes send streams to files on the
- 
 
 156 
    transport mount point. Honors `$config->{dryrun}`.
- 
 
 157 
  - Returns: `$newStatus` ARRAYREF of updated status lines.
340
158 
 
- 
 
 159 
- `cleanup($config, $message)`
341 
  - Parameters: config, status file contents
 160 
  - Performs final cleanup and reporting actions.
- 
 
 161 
  - Behavior: logs disk usage and zpool list, attempts to unmount the transport drive, sends report via `sendReport`, and
- 
 
 162 
    optionally shuts down the machine if configured. Honors `dryrun`.
342
163 
 
- 
 
 164 
- `updateTarget($config)`
- 
 
 165 
  - Reads files from the transport disk and feeds them into `zfs receive` to update target datasets.
- 
 
 166 
  - Behavior: detects file->dataset mapping via the filename (uses `dirnameToFileName` reversal), optionally decrypts
343 
  - Returns: hashref of new "last snapshot used" for new Status File
 167 
    with `openssl enc -d` if encryption was used, and calls `zfs receive -F` for each file.
344 
 
 168 
 
345 
- cleanDirectory
 169 
## Main flow summary
346
170 
 
- 
 
 171 
1. Load YAML config using `ZFS_Utils::loadConfig` with the default config provided in the script.
- 
 
 172 
2. Parse CLI flags (dryrun, verbose, help, version). CLI flags override config when present.
- 
 
 173 
3. Determine whether the running host matches `source.hostname` or `target.hostname` and set `runningAs` accordingly.
- 
 
 174 
4. Mount the transport drive (fatal error if not found) using `ZFS_Utils::mountDriveByLabel`.
- 
 
 175 
5. If running as source:
347 
  - Removes all files from a given directory. Will not touch subdirectories. Used to clean Transport Disk before source server begins transfer
 176 
   - Clean transport directory (non-recursive), produce send streams and write to files on transport drive.
- 
 
 177 
   - Update status file with newest snapshots.
- 
 
 178 
6. If running as target:
- 
 
 179 
   - If `target.geli` present, attempt to decrypt/mount GELI disks (via `ZFS_Utils::mountGeli`).
- 
 
 180 
   - Update target datasets by reading files from transport and running `zfs receive`.
- 
 
 181 
7. Run `cleanup()` to unmount, send reports, and optionally shutdown.
348
182 
 
349 
  - Parameters: directory name
 183 
## Logging & Reports
350
184 
 
- 
 
 185 
- The script uses `ZFS_Utils::logMsg` throughout. Default log path is set from `$config->{log_file}` and
- 
 
 186 
  the module exposes `$logFileName` and `$displayLogsOnConsole` for customizing behavior at runtime.
- 
 
 187 
- Reports can be saved to a drive (via `mountDriveByLabel`) and/or emailed via `/usr/sbin/sendmail` using
351 
  - Returns: nothing
 188 
  `ZFS_Utils::sendReport`.
352 
 
 189 
 
353 
- fatalError
 190 
## Security notes
354
191 
 
- 
 
 192 
- GELI combined keys are created by XOR'ing a remote binary key and a local 256-bit hex key — the resulting
355 
  - Adds a message to the log, then performs a die. Used for critical failure
 193 
  key is written with mode `0600`. Keep these files and the key disk physically secure.
- 
 
 194 
- Transport encryption uses `openssl enc -aes-256-cbc`; manage the encryption key material carefully.
356
195 
 
357 
  - Parameters: string to add to log
 196 
## Example quick-run checklist
358
197 
 
- 
 
 198 
1. Edit `sneakernet.conf.yaml` (create from defaults if necessary) and confirm `transport.mount_point` and `label`.
- 
 
 199 
2. On source: run `perl sneakernet --dryrun --verbose` to validate the planned commands.
359 
  - Returns: nothing, kills program
 200 
3. On source: run without `--dryrun` to execute replication.
- 
 
 201 
4. Physically move the drive to the target, insert it, and run the script on the target host.
360 
 
 202 
 
361 
Contributing
 203 
## Troubleshooting
- 
 
 204 
 
- 
 
 205 
- If the transport drive does not mount, check that the GPT label matches `transport.label` and that the filesystem
- 
 
 206 
  type matches `transport.fstype`.
- 
 
 207 
- If GELI attach fails, verify the keyfile exists on the secure key disk and that the local key hex string is correct
- 
 
 208 
  (exactly 64 hex characters) and that the combined keyfile is created at the configured `target` path.
- 
 
 209 
- Use `--dryrun` and `--verbose` to inspect command strings before running them.
362 
 
 210 
 
363 
- Follow semantic versioning for manifest schema changes.
 - 
 
364 
- Tests should simulate inconsistent snapshots in child datasets
 - 
 
- 
 
 211 
---
365 
 
 212 
 
366 
License
 213 
Document last updated: 2025-12-15
367 
 
 214 
 
368 
- Choose an appropriate license (MIT, Apache-2.0, etc.) for your project.
 - 
 
369 
 
 215