Subversion Repositories perlutils

# perlConfigFileUtility

A Perl utility for managing configuration files with template merging, interactive editing, and format conversion capabilities.

## Overview

`perlConfigFileUtility` is designed to simplify configuration file management by:
- Merging template structures with existing configuration files
- Providing an interactive hierarchical editor for modifying configurations
- Converting between YAML and JSON formats
- Automatically backing up config files when saving changes

## Background

This utility solves two primary problems:

1. **User-Friendly Editing**: Enables users unfamiliar with JSON/YAML syntax to safely edit configuration files through an interactive interface.

2. **Version Compatibility**: Handles configuration file evolution across software versions. When new key/value pairs are added to the application, the utility merges them with existing config files, ensuring older configurations remain compatible with newer software.

### Implementation Approach

- Default configuration is stored as a Perl data structure in a separate file (loaded via `do` statement in project script)
- This template can be shared across multiple applications
- The utility loads existing YAML/JSON configs, merges missing values from the template, and presents an interactive editor
- Format conversion (YAML ↔ JSON) was added as a natural extension of the merge functionality

## Features

- **Template Merging**: Combine Perl data structure templates with existing YAML/JSON config files
- **Interactive Editor**: Navigate and edit nested hashes and arrays with a menu-driven interface
- **Dynamic Modifications**: Add new keys to hashes and elements to arrays on the fly
- **Delete Operations**: Remove keys from hashes and elements from arrays, including all nested children
- **Format Conversion**: Seamlessly convert between YAML and JSON configuration formats
- **Safe Updates**: Automatic backup creation before saving changes

## Usage

```bash
# Basic usage
perlConfigFileUtility [-t <template_file>] [-c <config_file>] [-o <output_file>] [-e]

# Positional arguments (backward compatibility)
perlConfigFileUtility <template_file> [config_file]
```

## Example of template file

The template file is a Perl hash reference that defines the complete configuration structure with default values and inline documentation. Below is a simplified excerpt from the `sneakernet` project's datastructure file (see `../zfs_utils/sneakernet/sneakernet.datastructure` for the full example).

```perl
# Default configuration structure for sneakernet
# This file is loaded by the sneakernet script to provide default configuration values
# Variables from $programDefinition hashref (scriptDirectory, scriptFullPath) will be interpolated at runtime

{
   'dryrun' => 0, # if set to 1, will not perform any changes, just log what would be done
   'verbosity' => 1, # verbosity level for logging
   'debug' => 0, # set to number greater than 0 for debugging program
   'status_file' => "<scriptDirectory>/sneakernet_target.status", # file created on source server to track last copied dataset
   'log_file' => "<scriptFullPath>.log",
   'displayLogsOnTTY' => '', # if set to path of a tty, log messages will be sent there also. Example: /dev/ttyv1
   'displayLogsOnConsole' => 1, # If set to non-zero, will send log messages to screen in addition to log file
   'source' => { # information about source server
      'hostname' => '', # used to see if we are on source
      'poolname' => 'pool', # name of the ZFS default parent pool to export
      'cleanUpScriptsDir' => '<scriptDirectory>/cleanupScripts', # location on disk where scripts to be sent to target are located
      'report' => { # if set, will generate a report via email or by storing on a drive
         'email' => '', # if set, will email the report to this address
         'subject' => '', # subject of the report email (will be auto-generated if empty)
         'targetDrive' => { # if set, will store the report on this drive
            'label' => '', # the GPT or msdos label of the report drive, REQUIRED
            'fstype' => '', # filesystem type of the report drive, default is msdos
            'check_interval' => 15, # how often to check for the disk (seconds), message displayed every interval
            'wait_timeout' => 300, # time to wait for the disk to appear in seconds
            'mount_point' => '', # where to mount the report drive, default is /mnt/label
         }
      }
   },
   # ... additional sections: 'target', 'transport', 'datasets' (see full file)
}
```

### How it works

The template file serves dual purposes:

1. **Default Values**: The Perl script loads it at runtime using `$config = do 'scriptname.datastructure'`, which returns the hash reference with all defaults.

2. **Configuration Schema**: The inline comments document each setting's purpose, forming a self-documenting configuration schema that can be exported to YAML format (e.g., `scriptname.conf.yaml`).

When the datastructure changes, this tool can non-destructively merge the updated template with existing configuration files, preserving user customizations while adding new keys and documentation. The interactive edit mode (`-e`) provides safe, menu-driven configuration management.

## Options

| Option | Description |
|--------|-------------|
| `-t, --template <file>` | Template file (Perl hashref) |
| `-c, --config <file>` | Config file (YAML or JSON) |
| `-o, --output <file>` | Output file (default: STDOUT) |
| `-e, --edit` | Interactive edit mode |
| `-v, --version` | Show version information |
| `-h, --help` | Show help message |

## Examples

### Merge Template with Existing Config
```bash
perlConfigFileUtility -t template.pl -c config.yaml -o output.yaml
```

### Interactive Editing
```bash
perlConfigFileUtility -c config.yaml -e
```
Navigate through nested structures, modify values, add new keys/elements, or delete existing ones.

### Format Conversion
```bash
# YAML to JSON
perlConfigFileUtility -c config.yaml -o config.json

# JSON to YAML
perlConfigFileUtility -c config.json -o config.yaml
```

### Update Config with New Template Keys
```bash
# Merge new template keys into existing config, preserving user values
perlConfigFileUtility -t myapp.datastructure -c myapp.conf.yaml -o myapp.conf.yaml
```

### Start Fresh from Template
```bash
# Export template as YAML config
perlConfigFileUtility -t myapp.datastructure -o myapp.conf.yaml
```

## Interactive Editor Commands

When in edit mode (`-e`), the following commands are available:

- **Number (1-N)**: Select and edit the numbered item
- **0 or Enter**: Go back to previous level
- **a**: Add new key (in hash) or element (in array)
- **d**: Delete a key (from hash) or element (from array)
- **q**: Quit and optionally save changes

When editing:
- **Scalars**: Enter new value or press Enter to keep current
- **Hashes**: Navigate into nested structure
- **Arrays**: Navigate into nested structure or edit elements

## Requirements

### Perl Modules

**Required:**
- Perl 5.x or higher
- `File::Slurp` - File reading utilities
- `Data::Dumper` - Data structure serialization
- `Getopt::Long` - Command-line option parsing

**At least one YAML library:**
- `YAML::XS` (recommended, fastest)
- `YAML::Tiny`
- `YAML`

**At least one JSON library:**
- `JSON::XS` (recommended, fastest)
- `JSON::PP`
- `JSON`

### Installation

**Using CPAN:**
```bash
cpan File::Slurp YAML::XS JSON::XS
```

**Using cpanm:**
```bash
cpanm File::Slurp YAML::XS JSON::XS
```

**Debian/Ubuntu:**
```bash
sudo apt-get install libfile-slurp-perl libyaml-perl libjson-xs-perl
```

**RedHat/CentOS:**
```bash
sudo yum install perl-File-Slurp perl-YAML perl-JSON-XS
```

**FreeBSD:**
```bash
sudo pkg install p5-File-Slurp p5-YAML p5-JSON-XS
```

### Source Code

This script is part of the Subversion repository and may be downloaded, checked out, or exported from:

```bash
# Checkout the entire repository
svn checkout http://svn.dailydata.net/svn/perlutils/trunk perlutils

# Export without version control metadata
svn export http://svn.dailydata.net/svn/perlutils/trunk perlutils

# Download just this script
svn export http://svn.dailydata.net/svn/perlutils/trunk/perlConfigFileUtility
```

## Use Cases

### 1. User-Friendly Configuration Editing

Enable non-technical users to modify complex configuration files without risking syntax errors:
```bash
perlConfigFileUtility -c /etc/myapp/config.yaml -e
```

### 2. Configuration Version Management

When software is updated with new configuration options, merge them into existing configs:
```bash
# v2.0 adds new keys to template
perlConfigFileUtility -t myapp-v2.datastructure -c myapp-v1.conf.yaml -o myapp-v2.conf.yaml
```
User customizations are preserved while new defaults are added.

### 3. Configuration Format Migration

Convert existing configs between formats:
```bash
# Organization switches from JSON to YAML
for file in configs/*.json; do
  perlConfigFileUtility -c "$file" -o "${file%.json}.yaml"
done
```

### 4. Configuration Deployment

Generate initial configuration files from templates:
```bash
perlConfigFileUtility -t defaults.pl -o /etc/newapp/config.yaml
```

## Technical Details

### Template Files

Template files are Perl data structures (hashrefs) that define the configuration schema:

```perl
{
   'setting1' => 'default_value',
   'section' => {
      'nested_setting' => 123,
      'array_setting' => ['item1', 'item2']
   }
}
```

Inline comments in templates serve as documentation and are preserved when exported to YAML.

### Merge Behavior

- **Missing keys**: Added from template
- **Existing keys**: Preserved from config (user values not overwritten)
- **Nested hashes**: Recursively merged
- **Arrays**: Extended if template is longer; existing elements preserved
- **Type mismatches**: Config value takes precedence

### Backup Strategy

When saving changes, the original file is renamed with `.bak` extension:
```
config.yaml → config.yaml.bak  (old version)
config.yaml → (new version)
```

## Troubleshooting

**No YAML/JSON library error:**
Install at least one YAML and one JSON library using your system package manager or CPAN.

**Template must be a hashref error:**
Ensure template file contains a single hashref structure (starts with `{` and ends with `}`).

**Config must be a hashref error:**
The utility only works with hash-based configurations, not arrays or scalars at the root level.

**Permission denied errors:**
Ensure you have write permissions for the output directory when saving files.

## License

Simplified BSD License (FreeBSD License)  
Copyright (c) 2026, Daily Data Inc.

## Author

R. W. Rodolico <rodo@dailydata.net>

## Version

1.1.0 (January 2026)

### Version History

- **1.1.0** (2026-01-15): Added ability to delete keys from hashes and elements from arrays
- **1.0** (2026-01-13): Initial release with merge, edit, and format conversion capabilities