Subversion Repositories perlutils

# perlConfigFileUtility

A Perl utility for managing configuration files with template merging, interactive editing, and format conversion capabilities.

## Overview

`perlConfigFileUtility` is designed to simplify configuration file management by:

- Merging template structures with existing configuration files

- Providing an interactive hierarchical editor for modifying configurations

- Converting between YAML and JSON formats

- Automatically backing up config files when saving changes

## Background

This utility solves two primary problems:

1. **User-Friendly Editing**: Enables users unfamiliar with JSON/YAML syntax to safely edit configuration files through an interactive interface.

2. **Version Compatibility**: Handles configuration file evolution across software versions. When new key/value pairs are added to the application, the utility merges them with existing config files, ensuring older configurations remain compatible with newer software.

### Implementation Approach

- Default configuration is stored as a Perl data structure in a separate file (loaded via `do` statement in project script)

- This template can be shared across multiple applications

- The utility loads existing YAML/JSON configs, merges missing values from the template, and presents an interactive editor

- Format conversion (YAML ↔ JSON) was added as a natural extension of the merge functionality

## Features

- **Template Merging**: Combine Perl data structure templates with existing YAML/JSON config files

- **Interactive Editor**: Navigate and edit nested hashes and arrays with a menu-driven interface

- **Dynamic Modifications**: Add new keys to hashes and elements to arrays on the fly

- **Rename Keys**: Rename hash keys while preserving their values in interactive edit mode

- **Delete Operations**: Remove keys from hashes and elements from arrays, including all nested children

- **Config Comparison**: Load and compare multiple config files to identify differences

- **Format Conversion**: Seamlessly convert between YAML and JSON configuration formats

- **Safe Updates**: Automatic backup creation before saving changes

## Usage

```bash

# Basic usage

perlConfigFileUtility [-t <template_file>] [-c <config_file>] [-o <output_file>] [-e] [-C]

# Positional arguments (backward compatibility)

perlConfigFileUtility <template_file> [config_file]

# Compare multiple config files

perlConfigFileUtility -C -c <file1> -c <file2> [-c <file3> ...]

```

## Example of template file

The template file is a Perl hash reference that defines the complete configuration structure with default values and inline documentation. Below is a simplified excerpt from the `sneakernet` project's datastructure file (see `../zfs_utils/sneakernet/sneakernet.datastructure` for the full example).

```perl

# Default configuration structure for sneakernet

# This file is loaded by the sneakernet script to provide default configuration values

# Variables from $programDefinition hashref (scriptDirectory, scriptFullPath) will be interpolated at runtime

{

   'dryrun' => 0, # if set to 1, will not perform any changes, just log what would be done

   'verbosity' => 1, # verbosity level for logging

   'debug' => 0, # set to number greater than 0 for debugging program

   'status_file' => "<scriptDirectory>/sneakernet_target.status", # file created on source server to track last copied dataset

   'log_file' => "<scriptFullPath>.log",

   'displayLogsOnTTY' => '', # if set to path of a tty, log messages will be sent there also. Example: /dev/ttyv1

   'displayLogsOnConsole' => 1, # If set to non-zero, will send log messages to screen in addition to log file

   'source' => { # information about source server

      'hostname' => '', # used to see if we are on source

      'poolname' => 'pool', # name of the ZFS default parent pool to export

      'cleanUpScriptsDir' => '<scriptDirectory>/cleanupScripts', # location on disk where scripts to be sent to target are located

      'report' => { # if set, will generate a report via email or by storing on a drive

         'email' => '', # if set, will email the report to this address

         'subject' => '', # subject of the report email (will be auto-generated if empty)

         'targetDrive' => { # if set, will store the report on this drive

            'label' => '', # the GPT or msdos label of the report drive, REQUIRED

            'fstype' => '', # filesystem type of the report drive, default is msdos

            'check_interval' => 15, # how often to check for the disk (seconds), message displayed every interval

            'wait_timeout' => 300, # time to wait for the disk to appear in seconds

            'mount_point' => '', # where to mount the report drive, default is /mnt/label

         }

      }

   },

   # ... additional sections: 'target', 'transport', 'datasets' (see full file)

}

```

### How it works

The template file serves dual purposes:

1. **Default Values**: The Perl script loads it at runtime using `$config = do 'scriptname.datastructure'`, which returns the hash reference with all defaults.

2. **Configuration Schema**: The inline comments document each setting's purpose, forming a self-documenting configuration schema that can be exported to YAML format (e.g., `scriptname.conf.yaml`).

When the datastructure changes, this tool can non-destructively merge the updated template with existing configuration files, preserving user customizations while adding new keys and documentation. The interactive edit mode (`-e`) provides safe, menu-driven configuration management.

## Options

| Option | Description |

|--------|-------------|

| `-t, --template <file>` | Template file (Perl hashref) |

| `-c, --config <file>` | Config file (YAML or JSON) - can specify multiple for comparison |

| `-o, --output <file>` | Output file (default: STDOUT) |

| `-e, --edit` | Interactive edit mode |

| `-C, --compare` | Compare multiple config files and show differences |

| `-v, --version` | Show version information |

| `-h, --help` | Show help message |

## Examples

### Merge Template with Existing Config

```bash

perlConfigFileUtility -t template.pl -c config.yaml -o output.yaml

```

### Interactive Editing

```bash

perlConfigFileUtility -c config.yaml -e

```

Navigate through nested structures, modify values, add new keys/elements, rename keys, or delete existing ones.

### Compare Multiple Config Files

```bash

# Compare two config files

perlConfigFileUtility -C -c production.yaml -c staging.yaml

# Compare three or more files

perlConfigFileUtility -C -c prod.yaml -c staging.yaml -c dev.yaml

# Save comparison report to file

perlConfigFileUtility -C -c config1.yaml -c config2.yaml -o comparison.txt

```

Shows all differences including missing keys and differing values.

### Format Conversion

```bash

# YAML to JSON

perlConfigFileUtility -c config.yaml -o config.json

# JSON to YAML

perlConfigFileUtility -c config.json -o config.yaml

```

### Update Config with New Template Keys

```bash

# Merge new template keys into existing config, preserving user values

perlConfigFileUtility -t myapp.datastructure -c myapp.conf.yaml -o myapp.conf.yaml

```

### Start Fresh from Template

```bash

# Export template as YAML config

perlConfigFileUtility -t myapp.datastructure -o myapp.conf.yaml

```

## Interactive Editor Commands

When in edit mode (`-e`), the following commands are available:

- **Number (1-N)**: Select and edit the numbered item

- **0 or Enter**: Go back to previous level

- **a**: Add new key (in hash) or element (in array)

- **r**: Rename a key (in hash only)

- **d**: Delete a key (from hash) or element (from array)

- **q**: Quit and optionally save changes

When editing:

- **Scalars**: Enter new value or press Enter to keep current

- **Hashes**: Navigate into nested structure

- **Arrays**: Navigate into nested structure or edit elements

## Requirements

### Perl Modules

**Required:**

- Perl 5.x or higher

- `File::Slurp` - File reading utilities

- `Data::Dumper` - Data structure serialization

- `Getopt::Long` - Command-line option parsing

**At least one YAML library:**

- `YAML::XS` (recommended, fastest)

- `YAML::Tiny`

- `YAML`

**At least one JSON library:**

- `JSON::XS` (recommended, fastest)

- `JSON::PP`

- `JSON`

### Installation

**Using CPAN:**

```bash

cpan File::Slurp YAML::XS JSON::XS

```

**Using cpanm:**

```bash

cpanm File::Slurp YAML::XS JSON::XS

```

**Debian/Ubuntu:**

```bash

sudo apt-get install libfile-slurp-perl libyaml-perl libjson-xs-perl

```

**RedHat/CentOS:**

```bash

sudo yum install perl-File-Slurp perl-YAML perl-JSON-XS

```

**FreeBSD:**

```bash

sudo pkg install p5-File-Slurp p5-YAML p5-JSON-XS

```

### Source Code

This script is part of the Subversion repository and may be downloaded, checked out, or exported from:

```bash

# Checkout the entire repository

svn checkout http://svn.dailydata.net/svn/perlutils/trunk perlutils

# Export without version control metadata

svn export http://svn.dailydata.net/svn/perlutils/trunk perlutils

# Download just this script

svn export http://svn.dailydata.net/svn/perlutils/trunk/perlConfigFileUtility

```

## Use Cases

### 1. User-Friendly Configuration Editing

Enable non-technical users to modify complex configuration files without risking syntax errors:

```bash

perlConfigFileUtility -c /etc/myapp/config.yaml -e

```

### 2. Configuration Version Management

When software is updated with new configuration options, merge them into existing configs:

```bash

# v2.0 adds new keys to template

perlConfigFileUtility -t myapp-v2.datastructure -c myapp-v1.conf.yaml -o myapp-v2.conf.yaml

```

User customizations are preserved while new defaults are added.

### 3. Configuration Format Migration

Convert existing configs between formats:

```bash

# Organization switches from JSON to YAML

for file in configs/*.json; do

  perlConfigFileUtility -c "$file" -o "${file%.json}.yaml"

done

```

### 4. Configuration Deployment

Generate initial configuration files from templates:

```bash

perlConfigFileUtility -t defaults.pl -o /etc/newapp/config.yaml

```

### 5. Configuration Comparison and Auditing

Compare config files across environments to identify discrepancies:

```bash

# Compare production vs staging configurations

perlConfigFileUtility -C -c /etc/app/prod.yaml -c /etc/app/staging.yaml

# Audit multiple server configs

perlConfigFileUtility -C -c server1.yaml -c server2.yaml -c server3.yaml -o audit-report.txt

```

Identifies missing keys, type mismatches, and value differences for troubleshooting and compliance.

## Technical Details

### Template Files

Template files are Perl data structures (hashrefs) that define the configuration schema:

```perl

{

   'setting1' => 'default_value',

   'section' => {

      'nested_setting' => 123,

      'array_setting' => ['item1', 'item2']

   }

}

```

Inline comments in templates serve as documentation and are preserved when exported to YAML.

### Merge Behavior

- **Missing keys**: Added from template

- **Existing keys**: Preserved from config (user values not overwritten)

- **Nested hashes**: Recursively merged

- **Arrays**: Extended if template is longer; existing elements preserved

- **Type mismatches**: Config value takes precedence

### Backup Strategy

When saving changes, the original file is renamed with `.bak` extension:

```

config.yaml → config.yaml.bak  (old version)

config.yaml → (new version)

```

## Troubleshooting

**No YAML/JSON library error:**

Install at least one YAML and one JSON library using your system package manager or CPAN.

**Template must be a hashref error:**

Ensure template file contains a single hashref structure (starts with `{` and ends with `}`).

**Config must be a hashref error:**

The utility only works with hash-based configurations, not arrays or scalars at the root level.

**Permission denied errors:**

Ensure you have write permissions for the output directory when saving files.

## License

Simplified BSD License (FreeBSD License)  

Copyright (c) 2026, Daily Data Inc.

## Author

R. W. Rodolico <rodo@dailydata.net>

## Version

1.2.0 (January 2026)

### Version History

- **1.2.0** (2026-01-18): 

  - Added config file comparison mode to identify differences across multiple files

  - Added ability to rename hash keys in interactive edit mode

  - Refactored long functions for improved code readability and maintainability

- **1.1.0** (2026-01-15): Added ability to delete keys from hashes and elements from arrays

- **1.0** (2026-01-13): Initial release with merge, edit, and format conversion capabilities