Rev 6 | Blame | Compare with Previous | Last modification | View Log | Download | RSS feed
# configFileEditor
A Perl utility for managing configuration files with template merging, interactive editing, and format conversion capabilities.
## Overview
`configFileEditor` is designed to simplify configuration file management by:
- Merging template structures with existing configuration files
- Providing an interactive hierarchical editor for modifying configurations
- Converting between YAML and JSON formats
- Automatically backing up config files when saving changes
## Background
This utility solves two primary problems:
1. **User-Friendly Editing**: Enables users unfamiliar with JSON/YAML syntax to safely edit configuration files through an interactive interface.
2. **Version Compatibility**: Handles configuration file evolution across software versions. When new key/value pairs are added to the application, the utility merges them with existing config files, ensuring older configurations remain compatible with newer software.
### Implementation Approach
- Default configuration is stored as a Perl data structure in a separate file (loaded via `do` statement in project script)
- This template can be shared across multiple applications
- The utility loads existing YAML/JSON configs, merges missing values from the template, and presents an interactive editor
- Format conversion (YAML ↔ JSON) was added as a natural extension of the merge functionality
## Features
- **Template Merging**: Combine Perl data structure templates with existing YAML/JSON config files
- **Interactive Editor**: Navigate and edit nested hashes and arrays with a menu-driven interface
- **Dynamic Modifications**: Add new keys to hashes and elements to arrays on the fly
- **Rename Keys**: Rename hash keys while preserving their values in interactive edit mode
- **Delete Operations**: Remove keys from hashes and elements from arrays, including all nested children
- **Config Comparison**: Load and compare multiple config files to identify differences
- **Format Conversion**: Seamlessly convert between YAML and JSON configuration formats
- **Safe Updates**: Automatic backup creation before saving changes
## Usage
```bash
# Basic usage
configFileEditor [-t <template_file>] [-c <config_file>] [-o <output_file>] [-e] [-C]
# Positional arguments (backward compatibility)
configFileEditor <template_file> [config_file]
# Compare multiple config files
configFileEditor -C -c <file1> -c <file2> [-c <file3> ...]
```
## Example of template file
The template file is a Perl hash reference that defines the complete configuration structure with default values and inline documentation. Below is a simplified excerpt from the `sneakernet` project's datastructure file (see `../zfs_utils/sneakernet/sneakernet.datastructure` for the full example).
```perl
# Default configuration structure for sneakernet
# This file is loaded by the sneakernet script to provide default configuration values
# Variables from $programDefinition hashref (scriptDirectory, scriptFullPath) will be interpolated at runtime
{
'dryrun' => 0, # if set to 1, will not perform any changes, just log what would be done
'verbosity' => 1, # verbosity level for logging
'debug' => 0, # set to number greater than 0 for debugging program
'status_file' => "<scriptDirectory>/sneakernet_target.status", # file created on source server to track last copied dataset
'log_file' => "<scriptFullPath>.log",
'displayLogsOnTTY' => '', # if set to path of a tty, log messages will be sent there also. Example: /dev/ttyv1
'displayLogsOnConsole' => 1, # If set to non-zero, will send log messages to screen in addition to log file
'source' => { # information about source server
'hostname' => '', # used to see if we are on source
'poolname' => 'pool', # name of the ZFS default parent pool to export
'cleanUpScriptsDir' => '<scriptDirectory>/cleanupScripts', # location on disk where scripts to be sent to target are located
'report' => { # if set, will generate a report via email or by storing on a drive
'email' => '', # if set, will email the report to this address
'subject' => '', # subject of the report email (will be auto-generated if empty)
'targetDrive' => { # if set, will store the report on this drive
'label' => '', # the GPT or msdos label of the report drive, REQUIRED
'fstype' => '', # filesystem type of the report drive, default is msdos
'check_interval' => 15, # how often to check for the disk (seconds), message displayed every interval
'wait_timeout' => 300, # time to wait for the disk to appear in seconds
'mount_point' => '', # where to mount the report drive, default is /mnt/label
}
}
},
# ... additional sections: 'target', 'transport', 'datasets' (see full file)
}
```
### How it works
The template file serves dual purposes:
1. **Default Values**: The Perl script loads it at runtime using `$config = do 'scriptname.datastructure'`, which returns the hash reference with all defaults.
2. **Configuration Schema**: The inline comments document each setting's purpose, forming a self-documenting configuration schema that can be exported to YAML format (e.g., `scriptname.conf.yaml`).
When the datastructure changes, this tool can non-destructively merge the updated template with existing configuration files, preserving user customizations while adding new keys and documentation. The interactive edit mode (`-e`) provides safe, menu-driven configuration management.
## Options
| Option | Description |
|--------|-------------|
| `-t, --template <file>` | Template file (Perl hashref) |
| `-c, --config <file>` | Config file (YAML or JSON) - can specify multiple for comparison |
| `-o, --output <file>` | Output file (default: STDOUT) |
| `-e, --edit` | Interactive edit mode |
| `-C, --compare` | Compare multiple config files and show differences |
| `-v, --version` | Show version information |
| `-h, --help` | Show help message |
## Examples
### Merge Template with Existing Config
```bash
configFileEditor -t template.pl -c config.yaml -o output.yaml
```
### Interactive Editing
```bash
configFileEditor -c config.yaml -e
```
Navigate through nested structures, modify values, add new keys/elements, rename keys, or delete existing ones.
### Compare Multiple Config Files
```bash
# Compare two config files
configFileEditor -C -c production.yaml -c staging.yaml
# Compare three or more files
configFileEditor -C -c prod.yaml -c staging.yaml -c dev.yaml
# Save comparison report to file
configFileEditor -C -c config1.yaml -c config2.yaml -o comparison.txt
```
Shows all differences including missing keys and differing values.
### Format Conversion
```bash
# YAML to JSON
configFileEditor -c config.yaml -o config.json
# JSON to YAML
configFileEditor -c config.json -o config.yaml
```
### Update Config with New Template Keys
```bash
# Merge new template keys into existing config, preserving user values
configFileEditor -t myapp.datastructure -c myapp.conf.yaml -o myapp.conf.yaml
```
### Start Fresh from Template
```bash
# Export template as YAML config
configFileEditor -t myapp.datastructure -o myapp.conf.yaml
```
## Interactive Editor Commands
When in edit mode (`-e`), the following commands are available:
- **Number (1-N)**: Select and edit the numbered item
- **0 or Enter**: Go back to previous level
- **a**: Add new key (in hash) or element (in array)
- **r**: Rename a key (in hash only)
- **d**: Delete a key (from hash) or element (from array)
- **q**: Quit and optionally save changes
When editing:
- **Scalars**: Enter new value or press Enter to keep current
- **Hashes**: Navigate into nested structure
- **Arrays**: Navigate into nested structure or edit elements
## Requirements
### Perl Modules
**Required:**
- Perl 5.x or higher
- `File::Slurp` - File reading utilities
- `Data::Dumper` - Data structure serialization
- `Getopt::Long` - Command-line option parsing
**At least one YAML library:**
- `YAML::XS` (recommended, fastest)
- `YAML::Tiny`
- `YAML`
**At least one JSON library:**
- `JSON::XS` (recommended, fastest)
- `JSON::PP`
- `JSON`
### Installation
**Using CPAN:**
```bash
cpan File::Slurp YAML::XS JSON::XS
```
**Using cpanm:**
```bash
cpanm File::Slurp YAML::XS JSON::XS
```
**Debian/Ubuntu:**
```bash
sudo apt-get install libfile-slurp-perl libyaml-perl libjson-xs-perl
```
**RedHat/CentOS:**
```bash
sudo yum install perl-File-Slurp perl-YAML perl-JSON-XS
```
**FreeBSD:**
```bash
sudo pkg install p5-File-Slurp p5-YAML p5-JSON-XS
```
### Source Code
This script is part of the Subversion repository and may be downloaded, checked out, or exported from:
```bash
# Checkout the entire repository
svn checkout http://svn.dailydata.net/svn/perlutils/trunk perlutils
# Export without version control metadata
svn export http://svn.dailydata.net/svn/perlutils/trunk perlutils
# Download just this script
svn export http://svn.dailydata.net/svn/perlutils/trunk/configFileEditor
```
## Use Cases
### 1. User-Friendly Configuration Editing
Enable non-technical users to modify complex configuration files without risking syntax errors:
```bash
configFileEditor -c /etc/myapp/config.yaml -e
```
### 2. Configuration Version Management
When software is updated with new configuration options, merge them into existing configs:
```bash
# v2.0 adds new keys to template
configFileEditor -t myapp-v2.datastructure -c myapp-v1.conf.yaml -o myapp-v2.conf.yaml
```
User customizations are preserved while new defaults are added.
### 3. Configuration Format Migration
Convert existing configs between formats:
```bash
# Organization switches from JSON to YAML
for file in configs/*.json; do
configFileEditor -c "$file" -o "${file%.json}.yaml"
done
```
### 4. Configuration Deployment
Generate initial configuration files from templates:
```bash
configFileEditor -t defaults.pl -o /etc/newapp/config.yaml
```
### 5. Configuration Comparison and Auditing
Compare config files across environments to identify discrepancies:
```bash
# Compare production vs staging configurations
configFileEditor -C -c /etc/app/prod.yaml -c /etc/app/staging.yaml
# Audit multiple server configs
configFileEditor -C -c server1.yaml -c server2.yaml -c server3.yaml -o audit-report.txt
```
Identifies missing keys, type mismatches, and value differences for troubleshooting and compliance.
## Technical Details
### Template Files
Template files are Perl data structures (hashrefs) that define the configuration schema:
```perl
{
'setting1' => 'default_value',
'section' => {
'nested_setting' => 123,
'array_setting' => ['item1', 'item2']
}
}
```
Inline comments in templates serve as documentation and are preserved when exported to YAML.
### Merge Behavior
- **Missing keys**: Added from template
- **Existing keys**: Preserved from config (user values not overwritten)
- **Nested hashes**: Recursively merged
- **Arrays**: Extended if template is longer; existing elements preserved
- **Type mismatches**: Config value takes precedence
### Backup Strategy
When saving changes, the original file is renamed with `.bak` extension:
```
config.yaml → config.yaml.bak (old version)
config.yaml → (new version)
```
## Troubleshooting
**No YAML/JSON library error:**
Install at least one YAML and one JSON library using your system package manager or CPAN.
**Template must be a hashref error:**
Ensure template file contains a single hashref structure (starts with `{` and ends with `}`).
**Config must be a hashref error:**
The utility only works with hash-based configurations, not arrays or scalars at the root level.
**Permission denied errors:**
Ensure you have write permissions for the output directory when saving files.
## License
Simplified BSD License (FreeBSD License)
Copyright (c) 2026, Daily Data Inc.
## Author
R. W. Rodolico <rodo@dailydata.net>
## Version
1.2.0 (January 2026)
### Version History
- **1.2.0** (2026-01-18):
- Added config file comparison mode to identify differences across multiple files
- Added ability to rename hash keys in interactive edit mode
- Refactored long functions for improved code readability and maintainability
- **1.1.0** (2026-01-15): Added ability to delete keys from hashes and elements from arrays
- **1.0** (2026-01-13): Initial release with merge, edit, and format conversion capabilities