Last modification

• Rev 47 2026-01-03 01:06:37
• Author: rodolico

• Log message:

  Made major cleanups and changes (see CHANGES.md) and updated documentation to decent markdown format. Had CoPilot create a bunch of tests in ./tests folder and threw in some real world data. Updates have not been tested on Real World machines yet

havirt - High Availability Virtual Machine Cluster Manager

Overview

havirt is a cluster management tool for KVM/libvirt hypervisors with shared storage. It provides safe, automated management of virtual machines across multiple nodes with resource validation, maintenance mode support, and automatic load balancing.

Key Features

• Cluster-wide VM management - Start, stop, and migrate VMs across nodes

• Resource validation - Prevents over-allocation of CPU and memory

• Safety controls - Prevents running the same VM on multiple nodes (avoids data corruption)

• Maintenance mode - Automatically evacuate VMs from nodes requiring maintenance

• Load balancing - Automatically distribute VMs across nodes based on resource usage

• Centralized configuration - Share VM configs across all nodes via NFS

• Testing mode - Safe testing without affecting production systems

Requirements

• Multiple hypervisors running KVM/libvirt (tested on Devuan/Debian Linux)

• Shared block storage (iSCSI or similar) accessible from all nodes

• NFS share mounted at the same path on all nodes (file locking required)

• SSH public key authentication between all nodes (passwordless)

• Perl modules: Data::Dumper, YAML::Tiny, FindBin

Quick Start

Installation

1. Check out from repository to shared NFS mount:

svn co http://svn.dailydata.net/svn/havirt/stable /media/shared/havirt

2. Create symlink on each node:

ln -s /media/shared/havirt/havirt /usr/local/bin/havirt

3. Install dependencies:

apt install -y libdata-dump-perl libyaml-tiny-perl

4. Generate initial config:

havirt

This creates config.yaml with safe defaults (dry-run mode enabled)

Initial Setup

See INSTALL.md for complete installation instructions including:

• SSH key setup for passwordless authentication

• NFS configuration requirements

• Node registration

• VM configuration storage

Common Tasks

Daily Operations

Scan cluster for running VMs:

havirt node scan --force

List all nodes and their resources:

havirt node list

List all VMs:

havirt domain list

Start a VM on a specific node:

havirt domain start vmname nodename

Migrate VM to another node:

havirt domain migrate vmname targetnode

Shutdown a VM gracefully:

havirt domain shutdown vmname

Maintenance Operations

Put node in maintenance mode (evacuates all VMs):

havirt node maintenance nodename 1

Take node out of maintenance:

havirt node maintenance nodename 0

Balance cluster (redistribute VMs):

havirt cluster balance

Check cluster statistics:

havirt cluster stats

Adding New VMs

1. Generate a virt-install template:

havirt domain new myvm

2. Create the VM using the generated command

3. Scan to discover the new VM:

havirt node scan --force

4. Update VM metadata:

havirt domain update myvm

Configuration

config.yaml

The configuration file is auto-generated at first run. Key settings:

flags:

dryrun: 1 # Set to 0 to execute commands (default=1 for safety)

verbose: 1 # Show detailed output

debug: 0 # Debug level (0-3)

testing: 0 # Use test data instead of live systems

node reserved memory: 8388608 # Reserve 8GB per node

min scan time: 300 # Minimum seconds between scans

Directory Structure

havirt/

├── havirt # Main executable

├── *.pm # Module files (cluster, domain, node)

├── config.yaml # Configuration (auto-generated)

├── conf/ # VM XML configurations (shared across nodes)

├── var/ # Runtime data

│ ├── status.yaml # Cluster state database

│ └── lastscan # Last scan timestamp

└── tests/ # Test suite

Safety Features

Dry-Run Mode (Default)

By default, havirt operates in dry-run mode and only displays commands without executing them. To execute commands:

• Edit config.yaml and set dryrun: 0, or

• Use --dryrun 0 flag: havirt --dryrun 0 domain start vmname node1

Resource Validation

Before starting or migrating VMs, havirt verifies:

• Target node has sufficient free memory

• Target node has sufficient CPU threads

• VM is not already running elsewhere

Maintenance Mode

Nodes in maintenance mode:

• Cannot have VMs started on them

• Cannot be selected as migration targets

• Automatically evacuate all VMs during cluster balance

Automation

Cron Job Setup

Regular scanning keeps the cluster database current:

/etc/cron.d/havirt

/5 * root /usr/local/bin/havirt node scan 2>&1 | logger -t havirt

See havirt.sample.cron for more examples.

Testing

A comprehensive test suite is available in the tests/ directory:

cd tests

./test_havirt_safe.sh # Safe tests using --testing mode

./test_integration.pl # Unit tests for modules

See tests/TESTING.md for details.

Documentation

• INSTALL.md - Complete installation and setup guide

• USAGE.md - Detailed command reference for all modules

• PROGRAMMING.md - Development documentation

• CHANGES.md - Version history and changelog

• tests/TESTING.md - Testing documentation

Architecture

havirt extends virsh with cluster awareness:

| Task | virsh | havirt |

|------|-------|--------|

| Start VM | Starts on current node | Validates resources, checks cluster-wide, starts safely |

| Config location | /etc/libvirt/qemu/ | Shared conf/ directory |

| Cluster awareness | None | Full cluster state tracking |

| Safety checks | None | Prevents duplicate VMs, validates resources |

Troubleshooting

Enable verbose output:

havirt --verbose 1 [command]

Enable debug output:

havirt --debug 3 [command]

Force rescan if status seems incorrect:

havirt node scan --force

Check cluster state:

cat var/status.yaml

Support and Development

• Repository: http://svn.dailydata.net/svn/havirt/

• Stable branch: Use stable for production

• Development: trunk contains latest development (may be unstable)

License

Copyright 2024-2026 Daily Data, Inc.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that redistributions retain the copyright notice and disclaimer. See source files for complete license text (BSD-style license).