Subversion Repositories havirt

# havirt - High Availability Virtual Machine Cluster Manager

## Overview

**havirt** is a cluster management tool for KVM/libvirt hypervisors with shared storage. It provides safe, automated management of virtual machines across multiple nodes with resource validation, maintenance mode support, and automatic load balancing.

### Key Features

- **Cluster-wide VM management** - Start, stop, and migrate VMs across nodes
- **Resource validation** - Prevents over-allocation of CPU and memory
- **Safety controls** - Prevents running the same VM on multiple nodes (avoids data corruption)
- **Maintenance mode** - Automatically evacuate VMs from nodes requiring maintenance
- **Load balancing** - Automatically distribute VMs across nodes based on resource usage
- **Centralized configuration** - Share VM configs across all nodes via NFS
- **Testing mode** - Safe testing without affecting production systems

## Requirements

- **Multiple hypervisors** running KVM/libvirt (tested on Devuan/Debian Linux)
- **Shared block storage** (iSCSI or similar) accessible from all nodes
- **NFS share** mounted at the same path on all nodes (file locking required)
- **SSH public key authentication** between all nodes (passwordless)
- **Perl modules**: `Data::Dumper`, `YAML::Tiny`, `FindBin`

## Quick Start

### Installation

1. Check out from repository to shared NFS mount:
   ```bash
   svn co http://svn.dailydata.net/svn/havirt/stable /media/shared/havirt
   ```

2. Create symlink on each node:
   ```bash
   ln -s /media/shared/havirt/havirt /usr/local/bin/havirt
   ```

3. Install dependencies:
   ```bash
   apt install -y libdata-dump-perl libyaml-tiny-perl
   ```

4. Generate initial config:
   ```bash
   havirt
   ```
   This creates `config.yaml` with safe defaults (dry-run mode enabled)

### Initial Setup

See [INSTALL.md](INSTALL.md) for complete installation instructions including:
- SSH key setup for passwordless authentication
- NFS configuration requirements
- Node registration
- VM configuration storage

## Common Tasks

### Daily Operations

**Scan cluster for running VMs:**
```bash
havirt node scan --force
```

**List all nodes and their resources:**
```bash
havirt node list
```

**List all VMs:**
```bash
havirt domain list
```

**Start a VM on a specific node:**
```bash
havirt domain start vmname nodename
```

**Migrate VM to another node:**
```bash
havirt domain migrate vmname targetnode
```

**Shutdown a VM gracefully:**
```bash
havirt domain shutdown vmname
```

### Maintenance Operations

**Put node in maintenance mode (evacuates all VMs):**
```bash
havirt node maintenance nodename 1
```

**Take node out of maintenance:**
```bash
havirt node maintenance nodename 0
```

**Balance cluster (redistribute VMs):**
```bash
havirt cluster balance
```

**Check cluster statistics:**
```bash
havirt cluster stats
```

### Adding New VMs

1. Generate a virt-install template:
   ```bash
   havirt domain new myvm
   ```

2. Create the VM using the generated command

3. Scan to discover the new VM:
   ```bash
   havirt node scan --force
   ```

4. Update VM metadata:
   ```bash
   havirt domain update myvm
   ```

## Configuration

### config.yaml

The configuration file is auto-generated at first run. Key settings:

```yaml
flags:
  dryrun: 1          # Set to 0 to execute commands (default=1 for safety)
  verbose: 1         # Show detailed output
  debug: 0           # Debug level (0-3)
  testing: 0         # Use test data instead of live systems
  
node reserved memory: 8388608   # Reserve 8GB per node
min scan time: 300              # Minimum seconds between scans
```

### Directory Structure

```
havirt/
├── havirt              # Main executable
├── *.pm                # Module files (cluster, domain, node)
├── config.yaml         # Configuration (auto-generated)
├── conf/               # VM XML configurations (shared across nodes)
├── var/                # Runtime data
│   ├── status.yaml     # Cluster state database
│   └── lastscan        # Last scan timestamp
└── tests/              # Test suite
```

## Safety Features

### Dry-Run Mode (Default)

By default, havirt operates in dry-run mode and only displays commands without executing them. To execute commands:

- Edit `config.yaml` and set `dryrun: 0`, or
- Use `--dryrun 0` flag: `havirt --dryrun 0 domain start vmname node1`

### Resource Validation

Before starting or migrating VMs, havirt verifies:
- Target node has sufficient free memory
- Target node has sufficient CPU threads
- VM is not already running elsewhere

### Maintenance Mode

Nodes in maintenance mode:
- Cannot have VMs started on them
- Cannot be selected as migration targets
- Automatically evacuate all VMs during cluster balance

## Automation

### Cron Job Setup

Regular scanning keeps the cluster database current:

```bash
# /etc/cron.d/havirt
*/5 * * * * root /usr/local/bin/havirt node scan 2>&1 | logger -t havirt
```

See `havirt.sample.cron` for more examples.

## Testing

A comprehensive test suite is available in the `tests/` directory:

```bash
cd tests
./test_havirt_safe.sh     # Safe tests using --testing mode
./test_integration.pl     # Unit tests for modules
```

See [tests/TESTING.md](tests/TESTING.md) for details.

## Documentation

- **[INSTALL.md](INSTALL.md)** - Complete installation and setup guide
- **[USAGE.md](USAGE.md)** - Detailed command reference for all modules
- **[PROGRAMMING.md](PROGRAMMING.md)** - Development documentation
- **[CHANGES.md](CHANGES.md)** - Version history and changelog
- **[tests/TESTING.md](tests/TESTING.md)** - Testing documentation

## Architecture

havirt extends virsh with cluster awareness:

| Task | virsh | havirt |
|------|-------|--------|
| Start VM | Starts on current node | Validates resources, checks cluster-wide, starts safely |
| Config location | `/etc/libvirt/qemu/` | Shared `conf/` directory |
| Cluster awareness | None | Full cluster state tracking |
| Safety checks | None | Prevents duplicate VMs, validates resources |

## Troubleshooting

**Enable verbose output:**
```bash
havirt --verbose 1 [command]
```

**Enable debug output:**
```bash
havirt --debug 3 [command]
```

**Force rescan if status seems incorrect:**
```bash
havirt node scan --force
```

**Check cluster state:**
```bash
cat var/status.yaml
```

## Support and Development

- **Repository:** http://svn.dailydata.net/svn/havirt/
- **Stable branch:** Use `stable` for production
- **Development:** `trunk` contains latest development (may be unstable)

## License

Copyright 2024-2026 Daily Data, Inc.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that redistributions retain the copyright notice and disclaimer. See source files for complete license text (BSD-style license).