Subversion Repositories havirt

# havirt - High Availability Virtual Machine Cluster Manager

## Overview

**havirt** is a cluster management tool for KVM/libvirt hypervisors with shared storage. It provides safe, automated management of virtual machines across multiple nodes with resource validation, maintenance mode support, and automatic load balancing.

### Key Features

- **Cluster-wide VM management** - Start, stop, and migrate VMs across nodes

- **Resource validation** - Prevents over-allocation of CPU and memory

- **Safety controls** - Prevents running the same VM on multiple nodes (avoids data corruption)

- **Maintenance mode** - Automatically evacuate VMs from nodes requiring maintenance

- **Load balancing** - Automatically distribute VMs across nodes based on resource usage

- **Centralized configuration** - Share VM configs across all nodes via NFS

- **Testing mode** - Safe testing without affecting production systems

## Requirements

- **Multiple hypervisors** running KVM/libvirt (tested on Devuan/Debian Linux)

- **Shared block storage** (iSCSI or similar) accessible from all nodes

- **NFS share** mounted at the same path on all nodes (file locking required)

- **SSH public key authentication** between all nodes (passwordless)

- **Perl modules**: `Data::Dumper`, `YAML::Tiny`, `FindBin`

## Quick Start

### Installation

1. Check out from repository to shared NFS mount:

   ```bash

   svn co http://svn.dailydata.net/svn/havirt/stable /media/shared/havirt

   ```

2. Create symlink on each node:

   ```bash

   ln -s /media/shared/havirt/havirt /usr/local/bin/havirt

   ```

3. Install dependencies:

   ```bash

   apt install -y libdata-dump-perl libyaml-tiny-perl

   ```

4. Generate initial config:

   ```bash

   havirt

   ```

   This creates `config.yaml` with safe defaults (dry-run mode enabled)

### Initial Setup

See [INSTALL.md](INSTALL.md) for complete installation instructions including:

- SSH key setup for passwordless authentication

- NFS configuration requirements

- Node registration

- VM configuration storage

## Common Tasks

### Daily Operations

**Scan cluster for running VMs:**

```bash

havirt node scan --force

```

**List all nodes and their resources:**

```bash

havirt node list

```

**List all VMs:**

```bash

havirt domain list

```

**Start a VM on a specific node:**

```bash

havirt domain start vmname nodename

```

**Migrate VM to another node:**

```bash

havirt domain migrate vmname targetnode

```

**Shutdown a VM gracefully:**

```bash

havirt domain shutdown vmname

```

### Maintenance Operations

**Put node in maintenance mode (evacuates all VMs):**

```bash

havirt node maintenance nodename 1

```

**Take node out of maintenance:**

```bash

havirt node maintenance nodename 0

```

**Balance cluster (redistribute VMs):**

```bash

havirt cluster balance

```

**Check cluster statistics:**

```bash

havirt cluster stats

```

### Adding New VMs

1. Generate a virt-install template:

   ```bash

   havirt domain new myvm

   ```

2. Create the VM using the generated command

3. Scan to discover the new VM:

   ```bash

   havirt node scan --force

   ```

4. Update VM metadata:

   ```bash

   havirt domain update myvm

   ```

## Configuration

### config.yaml

The configuration file is auto-generated at first run. Key settings:

```yaml

flags:

  dryrun: 1          # Set to 0 to execute commands (default=1 for safety)

  verbose: 1         # Show detailed output

  debug: 0           # Debug level (0-3)

  testing: 0         # Use test data instead of live systems

node reserved memory: 8388608   # Reserve 8GB per node

min scan time: 300              # Minimum seconds between scans

```

### Directory Structure

```

havirt/

├── havirt              # Main executable

├── *.pm                # Module files (cluster, domain, node)

├── config.yaml         # Configuration (auto-generated)

├── conf/               # VM XML configurations (shared across nodes)

├── var/                # Runtime data

│   ├── status.yaml     # Cluster state database

│   └── lastscan        # Last scan timestamp

└── tests/              # Test suite

```

## Safety Features

### Dry-Run Mode (Default)

By default, havirt operates in dry-run mode and only displays commands without executing them. To execute commands:

- Edit `config.yaml` and set `dryrun: 0`, or

- Use `--dryrun 0` flag: `havirt --dryrun 0 domain start vmname node1`

### Resource Validation

Before starting or migrating VMs, havirt verifies:

- Target node has sufficient free memory

- Target node has sufficient CPU threads

- VM is not already running elsewhere

### Maintenance Mode

Nodes in maintenance mode:

- Cannot have VMs started on them

- Cannot be selected as migration targets

- Automatically evacuate all VMs during cluster balance

## Automation

### Cron Job Setup

Regular scanning keeps the cluster database current:

```bash

# /etc/cron.d/havirt

*/5 * * * * root /usr/local/bin/havirt node scan 2>&1 | logger -t havirt

```

See `havirt.sample.cron` for more examples.

## Testing

A comprehensive test suite is available in the `tests/` directory:

```bash

cd tests

./test_havirt_safe.sh     # Safe tests using --testing mode

./test_integration.pl     # Unit tests for modules

```

See [tests/TESTING.md](tests/TESTING.md) for details.

## Documentation

- **[INSTALL.md](INSTALL.md)** - Complete installation and setup guide

- **[USAGE.md](USAGE.md)** - Detailed command reference for all modules

- **[PROGRAMMING.md](PROGRAMMING.md)** - Development documentation

- **[CHANGES.md](CHANGES.md)** - Version history and changelog

- **[tests/TESTING.md](tests/TESTING.md)** - Testing documentation

## Architecture

havirt extends virsh with cluster awareness:

| Task | virsh | havirt |

|------|-------|--------|

| Start VM | Starts on current node | Validates resources, checks cluster-wide, starts safely |

| Config location | `/etc/libvirt/qemu/` | Shared `conf/` directory |

| Cluster awareness | None | Full cluster state tracking |

| Safety checks | None | Prevents duplicate VMs, validates resources |

## Troubleshooting

**Enable verbose output:**

```bash

havirt --verbose 1 [command]

```

**Enable debug output:**

```bash

havirt --debug 3 [command]

```

**Force rescan if status seems incorrect:**

```bash

havirt node scan --force

```

**Check cluster state:**

```bash

cat var/status.yaml

```

## Support and Development

- **Repository:** http://svn.dailydata.net/svn/havirt/

- **Stable branch:** Use `stable` for production

- **Development:** `trunk` contains latest development (may be unstable)

## License

Copyright 2024-2026 Daily Data, Inc.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that redistributions retain the copyright notice and disclaimer. See source files for complete license text (BSD-style license).