Subversion Repositories havirt

Rev

Rev 28 | Show entire file | Ignore whitespace | Details | Blame | Last modification | View Log | RSS feed

Rev 28 Rev 47
Line 1... Line 1...
1 
# havirt
 1 
# havirt - High Availability Virtual Machine Cluster Manager
2 
 
 2 
 
3 
### Script to macro manage cluster of nodes (hypervisors) and the domains
 - 
 
4 
(virtuals) on them.
 3 
## Overview
5 
 
 4 
 
6 
Used as an extension to virsh, which it calls quite often, but with a
 5 
**havirt** is a cluster management tool for KVM/libvirt hypervisors with shared storage. It provides safe, automated management of virtual machines across multiple nodes with resource validation, maintenance mode support, and automatic load balancing.
7 
cluster of nodes (hypervisors) with shared block devices (tested with
 6 
 
8 
iSCSI).
 7 
### Key Features
9 
 
 8 
 
10 
Also, includes code to verify there are enough resources on a node
 9 
- **Cluster-wide VM management** - Start, stop, and migrate VMs across nodes
11 
for the requested action, and protection against running the same domain
 10 
- **Resource validation** - Prevents over-allocation of CPU and memory
12 
on multiple nodes simultaneously, which an result in block device
 11 
- **Safety controls** - Prevents running the same VM on multiple nodes (avoids data corruption)
13 
corruption.
 12 
- **Maintenance mode** - Automatically evacuate VMs from nodes requiring maintenance
14 
 
 13 
- **Load balancing** - Automatically distribute VMs across nodes based on resource usage
15 
All nodes must be able to make an ssh connection to all other nodes using
14 
- **Centralized configuration** - Share VM configs across all nodes via NFS
16 
public key (no passwords).
 15 
- **Testing mode** - Safe testing without affecting production systems
17 
 
 16 
 
18 
Very similar to virsh (on purpose); just adds some protection against
 17 
## Requirements
19 
running domains on multiple nodes, allowing monitoring, etc... Samples are:
 18 
 
20 
 
 19 
- **Multiple hypervisors** running KVM/libvirt (tested on Devuan/Debian Linux)
21 
    virsh start domainname # virsh way
 20 
- **Shared block storage** (iSCSI or similar) accessible from all nodes
22 
    havirt domain start domainname nodename
 21 
- **NFS share** mounted at the same path on all nodes (file locking required)
23 
 
 22 
- **SSH public key authentication** between all nodes (passwordless)
24 
These will both start domainname. The virsh command will start it on the
 23 
- **Perl modules**: `Data::Dumper`, `YAML::Tiny`, `FindBin`
25 
node you are currently logged into. The havirt command will first verify
 24 
 
26 
domainname is not running on any node, verify the target node has enough
 25 
## Quick Start
27 
resources, then start it on nodename (or the current node, if nodename not
26 
 
28 
specified) and modify it's state file to reflect the status change.
 27 
### Installation
29 
 
 28 
 
30 
The other difference is that the virsh command uses the configuration
 29 
1. Check out from repository to shared NFS mount:
31 
domainname.xml stored in /etc/virtlib/qemu, while havirt does a 'virsh
 30 
   ```bash
32 
create' using the domainname.xml stored in installdir/conf. This allows
 31 
   svn co http://svn.dailydata.net/svn/havirt/stable /media/shared/havirt
33 
sharing of the same domain configuration on all nodes, at the expense of
 32 
   ```
34 
some functionality.
 33 
 
35 
 
 34 
2. Create symlink on each node:
36 
By default, havirt will simply emit the command it will normally run. If the
 35 
   ```bash
37 
flag --dryrun is defaulted to false (0) in the config file, will actually
 36 
   ln -s /media/shared/havirt/havirt /usr/local/bin/havirt
38 
run the command(s) unless it is turned back on by passing as a cli
 37 
   ```
39 
parameter (see config.sample.yaml).
 38 
 
40 
 
 39 
3. Install dependencies:
41 
havirt creates two subdirectories, installdir/conf/ and installdir/var/ if
 40 
   ```bash
42 
they don't exist.
 41 
   apt install -y libdata-dump-perl libyaml-tiny-perl
43 
 
 42 
   ```
44 
   conf/ stores the xml configuration of all domains
 43 
 
45 
   var/ stores the state file (yaml) and some temporary files.
 44 
4. Generate initial config:
46 
 
 45 
   ```bash
47 
havirt automatically creates a configuration file (config.yaml) in the executables
 46 
   havirt
48 
directory if it doesn't exist, allowing you to modify the default behavior
 47 
   ```
49 
of the system.
 48 
   This creates `config.yaml` with safe defaults (dry-run mode enabled)
50 
 
 49 
 
51 
Can be downloaded via subversion at
 50 
### Initial Setup
52 
    http://svn.dailydata.net/svn/havirt/stable
 51 
 
53 
Highly recommended you do not use trunk, as we modify that on a regular
 52 
See [INSTALL.md](INSTALL.md) for complete installation instructions including:
54 
basis and check in broken code sometimes
 53 
- SSH key setup for passwordless authentication
- 
 
 54 
- NFS configuration requirements
- 
 
 55 
- Node registration
- 
 
 56 
- VM configuration storage
- 
 
 57 
 
- 
 
 58 
## Common Tasks
- 
 
 59 
 
- 
 
 60 
### Daily Operations
- 
 
 61 
 
- 
 
 62 
**Scan cluster for running VMs:**
- 
 
 63 
```bash
- 
 
 64 
havirt node scan --force
- 
 
 65 
```
- 
 
 66 
 
- 
 
 67 
**List all nodes and their resources:**
- 
 
 68 
```bash
- 
 
 69 
havirt node list
- 
 
 70 
```
- 
 
 71 
 
- 
 
 72 
**List all VMs:**
- 
 
 73 
```bash
- 
 
 74 
havirt domain list
- 
 
 75 
```
- 
 
 76 
 
- 
 
 77 
**Start a VM on a specific node:**
- 
 
 78 
```bash
- 
 
 79 
havirt domain start vmname nodename
- 
 
 80 
```
- 
 
 81 
 
- 
 
 82 
**Migrate VM to another node:**
- 
 
 83 
```bash
- 
 
 84 
havirt domain migrate vmname targetnode
- 
 
 85 
```
- 
 
 86 
 
- 
 
 87 
**Shutdown a VM gracefully:**
- 
 
 88 
```bash
- 
 
 89 
havirt domain shutdown vmname
- 
 
 90 
```
- 
 
 91 
 
- 
 
 92 
### Maintenance Operations
- 
 
 93 
 
- 
 
 94 
**Put node in maintenance mode (evacuates all VMs):**
- 
 
 95 
```bash
- 
 
 96 
havirt node maintenance nodename 1
- 
 
 97 
```
- 
 
 98 
 
- 
 
 99 
**Take node out of maintenance:**
- 
 
 100 
```bash
- 
 
 101 
havirt node maintenance nodename 0
- 
 
 102 
```
- 
 
 103 
 
- 
 
 104 
**Balance cluster (redistribute VMs):**
- 
 
 105 
```bash
- 
 
 106 
havirt cluster balance
- 
 
 107 
```
- 
 
 108 
 
- 
 
 109 
**Check cluster statistics:**
- 
 
 110 
```bash
- 
 
 111 
havirt cluster stats
- 
 
 112 
```
- 
 
 113 
 
- 
 
 114 
### Adding New VMs
- 
 
 115 
 
- 
 
 116 
1. Generate a virt-install template:
- 
 
 117 
   ```bash
- 
 
 118 
   havirt domain new myvm
- 
 
 119 
   ```
- 
 
 120 
 
- 
 
 121 
2. Create the VM using the generated command
- 
 
 122 
 
- 
 
 123 
3. Scan to discover the new VM:
- 
 
 124 
   ```bash
- 
 
 125 
   havirt node scan --force
- 
 
 126 
   ```
- 
 
 127 
 
- 
 
 128 
4. Update VM metadata:
- 
 
 129 
   ```bash
- 
 
 130 
   havirt domain update myvm
- 
 
 131 
   ```
- 
 
 132 
 
- 
 
 133 
## Configuration
- 
 
 134 
 
- 
 
 135 
### config.yaml
- 
 
 136 
 
- 
 
 137 
The configuration file is auto-generated at first run. Key settings:
- 
 
 138 
 
- 
 
 139 
```yaml
- 
 
 140 
flags:
- 
 
 141 
  dryrun: 1          # Set to 0 to execute commands (default=1 for safety)
- 
 
 142 
  verbose: 1         # Show detailed output
- 
 
 143 
  debug: 0           # Debug level (0-3)
- 
 
 144 
  testing: 0         # Use test data instead of live systems
- 
 
 145
- 
 
 146 
node reserved memory: 8388608   # Reserve 8GB per node
- 
 
 147 
min scan time: 300              # Minimum seconds between scans
- 
 
 148 
```
- 
 
 149 
 
- 
 
 150 
### Directory Structure
- 
 
 151 
 
- 
 
 152 
```
- 
 
 153 
havirt/
- 
 
 154 
├── havirt              # Main executable
- 
 
 155 
├── *.pm                # Module files (cluster, domain, node)
- 
 
 156 
├── config.yaml         # Configuration (auto-generated)
- 
 
 157 
├── conf/               # VM XML configurations (shared across nodes)
- 
 
 158 
├── var/                # Runtime data
- 
 
 159 
│   ├── status.yaml     # Cluster state database
- 
 
 160 
│   └── lastscan        # Last scan timestamp
- 
 
 161 
└── tests/              # Test suite
- 
 
 162 
```
- 
 
 163 
 
- 
 
 164 
## Safety Features
- 
 
 165 
 
- 
 
 166 
### Dry-Run Mode (Default)
- 
 
 167 
 
- 
 
 168 
By default, havirt operates in dry-run mode and only displays commands without executing them. To execute commands:
- 
 
 169 
 
- 
 
 170 
- Edit `config.yaml` and set `dryrun: 0`, or
- 
 
 171 
- Use `--dryrun 0` flag: `havirt --dryrun 0 domain start vmname node1`
- 
 
 172 
 
- 
 
 173 
### Resource Validation
- 
 
 174 
 
- 
 
 175 
Before starting or migrating VMs, havirt verifies:
- 
 
 176 
- Target node has sufficient free memory
- 
 
 177 
- Target node has sufficient CPU threads
- 
 
 178 
- VM is not already running elsewhere
- 
 
 179 
 
- 
 
 180 
### Maintenance Mode
- 
 
 181 
 
- 
 
 182 
Nodes in maintenance mode:
- 
 
 183 
- Cannot have VMs started on them
- 
 
 184 
- Cannot be selected as migration targets
- 
 
 185 
- Automatically evacuate all VMs during cluster balance
- 
 
 186 
 
- 
 
 187 
## Automation
- 
 
 188 
 
- 
 
 189 
### Cron Job Setup
- 
 
 190 
 
- 
 
 191 
Regular scanning keeps the cluster database current:
- 
 
 192 
 
- 
 
 193 
```bash
- 
 
 194 
# /etc/cron.d/havirt
- 
 
 195 
*/5 * * * * root /usr/local/bin/havirt node scan 2>&1 | logger -t havirt
- 
 
 196 
```
- 
 
 197 
 
- 
 
 198 
See `havirt.sample.cron` for more examples.
- 
 
 199 
 
- 
 
 200 
## Testing
- 
 
 201 
 
- 
 
 202 
A comprehensive test suite is available in the `tests/` directory:
- 
 
 203 
 
- 
 
 204 
```bash
- 
 
 205 
cd tests
- 
 
 206 
./test_havirt_safe.sh     # Safe tests using --testing mode
- 
 
 207 
./test_integration.pl     # Unit tests for modules
- 
 
 208 
```
- 
 
 209 
 
- 
 
 210 
See [tests/TESTING.md](tests/TESTING.md) for details.
- 
 
 211 
 
- 
 
 212 
## Documentation
- 
 
 213 
 
- 
 
 214 
- **[INSTALL.md](INSTALL.md)** - Complete installation and setup guide
- 
 
 215 
- **[USAGE.md](USAGE.md)** - Detailed command reference for all modules
- 
 
 216 
- **[PROGRAMMING.md](PROGRAMMING.md)** - Development documentation
- 
 
 217 
- **[CHANGES.md](CHANGES.md)** - Version history and changelog
- 
 
 218 
- **[tests/TESTING.md](tests/TESTING.md)** - Testing documentation
- 
 
 219 
 
- 
 
 220 
## Architecture
- 
 
 221 
 
- 
 
 222 
havirt extends virsh with cluster awareness:
- 
 
 223 
 
- 
 
 224 
| Task | virsh | havirt |
- 
 
 225 
|------|-------|--------|
- 
 
 226 
| Start VM | Starts on current node | Validates resources, checks cluster-wide, starts safely |
- 
 
 227 
| Config location | `/etc/libvirt/qemu/` | Shared `conf/` directory |
- 
 
 228 
| Cluster awareness | None | Full cluster state tracking |
- 
 
 229 
| Safety checks | None | Prevents duplicate VMs, validates resources |
- 
 
 230 
 
- 
 
 231 
## Troubleshooting
- 
 
 232 
 
- 
 
 233 
**Enable verbose output:**
- 
 
 234 
```bash
- 
 
 235 
havirt --verbose 1 [command]
- 
 
 236 
```
- 
 
 237 
 
- 
 
 238 
**Enable debug output:**
- 
 
 239 
```bash
- 
 
 240 
havirt --debug 3 [command]
- 
 
 241 
```
- 
 
 242 
 
- 
 
 243 
**Force rescan if status seems incorrect:**
- 
 
 244 
```bash
- 
 
 245 
havirt node scan --force
- 
 
 246 
```
- 
 
 247 
 
- 
 
 248 
**Check cluster state:**
- 
 
 249 
```bash
- 
 
 250 
cat var/status.yaml
- 
 
 251 
```
- 
 
 252 
 
- 
 
 253 
## Support and Development
- 
 
 254 
 
- 
 
 255 
- **Repository:** http://svn.dailydata.net/svn/havirt/
- 
 
 256 
- **Stable branch:** Use `stable` for production
- 
 
 257 
- **Development:** `trunk` contains latest development (may be unstable)
- 
 
 258 
 
- 
 
 259 
## License
- 
 
 260 
 
- 
 
 261 
Copyright 2024-2026 Daily Data, Inc.
- 
 
 262 
 
- 
 
 263 
Redistribution and use in source and binary forms, with or without modification, are permitted provided that redistributions retain the copyright notice and disclaimer. See source files for complete license text (BSD-style license).