Rev 41 | Blame | Compare with Previous | Last modification | View Log | Download | RSS feed
# havirt Usage Guide
## Command Syntax
```bash
havirt [flags] [module] [action] [arguments]
```
- **flags**: Optional parameters (can be placed anywhere in the command)
- **module**: One of `domain`, `node`, or `cluster`
- **action**: The operation to perform
- **arguments**: Action-specific parameters
### Global Flags
| Flag | Short | Description | Default |
| ------------ | ----- | ------------------------------------ | ------------ |
| `--dryrun` | `-n` | Show commands without executing | 1 (enabled) |
| `--nodryrun` | | Alias for -n 0 | 0 (disabled) |
| `--verbose` | `-v` | Show detailed output | 1 (enabled) |
| `--quiet` | `-q` | Suppress output | 0 (disabled) |
| `--debug` | `-d` | Debug level (0-3) | 0 |
| `--force` | `-y` | Force operation (bypass time checks) | 0 |
| `--format` | `-f` | Output format: `screen` or `tsv` | screen |
| `--target` | `-t` | Target specific node or domain | none |
| `--testing` | | Use test data (for development) | 0 |
| `--help` | `-h` | Show help | |
| `--version` | | Show version information | |
### Flag Placement
Flags can be placed anywhere in the command:
```bash
# All equivalent
havirt --format tsv domain list
havirt domain --format tsv list
havirt domain list --format tsv
```
### Getting Help
Help is available at multiple levels:
```bash
havirt --help # General help
havirt domain # Domain module help
havirt domain help # Domain module help
havirt node # Node module help
havirt cluster # Cluster module help
```
## Module Overview
Havirt is organized into three main modules:
### domain - Virtual Machine Management
Manage individual VMs (domains). Operations include starting, stopping, migrating, and listing VMs.
```bash
havirt domain [action] [arguments]
```
### node - Hypervisor Management
Manage hypervisor nodes. Operations include adding nodes, scanning for VMs, and viewing node resources.
```bash
havirt node [action] [arguments]
```
### cluster - Cluster Operations
Cluster-wide operations including statistics, load balancing, and iSCSI management.
```bash
havirt cluster [action] [arguments]
```
---
# Domain Module
Manage individual virtual machines across the cluster.
## domain list
Display all known VMs with their status and resource allocation.
**Syntax:**
```bash
havirt domain list [--format screen|tsv]
```
**Examples:**
```bash
# Display VMs in table format
havirt domain list
# Output as tab-separated values
havirt domain list --format tsv
# Redirect to file for processing
havirt domain list -f tsv > vms.txt
```
**Output:**
```
name memory vcpu node maintenance
webvm1 4194304 4 node1 0
db01 8388608 8 node2 0
testvm 2097152 2 node3 1
```
**Fields:**
- **name**: VM name
- **memory**: Allocated memory in KB
- **vcpu**: Number of virtual CPUs
- **node**: Currently running on this node (empty if stopped)
- **maintenance**: 1 = in maintenance mode, 0 = normal
## domain start
Start a VM on a specified node (or current node if not specified).
**Syntax:**
```bash
havirt domain start <domainname> [nodename]
```
**Safety Checks:**
- Verifies VM not running elsewhere in cluster
- Validates VM configuration exists in `conf/`
- Checks target node has sufficient resources
- Rescans all nodes before starting
**Examples:**
```bash
# Start VM on specific node
havirt domain start webvm1 node2
# Start VM on current node
havirt domain start webvm1
# Dry-run to see what would happen
havirt --dryrun 1 domain start webvm1 node2
```
**Common Errors:**
```
# VM already running
Error: webvm1 is already running on node1
# Configuration not found
Error: Configuration file conf/webvm1.xml not found
# Insufficient resources
Error: node2 does not have sufficient memory
```
## domain shutdown
Gracefully shutdown a running VM.
**Syntax:**
```bash
havirt domain shutdown <domainname>
```
**Behavior:**
- Sends ACPI shutdown signal to VM
- Waits for VM to shut down gracefully
**Examples:**
```bash
# Shutdown VM
havirt domain shutdown webvm1
# View shutdown command without executing
havirt --dryrun 1 domain shutdown webvm1
```
# domain migrate
Live migrate a VM from its current node to another node.
**Syntax:**
```bash
havirt domain migrate <domainname> [target-node]
```
**Behavior:**
- Performs live migration (VM stays running)
- If no target specified, selects least-loaded node automatically
- Updates cluster database after migration
- Validates target has sufficient resources
- **Note:** libvirt can fail on domain migration. If this occurs, you will either need to manually migrate or shut down, then start the domain on the target node.
**Examples:**
```bash
# Migrate to specific node
havirt domain migrate webvm1 node3
# Migrate to automatically selected node
havirt domain migrate webvm1
# Check what would happen
havirt --dryrun 1 domain migrate webvm1 node3
# Failure reported due to timeout, but domain actually migrated
havirt domain migrate webmv1 node2
# some error message here
havirt node scan --force
havirt domain list | grep webvm1
# in most cases, if timeout set too low, the final command will show
# the domain actually moved
```
**Migration Time:**
Typically 10-60 seconds depending on:
- VM memory size
- Network speed
- VM workload (memory changes during migration)
**Safety:**
- Source and target must both access shared storage
- Validates resources before starting migration
- Will not migrate to nodes in maintenance mode
- Will not migrate VMs in maintenance mode
## domain update
Update VM metadata by reading XML configuration files.
**Syntax:**
```bash
havirt domain update [domainname]
havirt domain update -t domainname # Target specific domain
```
**Behavior:**
- Reads XML files from `conf/` directory
- Extracts memory, vcpu, and other settings
- Updates `var/status.yaml` database
- Without argument: updates ALL domains
**Examples:**
```bash
# Update all domains
havirt domain update
# Update specific domain
havirt domain update webvm1
havirt -t webvm1 domain update
```
**When to Use:**
- After adding new VMs to the cluster
- After modifying VM XML configuration
- After restoring from backup
- If database seems out of sync
## domain new
Generate a `virt-install` command template for creating a new VM.
**Syntax:**
```bash
havirt domain new [domainname]
```
**Generated Information:**
- Unused VNC port
- Unique UUID
- Random MAC address (prefix: 00:16:3e - XEN OUI)
- Template from `virt-install.template`
**Examples:**
```bash
# Generate template
havirt domain new mynewvm
# Copy output and customize
havirt domain new mynewvm > create-vm.sh
vi create-vm.sh
bash create-vm.sh
```
**Warning**: MAC addresses are randomly generated and not guaranteed unique. Check for conflicts in large clusters.
## domain maintenance
Set or clear maintenance flag for a VM.
**Syntax:**
```bash
havirt domain maintenance <domainname> <on|off>
```
**Examples:**
```bash
# Enable maintenance mode
havirt domain maintenance webvm1 on
# Disable maintenance mode
havirt domain maintenance webvm1 off
# Check current status
havirt domain list | grep webvm1
```
**Effect:**
- Prevents automatic migration during cluster balancing
- Prevents automatic restart by keepalive scripts
- VM can still be manually started/stopped/migrated
---
# Node Module
Manage hypervisor nodes in the cluster.
## node list
Display all nodes with their resources and status.
**Syntax:**
```bash
havirt node list [--format screen|tsv]
```
**Examples:**
```bash
# Display nodes
havirt node list
# TSV format for scripts
havirt -f tsv node list
```
**Output:**
```
name memory cpu_count maintenance
node1 67108864 16 0
node2 67108864 16 0
node3 33554432 8 1
```
**Fields:**
- **name**: Node hostname
- **memory**: Total memory in KB
- **cpu_count**: Number of CPU threads
- **maintenance**: 1 = in maintenance, 0 = active
## node scan
Scan nodes to discover running VMs.
**Syntax:**
```bash
havirt node scan [nodename]
havirt node scan [-t nodename] [--force]
```
**Behavior:**
- Executes `virsh list` on each node
- Updates cluster database with running VMs
- By default: scans all nodes
- Respects 5-minute minimum between scans (unless `--force`)
**Examples:**
```bash
# Scan all nodes (respects time limit)
havirt node scan
# Force immediate scan
havirt node scan --force
# Scan specific node
havirt node scan node1
havirt -t node1 node scan
```
**Cron Job:**
```bash
# Add to /etc/cron.d/havirt
*/5 * * * * root /usr/local/bin/havirt node scan --quiet 2>&1 | logger -t havirt
```
**Time Limit:**
By default, scans are limited to once per 5 minutes (configurable in `config.yaml` as `min scan time`). Use `--force` to override.
## node add / node update
Add a new node or update existing node information.
**Syntax:**
```bash
havirt node add <nodename>
havirt node update <nodename>
havirt node update # Update all nodes
```
**Behavior:**
- Connects via SSH to node
- Executes `virsh nodeinfo`
- Extracts CPU and memory information
- Adds to or updates cluster database
**Examples:**
```bash
# Add new node
havirt node add node4
# Update specific node
havirt node update node1
# Update all nodes
havirt node update
```
**Requirements:**
- SSH connectivity must be configured
- libvirt must be running on target node
- Node must be accessible by hostname/alias
## node maintenance
Place node in or remove from maintenance mode.
**Syntax:**
```bash
havirt node maintenance <nodename> <on|off>
```
**Examples:**
```bash
# Enable maintenance mode
havirt node maintenance node2 on
# Run cluster balance to evacuate VMs
havirt cluster balance
# Disable maintenance mode when done
havirt node maintenance node2 off
```
**Effect:**
- Prevents new VMs from starting on node
- Prevents node being selected as migration target
- During cluster balance, all VMs evacuated from node
- Existing VMs continue running until migrated
**Typical Workflow:**
```bash
# 1. Enable maintenance
havirt node maintenance node2 on
# 2. Evacuate VMs
havirt cluster balance
# 3. Verify node empty
havirt domain list | grep node2
# 4. Perform maintenance...
# 5. Disable maintenance
havirt node maintenance node2 off
```
---
# Cluster Module
Cluster-wide operations and statistics.
## cluster stats
Display resource usage statistics for the entire cluster.
**Syntax:**
```bash
havirt cluster stats [--format screen|tsv]
```
**Examples:**
```bash
# Display cluster statistics
havirt cluster stats
# TSV format
havirt -f tsv cluster stats
```
**Output:**
```
node memory cpu_count memory_used memory_free cpu_used cpu_free
node1 67108864 16 25165824 41943040 8 8
node2 67108864 16 33554432 33554432 12 4
node3 67108864 16 8388608 58720256 4 12
```
**Fields:**
- **memory**: Total memory (KB)
- **memory_used**: Allocated to running VMs
- **memory_free**: Available for new VMs
- **cpu_count**: Total CPU threads
- **cpu_used**: Virtual CPUs allocated
- **cpu_free**: Available vCPU slots
## cluster balance
Automatically redistribute VMs across nodes for optimal resource utilization.
**Syntax:**
```bash
havirt cluster balance [--dryrun 0|1]
```
**Behavior:**
1. Evacuates all VMs from nodes in maintenance mode
2. Calculates memory imbalance across active nodes
3. Selects VMs to migrate from overloaded nodes
4. Migrates VMs to underutilized nodes
5. Repeats until cluster is balanced
**Examples:**
```bash
# See what would be balanced (safe)
havirt --dryrun 1 cluster balance
# Actually perform balancing
havirt --dryrun 0 cluster balance
# With verbose output
havirt --verbose 1 --dryrun 0 cluster balance
```
**When to Use:**
- After placing node in maintenance mode
- After adding new nodes to cluster
- When cluster becomes unbalanced
- As part of regular maintenance
**Safety:**
- Respects node maintenance flags
- Respects VM maintenance flags
- Validates resources before each migration
- Will not overload target nodes
## cluster iscsi
Manage iSCSI targets across the cluster.
**Syntax:**
```bash
havirt cluster iscsi # List targets
havirt cluster iscsi add <target-ip> # Add target
havirt cluster iscsi update [node] # Update sessions
```
**Examples:**
```bash
# Add iSCSI target
havirt cluster iscsi add 192.168.1.10
# List configured targets
havirt cluster iscsi
# Update one node
havirt cluster iscsi update node1
# Update all nodes
havirt cluster iscsi update
```
**Behavior:**
- Stores target in cluster configuration
- Executes `iscsiadm` commands on nodes
- Discovers and logs in to targets
- Updates persistent iSCSI configuration
---
# Common Workflows
## Adding a New VM
```bash
# 1. Generate virt-install command
havirt domain new mynewvm > create-vm.sh
# 2. Customize and run
vi create-vm.sh
bash create-vm.sh
# 3. Discover new VM
havirt node scan --force
# 4. Import configuration
havirt domain update mynewvm
# 5. Verify
havirt domain list | grep mynewvm
```
## Performing Node Maintenance
```bash
# 1. Enable maintenance mode
havirt node maintenance node2 on
# 2. Evacuate VMs
havirt --dryrun 0 cluster balance
# 3. Verify evacuation
havirt domain list | grep node2
# (should show no VMs)
# 4. Perform maintenance work...
# shutdown, update, reboot, etc.
# 5. Bring node back online
havirt node maintenance node2 off
# 6. Re-balance cluster
havirt --dryrun 0 cluster balance
```
## Migrating a VM
```bash
# 1. Check current location
havirt domain list | grep webvm1
# 2. Verify target has resources
havirt cluster stats
# 3. Migrate
havirt --dryrun 0 domain migrate webvm1 node3
# 4. Verify migration
havirt node scan --force
havirt domain list | grep webvm1
```
## Emergency VM Shutdown
```bash
# 1. Graceful shutdown
havirt --dryrun 0 domain shutdown vmname
# If that doesn't work after 60 seconds:
# 2. Find node VM is on
havirt domain list | grep vmname
# 3. Force destroy directly
ssh node1 "virsh destroy vmname"
# 4. Rescan cluster
havirt node scan --force
```
## Recovering from Split-Brain
If a VM appears running on multiple nodes:
```bash
# 1. Identify the problem
havirt --debug 2 domain list
# 2. Determine correct node (check VM console/logs)
# 3. Destroy on incorrect node(s)
ssh wrong-node "virsh destroy vmname"
# 4. Rescan
havirt node scan --force
# 5. Verify
havirt domain list | grep vmname
```
---
# Output Formats
## Screen Format (Default)
Fixed-width columns suitable for terminal viewing:
```bash
havirt domain list
```
```
name memory vcpu node maintenance
webvm1 4194304 4 node1 0
```
## TSV Format
Tab-separated values for scripting:
```bash
havirt --format tsv domain list
```
```
name memory vcpu node maintenance
webvm1 4194304 4 node1 0
```
**Use Cases:**
```bash
# Import into spreadsheet
havirt -f tsv domain list > vms.tsv
# Process with awk
havirt -f tsv domain list | awk '$4 == "node1" {print $1}'
# Count VMs per node
havirt -f tsv domain list | tail -n +2 | cut -f4 | sort | uniq -c
```
---
# Dry-Run Mode
By default, havirt operates in dry-run mode for safety.
## Checking Mode
```bash
# Check current setting
grep dryrun /media/shared/havirt/config.yaml
```
## Temporary Override
```bash
# Force dry-run (safe)
havirt --dryrun 1 domain start vm1 node1
# Force execution (one command only)
havirt --dryrun 0 domain start vm1 node1
```
## Permanent Change
```bash
# Edit config
vi /media/shared/havirt/config.yaml
# Change:
flags:
dryrun: 0 # Now executes commands by default
```
---
# Debugging
## Debug Levels
```bash
# Level 1: Basic operations
havirt --debug 1 node scan
# Level 2: Detailed information
havirt --debug 2 domain migrate vm1 node2
# Level 3: Full command trace
havirt --debug 3 cluster balance
```
## Verbose Output
```bash
# Enable verbose mode
havirt --verbose 1 node scan
# Disable verbose mode
havirt --verbose 0 node scan
```
## Checking Database
```bash
# View entire status database
cat /media/shared/havirt/var/status.yaml
# Check specific VM
grep -A 5 "vmname" /media/shared/havirt/var/status.yaml
# Check node population
grep -A 10 "nodePopulation" /media/shared/havirt/var/status.yaml
```
---
# Best Practices
## Regular Operations
1. **Run scans regularly**: Set up cron job for `node scan`
2. **Use dry-run first**: Always test with `--dryrun 1` before executing
3. **Monitor cluster stats**: Regularly check `cluster stats`
4. **Balance periodically**: Run `cluster balance` weekly or after major changes
5. **Update after changes**: Run `domain update` after VM config changes
## Safety
1. **Enable maintenance mode**: Before node maintenance, use `node maintenance`
2. **Verify evacuations**: Check `domain list` after enabling maintenance mode
3. **Test migrations**: Use `--dryrun 1` before migrating critical VMs
4. **Keep backups**: Backup `var/status.yaml` before major operations
5. **Document changes**: Log all manual migrations and changes
## Performance
1. **Limit scan frequency**: Default 5-minute minimum is reasonable
2. **Use TSV for scripts**: Faster parsing than screen format
3. **Target specific nodes**: Use `-t node` instead of scanning all
4. **Batch operations**: Group multiple updates together
## Troubleshooting
1. **Start with verbose**: Use `--verbose 1` for basic troubleshooting
2. **Use debug levels**: Progress from `--debug 1` to `--debug 3`
3. **Force rescans**: Use `--force` if data seems stale
4. **Check SSH**: Verify `ssh node` works manually
5. **Review logs**: Check `/var/log/syslog` for cron job output