Subversion Repositories havirt

# havirt Usage Guide

## Command Syntax

```bash

havirt [flags] [module] [action] [arguments]

```

- **flags**: Optional parameters (can be placed anywhere in the command)

- **module**: One of `domain`, `node`, or `cluster`

- **action**: The operation to perform

- **arguments**: Action-specific parameters

### Global Flags

| Flag         | Short | Description                          | Default      |

| ------------ | ----- | ------------------------------------ | ------------ |

| `--dryrun`   | `-n`  | Show commands without executing      | 1 (enabled)  |

| `--nodryrun` |       | Alias for -n 0                       | 0 (disabled) |

| `--verbose`  | `-v`  | Show detailed output                 | 1 (enabled)  |

| `--quiet`    | `-q`  | Suppress output                      | 0 (disabled) |

| `--debug`    | `-d`  | Debug level (0-3)                    | 0            |

| `--force`    | `-y`  | Force operation (bypass time checks) | 0            |

| `--format`   | `-f`  | Output format: `screen` or `tsv`     | screen       |

| `--target`   | `-t`  | Target specific node or domain       | none         |

| `--testing`  |       | Use test data (for development)      | 0            |

| `--help`     | `-h`  | Show help                            |              |

| `--version`  |       | Show version information             |              |

### Flag Placement

Flags can be placed anywhere in the command:

```bash

# All equivalent

havirt --format tsv domain list

havirt domain --format tsv list

havirt domain list --format tsv

```

### Getting Help

Help is available at multiple levels:

```bash

havirt --help              # General help

havirt domain              # Domain module help

havirt domain help         # Domain module help

havirt node                # Node module help

havirt cluster             # Cluster module help

```

## Module Overview

Havirt is organized into three main modules:

### domain - Virtual Machine Management

Manage individual VMs (domains). Operations include starting, stopping, migrating, and listing VMs.

```bash

havirt domain [action] [arguments]

```

### node - Hypervisor Management

Manage hypervisor nodes. Operations include adding nodes, scanning for VMs, and viewing node resources.

```bash

havirt node [action] [arguments]

```

### cluster - Cluster Operations

Cluster-wide operations including statistics, load balancing, and iSCSI management.

```bash

havirt cluster [action] [arguments]

```

---

# Domain Module

Manage individual virtual machines across the cluster.

## domain list

Display all known VMs with their status and resource allocation.

**Syntax:**

```bash

havirt domain list [--format screen|tsv]

```

**Examples:**

```bash

# Display VMs in table format

havirt domain list

# Output as tab-separated values

havirt domain list --format tsv

# Redirect to file for processing

havirt domain list -f tsv > vms.txt

```

**Output:**

```

         name   memory vcpu  node maintenance

      webvm1  4194304    4 node1           0

       db01   8388608    8 node2           0

   testvm    2097152    2 node3           1

```

**Fields:**

- **name**: VM name

- **memory**: Allocated memory in KB

- **vcpu**: Number of virtual CPUs

- **node**: Currently running on this node (empty if stopped)

- **maintenance**: 1 = in maintenance mode, 0 = normal

## domain start

Start a VM on a specified node (or current node if not specified).

**Syntax:**

```bash

havirt domain start <domainname> [nodename]

```

**Safety Checks:**

- Verifies VM not running elsewhere in cluster

- Validates VM configuration exists in `conf/`

- Checks target node has sufficient resources

- Rescans all nodes before starting

**Examples:**

```bash

# Start VM on specific node

havirt domain start webvm1 node2

# Start VM on current node

havirt domain start webvm1

# Dry-run to see what would happen

havirt --dryrun 1 domain start webvm1 node2

```

**Common Errors:**

```

# VM already running

Error: webvm1 is already running on node1

# Configuration not found

Error: Configuration file conf/webvm1.xml not found

# Insufficient resources

Error: node2 does not have sufficient memory

```

## domain shutdown

Gracefully shutdown a running VM.

**Syntax:**

```bash

havirt domain shutdown <domainname>

```

**Behavior:**

- Sends ACPI shutdown signal to VM

- Waits for VM to shut down gracefully

**Examples:**

```bash

# Shutdown VM

havirt domain shutdown webvm1

# View shutdown command without executing

havirt --dryrun 1 domain shutdown webvm1

```

# domain migrate

Live migrate a VM from its current node to another node.

**Syntax:**

```bash

havirt domain migrate <domainname> [target-node]

```

**Behavior:**

- Performs live migration (VM stays running)

- If no target specified, selects least-loaded node automatically

- Updates cluster database after migration

- Validates target has sufficient resources

- **Note:** libvirt can fail on domain migration. If this occurs, you will either need to manually migrate or shut down, then start the domain on the target node.

**Examples:**

```bash

# Migrate to specific node

havirt domain migrate webvm1 node3

# Migrate to automatically selected node

havirt domain migrate webvm1

# Check what would happen

havirt --dryrun 1 domain migrate webvm1 node3

# Failure reported due to timeout, but domain actually migrated

havirt domain migrate webmv1 node2

# some error message here

havirt node scan --force

havirt domain list | grep webvm1

# in most cases, if timeout set too low, the final command will show

# the domain actually moved

```

**Migration Time:**

Typically 10-60 seconds depending on:

- VM memory size

- Network speed

- VM workload (memory changes during migration)

**Safety:**

- Source and target must both access shared storage

- Validates resources before starting migration

- Will not migrate to nodes in maintenance mode

- Will not migrate VMs in maintenance mode

## domain update

Update VM metadata by reading XML configuration files.

**Syntax:**

```bash

havirt domain update [domainname]

havirt domain update -t domainname  # Target specific domain

```

**Behavior:**

- Reads XML files from `conf/` directory

- Extracts memory, vcpu, and other settings

- Updates `var/status.yaml` database

- Without argument: updates ALL domains

**Examples:**

```bash

# Update all domains

havirt domain update

# Update specific domain

havirt domain update webvm1

havirt -t webvm1 domain update

```

**When to Use:**

- After adding new VMs to the cluster

- After modifying VM XML configuration

- After restoring from backup

- If database seems out of sync

## domain new

Generate a `virt-install` command template for creating a new VM.

**Syntax:**

```bash

havirt domain new [domainname]

```

**Generated Information:**

- Unused VNC port

- Unique UUID

- Random MAC address (prefix: 00:16:3e - XEN OUI)

- Template from `virt-install.template`

**Examples:**

```bash

# Generate template

havirt domain new mynewvm

# Copy output and customize

havirt domain new mynewvm > create-vm.sh

vi create-vm.sh

bash create-vm.sh

```

**Warning**: MAC addresses are randomly generated and not guaranteed unique. Check for conflicts in large clusters.

## domain maintenance

Set or clear maintenance flag for a VM.

**Syntax:**

```bash

havirt domain maintenance <domainname> <on|off>

```

**Examples:**

```bash

# Enable maintenance mode

havirt domain maintenance webvm1 on

# Disable maintenance mode

havirt domain maintenance webvm1 off

# Check current status

havirt domain list | grep webvm1

```

**Effect:**

- Prevents automatic migration during cluster balancing

- Prevents automatic restart by keepalive scripts

- VM can still be manually started/stopped/migrated

---

# Node Module

Manage hypervisor nodes in the cluster.

## node list

Display all nodes with their resources and status.

**Syntax:**

```bash

havirt node list [--format screen|tsv]

```

**Examples:**

```bash

# Display nodes

havirt node list

# TSV format for scripts

havirt -f tsv node list

```

**Output:**

```

     name     memory  cpu_count maintenance

    node1  67108864         16           0

    node2  67108864         16           0

    node3  33554432          8           1

```

**Fields:**

- **name**: Node hostname

- **memory**: Total memory in KB

- **cpu_count**: Number of CPU threads

- **maintenance**: 1 = in maintenance, 0 = active

## node scan

Scan nodes to discover running VMs.

**Syntax:**

```bash

havirt node scan [nodename]

havirt node scan [-t nodename] [--force]

```

**Behavior:**

- Executes `virsh list` on each node

- Updates cluster database with running VMs

- By default: scans all nodes

- Respects 5-minute minimum between scans (unless `--force`)

**Examples:**

```bash

# Scan all nodes (respects time limit)

havirt node scan

# Force immediate scan

havirt node scan --force

# Scan specific node

havirt node scan node1

havirt -t node1 node scan

```

**Cron Job:**

```bash

# Add to /etc/cron.d/havirt

*/5 * * * * root /usr/local/bin/havirt node scan --quiet 2>&1 | logger -t havirt

```

**Time Limit:**

By default, scans are limited to once per 5 minutes (configurable in `config.yaml` as `min scan time`). Use `--force` to override.

## node add / node update

Add a new node or update existing node information.

**Syntax:**

```bash

havirt node add <nodename>

havirt node update <nodename>

havirt node update              # Update all nodes

```

**Behavior:**

- Connects via SSH to node

- Executes `virsh nodeinfo`

- Extracts CPU and memory information

- Adds to or updates cluster database

**Examples:**

```bash

# Add new node

havirt node add node4

# Update specific node

havirt node update node1

# Update all nodes

havirt node update

```

**Requirements:**

- SSH connectivity must be configured

- libvirt must be running on target node

- Node must be accessible by hostname/alias

## node maintenance

Place node in or remove from maintenance mode.

**Syntax:**

```bash

havirt node maintenance <nodename> <on|off>

```

**Examples:**

```bash

# Enable maintenance mode

havirt node maintenance node2 on

# Run cluster balance to evacuate VMs

havirt cluster balance

# Disable maintenance mode when done

havirt node maintenance node2 off

```

**Effect:**

- Prevents new VMs from starting on node

- Prevents node being selected as migration target

- During cluster balance, all VMs evacuated from node

- Existing VMs continue running until migrated

**Typical Workflow:**

```bash

# 1. Enable maintenance

havirt node maintenance node2 on

# 2. Evacuate VMs

havirt cluster balance

# 3. Verify node empty

havirt domain list | grep node2

# 4. Perform maintenance...

# 5. Disable maintenance

havirt node maintenance node2 off

```

---

# Cluster Module

Cluster-wide operations and statistics.

## cluster stats

Display resource usage statistics for the entire cluster.

**Syntax:**

```bash

havirt cluster stats [--format screen|tsv]

```

**Examples:**

```bash

# Display cluster statistics

havirt cluster stats

# TSV format

havirt -f tsv cluster stats

```

**Output:**

```

     node     memory  cpu_count memory_used memory_free cpu_used cpu_free

    node1  67108864         16    25165824    41943040        8        8

    node2  67108864         16    33554432    33554432       12        4

    node3  67108864         16     8388608    58720256        4       12

```

**Fields:**

- **memory**: Total memory (KB)

- **memory_used**: Allocated to running VMs

- **memory_free**: Available for new VMs

- **cpu_count**: Total CPU threads

- **cpu_used**: Virtual CPUs allocated

- **cpu_free**: Available vCPU slots

## cluster balance

Automatically redistribute VMs across nodes for optimal resource utilization.

**Syntax:**

```bash

havirt cluster balance [--dryrun 0|1]

```

**Behavior:**

1. Evacuates all VMs from nodes in maintenance mode

2. Calculates memory imbalance across active nodes

3. Selects VMs to migrate from overloaded nodes

4. Migrates VMs to underutilized nodes

5. Repeats until cluster is balanced

**Examples:**

```bash

# See what would be balanced (safe)

havirt --dryrun 1 cluster balance

# Actually perform balancing

havirt --dryrun 0 cluster balance

# With verbose output

havirt --verbose 1 --dryrun 0 cluster balance

```

**When to Use:**

- After placing node in maintenance mode

- After adding new nodes to cluster

- When cluster becomes unbalanced

- As part of regular maintenance

**Safety:**

- Respects node maintenance flags

- Respects VM maintenance flags

- Validates resources before each migration

- Will not overload target nodes

## cluster iscsi

Manage iSCSI targets across the cluster.

**Syntax:**

```bash

havirt cluster iscsi                    # List targets

havirt cluster iscsi add <target-ip>    # Add target

havirt cluster iscsi update [node]      # Update sessions

```

**Examples:**

```bash

# Add iSCSI target

havirt cluster iscsi add 192.168.1.10

# List configured targets

havirt cluster iscsi

# Update one node

havirt cluster iscsi update node1

# Update all nodes

havirt cluster iscsi update

```

**Behavior:**

- Stores target in cluster configuration

- Executes `iscsiadm` commands on nodes

- Discovers and logs in to targets

- Updates persistent iSCSI configuration

---

# Common Workflows

## Adding a New VM

```bash

# 1. Generate virt-install command

havirt domain new mynewvm > create-vm.sh

# 2. Customize and run

vi create-vm.sh

bash create-vm.sh

# 3. Discover new VM

havirt node scan --force

# 4. Import configuration

havirt domain update mynewvm

# 5. Verify

havirt domain list | grep mynewvm

```

## Performing Node Maintenance

```bash

# 1. Enable maintenance mode

havirt node maintenance node2 on

# 2. Evacuate VMs

havirt --dryrun 0 cluster balance

# 3. Verify evacuation

havirt domain list | grep node2

# (should show no VMs)

# 4. Perform maintenance work...

# shutdown, update, reboot, etc.

# 5. Bring node back online

havirt node maintenance node2 off

# 6. Re-balance cluster

havirt --dryrun 0 cluster balance

```

## Migrating a VM

```bash

# 1. Check current location

havirt domain list | grep webvm1

# 2. Verify target has resources

havirt cluster stats

# 3. Migrate

havirt --dryrun 0 domain migrate webvm1 node3

# 4. Verify migration

havirt node scan --force

havirt domain list | grep webvm1

```

## Emergency VM Shutdown

```bash

# 1. Graceful shutdown

havirt --dryrun 0 domain shutdown vmname

# If that doesn't work after 60 seconds:

# 2. Find node VM is on

havirt domain list | grep vmname

# 3. Force destroy directly

ssh node1 "virsh destroy vmname"

# 4. Rescan cluster

havirt node scan --force

```

## Recovering from Split-Brain

If a VM appears running on multiple nodes:

```bash

# 1. Identify the problem

havirt --debug 2 domain list

# 2. Determine correct node (check VM console/logs)

# 3. Destroy on incorrect node(s)

ssh wrong-node "virsh destroy vmname"

# 4. Rescan

havirt node scan --force

# 5. Verify

havirt domain list | grep vmname

```

---

# Output Formats

## Screen Format (Default)

Fixed-width columns suitable for terminal viewing:

```bash

havirt domain list

```

```

         name   memory vcpu  node maintenance

      webvm1  4194304    4 node1           0

```

## TSV Format

Tab-separated values for scripting:

```bash

havirt --format tsv domain list

```

```

name    memory  vcpu    node    maintenance

webvm1  4194304 4       node1   0

```

**Use Cases:**

```bash

# Import into spreadsheet

havirt -f tsv domain list > vms.tsv

# Process with awk

havirt -f tsv domain list | awk '$4 == "node1" {print $1}'

# Count VMs per node

havirt -f tsv domain list | tail -n +2 | cut -f4 | sort | uniq -c

```

---

# Dry-Run Mode

By default, havirt operates in dry-run mode for safety.

## Checking Mode

```bash

# Check current setting

grep dryrun /media/shared/havirt/config.yaml

```

## Temporary Override

```bash

# Force dry-run (safe)

havirt --dryrun 1 domain start vm1 node1

# Force execution (one command only)

havirt --dryrun 0 domain start vm1 node1

```

## Permanent Change

```bash

# Edit config

vi /media/shared/havirt/config.yaml

# Change:

flags:

  dryrun: 0  # Now executes commands by default

```

---

# Debugging

## Debug Levels

```bash

# Level 1: Basic operations

havirt --debug 1 node scan

# Level 2: Detailed information

havirt --debug 2 domain migrate vm1 node2

# Level 3: Full command trace

havirt --debug 3 cluster balance

```

## Verbose Output

```bash

# Enable verbose mode

havirt --verbose 1 node scan

# Disable verbose mode

havirt --verbose 0 node scan

```

## Checking Database

```bash

# View entire status database

cat /media/shared/havirt/var/status.yaml

# Check specific VM

grep -A 5 "vmname" /media/shared/havirt/var/status.yaml

# Check node population

grep -A 10 "nodePopulation" /media/shared/havirt/var/status.yaml

```

---

# Best Practices

## Regular Operations

1. **Run scans regularly**: Set up cron job for `node scan`

2. **Use dry-run first**: Always test with `--dryrun 1` before executing

3. **Monitor cluster stats**: Regularly check `cluster stats`

4. **Balance periodically**: Run `cluster balance` weekly or after major changes

5. **Update after changes**: Run `domain update` after VM config changes

## Safety

1. **Enable maintenance mode**: Before node maintenance, use `node maintenance`

2. **Verify evacuations**: Check `domain list` after enabling maintenance mode

3. **Test migrations**: Use `--dryrun 1` before migrating critical VMs

4. **Keep backups**: Backup `var/status.yaml` before major operations

5. **Document changes**: Log all manual migrations and changes

## Performance

1. **Limit scan frequency**: Default 5-minute minimum is reasonable

2. **Use TSV for scripts**: Faster parsing than screen format

3. **Target specific nodes**: Use `-t node` instead of scanning all

4. **Batch operations**: Group multiple updates together

## Troubleshooting

1. **Start with verbose**: Use `--verbose 1` for basic troubleshooting

2. **Use debug levels**: Progress from `--debug 1` to `--debug 3`

3. **Force rescans**: Use `--force` if data seems stale

4. **Check SSH**: Verify `ssh node` works manually

5. **Review logs**: Check `/var/log/syslog` for cron job output