# checkVirtuals

This script checks a list of libvirt domains and restarts any domain that remains down after a second consecutive check. It is intended to be run from cron on a regular interval, such as every five minutes.

## What it does

For each domain listed in the configuration file:

- If the domain is running, the script removes any stale `/tmp/<domain>.down` flag file.
- If the domain is not running and no flag exists yet, the script creates `/tmp/<domain>.down` and sends an email notification.
- If the domain is still not running on the next pass and the flag already exists, the script attempts to start it with `virsh start <domain>` and sends another email notification.
- If the domain comes back up, the flag is removed.

This creates a two-pass detection cycle so the script does not immediately restart a domain on the first missed check, but does restart it if the domain remains down on a later pass.

## Files

- `checkVirtuals` — the Perl script
- `checkVirtuals.cfg.sample` — sample configuration file
- `checkVirtuals.cron` — sample cron entry

## Configuration file

The script looks for a file named `checkVirtuals.cfg` in the same directory as the script.

A sample configuration is in `checkVirtuals.cfg.sample`.

Example:

```text
# Notification recipients
mailto: ops@example.com, admin@example.com

# Domains to monitor
enfocus.primarycolors.local
remote0
remote1
remote2
```

Rules:

- Blank lines are ignored.
- Lines beginning with `#` are ignored.
- Any line beginning with `mailto:` is treated as an email recipient list.
- Multiple email addresses may be separated by commas, spaces, or semicolons.
- All other lines are treated as domain names to check.

## Email notifications

The script sends mail to each configured recipient with the local `sendmail` binary if it is available; otherwise it falls back to `/usr/bin/mail`.

If neither is available, it logs a warning and skips email delivery.

Notification behavior:

1. First detection of a down domain
   - creates the `.down` flag file
   - sends an email stating that the domain is down

2. Restart attempt after the flag remains present
   - attempts `virsh start <domain>`
   - sends an email stating that the domain is still down and the restart is being attempted

## Cron setup

A sample cron entry is provided in `checkVirtuals.cron`.

Example:

```cron
*/5 * * * * root perl /path/to/checkVirtuals
```

This checks the configured domains every five minutes.

## Example mail output

```text
Subject: Virtual domain remote0 is down

The domain remote0 is not running.
Created flag file: /tmp/remote0.down
Checked at: Sun Aug 16 2026 12:00:00 GMT
```

```text
Subject: Virtual domain remote0 still down; attempting restart

The domain remote0 is still not running and the restart flag exists.
Attempting restart with: /usr/bin/virsh start remote0
Checked at: Sun Aug 16 2026 12:05:00 GMT
```

## Notes

- The script assumes the `virsh` command is available and the current user has permission to manage libvirt domains.
- The script expects the config file to exist in the same directory as the script.
- Domain names are matched against the output of `virsh list`.
- Flag files live under `/tmp` and are named as `/tmp/<domain>.down`.

## Security / operational considerations

- Ensure the cron job runs with appropriate privileges to execute `virsh`.
- Review the email recipients and keep the notification list limited to the people responsible for domain recovery.
- If a domain is intentionally stopped for maintenance, remove its entry from the config or temporarily remove the flag file to avoid unwanted restarts.