# Recommended ErrorLogFormat for a combined vhost error log

## Status: superseded, not pursued further (2026-09-15)

This was moved here from `apache-notes/` and `conf-available/` because it is no longer the planned path forward — kept for the reasoning and the live-test evidence, not as something to deploy. Do not `a2enconf` `fail2ban-error-log.conf` from this directory as a next step; if this ever gets revived, treat it as starting over, not resuming.

**Why abandoned:** the live test on `stargazer.rodolico.org`/`smartappliances.us` (2026-09-15, ~11:16–11:54) came back inconclusive and mildly discouraging rather than a clean confirm of either open question below — `other_vhosts_error.log` picked up only Apache's own startup/lifecycle messages during the window, not the one genuine per-request error that landed in the normal `error.log` at the same time, and neither test vhost happened to throw a real error of its own to give question 1 an actual test case. Disabling the config afterward wasn't followed by a lasting, unambiguous change either way. Neither open question below was ever answered.

Separately, the two most valuable things this would have unlocked turned out not to need it:

- The cross-site path-traversal/secrets-scan pattern (`AH10244: invalid URI path` in the error log) has an exact 1:1 shadow as a `400` status code in the access log — confirmed against `testing/allegra.log`. That, plus the same burst's `403`/`404` responses, is now caught generically by `filter.d/apache-vhost-scan-burst.conf` / `jail.d/apache-vhost-scan-burst.local`, built directly against the already-proven `other_vhosts_access.log` — no `ErrorLog` plumbing required.
- Three of the seven stock error-log filters this would have enabled (`apache-noscript`, `apache-botsearch`, `apache-nohome`) are dead ends regardless: `testing/allsites.log` (a full-day tail of `error.log` across ~60 ISPConfig sites of every architecture) showed `AH00128`/"File does not exist" never appears anywhere on this host, most likely suppressed below the global `LogLevel warn` threshold. No amount of combined-log plumbing surfaces a message that isn't logged in the first place.

What's left that a working combined error log would still add over `scan-burst`: `apache-auth` (`AH01630`, "client denied by server config") fires cleanly and often per-site in `allsites.log`, and repeated denials of that kind should already show up as repeated `403`s in the access log too — so it's likely also already covered by `scan-burst`'s generic 4xx-burst counting, just not confirmed. `apache-overflows` may not need vhost combining at all, since malformed-request-line errors are typically raised by Apache's core parser before vhost dispatch (unconfirmed). `apache-modsecurity`/`apache-shellshock` are inconclusive — no signal seen in the sample, possibly because the module isn't installed, not because the approach fails.

If this ever needs revisiting, the trigger would be: a real attack pattern that's visible in per-site `error.log` but has no access-log shadow at all (unlike the LFI/scan case above) — that's the one shape of problem `scan-burst` structurally cannot catch.

## Problem

`apache-notes/apache-logformat.md` solved this for the access log: ISPConfig gives every vhost its own access log, so a shared `other_vhosts_access.log` was added (via Debian's stock `other-vhosts-access-log.conf` mechanism) to get one place fail2ban can watch for all sites at once.

The stock Apache error-log jails (`apache-auth`, `apache-noscript`, `apache-overflows`, `apache-nohome`, `apache-botsearch`, `apache-modsecurity`, `apache-shellshock`) have the same underlying problem, confirmed 2026-09-15: every ISPConfig vhost on this host sets `ErrorLog /var/log/ispconfig/httpd/<sitename>/error.log`, so the global `%(apache_error_log)s` (`/var/log/apache2/error.log`) these jails watch mostly only sees default/catch-all-vhost errors, not per-site ones. Unlike the access-log case, Debian ships no `other-vhosts-error-log.conf` equivalent — there is no existing combined error log to fall back on.

## Two open questions this is meant to test, before any filter gets rewritten

This is deliberately a separate, smaller step from rewriting the seven filters above, because there are two load-bearing assumptions here that are NOT yet verified the way the access-log ones were (see the "CHECKED" note in `apache-logformat.md`) — get this file deployed and confirmed working first.

**1. Does a main-server-scope `ErrorLog` actually receive a vhost's errors when that vhost has its own `ErrorLog`?**

For the access log, this is proven: `other_vhosts_access.log` really does capture every ISPConfig site's traffic even though each site also has its own `CustomLog`, because `CustomLog` directives are additive across scopes — the main-server one and each vhost's one both fire independently. `ErrorLog` is a different core directive; whether it behaves the same way (additive across scopes) rather than "closest scope wins" has not been confirmed on this host. If it's additive, this works exactly like the access log did. If it isn't, the global `ErrorLog` set below will simply never see per-site errors, same as today, and this approach needs rethinking (most likely: file a request against ISPConfig's own vhost template instead).

**2. Does adding a second main-server-scope `ErrorLog` directive coexist with the one `/etc/apache2/apache2.conf` already sets (`ErrorLog ${APACHE_LOG_DIR}/error.log`), or silently replace it?**

This is the more important of the two to check FIRST, because getting it wrong is a regression, not just a dead end: if `ErrorLog` is last-directive-wins per scope rather than stackable, enabling `conf-available/fail2ban-error-log.conf` would redirect the *entire* main server's error output to `other_vhosts_error.log` and stop `/var/log/apache2/error.log` from receiving anything at all. **After enabling this, confirm `/var/log/apache2/error.log` is still being written to** (e.g. `sudo tail -f /var/log/apache2/error.log` while triggering something, or just watch its mtime) before drawing any conclusion about question 1.

If question 2 comes back negative (single ErrorLog, last-wins), `a2disconf fail2ban-error-log` immediately reverts it — the main error log resumes working the moment the fragment is disabled and Apache reloads, same rollback story as the access-log drop-in.

## The format

Unlike `CustomLog`, `ErrorLog` takes no inline format argument (`CustomLog path format` vs. just `ErrorLog path`) — the format is controlled separately by `ErrorLogFormat`, and that directive is scoped to wherever you declare it (server or vhost context), not to one specific `ErrorLog` target. Apache's own hard-coded default error format (used when `ErrorLogFormat` is unset) has no vhost-identifying field at all — unlike `vhost_combined`, which already carried `%v` — so without setting our own, a combined error log still couldn't be attributed to a site even if question 1 above turns out to be true.

```apache
ErrorLogFormat "[%t] [%-m:%l] [pid %P] [client %a] %v: %M"
ErrorLog ${APACHE_LOG_DIR}/other_vhosts_error.log
```

- `%t`, `%-m:%l`, `%P`, `%a` mirror Apache's traditional error-log look (timestamp; module:level; process ID; client address) — this is a deliberately simplified format, not a byte-for-byte reproduction of the stock default, and hasn't been fidelity-checked against it.
- `%v` is the one addition that matters here — the vhost's `ServerName` — exactly the role `%v` already played in `vhost_combined` for the access log.
- `%M` is the actual log message.

**Side effect to expect, not a bug:** because `ErrorLogFormat` is scoped by context rather than by target, setting it here (main server context) also becomes the *default* format for every vhost's own `ErrorLog` unless that vhost sets its own `ErrorLogFormat` — and ISPConfig's vhosts don't. Expect every per-site `error.log` under `/var/log/ispconfig/httpd/` to start showing this same `[client ...] <vhostname>: message` format too, not just `other_vhosts_error.log`. That's expected noise (each site's own error log gets a slightly more verbose line, nothing is hidden or broken), not something to fix.

Both lines are packaged as a drop-in: `Documentation/dead-ends/fail2ban-error-log.conf` (moved here with this file 2026-09-15, out of the now-inapplicable `conf-available/`).

## Deploying it (for testing)

```bash
sudo cp Documentation/dead-ends/fail2ban-error-log.conf /etc/apache2/conf-available/fail2ban-error-log.conf
sudo a2enconf fail2ban-error-log
sudo apache2ctl configtest
sudo service apache2 reload   # SysV; use "systemctl reload apache2" where systemd runs
```

There is no stock conf to disable first this time — unlike `other-vhosts-access-log`, Apache ships nothing analogous for the error log, so there's no `a2disconf` step before enabling ours.

To test:

1. Confirm `/var/log/apache2/error.log` is still being written to (question 2 above) — do this before anything else.
2. Trigger a real error on one ISPConfig vhost (e.g. request a nonexistent path so Apache logs a "File does not exist" line to that site's own `error.log`).
3. Check whether the same line also landed in `/var/log/apache2/other_vhosts_error.log`, tagged with that vhost's name via `%v` (question 1 above).

To roll back cleanly at any point: `sudo a2disconf fail2ban-error-log && sudo service apache2 reload`.

(Historical plan, not acted on — see "Status" above.) The original intent was: once both questions came back confirmed, the seven error-log filters would get the same `-vhost-` treatment as `apache-vhost-badbots`/`apache-vhost-fakegooglebot` (see README.md "Naming convention"): `apache-vhost-auth`, `apache-vhost-noscript`, `apache-vhost-overflows`, `apache-vhost-nohome`, `apache-vhost-botsearch`, `apache-vhost-modsecurity`, `apache-vhost-shellshock`. That never happened — `apache-vhost-scan-burst` covers the highest-value case (cross-site scans) a different way, and the rest range from dead-end to unconfirmed, per "Status" above.