logrotate failing silently for weeks: how one `|| true` hid 3 bugs and stopped all log rotation

The symptom: disk filling, logs not rotating

A server with disk creeping up for no obvious reason. Digging in, the logs in /var/log were no longer rotating: weeks-old files, some in the gigabytes, with none of the usual .1, .2.gz suffixes. logrotate was installed, scheduled, and — per systemd — "running". Running and failing.

The treacherous detail: logrotate processes config files in sequence and, on hitting a fatal error, returns exit code 1. To the scheduler (cron / systemd timer), that's just "the unit failed" — no alarm, no page. And because an error in a single stanza taints the whole run's status, the impression was that NOTHING rotated, when in fact a handful of broken configs were poisoning the result.

Making the silence visible: logrotate in debug mode

logrotate doesn't shout by default. But debug mode (-d, which is a dry-run) plus verbose (-v) show exactly where it trips, without changing anything:

# -d = debug/dry-run (writes nothing), -v = verbose
sudo logrotate -dv /etc/logrotate.conf 2>&1 | less

# To see only the errors:
sudo logrotate -dv /etc/logrotate.conf 2>&1 | grep -iE "error|warning|skipping|duplicate"

Out came three distinct errors, from three independent causes. Each, on its own, was enough to return exit 1.

Bug 1: duplicate config across two include directories

The main config included two directories: the standard /etc/logrotate.d/ and an internal automation directory, /opt/ops/logrotate.d/. Both had stanzas for the same files (syslog and the AIDE logs). logrotate treats the same logfile declared twice as a fatal error:

error: /opt/ops/logrotate.d/aide:1 duplicate log entry for /var/log/aide/aide.log

Fix: one logfile, one owner. We removed the duplicate from the automation directory and left each file in a single stanza. Simple rule: if you include more than one directory, make sure they don't overlap.

Bug 2: a world-writable directory logrotate refuses to touch

The PHP-FPM log directory had drwxrwxrwx (777) permissions. For safety, logrotate refuses to rotate files in directories writable by anyone — unless you explicitly tell it which user/group to operate as, via the su directive:

error: skipping "/var/log/php-fpm/error.log" because parent directory has insecure permissions
(It's world writable or writable by group which is not "root") Set "su" directive in config file
to tell logrotate which user/group should be used for rotation.

There were two paths: fix the directory permission (the right one) or declare su in the stanza. We did both — tightened the permission to 755 and added the directive as a safety net:

/var/log/php-fpm/*.log {
    su root adm
    weekly
    rotate 8
    compress
    missingok
    notifempty
}

Bug 3: the empty glob that's fatal even with missingok

The subtlest gotcha. A stanza pointed at /var/log/aide/*.log. The directory existed but was empty — no file matched the glob. Intuition says "I have missingok, so I'm fine". Wrong: missingok covers a missing literal path, but a glob that matches nothing is treated as a fatal error by logrotate on many versions:

error: /opt/ops/logrotate.d/aide:1 glob failed for /var/log/aide/*.log

Since AIDE hadn't yet generated logs at that path, the whole stanza sat there waiting to fail. Fix: remove the stanza until files exist, or point at the correct literal path. Optimistic glob + empty directory = silent exit 1.

The cause behind the cause: the `|| true` that blinded everything

Even after rotation worked again, we still had to understand why nobody had noticed the problem before. The answer was in the postrotate of one stanza — a hook running a Python script to ship log metrics to a bucket:

postrotate
    /usr/bin/python3 /opt/ops/ship-logs.py >/dev/null 2>&1 || true
endscript

That >/dev/null 2>&1 || true throws away all output and forces success no matter what. Running the script by hand, what was hidden showed up instantly:

$ /usr/bin/python3 /opt/ops/ship-logs.py
Traceback (most recent call last):
  File "/opt/ops/ship-logs.py", line 3, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'

The Python environment had been rebuilt during some upgrade and lost its dependencies. The script failed on every run — but the || true swallowed the error, and postrotate "passed". Fix: reinstall the dependencies and stop masking the error:

pip install pandas pyarrow boto3
# pandas 2.3.3, Python 3.9.18

# and in postrotate: let the error surface (logrotate records the hook failure)
postrotate
    /usr/bin/python3 /opt/ops/ship-logs.py
endscript

|| true is not error handling — it's error suppression. Use it only when a failure is genuinely irrelevant, and never glued to a 2>&1 that also erases the message. Otherwise you're building a silence that will cost you dearly on a bad day.

How to test rotation before it bites you

After any change to a logrotate config, validate it without waiting for the 3am cron:

# 1. Dry-run: shows what it would do, writes nothing
sudo logrotate -dv /etc/logrotate.conf

# 2. Force a real run and check the exit code
sudo logrotate -fv /etc/logrotate.conf ; echo "exit: $?"

# 3. Timer health in systemd (a failure does NOT become an alert on its own!)
systemctl status logrotate.timer
journalctl -u logrotate.service --since "yesterday" | grep -i error

A non-zero exit there is your only early warning. If nobody watches that exit code, the failure stays invisible until the disk fills.

Lessons

1. One broken stanza taints the whole run. logrotate returns exit 1 on the first fatal error; to cron/systemd that's just "failed", with no hint which config. Treat the exit code as a monitorable signal.

2. missingok does not cover an empty glob. Missing literal path: fine. A glob that matches nothing: fatal error. Don't create optimistic stanzas for logs that don't exist yet.

3. A world-writable directory needs `su` (or correct perms). logrotate refuses to rotate logs in a 777 directory for safety. Tighten the permission and/or declare su user group.

4. `|| true` with `2>&1` is a time bomb. It hides the failure AND the message. If a hook can fail in a way that matters, let the error surface and be recorded.

When a green signal lies, the pattern repeats across layers — as in a site returning 500 with a healthy WAF. The instrument said "ok"; reality didn't.

Conclusion

None of the three bugs was hard — a duplicate include, a loose permission, an empty glob. What made them dangerous was their sum with a || true that erased the only evidence. Infrastructure maintenance is, in large part, refusing convenient silences: making the error surface today, in the dry-run, instead of letting it surface on its own three weeks from now, with the disk at 100% and the service going down.