Rebooting 2× a day: the rate limit that didn't work because nginx saw the wrong IP (real_ip behind a CDN)

The symptom: daily reboots under scraping

An EC2 t4g.medium (2 vCPU) serving a WordPress multisite behind a CDN rebooted twice a day. Under load, the 10 PHP-FPM workers saturated at ~100% CPU, the load climbed to 4.70 (on a 2 vCPU box), and average response time hit 15.77s. It was distributed scraping — many IPs, few requests each — and the rate limit blocked nothing.

The cause: nginx counted the CDN's IP

There was a 12 req/min rate-limit zone by $binary_remote_addr. The problem: behind a CDN, the $remote_addr nginx sees is the CDN PoP's IP, not the client's. Since all traffic arrives "from the CDN", the limit aggregates thousands of clients (and attackers) under a few IPs — and never fires correctly. The CDN had added 2 new PoPs whose ranges weren't in set_real_ip_from, so real_ip wasn't even rebuilt for them.

# nginx must trust the CDN's ranges to read the real IP from X-Forwarded-For
set_real_ip_from 198.51.100.0/24;   # CDN PoP (example)
set_real_ip_from 203.0.113.0/24;    # new PoP that was missing → real_ip not rebuilt
real_ip_header   X-Forwarded-For;
real_ip_recursive on;

# only then does the rate limit count the right client:
limit_req_zone $binary_remote_addr zone=perip:10m rate=12r/m;

Per-IP rate limiting behind a CDN only works if nginx rebuilds the real IP first. Without set_real_ip_from covering ALL PoPs, you limit the CDN — not the attacker.

The mistake we made: blocking the CDN itself

In the rush, one attempt was to block the ranges "the traffic" came from — which were the CDN's PoPs. That takes the whole site down for everyone. The lesson: before blocking a range, confirm whether it's the attacker's or your own edge. With real_ip correct, the actual source IPs appear and the block aims at the right place.

The second tweak: the pagination regex

The scraping hit deep pagination URLs. The rule detecting the pattern used {3,} (3+ digits), letting 2-digit pages through. Tightening to {2,} caught most of the abuse without affecting legitimate browsing:

# before: only caught /page/100+;  after: catches /page/10+
location ~* "/page/[0-9]{2,}" { limit_req zone=perip burst=5 nodelay; }

The result

Average response time : 15.77s  ->  < 1s
Load (2 vCPU)         : 4.70    ->  0.24
PHP-FPM CPU           : ~100%   ->  normal
Reboots/day           : 2       ->  0

Operational detail: journald wasn't persistent, so logs from the prior crashes were lost. Persisting the journal (Storage=persistent) was part of the fix — without logs, every reboot was a fresh mystery.

Lessons

1. set_real_ip_from must cover ALL CDN PoPs. A new PoP outside the list breaks real_ip and blinds the rate limit.

2. Never block a range without confirming the source. Blocking your own CDN's IPs takes the site down for everyone.

3. Persistent journald or you lose the evidence. Without Storage=persistent, each reboot erases the reason for the previous one.

Conclusion

A rate limit is only as good as the IP it counts. Behind a CDN, nginx sees the PoP — and a per-IP limit becomes useless until you rebuild the real IP with set_real_ip_from for every edge range. With the right IP, 12 req/min meant 12 req/min per client again, and the server stopped rebooting.