This morning on one of my servers I notice downtime of a few mins. So I checked Apache logs and noticed there were some hung requests which then caused a request pile up that eventually exeeded the MaxRequestWorkers setting (500). Generally only 10% of MaxRequestWorkers is in use. There was not any traffic/network spike present.
Investigating further I noticed the server was swapping at the same time as the Apache requests spike:
Memory looks like this:
The stuck queries (php-fpm) piled up until the MaxRequestWorkers error:
system stats:
A bit odd one, checking PHP-FPM as well.