Apache2 and nginx randomly consuming all memory, every week or so

I had a similar issue where a Wordpress site would start using tons of resources to the point where the server became largely unresponsive. Investigating the logs, I saw a few hundred attempts to access xmlrpc.php right around the same time as the memory footprint would balloon. The functions in xmlrpc.php can be abused as a force multiplier in brute force attacks using the system.multicall method.

This article is a more articulate description of how it works: https://blog.sucuri.net/2015/10/brute-force-amplification-attacks-against-wordpress-xmlrpc.html

More importantly, here are a few mitigation strategies from that article:

Protecting Yourself

I used to recommend people block all access to xmlrpc.php, but it was breaking some plugin’s functionality (mostly JetPack). With that in mind, if you are not using JetPack or any of the other plugin that require it XML-RPC, it might be a good idea to block direct access to it altogether.

If you can’t block XML-RPC, and you are using a WAF (web application firewall), I highly recommend blocking system.multicall requests. It is barely used in the wild and will protect you against these amplification methods.

I don't use any plugins that require access to xmlrpc.php, so I modified .htaccess to deny access. Since then, no more malicious actors successfully crushing the site. Here's the code if you'd like to give that a try:

Using the text editor of your choice, modify /var/www/html/.htaccess to include:

    <Files "xmlrpc.php">
    Order Allow,Deny
    deny from all
    </Files>

Wordpress has additional guidelines for hardening access to your site found here: https://wordpress.org/support/article/hardening-wordpress/

The Login Security Solution plugin for Wordpress may also help. I'd post the link, but I lack the reputation. Sorry!