MySQL memory usage is too high

Solution 1:

When it's about to collapse htop reports about 25Gb RES (96Gb VIRT). Then somehow MySQL is automatically restarted and we're back to square one.

If the OOM-killer is terminating it, that would be visible in syslog. Automatically restarting implies your init scripts or some other HA system intervened. You should find out what happened specifically.

Databases usually have large shared memory and small process memory, not the other way around.

Enable that slow query log and find what queries are not using indexes. Look at the applications using this database and determine what they were doing at the time. Database processes allocate tens of GB of RAM for a reason.