Swap being used when not necessary
Solution 1:
Do you have lot of stuff running on the server? If so, maybe consider a dedicated machine or VM for the DB.
Is this physical or VM/cloud server? Some cloud providers do NOT want you to use swap at all (so just putting that out there in case it is relevant): https://docs.rackspace.com/support/how-to/swap-space-on-cloud-servers/
There is some good discussion on this page for troubleshooting swap for MySQL or MariaDB -- maybe check the NUMA section toward the end: https://fromdual.com/do-not-underestimate-performance-impacts-of-swapping-on-numa-database-systems
This Q&A at MariaDB site leads to NUMA interleaving settings as a possible solution: https://mariadb.com/kb/en/swap-usage-wont-decrease/
A lot of discussion about memory and MariaDB: https://mariadb.com/kb/en/mariadb-memory-allocation/
And this is the authoritative youtube video for NUMA :-D https://youtu.be/KmtzQCSh6xk
Please upvote/accept this answer if it is helpful. Thanks.