Linux goes to swap while plenty of RAM installed

I have a linux box that is used as SAN storage. It is equipped with 14 SATA HDDs and the only software it run is iscsi target and zabbix agent. This is physical box, not VM.

The RAM size installed is 128 GB, swap was arranged of 4 GB and no-one ever care for its size when RAM is quite big, but as the box run (the uptime is 694 days) swap was used more and more, and now it used for ~80%.

The htop won't show anything bad, looks like OS uses RAM as cache etc. swappiness was set to 10, then to 1, none helped. Surely I can disable swap completely but looks like this will be bad move.

So my question is simple, how can I set the box to use RAM and not use swap, and why it uses swap at all while RAM is enough?

This is SAN, so it serves blocks not files, so cache can be done on block bases, and can easily be reduced if RAM is limited, it won't impact the logic of iSCSI anyway.

Zabbix says right now I have RAM available = 123.22 GB (hey, almost all of 128 GB!), swap used is 3.24 GB of 4 GB.

There is no problem. Even if there is still available RAM, the Linux Kernel will move memory pages which are hardly ever used into swap space.

It’s good to swap out memory pages that have been inactive for a while, keeping often-used data in cache; this is the desired situation of the Kernel.

You can have more control on this by using vfs_cache_pressure (which controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects) and swappiness (which controls how aggressive the kernel will swap memory pages: higher values will increase aggressiveness, lower values decrease the amount of swap).

You can find these settings in /etc/sysctl.conf.