How is my server hitting swap?

Under normal circumstances, linux will swap things out when they're not being used, even if there is still free space in RAM. Instead, it'll keep cached files. You can adjust this behaviour by changing vm.swappiness with systctl. More details on the exact details of how this works can be found here.

Linux aggressively swaps out address space which is not actively used for some time. It does so to make more RAM available to filesystem caches (well, actually SLAB caches) or applications using shared memories, a ram disk and so on. Usually, this is exactly what you want.

For further reading, I recommend Linux: Tuning Swappiness for a thorough explanation, although you probably won't have to touch the tunables mentioned there, ever.