What is the danger in having a fully used SWAP in an Ubuntu server?
Solution 1:
Swap basically serves two roles - firstly to move out less used 'pages' out of memory into storage so memory can be used more efficiently. Secondly if memory is insufficient, it acts to 'add' memory.
If its the former case, its ok. With the latter case, there's two possible scenarios. Firstly, you'd have increased disk use. If your disks arn't fast enough to keep up, then your system might end up thrashing, and you'd experience slowdowns as data is swapped in and out of memory. This would result in a bottleneck.
The second possibility is you might run out of memory, resulting in wierdness and crashes.
There's a few ways to work around this. Firstly, more swap. One option might be to increase the size of your swap partition when you next do maintainance, or to switch over to a swap file. Both these approaches are bandaiding the main issue, but should tide you over for the short run.
The real solution is to add more memory. There's no substitute for real butter ram, and if you have enough memory, you'll swap less.
If you for some reason still are swapping heavily despite maxing out your memory, consider using faster storage for swap - an SSD might be an option thats faster than having swap on a spinning drive. This will likely end up in a faster system than using a spinning drive, and with a large enough SSD, with a larger amount of spare space set you can maximize the lifespan of the drive and improve performance constancy, at the expense of some space. Its a throwback but having a seperate swap drive might be an option here.
If you want to do it right you might also want to work out what is using up that much ram, and why. Htop is a good start, and there's an entire question on that, then attempt to tune the program as needed. This is of course dependent on overall performance and other requirements and may be specific to the program, and what its used for
Solution 2:
The danger is the rather ominous sounding OOM Killer.
http://linux-mm.org/OOM_Killer