How can I test the security of my server?

Based on your tags, here's some basic advice:

OS (Linux)

• Apply updates/security patches including Kernel
• Checksum tool to detect file/permission changes like aide, fcheck, tripwire etc.
• Only enable network services you're actually using (check with netstat -tulpen)
• Sane user definition: who has root access?
• SSH: disable direct root logins
• Hard- or software firewall

PHP

• Use hardened PHP (Suhosin)
• Google the web for security best practices in PHP

MySQL

• Eiter let it run with Unix sockets or via TCP but only on localhost/your LAN
• Define a root password
• Define restricted users for every different application

That's just the pure basics written down in 2 minutes. Theres much more.

Download Backtrack and run FastTrack's AutoPwn against your server. It's a completely automated approach, but it's a great low effort way of finding the low-hanging fruit.

If you got web components, SkipFish is another great automated testing tool.

There are many tests you can perform and many tools available to test with. For starters you might like to run Nikto.

Although you may believe users can't upload files, a security hole in the applications or services may well prove otherwise, as many have learned the hard way. Always work on the assumption that your system is broken and vulnerable and look for ways to fix it, before someone else finds the holes for you.