Optimize a fresh Ubuntu server installation
This will very much depend on what you want to use the server for, so anything specific that we suggest may be irrelevant to you...
My general advice is to not start by doing a less than base install then removing packages though.
I tend to use Debian Stable for server installs, and I ensure that nothing other than the absolute essentials goes on in the initial setup (including deselecting the default "standard system" option when asked by tasksel during the install procedure) and I add packages and settings that are needed from that point. I assume that the same could be done with an Ubuntu server install as Ubuntu is very close to Debian in many respects.
First identify the services(ports) your server will provide, then disable everything else, there are a plenty of tools you can use to secure your server like bastille or tripwire. Depending on the services you are running, you have to look for specific tuning of the installed software.
First: sudo apt-get install rcconf
sudo rcconf
...disable all unnecessary services. And reboot system
Please excuse my bad english...
It sounds like you're concerned about security.
If this is the case, it is a good idea to take an md5sum and perhaps a copy of critical low-level system tools that would be used in case of a system compromise (such as ls, ps, netstat, etc AND md5sum) in their pristine state before you ever connect the system to the internet.
Then set up a process to monitor those files and notify you if the md5sums ever change. One of the first things most rootkits do is to replace these tools to hide their presence on your system.
There is a package called bastille which can be used for server-hardening.
It asks you a few questions about how the server will be used, and tightens up some of the default security settings.
Might be worth a look.