Ubuntu Server minimal package
Perhaps the Ubuntu "Minimal CD", aka the "minimal ISO image", is close enough to what you are looking for.
- "Installation/MinimalCD": http://help.ubuntu.com/community/Installation/MinimalCD
After doing a minimal install, I think there's only one more thing you need to do to allow ssh connection:
The Ubuntu Server Guide says (in the OpenSSH Server section) that
sudo apt-get install openssh-server
allows people to ssh to your server. You'll probably also want to to the other things suggested at How do I configure a new Ubuntu installation to accept ssh connections? .
EDIT:
If on one machine one installs the "Minimal CD", and then runs
sudo tasksel install server
and on another machine one installs the "Ubuntu Server CD", my understanding is that the two machines are practically the same.
If you know what the difference is between them, please post that difference to Is using Ubuntu Minimal any different from Server? .
See What does the "Basic Ubuntu Server" package contain in software selection during installation? for more details.