Mac OS X: how to run GUI apps on a headless machine

You need to set up a VNC server and connect using a client or buy the Apple Remote Desktop.

This does not work for GUI apps. There are some command line apps that will play music etc. but not iTunes.

If you have a machine that is headless and you can hit it with VNC or Screen Sharing to log in, then you can use open /Applications/iTunes.app to launch application from ssh in the current users GUI with out logging in to the GUI. However you need the right permissions.

You can combine setatakahashi's answer above with OS X's fast user switching, since you mentioned that you were prepared to first log into the machine you would like to control.

Log in to the machine you want to control. Fire up a VNC server like OSXVNC and set the port to 5902 or something of your choice that isn't the default 5900 used by the VNC server included in all Leopard Macs. (And which can be enabled by any admin user: Sharing pref pane=>Screen Sharing=> Allow VNC client to connect.) Set a password.

Enable fast user switching (Accounts pref pane=> Login options) and then, from the FUS icon in the menubar, choose "Login Window". The computer will then present Mac OS X's login window and allow others to log in and out normally without interrupting your session. (If the machine's preferences are set to display empty name and password fields, your active, hidden session will likely go unnoticed till someone logs in and clicks the FUS icon.) Your console session remains active, as does your VNC server. You can now control that session using a regular VNC client pointed at the right port on the remote machine. JollysFastVNC is a good, fast choice if your controller's also a Mac.

There are many security considerations, starting with a strong recommendation to use an SSH tunnel. Each account on the remote machine needs a password to stop users switching willy-nilly, users can kill your session by shutting down, sleeping, etc. Reliance on a wireless network might kill your connection, depending on how things are set up. The other stuff is probably outside the scope of this question. :)

Is the computer running truly headless (as in WindowServer is never started - usually Xserves) or is it simply a computer that doesn't have a user logged into?

The short answer is no. Mac OS X 10.5 will specifically disable applications from opening up and running over top of loginwindow. (Which was a rather annoying security hole when installers cough Flash cough would open a Finder window over loginwindow giving anybody walking by root access)