How can I add a custom url handler on Windows. Like iTunes itms://

If it's simple, you can do it via the command line:

ftype telnet # view current binding
ftype telnet=\path\to\putty.exe %1

Otherwise you'll need to use the registry as previously posted.

It's a registry keys that provides the custom URI handlers.

Here's an article on it on MSDN, to quote some of it:

HKEY_CLASSES_ROOT
     alert
          (Default) = "URL:Alert Protocol"
          URL Protocol = ""
          DefaultIcon
               (Default) = "alert.exe"
          shell
               open
                    command
                         (Default) = "C:\Program Files\Alert\alert.exe" "%1"

That would you allow to do

alert:some message

And your application would obviously receive the "some message" as its argument.

To change the telnet: protocol handler (or any other existing handler) on XP (may work for Vista - I haven't looked), go to Control Panel > Folder Options > File Types (Tab). Scroll down (not much, it's near the top) to find (NONE) URL:Telnet Protocol, and hit the "Advanced" button. From here on in, it's just like changing a file type association. Note that the target application needs to support the passing of the supplied URL as a parameter.

Unfortunately, the "New" button doesn't seem to allow the creation of new "URL" types, as it requires the entry of an extension. To create a new one, I've previously had to copy an existing entry from the registry (that is: find, export, edit in text editor, and re-import).