How can I register a custom protocol with xdg?

I've been struggling this morning trying to associate an application with a custom protocol, namely emacsclient and org-protocol.

I'm calling this protocol from a webbrowser bookmarklet, and I get the following behaviour :

In chromium,
the "Launch Application" dialog comes up, and calls xdg-open org-protocol://... which ends up firing a new chromium frame.

In firefox,
I've tried setting network.protocol-handler.app.org-protocol to an empty string or my emacsclient path, anyhow I get the following error message :
"Firefox doesn't know how to open this address, because the protocol (org-protocol) isn't associated with any program" without even showing any external application selection dialog.

I'm not using any desktop environment, so I need to make this work strictly with xdg, however, despite reading the shared mime info spec etc, I still can't fathom a working configuration.

The XDG method: Create an emacsclient.desktop file that handles the x-scheme-handler/org-protocol MIME type:

[Desktop Entry]
Name=Emacs Client
Exec=emacsclient %u
Icon=emacs-icon
Type=Application
Terminal=false
MimeType=x-scheme-handler/org-protocol;

(Note the MimeType= line above, which is for org-protocol: URIs.)

Put the file in ~/.local/share/applications or /usr/local/share/applications.

You might have to open the file ~/.local/share/applications/mimeapps.list and add a line like this:

x-scheme-handler/org-protocol=emacsclient.desktop

Add that under the [Default Applications] section.

@grawity is almost correct, but he overlooked two details:

1. After installing the .desktop file, you may need to manually run sudo update-desktop-database. (This may be done for you if you installed the .desktop file using xdg-desktop-menu install emacsclient.desktop)

   That was what was required to get my Firefox to display the "What should handle this protocol?" dialog rather than just displaying an error page.

2. At the time this question was asked, there was a bug in xdg-open which caused it to blindly hand URLs over to $BROWSER on "generic" (non-KDE, non-GNOME, non-Xfce, etc.) desktops and that produces exactly the behaviour @julien was experiencing.

   It should be fixed in modern versions of xdg-open but, if your desktop isn't using a modern version for some reason, here's the version of xdg-open I drop into /usr/local/bin to solve the problem.