How can I get Firefox to open torrent files with Transmission?

Method (I): Using Firefox's Preferences menu: (preferred)

A. From Firefox's global menu, select Edit -> Preferences (or press Alt + E followed by N) to open Firefox Preferences.

(Snapshot taken in Ubuntu 12.04)

B. Select the Applications menu and search for BitTorrent seed file or application/x-bittorrent or torrent. Change the Always ask or Use gedit (default) to transmission-gtk by selecting Use other... and click File System in the side bar, then browse to /usr/bin.

C. In /usr/bin search for transmission-gtk and double click to open.

D. That's it! Now the .torrent files will be opened with transmission.

Method (2): Editing the mimeTypes.rdf file:

Step 1:

Open the mimeTypes.rdf file with text editor of your choice:

gedit ~/.mozilla/firefox/ythemo16.default/mimeTypes.rdf &

(Save a backup of this file.) Search for the container:

<RDF:Seq RDF:about="urn:mimetypes:root">
 ...
 ...

</RDF:Seq>

and insert this:

<RDF:li RDF:resource="urn:mimetype:application/x-bittorrent"/>

so that, now the container looks something like this:

<RDF:Seq RDF:about="urn:mimetypes:root">
 ...
 ...

 <RDF:li RDF:resource="urn:mimetype:application/x-bittorrent"/>
</RDF:Seq>

(step 1 snapshot)

Step 2:

Now, within the container:

<RDF:RDF ... ...>
 ...
 ...
 
 ...
 ...

</RDF:RDF>

insert these:

<RDF:Description RDF:about="urn:mimetype:handler:application/x-bittorrent"
               NC:alwaysAsk="false"
               NC:saveToDisk="false">
 <NC:possibleApplication RDF:resource="urn:handler:local:/usr/bin/transmission-gtk"/>
 <NC:externalApplication RDF:resource="urn:mimetype:externalApplication:application/x-bittorrent"/>
</RDF:Description>
<RDF:Description RDF:about="urn:mimetype:externalApplication:application/x-bittorrent"
               NC:prettyName="transmission-gtk"
               NC:path="/usr/bin/transmission-gtk" />
<RDF:Description RDF:about="urn:handler:local:/usr/bin/transmission-gtk"
               NC:prettyName="transmission-gtk"
               NC:path="/usr/bin/transmission-gtk" />

<RDF:Description RDF:about="urn:mimetype:application/x-bittorrent"
               NC:fileExtensions="torrent"
               NC:description="BitTorrent seed file"
               NC:value="application/x-bittorrent"
               NC:editable="true">
    <NC:handlerProp RDF:resource="urn:mimetype:handler:application/x-bittorrent"/>
</RDF:Description>

so that the container will look something like this:

<RDF:RDF ... ...>
 ...
 ...
 
 ...
 ...

 <RDF:Description RDF:about="urn:mimetype:handler:application/x-bittorrent"
               NC:alwaysAsk="false"
               NC:saveToDisk="false">
  <NC:possibleApplication RDF:resource="urn:handler:local:/usr/bin/transmission-gtk"/>
  <NC:externalApplication RDF:resource="urn:mimetype:externalApplication:application/x-bittorrent"/>
 </RDF:Description>
 <RDF:Description RDF:about="urn:mimetype:externalApplication:application/x-bittorrent"
               NC:prettyName="transmission-gtk"
               NC:path="/usr/bin/transmission-gtk" />
 <RDF:Description RDF:about="urn:handler:local:/usr/bin/transmission-gtk"
               NC:prettyName="transmission-gtk"
               NC:path="/usr/bin/transmission-gtk" />

 <RDF:Description RDF:about="urn:mimetype:application/x-bittorrent"
               NC:fileExtensions="torrent"
               NC:description="BitTorrent seed file"
               NC:value="application/x-bittorrent"
               NC:editable="true">
    <NC:handlerProp RDF:resource="urn:mimetype:handler:application/x-bittorrent"/>
 </RDF:Description>
</RDF:RDF>

(step 2 snapshot)

Step 3:

Save and close the file. Then restart firefox.

Open the Firefox preferences window. Go to the Applications tab and you should see magnet on the list. Click on Always ask and if Transmission does not appear as an option then click on Use other and navigate to /usr/bin on your file system. Look for transmission or transmission-gtk and click that.

Source: How to make Firefox open magnet-links in Transmission

If magnet didn't show up in the applications tab, then you should do this via about:config as explained below:

• Type about:config into the address bar and press Enter.
• Right-click -> New -> Boolean -> Name: network.protocol-handler.external.magnet -> Value -> true
• Right-click -> New -> String -> Name: network.protocol-handler.app.magnet -> Value -> /usr/bin/transmission
• Ensure network.protocol-handler.expose-all is set to true

Source: Magnet links in Firefox using Transmission

Another more user-friendly way to add a new file type to the Edit -> Preferences -> Applications window is by downloading a file that has the file format you want to add, selecting any option, and selecting the "Do this automatically for files like this from now on." checkbox. Source: firefox doc.

Next time you download a torrent file, and the window pops up to ask what to do, give the path for the file to open with: /usr/bin/transmission-gtk and select it. All done.