How do I change which app Transmission uses on the "open folder" option?

Solution 1:

  1. From the Applications menu, open MIME Type Editor
  2. Search for inode
  3. Change "inode/directory" to your preferred file manager.

From now on Transmission will use your chosen file manager.