Where can I find a Portal Gem, and how does it work?

Solution 1:

Portal gem is the only unique gem in the game, and, you guessed it, it opens a town portal =)

The upside: Never have to use a TP scroll again! You can save them and trade them for ID scrolls! Fits in any color slot. And you don't really need to "bind a key"... You can just place it on one of the keys when you need it, or put it on your secondary weapons. (that's what I do)

The downside: It has a cast time. At 20% increased quailty, it becomes 40% decreased cast time.

And yes, they are quite rare.

Sources: I have one, wiki

Alas, I died, so no longer have my precious Portal Gem.

Solution 2:

Yes, they are quite rare drop which you can get from almost any monster in any area of any difficulty.

The gem can be put in any socket, no matter the color. And yes, if you want to use it you have to put it under skill slot (if you already use all of them, then just replace one of the skill with teleport when you'll need to use it and bring back the skill to that slot once you're done teleporting).

It's also worth noting that the portal doesn't open instantly like when using portal scroll, it takes 2.5 seconds (lesser if it's quality gem) to open the portal when using the gem so it's not recommended that you try to use it as an emergency escape as it might get you killed.