I can't find Infinium. Where/How can I find them?

Infinium Ore

It is rare indeed. And the latest patch made it more rare. But they still can be found more commonly in The Fae Wilds and near the edge of the map at Candoria and The Cursed Vale.

Infinium Ore is more common in underground caverns than on the surface. It is also most common in the following biomes: Fae Wilds, Candoria, and Cursed Vale.

"...what is the most efficient way to get them"

I would suggest going to Candoria(it's just my preference, because I see infinium easily here - mostly in chocolate or caves) during tuesday because you have +50% mining ore bonus. You can find infinium also in Medieval Highlands and other biomes mentioned by others. In other biomes it's quite rare.

The best way to mine ore in general is to either use crafted bombs/shooting dragons/high lasermancy gear or you can use Boomranger class whose skill 1 is a bomb that destroys a big area of blocks above it. The optimal way to use this bomb is to dig few blocks under the stack of ore you wanna get, place the bomb and also use your right click(when you are still underground so the boomerang comes back to you instantly) so you minimize the cooldown of bomb. This usually doesn't get down all infinium blocks but they are damaged a lot so it's easy to laser them down with normal gear.

If you want to look for all ores Dragonfire peaks is good because of the huge and full underground caves. But mostly it consists of shapestone and primordal flame as said above.