Why bother with Meteorite armour?

Its pretty much just for using the space gun, which Meteor Armor makes cost 0 mana to shoot, plus as it counts as a magic attack the +15% magic attack bonus also applies.

Like you say, the defense it provides is fairly low - if you are using Silver Armor then I'd say that loosing 3 defense might be worth it for the space gun (never used it myself so I'm not sure how good it is), however either way you will probably be getting the Shadow Armor fairly soon which offers a lot more defense, so you won't be wearing it for long.

Personally I'd make a Phaseblade, and possibly some Meteor Shot first, and then make the armor if you have any left over - Meteorite Ore is fairly abundant and isn't really used for that much stuff, so sooner or later you will have enough to make some eventually anyway.

If you get a bit more Meteorite Ore, you can build yourself a Space Gun, which is a rapid-fire gun that does moderate-to-low damage but requires only magic to shoot.

It's a decent pick against certain bosses, though there's definitely more powerful options out there. Especially if you have enough money for a minishark and some bullets (not to mention building a Star Canon).

Is it useless though? Hardly. There are more magical weapons in Terraria than just the Space Gun after all. Demon Scythe, for one, is a very useful spell you can get from Hell. The Aqua Scepter and Magic Missile from the dungeon have their own uses for sure. That Orb of Light is somewhat handy in certain cases, if you feel like you need a constant nearby light source. And I found myself using Vilethorn a LOT against bosses, due to its ability to linger and cause multi-hit damage, to say nothing of the new Crimson magic spell (which doesn't take up much magic to use, so YMMV).

Still, your mana will regenerate on its own regardless of whether or not you wear Meteor Armor, and the added speed at which it increases can only really be useful for a magic-heavy armament. While it CAN be used in that way, you may be better off saving your meteor ore for Shots.