Is there any downside to metal armor over leather armor?

Yes as the sneak factor of an armor is also determined by its weight-class. The heavier the armor, the less sneaky it is. You can also modify it to increase its stealth if the metal armor proves to be heavier.

Metal armor tends to be heavier than a corresponding piece of leather armor. However, this may not be true depending on the mods. Lighter armor is better for sneaking, so assuming the same mods, leather armor will be sneakier.

However, this weight issue isn't particular to metal armor. Any heavier armor, including a piece of leather armor with heavier mods, will be less sneaky than a lighter piece.

What is particular to metal armor vs leather armor are the resistances. Leather armor has high energy resist and low physical resist. Metal is the opposite, with high physical resist and low energy resistance. This is the main difference.