What is the penalty % for damage when using the wrong weapon?

The reload on a sniper rifle takes too long (longer than my charge cooldown), so times I stick with a fast reloading SMG against a big armored target such as a brute. How much penalty am I under if I attack a barrier with a sniper rifle or an armored target with a SMG?

You're no specifically penalised, you just do miss out of the damage bonus that a sniper rifle would have against an armoured target.

If you look at this wiki page for the first sniper rifle (the Mantis) you can see it has a ×1.5 damage multiplier against armoured foes and unmodified damage (×1.0) for shields and barriers, as is the case for most of the sniper rifles.

An SMG (for example) would be the other way around, normally with a ×1.5 multiplier against shields and barriers, but unmodified damage (×1.0) against armour.

If you want to think in raw numbers (and assuming your SMG and sniper have identical DPS including reload times) you are therefore losing 33% (×1.5 → ×1.0) of your damage against an armoured foe when using most SMGs compared to most sniper rifles.

Mass Effect 3 does not have the multipliers like ME2 did. There are two game mechanics in ME3 that are relevant to this question. The first is armor resistance, where the damage of any shot hitting armor is reduced by a constant amount (amount varies by difficulty and possibly by enemy). So weapons that rapidly fire lots of low-damage bullets will do much less damage to armor. But a hard-hitting sniper, for instance, will hardly notice the reduction. There are a number of abilities, upgrades, and modifications that affect this resistance.

The second game mechanic that has, in a way, replaced the damage bonuses is "shield gating". This limits blow-through damage from a single shot when destroying shields or barriers. The severity of the limitation varies by difficulty, with no blow through damage allowed on Insanity. This mechanic tends to limit hard-hitting, slow-firing weapons much more than fast-firing ones, since they stand to waste much more damage.

Note that in both these cases, each shotgun pellet is considered an individual shot. The downside of this is that each pellet will suffer reduced damage against armor. The upside is that with one blast, a powerful shotgun can both destroy a shield and the health of the target underneath.