When is it appropriate to say that a garment looks "flattering"

Recently, I've begun hearing the word "flattering" in apparently sincere compliments. In one scenario, a friend complimented someone on their new shirt by saying that it "looked very flattering." Now, I hang around a lot of engineers so social nuance isn't always in abundance but I've always regarded "flattering" as a word that denotes insincerity. Now if I wanted to coyly receive a compliment, I might say "Oh you flatter me;" that's cool. But when calling a piece of clothing "flattering" I always interpreted the statement as meaning that the garment made the person look good in spite of himself, as though it's hiding physical flaws or somehow compensating for an otherwise unfortunate appearance.

One non-negative context where I MIGHT use it is if I was helping a friend pick out a suit (or something) and he asked for some honest feedback because he was worried that the trousers looked bad on him. Under such a situation I might say "Don't worry, they really are very flattering." But as a casual compliment? At best it sounds tone-def and at worst, sounds like a catty mean-girls insult.

One scenario where I couldn't even imagine saying this is to someone of the opposite sex. Only if I'd grown tired of living, would I try to compliment a woman by saying that "her dress looks very flattering." That her dress "looks great?" yes; "Nice?" fine; but flattering? Yeesh, I wouldn't chance that without a cup and a mouth-guard.

What say stack exchange? Am I missing out on a charming compliment or adhering to prudent word-choice (and maybe even basic self-preservation)?

You are confusing flattery with flattering. From NOAD:

flattery (n.) excessive and insincere praise, esp. that given to further one's own interests

flattering (adj.) (of a person or their remarks) full of praise and compliments : the article began with some flattering words about us.
• pleasing; gratifying : [with infinitive ] it was flattering to have a pretty girl like Frances so obviously fond of him.
• (esp. of a garment or color) enhancing someone's appearance
• (of a picture or portrait) giving an unrealistically favorable impression of someone or something : that's a flattering picture of him.

Looking at those definitions, it's readily apparent that flattery has that connotation of insincerity that you mention; however, for the most part, in the case of flattering, the praise seems sincere. Even in the case of the flattering picture, you're still acknowledging that the depicted does in fact look good.

Remember the old adage: Flattery will get you nowhere – and for good reason. But a flattering compliment might get you somewhere. That said, you're not the first to mix these two words up. Keep that mouthguard handy when you use that adjective in a compliment, just in case.