Solution 1:

It's something you wouldn't say to your grandmother, perhaps, but it is a very common euphemism people use to avoid saying something stronger. It's used in everything from pop songs to commentaries by TV talk show hosts.

Seriously, in most walks of society screwed is considered merely informal these days, not rude.

Solution 2:

Robusto's response addresses the rudeness aspect of your question.

With regard to the second part, whether I'm screwed is "used when someone tries to say they made a mistake": I think you're confusing I'm screwed (which as the comments tell you means approximately "Aw, jeez, I'm in trouble") with I screwed up, which does mean "I've made a mistake".

Another version of screwed up, made famous in The Right Stuff, is screwed the pooch; this is somewhat intensive, but the conventional intensifier is royal: "I screwed up royally."

Both expressions (and the "stronger" version to which Robusto alludes) may be used in second or third person as well as first, in either the singular or plural, and in any tense, aspect or mood.

Screw up is also frequently used adjectivally ("That's a really screwed-up piece of design") and nominally ("I'm getting tired of Carol's screw-ups"). The nominal version of screw in this sense is more often the gerund screwing than the infinitive screw (which tends to be reserved for the sexual sense); I cannot recall encountering an adjectival use.

Solution 3:

I do believe that when you are saying "I'm screwed" or "He really screwed me" you are using a euphemism which has its roots in the sex act, which is sometimes referred to as "screwing". In this case, it is a less-vulgar version of saying "I'm fucked" or "He really fucked me". It is using the sex act as a metaphor for domination, which apparently is NEVER acceptable in polite company. Wiktionary says:

The modern sense of screwed originates in the mid-1600's with a sense of "to screw" as a means of "exerting pressure or coercion", probably in reference to instruments of torture (e.g. thumbscrews).[1] It quickly gained a wider general sense of "in a bind; in unfortunate inescapable circumstances" When the verb "to screw" gained a sexual connotation in the early 1700's,[2] it joined the long-lasting association of sexual imagery as a metaphor for domination, leading to screwed gaining synonyms like fucked and shagged. On a more general note, this is a prime example of the frequent tendency for verb participles to evolve into adjectives.

As for the term "screw the pooch", here's a quote from Jacob May, the DJ who first coined the term back in the '50s recounting how that came about:

John Rawlings was one of two roommates who were architecture students. In the spring of 1950 it was time for his project to complete the semester. He procrastinated. Apparently all architecture students do. He was going to be late even starting his charrette. So to be helpful I said the following: JACK: You're late, John, you're fouling up. You are fucking the dog. JOHN: Really, you are so vulgar and coarse, I just don't want to hear it. JACK: You're still late. Is this better? You are screwing the pooch. JOHN: (shrill laughter)

So FtD and its nicer version, "screw the pooch" mean to screw up EXTREMELY BADLY, somewhat how FUBAR means something ""f'd" up beyond all reason" Worse than we can describe in words. So bad that it would be as big a screw up as having intercourse with a dog. Any reference to the sex act at ALL traditionally is considered vulgarity, ESPECIALLY with animals, I would think.

To answer you, saying "I'm screwed" technically IS vulgar but has become more accepted in current society. However, you would not use the term "I'm screwed" to say that you made a mistake; rather you would say that you "screwed-up". You might actually be "screwed" because of your "screw-up", meaning you might find yourself "beset with unfortunate circumstances that seem difficult or impossible to overcome or in imminent danger" due to whatever you did that left something "Broken, damaged, inoperative or having only partial functioning". Hope this helps