What's the difference in usage between "annoyed at", "annoyed with", and "annoyed by"?

I am annoyed by a specific behavior.

I am annoyed with the person who's doing it.

I am annoyed at a requirement.

There are several ways to use annoyed as an adjective and you can also use the verb to annoy. I think it helps to consider these forms separately.

First the adjective:

I am annoyed by elephants.
Elephants in general annoy me.

I am annoyed with my sister.
My sister has done something to annoy me.

I am annoyed at my sister.
(identical meaning to with)

And the verb:

She is annoying me by drumming.
by + gerund (action that annoys)

She is annoying me with her drum.
with + a noun (thing that annoys)

At is not used with the verb. In the BNC there are zero occurrences of annoyed (v.) at and 67 occurrences of annoyed (adj.) at.

In the second sentence there is the potential for two different meanings, when the words at and with are interchanged:

1. He was annoyed at the children. – No ambiguity.

2. He was annoyed with the children. – Ambiguous, because it is now unclear as to whether the children were also annoyed.

For those who have English as a second language this could be confusing, because the word with is more often used to join nouns.

As one responder has mentioned, "annoyed by" can be used for statements of general truth ("I am annoyed by loud music"), whereas "annoyed with" and "annoyed at" are the result of the target's past behavior: "I am annoyed with/at my sister because of something that she did."

Try this with other similar verbs and adjectives: frustrated, irritated, angry, happy. Sometimes "at" works, but "with" always does.