Hearing equivalent for "speaker"

I think the word you are after is listener in everyday terminology.

The speaker speaks to an audience. In the audience, you may find some very attentive listeners. Everybody in the room will hear the talk, so it might be argued that hearer is a possibility.

It depends on the interest of the audience member, though.

Hearing is something you cannot control. Your ears will pick up the sound and send it to your brain. So everyone who hears the sounds will be a hearer.

Listening, though, is something that you can control. You can decide to tune out and not pay attention, or you can decide to lap up every sound or word. If you are interested and follow the talk, you will probably describe yourself as a listener.

This also manifests in the set term avid listener. I'm not sure if such a construct exists for hearing.

Edited Answer, see below.

Every time I've studied conversations patterns and rules of speaking, the other person has always been called either hearer or addressee. Here I'll quote an example from my notes:

"Speakers and hearers constantly adjust their internal registry of deictics to keep up with the conversation."

or

"Directives = Speech acts in which the words are aimed at making the hearer do something;"

I'm not really sure about listener, but honestly I can't recall using it.

EDIT:

Since I've been downvoted, I'll link a further reference which proves me right. The book, Speech acts: an essay in the philosophy of language, is written by John R. Searle, a rather known name in the Linguistics field and to whoever studied Linguistics in an academic setting.

Speech Acts are utterances that don't simply convey information, but can also "perform actions" such as "I hereby declare you company and wife." (This one, for example, is a Declaration according to Searle's classification of Speech Acts.

If you search words in that book, there are 34 entries for "hearer" and 0 entries for "listener".