Is there a single word that can refer to both listeners or readers?
One pair of terms that covers speakers/writers and listeners/readers is producers and receivers. See the extract below from the Wikipedia article on text linguistics. Text in this context refers to both written and spoken language.
Text producers often speculate on the receiver's attitude of acceptability and present texts that maximize the probability that the receivers will respond as desired by the producers.
An alternative pair listed under the entry on Communication in the Longman Dictionary of Language Teaching and Applied Linguistics (p97) is sender / receiver:
In an act of communication there is usually at least one speaker or sender, a message which is transmitted, and a person or persons for whom this message is intended (the receiver).