how can you say 'the person I'm talking with' with one word?
Solution 1:
The word is interlocutor. "You have to hear/listen to your interlocutor."
But it can refer to anyone taking part in a conversation.
Solution 2:
"collocutor," meaning "The person with/to whom one is speaking."
collocutor noun col·loc·u·tor \kəˈläkyətə(r), kōˈ- ˈkäləˌkyütə(r)\ Definition of collocutor
: a person to or with whom one speaks
© 2015 Merriam-Webster, Incorporated