'I came' vs. 'I went'

I was recently speaking to my friend over the phone. She asked about the whereabouts of my other friend, to which I replied "I don't know. I went home already." (Keep in mind that while saying this, I was already at home. After hearing this, my mother proceeded to comment on my incorrect grammar in the above dialogue. I was just wondering if 'I came back home' should be used instead of 'I went back home', and if the latter is grammatically incorrect in my situation. Thank you for answering.

In general you "come here" and "go there". As "went" is the past participle of "to go" I can see your mother's point. However, in my idiolect (I'm a native British English speaker) what you said sounds completely normal to me.

I think in "go home" from work (/school etc.), the focus is on leaving work rather than arriving at home. Hence it is "go {somewhere} from work" rather than "come here {from work}".