Have been to/have gone to
Yes, that's correct. The verb GO has two past participles. Gone implies that the travellers are still in the place they went to. Been implies that they returned from their trip.
Because the first sentence stipulates that the speaker is now back, been is required here.