"alone with someone"

Solution 1:

Alone means in the Free Dictionary Link:

Exclusively; only: The burden of proof rests on the prosecution alone.

And it addes, alone emphasizes being apart from others but does not necessarily imply unhappiness.

If you are traveling only with your wife excluding your parents or your children, you and your wife can be considered as one group of travellers. I don't see any reason alone cannot be used there.

One of examples in Google Search shows the similar usage as above.

Solution 2:

"Alone with" implies a private or intimate occasion.

"Finally after the children were in bed, I could be alone with my wife".

"I was determined to get five minutes alone with my new manager".

You could say "I like to travel with only my wife for company".