Does the order matter when talking to the ghost travelers?
No, it doesn't. The speeches will be the same regardless of how many of the other character you've talked to or what physical order you find them in. Each time you find a new traveler you'll see the next dialog for that traveler. All travelers can be found in any order.