How to tell if someone is likely to attack?

Unfortunately there's not a good way to tell this. In general, if you're using travel mode and a group stops you, they're going to be hostile (even if sometimes they seem like they're not).

Also, if they're at a campsite they're probably bandits. I've been to campsites where the bandits are totally neutral towards me, but if I have a companion from an entity they're enemies with a fight will start.

It's generally possible to have a conversation with intelligent hostiles before a combat starts, which makes it even harder to tell what's going on! You can try to ask questions about the area or the local ruler to identify which entity they're a part of (if you're at a campsite and they're telling you that they're in charge and they're right in all things, it means they're their own entity). However, this isn't very useful when you meet a traveller who tells you that they're on a secret mission, and to ask you about the area when they get home.

The yielding/combat escalation system can lead to some weird situations where your yielded enemies will only spit at you and call you a murderer, but they're happy to join you for adventure.

I personally find the most reliable way to tell if a group is hostile is to take good companions with me and be ready for a fight. As you discovered, companions are particularly good at detecting enemies, usually by starting a fight with them!