Is there a way to tell if an alien is aggressive or friendly on Starbound?

Unfortunately you can't tell if an alien is aggressive just by looking at them, but there is an audio cue you can use: hostile aliens will face you and roar/growl/squeak/whatever-their-sound-is before they start their attack. While passive aliens also make noises, your presence isn't the trigger and more often than not they will remain silent as you approach. If you aren't sure about an alien's disposition then cautiously move toward it and listen to its reaction.

The only way to tell whether a creature is hostile or not is to walk near them. On a new planet when you discover a creature you don't know yet, walk up to them slowly. If they don't attack you by the time you're almost close enough to smack them with your sword, they're friendly.