How do I know what to say to enemies?

It seems fairly random what enemies like to hear and don't like to hear. I very frequently fail to negotiate with enemies and either just get some item or have them just attack me again or even call for backup. For instance, I had to hold up Apsaras 4-5 times before I successfully negotiated with her and Onmoraki called for backup after attacking me when I failed negotiations twice in a row, but I was able to negotiate with Slime on my first try.

Are there any sorts of signs I should be looking for in their dialogue or the options I'm given when I'm holding enemies up and attempting to get them to lend me their power? How do I know which dialogue choice is the one that they will like to hear?

The negotiations, like most things in games, follow a pattern. This Kotaku guide explains these patterns from the game's own tutorial :

• Upbeat -> Joking, not serious or vague
• Timid -> Kind, not vague or joking
• Irritable -> Serious, not vague or kind
• Gloomy -> Vague, not serious or joking

Personalities can be checked by selecting the shadow on the "Hold Up" screen or using the analysis option (L2 menu).

Negotiations are semi-random, but there seems to be a set of questions for each personality type (it's shown in the upper left when holding up a shadow). So far it seems to be two questions and then the shadow decides what to do with you. You should begin each negotiation by carefully considering the aforementioned personality type - this gives you a hint into the general philosophy of the enemy, but it won't tell you the whole story.

You'll also need to pay attention to what they say and how they talk. Enemies that speak in a childish manner might be more receptive to certain response than enemies which speak like old people.

As you progress your vows with your party members, they will gain abilities which can assist you in negotiations as well.