How to use plain text smileys (disable automatic emoji substitution) in Messages.app?
Go to the Edit menu, submenu Substitutions, then uncheck Emoji.
Run the following command (source) in Terminal:
defaults write com.apple.messageshelper.MessageController SOInputLineSettings -dict-add "automaticEmojiSubstitutionEnablediMessage" -bool false
Then restart Messages.app.