Handoff stopped working for messages

As a starting point, my suggestion would be to sign out of your account on the iMac, but not from the iCloud preference pane. Instead, sign out of Messages as follows:

1. Open the Messages app on your iMac
2. Go to Messages > Preferences...
3. Select the Accounts tab
4. Select your Apple ID/iCloud account
5. Click on the Sign Out button
6. At the next prompt, click on the blue Sign Out button
7. Once you're signed out, Quit Messages
8. Wait a couple of minutes repeat Steps 1 to 3
9. Go through the process of re-adding your account

Once you've done this you should get a notification on your iPhone saying that your account is now being used on an iMac. Once this happens, test to see if everything is working okay.