Is it safe to remove ~/Library/Mail folders

By safe, I assume you mean "will not delete message on the server?" That's correct, you can go ahead and delete those - think of the as a "cache" of your emails. As long as you aren't deleting and emptying the trash from within whatever mail client you are using, they will remain on the server.

Do consider a backup before deleting ANYTHING from ~/Library (or any other Libraries on your Mac).

In this specific case, I would explicitly rebuild the mailbox indexes just in case the system will clean them up once you're done deleting things...

By safe, I assumed you meant "will this screw up my computer?"

The answer is no, it will not mess up your computer... your computer will be just fine. Mail.app stores it's data in ~/Library/Mail/* Removing this folder will not mess up the way the computer behaves, and if you launch Mail.app in the future, it will simply recreate these folders as needed.

You may however want to back up the files in case you realize that there was something in there that you needed...