Why is exFAT so slow on Mac?
exFAT is a file system optimized for flash drives, meaning storage devices without moving parts where seek times are constant. It sounds like you're using it for a mechanical hard drive – which is what's causing the poor performance that you're experiencing.
You should reformat the drive to HFS+ or FAT32 and only use exFAT for solid-state drives.
ref: http://en.wikipedia.org/wiki/ExFAT