Mail Queue getting huge even though mails are sent but slow and sending without spamassassin does not work
I figured it out and that was due to slow DNS resolving problem of the server.
It took around even 5 minutes to send a single email to Gmail.
I fixed it by changing the location of the server to another place.