Reverse dns for web/mail server with single ip

Solution 1:

If your mail server identifies itself as mail.domain.com, that should be what your PTR record has. It is perfectly fine, for your mail server to identify itself as frog.domain.com when sending and receiving mail. You have an A record for frog.domain.com, and that is the address which will pass rDNS validation. Given your pointer record, you should configure your mail server to identify itself as frog.domain.com.

You can leave your MX record pointing to mail.domain.com as mail servers shouldn't be verifying the name of the server they are sending to.

Most legitimate mail servers pass rDNS validation, as yours will when using frog.domain.com.

Consider setting up SPF records to protect your domains. Domains not sending mails should have a policy which prevents their use by mail servers or as sender domains. In your case I would allow mail from frog.domain.com and domain.com, and not allow mail from www.domain.com. If you are using mail.domain.com as convenience to access your mail server to send and receive mail, then I would consider not allowing it to send mail. (Mail will originate from frog.domain.com.

You could change your PTR record to mail.domain.com, which is a more traditional server name. In that case, I configure SPF to prevent mail from frog.domain.com, and allow it from mail.domain.com.

Once you get this working to your satisfaction, look into adding DKIM and DMARC functionality.