Parse undelivered mail headers (bounced mail)
What is the best way to parse the headers of bounced (undeliverable) email that is sent back to my server and determine whether it is a soft or hard bounce?
I only send opt-in emails to my users, but occasionally some email addresses go stale. When an email bounces back to my server, I would like to find why it bounced (soft/hard). Then I can deal with it appropriately in my database and/or flag the user to update their email when the log in next.
I'm using Ubuntu and Postfix. I have successfully implemented VERP with aliases and virtual aliases. So bounced emails have a return-path of [email protected], and I can pipe them to a script.
Now that I have VERP setup, I know who the original email was sent to, but I need to parse the returned mail headers to figure out if it is a soft bounce or hard bounce.
What is the best way to handle this? As I understand it, not all mail servers play by the same rules, and headers can have a variety of formats. Is there some open source project that keeps track of these types of things? Something simple I can implement that will categorize the majority of bounces properly?
I'm trying to protect the reputation of my mail server, so any help is much appreciated!
As RFC3463 explains, status codes beginning with 5 are used for permanent failures and 4 for persistent transient failures. Instead of trying to parse several messages with different formats you could rely on server logs and try something like this:
grep " dsn=5." /var/log/mail.log | grep -o -P " to=<(.+?)>" | sort | uniq -c
This will find permanent errors from mail.log (Postfix format) and give the addresses and amount of bounces on every address. You can also use " dsn=4." to get addresses with temporary errors.
Generally there are two types of bounces
- The bounces caused by directrejection of remote mail server when your postfix deliver the email.
- The bounces caused by remote server (next-hop server after your postfix) fails to deliver the message to final recipients.
The first case was already covered by excellent answer by Esa Jokinen above. Your best bet is parsing maillog.
The second case was special case of bounces. The example scenario:
- You send email with recipient [email protected] to mail.example.com server.
- In mail.example.com, [email protected] was aliased to [email protected] and must be forwarded to mail.example.net.
- Someday mail.example.net reject the your message so mail.example.com must send bounces to your server.
- Unfortunately maillog in your server will have "dsn=2" because mail.example.com already accepted the message but failed to forward it to mail.example.net.
Here the example of second type bounces email. There is forwarding rule Yahoo mail server [email protected] -> [email protected]. Unfortunately mail server of example.net reject the message :(
From MAILER-DAEMON Thu Mar 5 05:07:26 2015
Return-Path: <>
X-Original-To: [email protected]
Delivered-To: [email protected]
Received: from nm21-vm7.bullet.mail.gq1.yahoo.com (nm21-vm7.bullet.mail.gq1.yahoo.com [98.136.217.54])
(using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits))
(No client certificate requested)
by mx.example.org (Postfix) with ESMTPS id D6365565FC
for <[email protected]>; Thu, 5 Mar 2015 05:07:25 +0700 (WIT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=bounce; t=1425506842; bh=zk/tWZNl6c36dmlPDmakM9ekK8cHVJANXMmSdsbkcWc=; h=From:To:Date:Subject:From:Subject; b=Im95h1qTg6qN3yUI7vF1fXtJ0SbUnzv8rUPwLbpNwxGPN2p8wfosXJzQgJ3nzr4L4ZQ50P2d9E9U4jEUNtnyi7nlFd5kKbtiVuda4H56h1PFnt+7wSpgHcd5Irs/lLODumb6ZZSEpCOWttcB9+JLaDfEUUPjGcbR+xww4XeH5Eo=
From: [email protected]
To: [email protected]
Date: Wed, 04 Mar 2015 22:07:22 -0000
Subject: Failure Notice
X-Yahoo-Newman-Property: bmbounce
Sorry, we were unable to deliver your message to the following address.
<[email protected]>:
Remote host said:
550 5.1.1 User unknown
[RCPT_TO]
For this case, your only method is parsing the bounces message. Unfortunately there are no standard bounces format, so you must parsing the body and determine the rejection caused.
The feature checklist of your postfix bounce parsing:
- Check if VERP address was valid. You don't want to parse invalid message.
- Parse the body, determine if they are soft or hard rejection.
For the second feature, you can google some common rejection message. The example is this bounce-regex-list.xml by Jakub Liska.
Esa Jokinen made a good point in the comment below about these two bounce types. If your goal is keep the server reputation, then dealing the first bounce type should be enough. The second bounce was about cleaning your lists. So dead email should be erased thus freeing some resources in your server.
Some mailing list managers such as PHPlist and Mailman also deal with this bounce problem with parsing the email body as they have no resources to parsing the maillog.