How to use sniffer to troubleshoot SMTP traffic?
Solution 1:
Step 1 is to start grabbing packets in front of the spam appliance, where it theoretically receives them after passing the firewall. You want to set your capture filter to "port 25". Wireshark's dissector is good enough to do most of the analysis its self. Right-click on a packet, select "View TCP session" to get a complete transcript of a SMTP session.
If you see no packets, then your firewall really isn't passing SMTP traffic. You may want to send a few messages yourself to prove your capture filter is working. Repeat the test in front of your firewall if possible to prove you're getting any at all.
If all you see are SYN packets to the appliance with no response, then you've found your culprit. The appliance has gone on vacation.
If you see full SMTP conversations that look normal, repeat the test on the outbound port of your appliance.
If the outbound port shows no SMTP traffic, then you know the problem is inside the appliance somewhere. This should help convince vendor support that it really is their problem.
If you see complete traffic outbound, then the problem is in your mail system somewhere.
Solution 2:
How about the wireshark wiki on SMTP? And here is a blog post for beginners.
Solution 3:
Couple other options...
1) You could always try manual telnet tests to port 25 on the outside and go through the SMTP conversation manually. Details here: http://support.microsoft.com/kb/153119
2) I'm not familiar with barracuda boxes but on most linux email gateways, you can just run tcpdump (something like tcpdump port 25 -s 0 -w foo.pcap).
-M