How to find out that SMTP server is in open relay?

As juwi has in his answer, just make a network connection to tcp/25 of the server in question. You can use telnet for that, or netcat.

Here's an example of an SMTP transaction:

http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol#SMTP_transport_example

A relay is closed if it fails after you try to submit a RCPT that the server is not configured to accept. A relay is open if it accepts whatever domain you specify in an RCPT line.

I don't know what the nmap scan is doing to determine whether you have an open relay. One thing to keep in mind is that your SMTP server may be configured to accept all traffic from the IP you're testing from, so, yes, it's open, but not to the rest of the world, so that may be OK. You should run the test from a remote box to be sure.

Microsoft procedure for detecting open relays - manual telnet commands, too much to post up here. http://support.microsoft.com/kb/324958

Automated test http://www.mailradar.com/openrelay/ However when I've used this with Microsoft IIS6, the tests with the email address surrounded with speechmarks (e.g. method 7 RCPT TO: <"[email protected]">) appear to fail but don't actually get sent out. If you find yourself in this situation you may wish to do further investigating of settings and actually verifying these aren't ending up somewhere you don't want them to. 178.209.36.55 however sails through all these tests.

Once you are happy you are not running an open relay, the next step is to ensure you aren't backscattering. Backscatter is when a third party sends an email to an undeliverable address with a forged sender header so a NDR is sent to the apparent sender (forged).

This often comes about as a result of a server configured to relay mail for a whole domain without a list of valid users so it will accept the email initially, then will not be able to relay it to a server with a list of valid users.

Simple test - I'm going to call your external personal email address [email protected] and your domain is domain.com

ehlo hotmail.com
mail from [email protected]
rcpt to:[email protected]
data
subject:subject
text
.

Then check the [email protected] account (probably in the junk folder) for a NDR from something like [email protected].

This is code I had in my script-folder. Can't remember where I got it. Credits to original author:

#!/usr/bin/perl -w
#Script to check for Open Relay EMAIL Servers on port 25
#Author: Felipe Ferreira  fel.h2o(at)gmail Date: 18/02/2009
#ref. http://www.perlmonks.org/index.pl?node_id=718552
#TODO: 
#1. Should get from a list of IPs, should report any Open servers to a .txt file
#smtp->code() and message() are the responses of the server!

use Net::SMTP;
my $host;
my $filename = "servers.txt";

#open .txt file with all servers names or IPs
#open($filehandle, "<" . $filename);
open FILE, "<servers.txt" or die $!;
while (my $host = <FILE>) {
    print "Testing: $host";
    my $smtp = Net::SMTP->new($host, 
        Hello => "admin", 
        Timeout => 3,
        Debug => 0);
    if (!$smtp) {       print STDOUT "No connection to $host \n";

    }
    else {
    $smtp->mail('[email protected]');
    $smtp->to('[email protected]')
       or 
       print "SERVIDOR OK: $host RCPT TO: ", $smtp->code(), " ", $smtp->message();
    if ($smtp->code() eq "250" ) {
       print "SERVIDOR CRITICAL: $host Tiene problema de OPEN RELAY!!!";
    }
#   $smtp->quit;    
    }
} #Next host
close FILE;

You can just log on manually using telnet on Port 25.

Then you say HELO and try MAIL FROM: <mailadress> next up RCPT TO: <mailaddress> At that point it should tell you that it rejected the address because it denied access if it is not an open relay. If it is open it will do what you just told it - send out an email.