550 Invalid HELO/EHLO must contain a FQDN or IPv6

Solution 1:

If the FQDN of the server itself is something to the effect of "server.domain.local" (meaning it's only valid on your internal network), and the receiving server performs a rDNS or SPF check against the incoming email, it may continue to block the email. The way to overcome this is to set the outgoing FQDN on your server in the properties of the SMTP virtual server to something that is going to be valid externally, such as "mail.domain.com". Make sure to set up a rDNS and SPF record as well as an A record for this server in your external DNS forward and lookup zones (you will need to get your ISP to set up the rDNS record for you).

Solution 2:

You need to define a Fully Qualified Domain Name for your Server.

I guess here:

System - Computer Name - Change - More - DNS Suffix.

reference

And since I can't find anything in windows 2008 about the SMTP server, I will give you this windows 2003 document on the DNS records required to setup a FQDN.