How do you detect a spambot on your network?

Solution 1:

First, you need to stop the spam.
a- set your firewall to not allow outbound SMTP/POP except from the email server.
b- set your mail server to not allow outbound relay.

Then, you need to find the problem machine(s).
1- Look at the firewall logs to see which machine(s) are actually trying to do outbound mail and getting blocked. Those machines are infected.
2- Make sure each machine has current A/V, and do a thorough scan on each machine.
3- You may want to implement the Windows Firewall on each machine.
4- If still not found you will need to use a sniffer.

Note: I don't think that DNS and email on the same server is an issue.

Solution 2:

The problem could be that that your exchange server is allowing RELAY. Make sure that setting is turned off or set only the IPs which are allowed to relay via that server. Your network design should only allow the exchange server to send traffic out of your network via port 25.

Most spambots use port 25. Once you have setup like that, then if any other machine try to send via port 25 it will show up in the firewall logs.

Good Luck!

Solution 3:

1. You should sniff your network traffic. There are many nice tools available, all the way from plain packet dumps similar to tcpdump all the way to fancy GUI visualization apps. Typically you will need to either: a) plug into a special port on your switch, b) configure another port to see all traffic, or c) do the sniffing from your firewall/router. First just focus on finding SMTP traffic to the outside world from any machine that's not the Exchange sever. Later you should examine all traffic to see if anything else is going on: IRC from someone's machine who doesn't even know what IRC is, for instance.
2. Write a plain language statement of what traffic should be allowed out of your network, and implement outbound rules at your firewall/router, with logging. You will be surprised how well this works. And it will also mean that you will know when something bad is happening before you hear it from an outside party!

Solution 4:

If you have a firewall, a simple solution is to block all outbound port 25 traffic except for your Exchange server. Individual machines are likely trying to send spam on their own. Once you've put the block in place, check the firewall logs to see which IP is trying, and failing, to hit port 25 outbound.