Excessive Outbound DNS Traffic

I have a VPS system which I have had for 3 years on one host without issue. Recently, the host started sending an extreme amount of outbound DNS traffic to 31.193.132.138. Due to the way that Linode responded to this, I have recently left Linode and moved to 6sync. The server was completely rebuilt on 6sync with the exception of postfix mail configurations.

Currently, the daemons run are as follows:

sshd nginx postfix dovecot

php5-fpm (localhost only) spampd (localhost only) clamsmtpd (localhost only)

Given that the server was 100% rebuilt, I can't find any serious exploits against the above stated daemons, passwords have changed, ssh keys don't even exist on the rebuild yet, etc... it seems extremely unlikely that this is a compromise which is being used to DoS the address.

The provided IP is noted online as a known SPAM source. My initial assumption was that it was attempting to use my postfix server as a relay, and the bogus addresses it was providing were domains with that IP registered as their nameservers. I would imagine given my postfix configuration that DNS queries for things such as SPF information would come in with equal or greater amount than the number of attempted spam e-mails sent.

Both Linode and 6Sync have said that the outbound traffic is extremely disproportionate. The following is all the information I received from Linode regarding the outbound traffic:

21:28:28.647263 IP 97.107.134.33.32775 > 31.193.132.138.53: 28720 op8+% [b2&3=0x4134] [17267a] [30550q] [28773n] [14673au][|domain]
21:28:28.647264 IP 97.107.134.33 > 31.193.132.138: udp
21:28:28.647264 IP 97.107.134.33.32775 > 31.193.132.138.53: 28720 op8+% [b2&3=0x4134] [17267a] [30550q] [28773n] [14673au][|domain]
21:28:28.647265 IP 97.107.134.33 > 31.193.132.138: udp
21:28:28.647265 IP 97.107.134.33.32775 > 31.193.132.138.53: 28720 op8+% [b2&3=0x4134] [17267a] [30550q] [28773n] [14673au][|domain]
21:28:28.647266 IP 97.107.134.33 > 31.193.132.138: udp

6sync cannot confirm whether or not the recent spike in outbound traffic was to the same IP or over DNS, but I have presumed as such. For now my server is blocking the entire 31.0.0.0/8 subnet to help deter this while I figure it out.

Anyone have any idea what is going on?

not an answer, just some random thoughts:

• can you see this traffic when you just run tcpdump on your [virtual] network interface? if so - can you try to figure out if there's any daily/hourly pattern? you can create iptables rule to count the traffic and then allow munin plugin to collect the stats..
• can you try to determine which app is generating this traffic? i see two methods here:
  • brutal method is to wait till the traffic shows and start killing the applications one after another.
  • gentle approach - use iptables on the OUTPUT chain and owner match to log output packets going on the port 53 to your syslog. something like: iptables -I OUTPUT -p udp --dport 53 --match owner --uid-owner 33 -j LOG --log-prefix "uid 33" applied to all your used uids. check syslog to see which process generates the unwanted traffic.
• do you have local dns server [eg bind] running? if so:
  • sniff also on the loopback to see what app might be sending requests that cause unwanted traffic.
  • can external servers talk to your dns server? if so - maybe it's some sort of back-scatter attack where your server receives packets from spoofed addresses and responds bombarding a victim.
• are you 110% sure your php code was not altered? can it be that some of your scripts contain few malicious lines?

I don't (yet) know what the traffic is, but I can confirm that it is not conformant DNS traffic.

The 12 byte header has:

• a 2 byte ID field (28720 - 0x7030)
• a 2 byte flags fields (0x4134)
• 4 * 2 byte record counts [17267a] [30550q] [28773n] [14673au]

The flags on a normal recursive query should be 0x0100. The 4 counts should be (1, 0, 0, 0+).