Abnormal sendmail activity using up the server memory on my Ubuntu 12.04 server

spam sendmail low-memory ubuntu-12.04 wordpress

Environment

  • Rackspace
  • Ubuntu 12.04
  • Wordpress
  • MySql

The issue

I have been experiencing quite serious out-memory-issues in the last couple of days.

While I resolved one possible cause the issue I still get a very suspicious activity of sendmail.

Any recommendations on how to tackle this issue? I think that must be some malware, but I no experience on resolving this kind of attacks.

htop

  1  [|||||||||||||||||||||||||                                                          27.0%]     Tasks: 101, 50 thr; 1 running
  2  [|||||||||||||||||||||||||||||||||||||||||                                          45.7%]     Load average: 12.96 12.55 11.95 
  Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||1183/1995MB]     Uptime: 09:53:28
  Swp[||||                                                                           93/2047MB]

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
19704 root       20   0  120M 25328  2896 S  2.0  1.2  0:46.16 sendmail: MTA: ./s6HH4rLv009027 gmail.co.: user open
 3298 root       20   0   99M  5612  1684 S  2.0  0.3  2:46.31 sendmail: MTA: s6OABpf4003298 localhost [127.0.0.1]: DATA
 3301 root       20   0   99M  5544  1684 S  2.0  0.3  2:40.89 sendmail: MTA: s6OAGAAh003301 localhost [127.0.0.1]: DATA
19510 root       20   0 26488  2568  1212 R  2.0  0.1  0:23.73 htop
  771 syslog     20   0  244M  3892   516 S  1.0  0.2  2:22.43 rsyslogd -c5
 1226 smmsp      20   0  133M 56328  1396 S  0.0  2.8  1:56.85 sendmail: MSP: ./s6K1OdvJ030780 [127.0.0.1]: client DATA status
32488 root       20   0  102M  7168  2748 S  0.0  0.4  0:00.02 sendmail: MTA: ./s6OAcr6I032488 aspmx.l.google.com.: client EHLO
31723 www-data   39  19  448M 72676 47276 S  0.0  3.6  0:01.14 /usr/sbin/apache2 -k start
29624 root       20   0  120M 25916  2884 S  0.0  1.3  0:29.65 sendmail: MTA: ./s6NHPdHs002287 todito.com.: user open
  898 mysql      20   0 1315M  105M  3296 S  0.0  5.3 23:25.23 /usr/sbin/mysqld
30966 root       20   0  101M  5092   460 D  0.0  0.2  0:01.52 sendmail: MTA: running queue: /var/spool/mqueue
 5013 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:25.58 /usr/sbin/mysqld
25504 root       20   0  120M 25904  2900 S  0.0  1.3  0:24.57 sendmail: MTA: ./s6JHcEdS028616 hotamil.com.: user open
 1033 root       20   0  630M  6228  2356 S  0.0  0.3  1:17.85 /usr/local/bin/driveclient --daemon
 1062 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.50 /usr/local/bin/driveclient --daemon
 1082 newrelic   20   0  107M  1576  1072 S  0.0  0.1  0:46.81 /usr/sbin/nrsysmond -c /etc/newrelic/nrsysmond.cfg -p /var/run/nrsysmond.pid
 1089 newrelic   20   0  107M  1576  1072 S  0.0  0.1  0:46.80 /usr/sbin/nrsysmond -c /etc/newrelic/nrsysmond.cfg -p /var/run/nrsysmond.pid
  822 syslog     20   0  244M  3892   516 S  0.0  0.2  1:35.12 rsyslogd -c5
 1061 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.80 /usr/local/bin/driveclient --daemon
 8532 root       20   0  105M  9444   460 D  0.0  0.5  0:06.40 sendmail: MTA: running queue: /var/spool/mqueue
31711 www-data   39  19  445M 75316 52764 S  0.0  3.7  0:01.50 /usr/sbin/apache2 -k start
27927 root       20   0  120M 25904  2900 S  0.0  1.3  0:32.35 sendmail: MTA: ./s6NKLEhE005721 yahoo.co.: user open
13821 mysql      20   0 1315M  105M  3296 S  0.0  5.3  2:25.39 /usr/sbin/mysqld
31924 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:49.12 /usr/sbin/mysqld
31713 www-data   39  19  446M 68484 45496 S  0.0  3.4  0:00.79 /usr/sbin/apache2 -k start
 4195 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:29.08 /usr/sbin/mysqld
 9799 mysql      20   0 1315M  105M  3296 S  0.0  5.3  2:29.95 /usr/sbin/mysqld
 2664 smmsp      20   0  133M 56424  1476 D  0.0  2.8  1:52.68 sendmail: MSP: ./s6K3MC7s027126 [127.0.0.1]: client DATA status
  853 syslog     20   0  244M  3892   516 S  0.0  0.2  0:47.23 rsyslogd -c5
31714 www-data   39  19  446M 68404 45420 S  0.0  3.3  0:00.73 /usr/sbin/apache2 -k start
31903 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:47.96 /usr/sbin/mysqld
 1063 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.40 /usr/local/bin/driveclient --daemon
31600 www-data   39  19  448M 71340 46228 S  0.0  3.5  0:00.92 /usr/sbin/apache2 -k start
 4308 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:28.28 /usr/sbin/mysqld
 1064 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.41 /usr/local/bin/driveclient --daemon
31727 www-data   39  19  447M 70324 45756 S  0.0  3.4  0:00.84 /usr/sbin/apache2 -k start
31725 www-data   39  19  447M 70340 45756 S  0.0  3.4  0:00.86 /usr/sbin/apache2 -k start
31724 www-data   39  19  447M 70548 45932 S  0.0  3.5  0:00.84 /usr/sbin/apache2 -k start
 1715 mysql      20   0 1315M  105M  3296 S  0.0  5.3  3:05.00 /usr/sbin/mysqld
23774 root       39  19  425M  6636  4676 S  0.0  0.3  0:06.00 /usr/sbin/apache2 -k start
 1065 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.35 /usr/local/bin/driveclient --daemon
 1060 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.43 /usr/local/bin/driveclient --daemon
F1Help  F2Setup F3SearchF4FilterF5Tre

huge /var/mail

root@web:/var/mail# ls -alh
total 1.2G
drwxrwsrwt  2 root     mail 4.0K Jul 24 10:51 .
drwxr-xr-x 15 root     root 4.0K Jul 24 00:45 ..
-rw-rw----  1 munin    mail  83K Jul 19 18:48 munin
-rw-------  1 root     mail 1.1G Jul 24 10:51 root
-rw-rw----  1 www-data mail  98M Jul 23 22:34 www-data

My root mail account is continuously sending emails

NOTE: I have replace my domain by filtered.com

Return-Path: <MAILER-DAEMON>
Received: from localhost (localhost)
    by web.filtered.com (8.14.4/8.14.4/Debian-2ubuntu2) id s6OAqpuv010033;
    Thu, 24 Jul 2014 10:52:51 GMT
Date: Thu, 24 Jul 2014 10:52:51 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON>
Message-Id: <[email protected]>
To: <[email protected]>
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
    boundary="s6OAqpuv010033.1406199171/web.filtered.com"
Subject: Returned mail: see transcript for details
Auto-Submitted: auto-generated (failure)

--s6OAqpuw010033.1406199172/web.filtered.com--

From MAILER-DAEMON  Thu Jul 24 10:52:53 2014
Return-Path: <MAILER-DAEMON>
Received: from localhost (localhost)
    by web.filtered.com (8.14.4/8.14.4/Debian-2ubuntu2) id s6OAqq6J010047;
    Thu, 24 Jul 2014 10:52:53 GMT
Date: Thu, 24 Jul 2014 10:52:53 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON>
Message-Id: <[email protected]>
To: postmaster
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
    boundary="s6OAqq6J010047.1406199173/web.filtered.com"
Subject: Postmaster notify: see transcript for details
Auto-Submitted: auto-generated (postmaster-notification)

This is a MIME-encapsulated message

--s6OAqq6J010047.1406199173/web.filtered.com

The original message was received at Thu, 24 Jul 2014 10:52:52 GMT
from localhost
with id s6OAqq6I010047

   ----- The following addresses had permanent fatal errors -----
<[email protected]>
    (reason: 550-5.1.1 The email account that you tried to reach does not exist. Please try)

   ----- Transcript of session follows -----
... while talking to aspmx.l.google.com.:
>>> RCPT To:<[email protected]>
<<< 550-5.1.1 The email account that you tried to reach does not exist. Please try
<<< 550-5.1.1 double-checking the recipient's email address for typos or
<<< 550-5.1.1 unnecessary spaces. Learn more at
<<< 550 5.1.1 http://support.google.com/mail/bin/answer.py?answer=6596 sq8si14059110obc.83 - gsmtp
550 5.1.1 <[email protected]>... User unknown
>>> DATA
<<< 503 5.5.1 RCPT first. sq8si14059110obc.83 - gsmtp

--s6OAqq6J010047.1406199173/web.filtered.com
Content-Type: message/delivery-status

Reporting-MTA: dns; web.filtered.com
Received-From-MTA: DNS; localhost
Arrival-Date: Thu, 24 Jul 2014 10:52:52 GMT

Final-Recipient: RFC822; [email protected]
Action: failed
Status: 5.1.1
Remote-MTA: DNS; aspmx.l.google.com
Diagnostic-Code: SMTP; 550-5.1.1 The email account that you tried to reach does not exist. Please try
Last-Attempt-Date: Thu, 24 Jul 2014 10:52:53 GMT

--s6OAqq6J010047.1406199173/web.filtered.com
Content-Type: text/rfc822-headers

Return-Path: <MAILER-DAEMON>
Received: from localhost (localhost)
    by web.filtered.com (8.14.4/8.14.4/Debian-2ubuntu2) id s6OAqq6I010047;
    Thu, 24 Jul 2014 10:52:52 GMT
Date: Thu, 24 Jul 2014 10:52:52 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON>
Message-Id: <[email protected]>
To: <[email protected]>
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
    boundary="s6OAqq6I010047.1406199172/web.filtered.com"
Subject: Returned mail: see transcript for details
Auto-Submitted: auto-generated (failure)

--s6OAqq6J010047.1406199173/web.filtered.com--

ps -ef | grep sendmail

root@web:/var/mail# ps -ef | grep sendmail
smmsp     1226     1  0 00:45 ?        00:02:04 sendmail: MSP: ./s6KKDDVU014035 [127.0.0.1]: client DATA status
smmsp     2644  2641  0 01:00 ?        00:00:00 /bin/sh -c test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp
smmsp     2647  2644  0 01:00 ?        00:00:00 /bin/sh /usr/share/sendmail/sendmail cron-msp
smmsp     2664  2647  0 01:00 ?        00:01:58 sendmail: MSP: [127.0.0.1]: idle              
root      3298     1  1 07:57 ?        00:03:16 sendmail: MTA: s6OB1dam003298 localhost [127.0.0.1]: DATA
root      3301     1  1 07:57 ?        00:03:05 sendmail: MTA: server localhost [127.0.0.1] cmd read
root     19675     1  0 11:20 ?        00:00:00 sendmail: MTA: ./s6OBKJuv019675 aspmx.l.google.com.: client DATA 354
root     19689     1  0 11:20 ?        00:00:00 sendmail: MTA: ./s6OBKLuv019689 aspmx.l.google.com.: client DATA 354
root     19800     1  0 11:20 ?        00:00:00 sendmail: MTA: ./s6OBKbuv019800 aspmx.l.google.com.: client DATA 354
root     20178     1  0 11:21 ?        00:00:00 sendmail: MTA: ./s6OBLSuv020178 aspmx.l.google.com.: client DATA 354
root     20270     1  0 11:21 ?        00:00:00 sendmail: MTA: ./s6OBLZuv020270 aspmx.l.google.com.: client DATA 354
root     20537     1  0 11:21 ?        00:00:00 sendmail: MTA: ./s6OBM0uv020537 aspmx.l.google.com.: client DATA 354
root     20646     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBM5uv020646 aspmx.l.google.com.: client DATA 354
root     21006     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMZ6I021006 aspmx.l.google.com.: client DATA 354
root     21015     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMZ6I021015 aspmx.l.google.com.: client DATA 354
root     21027     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMauv021027 aspmx.l.google.com.: client DATA 354
root     21036     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMb6I021036 aspmx.l.google.com.: client DATA 354
root     21063     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMeuv021063 aspmx.l.google.com.: client DATA 354
root     21065     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021065 aspmx.l.google.com.: client DATA 354
root     21086     1  1 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021086 aspmx.l.google.com.: client DATA 354
root     21094     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021094 aspmx.l.google.com.: client DATA 354
root     21098     1  2 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021098 aspmx.l.google.com.: client DATA 354
root     21103     1  1 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021103 aspmx.l.google.com.: client DATA 354
root     21105     1  1 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMguv021105 aspmx.l.google.com.: client DATA 354
root     21108     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OB1dag003298 mx-eu.mail.am0.yahoodns.net.: client MAIL
root     21111     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021111 aspmx.l.google.com.: client RCPT
root     21113     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OAsOi1003301 mx-ha03.web.de.: client greeting
root     21117     1  1 11:22 ?        00:00:00 sendmail: MTA: ./s6OAsOi3003301 gmail-smtp-in.l.google.com.: client DATA status
root     21123     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OAsOi5003301 gmail-smtp-in.l.google.com.: client EHLO
root     21127 18604  0 11:22 pts/0    00:00:00 grep --color=auto sendmail

Sendmail status

root@web:/var/mail# /etc/init.d/sendmail status
MSP: is run via cron (20m)
MTA: is not running
QUE: Same as MTA

/var/spool/mqueue

root@web:/var/spool# ls -alh
total 48M
drwxr-xr-x  7 root  root  4.0K Mar 29  2013 .
drwxr-xr-x 15 root  root  4.0K Jul 24 00:45 ..
drwxr-xr-x  5 root  root  4.0K May  1  2012 cron
lrwxrwxrwx  1 root  root     7 May  1  2012 mail -> ../mail
drwxr-s---  2 smmta smmsp  14M Jul 24 11:44 mqueue
drwxrws---  2 smmsp smmsp  34M Jul 24 12:25 mqueue-client
drwxr-xr-x  2 root  root  4.0K Apr 13  2012 plymouth
drwxr-xr-x  2 root  root  4.0K Mar 30  2012 rsyslog


root@web:/var/spool# du -h -d 1
4.0K    ./plymouth
1.6G    ./mqueue    <=====
4.0K    ./rsyslog
One message from /var/spool/mqueue 
root@web:/var/spool/mqueue# more qfs6OBTUZY003298 
V8
T1406201622
K1406201622
N1
P120781
I202/1/476577
MDeferred: 421 4.7.1 : (DYN:T1) http://postmaster.info.aol.com/errors/421dynt1.html
Fbs
$_localhost [127.0.0.1]
$rESMTP
$sweb.anybots.com
${daemon_flags}
${if_addr}127.0.0.1
S<[email protected]>
MDeferred: 421 4.7.1 : (DYN:T1) http://postmaster.info.aol.com/errors/421dynt1.html
rRFC822; [email protected]
RPFD:<[email protected]>
H?P?Return-Path: <?g>
H??Received: from web.anybots.com (localhost [127.0.0.1])
    by web.anybots.com (8.14.4/8.14.4/Debian-2ubuntu2) with ESMTP id s6OBTUZY003298
    for <[email protected]>; Thu, 24 Jul 2014 11:33:42 GMT
H??Received: (from www-data@localhost)
    by web.anybots.com (8.14.4/8.14.4/Submit) id s6JHVJId026134;
    Sat, 19 Jul 2014 17:31:19 GMT
H??Date: Sat, 19 Jul 2014 17:31:19 GMT
H??Message-Id: <[email protected]>
H??X-Authentication-Warning: web.anybots.com: www-data set sender to [email protected] using -f
H??To: [email protected]
H??Subject: Fw:  Hi Generic Drugs Online Products
H??X-PHP-Originating-Script: 33:dirs.php
H??From: "Patty Jennings" <[email protected]>
H??Reply-To:"Patty Jennings" <[email protected]>
H??X-Priority: 3 (Normal)
H??MIME-Version: 1.0
H??Content-Type: text/html; charset="iso-8859-1"
H??Content-Transfer-Encoding: 8bit
.

Your problem may be caused by HUGE number of (spam) messages in both sendmail queues.
(see https://serverfault.com/a/490890/163277 )

Check number of message in both sendmail queues 

sendmail -O QueueSortOrder=none -Am -bp
sendmail -O QueueSortOrder=none -Ac -bp

The most memory consuming sendmail process looks like MTA queue processing (-Am). The remaning looks like transfers from MSA to MTA queue and first time delivery attempts to external servers after such transfer.

You may use qtool.pl script to move messages send by www-data (web server) to another queue/directory. It is provided in contrib directory of sendmail.org distribution and in sendmail-base package by Debian-Linux.

Related

VSS does not truncate Exchange 2013 Logs on Server 2012
Gateway gets stripped from DHCP Reply through OpenVPN tunnel
Difference between /dev/udp and netcat
Connect to RDS from EC2 instance -- both inside VPC
Zabbix - service restart
btrfs Integrity and Stress Testing
On IPv6 linux router, autoconf and accept router advertisements for single interface
Amazon AWS IAM Policy for single VPC Subnet
Serve a custom 404 page generated by PHP
Nginx/Apache: set HSTS only if X-Forwarded-Proto is https
Rename a GlusterFS Peer
How to see which asp.net application in a pool is using the most memory