How to delete stuck mail in active/incoming queue - postfix

After DoS postfix attack we have the incoming and active queues full of mails:

drwx------.  2 postfix root     1007616 nov  5 17:01 active
drwx------.  2 postfix root        4096 nov  5 11:31 bounce
drwx------.  2 postfix root        4096 feb 20  2014 corrupt
drwx------. 18 postfix root        4096 jun 30  2014 defer
drwx------. 18 postfix root        4096 jun 30  2014 deferred
drwx------.  2 postfix root        4096 sep  8 10:41 flush
drwx------.  2 postfix root        4096 feb 20  2014 hold
drwx------.  2 postfix root     1093632 nov  5 17:01 incoming
drwx-wx---.  2 postfix postdrop    4096 nov  5 17:01 maildrop
drwxr-xr-x.  2 root    root        4096 nov  5 16:49 pid
drwx------.  2 postfix root        4096 nov  5 16:49 private
drwx--x---.  2 postfix postdrop    4096 nov  5 16:49 public
drwx------.  2 postfix root        4096 feb 20  2014 saved
drwx------.  2 postfix root        4096 feb 20  2014 trace

Active queue:

[root@revres]# ls -la /var/spool/postfix/active/
total 992
drwx------.  2 postfix root 1007616 nov  5 17:01 .
drwxr-xr-x. 16 root    root    4096 nov  5 09:06 ..

Incoming queue:

[root@revres]# ls -la /var/spool/postfix/incoming/
total 1076
drwx------.  2 postfix root 1093632 nov  5 17:01 .
drwxr-xr-x. 16 root    root    4096 nov  5 09:06 ..

Running the postsuper -d ALL command doesn't delete anything, nor gives any output.

Is there any other way to empty those boxes?

If the ls -la only shows the two "files" . and .. then it is empty.

If you then say: "Why is . so big when it is empty"? Then the answer is: That is usual in ext3 or ext4 file systems. They reserve space for the inodes present in the directory. And even when all files are deleted (the inodes are gone) the reserved space for managing the inodes is still present. So nothing to worry about. (And even if: It is only one megabyte "big")

I'm pretty sure that after that attack your postfix lost consistency. The queues are actually in memory data structures, so messages might be on the disk, but postfix might not be aware of them. I'd recommend you to stop the postfix service, run postsuper -s (which repairs and checks the files structure) and start it again.

Same problem.

sudo ls -ln /var/spool/postfix/incoming shows 1472 files.

#sudo ls /var/spool/postfix/incoming/ -ln
total 1472
-rw------- 1 89 89   8192 Feb 14 15:38 0007B120A83
-rw------- 1 89 89      0 Feb 14 16:38 0030E120A9B
-rw------- 1 89 89   4096 Feb 14 18:04 04548120AE7
-rw------- 1 89 89 102400 Feb 14 16:34 069CA120A94
-rw------- 1 89 89      0 Feb 14 17:56 06E53120ADF
-rw------- 1 89 89      0 Feb 14 17:10 08ADF120AB6
-rw------- 1 89 89      0 Feb 14 18:36 09A56120B24
-rw------- 1 89 89      0 Feb 14 18:32 0B0D0120B11
-rw------- 1 89 89  36864 Feb 14 16:43 0BC4D120A9A
-rw------- 1 89 89      0 Feb 14 19:01 0C150120B3E
-rw------- 1 89 89      0 Feb 14 18:30 0CED5120B16

Restarted MailScanner and postfix services. Was also getting a bunch of errors from the Exchange 2010 server for which I am filtering and acting as a gateway.

 Out: 250 2.1.5 Ok
 In:  DATA
 Out: 354 End data with <CR><LF>.<CR><LF>
 Out: 451 4.3.0 Error: queue file write error
 In:  RSET
 Out: 421 4.3.0 Mail system error

mailq or postqueue -p shows an empty queue...

#mailq
Mail queue is empty

You can see to the minute when it hit the wall. Unfotunately, I'm running the EFA Project on Centos 6.8, so although:

#yum info postfix
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * EFA: dl4.efa-project.org
 * base: mirror.fusioncloud.co
 * epel: archive.linux.duke.edu
 * extras: mirrors.lga7.us.voxel.net
 * updates: mirrors.evowise.com
Installed Packages
Name        : postfix
Arch        : x86_64
Epoch       : 2
Version     : 3.1.3
Release     : 1.efa.el6
Size        : 14 M
Repo        : installed
From repo   : EFA
Summary     : Postfix Mail Transport Agent
URL         : http://www.postfix.org
License     : IBM
Description : Postfix is a Mail Transport Agent (MTA), supporting LDAP, SMTP AUTH (SASL),
            : TLS built for Email Filter Appliance (EFA)

I can't find the postfix-perl scripts packaged for this OS. I tried to fudge the Fedora packages, but my rpm-foo is very weak.

... EDITED ...

Snagging the ID from the filename and attempting to view it using postcat yields.

#postcat -vq 0007B120A83
postcat: name_mask: ipv4
postcat: inet_addr_local: configured 2 IPv4 addresses
postcat: fatal: open queue file 0007B120A83: Permission denied
[ssmith@foster-spam ~]$ sudo postcat -vq 0007B120A83
postcat: name_mask: ipv4
postcat: inet_addr_local: configured 2 IPv4 addresses
*** ENVELOPE RECORDS incoming/0007B120A83 ***
message_size:               0               0               0               0               0               0
postcat: fatal: invalid size record:               0               0               0               0               0               0

Searching the maillog:

#sudo grep -i '0007B120A83' /var/log/maillog
Feb 14 15:36:36 foster-spam postfix/smtpd[16368]: 0007B120A83: client=foster-mail.foster2007.local[10.0.2.28]:63650
Feb 14 15:36:36 foster-spam postfix/cleanup[16371]: 0007B120A83: hold: header Received: from mail.fosterfuels.com (foster-mail.foster2007.local [10.0.2.28])??(using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits))??(No client certificate requested)??by mx.fosterfuels.com  from foster-mail.foster2007.local[10.0.2.28]:63650; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<mail.fosterfuels.com>
Feb 14 15:36:36 foster-spam postfix/cleanup[16371]: 0007B120A83: message-id=<6B24BD0263D83043837040657FCAC53414F05903@foster-mail.FOSTER2007.local>
Feb 14 19:41:51 foster-spam postfix/postsuper[20067]: queue: 0007B120A83
Feb 14 19:41:51 foster-spam postfix/postsuper[20067]: fatal: invalid directory name: 0007B120A83

I think at this point I might assume that all those files are just trash and hope like crazy that no mail was eaten during the DOS/virus deluge that kicked this off...

Check /var/spool/postfix/defer and deferred - make sure they are blank before (re)starting postfix.

A directory file is a special file that ONLY expands and never contracts as they fill with more and more files and directories underneath it. I have experienced this problem especially with files in /tmp with processes having run-away scripting that created thousands of intermediate files.

If you want to reduce the size of a directory file, the basic steps are as follows:

1. Terminate any processes using bigdir

2. mv bigdir bigdir.x

3. mkdir bigdir

4. mv bigdir.x/* bigdir ;; move existing files to the new smaller dir

5. mv bigdir.x/.[a-zA-Z0-9]* bigdir ;; copy hidden files but not . and .

6. Change permissions, ACLs, SEL labeling to match biddir.x

7. rmdir bigdir.x ;; bigdir.x should be empty

8. You are free to (re)start any processes using bigdir

The resulting directory will be much smaller than the original.