MSMQ very slow to receive messages
Whenever someone says they have over a million messages the alarm klaxons go off! Messages require kernel (paged pool) memory to be managed. If you have such a vast number of messages, you may be exhausting what is available on the clustered server. An optimal number for number of messages in a queue is zero - basically make sure you can normally process messages faster than they can arrive.
I would recommend shutting down the web servers and completely processing the backlog of messages before bring them back online again.
Reference Item 4 of this blog post: http://blogs.msdn.com/johnbreakwell/archive/2006/09/18/insufficient-resources-run-away-run-away.aspx
Cheers John Breakwell (MSFT)
I asked one of our sysadmins and he said our magic point was 4 web servers max hitting MSMQ box on virtual machines, then they moved to hardware box to solve. Also try packet capture to see what is going on. Is there much in authentication going to AD also? With how chatty MSMQ is, you need to limit network paths and possibly authentication path.
HTH, Chuck.
Referencing your comment about lack of remote administration, yes, it's not a great story with MSMQ and perf counters. For anyone following the thread and wanting to know what combinations of OSes work then have a look at the Motley Queue blog:
MSMQ 4.0 Performance Counters and the NetNameForPerfCounters Registry Key http://blogs.msdn.com/motleyqueue/archive/2007/12/14/msmq-4-0-performance-counters-and-the-netnameforperfcounters-registry-key.aspx
Cheers John Breakwell (MSFT)