Virtualbox HTTP load testing, host CPU overload issues

I'm doing HTTP load testing benchmarks (using Apache Benchmark and Siege) on a small Java EE 1.7.0 / Tomcat 7.0.26 application running on a Debian Squeeze 6.0.4 x64 virtualized with Virtualbox 4.1.8. The computer host is Ubuntu 11.10 x64.

I've modified those parameters in the Tomcat server.xml :

<Connector 
    port="8080" 
    protocol="HTTP/1.1"
    connectionTimeout="200000"
    redirectPort="8443" 
    acceptCount="2000"
    maxThreads="150"
    minSpareThreads="50" />

The application executed on the server takes around 300ms.

This app is running well until a certain amount of concurrent connections like those one :

ab -n 500 -c 150 http://xx.xx.xx.xx:8080/myapp/
ab -n 1000 -c 50 http://xx.xx.xx.xx:8080/myapp/
siege -b -c 100 -r 20 http://xx.xx.xx.xx:8080/myapp/ 

A lot of socket connection timed out happens and this completly overload the host processor (but the CPU load inside the VM is normal).

Doing an htop on the host, i can see that the Virtualbox processus is running under 300% CPU and never come down even after the load test is finished. (I've allocated 4 processors to the VM, if I allocate only one processor, CPU load goes under 100%).

Restarting Tomcat don't do anything, i'm forced to restart the whole VM.

I've tryed to launch those ab/siege commands locally on the VM and everything goes well.

I first thought it was related to a linux network limit as explained here: Running some benchmarks using ab, and tomcat starts to really slow down So I've modified those TCP parameters :

echo 15 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

It seems to be better, but it continues to overload the host CPU and output socket connections time out at a certain amount of concurrent connections.

I'm wondering if this is not related to how Virtualbox handles external concurrent connections.

Solution 1:

I've seen this problem before when you have a loop that causes your network traffic to be re-sent forever. Try checking to see if you're getting a absurdly high amount of traffic going through any of your network interfaces.

If that doesn't work, does the CPU usage stop when the guest OS is stopped, or do you need to shutdown the whole VM container?

Solution 2:

First of all you don't say anything about the VBox host. How powerful is it? When I do VBox load testing I use a machine with 12 cores and 32G of RAM to run a virtual server with 2 to 8 cores and up to 12G of RAM. In otherwords, the underlying host is more powerful than the VBox guest so I am reasonably sure that the guest has dedicated cores and RAM.

Next, I tune both the host and the guest OSes to handle load. Things like the max open files, max sockets being opened, and various TCP/IP settings like buffers. These need to be set right to do load testing. If you graph your data with a range of loads, you will see it get suddenly much worse when you hit an OS limit. Fix this before you load test your app.