Load testing at 350,000 requests per minute [closed]

I need a way to reproduce the sudden burst of traffic our site experienced a while ago, to have a chance to keep our stack working.

Our load balancer reported at a certain point that some 350k requests were performed per minute, trashing everything, from the operating system to knocking off the backends. I tried looking into services that performs this kind of burst test, but it appears that when tweaking the test settings to match the desired conditions it would simply cost too much.

How can I load test my stack in a way that matches these conditions?

So, this is 5,833 requests per second, which is a lot, but its doable. I would recommend using a tool called siege. Experiment with different concurrency options until you reach a peak Transactions per second: rate in siege's output. You can also make use of ab (ApacheBench) from the Apache Project, but siege is much more powerful.

Then, add more machines until you can get to a total of, say, 6,000. You can probably make cost-effective use of Amazon EC2, or even just set up a few machines on different internet uplinks in different locations with ssh access to them. You could make use of cron or at to start siege at the same time on all of the machines (ensure that the clocks are in sync first).

As far as making use of an external company to do the test, yes, this might be prohibitively expensive, and I don't think I'm allowed to point to any particular company, but I've worked with a few in the past. This is definitely something you can do on your own with a few hours of EC2 time.

Edit

As @HBruijn awesomely pointed out, there is a great open-source tool called Bees with Machine Guns which will automate the creation, teardown, and coordination of load generation via EC2. This sounds like the right approach for your situation.