How to play traffic against a shadow network?
I'd call it "load testing via session replaying", personally. I don't know of any simple catch-all term for this kind of testing technique.
The basic strategy that I've seen employed for this kind of load testing is to ingest log files from the production system and replay them on a test system.
You can use tools like JMeter or Apache Bench to replay requests from log files. If you're looking at replaying very complex client / server interactions (with specific timing details based on the original log stream) in hopes of really exercising the innards of your application (looking for race conditions, timing-related bugs, etc) you might look at writing application-specific testing tools that simulate clients at scale.
You're not going to be able to simply capture boatloads of raw network traffic and "replay" it with any TCP or IP-based protocol. TCP sequence numbers aren't going to match the original captured traffic and it's not going to work. IP-layer captures are going to be problematic because your simulated clients will need to answer for the captured sender's IP address. You'd be better off capturing traffic closer to layer 7 and using that to replay sessions because, otherwise, you're looking at writing a TCP simulator, too. (I could imagine using something like tshark to bust out the layer 7 data and timing from a TCP stream and replaying that, for example.)
Simply replaying network traffic simulates load but doesn't necessarily capture defects. Your simulated client would need to receive responses from the test server and parse them for correctness if you wanted load-test any test that the application is responding properly. Since your application is going to generate dynamic response data it's unlikely that your simulated client can simply compare the test server's response to the logged response from the production server. This is where you're going to get into writing a test harness specific to your application and its output.
You use a service like BrowserMob which simulates a lot people simultaneously accessing your website at once. These services don't replay logged traffic, because then you'd be missing the client side of the conversation. E.g, your servers would be trying to send packets to computers on the Internet that aren't expecting to receive them. But what these companies do is study the logs (generally at an application-level, not packet-level) and use that information to figure out which pages people are clicking on, how often, and in what sequence. This data is used to write scripts/macros which BrowserMob then repeats.
ApacheBench, as mentioned by another user, isn't really used much these days. It was more helpful 10 years ago when you just needed to figure out how quickly a static HTML document or JPEG can be served up under a heavy load. It's not a whole lot different than a bunch of people clicking reload, reload, reload over and over again on their web browser. You need something a bit smarter when testing a web app that has a more complex workflow.
I don't think you could do this at a network layer, though you could possibly get a specialized kernel for a hardware load balancer to handle the second server. Basically web traffic (TCP) will require an acknowledgement of each packet that is sent/received. So if a user sends a packet to your network, it would get duplicated to both your prod network, and your shadow network. The servers in each network reply, and the prod server's packet is forwarded back to your machine which shoots back an acknowledgement, and they merrily carry on their conversation. However if you drop your shadow server's packet, it won't see an acknowledgement. So, it will try resending it, and at the same time slow down its transmission speeds for all network activity (this is called windowing). It will keep retrying to send it until it times out, and the session is torn down. Honestly, you wouldn't even be able to complete a handshake to establish a connection in the first place.
About the closest you could come to this would be forwarding the original synchronization packet to your shadow server and then set the default gateway for those boxes as some non-existant location. Then anytime a user would try to set up a connection they'd get a real server on your prod network, and at the very least you'd send a syn packet to the shadow network. Darn, now you have me wondering how you could make this work too :)
I was able to ask @adrianco about this at a Netflix meetup.
The answer was that they wrote their own tool, which is basically a ServletFilter (sorry, Java-specific terminology) that recreates the current request and does an asynchronous fire-and-forget invocation on a target server.
The benefits are:
- 'Real World' traffic patterns against your test ("dark") infrastructure
- No need to record and then replay
The drawback:
- Gotta have the threads/CPU cycles to spare on your production boxes
- Latency on your test infrastructure could back up and affect your production boxes