Running a test suite with over a million test cases

I don't know any specifics about Python, but you might want to try spinning up each test case as a separate process. This will allow you to recover the memory from each process as it completes, and may allow you to run your million test cases safely on one machine.

Generally, when a program needs more memory from the OS, it takes it, but it cannot give it back until it exits. This is likely why your process falls over.

Quick processes that you can spin up and then kill, will alleviate this memory problem.... and then it is a pretty short step from there to running each of these test cases through rsh on a different machine entirely.

I hope that helps a little.