Xargs and Wget stops working after an hour

Running script with Cygwin on Windows XP with Dual Core and 4GB Ram

cat url_list.txt | xargs -P50 wget -i

I am trying to trawl through 4GB of URL to download (approx 43 Million)

Works okay for about the first hour, then the Bash shell and downloads stop even though its only 2% through the URL list.

Any ideas at what could be wrong?

What is the best way to debug why this is stoping after an hour?


It's possible wget is taking time to download some of the files. Are there any wget/xargs processes in memory during the period that it appears to be hung? If so, is it the full 50 processes as you allocated with the -P50 flag to xargs, or has it somehow creeped up over that number or less than that number and no new instances are being spawned properly? Although it's being run under cygwin, take a look at the process list in windows itself, as each wget download should launch an instance in the task manager.