SocketException in Mongo logs
First off, take a look and follow the steps outlined here:
http://www.mongodb.org/display/DOCS/Troubleshooting#Troubleshooting-Socketerrorsinshardedclustersandreplicasets
Next, look for any ulimit issues on the target host (a new file descriptor is required for a new socket and can cause the error):
http://www.mongodb.org/display/DOCS/Too+Many+Open+Files
Finally, there are a couple of issues related to idle connections being used when they should not be, and that can also contribute to this type of issue:
https://jira.mongodb.org/browse/SERVER-5793
Until SERVER-5632 is complete, the only remedy here is to flush the connections by restarting the mongod/mongos processes.
I got this to stop happening by adding a batchSize option arg to my cursor query
e.g. before:
var cursor = db.collection('images').find( {}, {"_id": 1} ).sort({"_id": 1});
after:
var cursor = db.collection('images').find( {}, {"_id": 1} ).sort({"_id": 1}).batchSize(1000);
This made it run smoother (not as many start/stops) and with fewer nasty messages. None in fact.