Should I disable turn off MongoDB profiler in production machine?
The default according to [1] is collect profile data when execution time > 100ms
But one of the issue is when majority of queries' execution time > 100ms, our system start to overload and I am not sure if the profiler make our system even more busy.
I am thinking to disable or set a large value, e.g. 2 sec. Seems the default of 100ms is too small?
[1] http://docs.mongodb.org/manual/tutorial/manage-the-database-profiler/
Solution 1:
First, the profiler is off by default. Slow queries (by default that is any query that takes over 100ms, as you mention) will be logged to the MongoDB log, but that is not the same as the profiler and will not take up a lot of resources (it is essentially just writing a line of text to a file outside the DB, no locking etc.). It is generally safe to leave this as-is and the benefits of spotting slow queries generally far outweigh the downside of having a larger log file.
Second, when the profiler is enabled, it writes all the data it collects to the system.profile collection. At the default level of 0 it is disabled, at level 1 it will log all "slow" operations based on that same limit, and at level 2 it will log all operations, regardless of speed.
As you can imagine, having it set to 2 is very resource intensive - you would be doubling your write volume, and adding a write for every read. Depending on your workload, a level of 1 can be intensive too. In general, I would recommend the profiler only under controlled circumstances (maintenance window, debugging, development/QA environment).
Although you can consider the two methods of recording slow ops as completely sepearate, currently, the slowms value will affect both methods in terms of what is considered a slow query.
Hence, if you were to set the value higher (2 seconds), it would decrease the amount of operations written to your log. The downside to that is that you will miss all operations that are taking up to that amount of time, generally considered an eternity in modern database terms. The 100ms value is generally a decent starter, and I have even seen people lower it based on their application requirements in terms of timeouts, but it is rare that increasing the value is worth it.