what does locked % in mongostat mean?

When running mongostat to look at our mongo database, I'll frequently see the locked % number jump up, sometimes as high as 80%. Here's a few example rows:

insert  query update delete getmore command flushes mapped  vsize    res faults locked % idx miss %     qr|qw   ar|aw  netIn netOut  conn       time 
 0     10      7      0       0      17       0  2.27g     5g  1.63g      0      0.1          0       0|0     0|0    22k     7k    83   15:46:10 
 0     21      7      1       0      23       0  2.27g     5g  1.73g      0      0.1          0       0|0     0|1    11k   424k    83   15:46:11 
 0     28     10      3       0      28       0  2.27g     5g  1.73g      0     26.9          0       0|0     0|0    33k   196k    83   15:46:12 
 0     17      6      3       0      13       0  2.27g     5g  1.72g      0     18.2          0       0|0     0|0    11k    10k    83   15:46:13 
 0     18      5      1       0      11       0  2.27g     5g  1.73g      0      0.1          0       0|0     0|0    23k   362k    83   15:46:14 

The help for mongostat says it's

locked - percent of time in global write lock

My understanding is that writes lock the whole database. If I have two databases on the same mongo server (A and B), will writes on A block writes on B? Does half the time on one database mean it'll show 25% (like using half of one core on a dual-core box), or will it show 50%?

Yes, MongoDB utilizes a global write lock (server-wide), so "lock %" as reported by mongostat represents the total amount of time the server spent in global write lock during the last sample period (one second) -- for all ops on all dbs.

As you do more write operations such as updates, inserts, deletions, and evals, this value is going to increase.

The numbers in your sample above aren't alarming for a standalone server.

Regular readings of 80%, though, would suggest you may want to balance your reads and writes across more servers or shard your collection(s), though I would look to sharding to relieve storage problems before performance ones.

In general, keys to optimization include correct and comprehensive indexing, limiting results, using the profiler and explain() to uncover bottlenecks, and making sure you're using the driver appropriately (experimenting with the cursor batch size can produce significant improvements).

The high write lock percentages isn't the sole indication of performance. You should combine this number with your write queue which indicate the number of inserts/updates/upserts and remove operations that are queued (in total) to acquire a lock. If both the numbers remain consistently high there is a problem which should be addressed immediately. If that the case, start by pulling out the topmost slow queries, analyze and try to tune them. When no further tuning is possible, turn to the resources allocated on the server.