are following assertions about linux load and tomcat right?
1) Yes: Tomcat uses a thread to process a request.
2) Yes: Oracle JVM and OpenJDK JVM work with native threads only since JRE 1.3
3) Yes: Linux's run queue contains processes and threads (id native threads) the same way
4) Yes: Load average provides the average number of process/threads in the run queue
5) N/A
6) Yes: On Linux the load average counts processes/threads in state running, runnable, and uninterruptible sleep
7) This needs rephrasing. You seem to know how swap space works. So the big point: I/O to normal files is usually done with the same mechanism (not similar - literally the same one). This is called mmap
. When your application wants to write to a file abc.txt
it just changes bytes at a predefined memory address. The memory page (4096 bytes) is marked dirty
. Soon it becomes written by a background daemon to filesystem (and not to swap space). And when application just opens file for reading it gets the memory address with initially no memory page. When it actually accesses the memory there the kernel reads the pages of the file and make them appear there. This is how general disk cache works in Linux - it is very much paging, nothing more.
Now the network part. Linux has a couple of ways to mount filesystems that don't reside on local SAS/SATA/FC disks, but in fact communicate over the network (for example NFS or CIFS). So paging the a.txt
in fact causes packets to fly over the network. Your process doesn't have a socket in this specific case, just a mmapped address in its own address space. Everything it does is moving bytes between memory locations.
Uninterruptible sleep is defined as waiting for that mechanism - whether the filesystem is mounted from disk or from network. Lets say we're reading and we're waiting for the next page - the process needs to exist and needs to own the memory, because the memory has to be ready to receive the bytes being read from the disk (think: DMA in progress). That's why it stays despite you kill -9
it.
And the normal network traffic done with sockets owned by a process is a standard sleep, this one not counted in the load average.
A process could also read disks "the old way" without mmap
, this would be also standard sleep. It's quite rare.
"Precise" is not a useful thing to achieve with load average, in my opinion. It is an algorithm, but somewhat arbitrarily developed aggregation of what a slow system feels like. From sched/loadavg.c:
This file contains the magic bits required to compute the global loadavg figure. Its a silly number but people think its important.
In particular, that Linux Journal article is a bit misleading to talk about load average as a CPU load metric. Since the TASK_UNINTERRUPTIBLE addition circa 1993, off CPU states such as I/O or certain locks held. Which makes it more of a system load metric. (And this is specific to Linux.)
Gregg's post you referenced, Linux Load Averages: Solving the Mystery did not have a complete list of these because
Nowadays, in Linux 4.12, there are nearly 400 codepaths that set TASK_UNINTERRUPTIBLE, including some lock primitives.
Rather that a complete list, that shows how to measure kernel uninterruptible via an off-CPU flame graph. Which is useful to understand why your workload's load average might be high even if CPU utilization may be low.
Good performance monitoring profiles application user response time, for the complete request.
Find any correlation of response time with load average. It may be that 6 load on a 4 CPU system is a symptom of degraded performance of your workload. (I made this number up, look at your data.)
When load is high, confirm the application is responding slowly, then dig deeper into every resource of the system to find out why.