Tomcat crashing after a few hours
I'm running a CentOS 6.5 64 bit OS with Tomcat and one web application. I use Apache (httpd) as a reverse proxy (ProxyPass) to the application. Until recently, it (Tomcat) has been running fine without crashing. Recently, I added a Joomla site and PHP (obviously) to the http server. It was shortly after that where I started getting Tomcat crashes.
When it crashes, Tomcat doesn't leave evidence in the Catalina.out. I finally found output in /var/log/messages. I was hoping someone could have a cursory look and point me in the right direction:
============
Jul 23 01:19:31 myservername kernel: httpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 23 01:19:31 myservername kernel: httpd cpuset=/ mems_allowed=0
Jul 23 01:19:31 myservername kernel: Pid: 6104, comm: httpd Not tainted
2.6.32-431.1.2.0.1.el6.x86_64 #1
Jul 23 01:19:31 myservername kernel: Call Trace:
Jul 23 01:19:31 myservername kernel: [<ffffffff810d05c1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Jul 23 01:19:31 myservername kernel: [<ffffffff81122970>] ? dump_header+0x90/0x1b0
Jul 23 01:19:31 myservername kernel: [<ffffffff8122799c>] ? security_real_capable_noaudit+0x3c/0x70
Jul 23 01:19:31 myservername kernel: [<ffffffff81122df2>] ? oom_kill_process+0x82/0x2a0
Jul 23 01:19:31 myservername kernel: [<ffffffff81122d31>] ? select_bad_process+0xe1/0x120
Jul 23 01:19:31 myservername kernel: [<ffffffff81123230>] ? out_of_memory+0x220/0x3c0
Jul 23 01:19:31 myservername kernel: [<ffffffff8112fb4c>] ? __alloc_pages_nodemask+0x8ac/0x8d0
Jul 23 01:19:31 myservername kernel: [<ffffffff81167aaa>] ? alloc_pages_current+0xaa/0x110
Jul 23 01:19:31 myservername kernel: [<ffffffff8111fd67>] ? __page_cache_alloc+0x87/0x90
Jul 23 01:19:31 myservername kernel: [<ffffffff8111f74e>] ? find_get_page+0x1e/0xa0
Jul 23 01:19:31 myservername kernel: [<ffffffff81120d07>] ? filemap_fault+0x1a7/0x500
Jul 23 01:19:31 myservername kernel: [<ffffffff8114a094>] ? __do_fault+0x54/0x530
Jul 23 01:19:31 myservername kernel: [<ffffffff8114a667>] ? handle_pte_fault+0xf7/0xb00
Jul 23 01:19:31 myservername kernel: [<ffffffff811214a0>] ? generic_file_aio_read+0x380/0x700
Jul 23 01:19:31 myservername kernel: [<ffffffff8114b29a>] ? handle_mm_fault+0x22a/0x300
Jul 23 01:19:31 myservername kernel: [<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480
Jul 23 01:19:31 myservername kernel: [<ffffffff8100988e>] ? __switch_to+0x26e/0x320
Jul 23 01:19:31 myservername kernel: [<ffffffff81527920>] ? thread_return+0x4e/0x76e
Jul 23 01:19:31 myservername kernel: [<ffffffff8152d46e>] ? do_page_fault+0x3e/0xa0
Jul 23 01:19:31 myservername kernel: [<ffffffff8152a825>] ? page_fault+0x25/0x30
Jul 23 01:19:31 myservername kernel: Mem-Info:
Jul 23 01:19:31 myservername kernel: Node 0 DMA per-cpu:
Jul 23 01:19:31 myservername kernel: CPU 0: hi: 0, btch: 1 usd: 0
Jul 23 01:19:31 myservername kernel: Node 0 DMA32 per-cpu:
Jul 23 01:19:31 myservername kernel: CPU 0: hi: 186, btch: 31 usd: 30
Jul 23 01:19:31 myservername kernel: active_anon:112069 inactive_anon:56 isolated_anon:0
Jul 23 01:19:31 myservername kernel: active_file:17 inactive_file:465 isolated_file:0
Jul 23 01:19:31 myservername kernel: unevictable:0 dirty:1 writeback:0 unstable:0
Jul 23 01:19:31 myservername kernel: free:1181 slab_reclaimable:1556 slab_unreclaimable:5516
Jul 23 01:19:31 myservername kernel: mapped:78 shmem:58 pagetables:2575 bounce:0
Jul 23 01:19:31 myservername kernel: Node 0 DMA free:2040kB min:84kB low:104kB high:124kB active_anon:13500kB inactive_anon:0kB active_file:12kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15352kB mlocked:0kB dirty:0kB writeback:0kB mapped:24kB shmem:0kB slab_reclaimable:40kB slab_unreclaimable:76kB kernel_stack:16kB pagetables:48kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:6 all_unreclaimable? no
Jul 23 01:19:31 myservername kernel: lowmem_reserve[]: 0 489 489 489
Jul 23 01:19:31 myservername kernel: Node 0 DMA32 free:2684kB min:2784kB low:3480kB high:4176kB active_anon:434776kB inactive_anon:224kB active_file:56kB inactive_file:1848kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500948kB mlocked:0kB dirty:4kB writeback:0kB mapped:288kB shmem:232kB slab_reclaimable:6184kB slab_unreclaimable:21988kB kernel_stack:944kB pagetables:10252kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:130 all_unreclaimable? yes
Jul 23 01:19:31 myservername kernel: lowmem_reserve[]: 0 0 0 0
Jul 23 01:19:31 myservername kernel: Node 0 DMA: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2040kB
Jul 23 01:19:31 myservername kernel: Node 0 DMA32: 193*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2684kB
Jul 23 01:19:31 myservername kernel: 547 total pagecache pages
Jul 23 01:19:31 myservername kernel: 0 pages in swap cache
Jul 23 01:19:31 myservername kernel: Swap cache stats: add 0, delete 0, find 0/0
Jul 23 01:19:31 myservername kernel: Free swap = 0kB
Jul 23 01:19:31 myservername kernel: Total swap = 0kB
Jul 23 01:19:31 myservername kernel: 131068 pages RAM
Jul 23 01:19:31 myservername kernel: 5503 pages reserved
Jul 23 01:19:31 myservername kernel: 13454 pages shared
Jul 23 01:19:31 myservername kernel: 119917 pages non-shared
Jul 23 01:19:31 myservername kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 23 01:19:31 myservername kernel: [ 325] 0 325 2727 168 0 -17 -1000 udevd
Jul 23 01:19:31 myservername kernel: [ 602] 0 602 2726 167 0 -17 -1000 udevd
Jul 23 01:19:31 myservername kernel: [ 878] 0 878 23300 78 0 -17 -1000 auditd
Jul 23 01:19:31 myservername kernel: [ 894] 0 894 62368 729 0 0 0 rsyslogd
Jul 23 01:19:31 myservername kernel: [ 1027] 0 1027 29325 153 0 0 0 crond
Jul 23 01:19:31 myservername kernel: [ 1050] 0 1050 1016 21 0 0 0 mingetty
Jul 23 01:19:31 myservername kernel: [ 1052] 0 1052 1016 22 0 0 0 mingetty
Jul 23 01:19:31 myservername kernel: [ 1054] 0 1054 1016 22 0 0 0 mingetty
Jul 23 01:19:31 myservername kernel: [ 1056] 0 1056 1016 22 0 0 0 mingetty
Jul 23 01:19:31 myservername kernel: [ 1058] 0 1058 1016 21 0 0 0 mingetty
Jul 23 01:19:31 myservername kernel: [ 1189] 38 1189 7679 158 0 0 0 ntpd
Jul 23 01:19:31 myservername kernel: [19331] 99 19331 11317 405 0 0 0 openvpn
Jul 23 01:19:31 myservername kernel: [19336] 0 19336 2726 167 0 -17 -1000 udevd
Jul 23 01:19:31 myservername kernel: [27296] 0 27296 16652 178 0 -17 -1000 sshd
Jul 23 01:19:31 myservername kernel: [29754] 0 29754 1016 22 0 0 0 mingetty
Jul 23 01:19:31 myservername kernel: [14500] 0 14500 20321 226 0 0 0 master
Jul 23 01:19:31 myservername kernel: [14503] 89 14503 20384 233 0 0 0 qmgr
Jul 23 01:19:31 myservername kernel: [ 3155] 0 3155 27042 66 0 0 0 mysqld_safe
Jul 23 01:19:31 myservername kernel: [ 3257] 27 3257 160134 4405 0 0 0 mysqld
Jul 23 01:19:31 myservername kernel: [ 6102] 0 6102 76169 1432 0 0 0 httpd
Jul 23 01:19:31 myservername kernel: [ 6104] 48 6104 104399 5480 0 0 0 httpd
Jul 23 01:19:31 myservername kernel: [ 6105] 48 6105 104565 5583 0 0 0 httpd
Jul 23 01:19:31 myservername kernel: [ 6106] 48 6106 104301 5322 0 0 0 httpd
Jul 23 01:19:31 myservername kernel: [ 6107] 48 6107 104630 5682 0 0 0 httpd
Jul 23 01:19:31 myservername kernel: [ 6108] 48 6108 104491 5570 0 0 0 httpd
Jul 23 01:19:31 myservername kernel: [ 6109] 48 6109 104563 5570 0 0 0 httpd
Jul 23 01:19:31 myservername kernel: [ 6110] 48 6110 104546 5639 0 0 0 httpd
Jul 23 01:19:31 myservername kernel: [ 6111] 48 6111 104362 5469 0 0 0 httpd
Jul 23 01:19:31 myservername kernel: [ 6171] 48 6171 101911 3019 0 0 0 httpd
Jul 23 01:19:31 myservername kernel: [ 6186] 48 6186 101530 2632 0 0 0 httpd
Jul 23 01:19:31 myservername kernel: [ 6522] 48 6522 104451 5479 0 0 0 httpd
Jul 23 01:19:31 myservername kernel: [ 6835] 501 6835 317935 55623 0 0 0 java
Jul 23 01:19:31 myservername kernel: [ 7422] 48 7422 104365 5444 0 0 0 httpd
Jul 23 01:19:31 myservername kernel: [ 7656] 89 7656 20341 220 0 0 0 pickup
Jul 23 01:19:31 myservername kernel: Out of memory: Kill process 6835 (java) score 444 or sacrifice child
Jul 23 01:19:31 myservername kernel: Killed process 6835, UID 501, (java) total-vm:1271740kB, anon-rss:222432kB, file-rss:60kB
============
I already raised the heap settings from the default, but that didn't help:
-Xms256m -Xmx512m
I assume that the right direction is to start profiling/monitoring the JVM with one of the tools that comes with the JDK/JRE and see where the leak is. It's just odd that I never had this problem before fiddling with the new Joomla site. I'm not familiar with profiling/monitoring, so I'd appreciate suggestions on which tool to use for my particular situation, given the log file snippet included.
Solution 1:
It seems that there is no more memory available to use on your system. Please check if you have swap memory on your system with below command
free -m
output should be as below if you have swap enabled.
total used free shared buffers cached
Mem: 1876 1798 77 0 119 1440
-/+ buffers/cache: 237 1638
Swap: 4607 0 4607
If you can't see swap in your file, you have to create swapfile. Here is a guide to create and enable swapfile.
http://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/
Also you can configure javamelody with your tomcat application to identify issues with your tomcat application
--- UPDATE
I've wrote a post about swap memory, if you any question please comment on the below post
https://thinkersbase.blogspot.in/2018/03/create-linux-swap.html
Solution 2:
Here is the cause:
Jul 23 01:19:31 myservername kernel: [<ffffffff81122df2>] ? oom_kill_process+0x82/0x2a0
There is not enough ram. oom_kill_process is the kernel function which kills some processes if there is no ram enough. Something seems leaking, or you have very few memory. Is your swap really enabled and working? (cat /proc/swaps).
Extension: some are saying, that enabling swap makes your system slower. That is false, don't believe that, see my comment.