OOM invoked with plenty of free swap
I am running an ARM router with i2p and tor on it - a Netgear R7000. Of course I've added a full 512 MB of SWAP to it to prevent OOMs, understanding that it could slow down the system... But then I still get OOM killer starting with plenty of SWAP free, and killing tor! More interesting, after killing tor it seems that the system is OK for an unlimited amount of time... But still seems like tor couldn't be swapped. I even tried to turn off the overcommit, didn't helped at all.
See the dmesg log below
resetbutton invoked oom-killer: gfp_mask=0x2000d0, order=0, oom_score_adj=0
CPU: 0 PID: 1500 Comm: resetbutton Tainted: P 3.10.79 #381
Backtrace:
[<c0015cb8>] (dump_backtrace+0x0/0x118) from [<c0015ec0>] (show_stack+0x18/0x1c)
r6:c7a4cdc0 r5:00000000 r4:c6818000 r3:00000000
[<c0015ea8>] (show_stack+0x0/0x1c) from [<c012e5c0>] (dump_stack+0x24/0x28)
[<c012e59c>] (dump_stack+0x0/0x28) from [<c007c5cc>] (dump_header.isra.13+0x84/0x194)
[<c007c548>] (dump_header.isra.13+0x0/0x194) from [<c007c958>] (oom_kill_process+0x90/0x3e8)
[<c007c8c8>] (oom_kill_process+0x0/0x3e8) from [<c007d17c>] (out_of_memory+0x2c0/0x304)
[<c007cebc>] (out_of_memory+0x0/0x304) from [<c007fcbc>] (__alloc_pages_nodemask+0x5b0/0x694)
[<c007f70c>] (__alloc_pages_nodemask+0x0/0x694) from [<c00a2464>] (cache_alloc_refill+0x2c8/0x600)
[<c00a219c>] (cache_alloc_refill+0x0/0x600) from [<c00a214c>] (kmem_cache_alloc+0x88/0xd8)
[<c00a20c4>] (kmem_cache_alloc+0x0/0xd8) from [<c00afdd4>] (getname_flags+0x2c/0x104)
r7:b6f5509c r6:00000000 r5:00000000 r4:00020000
[<c00afda8>] (getname_flags+0x0/0x104) from [<c00afec4>] (getname+0x18/0x1c)
[<c00afeac>] (getname+0x0/0x1c) from [<c00a5270>] (do_sys_open+0xbc/0x178)
[<c00a51b4>] (do_sys_open+0x0/0x178) from [<c00a5350>] (SyS_open+0x24/0x28)
r9:c6818000 r8:c0008d24 r7:00000005 r6:00000000 r5:00000000
r4:00000000
[<c00a532c>] (SyS_open+0x0/0x28) from [<c0008ba0>] (ret_fast_syscall+0x0/0x30)
Mem-info:
Normal per-cpu:
CPU 0: hi: 42, btch: 7 usd: 0
CPU 1: hi: 42, btch: 7 usd: 0
HighMem per-cpu:
CPU 0: hi: 42, btch: 7 usd: 0
CPU 1: hi: 42, btch: 7 usd: 0
active_anon:10263 inactive_anon:12797 isolated_anon:0
active_file:957 inactive_file:1335 isolated_file:0
unevictable:151 dirty:0 writeback:0 unstable:0
free:13787 slab_reclaimable:912 slab_unreclaimable:5734
mapped:1171 shmem:0 pagetables:283 bounce:0
free_cma:0
Normal free:26184kB min:32768kB low:40960kB high:49152kB active_anon:0kB inactive_anon:0kB active_file:96kB inactive_file:64kB unevictable:64kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:124196kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:3648kB slab_unreclaimable:22936kB kernel_stack:1192kB pagetables:1132kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4440 all_unreclaimable? no
lowmem_reserve[]: 0 1024 1024
HighMem free:28656kB min:128kB low:8772kB high:17416kB active_anon:41052kB inactive_anon:51512kB active_file:3764kB inactive_file:5244kB unevictable:540kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:131072kB mlocked:0kB dirty:0kB writeback:0kB mapped:4684kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
Normal: 102*4kB (UEM) 118*8kB (UEM) 360*16kB (UM) 158*32kB (UM) 28*64kB (UEM) 8*128kB (UM) 4*256kB (UM) 4*512kB (UM) 0*1024kB 0*2048kB 2*4096kB (R) = 26248kB
HighMem: 1177*4kB (UMR) 1620*8kB (UMR) 225*16kB (UMR) 30*32kB (UMR) 29*64kB (UMR) 8*128kB (UMR) 1*256kB (U) 2*512kB (U) 0*1024kB 1*2048kB (U) 0*4096kB = 28436kB
9347 total pagecache pages
6914 pages in swap cache
Swap cache stats: add 484419, delete 477505, find 187469/228822
Free swap = 408788kB
Total swap = 524284kB
65536 pages of RAM
13904 free pages
1661 reserved pages
6646 slab pages
549590 pages shared
6923 pages swap cached
[ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[ 622] 0 622 189 7 3 13 0 hotplug2
[ 626] 0 626 199 30 3 6 0 mstpd
[ 971] 0 971 217 9 3 8 0 dropbear
[ 987] 0 987 297 105 3 30 0 dnsmasq
[ 1112] 0 1112 315 29 4 30 0 ttraff
[ 1195] 0 1195 209 25 3 6 0 dhcp6c
[ 1288] 0 1288 755 3 4 83 0 httpd
[ 1330] 0 1330 315 26 3 18 0 process_monitor
[ 1334] 0 1334 361 44 3 38 0 upnp
[ 1337] 0 1337 334 50 4 9 0 wland
[ 1338] 0 1338 190 42 3 3 0 igmprt
[ 1339] 0 1339 291 10 3 6 0 udhcpc
[ 1449] 0 1449 182 27 3 3 0 cron
[ 1457] 0 1457 370 38 3 32 0 nas
[ 1460] 0 1460 370 36 4 32 0 nas
[ 1495] 0 1495 291 11 3 6 0 syslogd
[ 1497] 0 1497 291 15 3 6 0 klogd
[ 1500] 0 1500 334 32 3 12 0 resetbutton
[ 1684] 65534 1684 1008 18 5 499 0 tinyproxy
[ 1690] 65534 1690 1021 9 5 508 0 tinyproxy
[ 1691] 65534 1691 1021 9 5 508 0 tinyproxy
[ 1692] 65534 1692 1021 9 5 508 0 tinyproxy
[ 1693] 65534 1693 1008 6 5 502 0 tinyproxy
[ 1694] 65534 1694 1008 6 5 502 0 tinyproxy
[ 1695] 65534 1695 1008 6 5 502 0 tinyproxy
[ 1696] 65534 1696 1008 6 5 502 0 tinyproxy
[ 1697] 65534 1697 1008 6 5 502 0 tinyproxy
[ 1698] 65534 1698 1008 6 5 502 0 tinyproxy
[ 1699] 65534 1699 1008 6 5 502 0 tinyproxy
[ 1711] 106 1711 36556 12756 73 10236 0 tor
[ 1730] 13 1730 580 9 4 38 0 polipo
[ 1806] 1000 1806 2828 41 5 124 0 wrapper
[ 1808] 1000 1808 59927 7979 69 13797 0 java
[ 1921] 0 1921 241 18 3 31 0 dropbear
[ 1923] 0 1923 293 9 3 9 0 sh
[ 3705] 0 3705 804 5 4 86 0 bash
[ 5859] 0 5859 320 61 3 12 0 radvd
[ 5860] 0 5860 320 18 3 18 0 radvd
Out of memory: Kill process 1711 (tor) score 118 or sacrifice child
Killed process 1711 (tor) total-vm:146224kB, anon-rss:49036kB, file-rss:1988kB
tor: page allocation failure: order:0, mode:0x20058
CPU: 1 PID: 1711 Comm: tor Tainted: P 3.10.79 #381
Backtrace:
[<c0015cb8>] (dump_backtrace+0x0/0x118) from [<c0015ec0>] (show_stack+0x18/0x1c)
r6:00000000 r5:00000000 r4:00020058 r3:00000000
[<c0015ea8>] (show_stack+0x0/0x1c) from [<c012e5c0>] (dump_stack+0x24/0x28)
[<c012e59c>] (dump_stack+0x0/0x28) from [<c007def4>] (warn_alloc_failed+0x100/0x110)
[<c007ddf4>] (warn_alloc_failed+0x0/0x110) from [<c007faf0>] (__alloc_pages_nodemask+0x3e4/0x694)
r3:00040001 r2:00000000
r7:c0405180 r6:00000001 r5:00000000 r4:00020058
[<c007f70c>] (__alloc_pages_nodemask+0x0/0x694) from [<c007a274>] (find_or_create_page+0x48/0x94)
[<c007a22c>] (find_or_create_page+0x0/0x94) from [<c00cfc08>] (__getblk+0x16c/0x308)
r9:00000000 r8:0223764c r7:c7418110 r6:00000001 r5:00001000
r4:00000000
[<c00cfa9c>] (__getblk+0x0/0x308) from [<c00d0668>] (__bread+0x1c/0xd8)
[<c00d064c>] (__bread+0x0/0xd8) from [<bf5e96c8>] (_8+0x208/0x4a0 [ext2])
r4:c6bb5b58
[<bf5e963c>] (_8+0x17c/0x4a0 [ext2]) from [<bf5ea2e0>] (_35+0x78/0x884 [ext2])
[<bf5ea268>] (_35+0x0/0x884 [ext2]) from [<c00d6da8>] (do_mpage_readpage+0x268/0x6d4)
[<c00d6b40>] (do_mpage_readpage+0x0/0x6d4) from [<c00d72d4>] (mpage_readpages+0xc0/0x104)
[<c00d7214>] (mpage_readpages+0x0/0x104) from [<bf5e99ac>] (_61+0x24/0x28 [ext2])
[<bf5e9988>] (_61+0x0/0x28 [ext2]) from [<c0082f08>] (__do_page_cache_readahead+0x170/0x1f8)
[<c0082d98>] (__do_page_cache_readahead+0x0/0x1f8) from [<c0083428>] (ra_submit+0x30/0x38)
[<c00833f8>] (ra_submit+0x0/0x38) from [<c007bb4c>] (filemap_fault+0x1bc/0x434)
[<c007b990>] (filemap_fault+0x0/0x434) from [<c008e5a4>] (__do_fault+0xac/0x490)
[<c008e4f8>] (__do_fault+0x0/0x490) from [<c0091284>] (handle_pte_fault+0x78/0x6d4)
[<c009120c>] (handle_pte_fault+0x0/0x6d4) from [<c0091968>] (__handle_mm_fault+0x88/0x90)
[<c00918e0>] (__handle_mm_fault+0x0/0x90) from [<c00920e0>] (handle_mm_fault+0x28/0x2c)
[<c00920b8>] (handle_mm_fault+0x0/0x2c) from [<c001948c>] (do_page_fault+0x118/0x288)
[<c0019374>] (do_page_fault+0x0/0x288) from [<c0008580>] (do_DataAbort+0x40/0x9c)
[<c0008540>] (do_DataAbort+0x0/0x9c) from [<c0008998>] (__dabt_usr+0x38/0x40)
Exception stack(0xc6bb5fb0 to 0xc6bb5ff8)
5fa0: b6fe43d8 b6cfd864 00000005 00000053
5fc0: 00000001 b6fe43dd b6d0a358 b6cfd864 b866141c 00000000 b6fe43d8 00000000
5fe0: b6d0b658 be8798f0 b6cf2330 b6a4c388 20000010 ffffffff
r8:b866141c r7:b6cfd864 r6:ffffffff r5:20000010 r4:b6a4c388
Mem-info:
Normal per-cpu:
CPU 0: hi: 42, btch: 7 usd: 0
CPU 1: hi: 42, btch: 7 usd: 0
HighMem per-cpu:
CPU 0: hi: 42, btch: 7 usd: 0
CPU 1: hi: 42, btch: 7 usd: 0
active_anon:10263 inactive_anon:13171 isolated_anon:0
active_file:971 inactive_file:1322 isolated_file:0
unevictable:151 dirty:1 writeback:0 unstable:0
free:13621 slab_reclaimable:912 slab_unreclaimable:5739
mapped:1172 shmem:0 pagetables:283 bounce:0
free_cma:0
Normal free:27136kB min:32768kB low:40960kB high:49152kB active_anon:0kB inactive_anon:0kB active_file:40kB inactive_file:120kB unevictable:64kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:124196kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:3648kB slab_unreclaimable:22956kB kernel_stack:1192kB pagetables:1132kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:40 all_unreclaimable? no
lowmem_reserve[]: 0 1024 1024
HighMem free:27380kB min:128kB low:8772kB high:17416kB active_anon:41052kB inactive_anon:52684kB active_file:3764kB inactive_file:5248kB unevictable:540kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:131072kB mlocked:0kB dirty:4kB writeback:0kB mapped:4688kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
Normal: 96*4kB (UEM) 120*8kB (UEM) 360*16kB (UM) 182*32kB (UM) 32*64kB (UEM) 8*128kB (UM) 4*256kB (UM) 4*512kB (UM) 0*1024kB 0*2048kB 2*4096kB (R) = 27264kB
HighMem: 918*4kB (UMR) 1620*8kB (UMR) 225*16kB (UMR) 30*32kB (UMR) 29*64kB (UMR) 8*128kB (UMR) 1*256kB (U) 2*512kB (U) 0*1024kB 1*2048kB (U) 0*4096kB = 27400kB
9533 total pagecache pages
7099 pages in swap cache
Swap cache stats: add 484625, delete 477526, find 187576/228974
Free swap = 408868kB
Total swap = 524284kB
65536 pages of RAM
13939 free pages
1661 reserved pages
6652 slab pages
549523 pages shared
7099 pages swap cached
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
The sysctl vm related settings:
vm.admin_reserve_kbytes = 7914
vm.block_dump = 0
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.drop_caches = 0
vm.highmem_is_dirtyable = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 32 32
vm.max_map_count = 65530
vm.min_free_kbytes = 32768
vm.mmap_min_addr = 4096
vm.nr_pdflush_threads = 0
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_memory = 2
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.scan_unevictable_pages = 0
vm.stat_interval = 1
vm.swappiness = 100
vm.user_reserve_kbytes = 7914
vm.vfs_cache_pressure = 100
What am I doing wrong (except for using a SOHO router for tor, lol)?
PS: Using DD-WRT firmware
EDIT: Another OOM killer log - this time nothing about resetbutton:
kthreadd invoked oom-killer: gfp_mask=0x3000d0, order=1, oom_score_adj=0
CPU: 0 PID: 2 Comm: kthreadd Tainted: P 3.10.79 #381
Backtrace:
[<c0015cb8>] (dump_backtrace+0x0/0x118) from [<c0015ec0>] (show_stack+0x18/0x1c)
r6:c782fb80 r5:00000000 r4:c7834000 r3:00000000
[<c0015ea8>] (show_stack+0x0/0x1c) from [<c012e5c0>] (dump_stack+0x24/0x28)
[<c012e59c>] (dump_stack+0x0/0x28) from [<c007c5cc>] (dump_header.isra.13+0x84/0x194)
[<c007c548>] (dump_header.isra.13+0x0/0x194) from [<c007c958>] (oom_kill_process+0x90/0x3e8)
[<c007c8c8>] (oom_kill_process+0x0/0x3e8) from [<c007d17c>] (out_of_memory+0x2c0/0x304)
[<c007cebc>] (out_of_memory+0x0/0x304) from [<c007fcbc>] (__alloc_pages_nodemask+0x5b0/0x694)
[<c007f70c>] (__alloc_pages_nodemask+0x0/0x694) from [<c002bf70>] (copy_process.part.68+0xdc/0xe30)
[<c002be94>] (copy_process.part.68+0x0/0xe30) from [<c002cdd8>] (do_fork+0xe0/0x2bc)
[<c002ccf8>] (do_fork+0x0/0x2bc) from [<c002cfe4>] (kernel_thread+0x30/0x38)
r9:00000001 r8:00000000 r7:c7834000 r6:c0454014 r5:c04096a8
r4:c1109e88
[<c002cfb4>] (kernel_thread+0x0/0x38) from [<c004be50>] (kthreadd+0xe4/0x12c)
[<c004bd6c>] (kthreadd+0x0/0x12c) from [<c0008c38>] (ret_from_fork+0x14/0x3c)
Mem-info:
Normal per-cpu:
CPU 0: hi: 42, btch: 7 usd: 0
CPU 1: hi: 42, btch: 7 usd: 0
HighMem per-cpu:
CPU 0: hi: 42, btch: 7 usd: 0
CPU 1: hi: 42, btch: 7 usd: 0
active_anon:7196 inactive_anon:7223 isolated_anon:0
active_file:1054 inactive_file:1422 isolated_file:0
unevictable:199 dirty:3 writeback:0 unstable:0
free:22911 slab_reclaimable:969 slab_unreclaimable:6549
mapped:685 shmem:0 pagetables:210 bounce:0
free_cma:0
Normal free:29372kB min:32768kB low:40960kB high:49152kB active_anon:0kB inactive_anon:0kB active_file:44kB inactive_file:128kB unevictable:44kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:124196kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:3876kB slab_unreclaimable:26196kB kernel_stack:576kB pagetables:840kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1064 all_unreclaimable? no
lowmem_reserve[]: 0 1024 1024
HighMem free:62272kB min:128kB low:8772kB high:17416kB active_anon:28784kB inactive_anon:28892kB active_file:4068kB inactive_file:5676kB unevictable:752kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:131072kB mlocked:0kB dirty:12kB writeback:0kB mapped:2740kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
Normal: 465*4kB (UEM) 285*8kB (UEM) 219*16kB (UEM) 92*32kB (UEM) 51*64kB (UEM) 25*128kB (UEM) 12*256kB (UEM) 2*512kB (U) 0*1024kB 0*2048kB 2*4096kB (R) = 29340kB
HighMem: 5997*4kB (UM) 2743*8kB (UM) 478*16kB (UM) 27*32kB (UM) 2*64kB (M) 2*128kB (U) 1*256kB (U) 2*512kB (U) 0*1024kB 1*2048kB (U) 1*4096kB (R) = 62252kB
5489 total pagecache pages
2805 pages in swap cache
Swap cache stats: add 1237151, delete 1234346, find 539424/688368
Free swap = 484712kB
Total swap = 524284kB
65536 pages of RAM
23095 free pages
1661 reserved pages
7517 slab pages
557492 pages shared
2805 pages swap cached
[ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[ 622] 0 622 189 1 3 12 0 hotplug2
[ 626] 0 626 199 31 3 6 0 mstpd
[ 971] 0 971 217 1 3 7 0 dropbear
[ 987] 0 987 297 105 3 30 0 dnsmasq
[ 1112] 0 1112 315 61 4 10 0 ttraff
[ 1195] 0 1195 209 40 3 2 0 dhcp6c
[ 1288] 0 1288 755 1 4 82 0 httpd
[ 1330] 0 1330 315 38 3 17 0 process_monitor
[ 1334] 0 1334 366 89 3 11 0 upnp
[ 1337] 0 1337 334 50 4 9 0 wland
[ 1338] 0 1338 190 45 3 4 0 igmprt
[ 1339] 0 1339 291 30 3 5 0 udhcpc
[ 1449] 0 1449 182 32 3 2 0 cron
[ 1457] 0 1457 370 36 3 32 0 nas
[ 1460] 0 1460 370 34 4 32 0 nas
[ 1495] 0 1495 291 20 3 5 0 syslogd
[ 1497] 0 1497 291 37 3 2 0 klogd
[ 1500] 0 1500 334 32 3 12 0 resetbutton
[ 1684] 65534 1684 1008 15 5 499 0 tinyproxy
[ 1690] 65534 1690 1021 5 5 507 0 tinyproxy
[ 1691] 65534 1691 1021 5 5 507 0 tinyproxy
[ 1692] 65534 1692 1021 5 5 507 0 tinyproxy
[ 1693] 65534 1693 1008 4 5 501 0 tinyproxy
[ 1694] 65534 1694 1008 4 5 501 0 tinyproxy
[ 1695] 65534 1695 1008 4 5 501 0 tinyproxy
[ 1696] 65534 1696 1008 4 5 501 0 tinyproxy
[ 1697] 65534 1697 1008 4 5 501 0 tinyproxy
[ 1698] 65534 1698 1008 4 5 501 0 tinyproxy
[ 1699] 65534 1699 1008 4 5 501 0 tinyproxy
[ 1730] 13 1730 580 10 4 37 0 polipo
[ 1806] 1000 1806 2828 78 5 116 0 wrapper
[ 1921] 0 1921 241 51 3 19 0 dropbear
[ 1923] 0 1923 293 1 3 8 0 sh
[16658] 106 16643 33786 12722 65 7647 0 tor
[16661] 0 16661 804 4 4 85 0 bash
[17143] 0 17143 790 167 4 26 0 top
[20851] 0 20851 320 64 3 12 0 radvd
[20853] 0 20853 320 33 3 17 0 radvd
Out of memory: Kill process 16658 (tor) score 104 or sacrifice child
Killed process 16658 (tor) total-vm:135144kB, anon-rss:49088kB, file-rss:1800kB
tor: page allocation failure: order:0, mode:0x20058
CPU: 1 PID: 16658 Comm: tor Tainted: P 3.10.79 #381
Backtrace:
[<c0015cb8>] (dump_backtrace+0x0/0x118) from [<c0015ec0>] (show_stack+0x18/0x1c)
r6:00000000 r5:00000000 r4:00020058 r3:00000000
[<c0015ea8>] (show_stack+0x0/0x1c) from [<c012e5c0>] (dump_stack+0x24/0x28)
[<c012e59c>] (dump_stack+0x0/0x28) from [<c007def4>] (warn_alloc_failed+0x100/0x110)
[<c007ddf4>] (warn_alloc_failed+0x0/0x110) from [<c007faf0>] (__alloc_pages_nodemask+0x3e4/0x694)
r3:00040001 r2:00000000
r7:c0405180 r6:c6488000 r5:00000000 r4:00020058
[<c007f70c>] (__alloc_pages_nodemask+0x0/0x694) from [<c007a274>] (find_or_create_page+0x48/0x94)
[<c007a22c>] (find_or_create_page+0x0/0x94) from [<c00cfc08>] (__getblk+0x16c/0x308)
r9:00000000 r8:0220f844 r7:c7418110 r6:00000001 r5:00001000
r4:00000000
[<c00cfa9c>] (__getblk+0x0/0x308) from [<c00d0668>] (__bread+0x1c/0xd8)
[<c00d064c>] (__bread+0x0/0xd8) from [<bf5e96c8>] (_8+0x208/0x4a0 [ext2])
r4:c6489b58
[<bf5e963c>] (_8+0x17c/0x4a0 [ext2]) from [<bf5ea2e0>] (_35+0x78/0x884 [ext2])
[<bf5ea268>] (_35+0x0/0x884 [ext2]) from [<c00d6da8>] (do_mpage_readpage+0x268/0x6d4)
[<c00d6b40>] (do_mpage_readpage+0x0/0x6d4) from [<c00d72d4>] (mpage_readpages+0xc0/0x104)
[<c00d7214>] (mpage_readpages+0x0/0x104) from [<bf5e99ac>] (_61+0x24/0x28 [ext2])
[<bf5e9988>] (_61+0x0/0x28 [ext2]) from [<c0082f08>] (__do_page_cache_readahead+0x170/0x1f8)
[<c0082d98>] (__do_page_cache_readahead+0x0/0x1f8) from [<c0083428>] (ra_submit+0x30/0x38)
[<c00833f8>] (ra_submit+0x0/0x38) from [<c007bb4c>] (filemap_fault+0x1bc/0x434)
[<c007b990>] (filemap_fault+0x0/0x434) from [<c008e5a4>] (__do_fault+0xac/0x490)
[<c008e4f8>] (__do_fault+0x0/0x490) from [<c0091284>] (handle_pte_fault+0x78/0x6d4)
[<c009120c>] (handle_pte_fault+0x0/0x6d4) from [<c0091968>] (__handle_mm_fault+0x88/0x90)
[<c00918e0>] (__handle_mm_fault+0x0/0x90) from [<c00920e0>] (handle_mm_fault+0x28/0x2c)
[<c00920b8>] (handle_mm_fault+0x0/0x2c) from [<c001948c>] (do_page_fault+0x118/0x288)
[<c0019374>] (do_page_fault+0x0/0x288) from [<c000861c>] (do_PrefetchAbort+0x40/0x9c)
[<c00085dc>] (do_PrefetchAbort+0x0/0x9c) from [<c0008b58>] (ret_from_exception+0x0/0x10)
Exception stack(0xc6489fb0 to 0xc6489ff8)
9fa0: 0000021c 00000000 6df1d800 6df1d800
9fc0: b470b880 0000000f da4afeed b6fe04bc 00000040 5eedf00d b6fdb8f4 b6fba134
9fe0: 00000000 b470b858 b6f0ead4 b6f0ead4 60000010 ffffffff
r8:00000040 r7:b6fe04bc r6:ffffffff r5:60000010 r4:b6f0ead4
Mem-info:
Normal per-cpu:
CPU 0: hi: 42, btch: 7 usd: 0
CPU 1: hi: 42, btch: 7 usd: 0
HighMem per-cpu:
CPU 0: hi: 42, btch: 7 usd: 0
CPU 1: hi: 42, btch: 7 usd: 0
active_anon:7196 inactive_anon:7223 isolated_anon:0
active_file:1028 inactive_file:1457 isolated_file:0
unevictable:199 dirty:3 writeback:0 unstable:0
free:22844 slab_reclaimable:969 slab_unreclaimable:6547
mapped:685 shmem:0 pagetables:210 bounce:0
free_cma:0
Normal free:29092kB min:32768kB low:40960kB high:49152kB active_anon:0kB inactive_anon:0kB active_file:44kB inactive_file:128kB unevictable:44kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:124196kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:3876kB slab_unreclaimable:26188kB kernel_stack:576kB pagetables:840kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1024 1024
HighMem free:62252kB min:128kB low:8772kB high:17416kB active_anon:28784kB inactive_anon:28892kB active_file:4068kB inactive_file:5700kB unevictable:752kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:131072kB mlocked:0kB dirty:12kB writeback:0kB mapped:2740kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
Normal: 467*4kB (UEM) 285*8kB (UEM) 219*16kB (UEM) 93*32kB (UEM) 50*64kB (EM) 25*128kB (UEM) 11*256kB (UEM) 2*512kB (U) 0*1024kB 0*2048kB 2*4096kB (R) = 29060kB
HighMem: 5997*4kB (UM) 2743*8kB (UM) 478*16kB (UM) 27*32kB (UM) 2*64kB (M) 2*128kB (U) 1*256kB (U) 2*512kB (U) 0*1024kB 1*2048kB (U) 1*4096kB (R) = 62252kB
5489 total pagecache pages
2805 pages in swap cache
Swap cache stats: add 1237151, delete 1234346, find 539424/688368
Free swap = 484712kB
Total swap = 524284kB
65536 pages of RAM
23025 free pages
1661 reserved pages
7515 slab pages
819780 pages shared
2805 pages swap cached
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
And again, Free swap = 484712kB - huge amount not used. Why?
Solution 1:
Your problem actually lies here:
Normal
free:29372kB
min:32768kB
low:40960kB
high:49152kB
active_anon:0kB
inactive_anon:0kB
active_file:44kB
inactive_file:128kB
unevictable:44kB
isolated(anon):0kB
isolated(file):0kB
present:131072kB
managed:124196kB
mlocked:0kB
dirty:0kB
writeback:0kB
mapped:0kB
shmem:0kB
slab_reclaimable:3876kB
slab_unreclaimable:26196kB
kernel_stack:576kB
pagetables:840kB
unstable:0kB
bounce:0kB
free_cma:0kB
writeback_tmp:0kB
pages_scanned:1064
all_unreclaimable? no
The memory on your router is split up into zones, one zone -- HighMem contains 128MiB of memory, the other zone, Normal contains 128MiB of memory.
The zone are functionally independent of one another. The memory manager works in each zone in an isolated way.
oom-killer is invoked in the normal zone. You can practically eliminate 128Mib of memory when you are looking at your request.
Now to the actual anomaly:
free: 29372kB
min: 32768kB
The min field is the kernels interpretation of 'when I go below this threshold, I will have to do drastic things to reclaim memory -- like OOM-killer'.
Whats highly unusual about this field in your description is just how much is set as the min amount. This value typically floats at about 2% of your memory, in this case its 25% of the entire zone. Its no wonder you oom-kill so aggressively!
Lo and behlod when reviewing your question output the reason for the problem lays here:
vm.min_free_kbytes = 32768
Alter this value to be something way lower. Perhaps 1024 would be a more reasonable number.
You also shows some 'orphaned tcp socket' errors. You may also wish to reduce the values for net.ipv4.tcp_wmem and net.ipv4.tcp_rmem which will permit you handle more sockets (at the cost of latency).
So, to answer your question why are you killing processes with so much free swap - the answer to that is the vm.min_free_kb setting is ridiculously high for the amount of actual memory you have in each zone. Reduce this value from 32MiB to 1MiB.
Solution 2:
EDIT: This answer totally misses the problem. See Matthew Ife's post about the crazy high setting on vm.min_free_kbytes
The router and tor have been running for a while, correct?
I suspect you are being hit by external (memory) fragmentation. If you look in your OOM logs you can see that there are many chunks of available memory, but if none of them are as big as what tor is asking for, the malloc() will fail.
Normal: 465*4kB (UEM) 285*8kB (UEM) 219*16kB (UEM) 92*32kB (UEM) 51*64kB (UEM) 25*128kB (UEM) 12*256kB (UEM) 2*512kB (U) 0*1024kB 0*2048kB 2*4096kB (R) = 29340kB
HighMem: 5997*4kB (UM) 2743*8kB (UM) 478*16kB (UM) 27*32kB (UM) 2*64kB (M) 2*128kB (U) 1*256kB (U) 2*512kB (U) 0*1024kB 1*2048kB (U) 1*4096kB (R) = 62252kB
Edit: As Matthew points out, the offending malloc()s are asking for order-0 (single page) and order-1 (two page) allocations. I am clearly barking up the wrong tree.
There is a great explanation of what is happening over at elinux.org. Unfortunately the memory compaction which is mentioned in that presentation does not appear to be tunable in DD-WRT, though I can find changesets going back to 2010 which indicate it is in the kernel.
As to what you should do about tor I am less clear. Maybe you can run a watchdog for it, or opportunistically restart the process (which should help the kernel find space to clean up memory). It's also possible that having the swappiness set so high is slowing down the memory compaction process, but I'm speculating at this point.
Solution 3:
You have a couple of obvious problems.
First:
vm.swappiness = 100
You've told the kernel to swap as aggressively as possible, even when it's not strictly necessary to do so.
On an embedded device (which usually won't have swap at all) this is completely insane. This should be set to 0 (probably best) or 1.
Second:
vm.overcommit_memory = 2
Aside from preventing any overcommit at all, and preventing the system from using all free memory and swap, this also causes additional memory to be reserved for admin and user processes as specified in admin_reserve_kbytes and user_reserve_kbytes.
I can't find a single system I manage, embedded, desktop or server, where this setting is anything other than 0.
Finally, about DD-WRT... I'm not happy with them. They went overly commercial some time back and stopped making regular releases and updates. These days I recommend OpenWrt as a platform for building embedded systems.