Hello, openldap-technical@ readers.
I'm running openSUSE 11.4 with openldap 2.4.23 and libdb-4_8 (berkeleydb) in vmware environment. Last friday my slapd was killed by oom killer. Seems that it consumed all available memory. It is the first time i see that openldap was killed.
My database is rather small (15953 entries, 22 mb on disk id2entry.bdb + dn2id.bdb).
Could somebody point me to place where i can limit memory usage in openldap?
Here is part of log: Jan 20 18:23:42 ldap kernel: [5636879.604870] slapd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Jan 20 18:23:42 ldap kernel: [5636879.604894] slapd cpuset=/ mems_allowed=0 Jan 20 18:23:42 ldap kernel: [5636879.604905] Pid: 11984, comm: slapd Not tainted 2.6.37.6-0.5-desktop #1 Jan 20 18:23:42 ldap kernel: [5636879.604907] Call Trace: Jan 20 18:23:42 ldap kernel: [5636879.605070] [<ffffffff810059b9>] dump_trace+0x79/0x340 Jan 20 18:23:42 ldap kernel: [5636879.605144] [<ffffffff81521752>] dump_stack+0x69/0x6f Jan 20 18:23:42 ldap kernel: [5636879.605207] [<ffffffff810fd4e2>] dump_header+0xa2/0x240 Jan 20 18:23:42 ldap kernel: [5636879.605234] [<ffffffff810fdbe4>] oom_kill_process+0xa4/0x1d0 Jan 20 18:23:42 ldap kernel: [5636879.605246] [<ffffffff810fe03c>] out_of_memory+0x10c/0x250 Jan 20 18:23:42 ldap kernel: [5636879.605251] [<ffffffff811030bb>] __alloc_pages_nodemask+0x6fb/0x710 Jan 20 18:23:42 ldap kernel: [5636879.605281] [<ffffffff81139089>] alloc_pages_current+0x99/0x110 Jan 20 18:23:42 ldap kernel: [5636879.605302] [<ffffffff81105bb1>] __do_page_cache_readahead+0xe1/0x230 Jan 20 18:23:42 ldap kernel: [5636879.605313] [<ffffffff8110603c>] ra_submit+0x1c/0x30 Jan 20 18:23:42 ldap kernel: [5636879.605317] [<ffffffff810fc937>] filemap_fault+0x487/0x4a0 Jan 20 18:23:42 ldap kernel: [5636879.605337] [<ffffffff8111c202>] __do_fault+0x52/0x560 Jan 20 18:23:42 ldap kernel: [5636879.605349] [<ffffffff81120cd1>] handle_mm_fault+0x1a1/0x410 Jan 20 18:23:42 ldap kernel: [5636879.605369] [<ffffffff81527dd9>] do_page_fault+0x1a9/0x520 Jan 20 18:23:42 ldap kernel: [5636879.605382] [<ffffffff8152520f>] page_fault+0x1f/0x30 Jan 20 18:23:42 ldap kernel: [5636879.605481] [<00007ff68a86fffe>] 0x7ff68a86fffe Jan 20 18:23:42 ldap kernel: [5636879.605483] Mem-Info: Jan 20 18:23:42 ldap kernel: [5636879.605485] Node 0 DMA per-cpu: Jan 20 18:23:42 ldap kernel: [5636879.605490] CPU 0: hi: 0, btch: 1 usd: 0 Jan 20 18:23:42 ldap kernel: [5636879.605491] Node 0 DMA32 per-cpu: Jan 20 18:23:42 ldap kernel: [5636879.605493] CPU 0: hi: 186, btch: 31 usd: 31 Jan 20 18:23:42 ldap kernel: [5636879.605498] active_anon:55520 inactive_anon:55653 isolated_anon:0 Jan 20 18:23:42 ldap kernel: [5636879.605499] active_file:75 inactive_file:114 isolated_file:96 Jan 20 18:23:42 ldap kernel: [5636879.605500] unevictable:0 dirty:18 writeback:10 unstable:0 Jan 20 18:23:42 ldap kernel: [5636879.605500] free:1190 slab_reclaimable:1125 slab_unreclaimable:3921 Jan 20 18:23:42 ldap kernel: [5636879.605501] mapped:59 shmem:8 pagetables:4707 bounce:0 Jan 20 18:23:42 ldap kernel: [5636879.605503] Node 0 DMA free:2044kB min:84kB low:104kB high:124kB active_anon:6072kB inactive_anon:6424kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15684kB mlocked:0kB dirty:4kB writeback:0kB mapped:8kB shmem:0kB slab_reclaimable:28kB slab_unreclaimable:588kB kernel_stack:360kB pagetables:268kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1 all_unreclaimable? yes Jan 20 18:23:42 ldap kernel: [5636879.605515] lowmem_reserve[]: 0 489 489 489 Jan 20 18:23:42 ldap kernel: [5636879.605518] Node 0 DMA32 free:2716kB min:2784kB low:3480kB high:4176kB active_anon:216008kB inactive_anon:216188kB active_file:300kB inactive_file:456kB unevictable:0kB isolated(anon):0kB isolated(file):384kB present:500896kB mlocked:0kB dirty:68kB writeback:40kB mapped:228kB shmem:32kB slab_reclaimable:4472kB slab_unreclaimable:15096kB kernel_stack:1032kB pagetables:18560kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1154 all_unreclaimable? yes Jan 20 18:23:42 ldap kernel: [5636879.605527] lowmem_reserve[]: 0 0 0 0 Jan 20 18:23:42 ldap kernel: [5636879.605529] Node 0 DMA: 77*4kB 3*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2044kB Jan 20 18:23:42 ldap kernel: [5636879.605542] Node 0 DMA32: 677*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2716kB Jan 20 18:23:42 ldap kernel: [5636879.605550] 20332 total pagecache pages Jan 20 18:23:42 ldap kernel: [5636879.605551] 20056 pages in swap cache Jan 20 18:23:42 ldap kernel: [5636879.605553] Swap cache stats: add 681283, delete 661227, find 13544302/13638161 Jan 20 18:23:42 ldap kernel: [5636879.605555] Free swap = 0kB Jan 20 18:23:42 ldap kernel: [5636879.605556] Total swap = 262140kB Jan 20 18:23:42 ldap kernel: [5636879.608604] 131056 pages RAM Jan 20 18:23:42 ldap kernel: [5636879.608607] 5288 pages reserved Jan 20 18:23:42 ldap kernel: [5636879.608609] 33759 pages shared Jan 20 18:23:42 ldap kernel: [5636879.608610] 122516 pages non-shared Jan 20 18:23:42 ldap kernel: [5636879.608612] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Jan 20 18:23:42 ldap kernel: [5636879.608627] [ 320] 0 320 5295 1 0 -17 -1000 udevd Jan 20 18:23:42 ldap kernel: [5636879.608634] [ 816] 100 816 6904 1 0 0 0 dbus-daemon Jan 20 18:23:42 ldap kernel: [5636879.608638] [ 1902] 0 1902 60161 1038 0 0 0 radiusd Jan 20 18:23:42 ldap kernel: [5636879.608641] [ 1949] 0 1949 5142 28 0 0 0 master Jan 20 18:23:42 ldap kernel: [5636879.608644] [ 1971] 51 1971 5225 26 0 0 0 qmgr Jan 20 18:23:42 ldap kernel: [5636879.608648] [ 2030] 0 2030 3871 21 0 0 0 cron Jan 20 18:23:42 ldap kernel: [5636879.608651] [ 2034] 0 2034 4179 16 0 0 0 rpcbind Jan 20 18:23:42 ldap kernel: [5636879.608654] [ 2044] 0 2044 20429 76 0 0 0 vmtoolsd Jan 20 18:23:42 ldap kernel: [5636879.608658] [ 2045] 0 2045 992 0 0 0 0 startpar Jan 20 18:23:42 ldap kernel: [5636879.608661] [ 2049] 0 2049 990 2 0 0 0 mingetty Jan 20 18:23:42 ldap kernel: [5636879.608664] [ 2050] 0 2050 990 2 0 0 0 mingetty Jan 20 18:23:42 ldap kernel: [5636879.608667] [ 2051] 0 2051 990 2 0 0 0 mingetty Jan 20 18:23:42 ldap kernel: [5636879.608670] [ 2052] 0 2052 990 2 0 0 0 mingetty Jan 20 18:23:42 ldap kernel: [5636879.608674] [ 2053] 0 2053 990 2 0 0 0 mingetty Jan 20 18:23:42 ldap kernel: [5636879.608677] [ 2054] 0 2054 990 2 0 0 0 mingetty Jan 20 18:23:42 ldap kernel: [5636879.608689] [24805] 74 24805 3944 32 0 0 0 ntpd Jan 20 18:23:42 ldap kernel: [5636879.608692] [24855] 0 24855 11339 27 0 -17 -1000 sshd Jan 20 18:23:42 ldap kernel: [5636879.608695] [28273] 0 28273 29976 90 0 0 0 rsyslogd Jan 20 18:23:42 ldap kernel: [5636879.608699] [ 5665] 105 5665 4121 9 0 0 0 nrpe Jan 20 18:23:42 ldap kernel: [5636879.608702] [29623] 0 29623 2844 0 0 0 0 mysqld_safe Jan 20 18:23:42 ldap kernel: [5636879.608706] [29744] 60 29744 55304 999 0 0 0 mysqld Jan 20 18:23:42 ldap kernel: [5636879.608709] [30102] 0 30102 5294 2 0 -17 -1000 udevd Jan 20 18:23:42 ldap kernel: [5636879.608713] [30103] 0 30103 5294 1 0 -17 -1000 udevd Jan 20 18:23:42 ldap kernel: [5636879.608717] [30788] 0 30788 4127 1 0 0 0 automount Jan 20 18:23:42 ldap kernel: [5636879.608721] [31053] 0 31053 31139 6849 0 0 0 puppetd Jan 20 18:23:42 ldap kernel: [5636879.608724] [10627] 0 10627 63600 828 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608727] [23310] 76 23310 300113 76212 0 0 0 slapd Jan 20 18:23:42 ldap kernel: [5636879.608737] [32555] 51 32555 13882 166 0 0 0 pickup Jan 20 18:23:42 ldap kernel: [5636879.608742] [ 1137] 30 1137 67563 1005 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608745] [ 1156] 30 1156 67563 1004 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608748] [ 1158] 30 1158 67563 1004 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608752] [ 1160] 30 1160 67563 1005 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608755] [ 1161] 30 1161 67563 1005 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608758] [ 1164] 30 1164 67563 1005 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608761] [ 1166] 30 1166 67563 1005 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608765] [ 1169] 30 1169 67563 1005 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608769] [ 1171] 30 1171 67563 1005 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608777] [ 1172] 30 1172 67563 1003 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608780] [ 1175] 30 1175 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608786] [ 1181] 30 1181 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608789] [ 1183] 30 1183 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608792] [ 1188] 30 1188 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608795] [ 1189] 30 1189 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608798] [ 1190] 30 1190 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608802] [ 1191] 30 1191 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608805] [ 1192] 30 1192 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608808] [ 1193] 30 1193 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608811] [ 1194] 30 1194 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608814] [ 1195] 30 1195 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608822] [ 1196] 30 1196 67563 1001 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608825] [ 1197] 30 1197 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608829] [ 1198] 30 1198 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608832] [ 1199] 30 1199 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608835] [ 1200] 30 1200 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608838] [ 1201] 30 1201 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608841] [ 1202] 30 1202 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608844] [ 1203] 30 1203 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608847] [ 1204] 30 1204 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608851] [ 1205] 30 1205 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608859] [ 1206] 30 1206 67563 1003 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608862] [ 1207] 30 1207 67563 999 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608865] [ 1208] 30 1208 57304 1069 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608868] [ 1209] 30 1209 67563 963 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608872] [ 1210] 30 1210 67563 962 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608875] [ 1211] 30 1211 67563 963 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608878] [ 1212] 30 1212 67563 963 0 0 0 httpd2-prefork Jan 20 18:23:42 ldap kernel: [5636879.608906] Out of memory: Kill process 23310 (slapd) score 685 or sacrifice child Jan 20 18:23:42 ldap kernel: [5636879.608912] Killed process 23310 (slapd) total-vm:1200452kB, anon-rss:304848kB, file-rss:0kB Jan 20 18:23:42 ldap kernel: [5636879.645240] httpd2-prefork invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Jan 20 18:23:42 ldap kernel: [5636879.645250] httpd2-prefork cpuset=/ mems_allowed=0 Jan 20 18:23:42 ldap kernel: [5636879.645254] Pid: 1206, comm: httpd2-prefork Not tainted 2.6.37.6-0.5-desktop #1 Jan 20 18:23:42 ldap kernel: [5636879.645256] Call Trace: Jan 20 18:23:42 ldap kernel: [5636879.645279] [<ffffffff810059b9>] dump_trace+0x79/0x340 Jan 20 18:23:42 ldap kernel: [5636879.645285] [<ffffffff81521752>] dump_stack+0x69/0x6f Jan 20 18:23:42 ldap kernel: [5636879.645291] [<ffffffff810fd4e2>] dump_header+0xa2/0x240 Jan 20 18:23:42 ldap kernel: [5636879.645296] [<ffffffff810fdbe4>] oom_kill_process+0xa4/0x1d0 Jan 20 18:23:42 ldap kernel: [5636879.645301] [<ffffffff810fe03c>] out_of_memory+0x10c/0x250 Jan 20 18:23:42 ldap kernel: [5636879.645305] [<ffffffff811030bb>] __alloc_pages_nodemask+0x6fb/0x710
--On Monday, January 23, 2012 12:47 PM +0400 Igor Zinovik zinovik.igor@gmail.com wrote:
Hello, openldap-technical@ readers.
I'm running openSUSE 11.4 with openldap 2.4.23 and libdb-4_8 (berkeleydb) in vmware environment. Last friday my slapd was killed by oom killer. Seems that it consumed all available memory. It is the first time i see that openldap was killed.
My database is rather small (15953 entries, 22 mb on disk id2entry.bdb
- dn2id.bdb).
The OOM killers function by killing the *next* process to allocate memory. It does not mean that slapd consumed all available memory.
In any case, as has been noted a substantial number of times on this list:
a) Upgrade to a recent release (say 2.4.29) b) Use an alternative memory allocated to glibc (for example, tcmalloc from the google perf tools suite, or hoard)
Glibc is extremely inefficient in managing memory for OpenLDAP, which only leads to substantial problems over time.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
b) Use an alternative memory allocated to glibc (for example, tcmalloc from the google perf tools suite, or hoard)
Glibc is extremely inefficient in managing memory for OpenLDAP, which only leads to substantial problems over time.
I guess this should also be not necessary anymore with back-mdb.
Ciao, Michael.
Michael Ströder wrote:
Quanah Gibson-Mount wrote:
b) Use an alternative memory allocated to glibc (for example, tcmalloc from the google perf tools suite, or hoard)
Glibc is extremely inefficient in managing memory for OpenLDAP, which only leads to substantial problems over time.
I guess this should also be not necessary anymore with back-mdb.
It should take many orders of magnitude longer for the problem to manifest, but even though the mdb backend is doing few/zero mallocs, the frontend still has malloc activity of its own.
openldap-technical@openldap.org