Hello,
I could update both systems during my vacation to 2.4.33. Both servers had 16GByte RAM. The system crashed again randomly (as expected). So we increased the memory on one server to 24 GByte RAM. No effect, this server crashes too, sometime till 5 times a day.
I installed a test machine (VM), but only with 7GByte RAM and could reproduce the crashes with one/two parallel searches. So I tried tcmalloc, but with the same result.
I digged a little deeper and the results are, that not the physical memory from the system is insufficient.
First - slapd tries to allocate memory more then available, but not use it. It is enough physical memory available during slapd crashes.
I ran the commands continuously with while and sleep 1: free -m, and pmap $PID_OF_LDAP and vmstat 1
The last memory information before the crash was:
pmap: -------------------------- 21954: slapd START SIZE RSS PSS DIRTY SWAP PERM MAPPING 00007f63f4904000 4K 0K 0K 0K 0K rw-p [anon] 00007f63f4905000 4K 0K 0K 0K 0K ---p [anon] 00007f63f4906000 16384K 2080K 2080K 2080K 0K rw-p [anon] 00007f63f5906000 4K 0K 0K 0K 0K ---p [anon] 00007f63f5907000 16384K 4120K 4120K 4120K 0K rw-p [anon] 00007f63f6907000 4K 0K 0K 0K 0K ---p [anon] 00007f63f6908000 16384K 4120K 4120K 4120K 0K rw-p [anon] 00007f63f7908000 4K 0K 0K 0K 0K ---p [anon] 00007f63f7909000 16384K 4104K 4104K 4104K 16K rw-p [anon] 00007f63f8909000 4K 0K 0K 0K 0K ---p [anon] 00007f63f890a000 16384K 8K 8K 8K 4K rw-p [anon] 00007f63f990a000 24K 8K 8K 8K 0K rw-s /SYSV0000000c 00007f63f9910000 544K 44K 44K 44K 0K rw-s /SYSV0000000b 00007f63f9998000 2304K 12K 12K 12K 0K rw-s /SYSV0000000a 00007f63f9bd8000 2097152K 2097052K 2097052K 2097052K 0K rw-s /SYSV00000009 00007f6479bd8000 78008K 60508K 60508K 60508K 0K rw-s /SYSV00000008 00007f647e806000 16K 0K 0K 0K 0K r-xp /usr/lib64/sasl2/libplain.so.2.0.22 00007f647e80a000 2044K 0K 0K 0K 0K ---p /usr/lib64/sasl2/libplain.so.2.0.22 00007f647ea09000 4K 0K 0K 0K 4K r--p /usr/lib64/sasl2/libplain.so.2.0.22 00007f647ea0a000 4K 0K 0K 0K 4K rw-p /usr/lib64/sasl2/libplain.so.2.0.22 00007f647ea0b000 16K 0K 0K 0K 0K r-xp /usr/lib64/sasl2/libanonymous.so.2.0.22 00007f647ea0f000 2044K 0K 0K 0K 0K ---p /usr/lib64/sasl2/libanonymous.so.2.0.22 00007f647ec0e000 4K 0K 0K 0K 4K r--p /usr/lib64/sasl2/libanonymous.so.2.0.22 00007f647ec0f000 4K 0K 0K 0K 4K rw-p /usr/lib64/sasl2/libanonymous.so.2.0.22 00007f647ec10000 20K 0K 0K 0K 0K r-xp /usr/lib64/sasl2/libsasldb.so.2.0.22 00007f647ec15000 2044K 0K 0K 0K 0K ---p /usr/lib64/sasl2/libsasldb.so.2.0.22 00007f647ee14000 4K 0K 0K 0K 4K r--p /usr/lib64/sasl2/libsasldb.so.2.0.22 00007f647ee15000 4K 0K 0K 0K 4K rw-p /usr/lib64/sasl2/libsasldb.so.2.0.22 00007f647ee16000 16K 0K 0K 0K 0K r-xp /usr/lib64/sasl2/liblogin.so.2.0.22 00007f647ee1a000 2044K 0K 0K 0K 0K ---p /usr/lib64/sasl2/liblogin.so.2.0.22 00007f647f019000 4K 0K 0K 0K 4K r--p /usr/lib64/sasl2/liblogin.so.2.0.22 00007f647f01a000 4K 0K 0K 0K 4K rw-p /usr/lib64/sasl2/liblogin.so.2.0.22 00007f647f01b000 8K 0K 0K 0K 0K r-xp /lib64/libkeyutils-1.2.so 00007f647f01d000 2044K 0K 0K 0K 0K ---p /lib64/libkeyutils-1.2.so 00007f647f21c000 4K 0K 0K 0K 4K r--p /lib64/libkeyutils-1.2.so 00007f647f21d000 4K 0K 0K 0K 4K rw-p /lib64/libkeyutils-1.2.so 00007f647f21e000 28K 0K 0K 0K 0K r-xp /usr/lib64/libkrb5support.so.0.1 00007f647f225000 2048K 0K 0K 0K 0K ---p /usr/lib64/libkrb5support.so.0.1 00007f647f425000 4K 0K 0K 0K 4K r--p /usr/lib64/libkrb5support.so.0.1 00007f647f426000 4K 0K 0K 0K 4K rw-p /usr/lib64/libkrb5support.so.0.1 00007f647f427000 12K 0K 0K 0K 0K r-xp /lib64/libcom_err.so.2.1 00007f647f42a000 2044K 0K 0K 0K 0K ---p /lib64/libcom_err.so.2.1 00007f647f629000 4K 0K 0K 0K 4K r--p /lib64/libcom_err.so.2.1 00007f647f62a000 4K 0K 0K 0K 4K rw-p /lib64/libcom_err.so.2.1 00007f647f62b000 144K 0K 0K 0K 0K r-xp /usr/lib64/libk5crypto.so.3.1 00007f647f64f000 2044K 0K 0K 0K 0K ---p /usr/lib64/libk5crypto.so.3.1 00007f647f84e000 8K 0K 0K 0K 8K r--p /usr/lib64/libk5crypto.so.3.1 00007f647f850000 4K 0K 0K 0K 4K rw-p /usr/lib64/libk5crypto.so.3.1 00007f647f851000 632K 0K 0K 0K 0K r-xp /usr/lib64/libkrb5.so.3.3 00007f647f8ef000 2044K 0K 0K 0K 0K ---p /usr/lib64/libkrb5.so.3.3 00007f647faee000 12K 0K 0K 0K 12K r--p /usr/lib64/libkrb5.so.3.3 00007f647faf1000 8K 0K 0K 0K 8K rw-p /usr/lib64/libkrb5.so.3.3 00007f647faf3000 172K 0K 0K 0K 0K r-xp /usr/lib64/libgssapi_krb5.so.2.2 00007f647fb1e000 2048K 0K 0K 0K 0K ---p /usr/lib64/libgssapi_krb5.so.2.2 00007f647fd1e000 4K 0K 0K 0K 4K r--p /usr/lib64/libgssapi_krb5.so.2.2 00007f647fd1f000 4K 0K 0K 0K 4K rw-p /usr/lib64/libgssapi_krb5.so.2.2 00007f647fd20000 28K 0K 0K 0K 0K r-xp /usr/lib64/sasl2/libgssapiv2.so.2.0.22 00007f647fd27000 2044K 0K 0K 0K 0K ---p /usr/lib64/sasl2/libgssapiv2.so.2.0.22 00007f647ff26000 4K 0K 0K 0K 4K r--p /usr/lib64/sasl2/libgssapiv2.so.2.0.22 00007f647ff27000 4K 0K 0K 0K 4K rw-p /usr/lib64/sasl2/libgssapiv2.so.2.0.22 00007f647ff28000 48K 0K 0K 0K 0K r-xp /usr/lib64/sasl2/libdigestmd5.so.2.0.22 00007f647ff34000 2044K 0K 0K 0K 0K ---p /usr/lib64/sasl2/libdigestmd5.so.2.0.22 00007f6480133000 4K 0K 0K 0K 4K r--p /usr/lib64/sasl2/libdigestmd5.so.2.0.22 00007f6480134000 4K 0K 0K 0K 4K rw-p /usr/lib64/sasl2/libdigestmd5.so.2.0.22 00007f6480135000 84K 0K 0K 0K 0K r-xp /lib64/libz.so.1.2.3 00007f648014a000 2044K 0K 0K 0K 0K ---p /lib64/libz.so.1.2.3 00007f6480349000 4K 0K 0K 0K 4K r--p /lib64/libz.so.1.2.3 00007f648034a000 4K 0K 0K 0K 4K rw-p /lib64/libz.so.1.2.3 00007f648034b000 84K 0K 0K 0K 0K r-xp /lib64/libgcc_s.so.1 00007f6480360000 2044K 0K 0K 0K 0K ---p /lib64/libgcc_s.so.1 00007f648055f000 4K 0K 0K 0K 4K r--p /lib64/libgcc_s.so.1 00007f6480560000 4K 0K 0K 0K 4K rw-p /lib64/libgcc_s.so.1 00007f6480561000 940K 0K 0K 0K 0K r-xp /usr/lib64/libstdc++.so.6.0.16 00007f648064c000 2048K 0K 0K 0K 0K ---p /usr/lib64/libstdc++.so.6.0.16 00007f648084c000 32K 0K 0K 0K 32K r--p /usr/lib64/libstdc++.so.6.0.16 00007f6480854000 8K 0K 0K 0K 8K rw-p /usr/lib64/libstdc++.so.6.0.16 00007f6480856000 84K 0K 0K 0K 4K rw-p [anon] 00007f648086b000 36K 28K 28K 0K 0K r-xp /lib64/libunwind.so.7.0.0 00007f6480874000 2044K 0K 0K 0K 0K ---p /lib64/libunwind.so.7.0.0 00007f6480a73000 4K 4K 4K 4K 0K r--p /lib64/libunwind.so.7.0.0 00007f6480a74000 4K 4K 4K 4K 0K rw-p /lib64/libunwind.so.7.0.0 00007f6480a75000 16K 4K 4K 4K 0K rw-p [anon] 00007f6480a79000 1460K 444K 38K 0K 0K r-xp /lib64/libc-2.11.3.so 00007f6480be6000 2048K 0K 0K 0K 0K ---p /lib64/libc-2.11.3.so 00007f6480de6000 16K 16K 16K 8K 0K r--p /lib64/libc-2.11.3.so 00007f6480dea000 4K 4K 4K 4K 0K rw-p /lib64/libc-2.11.3.so 00007f6480deb000 20K 16K 16K 16K 4K rw-p [anon] 00007f6480df0000 36K 28K 10K 0K 0K r-xp /lib64/libwrap.so.0.7.6 00007f6480df9000 2044K 0K 0K 0K 0K ---p /lib64/libwrap.so.0.7.6 00007f6480ff8000 4K 4K 4K 0K 0K r--p /lib64/libwrap.so.0.7.6 00007f6480ff9000 4K 4K 4K 4K 0K rw-p /lib64/libwrap.so.0.7.6 00007f6480ffa000 4K 4K 4K 4K 0K rw-p [anon] 00007f6480ffb000 76K 4K 0K 0K 0K r-xp /lib64/libresolv-2.11.3.so 00007f648100e000 2048K 0K 0K 0K 0K ---p /lib64/libresolv-2.11.3.so 00007f648120e000 4K 0K 0K 0K 4K r--p /lib64/libresolv-2.11.3.so 00007f648120f000 4K 0K 0K 0K 4K rw-p /lib64/libresolv-2.11.3.so 00007f6481210000 8K 0K 0K 0K 0K rw-p [anon] 00007f6481212000 48K 4K 0K 0K 0K r-xp /lib64/libcrypt-2.11.3.so 00007f648121e000 2044K 0K 0K 0K 0K ---p /lib64/libcrypt-2.11.3.so 00007f648141d000 4K 0K 0K 0K 4K r--p /lib64/libcrypt-2.11.3.so 00007f648141e000 4K 0K 0K 0K 4K rw-p /lib64/libcrypt-2.11.3.so 00007f648141f000 184K 0K 0K 0K 0K rw-p [anon] 00007f648144d000 1484K 4K 4K 0K 0K r-xp /usr/lib64/libcrypto.so.0.9.8 00007f64815c0000 2044K 0K 0K 0K 0K ---p /usr/lib64/libcrypto.so.0.9.8 00007f64817bf000 64K 0K 0K 0K 64K r--p /usr/lib64/libcrypto.so.0.9.8 00007f64817cf000 100K 0K 0K 0K 100K rw-p /usr/lib64/libcrypto.so.0.9.8 00007f64817e8000 16K 0K 0K 0K 12K rw-p [anon] 00007f64817ec000 308K 4K 4K 0K 0K r-xp /usr/lib64/libssl.so.0.9.8 00007f6481839000 2044K 0K 0K 0K 0K ---p /usr/lib64/libssl.so.0.9.8 00007f6481a38000 8K 0K 0K 0K 8K r--p /usr/lib64/libssl.so.0.9.8 00007f6481a3a000 24K 0K 0K 0K 24K rw-p /usr/lib64/libssl.so.0.9.8 00007f6481a40000 8K 4K 0K 0K 0K r-xp /lib64/libdl-2.11.3.so 00007f6481a42000 2048K 0K 0K 0K 0K ---p /lib64/libdl-2.11.3.so 00007f6481c42000 4K 0K 0K 0K 4K r--p /lib64/libdl-2.11.3.so 00007f6481c43000 4K 0K 0K 0K 4K rw-p /lib64/libdl-2.11.3.so 00007f6481c44000 100K 44K 36K 0K 0K r-xp /usr/lib64/libsasl2.so.2.0.22 00007f6481c5d000 2048K 0K 0K 0K 0K ---p /usr/lib64/libsasl2.so.2.0.22 00007f6481e5d000 4K 4K 4K 4K 0K r--p /usr/lib64/libsasl2.so.2.0.22 00007f6481e5e000 4K 4K 4K 4K 0K rw-p /usr/lib64/libsasl2.so.2.0.22 00007f6481e5f000 84K 4K 0K 0K 0K r-xp /lib64/libnsl-2.11.3.so 00007f6481e74000 2044K 0K 0K 0K 0K ---p /lib64/libnsl-2.11.3.so 00007f6482073000 4K 0K 0K 0K 4K r--p /lib64/libnsl-2.11.3.so 00007f6482074000 4K 0K 0K 0K 4K rw-p /lib64/libnsl-2.11.3.so 00007f6482075000 8K 0K 0K 0K 0K rw-p [anon] 00007f6482077000 364K 4K 0K 0K 0K r-xp /lib64/libm-2.11.3.so 00007f64820d2000 2044K 0K 0K 0K 0K ---p /lib64/libm-2.11.3.so 00007f64822d1000 4K 0K 0K 0K 4K r--p /lib64/libm-2.11.3.so 00007f64822d2000 120K 0K 0K 0K 8K rw-p /lib64/libm-2.11.3.so 00007f64822f0000 96K 4K 4K 0K 0K r-xp /usr/lib64/libslp.so.1.0.0 00007f6482308000 2044K 0K 0K 0K 0K ---p /usr/lib64/libslp.so.1.0.0 00007f6482507000 4K 0K 0K 0K 4K r--p /usr/lib64/libslp.so.1.0.0 00007f6482508000 4K 0K 0K 0K 4K rw-p /usr/lib64/libslp.so.1.0.0 00007f6482509000 92K 76K 26K 0K 0K r-xp /lib64/libpthread-2.11.3.so 00007f6482520000 2048K 0K 0K 0K 0K ---p /lib64/libpthread-2.11.3.so 00007f6482720000 4K 4K 4K 4K 0K r--p /lib64/libpthread-2.11.3.so 00007f6482721000 4K 4K 4K 4K 0K rw-p /lib64/libpthread-2.11.3.so 00007f6482722000 16K 4K 4K 4K 0K rw-p [anon] 00007f6482726000 400K 4K 4K 0K 0K r-xp /usr/lib64/libodbc.so.1.0.0 00007f648278a000 2044K 0K 0K 0K 0K ---p /usr/lib64/libodbc.so.1.0.0 00007f6482989000 4K 0K 0K 0K 4K r--p /usr/lib64/libodbc.so.1.0.0 00007f648298a000 28K 0K 0K 0K 28K rw-p /usr/lib64/libodbc.so.1.0.0 00007f6482991000 4K 0K 0K 0K 0K rw-p [anon] 00007f6482992000 1224K 196K 187K 0K 0K r-xp /usr/lib64/libdb-4.5.so 00007f6482ac4000 2048K 0K 0K 0K 0K ---p /usr/lib64/libdb-4.5.so 00007f6482cc4000 12K 4K 4K 4K 8K r--p /usr/lib64/libdb-4.5.so 00007f6482cc7000 12K 12K 12K 12K 0K rw-p /usr/lib64/libdb-4.5.so 00007f6482cca000 36K 4K 0K 0K 0K r-xp /usr/lib64/libltdl.so.7.2.0 00007f6482cd3000 2044K 0K 0K 0K 0K ---p /usr/lib64/libltdl.so.7.2.0 00007f6482ed2000 4K 0K 0K 0K 4K r--p /usr/lib64/libltdl.so.7.2.0 00007f6482ed3000 4K 0K 0K 0K 4K rw-p /usr/lib64/libltdl.so.7.2.0 00007f6482ed4000 56K 44K 14K 0K 0K r-xp /usr/lib64/liblber-2.4.so.2.8.5 00007f6482ee2000 2044K 0K 0K 0K 0K ---p /usr/lib64/liblber-2.4.so.2.8.5 00007f64830e1000 4K 4K 4K 4K 0K r--p /usr/lib64/liblber-2.4.so.2.8.5 00007f64830e2000 4K 4K 4K 4K 0K rw-p /usr/lib64/liblber-2.4.so.2.8.5 00007f64830e3000 308K 64K 64K 0K 0K r-xp /usr/lib64/libldap_r-2.4.so.2.8.5 00007f6483130000 2048K 0K 0K 0K 0K ---p /usr/lib64/libldap_r-2.4.so.2.8.5 00007f6483330000 4K 4K 4K 4K 0K r--p /usr/lib64/libldap_r-2.4.so.2.8.5 00007f6483331000 8K 8K 8K 8K 0K rw-p /usr/lib64/libldap_r-2.4.so.2.8.5 00007f6483333000 12K 8K 8K 8K 4K rw-p [anon] 00007f6483336000 288K 96K 96K 0K 0K r-xp /usr/local/lib/libtcmalloc.so.4.1.0 00007f648337e000 2044K 0K 0K 0K 0K ---p /usr/local/lib/libtcmalloc.so.4.1.0 00007f648357d000 4K 4K 4K 4K 0K r--p /usr/local/lib/libtcmalloc.so.4.1.0 00007f648357e000 4K 4K 4K 4K 0K rw-p /usr/local/lib/libtcmalloc.so.4.1.0 00007f648357f000 156K 92K 92K 88K 36K rw-p [anon] 00007f64835a6000 124K 40K 0K 0K 0K r-xp /lib64/ld-2.11.3.so 00007f6483708000 212K 0K 0K 0K 0K r--s /var/run/nscd/group 00007f648373d000 212K 0K 0K 0K 0K r--s /var/run/nscd/passwd 00007f6483772000 212K 0K 0K 0K 0K r--s /var/run/nscd/dbvbS4oY 00007f64837a7000 48K 28K 28K 24K 20K rw-p [anon] 00007f64837bc000 24K 8K 8K 8K 0K rw-s /SYSV00000007 00007f64837c2000 8K 4K 4K 4K 4K rw-p [anon] 00007f64837c4000 4K 4K 4K 4K 0K r--p /lib64/ld-2.11.3.so 00007f64837c5000 4K 4K 4K 4K 0K rw-p /lib64/ld-2.11.3.so 00007f64837c6000 4K 4K 4K 4K 0K rw-p [anon] 00007f64837c7000 1944K 456K 456K 0K 0K r-xp /usr/lib/openldap/slapd 00007f6483bac000 20K 8K 8K 8K 12K r--p /usr/lib/openldap/slapd 00007f6483bb1000 48K 24K 24K 24K 24K rw-p /usr/lib/openldap/slapd 00007f6483bbd000 3812K 1052K 1052K 984K 1788K rw-p [heap] 00007f6483f76000 5004408K 4574484K 4574484K 4574392K 257508K rw-p [heap] 00007fff9f532000 132K 4K 4K 4K 32K rw-p [stack] 00007fff9f5b6000 4K 4K 0K 0K 0K r-xp [vdso] ffffffffff600000 4K 0K 0K 0K 0K r-xp [vsyscall] Total: 7350688K 6749464K 6748875K 6747724K 259932K
5091328K writable-private, 80668K readonly-private, 2178692K shared, and 4900700K referenced
================================================================================
last free -m before crash: ------------------------------- total used free shared buffers cached Mem: 6928 6813 115 0 0 2180 -/+ buffers/cache: 4632 2296 Swap: 2047 288 1759
last free after crash: -------------------------------
total used free shared buffers cached Mem: 6928 4978 1950 0 0 2166 -/+ buffers/cache: 2810 4117 Swap: 2047 42 2005 ================================================================================
and vmstat: ------------------------------- procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 1 4 295284 110148 540 2255804 108 0 13428 0 796 1373 3 5 27 65 0 0 3 295260 118140 608 2242804 36 0 9308 0 2227 1309 2 5 23 70 0 0 3 295224 116668 680 2237396 48 0 4008 0 1253 2002 3 6 52 39 0 1 0 295164 116296 736 2232588 128 0 6028 0 1083 954 3 14 45 39 0 0 0 42332 4742716 768 2219044 784 0 1104 0 1428 1178 3 23 63 11 0 0 0 42332 4743020 768 2219168 0 0 0 0 253 97 0 0 100 0 0 0 0 42332 4743408 768 2219160 0 0 0 0 201 98 0 0 100 0 0 0 1 42300 4743408 768 2219140 216 0 216 0 295 150 0 0 99 1 0
What I see, that slapd had reserved "Total: 7350688K" (overcommitted?), but only referenced 4900700K. Why does slapd reserve so much memory and use it not? Because of this, I changed the default values for memory overcomittment
from: vm.overcommit_memory = 0 (guess, default) vm.overcommit_ratio = 50
to: vm.overcommit_memory=2 (strict overcommit) vm.overcommit_ratio=100
but it crashed again (ch_malloc of 34545 bytes failed): . . 00007f2bf8e1a000 5019920K 4597460K 4597460K 4597116K 333472K rw-p [heap] 00007fffe86c5000 132K 0K 0K 0K 36K rw-p [stack] 00007fffe87af000 4K 4K 0K 0K 0K r-xp [vdso] ffffffffff600000 4K 0K 0K 0K 0K r-xp [vsyscall] Total: 7333420K 6766996K 6766484K 6765536K 336192K
5074068K writable-private, 80660K readonly-private, 2178692K shared, and 4681876K referenced
So in spite of increase the memory from 16GByte to 24 GByte (one of the production server - configuration described in the first mail) same results.
Is there anywhere a memory leak - how can I find out this?
Thanks Meike
P.S., mdb is not the option now, we need time to evaluate this ...