I have followed slapd for several days now. The last few days slapd stopped a couple of time. Last day it worked for more than 24 hours without a problem.
attaching gdb to slapd: Attaching to process 3845 Reading symbols from /usr/sbin/slapd...Reading symbols from /usr/lib/debug/usr/sbin/slapd...done. (no debugging symbols found)...done. Reading symbols from /usr/lib/libldap_r-2.4.so.2...Reading symbols from /usr/lib/debug/usr/lib/libldap_r-2.4.so.2.5.6...done. (no debugging symbols found)...done. Loaded symbols for /usr/lib/libldap_r-2.4.so.2 Reading symbols from /usr/lib/liblber-2.4.so.2...Reading symbols from /usr/lib/debug/usr/lib/liblber-2.4.so.2.5.6...done. (no debugging symbols found)...done. Loaded symbols for /usr/lib/liblber-2.4.so.2 Reading symbols from /usr/lib/libdb-4.8.so...Reading symbols from /usr/lib/debug/usr/lib/libdb-4.8.so.debug...done. (no debugging symbols found)...done. Loaded symbols for /usr/lib/libdb-4.8.so Reading symbols from /usr/lib/libodbc.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libodbc.so.1 Reading symbols from /usr/lib/libslp.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libslp.so.1 Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libsasl2.so.2 Reading symbols from /usr/lib/libgnutls.so.26...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgnutls.so.26 Reading symbols from /lib/i686/cmov/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/i686/cmov/libcrypt.so.1 Reading symbols from /lib/i686/cmov/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/i686/cmov/libresolv.so.2 Reading symbols from /usr/lib/libltdl.so.7...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libltdl.so.7 Reading symbols from /lib/libwrap.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libwrap.so.0 Reading symbols from /lib/i686/cmov/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] [New Thread 0xb0cffb70 (LWP 7018)] [New Thread 0xb1bffb70 (LWP 4253)] [New Thread 0xb3cdcb70 (LWP 3850)] [New Thread 0xb40ddb70 (LWP 3849)] [New Thread 0xb44deb70 (LWP 3848)] [New Thread 0xb51e1b70 (LWP 3847)] [New Thread 0xb55e2b70 (LWP 3846)] Loaded symbols for /lib/i686/cmov/libpthread.so.0 Reading symbols from /lib/i686/cmov/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/i686/cmov/libc.so.6 Reading symbols from /lib/i686/cmov/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/i686/cmov/libnsl.so.1 Reading symbols from /lib/i686/cmov/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/i686/cmov/libdl.so.2 Reading symbols from /usr/lib/libtasn1.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libtasn1.so.3 Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /usr/lib/libgcrypt.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgcrypt.so.11 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /usr/lib/libgpg-error.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgpg-error.so.0 Reading symbols from /lib/i686/cmov/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/i686/cmov/libnss_files.so.2 Reading symbols from /lib/i686/cmov/libnss_compat.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/i686/cmov/libnss_compat.so.2 Reading symbols from /lib/i686/cmov/libnss_nis.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/i686/cmov/libnss_nis.so.2 Reading symbols from /lib/libnss_ldap.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_ldap.so.2 Reading symbols from /usr/lib/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.3 Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libcom_err.so.2 Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi_krb5.so.2 Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libk5crypto.so.3 Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5support.so.0 Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libkeyutils.so.1 Reading symbols from /usr/lib/sasl2/libsasldb.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/sasl2/libsasldb.so.2 Reading symbols from /usr/lib/sasl2/libcrammd5.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/sasl2/libcrammd5.so.2 Reading symbols from /usr/lib/sasl2/libanonymous.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/sasl2/libanonymous.so.2 Reading symbols from /usr/lib/sasl2/libdigestmd5.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/sasl2/libdigestmd5.so.2 Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.8 Reading symbols from /usr/lib/sasl2/libplain.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/sasl2/libplain.so.2 Reading symbols from /usr/lib/sasl2/liblogin.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/sasl2/liblogin.so.2 Reading symbols from /usr/lib/sasl2/libntlm.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/sasl2/libntlm.so.2 Reading symbols from /usr/lib/ldap/back_bdb-2.4.so.2...Reading symbols from /usr/lib/debug/usr/lib/ldap/back_bdb-2.4.so.2.5.6...done. (no debugging symbols found)...done. Loaded symbols for /usr/lib/ldap/back_bdb-2.4.so.2 Reading symbols from /usr/lib/ldap/syncprov-2.4.so.2...Reading symbols from /usr/lib/debug/usr/lib/ldap/syncprov-2.4.so.2.5.6...done. (no debugging symbols found)...done. Loaded symbols for /usr/lib/ldap/syncprov-2.4.so.2 0xb782b424 in __kernel_vsyscall ()
Program received signal SIGPIPE, Broken pipe.
Program received signal SIGPIPE, Broken pipe.
Program received signal SIGPIPE, Broken pipe.
.. few hundred time same messages ...
Program received signal SIGABRT, Aborted. [Thread 0xb55e2b70 (LWP 3846) exited] [Thread 0xafbffb70 (LWP 11193) exited] [Thread 0xb51e1b70 (LWP 3847) exited] [Thread 0xb44deb70 (LWP 3848) exited] [Thread 0xb40ddb70 (LWP 3849) exited] [Thread 0xb3cdcb70 (LWP 3850) exited] [Thread 0xb1bffb70 (LWP 4253) exited] [Thread 0xb0cffb70 (LWP 7018) exited]
Last messages in log-file Mar 3 05:17:38 ux-254 slapd[3845]: connection_read(45): no connection! Mar 3 05:17:38 ux-254 slapd[3845]: connection_read(45): no connection! Mar 3 05:17:54 ux-254 slapd[3845]: send_search_entry: conn 52805 ber write failed.
This is the first time I saw a SIGABRT. All other times nothing but SIGPIPE or sometimes a SIGTERM. As you can see slapd stops late in the night, everybody sleeping, hardly users working.
As I wrote earlier, I use a perl daemon wich checks the process: pid combined with processtable. If the process is gone in the processtable or in defunc or stop state it will restart slapd. This happens within 20 seconds. This way we manage to minimize the impact of the problems with slapd. Perhaps the SIGTERM is the result of the processmonitor. If that is the case it must be the result of a defunct or stop condition in the process table. But we can't work without this tools to restart slapd as quickly as possible.
Op 28-2-2011 14:30, Howard Chu schreef:
Ruud Baart wrote:
Thank you all for the fast answers.
I now attached gdb to slapd and get some results:
(gdb) handle all nostop (gdb) continue Continuing.
Program received signal SIGPIPE, Broken pipe.
Program received signal SIGPIPE, Broken pipe. [New Thread 0xb01f6b70 (LWP 1548)]
Program received signal SIGPIPE, Broken pipe. .. more of the same .. Program received signal SIGPIPE, Broken pipe.
Program received signal SIGPIPE, Broken pipe.
---Type<return> to continue, or q<return> to quit--- Program received signal SIGPIPE, Broken pipe.
Program received signal SIGTERM, Terminated.
This is not a crash or any error in slapd; some external command was used to kill the slapd process.
[New Thread 0xaf4f3b70 (LWP 1968)] [New Thread 0xaf0f2b70 (LWP 1969)] [New Thread 0xaecf1b70 (LWP 1970)] [New Thread 0xae8f0b70 (LWP 1971)] [Thread 0xaf4f3b70 (LWP 1968) exited] [Thread 0xb1bfcb70 (LWP 1474) exited] [Thread 0xb5268b70 (LWP 1462) exited] [Thread 0xaf0f2b70 (LWP 1969) exited] [Thread 0xb4565b70 (LWP 1463) exited] [Thread 0xb01f6b70 (LWP 1548) exited] [Thread 0xb5669b70 (LWP 1461) exited] [Thread 0xb17fbb70 (LWP 1475) exited] [Thread 0xae8f0b70 (LWP 1971) exited] [Thread 0xb28ffb70 (LWP 1464) exited] [Thread 0xaecf1b70 (LWP 1970) exited]
Program exited normally.