hello all,
I upgraded to OpenLDAP 2.4.43 three weeks ago. It runs on debian squeeze 64bits lts it is built from sources with the fix from ITS#8330
Since the upgrade, it keeps crashing regularly (I would say once every 2 to 5 days) However, there isn't anything in the logs: no message, no error, no nothing
slapd no longer runs, I restart it, and it runs ok until the next crash
now I don't know how to debug that. I could downgrade to the previous version (2.4.41) which worked fine, but as our university is closed until 01/11, this is the perfect time to investigate. I still don't know if it's a bug in 2.4.43 or a misconfiguration for which previous versions was forgiving
What are your advice to track down the cause for the crash?
PS: here is a bit of context: one master with 2 mdb databases: one accesslog, one for the data two slaves (I mean complete replication, even cn=config) the master has crashed only once the slaves keep crashing regularly the slaves are complete replicas of the master, and I wonder if it could be a source of problems: I'll give details in another thread
Am Mon, 28 Dec 2015 10:31:35 +0400 schrieb Jephte Clain jephte.clain@univ-reunion.fr:
hello all,
I upgraded to OpenLDAP 2.4.43 three weeks ago. It runs on debian squeeze 64bits lts it is built from sources with the fix from ITS#8330
Since the upgrade, it keeps crashing regularly (I would say once every 2 to 5 days) However, there isn't anything in the logs: no message, no error, no nothing
slapd no longer runs, I restart it, and it runs ok until the next crash
now I don't know how to debug that. I could downgrade to the previous version (2.4.41) which worked fine, but as our university is closed until 01/11, this is the perfect time to investigate. I still don't know if it's a bug in 2.4.43 or a misconfiguration for which previous versions was forgiving
What are your advice to track down the cause for the crash?
Check whether openldap has been built with --enable-debug=yes. If so set ulimit -c unlimited, which hopefully will create a core dump on a crash.
-Dieter
I expirience the very same on FreeBSD since quite some versions already - in fact since 2010. I had to develope a workaround which is a monitoring script. It starts the service again everytime it died. The root cause could never be found. Please find the link below for further information:
https://forums.freebsd.org/threads/openldap-slapd-dies-sporadically.47634/
Best regards
Am 28.12.15 um 10:28 schrieb Dieter Klünter:
Am Mon, 28 Dec 2015 10:31:35 +0400 schrieb Jephte Clain jephte.clain@univ-reunion.fr:
hello all,
I upgraded to OpenLDAP 2.4.43 three weeks ago. It runs on debian squeeze 64bits lts it is built from sources with the fix from ITS#8330
Since the upgrade, it keeps crashing regularly (I would say once every 2 to 5 days) However, there isn't anything in the logs: no message, no error, no nothing
slapd no longer runs, I restart it, and it runs ok until the next crash
now I don't know how to debug that. I could downgrade to the previous version (2.4.41) which worked fine, but as our university is closed until 01/11, this is the perfect time to investigate. I still don't know if it's a bug in 2.4.43 or a misconfiguration for which previous versions was forgiving
What are your advice to track down the cause for the crash?
Check whether openldap has been built with --enable-debug=yes. If so set ulimit -c unlimited, which hopefully will create a core dump on a crash.
-Dieter
Le 28/12/2015 13:28, Dieter Klünter a écrit :
Am Mon, 28 Dec 2015 10:31:35 +0400 schrieb Jephte Clain jephte.clain@univ-reunion.fr:
hello all,
I upgraded to OpenLDAP 2.4.43 three weeks ago. It runs on debian squeeze 64bits lts it is built from sources with the fix from ITS#8330
Since the upgrade, it keeps crashing regularly (I would say once every 2 to 5 days) However, there isn't anything in the logs: no message, no error, no nothing
slapd no longer runs, I restart it, and it runs ok until the next crash
now I don't know how to debug that. (...) What are your advice to track down the cause for the crash?
Check whether openldap has been built with --enable-debug=yes. If so set ulimit -c unlimited, which hopefully will create a core dump on a crash.
-Dieter
very interesting. Yes it is compiled with enable-debug I have just enabled core dumps
according to the internet (tm) [grin], this is how I get a stracktrace, right?
gdb --core /path/to/THECOREDUMP <<<"thread apply all bt full"
thanks! regards, Jephté
Le 29/12/2015 09:34, Jephte Clain a écrit :
Le 28/12/2015 13:28, Dieter Klünter a écrit :
Am Mon, 28 Dec 2015 10:31:35 +0400 schrieb Jephte Clain jephte.clain@univ-reunion.fr:
hello all,
I upgraded to OpenLDAP 2.4.43 three weeks ago. It runs on debian squeeze 64bits lts it is built from sources with the fix from ITS#8330
Since the upgrade, it keeps crashing regularly (I would say once every 2 to 5 days) However, there isn't anything in the logs: no message, no error, no nothing
slapd no longer runs, I restart it, and it runs ok until the next crash
now I don't know how to debug that. (...) What are your advice to track down the cause for the crash?
Check whether openldap has been built with --enable-debug=yes. If so set ulimit -c unlimited, which hopefully will create a core dump on a crash.
-Dieter
very interesting. Yes it is compiled with enable-debug I have just enabled core dumps
according to the internet (tm) [grin], this is how I get a stracktrace, right?
gdb --core /path/to/THECOREDUMP <<<"thread apply all bt full"
re- it seems that this command is better:
gdb /usr/sbin/slapd -c /path/to/core <<<"thread apply all bt full"
to associate the core with the program that generated it can someone experienced confirm this?
also, in your experience, in case of a crash, can the debugging packages (i.e. slapd-dbg and libldap-dbg on debian) be installed a posteriori to enable better traceback informations?
thanks again for any help. regards, Jephté CLAIN
thanks! regards, Jephté
--On Monday, December 28, 2015 10:31 AM +0400 Jephte Clain jephte.clain@univ-reunion.fr wrote:
hello all,
I upgraded to OpenLDAP 2.4.43 three weeks ago. It runs on debian squeeze 64bits lts it is built from sources with the fix from ITS#8330
Since the upgrade, it keeps crashing regularly (I would say once every 2 to 5 days) However, there isn't anything in the logs: no message, no error, no nothing
slapd no longer runs, I restart it, and it runs ok until the next crash
Do you have stats logging enabled? If so, did slapd report it was being told to shut down, or did it just disappear during an operation? Is the OS killing slapd for OOM reasons? As Dieter noted, it is well advised to allow your system to create core files. You may want to read over https://wiki.zimbra.com/wiki/Enabling_Core_Files and correctly enable core file generation for your system.
--Quanah
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Le 28/12/2015 19:14, Quanah Gibson-Mount a écrit :
--On Monday, December 28, 2015 10:31 AM +0400 Jephte Clain jephte.clain@univ-reunion.fr wrote:
hello all,
I upgraded to OpenLDAP 2.4.43 three weeks ago. It runs on debian squeeze 64bits lts it is built from sources with the fix from ITS#8330
Since the upgrade, it keeps crashing regularly (I would say once every 2 to 5 days) However, there isn't anything in the logs: no message, no error, no nothing
slapd no longer runs, I restart it, and it runs ok until the next crash
Do you have stats logging enabled? If so, did slapd report it was being told to shut down, or did it just disappear during an operation? Is the OS killing slapd for OOM reasons? As Dieter noted, it is well advised to allow your system to create core files. You may want to read over https://wiki.zimbra.com/wiki/Enabling_Core_Files and correctly enable core file generation for your system.
hello,
thanks for the response yes, stats logging is enabled, and yes slapd just disappear during an operation I don't think slapd is killed for OOM reasons, it is running alone in the VM, with 4Go RAM (not much, but enough for us) How would I know? I just had a look at syslog, kern.log, dmesg, and daemon.log , but couldn't find anything related to OOM killing
Following Dieter's advice and the interesting article above, I enabled core dumping. I'll report if I find anything interesting
thanks again for your useful help! best regards, Jephté CLAIN
openldap-technical@openldap.org
-
Dieter Klünter -
Jephte Clain -
Leander Schäfer -
Quanah Gibson-Mount