Hi,
We recently left the debian openldap, and compiled the 2.4.39 and installed on our servers.
The old slapd failed multiple times a day, and now the new one too.
The log says: Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and initiated. Mar 15 07:06:25 ldap1 slapd[2798]: conn=72598 op=1 DISCONNECT tag=120 err=52 text= ... Mar 15 07:06:25 ldap1 slapd[2798]: conn=72570 op=3 DISCONNECT tag=120 err=52 text= Mar 15 07:06:25 ldap1 slapd[2798]: conn=72583 fd=45 closed (slapd shutdown) .... Mar 15 07:06:25 ldap1 slapd[2798]: conn=72602 fd=59 closed (slapd shutdown) Mar 15 07:06:25 ldap1 slapd[2798]: conn=72603 fd=60 closed (slapd shutdown) Mar 15 07:06:25 ldap1 slapd[2798]: slapd shutdown: waiting for 0 operations/tasks to finish Mar 15 07:06:25 ldap1 slapd[2798]: slapd stopped.
I've run out of the ideas.
The configure was with the following switches: --enable-crypt --enable-spasswd --enable-modules --enable-ppolicy --enable-syncprov --enable-unique --with-cyrus-sasl using libdb5.1-dev (deb 5.1.29-5)
Can you give me some pointers where to search for errors?
thanks, Dancsa
Hi,
On Sat, 15 Mar 2014, GALAMBOS Daniel wrote:
Hi,
We recently left the debian openldap, and compiled the 2.4.39 and installed on our servers.
The old slapd failed multiple times a day, and now the new one too.
It doesn't fail. See below ...
The log says: Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and initiated.
your slapd received a HUP signal and has stopped gracefully.
<snipp/>
I've run out of the ideas.
The configure was with the following switches: --enable-crypt --enable-spasswd --enable-modules --enable-ppolicy --enable-syncprov --enable-unique --with-cyrus-sasl using libdb5.1-dev (deb 5.1.29-5)
Can you give me some pointers where to search for errors?
do you have a cronjob using kill -HUP to the slapd pid for some kind of log rotation setup ?
Check your logrotate configs etc...
Greetings Christian
Thanks, but there is only one ldap releated cronjob, it monitors the slapd resource usage, but that did not do anything. For logging we use rsyslogd, and reloading, restarting or stopping the rsyslogd doesn't affect the slapd. Also there are no logrotate config which restarts the slapd, only reloades the rsyslog.
These were also checked by one of our senior sysadmins.
Most of the slapd shutdowns happens at 7:01, 8:01 ... etc when other machines uses the ldap more than usually(syncing users to other system what can't use ldap, generating mail identities to users...), but there are "only" 3-7 shutdown per day. There are sometimes a day, when it survives the whole day.
The slapd runs on a VM with 6 GiB memory (slapd uses 7-900MiB), the VM is dedicated for the slapd.
Dancsa
On 2014-03-15 14:23, Christian Kratzer wrote:
Hi,
On Sat, 15 Mar 2014, GALAMBOS Daniel wrote:
Hi,
We recently left the debian openldap, and compiled the 2.4.39 and installed on our servers.
The old slapd failed multiple times a day, and now the new one too.
It doesn't fail. See below ...
The log says: Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and initiated.
your slapd received a HUP signal and has stopped gracefully.
<snipp/> > I've run out of the ideas. > > The configure was with the following switches: --enable-crypt > --enable-spasswd --enable-modules --enable-ppolicy --enable-syncprov > --enable-unique --with-cyrus-sasl using libdb5.1-dev (deb 5.1.29-5) > > Can you give me some pointers where to search for errors?
do you have a cronjob using kill -HUP to the slapd pid for some kind of log rotation setup ?
Check your logrotate configs etc...
Greetings Christian
Hi,
On Sat, 15 Mar 2014, GALAMBOS Daniel wrote:
Thanks, but there is only one ldap releated cronjob, it monitors the slapd resource usage, but that did not do anything. For logging we use rsyslogd, and reloading, restarting or stopping the rsyslogd doesn't affect the slapd. Also there are no logrotate config which restarts the slapd, only reloades the rsyslog.
These were also checked by one of our senior sysadmins.
well you need to keep searching then.
The message:
Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and initiated.
you are sseing is from slapd receiving a HUP signal, so something must be sending it.
1. check again all user crontabs and the system crontab once more 2. search your disk for anything referencing slapd.pid ( grep -r slapd.pid / ) 3. check the dmesg output for OOM killer in case linux is killing slapd because of memory usage 4. check all other logs around the time slapd is being stopped 5. check all scripts under /etc/logratate.d 6. anything else you can think of
Most of the slapd shutdowns happens at 7:01, 8:01 ... etc when other machines uses the ldap more than usually(syncing users to other system what can't use ldap, generating mail identities to users...), but there are "only" 3-7 shutdown per day. There are sometimes a day, when it survives the whole day.
could well be linux OOM or anything else that rate limits processes based on load...
The slapd runs on a VM with 6 GiB memory (slapd uses 7-900MiB), the VM is dedicated for the slapd.
Greetings Christian
<snipp/>
On Sat, 2014-03-15 at 17:09 +0100, Christian Kratzer wrote:
The message:
Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and initiated.
you are sseing is from slapd receiving a HUP signal, so something must be sending it.
HUP, INT, TERM or TRAP. TRAP is for debugging, not sure why that's included. Or SIGBREAK if it is defined and HAVE_NT_SERVICE_MANAGER is not. Looks like some Windows thing. Maybe it should only be included if defined(HAVE_NT_SERVICE_MANAGER).
We had a similar issue a few months back. I discovered that it was dying at about the same time. Come to find out it was syslog rotation that was doing it. Syslog sends a HUP signal to rotate the logs and restart a daemon. I had to put a sleep statement in the syslog for slapd.
-----Original Message----- From: openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] On Behalf Of Hallvard Breien Furuseth Sent: Monday, March 17, 2014 4:24 AM To: Christian Kratzer Cc: GALAMBOS Daniel; openldap-technical@openldap.org Subject: Re: slapd shuts down for no reason
On Sat, 2014-03-15 at 17:09 +0100, Christian Kratzer wrote:
The message:
Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and
initiated.
you are sseing is from slapd receiving a HUP signal, so something must be
sending it.
HUP, INT, TERM or TRAP. TRAP is for debugging, not sure why that's included. Or SIGBREAK if it is defined and HAVE_NT_SERVICE_MANAGER is not. Looks like some Windows thing. Maybe it should only be included if defined(HAVE_NT_SERVICE_MANAGER).
-- Hallvard
On Mon, 2014-03-17 at 08:30 -0400, Borresen, John - 0442 - MITLL wrote:
We had a similar issue a few months back. I discovered that it was dying at about the same time. Come to find out it was syslog rotation that was doing it. Syslog sends a HUP signal to rotate the logs and restart a daemon. I had to put a sleep statement in the syslog for slapd.
Your syslog rotates logs? On our host (RedHat) it's logrotate which does that. And it's logrotate which must kill -HUP syslogd, to make syslogd close and reopen the log. From logrotate.conf:
notifempty missingok create 0640 ldap cerebrum start 0
/ldap/log/syslog/openldap.log { size=250M rotate 2500 compress delaycompress lastaction /bin/kill -s HUP syslogd endscript }
Fall back to legacy unix...sorry logrotate is more appropriate. Yes, yours is very similar to how ours looked and it would stop the daemon fine, and rotate the log but would not restart. I modified it to look like this:
.... /etc/init.d/slapd stop; sleep 5; /etc/init.d/slapd start ....
Commenting out the "/bin/kill -s HUP syslogd".
Now, it does what it is supposed to do.
John
-----Original Message----- From: Hallvard Breien Furuseth [mailto:h.b.furuseth@usit.uio.no] Sent: Tuesday, March 18, 2014 4:26 AM To: Borresen, John - 0442 - MITLL Cc: openldap-technical@openldap.org Subject: RE: slapd shuts down for no reason
On Mon, 2014-03-17 at 08:30 -0400, Borresen, John - 0442 - MITLL wrote:
We had a similar issue a few months back. I discovered that it was dying at about the same time. Come to find out it was syslog rotation that was doing it. Syslog sends a HUP signal to rotate the logs and restart a daemon. I had to put a sleep statement in the syslog for slapd.
Your syslog rotates logs? On our host (RedHat) it's logrotate which does that. And it's logrotate which must kill -HUP syslogd, to make syslogd close and reopen the log. From logrotate.conf:
notifempty missingok create 0640 ldap cerebrum start 0
/ldap/log/syslog/openldap.log { size=250M rotate 2500 compress delaycompress lastaction /bin/kill -s HUP syslogd endscript }
Hi,
On Tue, 18 Mar 2014, Borresen, John - 0442 - MITLL wrote:
Fall back to legacy unix...sorry logrotate is more appropriate. Yes, yours is very similar to how ours looked and it would stop the daemon fine, and rotate the log but would not restart. I modified it to look like this:
.... /etc/init.d/slapd stop; sleep 5; /etc/init.d/slapd start
why are you stopping and starting slapd ? This does not make any sense.
Greetings Christian
....
Commenting out the "/bin/kill -s HUP syslogd".
Now, it does what it is supposed to do.
John
-----Original Message----- From: Hallvard Breien Furuseth [mailto:h.b.furuseth@usit.uio.no] Sent: Tuesday, March 18, 2014 4:26 AM To: Borresen, John - 0442 - MITLL Cc: openldap-technical@openldap.org Subject: RE: slapd shuts down for no reason
On Mon, 2014-03-17 at 08:30 -0400, Borresen, John - 0442 - MITLL wrote:
We had a similar issue a few months back. I discovered that it was dying at about the same time. Come to find out it was syslog rotation that was doing it. Syslog sends a HUP signal to rotate the logs and restart a daemon. I had to put a sleep statement in the syslog for slapd.
Your syslog rotates logs? On our host (RedHat) it's logrotate which does that. And it's logrotate which must kill -HUP syslogd, to make syslogd close and reopen the log. From logrotate.conf:
notifempty missingok create 0640 ldap cerebrum start 0
/ldap/log/syslog/openldap.log { size=250M rotate 2500 compress delaycompress lastaction /bin/kill -s HUP syslogd endscript }
Only for logrotate. The -HUP in the logrotate.conf was not working. Even on the command line kill the slapd process with a -HUP was not working. Putting the /etc/init.d/slapd stop; sleep 5, /etc/init.d/slapd start does make sense when nothing else was working. We tried numerous different iterations, this was the only thing that worked for us. It worked...so, thus it makes sense.
-----Original Message----- From: Christian Kratzer [mailto:ck-lists@cksoft.de] Sent: Tuesday, March 18, 2014 8:51 AM To: Borresen, John - 0442 - MITLL Cc: Hallvard Breien Furuseth; openldap-technical@openldap.org Subject: RE: slapd shuts down for no reason
Hi,
On Tue, 18 Mar 2014, Borresen, John - 0442 - MITLL wrote:
Fall back to legacy unix...sorry logrotate is more appropriate. Yes, yours is very similar to how ours looked and it would stop the daemon fine, and rotate the log but would not restart. I modified it to look
like this:
.... /etc/init.d/slapd stop; sleep 5; /etc/init.d/slapd start
why are you stopping and starting slapd ? This does not make any sense.
Greetings Christian
....
Commenting out the "/bin/kill -s HUP syslogd".
Now, it does what it is supposed to do.
John
-----Original Message----- From: Hallvard Breien Furuseth [mailto:h.b.furuseth@usit.uio.no] Sent: Tuesday, March 18, 2014 4:26 AM To: Borresen, John - 0442 - MITLL Cc: openldap-technical@openldap.org Subject: RE: slapd shuts down for no reason
On Mon, 2014-03-17 at 08:30 -0400, Borresen, John - 0442 - MITLL wrote:
We had a similar issue a few months back. I discovered that it was dying at about the same time. Come to find out it was syslog rotation that was doing it. Syslog sends a HUP signal to rotate the logs and restart a daemon. I had to put a sleep statement in the syslog
for slapd.
Your syslog rotates logs? On our host (RedHat) it's logrotate which does that. And it's logrotate which must kill -HUP syslogd, to make syslogd close and reopen the log. From logrotate.conf:
notifempty missingok create 0640 ldap cerebrum start 0
/ldap/log/syslog/openldap.log { size=250M rotate 2500 compress delaycompress lastaction /bin/kill -s HUP syslogd endscript }
It still doesn't make sense to stop slapd in logrotate. Seems I was unclear: It's the process which writes to the log which must be HUPped, so it'll stop writing to the rotated logfile. That's syslogd, not slapd. Unless you're doing 'slapd 2>slapd.log &'. (Or it's rsyslogd or klogd or whatever. Ask 'ps'.)
On Tue, 2014-03-18 at 08:56 -0400, Borresen, John - 0442 - MITLL wrote:
Only for logrotate. (...)
Christian Kratzer wrote:
why are you stopping and starting slapd ? This does not make any sense.
Hallvard Breien Furuseth wrote:
On Sat, 2014-03-15 at 17:09 +0100, Christian Kratzer wrote:
The message:
Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and initiated.
you are sseing is from slapd receiving a HUP signal, so something must be sending it.
HUP, INT, TERM or TRAP. TRAP is for debugging, not sure why that's included. Or SIGBREAK if it is defined and HAVE_NT_SERVICE_MANAGER is not. Looks like some Windows thing. Maybe it should only be included if defined(HAVE_NT_SERVICE_MANAGER).
I think you've misread something, SIGBREAK is already #if defined(HAVE_NT_SERVICE_MANAGER).
On Mon, 2014-03-17 at 10:02 -0700, Howard Chu wrote:
Hallvard Breien Furuseth wrote:
On Sat, 2014-03-15 at 17:09 +0100, Christian Kratzer wrote:
you are sseing is from slapd receiving a HUP signal, so something must be sending it.
HUP, INT, TERM or TRAP. TRAP is for debugging, not sure why that's included. Or SIGBREAK if it is defined and HAVE_NT_SERVICE_MANAGER is not. Looks like some Windows thing. Maybe it should only be included if defined(HAVE_NT_SERVICE_MANAGER).
I think you've misread something, SIGBREAK is already #if defined(HAVE_NT_SERVICE_MANAGER).
No, I phrased it badly. main.c sets a SIGBREAK handler if SIGBREAK exists. Then the signal handler sets slapd_shutdown if !defined(HAVE_NT_SERVICE_MANAGER). Could be intentional, that could be what the comment in slap_sig_shutdown() means.
Check for OOM messages in your system log. The OOM manager will kill whatever the next process is to allocate memory when it runs out regardless of how much or little that process is using. Whatever the case there is no slapd bug here. Something on your system is specifically telling slapd to stop and it is doing as instructed.
--Quanah
On Mar 15, 2014, at 8:54 AM, GALAMBOS Daniel dancsa@dancsa.hu wrote:
Thanks, but there is only one ldap releated cronjob, it monitors the slapd resource usage, but that did not do anything. For logging we use rsyslogd, and reloading, restarting or stopping the rsyslogd doesn't affect the slapd. Also there are no logrotate config which restarts the slapd, only reloades the rsyslog.
These were also checked by one of our senior sysadmins.
Most of the slapd shutdowns happens at 7:01, 8:01 ... etc when other machines uses the ldap more than usually(syncing users to other system what can't use ldap, generating mail identities to users...), but there are "only" 3-7 shutdown per day. There are sometimes a day, when it survives the whole day.
The slapd runs on a VM with 6 GiB memory (slapd uses 7-900MiB), the VM is dedicated for the slapd.
Dancsa
On 2014-03-15 14:23, Christian Kratzer wrote: Hi,
On Sat, 15 Mar 2014, GALAMBOS Daniel wrote:
Hi,
We recently left the debian openldap, and compiled the 2.4.39 and installed on our servers.
The old slapd failed multiple times a day, and now the new one too.
It doesn't fail. See below ...
The log says: Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and initiated.
your slapd received a HUP signal and has stopped gracefully.
<snipp/> > I've run out of the ideas. > > The configure was with the following switches: --enable-crypt > --enable-spasswd --enable-modules --enable-ppolicy --enable-syncprov > --enable-unique --with-cyrus-sasl using libdb5.1-dev (deb 5.1.29-5) > > Can you give me some pointers where to search for errors?
do you have a cronjob using kill -HUP to the slapd pid for some kind of log rotation setup ?
Check your logrotate configs etc...
Greetings Christian
openldap-technical@openldap.org
-
Borresen, John - 0442 - MITLL -
Christian Kratzer -
GALAMBOS Daniel -
Hallvard Breien Furuseth -
Howard Chu -
Quanah Gibson-Mount