2:58 a.m.
Hi,
We recently left the debian openldap, and compiled the 2.4.39 and
installed on our servers.
The old slapd failed multiple times a day, and now the new one too.
The log says:
Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and initiated.
Mar 15 07:06:25 ldap1 slapd[2798]: conn=72598 op=1 DISCONNECT tag=120
err=52 text=
...
Mar 15 07:06:25 ldap1 slapd[2798]: conn=72570 op=3 DISCONNECT tag=120
err=52 text=
Mar 15 07:06:25 ldap1 slapd[2798]: conn=72583 fd=45 closed (slapd shutdown)
....
Mar 15 07:06:25 ldap1 slapd[2798]: conn=72602 fd=59 closed (slapd shutdown)
Mar 15 07:06:25 ldap1 slapd[2798]: conn=72603 fd=60 closed (slapd shutdown)
Mar 15 07:06:25 ldap1 slapd[2798]: slapd shutdown: waiting for 0
operations/tasks to finish
Mar 15 07:06:25 ldap1 slapd[2798]: slapd stopped.
I've run out of the ideas.
The configure was with the following switches: --enable-crypt
--enable-spasswd --enable-modules --enable-ppolicy --enable-syncprov
--enable-unique --with-cyrus-sasl using libdb5.1-dev (deb 5.1.29-5)
Can you give me some pointers where to search for errors?
thanks,
Dancsa
6:23 a.m.
Hi,
On Sat, 15 Mar 2014, GALAMBOS Daniel wrote:
Hi,
We recently left the debian openldap, and compiled the 2.4.39 and
installed on our servers.
The old slapd failed multiple times a day, and now the new one too.
It doesn't fail. See below ...
The log says:
Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and initiated.
your slapd received a HUP signal and has stopped gracefully.
<snipp/>
I've run out of the ideas.
The configure was with the following switches: --enable-crypt
--enable-spasswd --enable-modules --enable-ppolicy --enable-syncprov
--enable-unique --with-cyrus-sasl using libdb5.1-dev (deb 5.1.29-5)
Can you give me some pointers where to search for errors?
do you have a cronjob using kill -HUP to the slapd pid for some kind of log rotation setup
?
Check your logrotate configs etc...
Greetings
Christian
--
Christian Kratzer CK Software GmbH
Email: ck(a)cksoft.de Wildberger Weg 24/2
Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden
Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart
Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/
8:40 a.m.
Thanks, but there is only one ldap releated cronjob, it monitors the
slapd resource usage, but that did not do anything. For logging we use
rsyslogd, and reloading, restarting or stopping the rsyslogd doesn't
affect the slapd. Also there are no logrotate config which restarts the
slapd, only reloades the rsyslog.
These were also checked by one of our senior sysadmins.
Most of the slapd shutdowns happens at 7:01, 8:01 ... etc when other
machines uses the ldap more than usually(syncing users to other system
what can't use ldap, generating mail identities to users...), but there
are "only" 3-7 shutdown per day. There are sometimes a day, when it
survives the whole day.
The slapd runs on a VM with 6 GiB memory (slapd uses 7-900MiB), the VM
is dedicated for the slapd.
Dancsa
On 2014-03-15 14:23, Christian Kratzer wrote:
Hi,
On Sat, 15 Mar 2014, GALAMBOS Daniel wrote:
> Hi,
>
> We recently left the debian openldap, and compiled the 2.4.39 and
> installed on our servers.
>
> The old slapd failed multiple times a day, and now the new one too.
It doesn't fail. See below ...
> The log says:
> Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and
> initiated.
your slapd received a HUP signal and has stopped gracefully.
<snipp/>
> I've run out of the ideas.
>
> The configure was with the following switches: --enable-crypt
> --enable-spasswd --enable-modules --enable-ppolicy --enable-syncprov
> --enable-unique --with-cyrus-sasl using libdb5.1-dev (deb 5.1.29-5)
>
> Can you give me some pointers where to search for errors?
do you have a cronjob using kill -HUP to the slapd pid for some kind
of log rotation setup ?
Check your logrotate configs etc...
Greetings
Christian
9:09 a.m.
Hi,
On Sat, 15 Mar 2014, GALAMBOS Daniel wrote:
Thanks, but there is only one ldap releated cronjob, it monitors
the
slapd resource usage, but that did not do anything. For logging we use
rsyslogd, and reloading, restarting or stopping the rsyslogd doesn't
affect the slapd. Also there are no logrotate config which restarts the
slapd, only reloades the rsyslog.
These were also checked by one of our senior sysadmins.
well you need to keep searching then.
The message:
Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and initiated.
you are sseing is from slapd receiving a HUP signal, so something must be sending it.
1. check again all user crontabs and the system crontab once more
2. search your disk for anything referencing slapd.pid ( grep -r slapd.pid / )
3. check the dmesg output for OOM killer in case linux is killing slapd because of memory
usage
4. check all other logs around the time slapd is being stopped
5. check all scripts under /etc/logratate.d
6. anything else you can think of
Most of the slapd shutdowns happens at 7:01, 8:01 ... etc when other
machines uses the ldap more than usually(syncing users to other system
what can't use ldap, generating mail identities to users...), but there
are "only" 3-7 shutdown per day. There are sometimes a day, when it
survives the whole day.
could well be linux OOM or anything else that rate limits processes based on load...
The slapd runs on a VM with 6 GiB memory (slapd uses 7-900MiB), the VM
is dedicated for the slapd.
Greetings
Christian
<snipp/>
--
Christian Kratzer CK Software GmbH
Email: ck(a)cksoft.de Wildberger Weg 24/2
Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden
Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart
Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/
1:24 a.m.
On Sat, 2014-03-15 at 17:09 +0100, Christian Kratzer wrote:
The message:
Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and initiated.
you are sseing is from slapd receiving a HUP signal, so something must be sending it.
HUP, INT, TERM or TRAP. TRAP is for debugging, not sure
why that's included.
Or SIGBREAK if it is defined and HAVE_NT_SERVICE_MANAGER
is not. Looks like some Windows thing. Maybe it should
only be included if defined(HAVE_NT_SERVICE_MANAGER).
--
Hallvard
5:30 a.m.
We had a similar issue a few months back. I discovered that it was dying at
about the same time. Come to find out it was syslog rotation that was doing
it. Syslog sends a HUP signal to rotate the logs and restart a daemon. I
had to put a sleep statement in the syslog for slapd.
-----Original Message-----
From: openldap-technical-bounces(a)OpenLDAP.org
[mailto:openldap-technical-bounces@OpenLDAP.org] On Behalf Of Hallvard
Breien Furuseth
Sent: Monday, March 17, 2014 4:24 AM
To: Christian Kratzer
Cc: GALAMBOS Daniel; openldap-technical(a)openldap.org
Subject: Re: slapd shuts down for no reason
On Sat, 2014-03-15 at 17:09 +0100, Christian Kratzer wrote:
The message:
Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and
initiated.
you are sseing is from slapd receiving a HUP signal, so something must be
sending
it.
HUP, INT, TERM or TRAP. TRAP is for debugging, not sure why that's
included.
Or SIGBREAK if it is defined and HAVE_NT_SERVICE_MANAGER is not. Looks like
some Windows thing. Maybe it should only be included if
defined(HAVE_NT_SERVICE_MANAGER).
--
Hallvard
1:25 a.m.
On Mon, 2014-03-17 at 08:30 -0400, Borresen, John - 0442 - MITLL wrote:
We had a similar issue a few months back. I discovered that it was
dying at
about the same time. Come to find out it was syslog rotation that was doing
it. Syslog sends a HUP signal to rotate the logs and restart a daemon. I
had to put a sleep statement in the syslog for slapd.
Your syslog rotates logs? On our host (RedHat) it's logrotate which
does that. And it's logrotate which must kill -HUP syslogd, to make
syslogd close and reopen the log. From logrotate.conf:
notifempty
missingok
create 0640 ldap cerebrum
start 0
/ldap/log/syslog/openldap.log {
size=250M
rotate 2500
compress
delaycompress
lastaction
/bin/kill -s HUP syslogd
endscript
}
5:16 a.m.
Fall back to legacy unix...sorry logrotate is more appropriate. Yes, yours
is very similar to how ours looked and it would stop the daemon fine, and
rotate the log but would not restart. I modified it to look like this:
....
/etc/init.d/slapd stop; sleep 5; /etc/init.d/slapd start
....
Commenting out the "/bin/kill -s HUP syslogd".
Now, it does what it is supposed to do.
John
-----Original Message-----
From: Hallvard Breien Furuseth [mailto:h.b.furuseth@usit.uio.no]
Sent: Tuesday, March 18, 2014 4:26 AM
To: Borresen, John - 0442 - MITLL
Cc: openldap-technical(a)openldap.org
Subject: RE: slapd shuts down for no reason
On Mon, 2014-03-17 at 08:30 -0400, Borresen, John - 0442 - MITLL wrote:
We had a similar issue a few months back. I discovered that it was
dying at about the same time. Come to find out it was syslog rotation
that was doing it. Syslog sends a HUP signal to rotate the logs and
restart a daemon. I had to put a sleep statement in the syslog for slapd.
Your syslog rotates logs? On our host (RedHat) it's logrotate which does
that. And it's logrotate which must kill -HUP syslogd, to make syslogd
close and reopen the log. From logrotate.conf:
notifempty
missingok
create 0640 ldap cerebrum
start 0
/ldap/log/syslog/openldap.log {
size=250M
rotate 2500
compress
delaycompress
lastaction
/bin/kill -s HUP syslogd
endscript
}
5:51 a.m.
Hi,
On Tue, 18 Mar 2014, Borresen, John - 0442 - MITLL wrote:
Fall back to legacy unix...sorry logrotate is more appropriate. Yes,
yours
is very similar to how ours looked and it would stop the daemon fine, and
rotate the log but would not restart. I modified it to look like this:
....
/etc/init.d/slapd stop; sleep 5; /etc/init.d/slapd start
why are you stopping and starting slapd ? This does not make any sense.
Greetings
Christian
....
Commenting out the "/bin/kill -s HUP syslogd".
Now, it does what it is supposed to do.
John
-----Original Message-----
From: Hallvard Breien Furuseth [mailto:h.b.furuseth@usit.uio.no]
Sent: Tuesday, March 18, 2014 4:26 AM
To: Borresen, John - 0442 - MITLL
Cc: openldap-technical(a)openldap.org
Subject: RE: slapd shuts down for no reason
On Mon, 2014-03-17 at 08:30 -0400, Borresen, John - 0442 - MITLL wrote:
> We had a similar issue a few months back. I discovered that it was
> dying at about the same time. Come to find out it was syslog rotation
> that was doing it. Syslog sends a HUP signal to rotate the logs and
> restart a daemon. I had to put a sleep statement in the syslog for slapd.
Your syslog rotates logs? On our host (RedHat) it's logrotate which does
that. And it's logrotate which must kill -HUP syslogd, to make syslogd
close and reopen the log. From logrotate.conf:
notifempty
missingok
create 0640 ldap cerebrum
start 0
/ldap/log/syslog/openldap.log {
size=250M
rotate 2500
compress
delaycompress
lastaction
/bin/kill -s HUP syslogd
endscript
}
--
Christian Kratzer CK Software GmbH
Email: ck(a)cksoft.de Wildberger Weg 24/2
Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden
Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart
Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/
5:56 a.m.
Only for logrotate. The -HUP in the logrotate.conf was not working. Even
on the command line kill the slapd process with a -HUP was not working.
Putting the /etc/init.d/slapd stop; sleep 5, /etc/init.d/slapd start does
make sense when nothing else was working. We tried numerous different
iterations, this was the only thing that worked for us. It worked...so,
thus it makes sense.
-----Original Message-----
From: Christian Kratzer [mailto:ck-lists@cksoft.de]
Sent: Tuesday, March 18, 2014 8:51 AM
To: Borresen, John - 0442 - MITLL
Cc: Hallvard Breien Furuseth; openldap-technical(a)openldap.org
Subject: RE: slapd shuts down for no reason
Hi,
On Tue, 18 Mar 2014, Borresen, John - 0442 - MITLL wrote:
Fall back to legacy unix...sorry logrotate is more appropriate. Yes,
yours is very similar to how ours looked and it would stop the daemon
fine, and rotate the log but would not restart. I modified it to look
like this:
....
/etc/init.d/slapd stop; sleep 5; /etc/init.d/slapd start
why are you stopping and starting slapd ? This does not make any sense.
Greetings
Christian
....
Commenting out the "/bin/kill -s HUP syslogd".
Now, it does what it is supposed to do.
John
-----Original Message-----
From: Hallvard Breien Furuseth [mailto:h.b.furuseth@usit.uio.no]
Sent: Tuesday, March 18, 2014 4:26 AM
To: Borresen, John - 0442 - MITLL
Cc: openldap-technical(a)openldap.org
Subject: RE: slapd shuts down for no reason
On Mon, 2014-03-17 at 08:30 -0400, Borresen, John - 0442 - MITLL wrote:
> We had a similar issue a few months back. I discovered that it was
> dying at about the same time. Come to find out it was syslog
> rotation that was doing it. Syslog sends a HUP signal to rotate the
> logs and restart a daemon. I had to put a sleep statement in the syslog
for
slapd.
Your syslog rotates logs? On our host (RedHat) it's logrotate which
does that. And it's logrotate which must kill -HUP syslogd, to make
syslogd close and reopen the log. From logrotate.conf:
notifempty
missingok
create 0640 ldap cerebrum
start 0
/ldap/log/syslog/openldap.log {
size=250M
rotate 2500
compress
delaycompress
lastaction
/bin/kill -s HUP syslogd
endscript
}
--
Christian Kratzer CK Software GmbH
Email: ck(a)cksoft.de Wildberger Weg 24/2
Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden
Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart
Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/
9:57 a.m.
It still doesn't make sense to stop slapd in logrotate. Seems I
was unclear: It's the process which writes to the log which must
be HUPped, so it'll stop writing to the rotated logfile. That's
syslogd, not slapd. Unless you're doing 'slapd 2>slapd.log &'.
(Or it's rsyslogd or klogd or whatever. Ask 'ps'.)
On Tue, 2014-03-18 at 08:56 -0400, Borresen, John - 0442 - MITLL wrote:
Only for logrotate. (...)
Christian Kratzer wrote:
> why are you stopping and starting slapd ? This does not make any
> sense.
10:02 a.m.
Hallvard Breien Furuseth wrote:
On Sat, 2014-03-15 at 17:09 +0100, Christian Kratzer wrote:
> The message:
>
> Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and initiated.
>
> you are sseing is from slapd receiving a HUP signal, so something must be sending
it.
HUP, INT, TERM or TRAP. TRAP is for debugging, not sure
why that's included.
Or SIGBREAK if it is defined and HAVE_NT_SERVICE_MANAGER
is not. Looks like some Windows thing. Maybe it should
only be included if defined(HAVE_NT_SERVICE_MANAGER).
I think you've misread something, SIGBREAK is already #if
defined(HAVE_NT_SERVICE_MANAGER).
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
1:15 a.m.
On Mon, 2014-03-17 at 10:02 -0700, Howard Chu wrote:
Hallvard Breien Furuseth wrote:
> On Sat, 2014-03-15 at 17:09 +0100, Christian Kratzer wrote:
>> you are sseing is from slapd receiving a HUP signal, so something must be
sending it.
>
> HUP, INT, TERM or TRAP. TRAP is for debugging, not sure
> why that's included.
> Or SIGBREAK if it is defined and HAVE_NT_SERVICE_MANAGER
> is not. Looks like some Windows thing. Maybe it should
> only be included if defined(HAVE_NT_SERVICE_MANAGER).
I think you've misread something, SIGBREAK is already #if
defined(HAVE_NT_SERVICE_MANAGER).
No, I phrased it badly. main.c sets a SIGBREAK handler
if SIGBREAK exists. Then the signal handler sets
slapd_shutdown if !defined(HAVE_NT_SERVICE_MANAGER).
Could be intentional, that could be what the comment
in slap_sig_shutdown() means.
12:44 p.m.
Check for OOM messages in your system log. The OOM manager will kill whatever the next
process is to allocate memory when it runs out regardless of how much or little that
process is using. Whatever the case there is no slapd bug here. Something on your system
is specifically telling slapd to stop and it is doing as instructed.
--Quanah
On Mar 15, 2014, at 8:54 AM, GALAMBOS Daniel <dancsa(a)dancsa.hu>
wrote:
Thanks, but there is only one ldap releated cronjob, it monitors the
slapd resource usage, but that did not do anything. For logging we use
rsyslogd, and reloading, restarting or stopping the rsyslogd doesn't
affect the slapd. Also there are no logrotate config which restarts the
slapd, only reloades the rsyslog.
These were also checked by one of our senior sysadmins.
Most of the slapd shutdowns happens at 7:01, 8:01 ... etc when other
machines uses the ldap more than usually(syncing users to other system
what can't use ldap, generating mail identities to users...), but there
are "only" 3-7 shutdown per day. There are sometimes a day, when it
survives the whole day.
The slapd runs on a VM with 6 GiB memory (slapd uses 7-900MiB), the VM
is dedicated for the slapd.
Dancsa
> On 2014-03-15 14:23, Christian Kratzer wrote:
> Hi,
>
>> On Sat, 15 Mar 2014, GALAMBOS Daniel wrote:
>>
>> Hi,
>>
>> We recently left the debian openldap, and compiled the 2.4.39 and
>> installed on our servers.
>>
>> The old slapd failed multiple times a day, and now the new one too.
>
> It doesn't fail. See below ...
>
>> The log says:
>> Mar 15 07:06:25 ldap1 slapd[2798]: daemon: shutdown requested and
>> initiated.
>
> your slapd received a HUP signal and has stopped gracefully.
>
> <snipp/>
>> I've run out of the ideas.
>>
>> The configure was with the following switches: --enable-crypt
>> --enable-spasswd --enable-modules --enable-ppolicy --enable-syncprov
>> --enable-unique --with-cyrus-sasl using libdb5.1-dev (deb 5.1.29-5)
>>
>> Can you give me some pointers where to search for errors?
>
> do you have a cronjob using kill -HUP to the slapd pid for some kind
> of log rotation setup ?
>
> Check your logrotate configs etc...
>
> Greetings
> Christian
3485
days inactive
3488
days old
openldap-technical@openldap.org
13 comments
6 participants
participants (6)
-
Borresen, John - 0442 - MITLL -
Christian Kratzer -
GALAMBOS Daniel -
Hallvard Breien Furuseth -
Howard Chu -
Quanah Gibson-Mount