--000e0cd48c428f954504a23eaa22
Content-Type: text/plain; charset=UTF-8
First of all let me apologize for the state of the config.
As you say the logs don't show any crash yet the OpenLDAP server is not
accepting any new connections after 20:36:37, there should be hundreds of
connections being logged there but instead the server hangs completely and
nothing is logged, even on log level 1.
Perhaps someone could suggest a way of getting more information than what
loglevel 1 shows?
Would the only other step be a debug build?
Jarl
On Sun, May 1, 2011 at 4:01 AM, Howard Chu hyc@symas.com wrote:
jarl@dallur.com wrote:
Full_Name: Jarl Stefansson
Version: 2.4.22
OS: Centos 5.5
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (81.15.35.75)
Two servers running OpenLDAP in master/master using syncrepl, roughly once
per
week one of the servers stops responding to new connections, I tried
logging
with loglevel=1 and this is all I got, all suggestions and comments
appreciated.
This is most likely a "load" related problem since we never experience
this
problem in the testlab, only production.:
Your log doesn't indicate any kind of crash. It only shows that a shutdown
was requested, and apparently your startup scripts didn't wait for the
shutdown to complete before starting again.
Apr 23 20:36:37 ldap01 slapd2.4[32233]:
bdb_dn2entry("macaddress=1\2C6\2C00:00:00:00:00:01,ou=xxx,ou=xx,o=xxxx")
Apr 23 20:36:37 ldap01 slapd2.4[32233]: =>
bdb_dn2id("macaddress=1\2C6\2C00:00:00:00:00:01,ou=xxx,ou=xx,o=xxxx")
Apr 23 20:36:37 ldap01 slapd2.4[32233]:<= bdb_dn2id: get failed:
DB_NOTFOUND:
No matching key/data pair found (-30988)
Apr 23 20:36:37 ldap01 slapd2.4[32233]: => bdb_dn2id_add 0x2fed7:
"macaddress=1\2C6\2C00:00:00:00:00:01,ou=xxx,ou=xx,o=xxxx"
Apr 23 20:37:44 ldap01 slapd2.4[32233]: slap_listener_activate(8):
Apr 23 21:35:42 ldap01 slapd2.4[32233]: connection_close: conn=1025 sd=16
Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=1022 sd=17
Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=1037 sd=18
Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=1038 sd=20
Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=1039 sd=25
Apr 24 12:02:23 ldap01 slapd2.4[32233]: daemon: shutdown requested and
initiated. I have two Centos 5 servers running Openldap 2.4.22 with
master/master syncrepl setup,
Apr 24 12:02:23 ldap01 slapd2.4[32233]: connection_close: conn=1005 sd=21
Apr 24 12:02:23 ldap01 slapd2.4[32233]: slapd shutdown: waiting for 59
operations/tasks to finish
Apr 24 12:02:34 ldap01 slapd2.4[26503]: bdb_db_open: database "": unclean
shutdown detected; attempting recovery.
Apr 24 12:02:35 ldap01 slapd2.4[26503]: slapd starting
Your config is a mess, with global directives and DB-specific directives
interleaved. That may not be the cause of any specific problems, but it
shows sloppiness on the part of the sysadmins.
---------------------- Relevant Config
moduleload syncprov.la
TLSCertificateFile /etc/pki/tls/private/ldap.pem
TLSCertificateKeyFile /etc/pki/tls/private/ldap.pem
TLSCACertificateFile /etc/pki/tls/private/ldap.pem
serverID 001
database bdb
sizelimit 50000
cachesize 10000
checkpoint 256 5
syncrepl rid=001
provider=ldap://10.10.10.10:389
bindmethod=simple
binddn="uid=xxxxrep,ou=xxxxxx,o=xxxxxx"
credentials=MyPassword
searchbase="o=xxxx"
schemachecking=on
sizelimit="unlimited"
timelimit="unlimited"
type=refreshAndPersist
interval=00:00:00:10
retry="5 5 60 +"
attrs="*,+"
mirrormode on
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100
idletimeout 3600
threads 32
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
--
Regards
Jarl
jarl@dallur.com
--000e0cd48c428f954504a23eaa22
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
First of all let me=C2=A0apologize=C2=A0for the state of the config.<div><b=
r></div><div>As you say the logs don't show any crash yet the OpenLDAP =
server is not accepting any new connections after=C2=A0<meta http-equiv=3D"=
content-type" content=3D"text/html; charset=3Dutf-8">20:36:37, there should=
be hundreds of connections being logged there but instead the server hangs=
completely and nothing is logged, even on log level 1.</div>
<div><br></div><div>Perhaps someone could suggest a way of getting more inf=
ormation than what loglevel 1 shows?</div><div><br></div><div>Would the onl=
y other step be a debug build?</div><div><br></div><div>Jarl<br><br><div cl=
ass=3D"gmail_quote">
On Sun, May 1, 2011 at 4:01 AM, Howard Chu <span dir=3D"ltr"><<a href=3D=
"mailto:hyc@symas.com">hyc@symas.com</a>></span> wrote:<br><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;=
padding-left:1ex;">
<div class=3D"im"><a href=3D"mailto:jarl@dallur.com" target=3D"_blank">jarl=
@dallur.com</a> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
Full_Name: Jarl Stefansson<br>
Version: =C2=A02.4.22<br>
OS: Centos 5.5<br>
URL: <a href=3D"ftp://ftp.openldap.org/incoming/" target=3D"_blank">ftp://f=
tp.openldap.org/incoming/</a><br>
Submission from: (NULL) (81.15.35.75)<br>
<br>
<br>
Two servers running OpenLDAP in master/master using syncrepl, roughly once =
per<br>
week one of the servers stops responding to new connections, I tried loggin=
g<br>
with loglevel=3D1 and this is all I got, all suggestions and comments<br>
appreciated.<br>
<br>
This is most likely a "load" related problem since we never exper=
ience this<br>
problem in the testlab, only production.:<br>
</blockquote>
<br></div>
Your log doesn't indicate any kind of crash. It only shows that a shutd=
own was requested, and apparently your startup scripts didn't wait for =
the shutdown to complete before starting again.<div class=3D"im"><br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
Apr 23 20:36:37 ldap01 slapd2.4[32233]:<br>
bdb_dn2entry("macaddress=3D1\2C6\2C00:00:00:00:00:01,ou=3Dxxx,ou=3Dxx,=
o=3Dxxxx")<br>
Apr 23 20:36:37 ldap01 slapd2.4[32233]: =3D><br>
bdb_dn2id("macaddress=3D1\2C6\2C00:00:00:00:00:01,ou=3Dxxx,ou=3Dxx,o=
=3Dxxxx")<br>
Apr 23 20:36:37 ldap01 slapd2.4[32233]:<=3D bdb_dn2id: get failed: DB_NO=
TFOUND:<br>
No matching key/data pair found (-30988)<br>
Apr 23 20:36:37 ldap01 slapd2.4[32233]: =3D> =C2=A0bdb_dn2id_add 0x2fed7=
:<br>
"macaddress=3D1\2C6\2C00:00:00:00:00:01,ou=3Dxxx,ou=3Dxx,o=3Dxxxx"=
;<br>
Apr 23 20:37:44 ldap01 slapd2.4[32233]: slap_listener_activate(8):<br>
Apr 23 21:35:42 ldap01 slapd2.4[32233]: connection_close: conn=3D1025 sd=3D=
16<br>
Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=3D1022 sd=3D=
17<br>
Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=3D1037 sd=3D=
18<br>
Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=3D1038 sd=3D=
20<br>
Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=3D1039 sd=3D=
25<br>
<br>
Apr 24 12:02:23 ldap01 slapd2.4[32233]: daemon: shutdown requested and<br>
initiated. I have two Centos 5 servers running Openldap 2.4.22 with<br>
master/master syncrepl setup,<br>
Apr 24 12:02:23 ldap01 slapd2.4[32233]: connection_close: conn=3D1005 sd=3D=
21<br>
Apr 24 12:02:23 ldap01 slapd2.4[32233]: slapd shutdown: waiting for 59<br>
operations/tasks to finish<br>
Apr 24 12:02:34 ldap01 slapd2.4[26503]: bdb_db_open: database "":=
unclean<br>
shutdown detected; attempting recovery.<br>
Apr 24 12:02:35 ldap01 slapd2.4[26503]: slapd starting<br>
</blockquote>
<br></div>
Your config is a mess, with global directives and DB-specific directives in=
terleaved. That may not be the cause of any specific problems, but it shows=
sloppiness on the part of the sysadmins.<div><div></div><div class=3D"h5">
<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
---------------------- Relevant Config<br>
------------------------------------------------<br>
moduleload =C2=A0 =C2=A0 <a href=3D"
http://syncprov.la" target=3D"_blank">s=
yncprov.la</a><br>
TLSCertificateFile =C2=A0 =C2=A0 =C2=A0/etc/pki/tls/private/ldap.pem<br>
TLSCertificateKeyFile =C2=A0 /etc/pki/tls/private/ldap.pem<br>
TLSCACertificateFile =C2=A0 =C2=A0/etc/pki/tls/private/ldap.pem<br>
<br>
serverID =C2=A0 =C2=A0 =C2=A0 =C2=A0001<br>
database =C2=A0 =C2=A0 =C2=A0 =C2=A0bdb<br>
sizelimit =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A050000<br>
cachesize 10000<br>
checkpoint 256 5<br>
<br>
<br>
syncrepl =C2=A0 =C2=A0 =C2=A0 =C2=A0rid=3D001<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 provider=3Dldap://=
<a href=3D"
http://10.10.10.10:389" target=3D"_blank">10.10.10.10:389</a><br=
>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bindmethod=3Dsimpl=
e<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 binddn=3D"uid=
=3Dxxxxrep,ou=3Dxxxxxx,o=3Dxxxxxx"<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 credentials=3DMyPa=
ssword<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 searchbase=3D"=
;o=3Dxxxx"<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 schemachecking=3Do=
n<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sizelimit=3D"=
unlimited"<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 timelimit=3D"=
unlimited"<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 type=3DrefreshAndP=
ersist<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 interval=3D00:00:0=
0:10<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 retry=3D"5 5 =
60 +"<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 attrs=3D"*,+&=
quot;<br>
<br>
mirrormode on<br>
overlay syncprov<br>
syncprov-checkpoint =C2=A0 =C2=A0 100 =C2=A0 =C2=A0 10<br>
syncprov-sessionlog =C2=A0 =C2=A0 100<br>
<br>
idletimeout 3600<br>
threads 32<br>
<br>
<br>
</blockquote></div></div><font color=3D"#888888">
-- <br>
=C2=A0-- Howard Chu<br>
=C2=A0CTO, Symas Corp. =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"http:=
//www.symas.com" target=3D"_blank">
http://www.symas.com</a><br>
=C2=A0Director, Highland Sun =C2=A0 =C2=A0 <a href=3D"
http://highlandsun.c=
om/hyc/" target=3D"_blank">
http://highlandsun.com/hyc/</a><br>
=C2=A0Chief Architect, OpenLDAP =C2=A0<a href=3D"
http://www.openldap.org/p=
roject/" target=3D"_blank">
http://www.openldap.org/project/</a><br>
</font></blockquote></div><br><br clear=3D"all"><br>-- <br>Regards<br><br>J=
arl<br><a href=3D"mailto:jarl@dallur.com">jarl@dallur.com</a><br>
</div>
--000e0cd48c428f954504a23eaa22--
jarl＠dallur.com