openldap-bugs October 2009

openldap-bugs@openldap.org

34 participants
223 discussions

Re: (ITS#6310) Slapd with pcache crashes under load
by karavelov＠spnet.net 06 Oct '09

06 Oct '09

Hello, I have somehow changed the setup: The first process is a ldap to sql server withpout pcache the config file is here: http://purgatory.spnet.net/~karavelov/abort/slapd.conf The second process is a pure cache that reads from the first process using unix domain socket. The config file is here: http://purgatory.spnet.net/~karavelov/abort/slapd-proxy.conf The sql backend runs without problems. But the proxy process crashes once or twice a day on assertion: The output of the process is here: http://purgatory.spnet.net/~karavelov/abort/log It was running in valgrind, so here is its output: http://purgatory.spnet.net/~karavelov/abort/vg The process has dumped a core. I have made a debugging session with it. Here it is: http://purgatory.spnet.net/~karavelov/abort/gdb I am not sure what to look for. The process aborts on the same asserrtion even it is not run in valgrind. May be I should try with libdb-4.8? Thanks in advance Luben

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by masarati＠aero.polimi.it 05 Oct '09

05 Oct '09

masarati(a)aero.polimi.it wrote: > Another quick check you could probably do relatively quickly is zero out > that "e" pointer before calling be_entry_get_rw() within refresh_merge(). Forget about it: it's already done from within be_entry_get_rw() itself. The fact you get a corrupted "e" sounds odd, though. It would be interesting to track how and where it is set. This should appear from valgrind's inspection when *e is dereferenced. p.

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by masarati＠aero.polimi.it 05 Oct '09

05 Oct '09

luben karavelov wrote: > masarati(a)aero.polimi.it wrote: >> >> Thanks for collecting this info. The valgrind output could be of some >> use, but unfortunately I don't have time right now to set up a working >> RDBMS and extensively debug things. I'll keep this on my todo list. >> >> You should please re-run valgrind with --num-callers=30 or more, because >> in some cases errors are in too nested functions to get a clear idea of >> whether the issue is caused by garbage fed by slapd/back-sql or by errors >> inside the RDBMS/ODBC layers. The fact that valgrind systematically >> complains about internals of the RDBMS/ODBC reading past the end of >> memory >> chunks malloc'ed by slapd could be related to passing some non-nul >> terminated bervals that are dealt with as strings. Having a longer call >> stack could help tracking those occurrences. However, those issues >> should >> not be critical, since there's no invalid writes. >> >> Also, you should walk through the list of attributes being returned, to >> provide a hint about whether back-sql is computing a screwed attrlist or >> so. Along the lines of your current gdb session, you should get to frame >> #5, refresh_merge() in pcache.c, and print *e->e_attrs, >> *e->e_attrs->a_desc, *e->e_attrs->a_vals[0]; then move to >> e->e_attrs->a_next and repeat the prints to the end of the list. The >> fact >> you get a value of "a" equal to 0x500000000 looks definitely odd to >> me, as >> that attr list should result from be_entry_get_rw(), which in turn should >> collect it from the local database. Unless valgrind reveals some oddity >> in back-sql, the behavior you notice should not depend on the specific >> remote database you're using, but rather from the local one. >> >> p. > > Hello, > Tomorrow I will make a setup with pure sql process and a pure pcache > daemon that reads from the first over unix domain socket. In this manner > it will be clear if the crashing part is related to back-sql and the > database drivers/ODBC manager or not. > > Meanwhile, you could find the requested debugging session here: > http://purgatory.spnet.net/~karavelov/attr_list/gdb-1 > > It seems that the "e" pointer is corrupted. Good catch. > Tomorrow I will start it > through valgrind with more back-frames as requested Another quick check you could probably do relatively quickly is zero out that "e" pointer before calling be_entry_get_rw() within refresh_merge(). Thanks, p.

1 0

Re: (ITS#6320)
by j＠telepaths.org 04 Oct '09

04 Oct '09

Michael, Yes, that has been a theory of mine - I am investigating the =20 possibility, however I should point out that we recently =20 decommissioned our old LDAP system, which used slapd on Etch - it uses =20= the same firewalls for out-of-zone-access, and was not affected by =20 this problem. However, I should also point out that since my last update, the =20 problem has not returned. Using a six-member multimaster mesh has =20 proven fruitful so far, but we'll see how it plays out over the next =20 couple of days. Thanks for your help ... On Oct 4, 2009, at 06:00 , Michael Str=F6der wrote: > j(a)telepaths.org wrote: >> On Oct 2, 2009, at 23:00 , Howard Chu wrote: >> >>> j(a)telepaths.org wrote: >>>> My last email was likely indecipherable, so re-sending as plain =20 >>>> text: >>>> >>>> ##### >>>> >>>> This may sound like a strange question, but couldn't seem to find =20= >>>> the >>>> answer in the docs: >>>> >>>> Would the global "idletimeout" parameter interfere with >>>> 'refreshAndPersist' operations in any way? >>> No, that would be stupid. >>> >> Yes Howard -- that would be incredibly stupid. But I had to ask. >> [..] >> SOME of the servers receiving updates are separated by firewalls, >> yes ... HOWEVER (a) there are special policies in place, allowing >> these hosts to communicate via LDAP[S]; this can be confirmed by >> running ldapsearch routines from every possible vantage point TO =20 >> every >> possible provider, etc., > > Note that firewalls are doing connection tracking enforcing their =20 > own timeouts > for idle connections. So please check whether the firewall =20 > configuration > really matches your slapd idle timeout requirements. > > Ciao, Michael.

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by karavelov＠spnet.net 04 Oct '09

04 Oct '09

masarati(a)aero.polimi.it wrote: > > Thanks for collecting this info. The valgrind output could be of some > use, but unfortunately I don't have time right now to set up a working > RDBMS and extensively debug things. I'll keep this on my todo list. > > You should please re-run valgrind with --num-callers=30 or more, because > in some cases errors are in too nested functions to get a clear idea of > whether the issue is caused by garbage fed by slapd/back-sql or by errors > inside the RDBMS/ODBC layers. The fact that valgrind systematically > complains about internals of the RDBMS/ODBC reading past the end of memory > chunks malloc'ed by slapd could be related to passing some non-nul > terminated bervals that are dealt with as strings. Having a longer call > stack could help tracking those occurrences. However, those issues should > not be critical, since there's no invalid writes. > > Also, you should walk through the list of attributes being returned, to > provide a hint about whether back-sql is computing a screwed attrlist or > so. Along the lines of your current gdb session, you should get to frame > #5, refresh_merge() in pcache.c, and print *e->e_attrs, > *e->e_attrs->a_desc, *e->e_attrs->a_vals[0]; then move to > e->e_attrs->a_next and repeat the prints to the end of the list. The fact > you get a value of "a" equal to 0x500000000 looks definitely odd to me, as > that attr list should result from be_entry_get_rw(), which in turn should > collect it from the local database. Unless valgrind reveals some oddity > in back-sql, the behavior you notice should not depend on the specific > remote database you're using, but rather from the local one. > > p. Hello, Tomorrow I will make a setup with pure sql process and a pure pcache daemon that reads from the first over unix domain socket. In this manner it will be clear if the crashing part is related to back-sql and the database drivers/ODBC manager or not. Meanwhile, you could find the requested debugging session here: http://purgatory.spnet.net/~karavelov/attr_list/gdb-1 It seems that the "e" pointer is corrupted. Tomorrow I will start it through valgrind with more back-frames as requested Best regards Luben

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by masarati＠aero.polimi.it 04 Oct '09

04 Oct '09

> hyc(a)symas.com wrote: >> I haven't touched back-sql in quite a while, and don't have a suitable >> config >> for testing this. I suggest you run under valgrind and see what errors >> it reports. > > Hello, > > I have run the server through valgrind. It crashed on the same structure > on different location. The gdb session could be found here: > http://purgatory.spnet.net/~karavelov/attr_list/gdb > > The output from valgrind could be found here: > http://purgatory.spnet.net/~karavelov/attr_list/vg > > In it the lines up to 2833 are output on server start. You could find > the address 0x500000000 (found from gdb session) around line 3180. > > I could see nothing interesting in the log file. > > I hope this info will help you. If I could gather more information, just > say. Thanks for collecting this info. The valgrind output could be of some use, but unfortunately I don't have time right now to set up a working RDBMS and extensively debug things. I'll keep this on my todo list. You should please re-run valgrind with --num-callers=30 or more, because in some cases errors are in too nested functions to get a clear idea of whether the issue is caused by garbage fed by slapd/back-sql or by errors inside the RDBMS/ODBC layers. The fact that valgrind systematically complains about internals of the RDBMS/ODBC reading past the end of memory chunks malloc'ed by slapd could be related to passing some non-nul terminated bervals that are dealt with as strings. Having a longer call stack could help tracking those occurrences. However, those issues should not be critical, since there's no invalid writes. Also, you should walk through the list of attributes being returned, to provide a hint about whether back-sql is computing a screwed attrlist or so. Along the lines of your current gdb session, you should get to frame #5, refresh_merge() in pcache.c, and print *e->e_attrs, *e->e_attrs->a_desc, *e->e_attrs->a_vals[0]; then move to e->e_attrs->a_next and repeat the prints to the end of the list. The fact you get a value of "a" equal to 0x500000000 looks definitely odd to me, as that attr list should result from be_entry_get_rw(), which in turn should collect it from the local database. Unless valgrind reveals some oddity in back-sql, the behavior you notice should not depend on the specific remote database you're using, but rather from the local one. p.

1 0

Re: (ITS#6320)
by michael＠stroeder.com 04 Oct '09

04 Oct '09

j(a)telepaths.org wrote: > On Oct 2, 2009, at 23:00 , Howard Chu wrote: > >> j(a)telepaths.org wrote: >>> My last email was likely indecipherable, so re-sending as plain text: >>> >>> ##### >>> >>> This may sound like a strange question, but couldn't seem to find the >>> answer in the docs: >>> >>> Would the global "idletimeout" parameter interfere with >>> 'refreshAndPersist' operations in any way? >> No, that would be stupid. >> > Yes Howard -- that would be incredibly stupid. But I had to ask. > [..] > SOME of the servers receiving updates are separated by firewalls, > yes ... HOWEVER (a) there are special policies in place, allowing > these hosts to communicate via LDAP[S]; this can be confirmed by > running ldapsearch routines from every possible vantage point TO every > possible provider, etc., Note that firewalls are doing connection tracking enforcing their own timeouts for idle connections. So please check whether the firewall configuration really matches your slapd idle timeout requirements. Ciao, Michael.

1 0

Re: (ITS#6320)
by j＠telepaths.org 04 Oct '09

04 Oct '09

Pierangelo, I offered to do post my configuration in my initial post. No one accepted my offer. Since you are now asking for it, I will gladly post it below. Two, thanks for the hint about editing passwords, I can assure you that all confidential data posted below will have been tainted *somehow*. I should also point out that today I made a change to my infrastructure that I hope will help the situation. Since I've noticed that most of the DIT discrepancies were limited to the standard Consumer boxes, and not the Providers, I have decided to do away entirely with the the standard Consumers. We now have six (6) virtually-identically configured Providers whom all replicate with the other five (5) respective hybrid Consumer/Providers. Essentially a six member multimaster mesh, all of whom can contact all of the other members perfectly via both LDAP and LDAPS. Here is a slapd.conf from ONE of the SIX members: ##### include /etc/ldap/schema/core.schema include /etc/ldap/schema/cosine.schema include /etc/ldap/schema/nis.schema include /etc/ldap/schema/inetorgperson.schema include /etc/ldap/schema/misc.schema include /etc/ldap/schema/openldap.schema include /etc/ldap/schema/duaconf.schema include /etc/ldap/schema/dyngroup.schema include /etc/ldap/schema/ppolicy.schema include /etc/ldap/schema/sudo.schema include /etc/ldap/schema/dhcp.schema include /etc/ldap/schema/samba.schema include /usr/share/doc/libpam-ldap/ldapns.schema include /etc/ldap/schema/hdb.schema include /etc/ldap/schema/uber.schema pidfile /var/run/slapd/slapd.pid argsfile /var/run/slapd/slapd.args tool-threads 4 loglevel none modulepath /usr/lib/ldap moduleload back_hdb moduleload back_relay moduleload rwm.la moduleload back_monitor.la moduleload syncprov moduleload accesslog serverID 100 ldaps://10.64.100.100:636/ serverID 107 ldaps://10.64.100.107:636/ serverID 108 ldaps://10.64.100.108:636/ serverID 811 ldaps://10.9.8.11:636/ serverID 812 ldaps://10.9.8.12:636/ serverID 814 ldaps://10.9.8.14:636/ TLSCertificateFile /etc/ldap/ssl/wildcard.site.example.com.crt TLSCertificateKeyFile /etc/ldap/ssl/wildcard.site.example.com.key TLSCACertificateFile /etc/ssl/certs/ca.cert disallow bind_anon sizelimit unlimited timelimit unlimited security tls=0 access to dn.subtree="cn=Subschema" by users read by * none stop access to dn.base="" by users read by * none stop defaultSearchBase dc=real,dc=example,dc=com sasl-realm SITE.EXAMPLE.COM sasl-host ds.site.example.com #sasl-secprops minssf=0 authz-regexp "uid=$.*$,cn=SITE.EXAMPLE.COM,cn=gssapi,cn=auth" "uid=$1,cn=plain,cn=auth,dc=site,dc=example,dc=com" authz-regexp "gidNumber=\\\0+uidNumber=\\ \0,cn=peercred,cn=external,cn=auth" "uid=writer,cn=plain,cn=auth,dc=real,dc=example,dc=com" backend hdb ########### Monitoring Database - For slapd/hdb performance data database monitor rootdn uid=monitor,cn=monitor rootpw {SSHA}encrypted-hash access to dn.subtree="cn=monitor" by group/groupOfUniqueNames/ uniqueMember ="cn=ldapadmin,cn=ldap,cn=groups,dc=real,dc=example,dc=com" read by dn.exact="uid=rootdn,cn=plain,cn=auth,dc=real,dc=example,dc=com" read ########### example.Log database hdb suffix cn=log rootdn "uid=log,cn=log" rootpw {SSHA}encrypted-hash directory /var/lib/ldap/log index reqStart,objectClass,entryCSN,reqResult eq dbconfig set_cachesize 0 2097152 0 overlay syncprov syncprov-nopresent TRUE syncprov-reloadhint TRUE access to dn.subtree="cn=log" by group/groupOfUniqueNames/ uniqueMember ="cn=ldapadmin,cn=ldap,cn=groups,dc=real,dc=example,dc=com" read by dn.base="uid=rootdn,cn=plain,cn=auth,dc=real,dc=example,dc=com" read by dn.base="uid=log,cn=log" read ########### example.real database hdb cachesize 10000 idlcachesize 30000 suffix "dc=real,dc=example,dc=com" checksum checkpoint 100 10 cachefree 20 rootdn "uid=rootdn,cn=plain,cn=auth,dc=real,dc=example,dc=com" rootpw {SSHA}encrypted-hash monitoring on directory "/var/lib/ldap/real" dncachesize 100000 dbconfig set_cachesize 1 0 2 dbconfig set_lg_max 10485760 dbconfig set_flags db_log_autoremove dbconfig set_lg_bsize 2097152 dbconfig set_lk_max_objects 1500 dbconfig set_lk_max_locks 1500 dbconfig set_lk_max_lockers 1500 index objectClass,structuralObjectClass eq index entryCSN,entryUUID eq index cn,uid,memberUid eq syncrepl rid=001 provider=ldaps://10.64.100.100:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist syncrepl rid=002 provider=ldaps://10.64.100.107:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist syncrepl rid=003 provider=ldaps://10.64.100.108:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist syncrepl rid=004 provider=ldaps://10.9.8.14:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist syncrepl rid=005 provider=ldaps://10.9.8.11:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist mirrormode true overlay syncprov syncprov-reloadhint TRUE #syncprov-checkpoint 10 5 syncprov-sessionlog 5000 overlay accesslog logdb cn=log logops writes logpurge 7+00:00 2+00:00 logsuccess TRUE include /etc/ldap/acls include /etc/ldap/relays #### Thanks again Jeff

1 0

Re: (ITS#6320)
by masarati＠aero.polimi.it 04 Oct '09

04 Oct '09

> OK, maybe I wasn't clear enough, so I'll rephrase my initial question: > > Is it normal for a 'slapd' server to ONLY correct itself via a restart > of the 'slapd' process? > > I asked because of my earlier findings today. Since waiting for the > system to correct the discrepancy in the DIT by itself proved to be a > futile exercise, and since a manual restart produced the results, it > seemed as though I was observing suboptimal behavior, and was asking > if this was indicative of something else. What about posting your *EXACT* configuration - not a snapshot of it, not a verbal description of how you defined it. Sometimes the devil is in the details. Only make sure you remove passwords, assuming they're correctly spelled in the original. p.

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by karavelov＠spnet.net 03 Oct '09

03 Oct '09

hyc(a)symas.com wrote: > I haven't touched back-sql in quite a while, and don't have a suitable config > for testing this. I suggest you run under valgrind and see what errors it reports. Hello, I have run the server through valgrind. It crashed on the same structure on different location. The gdb session could be found here: http://purgatory.spnet.net/~karavelov/attr_list/gdb The output from valgrind could be found here: http://purgatory.spnet.net/~karavelov/attr_list/vg In it the lines up to 2833 are output on server start. You could find the address 0x500000000 (found from gdb session) around line 3180. I could see nothing interesting in the log file. I hope this info will help you. If I could gather more information, just say. Luben

1 0

← Newer
1
...
16
17
18
19
20
21
22
23
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs October 2009