openldap-bugs

openldap-bugs@openldap.org

1 participants
20959 discussions

Re: (ITS#6320)
by j＠telepaths.org 05 Oct '09

05 Oct '09

Michael, Yes, that has been a theory of mine - I am investigating the =20 possibility, however I should point out that we recently =20 decommissioned our old LDAP system, which used slapd on Etch - it uses =20= the same firewalls for out-of-zone-access, and was not affected by =20 this problem. However, I should also point out that since my last update, the =20 problem has not returned. Using a six-member multimaster mesh has =20 proven fruitful so far, but we'll see how it plays out over the next =20 couple of days. Thanks for your help ... On Oct 4, 2009, at 06:00 , Michael Str=F6der wrote: > j(a)telepaths.org wrote: >> On Oct 2, 2009, at 23:00 , Howard Chu wrote: >> >>> j(a)telepaths.org wrote: >>>> My last email was likely indecipherable, so re-sending as plain =20 >>>> text: >>>> >>>> ##### >>>> >>>> This may sound like a strange question, but couldn't seem to find =20= >>>> the >>>> answer in the docs: >>>> >>>> Would the global "idletimeout" parameter interfere with >>>> 'refreshAndPersist' operations in any way? >>> No, that would be stupid. >>> >> Yes Howard -- that would be incredibly stupid. But I had to ask. >> [..] >> SOME of the servers receiving updates are separated by firewalls, >> yes ... HOWEVER (a) there are special policies in place, allowing >> these hosts to communicate via LDAP[S]; this can be confirmed by >> running ldapsearch routines from every possible vantage point TO =20 >> every >> possible provider, etc., > > Note that firewalls are doing connection tracking enforcing their =20 > own timeouts > for idle connections. So please check whether the firewall =20 > configuration > really matches your slapd idle timeout requirements. > > Ciao, Michael.

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by karavelov＠spnet.net 04 Oct '09

04 Oct '09

masarati(a)aero.polimi.it wrote: > > Thanks for collecting this info. The valgrind output could be of some > use, but unfortunately I don't have time right now to set up a working > RDBMS and extensively debug things. I'll keep this on my todo list. > > You should please re-run valgrind with --num-callers=30 or more, because > in some cases errors are in too nested functions to get a clear idea of > whether the issue is caused by garbage fed by slapd/back-sql or by errors > inside the RDBMS/ODBC layers. The fact that valgrind systematically > complains about internals of the RDBMS/ODBC reading past the end of memory > chunks malloc'ed by slapd could be related to passing some non-nul > terminated bervals that are dealt with as strings. Having a longer call > stack could help tracking those occurrences. However, those issues should > not be critical, since there's no invalid writes. > > Also, you should walk through the list of attributes being returned, to > provide a hint about whether back-sql is computing a screwed attrlist or > so. Along the lines of your current gdb session, you should get to frame > #5, refresh_merge() in pcache.c, and print *e->e_attrs, > *e->e_attrs->a_desc, *e->e_attrs->a_vals[0]; then move to > e->e_attrs->a_next and repeat the prints to the end of the list. The fact > you get a value of "a" equal to 0x500000000 looks definitely odd to me, as > that attr list should result from be_entry_get_rw(), which in turn should > collect it from the local database. Unless valgrind reveals some oddity > in back-sql, the behavior you notice should not depend on the specific > remote database you're using, but rather from the local one. > > p. Hello, Tomorrow I will make a setup with pure sql process and a pure pcache daemon that reads from the first over unix domain socket. In this manner it will be clear if the crashing part is related to back-sql and the database drivers/ODBC manager or not. Meanwhile, you could find the requested debugging session here: http://purgatory.spnet.net/~karavelov/attr_list/gdb-1 It seems that the "e" pointer is corrupted. Tomorrow I will start it through valgrind with more back-frames as requested Best regards Luben

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by masarati＠aero.polimi.it 04 Oct '09

04 Oct '09

> hyc(a)symas.com wrote: >> I haven't touched back-sql in quite a while, and don't have a suitable >> config >> for testing this. I suggest you run under valgrind and see what errors >> it reports. > > Hello, > > I have run the server through valgrind. It crashed on the same structure > on different location. The gdb session could be found here: > http://purgatory.spnet.net/~karavelov/attr_list/gdb > > The output from valgrind could be found here: > http://purgatory.spnet.net/~karavelov/attr_list/vg > > In it the lines up to 2833 are output on server start. You could find > the address 0x500000000 (found from gdb session) around line 3180. > > I could see nothing interesting in the log file. > > I hope this info will help you. If I could gather more information, just > say. Thanks for collecting this info. The valgrind output could be of some use, but unfortunately I don't have time right now to set up a working RDBMS and extensively debug things. I'll keep this on my todo list. You should please re-run valgrind with --num-callers=30 or more, because in some cases errors are in too nested functions to get a clear idea of whether the issue is caused by garbage fed by slapd/back-sql or by errors inside the RDBMS/ODBC layers. The fact that valgrind systematically complains about internals of the RDBMS/ODBC reading past the end of memory chunks malloc'ed by slapd could be related to passing some non-nul terminated bervals that are dealt with as strings. Having a longer call stack could help tracking those occurrences. However, those issues should not be critical, since there's no invalid writes. Also, you should walk through the list of attributes being returned, to provide a hint about whether back-sql is computing a screwed attrlist or so. Along the lines of your current gdb session, you should get to frame #5, refresh_merge() in pcache.c, and print *e->e_attrs, *e->e_attrs->a_desc, *e->e_attrs->a_vals[0]; then move to e->e_attrs->a_next and repeat the prints to the end of the list. The fact you get a value of "a" equal to 0x500000000 looks definitely odd to me, as that attr list should result from be_entry_get_rw(), which in turn should collect it from the local database. Unless valgrind reveals some oddity in back-sql, the behavior you notice should not depend on the specific remote database you're using, but rather from the local one. p.

1 0

Re: (ITS#6320)
by michael＠stroeder.com 04 Oct '09

04 Oct '09

j(a)telepaths.org wrote: > On Oct 2, 2009, at 23:00 , Howard Chu wrote: > >> j(a)telepaths.org wrote: >>> My last email was likely indecipherable, so re-sending as plain text: >>> >>> ##### >>> >>> This may sound like a strange question, but couldn't seem to find the >>> answer in the docs: >>> >>> Would the global "idletimeout" parameter interfere with >>> 'refreshAndPersist' operations in any way? >> No, that would be stupid. >> > Yes Howard -- that would be incredibly stupid. But I had to ask. > [..] > SOME of the servers receiving updates are separated by firewalls, > yes ... HOWEVER (a) there are special policies in place, allowing > these hosts to communicate via LDAP[S]; this can be confirmed by > running ldapsearch routines from every possible vantage point TO every > possible provider, etc., Note that firewalls are doing connection tracking enforcing their own timeouts for idle connections. So please check whether the firewall configuration really matches your slapd idle timeout requirements. Ciao, Michael.

1 0

Re: (ITS#6320)
by j＠telepaths.org 04 Oct '09

04 Oct '09

Pierangelo, I offered to do post my configuration in my initial post. No one accepted my offer. Since you are now asking for it, I will gladly post it below. Two, thanks for the hint about editing passwords, I can assure you that all confidential data posted below will have been tainted *somehow*. I should also point out that today I made a change to my infrastructure that I hope will help the situation. Since I've noticed that most of the DIT discrepancies were limited to the standard Consumer boxes, and not the Providers, I have decided to do away entirely with the the standard Consumers. We now have six (6) virtually-identically configured Providers whom all replicate with the other five (5) respective hybrid Consumer/Providers. Essentially a six member multimaster mesh, all of whom can contact all of the other members perfectly via both LDAP and LDAPS. Here is a slapd.conf from ONE of the SIX members: ##### include /etc/ldap/schema/core.schema include /etc/ldap/schema/cosine.schema include /etc/ldap/schema/nis.schema include /etc/ldap/schema/inetorgperson.schema include /etc/ldap/schema/misc.schema include /etc/ldap/schema/openldap.schema include /etc/ldap/schema/duaconf.schema include /etc/ldap/schema/dyngroup.schema include /etc/ldap/schema/ppolicy.schema include /etc/ldap/schema/sudo.schema include /etc/ldap/schema/dhcp.schema include /etc/ldap/schema/samba.schema include /usr/share/doc/libpam-ldap/ldapns.schema include /etc/ldap/schema/hdb.schema include /etc/ldap/schema/uber.schema pidfile /var/run/slapd/slapd.pid argsfile /var/run/slapd/slapd.args tool-threads 4 loglevel none modulepath /usr/lib/ldap moduleload back_hdb moduleload back_relay moduleload rwm.la moduleload back_monitor.la moduleload syncprov moduleload accesslog serverID 100 ldaps://10.64.100.100:636/ serverID 107 ldaps://10.64.100.107:636/ serverID 108 ldaps://10.64.100.108:636/ serverID 811 ldaps://10.9.8.11:636/ serverID 812 ldaps://10.9.8.12:636/ serverID 814 ldaps://10.9.8.14:636/ TLSCertificateFile /etc/ldap/ssl/wildcard.site.example.com.crt TLSCertificateKeyFile /etc/ldap/ssl/wildcard.site.example.com.key TLSCACertificateFile /etc/ssl/certs/ca.cert disallow bind_anon sizelimit unlimited timelimit unlimited security tls=0 access to dn.subtree="cn=Subschema" by users read by * none stop access to dn.base="" by users read by * none stop defaultSearchBase dc=real,dc=example,dc=com sasl-realm SITE.EXAMPLE.COM sasl-host ds.site.example.com #sasl-secprops minssf=0 authz-regexp "uid=$.*$,cn=SITE.EXAMPLE.COM,cn=gssapi,cn=auth" "uid=$1,cn=plain,cn=auth,dc=site,dc=example,dc=com" authz-regexp "gidNumber=\\\0+uidNumber=\\ \0,cn=peercred,cn=external,cn=auth" "uid=writer,cn=plain,cn=auth,dc=real,dc=example,dc=com" backend hdb ########### Monitoring Database - For slapd/hdb performance data database monitor rootdn uid=monitor,cn=monitor rootpw {SSHA}encrypted-hash access to dn.subtree="cn=monitor" by group/groupOfUniqueNames/ uniqueMember ="cn=ldapadmin,cn=ldap,cn=groups,dc=real,dc=example,dc=com" read by dn.exact="uid=rootdn,cn=plain,cn=auth,dc=real,dc=example,dc=com" read ########### example.Log database hdb suffix cn=log rootdn "uid=log,cn=log" rootpw {SSHA}encrypted-hash directory /var/lib/ldap/log index reqStart,objectClass,entryCSN,reqResult eq dbconfig set_cachesize 0 2097152 0 overlay syncprov syncprov-nopresent TRUE syncprov-reloadhint TRUE access to dn.subtree="cn=log" by group/groupOfUniqueNames/ uniqueMember ="cn=ldapadmin,cn=ldap,cn=groups,dc=real,dc=example,dc=com" read by dn.base="uid=rootdn,cn=plain,cn=auth,dc=real,dc=example,dc=com" read by dn.base="uid=log,cn=log" read ########### example.real database hdb cachesize 10000 idlcachesize 30000 suffix "dc=real,dc=example,dc=com" checksum checkpoint 100 10 cachefree 20 rootdn "uid=rootdn,cn=plain,cn=auth,dc=real,dc=example,dc=com" rootpw {SSHA}encrypted-hash monitoring on directory "/var/lib/ldap/real" dncachesize 100000 dbconfig set_cachesize 1 0 2 dbconfig set_lg_max 10485760 dbconfig set_flags db_log_autoremove dbconfig set_lg_bsize 2097152 dbconfig set_lk_max_objects 1500 dbconfig set_lk_max_locks 1500 dbconfig set_lk_max_lockers 1500 index objectClass,structuralObjectClass eq index entryCSN,entryUUID eq index cn,uid,memberUid eq syncrepl rid=001 provider=ldaps://10.64.100.100:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist syncrepl rid=002 provider=ldaps://10.64.100.107:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist syncrepl rid=003 provider=ldaps://10.64.100.108:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist syncrepl rid=004 provider=ldaps://10.9.8.14:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist syncrepl rid=005 provider=ldaps://10.9.8.11:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist mirrormode true overlay syncprov syncprov-reloadhint TRUE #syncprov-checkpoint 10 5 syncprov-sessionlog 5000 overlay accesslog logdb cn=log logops writes logpurge 7+00:00 2+00:00 logsuccess TRUE include /etc/ldap/acls include /etc/ldap/relays #### Thanks again Jeff

1 0

Re: (ITS#6320)
by masarati＠aero.polimi.it 04 Oct '09

04 Oct '09

> OK, maybe I wasn't clear enough, so I'll rephrase my initial question: > > Is it normal for a 'slapd' server to ONLY correct itself via a restart > of the 'slapd' process? > > I asked because of my earlier findings today. Since waiting for the > system to correct the discrepancy in the DIT by itself proved to be a > futile exercise, and since a manual restart produced the results, it > seemed as though I was observing suboptimal behavior, and was asking > if this was indicative of something else. What about posting your *EXACT* configuration - not a snapshot of it, not a verbal description of how you defined it. Sometimes the devil is in the details. Only make sure you remove passwords, assuming they're correctly spelled in the original. p.

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by karavelov＠spnet.net 03 Oct '09

03 Oct '09

hyc(a)symas.com wrote: > I haven't touched back-sql in quite a while, and don't have a suitable config > for testing this. I suggest you run under valgrind and see what errors it reports. Hello, I have run the server through valgrind. It crashed on the same structure on different location. The gdb session could be found here: http://purgatory.spnet.net/~karavelov/attr_list/gdb The output from valgrind could be found here: http://purgatory.spnet.net/~karavelov/attr_list/vg In it the lines up to 2833 are output on server start. You could find the address 0x500000000 (found from gdb session) around line 3180. I could see nothing interesting in the log file. I hope this info will help you. If I could gather more information, just say. Luben

1 0

Re: (ITS#6320)
by j＠telepaths.org 03 Oct '09

03 Oct '09

OK, maybe I wasn't clear enough, so I'll rephrase my initial question: Is it normal for a 'slapd' server to ONLY correct itself via a restart of the 'slapd' process? I asked because of my earlier findings today. Since waiting for the system to correct the discrepancy in the DIT by itself proved to be a futile exercise, and since a manual restart produced the results, it seemed as though I was observing suboptimal behavior, and was asking if this was indicative of something else. On Oct 2, 2009, at 23:02 , Howard Chu wrote: > j(a)telepaths.org wrote: >> IMPORTANT UPDATE: >> >> Is it normal for a 'slapd' server to correct itself via a restart of >> the slapd process? Currently, I have one Consumer that is 2-records >> behind in Synchronization - I restarted his process and the records >> immediately were replicated. This normally is NOT standard behavior. >> >> Can anyone explain? Is this expected? Indicative of another problem? > > That's what the "refresh" in "refreshAndPersist" is about. Any > freshly started consumer will automatically sync up to the current > state of the provider. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6320)
by j＠telepaths.org 03 Oct '09

03 Oct '09

On Oct 2, 2009, at 23:00 , Howard Chu wrote: > j(a)telepaths.org wrote: >> My last email was likely indecipherable, so re-sending as plain text: >> >> ##### >> >> This may sound like a strange question, but couldn't seem to find the >> answer in the docs: >> >> Would the global "idletimeout" parameter interfere with >> 'refreshAndPersist' operations in any way? > > No, that would be stupid. > Yes Howard -- that would be incredibly stupid. But I had to ask. >> I ask because I have had yet another instance where my three >> production consumers did not get a series of updates that it should >> have. This was hours after successfully adding an accesslog database >> to all Provider servers in question. >> >> When querying the log database: >> >> ldapsearch -b cn=log -xD uid=log,cn=log -w logpassword -s sub -H ldap://ldap-provider-in-question >> >> I see that the records-in-question were in fact updated as they >> should >> have .... but there's absolutely no information hinting towards WHY a >> remote server would fail to get updates (especially when all LDAP >> traffic in between the said-hosts is unrestricted and unmoderated). > > Are you sure? No firewalls in the way? > SOME of the servers receiving updates are separated by firewalls, yes ... HOWEVER (a) there are special policies in place, allowing these hosts to communicate via LDAP[S]; this can be confirmed by running ldapsearch routines from every possible vantage point TO every possible provider, etc., and (b) some of the replication issues observed occurred when all affected hosts (one or more providers and its consumer clients) were in the same local subnet, at which point the firewall is immaterial. I had asked my stupid question about the idletimeout, because this behavior doesn't strike me as simply "not working". Rather, it begins to work .... then just stops. So my next thought was that perhaps the firewall was causing the connections to time-out. But if that were true ... wouldn't both the "retry" and the "timeout" parameters compensate for this? I mean, the man page certainly doesn't contradict my assumption ... Jeff >> Hence why I began to scrutinize the 'idletimeout' parameter >> (previously set to 45 seconds, now has been removed entirely from all >> Providers AND Consumers alike). >> >> Still looking at, testing and contemplating this whole situation ... >> I'll keep you posted, but am curious if anyone has any input to my >> question above, thanks. >> >> ##### >> >> I am officially out of ideas - synchronization is entirely >> unreliable. >> Need guidance ASAP. Thanks ..... > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6320)
by hyc＠symas.com 03 Oct '09

03 Oct '09

j(a)telepaths.org wrote: > IMPORTANT UPDATE: > > Is it normal for a 'slapd' server to correct itself via a restart of > the slapd process? Currently, I have one Consumer that is 2-records > behind in Synchronization - I restarted his process and the records > immediately were replicated. This normally is NOT standard behavior. > > Can anyone explain? Is this expected? Indicative of another problem? That's what the "refresh" in "refreshAndPersist" is about. Any freshly started consumer will automatically sync up to the current state of the provider. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs