- openldap-bugs - openldap.org

Re: (ITS#6310) Slapd with pcache crashes under load
by karavelov＠spnet.net 07 Oct '09

07 Oct '09

masarati(a)aero.polimi.it wrote: > No. Some overlay along the lines apparently mucks with e_private in the > entry that back-ldap is sending. It's probably slapo-pcache. I need to > debug this. You say it happens seldom. Can you recognize anything > special in the operation that triggers the assertion failure? I mean: > either in terms of special data, or in terms of special state of the > cache? This could help narrow down some means to reproduce the issue. > > Thanks, p. > > > What is interesting is that most of the logs are something like this one: conn=6891 op=1 SRCH attr=spamassassin query template of incoming query = (&(objectClass=)(cn=)) Entering QC, querystr = (&(objectClass=mailUser)(cn=nobody)) Lock QC index = 0x7e0a7b8 QUERY ANSWERABLE (answered 3267 times) conn=6891 op=1 SEARCH RESULT tag=101 err=0 nentries=0 text= conn=6891 fd=32 closed (connection lost) query template of incoming query = (&(objectClass=)(uid=)) QUERY NOT ANSWERABLE QUERY NOT CACHEABLE query template of incoming query = (&(objectClass=)(uid=)) QUERY NOT ANSWERABLE QUERY NOT CACHEABLE query template of incoming query = (&(objectClass=)(uid=)) QUERY NOT ANSWERABLE QUERY NOT CACHEABLE query template of incoming query = (&(objectClass=)(uid=)) QUERY NOT ANSWERABLE QUERY NOT CACHEABLE query template of incoming query = (&(objectClass=)(uid=)) QUERY NOT ANSWERABLE QUERY NOT CACHEABLE conn=3573 op=812 SRCH base="dc=ciela.net,dc=hosting,dc=spnet,dc=net" scope=1 deref=0 filter="(&(objectClass=mailUser)(uid=diana))" conn=3573 op=812 SRCH attr=uid userPassword mailMessageStore uidNumber gidNumber query template of incoming query = (&(objectClass=)(uid=)) Entering QC, querystr = (&(objectClass=mailUser)(uid=diana)) Lock QC index = 0x7ee9fe8 Base of added query = dc=ciela.net,dc=hosting,dc=spnet,dc=net QUERY ANSWERABLE (answered 63 times) conn=3573 op=812 SEARCH RESULT tag=101 err=0 nentries=1 text= slapd: /home/luben/openldap-2.4.18-h/servers/slapd/entry.c:483: entry_clean: Assertion `e->e_private == ((void *)0)' failed. This sequence of : query template of incoming query = (&(objectClass=)(uid=)) QUERY NOT ANSWERABLE QUERY NOT CACHEABLE is something I could not explain. There is not a log entry for incomming search. I have restarted the cache with more logging (stats+trace+pcache) to see from where they come. I have some core dumps and if you tell me for what to look for, tomorrow I will debug them and see if there is a common pattern. Luben

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by masarati＠aero.polimi.it 06 Oct '09

06 Oct '09

> Hello, > I have somehow changed the setup: > > The first process is a ldap to sql server withpout pcache > the config file is here: > http://purgatory.spnet.net/~karavelov/abort/slapd.conf > > The second process is a pure cache that reads from the first process > using unix domain socket. The config file is here: > http://purgatory.spnet.net/~karavelov/abort/slapd-proxy.conf > > The sql backend runs without problems. But the proxy process crashes > once or twice a day on assertion: > > The output of the process is here: > http://purgatory.spnet.net/~karavelov/abort/log > > It was running in valgrind, so here is its output: > http://purgatory.spnet.net/~karavelov/abort/vg > > The process has dumped a core. I have made a debugging session with it. > Here it is: > http://purgatory.spnet.net/~karavelov/abort/gdb > > I am not sure what to look for. The process aborts on the same > asserrtion even it is not run in valgrind. No. Some overlay along the lines apparently mucks with e_private in the entry that back-ldap is sending. It's probably slapo-pcache. I need to debug this. You say it happens seldom. Can you recognize anything special in the operation that triggers the assertion failure? I mean: either in terms of special data, or in terms of special state of the cache? This could help narrow down some means to reproduce the issue. Thanks, p.

1 0

Re: (ITS#6324) slapd -d ? emits incorrect value for LDAP_DEBUG_ANY on x86_64
by masarati＠aero.polimi.it 06 Oct '09

06 Oct '09

> The 32-bit value 4294967295 for LDAP_DEBUG_ANY is also rejected. > lutil_atoix() declares i as type long and tries to check for overflow(?) > with this cast: > > if ( (long)(int)i != i ) { > > which always fails in this case on x86_64. lutil_atoux() might be the more > appropriate choice here. > > str2loglevel() seems to behave correctly by virtue of verb_to_mask() > returning an index into loglevel_ops, but I haven't checked extensively. > loglevel_init(), however, does specify a hardcoded -1 for the Any level > instead of LDAP_DEBUG_ANY. Please check the fix in HEAD. I can't check right now on x86_64, but I'll probably do it tomorrow. Apparently, I didn't break anything on 32 bit. Thanks, p.

1 0

Re: (ITS#6324) slapd -d ? emits incorrect value for LDAP_DEBUG_ANY on x86_64
by jwm＠horde.net 06 Oct '09

06 Oct '09

The 32-bit value 4294967295 for LDAP_DEBUG_ANY is also rejected. lutil_atoix() declares i as type long and tries to check for overflow(?) with this cast: if ( (long)(int)i != i ) { which always fails in this case on x86_64. lutil_atoux() might be the more appropriate choice here. str2loglevel() seems to behave correctly by virtue of verb_to_mask() returning an index into loglevel_ops, but I haven't checked extensively. loglevel_init(), however, does specify a hardcoded -1 for the Any level instead of LDAP_DEBUG_ANY. john -- John Morrissey _o /\ ---- __o jwm(a)horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

1 0

(ITS#6324) slapd -d ? emits incorrect value for LDAP_DEBUG_ANY on x86_64
by jwm＠horde.net 06 Oct '09

06 Oct '09

Full_Name: John Morrissey Version: HEAD OS: Linux x86_64 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2001:4978:194:0:21f:5bff:fee9:da92) slapd -d ? emits the incorrect numeric value for LDAP_DEBUG_ANY on x86_64: $ slapd -d \? Installed log subsystems: Any (18446744073709551615) [...] $ slapd -d 18446744073709551615 unrecognized log level "18446744073709551615" Debug levels are always treated as type int, which is four bytes on Linux x86_64. loglevel_print() in servers/slapd/bconfig.c seems to blame, since it's using a long-typed format (%lu) instead of an unsigned int-typed one (%u).

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by karavelov＠spnet.net 06 Oct '09

06 Oct '09

Hello, I have somehow changed the setup: The first process is a ldap to sql server withpout pcache the config file is here: http://purgatory.spnet.net/~karavelov/abort/slapd.conf The second process is a pure cache that reads from the first process using unix domain socket. The config file is here: http://purgatory.spnet.net/~karavelov/abort/slapd-proxy.conf The sql backend runs without problems. But the proxy process crashes once or twice a day on assertion: The output of the process is here: http://purgatory.spnet.net/~karavelov/abort/log It was running in valgrind, so here is its output: http://purgatory.spnet.net/~karavelov/abort/vg The process has dumped a core. I have made a debugging session with it. Here it is: http://purgatory.spnet.net/~karavelov/abort/gdb I am not sure what to look for. The process aborts on the same asserrtion even it is not run in valgrind. May be I should try with libdb-4.8? Thanks in advance Luben

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by masarati＠aero.polimi.it 05 Oct '09

05 Oct '09

masarati(a)aero.polimi.it wrote: > Another quick check you could probably do relatively quickly is zero out > that "e" pointer before calling be_entry_get_rw() within refresh_merge(). Forget about it: it's already done from within be_entry_get_rw() itself. The fact you get a corrupted "e" sounds odd, though. It would be interesting to track how and where it is set. This should appear from valgrind's inspection when *e is dereferenced. p.

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by masarati＠aero.polimi.it 05 Oct '09

05 Oct '09

luben karavelov wrote: > masarati(a)aero.polimi.it wrote: >> >> Thanks for collecting this info. The valgrind output could be of some >> use, but unfortunately I don't have time right now to set up a working >> RDBMS and extensively debug things. I'll keep this on my todo list. >> >> You should please re-run valgrind with --num-callers=30 or more, because >> in some cases errors are in too nested functions to get a clear idea of >> whether the issue is caused by garbage fed by slapd/back-sql or by errors >> inside the RDBMS/ODBC layers. The fact that valgrind systematically >> complains about internals of the RDBMS/ODBC reading past the end of >> memory >> chunks malloc'ed by slapd could be related to passing some non-nul >> terminated bervals that are dealt with as strings. Having a longer call >> stack could help tracking those occurrences. However, those issues >> should >> not be critical, since there's no invalid writes. >> >> Also, you should walk through the list of attributes being returned, to >> provide a hint about whether back-sql is computing a screwed attrlist or >> so. Along the lines of your current gdb session, you should get to frame >> #5, refresh_merge() in pcache.c, and print *e->e_attrs, >> *e->e_attrs->a_desc, *e->e_attrs->a_vals[0]; then move to >> e->e_attrs->a_next and repeat the prints to the end of the list. The >> fact >> you get a value of "a" equal to 0x500000000 looks definitely odd to >> me, as >> that attr list should result from be_entry_get_rw(), which in turn should >> collect it from the local database. Unless valgrind reveals some oddity >> in back-sql, the behavior you notice should not depend on the specific >> remote database you're using, but rather from the local one. >> >> p. > > Hello, > Tomorrow I will make a setup with pure sql process and a pure pcache > daemon that reads from the first over unix domain socket. In this manner > it will be clear if the crashing part is related to back-sql and the > database drivers/ODBC manager or not. > > Meanwhile, you could find the requested debugging session here: > http://purgatory.spnet.net/~karavelov/attr_list/gdb-1 > > It seems that the "e" pointer is corrupted. Good catch. > Tomorrow I will start it > through valgrind with more back-frames as requested Another quick check you could probably do relatively quickly is zero out that "e" pointer before calling be_entry_get_rw() within refresh_merge(). Thanks, p.

1 0

Re: (ITS#6320)
by j＠telepaths.org 05 Oct '09

05 Oct '09

Michael, Yes, that has been a theory of mine - I am investigating the =20 possibility, however I should point out that we recently =20 decommissioned our old LDAP system, which used slapd on Etch - it uses =20= the same firewalls for out-of-zone-access, and was not affected by =20 this problem. However, I should also point out that since my last update, the =20 problem has not returned. Using a six-member multimaster mesh has =20 proven fruitful so far, but we'll see how it plays out over the next =20 couple of days. Thanks for your help ... On Oct 4, 2009, at 06:00 , Michael Str=F6der wrote: > j(a)telepaths.org wrote: >> On Oct 2, 2009, at 23:00 , Howard Chu wrote: >> >>> j(a)telepaths.org wrote: >>>> My last email was likely indecipherable, so re-sending as plain =20 >>>> text: >>>> >>>> ##### >>>> >>>> This may sound like a strange question, but couldn't seem to find =20= >>>> the >>>> answer in the docs: >>>> >>>> Would the global "idletimeout" parameter interfere with >>>> 'refreshAndPersist' operations in any way? >>> No, that would be stupid. >>> >> Yes Howard -- that would be incredibly stupid. But I had to ask. >> [..] >> SOME of the servers receiving updates are separated by firewalls, >> yes ... HOWEVER (a) there are special policies in place, allowing >> these hosts to communicate via LDAP[S]; this can be confirmed by >> running ldapsearch routines from every possible vantage point TO =20 >> every >> possible provider, etc., > > Note that firewalls are doing connection tracking enforcing their =20 > own timeouts > for idle connections. So please check whether the firewall =20 > configuration > really matches your slapd idle timeout requirements. > > Ciao, Michael.

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by karavelov＠spnet.net 05 Oct '09

05 Oct '09

masarati(a)aero.polimi.it wrote: > > Thanks for collecting this info. The valgrind output could be of some > use, but unfortunately I don't have time right now to set up a working > RDBMS and extensively debug things. I'll keep this on my todo list. > > You should please re-run valgrind with --num-callers=30 or more, because > in some cases errors are in too nested functions to get a clear idea of > whether the issue is caused by garbage fed by slapd/back-sql or by errors > inside the RDBMS/ODBC layers. The fact that valgrind systematically > complains about internals of the RDBMS/ODBC reading past the end of memory > chunks malloc'ed by slapd could be related to passing some non-nul > terminated bervals that are dealt with as strings. Having a longer call > stack could help tracking those occurrences. However, those issues should > not be critical, since there's no invalid writes. > > Also, you should walk through the list of attributes being returned, to > provide a hint about whether back-sql is computing a screwed attrlist or > so. Along the lines of your current gdb session, you should get to frame > #5, refresh_merge() in pcache.c, and print *e->e_attrs, > *e->e_attrs->a_desc, *e->e_attrs->a_vals[0]; then move to > e->e_attrs->a_next and repeat the prints to the end of the list. The fact > you get a value of "a" equal to 0x500000000 looks definitely odd to me, as > that attr list should result from be_entry_get_rw(), which in turn should > collect it from the local database. Unless valgrind reveals some oddity > in back-sql, the behavior you notice should not depend on the specific > remote database you're using, but rather from the local one. > > p. Hello, Tomorrow I will make a setup with pure sql process and a pure pcache daemon that reads from the first over unix domain socket. In this manner it will be clear if the crashing part is related to back-sql and the database drivers/ODBC manager or not. Meanwhile, you could find the requested debugging session here: http://purgatory.spnet.net/~karavelov/attr_list/gdb-1 It seems that the "e" pointer is corrupted. Tomorrow I will start it through valgrind with more back-frames as requested Best regards Luben

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by masarati＠aero.polimi.it 04 Oct '09

04 Oct '09

> hyc(a)symas.com wrote: >> I haven't touched back-sql in quite a while, and don't have a suitable >> config >> for testing this. I suggest you run under valgrind and see what errors >> it reports. > > Hello, > > I have run the server through valgrind. It crashed on the same structure > on different location. The gdb session could be found here: > http://purgatory.spnet.net/~karavelov/attr_list/gdb > > The output from valgrind could be found here: > http://purgatory.spnet.net/~karavelov/attr_list/vg > > In it the lines up to 2833 are output on server start. You could find > the address 0x500000000 (found from gdb session) around line 3180. > > I could see nothing interesting in the log file. > > I hope this info will help you. If I could gather more information, just > say. Thanks for collecting this info. The valgrind output could be of some use, but unfortunately I don't have time right now to set up a working RDBMS and extensively debug things. I'll keep this on my todo list. You should please re-run valgrind with --num-callers=30 or more, because in some cases errors are in too nested functions to get a clear idea of whether the issue is caused by garbage fed by slapd/back-sql or by errors inside the RDBMS/ODBC layers. The fact that valgrind systematically complains about internals of the RDBMS/ODBC reading past the end of memory chunks malloc'ed by slapd could be related to passing some non-nul terminated bervals that are dealt with as strings. Having a longer call stack could help tracking those occurrences. However, those issues should not be critical, since there's no invalid writes. Also, you should walk through the list of attributes being returned, to provide a hint about whether back-sql is computing a screwed attrlist or so. Along the lines of your current gdb session, you should get to frame #5, refresh_merge() in pcache.c, and print *e->e_attrs, *e->e_attrs->a_desc, *e->e_attrs->a_vals[0]; then move to e->e_attrs->a_next and repeat the prints to the end of the list. The fact you get a value of "a" equal to 0x500000000 looks definitely odd to me, as that attr list should result from be_entry_get_rw(), which in turn should collect it from the local database. Unless valgrind reveals some oddity in back-sql, the behavior you notice should not depend on the specific remote database you're using, but rather from the local one. p.

1 0

Re: (ITS#6320)
by michael＠stroeder.com 04 Oct '09

04 Oct '09

j(a)telepaths.org wrote: > On Oct 2, 2009, at 23:00 , Howard Chu wrote: > >> j(a)telepaths.org wrote: >>> My last email was likely indecipherable, so re-sending as plain text: >>> >>> ##### >>> >>> This may sound like a strange question, but couldn't seem to find the >>> answer in the docs: >>> >>> Would the global "idletimeout" parameter interfere with >>> 'refreshAndPersist' operations in any way? >> No, that would be stupid. >> > Yes Howard -- that would be incredibly stupid. But I had to ask. > [..] > SOME of the servers receiving updates are separated by firewalls, > yes ... HOWEVER (a) there are special policies in place, allowing > these hosts to communicate via LDAP[S]; this can be confirmed by > running ldapsearch routines from every possible vantage point TO every > possible provider, etc., Note that firewalls are doing connection tracking enforcing their own timeouts for idle connections. So please check whether the firewall configuration really matches your slapd idle timeout requirements. Ciao, Michael.

1 0

Re: (ITS#6320)
by j＠telepaths.org 04 Oct '09

04 Oct '09

Pierangelo, I offered to do post my configuration in my initial post. No one accepted my offer. Since you are now asking for it, I will gladly post it below. Two, thanks for the hint about editing passwords, I can assure you that all confidential data posted below will have been tainted *somehow*. I should also point out that today I made a change to my infrastructure that I hope will help the situation. Since I've noticed that most of the DIT discrepancies were limited to the standard Consumer boxes, and not the Providers, I have decided to do away entirely with the the standard Consumers. We now have six (6) virtually-identically configured Providers whom all replicate with the other five (5) respective hybrid Consumer/Providers. Essentially a six member multimaster mesh, all of whom can contact all of the other members perfectly via both LDAP and LDAPS. Here is a slapd.conf from ONE of the SIX members: ##### include /etc/ldap/schema/core.schema include /etc/ldap/schema/cosine.schema include /etc/ldap/schema/nis.schema include /etc/ldap/schema/inetorgperson.schema include /etc/ldap/schema/misc.schema include /etc/ldap/schema/openldap.schema include /etc/ldap/schema/duaconf.schema include /etc/ldap/schema/dyngroup.schema include /etc/ldap/schema/ppolicy.schema include /etc/ldap/schema/sudo.schema include /etc/ldap/schema/dhcp.schema include /etc/ldap/schema/samba.schema include /usr/share/doc/libpam-ldap/ldapns.schema include /etc/ldap/schema/hdb.schema include /etc/ldap/schema/uber.schema pidfile /var/run/slapd/slapd.pid argsfile /var/run/slapd/slapd.args tool-threads 4 loglevel none modulepath /usr/lib/ldap moduleload back_hdb moduleload back_relay moduleload rwm.la moduleload back_monitor.la moduleload syncprov moduleload accesslog serverID 100 ldaps://10.64.100.100:636/ serverID 107 ldaps://10.64.100.107:636/ serverID 108 ldaps://10.64.100.108:636/ serverID 811 ldaps://10.9.8.11:636/ serverID 812 ldaps://10.9.8.12:636/ serverID 814 ldaps://10.9.8.14:636/ TLSCertificateFile /etc/ldap/ssl/wildcard.site.example.com.crt TLSCertificateKeyFile /etc/ldap/ssl/wildcard.site.example.com.key TLSCACertificateFile /etc/ssl/certs/ca.cert disallow bind_anon sizelimit unlimited timelimit unlimited security tls=0 access to dn.subtree="cn=Subschema" by users read by * none stop access to dn.base="" by users read by * none stop defaultSearchBase dc=real,dc=example,dc=com sasl-realm SITE.EXAMPLE.COM sasl-host ds.site.example.com #sasl-secprops minssf=0 authz-regexp "uid=$.*$,cn=SITE.EXAMPLE.COM,cn=gssapi,cn=auth" "uid=$1,cn=plain,cn=auth,dc=site,dc=example,dc=com" authz-regexp "gidNumber=\\\0+uidNumber=\\ \0,cn=peercred,cn=external,cn=auth" "uid=writer,cn=plain,cn=auth,dc=real,dc=example,dc=com" backend hdb ########### Monitoring Database - For slapd/hdb performance data database monitor rootdn uid=monitor,cn=monitor rootpw {SSHA}encrypted-hash access to dn.subtree="cn=monitor" by group/groupOfUniqueNames/ uniqueMember ="cn=ldapadmin,cn=ldap,cn=groups,dc=real,dc=example,dc=com" read by dn.exact="uid=rootdn,cn=plain,cn=auth,dc=real,dc=example,dc=com" read ########### example.Log database hdb suffix cn=log rootdn "uid=log,cn=log" rootpw {SSHA}encrypted-hash directory /var/lib/ldap/log index reqStart,objectClass,entryCSN,reqResult eq dbconfig set_cachesize 0 2097152 0 overlay syncprov syncprov-nopresent TRUE syncprov-reloadhint TRUE access to dn.subtree="cn=log" by group/groupOfUniqueNames/ uniqueMember ="cn=ldapadmin,cn=ldap,cn=groups,dc=real,dc=example,dc=com" read by dn.base="uid=rootdn,cn=plain,cn=auth,dc=real,dc=example,dc=com" read by dn.base="uid=log,cn=log" read ########### example.real database hdb cachesize 10000 idlcachesize 30000 suffix "dc=real,dc=example,dc=com" checksum checkpoint 100 10 cachefree 20 rootdn "uid=rootdn,cn=plain,cn=auth,dc=real,dc=example,dc=com" rootpw {SSHA}encrypted-hash monitoring on directory "/var/lib/ldap/real" dncachesize 100000 dbconfig set_cachesize 1 0 2 dbconfig set_lg_max 10485760 dbconfig set_flags db_log_autoremove dbconfig set_lg_bsize 2097152 dbconfig set_lk_max_objects 1500 dbconfig set_lk_max_locks 1500 dbconfig set_lk_max_lockers 1500 index objectClass,structuralObjectClass eq index entryCSN,entryUUID eq index cn,uid,memberUid eq syncrepl rid=001 provider=ldaps://10.64.100.100:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist syncrepl rid=002 provider=ldaps://10.64.100.107:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist syncrepl rid=003 provider=ldaps://10.64.100.108:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist syncrepl rid=004 provider=ldaps://10.9.8.14:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist syncrepl rid=005 provider=ldaps://10.9.8.11:636/ bindmethod=simple binddn="uid=syncrepl,cn=plain,cn=auth,dc=real,dc=example,dc=com" credentials=syncreplpass scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=real,dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshAndPersist mirrormode true overlay syncprov syncprov-reloadhint TRUE #syncprov-checkpoint 10 5 syncprov-sessionlog 5000 overlay accesslog logdb cn=log logops writes logpurge 7+00:00 2+00:00 logsuccess TRUE include /etc/ldap/acls include /etc/ldap/relays #### Thanks again Jeff

1 0

Re: (ITS#6320)
by masarati＠aero.polimi.it 04 Oct '09

04 Oct '09

> OK, maybe I wasn't clear enough, so I'll rephrase my initial question: > > Is it normal for a 'slapd' server to ONLY correct itself via a restart > of the 'slapd' process? > > I asked because of my earlier findings today. Since waiting for the > system to correct the discrepancy in the DIT by itself proved to be a > futile exercise, and since a manual restart produced the results, it > seemed as though I was observing suboptimal behavior, and was asking > if this was indicative of something else. What about posting your *EXACT* configuration - not a snapshot of it, not a verbal description of how you defined it. Sometimes the devil is in the details. Only make sure you remove passwords, assuming they're correctly spelled in the original. p.

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by karavelov＠spnet.net 03 Oct '09

03 Oct '09

hyc(a)symas.com wrote: > I haven't touched back-sql in quite a while, and don't have a suitable config > for testing this. I suggest you run under valgrind and see what errors it reports. Hello, I have run the server through valgrind. It crashed on the same structure on different location. The gdb session could be found here: http://purgatory.spnet.net/~karavelov/attr_list/gdb The output from valgrind could be found here: http://purgatory.spnet.net/~karavelov/attr_list/vg In it the lines up to 2833 are output on server start. You could find the address 0x500000000 (found from gdb session) around line 3180. I could see nothing interesting in the log file. I hope this info will help you. If I could gather more information, just say. Luben

1 0

Re: (ITS#6320)
by j＠telepaths.org 03 Oct '09

03 Oct '09

OK, maybe I wasn't clear enough, so I'll rephrase my initial question: Is it normal for a 'slapd' server to ONLY correct itself via a restart of the 'slapd' process? I asked because of my earlier findings today. Since waiting for the system to correct the discrepancy in the DIT by itself proved to be a futile exercise, and since a manual restart produced the results, it seemed as though I was observing suboptimal behavior, and was asking if this was indicative of something else. On Oct 2, 2009, at 23:02 , Howard Chu wrote: > j(a)telepaths.org wrote: >> IMPORTANT UPDATE: >> >> Is it normal for a 'slapd' server to correct itself via a restart of >> the slapd process? Currently, I have one Consumer that is 2-records >> behind in Synchronization - I restarted his process and the records >> immediately were replicated. This normally is NOT standard behavior. >> >> Can anyone explain? Is this expected? Indicative of another problem? > > That's what the "refresh" in "refreshAndPersist" is about. Any > freshly started consumer will automatically sync up to the current > state of the provider. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6320)
by j＠telepaths.org 03 Oct '09

03 Oct '09

On Oct 2, 2009, at 23:00 , Howard Chu wrote: > j(a)telepaths.org wrote: >> My last email was likely indecipherable, so re-sending as plain text: >> >> ##### >> >> This may sound like a strange question, but couldn't seem to find the >> answer in the docs: >> >> Would the global "idletimeout" parameter interfere with >> 'refreshAndPersist' operations in any way? > > No, that would be stupid. > Yes Howard -- that would be incredibly stupid. But I had to ask. >> I ask because I have had yet another instance where my three >> production consumers did not get a series of updates that it should >> have. This was hours after successfully adding an accesslog database >> to all Provider servers in question. >> >> When querying the log database: >> >> ldapsearch -b cn=log -xD uid=log,cn=log -w logpassword -s sub -H ldap://ldap-provider-in-question >> >> I see that the records-in-question were in fact updated as they >> should >> have .... but there's absolutely no information hinting towards WHY a >> remote server would fail to get updates (especially when all LDAP >> traffic in between the said-hosts is unrestricted and unmoderated). > > Are you sure? No firewalls in the way? > SOME of the servers receiving updates are separated by firewalls, yes ... HOWEVER (a) there are special policies in place, allowing these hosts to communicate via LDAP[S]; this can be confirmed by running ldapsearch routines from every possible vantage point TO every possible provider, etc., and (b) some of the replication issues observed occurred when all affected hosts (one or more providers and its consumer clients) were in the same local subnet, at which point the firewall is immaterial. I had asked my stupid question about the idletimeout, because this behavior doesn't strike me as simply "not working". Rather, it begins to work .... then just stops. So my next thought was that perhaps the firewall was causing the connections to time-out. But if that were true ... wouldn't both the "retry" and the "timeout" parameters compensate for this? I mean, the man page certainly doesn't contradict my assumption ... Jeff >> Hence why I began to scrutinize the 'idletimeout' parameter >> (previously set to 45 seconds, now has been removed entirely from all >> Providers AND Consumers alike). >> >> Still looking at, testing and contemplating this whole situation ... >> I'll keep you posted, but am curious if anyone has any input to my >> question above, thanks. >> >> ##### >> >> I am officially out of ideas - synchronization is entirely >> unreliable. >> Need guidance ASAP. Thanks ..... > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6320)
by hyc＠symas.com 03 Oct '09

03 Oct '09

j(a)telepaths.org wrote: > IMPORTANT UPDATE: > > Is it normal for a 'slapd' server to correct itself via a restart of > the slapd process? Currently, I have one Consumer that is 2-records > behind in Synchronization - I restarted his process and the records > immediately were replicated. This normally is NOT standard behavior. > > Can anyone explain? Is this expected? Indicative of another problem? That's what the "refresh" in "refreshAndPersist" is about. Any freshly started consumer will automatically sync up to the current state of the provider. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6320)
by hyc＠symas.com 03 Oct '09

03 Oct '09

j(a)telepaths.org wrote: > My last email was likely indecipherable, so re-sending as plain text: > > ##### > > This may sound like a strange question, but couldn't seem to find the > answer in the docs: > > Would the global "idletimeout" parameter interfere with > 'refreshAndPersist' operations in any way? No, that would be stupid. > I ask because I have had yet another instance where my three > production consumers did not get a series of updates that it should > have. This was hours after successfully adding an accesslog database > to all Provider servers in question. > > When querying the log database: > > ldapsearch -b cn=log -xD uid=log,cn=log -w logpassword -s sub -H ldap://ldap-provider-in-question > > I see that the records-in-question were in fact updated as they should > have .... but there's absolutely no information hinting towards WHY a > remote server would fail to get updates (especially when all LDAP > traffic in between the said-hosts is unrestricted and unmoderated). Are you sure? No firewalls in the way? > Hence why I began to scrutinize the 'idletimeout' parameter > (previously set to 45 seconds, now has been removed entirely from all > Providers AND Consumers alike). > > Still looking at, testing and contemplating this whole situation ... > I'll keep you posted, but am curious if anyone has any input to my > question above, thanks. > > ##### > > I am officially out of ideas - synchronization is entirely unreliable. > Need guidance ASAP. Thanks ..... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6320)
by j＠telepaths.org 02 Oct '09

02 Oct '09

IMPORTANT UPDATE: Is it normal for a 'slapd' server to correct itself via a restart of the slapd process? Currently, I have one Consumer that is 2-records behind in Synchronization - I restarted his process and the records immediately were replicated. This normally is NOT standard behavior. Can anyone explain? Is this expected? Indicative of another problem? Jeff

1 0

Re: (ITS#6320)
by j＠telepaths.org 02 Oct '09

02 Oct '09

My last email was likely indecipherable, so re-sending as plain text: ##### This may sound like a strange question, but couldn't seem to find the answer in the docs: Would the global "idletimeout" parameter interfere with 'refreshAndPersist' operations in any way? I ask because I have had yet another instance where my three production consumers did not get a series of updates that it should have. This was hours after successfully adding an accesslog database to all Provider servers in question. When querying the log database: ldapsearch -b cn=log -xD uid=log,cn=log -w logpassword -s sub -H ldap://ldap-provider-in-question I see that the records-in-question were in fact updated as they should have .... but there's absolutely no information hinting towards WHY a remote server would fail to get updates (especially when all LDAP traffic in between the said-hosts is unrestricted and unmoderated). Hence why I began to scrutinize the 'idletimeout' parameter (previously set to 45 seconds, now has been removed entirely from all Providers AND Consumers alike). Still looking at, testing and contemplating this whole situation ... I'll keep you posted, but am curious if anyone has any input to my question above, thanks. ##### I am officially out of ideas - synchronization is entirely unreliable. Need guidance ASAP. Thanks ..... Jeff

1 0

Re: (ITS#6322) slapd suddenly stops working, and starts using 100% CPU
by quanah＠zimbra.com 02 Oct '09

02 Oct '09

--On Friday, October 02, 2009 8:54 PM +0000 hyc(a)symas.com wrote: > strace is useless. Use gdb and get a trace of all running threads when > this occurs. > > http://www.openldap.org/faq/data/cache/59.html You also need to test against a current release of OpenLDAP, like OpenLDAP 2.4.18. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6321) segfault in 2.4.18 replica
by Howard Chu 02 Oct '09

02 Oct '09

Your trace shows pretty clearly that the crash occurred in your Kerberos library. You're going to need to start there to look for the problem.

1 0

Re: (ITS#6310) Slapd with pcache crashes under load
by hyc＠symas.com 02 Oct '09

02 Oct '09

karavelov(a)spnet.net wrote: > Full_Name: luben karavelov > Version: 2.4.18 - HEAD > OS: Debian Linux > URL: > Submission from: (NULL) (212.50.0.80) > > > Hello, > I have 2 running slapd servers for load balancing. The 2 instances are > configured with back-sql and pcache overlay. Every 5-15 minutes one of the > servers crashes. > > The config could be found here: > http://purgatory.spnet.net/~karavelov/slapd.conf.h2 > > A gdb session with the resulting core dump could be found here: > http://purgatory.spnet.net/~karavelov/gdb1 > > Without proxycache overlay the servers does not crash in this manner. > > The software tested is > 1. slapd 2.4.18 with backported pcache.c from HEAD > 2. slapd HEAD I haven't touched back-sql in quite a while, and don't have a suitable config for testing this. I suggest you run under valgrind and see what errors it reports. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6320)
by j＠telepaths.org 02 Oct '09

02 Oct '09

--Apple-Mail-4--815209214 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes This may sound like a strange question, but couldn't seem to find the answer in the docs: Would the global "idletimeout" parameter interfere with 'refreshAndPersist' operations in any way? I ask because I have had yet another instance where my three production consumers did not get a series of updates that it should have. This was hours after successfully adding an accesslog database to all Provider servers in question. When querying the log database: ldapsearch -b cn=log -xD uid=log,cn=log -w logpassword -s sub -H ldap://ldap-provider-in-question I see that the records-in-question were in fact updated as they should have .... but there's absolutely no information hinting towards WHY a remote server would fail to get updates (especially when all LDAP traffic in between the said-hosts is unrestricted and unmoderated). Hence why I began to scrutinize the 'idletimeout' parameter (previously set to 45 seconds, now has been removed entirely from all Providers AND Consumers alike). Still looking at, testing and contemplating this whole situation ... I'll keep you posted, but am curious if anyone has any input to my question above, thanks. Jeff On Oct 1, 2009, at 11:39 PM, Howard Chu wrote: > j(a)telepaths.org wrote: >> Sorry about that .... resending to ITS address: >> >> Regarding use of the accesslog DB ..... is this still an option when >> using 'mirrormode' ? I understand at least in the recent past, this >> option was not compatible with the accesslog overlay ..... > > That's not correct. The accesslog overlay can always be used for > logging purposes. It is just that using it for multimaster delta- > syncrepl is not yet supported. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ --Apple-Mail-4--815209214 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; = "><div>This may sound like a strange question, but couldn't seem to find = the answer in the docs:</div><div><br></div><div>Would the global = "idletimeout" parameter interfere with 'refreshAndPersist' operations in = any way?</div><div><br></div><div>I ask because I have had yet another = instance where my three production consumers did not get a series of = updates that it should have.  This was hours after successfully = adding an accesslog database to all Provider servers in = question.</div><div><br></div><div>When querying the log = database:</div><div><br></div><div style=3D"font-size: 14px; = ">ldapsearch -b cn=3Dlog -xD uid=3Dlog,cn=3Dlog -w logpassword -s sub -H = <a = href=3D"ldap://ldap-provider-in-question">ldap://ldap-provider-in-question= </a></div><div><br></div><div>I see that the records-in-question were in = fact updated as they should have .... but there's absolutely no = information hinting towards WHY a remote server would fail to get = updates (especially when all LDAP traffic in between the said-hosts is = unrestricted and unmoderated).  Hence why I began to scrutinize the = 'idletimeout' parameter (previously set to 45 seconds, now has been = removed entirely from all Providers AND Consumers = alike).</div><div><br></div><div>Still looking at, testing and = contemplating this whole situation ... I'll keep you posted, but am = curious if anyone has any input to my question above, = thanks.</div><div><br></div><div>Jeff</div><div><br></div><div><br></div><= div><br></div><br><div><div>On Oct 1, 2009, at 11:39 PM, Howard Chu = wrote:</div><br class=3D"Apple-interchange-newline"><blockquote = type=3D"cite"><div><a href=3D"mailto:j@telepaths.org">j(a)telepaths.org</a> = wrote:<br><blockquote type=3D"cite">Sorry about that .... =  resending to ITS address:<br></blockquote><blockquote = type=3D"cite"><br></blockquote><blockquote type=3D"cite">Regarding use = of the accesslog DB ..... is this still an option = when<br></blockquote><blockquote type=3D"cite">using 'mirrormode' ? =  I understand at least in the recent past, = this<br></blockquote><blockquote type=3D"cite">option was not compatible = with the accesslog overlay .....<br></blockquote><br>That's not correct. = The accesslog overlay can always be used for logging purposes. It is = just that using it for multimaster delta-syncrepl is not yet = supported.<br><br>-- <br>  -- Howard Chu<br>  CTO, Symas Corp. =           <a = href=3D"http://www.symas.com">http://www.symas.com</a><br> =  Director, Highland Sun     <a = href=3D"http://highlandsun.com/hyc/">http://highlandsun.com/hyc/</a><br> =  Chief Architect, OpenLDAP  <a = href=3D"http://www.openldap.org/project/">http://www.openldap.org/project/= </a><br></div></blockquote></div><br></body></html>= --Apple-Mail-4--815209214--

1 0

Re: (ITS#6322) slapd suddenly stops working, and starts using 100% CPU
by hyc＠symas.com 02 Oct '09

02 Oct '09

helge(a)monsternett.no wrote: > Full_Name: Helge Milde > Version: slapd 2.4.11 > OS: Debian 5 > URL: > Submission from: (NULL) (81.175.2.164) > > > Hello, > > In the last two days, slapd has randomly stopped working on our server, about > one time every two or three hours. Before this, slapd had been running without > any problems for about two months. slapd.conf has not been touched since then. > > What happens is that slapd stops taking queries (postfix, dovecot, NSS etc. > starts getting timeouts), and it starts using exactly 100% of one core until we > restart it. > After restart, everything works again (until this happens again). > > It seems to only happen when slapd is under "stress" (we have an 8-core system > with 8G RAM, and so slapd is really never under much "stress"; the biggest CPU > load I've seen is about 40% on one core). The last ldap search before slapd > stops working is always the same: > Oct 2 10:57:38 oi-mail slapd[24550]: conn=1543 op=1 SRCH > base="ou=groups,dc=web,dc=xxxx,dc=net" scope=1 deref=0 > filter="(&(objectClass=posixGroup))" > .. which should return about 1800 entries. Different connections might query > this simultaneously, and it's a query that happens every ~10 seconds on > average. > > I've tried to reproduce this by doing .. > # while [ 1 ]; do ldapsearch -x -H ldaps://.. -b > 'ou=groups,dc=web,dc=xxxx,dc=net' '(&(objectClass=posixAccount))'; done > .. for two minutes without any "luck" :). slapd just uses a bit more CPU, but > never stops working. > > With "loglevel stats stats2", I get nothing out of the logs: > Oct 2 08:42:15 oi-mail slapd[15836]: conn=34798 op=1 ENTRY > dn="cn=xxxx,ou=groups,dc=web,dc=xxxx,dc=net" > ... time of "crash" (we're mid-search at this point); nothing gets logged until > we do a restart: ... > Oct 2 08:42:30 oi-mail slapd[15836]: daemon: shutdown requested and initiated. > Oct 2 08:42:30 oi-mail slapd[15836]: conn=34391 fd=31 closed (slapd shutdown) > ... many entires like the one above, and then I seem to be getting the next > entry from the same connection, right before it closes: ... > Oct 2 08:42:30 oi-mail slapd[15836]: conn=34798 op=1 ENTRY > dn="cn=xxxx,ou=groups,dc=web,dc=xxxx,dc=net" > Oct 2 08:42:30 oi-mail slapd[15836]: conn=34798 fd=81 closed (slapd shutdown) > > I've tried logging more than just stats, but the interesting loglevels also > causes a big performance hit (trace and args), and I'm not really sure which of > the other loglevels could be helpful. > This is a live ldap server, on a server that also hosts other services, so I > simply can't take the performance hit. > > I also did a strace -p<slapd pid>, which only gives "futex(0x41c499e0, > FUTEX_WAIT," until I restart slapd, but the "restarting routine" could be > useful, so here it is: http://www.pastie.org/639170 strace is useless. Use gdb and get a trace of all running threads when this occurs. http://www.openldap.org/faq/data/cache/59.html -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#6323) smbk5pwd: Makefile links module for a different directory than it's installed to
by jonathan＠phillipoux.net 02 Oct '09

02 Oct '09

Full_Name: Jonathan Clarke Version: RE24 OS: Linux URL: ftp://ftp.openldap.org/incoming/jonathan-clarke-smbk5pwd-rpath-20091002.pat… Submission from: (NULL) (82.67.204.30) Greetings, The makefile in the smbk5pwd contrib module runs the libtool link phase with rtool=$(libdir), but then installs the module to $(moduledir), which can be different, and thus causes install to bork: ../../../libtool --mode=install cp smbk5pwd.la /usr/lib/openldap libtool: install: error: cannot install `smbk5pwd.la' to a directory not ending in /usr/lib The patch above fixes this, removing libdir completely from the Makefile as it doesn't seem to be needed.

1 0

Re: (ITS#6320)
by j＠telepaths.org 02 Oct '09

02 Oct '09

OK I have a fully-functional accesslog configured on all applicable Provider servers. The respective DIT is now being watched for all WRITES (delete, add, modify and modrdn). I made a change a few minutes ago to a test record, and it DID replicate to ALL slapd servers, so we're looking good in the "short- term". I will check back with you guys later today with any developments, but we should know if there's still a problem today, and maybe now I can get some visibility into it. Thanks again to both of you, Howard and Quanah. Jeff

1 0

Re: (ITS#6320)
by j＠telepaths.org 02 Oct '09

02 Oct '09

Ah, I see. Then the accesslog overlay can be loaded and used on the Provider, but Consumers won't query the Provider any differently than with regular SyncRepl. I will promptly try this, thanks Howard. Jeff On Oct 1, 2009, at 23:39 , Howard Chu wrote: > j(a)telepaths.org wrote: >> Sorry about that .... resending to ITS address: >> >> Regarding use of the accesslog DB ..... is this still an option when >> using 'mirrormode' ? I understand at least in the recent past, this >> option was not compatible with the accesslog overlay ..... > > That's not correct. The accesslog overlay can always be used for > logging purposes. It is just that using it for multimaster delta- > syncrepl is not yet supported. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#6322) slapd suddenly stops working, and starts using 100% CPU
by helge＠monsternett.no 02 Oct '09

02 Oct '09

Full_Name: Helge Milde Version: slapd 2.4.11 OS: Debian 5 URL: Submission from: (NULL) (81.175.2.164) Hello, In the last two days, slapd has randomly stopped working on our server, about one time every two or three hours. Before this, slapd had been running without any problems for about two months. slapd.conf has not been touched since then. What happens is that slapd stops taking queries (postfix, dovecot, NSS etc. starts getting timeouts), and it starts using exactly 100% of one core until we restart it. After restart, everything works again (until this happens again). It seems to only happen when slapd is under "stress" (we have an 8-core system with 8G RAM, and so slapd is really never under much "stress"; the biggest CPU load I've seen is about 40% on one core). The last ldap search before slapd stops working is always the same: Oct 2 10:57:38 oi-mail slapd[24550]: conn=1543 op=1 SRCH base="ou=groups,dc=web,dc=xxxx,dc=net" scope=1 deref=0 filter="(&(objectClass=posixGroup))" .. which should return about 1800 entries. Different connections might query this simultaneously, and it's a query that happens every ~10 seconds on average. I've tried to reproduce this by doing .. # while [ 1 ]; do ldapsearch -x -H ldaps://.. -b 'ou=groups,dc=web,dc=xxxx,dc=net' '(&(objectClass=posixAccount))'; done .. for two minutes without any "luck" :). slapd just uses a bit more CPU, but never stops working. With "loglevel stats stats2", I get nothing out of the logs: Oct 2 08:42:15 oi-mail slapd[15836]: conn=34798 op=1 ENTRY dn="cn=xxxx,ou=groups,dc=web,dc=xxxx,dc=net" ... time of "crash" (we're mid-search at this point); nothing gets logged until we do a restart: ... Oct 2 08:42:30 oi-mail slapd[15836]: daemon: shutdown requested and initiated. Oct 2 08:42:30 oi-mail slapd[15836]: conn=34391 fd=31 closed (slapd shutdown) ... many entires like the one above, and then I seem to be getting the next entry from the same connection, right before it closes: ... Oct 2 08:42:30 oi-mail slapd[15836]: conn=34798 op=1 ENTRY dn="cn=xxxx,ou=groups,dc=web,dc=xxxx,dc=net" Oct 2 08:42:30 oi-mail slapd[15836]: conn=34798 fd=81 closed (slapd shutdown) I've tried logging more than just stats, but the interesting loglevels also causes a big performance hit (trace and args), and I'm not really sure which of the other loglevels could be helpful. This is a live ldap server, on a server that also hosts other services, so I simply can't take the performance hit. I also did a strace -p <slapd pid>, which only gives "futex(0x41c499e0, FUTEX_WAIT," until I restart slapd, but the "restarting routine" could be useful, so here it is: http://www.pastie.org/639170 I understand that this probably isn't enough data to see what the problem is, and so I'm hoping to get some help in debugging this properly. As I've said, args/trace loglevel takes too much of a performance hit, but if this is the only way, we might reconcider. Interesting parts of slapd.conf below. We use slapd.conf instead of cn=config, everything goes over TLS, we're AFAIK only using LDAPv3 binding, and we only have one database entry in slapd.conf. moduleload back_hdb sizelimit 99999 tool-threads 8 threads 16 concurrency 32 backend hdb database hdb cachesize 60000 dbconfig set_cachesize 0 52428800 0 dbconfig set_lk_max_objects 1500 dbconfig set_lk_max_locks 1500 dbconfig set_lk_max_lockers 1500 lastmod on checkpoint 512 30 Best regards, Helge Milde

1 0

Re: (ITS#6320)
by hyc＠symas.com 02 Oct '09

02 Oct '09

j(a)telepaths.org wrote: > Sorry about that .... resending to ITS address: > > Regarding use of the accesslog DB ..... is this still an option when > using 'mirrormode' ? I understand at least in the recent past, this > option was not compatible with the accesslog overlay ..... That's not correct. The accesslog overlay can always be used for logging purposes. It is just that using it for multimaster delta-syncrepl is not yet supported. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6217) proxycache not returning cached data
by masarati＠aero.polimi.it 01 Oct '09

01 Oct '09

> Sounds more like a schema config issue then. Most likely you don't have > the > matching schema in the proxy server, so it is unable to store the entries > it > has received. That was my first thought as well, but the proxy's slapd.conf he posted earlier included some samba-related schema file (I'm blindly assuming that file was defining sambaSamAccount). p.

1 0

(ITS#6320)
by j＠telepaths.org 01 Oct '09

01 Oct '09

Sorry about that .... resending to ITS address: Regarding use of the accesslog DB ..... is this still an option when using 'mirrormode' ? I understand at least in the recent past, this option was not compatible with the accesslog overlay ..... Jeff

1 0

Re: (ITS#6217) proxycache not returning cached data
by hyc＠symas.com 01 Oct '09

01 Oct '09

Jim van Keulen wrote: > Howard Chu wrote: >> w.van.keulen(a)few.vu.nl wrote: >>> masarati(a)aero.polimi.it wrote: >>>>> Here are slapd.conf and proxycache.conf as attachments. >>>> >>>> I just wanted to feed back before I leave for a few days. I don't see >>>> anything critical in your files. You explicitly give the rootdn write >>>> permissions, which is useless, but apart from this everything looks >>>> fine. >>>> >>>> I've tested a simple setup and it works as expected. I used HEAD and >>>> re24 >>>> code; the latter is nearly identical to 2.4.17. I also tested with TLS. >>>> >>>> At this point, I'm a bit clueless. There must be something we're >>>> overlooking. >> >>> Here is some more info, that I just discovered. >>> If my filter rule is >>> '(&(objectClass=sambaSamAccount)(uid=wvk))' >>> the data is cached the first time, but not returned >>> the second and later time. However when my filter rule is >>> (&(objectClass=posixAccount)(uid=wvk))' >>> it works as expected. >> >> Sounds like you may have already had the entry in the cache DB but with >> an incompatible objectclass before, and the attempt to update it with >> the new objectclass failed. This seems to be related to ITS#6242 which >> was recently fixed in HEAD. Can you please retest with the latest >> overlays/pcache.c? >> > > As you requested I retested with HEAD. However the cache is still not > working correctly. > > When my filter rule is: > '(&(objectClass=posixAccount)(uid=<who>))' the named attributes are > cached and correctly returned from the cache for every uid from the > existing user base > When my filter rule is: > '(&(objectClass=sambaSamAccount)(uid=<who>))' the named attributes are > cached and correctly returned from the cache for a subset of the > existing user base only. For all uid's the query is cached (according to > the log), but for some no data is returned. Sounds more like a schema config issue then. Most likely you don't have the matching schema in the proxy server, so it is unable to store the entries it has received. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6320) Replication Issues
by quanah＠zimbra.com 01 Oct '09

01 Oct '09

--On Thursday, October 01, 2009 12:20 PM -0700 Jeff Doyle <j(a)telepaths.org> wrote: > Yes, that is the first thing attempted (both via using LOCAL4 -> syslog > and via running slapd via cmd line) > > We see a brand-new server replicate all records (sids, csns, etc all look > fine) without incident. An existing server will receive, as I said, all > of the changes ... at least for awhile. > > Nothing in the sync activity looks out of the ordinary (and I have read > quite a few posts to be able to discern what the items inside mean) > > When a replication error occurs, often nothing at all is seen inside the > logs (no error mesg, no segfault, nothing) to indicate a failure has > occurred. Oftentimes, when SyncRepl fails to replicate, it is NOT during > testing, so it is difficult for us to find out exactly WHEN the error > occurred, as it could have happened at any point within the past 24 hrs, > and up to a week ago. > > I can try to replicate this error deliberately. Unfortunately it is very > difficult to make happen at-will. My concern is that someone will try to > close this ticket before I have witnessed the event. Hm, I would suggest creating an accesslog database on the master that records all of your changes. That way, you can look at the CSN stamp of what failed and look at the change in the accesslog DB. It sounds like however that the replica syncrepl is simply failing to see the change occur on the master if it isn't logging anything about the missing change. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6320) Replication Issues
by j＠telepaths.org 01 Oct '09

01 Oct '09

Yes, that is the first thing attempted (both via using LOCAL4 -> syslog and via running slapd via cmd line) We see a brand-new server replicate all records (sids, csns, etc all look fine) without incident. An existing server will receive, as I said, all of the changes ... at least for awhile. Nothing in the sync activity looks out of the ordinary (and I have read quite a few posts to be able to discern what the items inside mean) When a replication error occurs, often nothing at all is seen inside the logs (no error mesg, no segfault, nothing) to indicate a failure has occurred. Oftentimes, when SyncRepl fails to replicate, it is NOT during testing, so it is difficult for us to find out exactly WHEN the error occurred, as it could have happened at any point within the past 24 hrs, and up to a week ago. I can try to replicate this error deliberately. Unfortunately it is very difficult to make happen at-will. My concern is that someone will try to close this ticket before I have witnessed the event. Jeff On Oct 1, 2009, at 12:14 PM, Quanah Gibson-Mount wrote: > --On Thursday, October 01, 2009 3:19 AM +0000 j(a)telepaths.org wrote: > >> >> --Apple-Mail-6--965444022 >> Content-Transfer-Encoding: 7bit >> Content-Type: text/plain; >> charset=us-ascii; >> format=flowed > > Have you set the loglevel on all servers to include "sync" level > logging? > > --Quanah > > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6320) Replication Issues
by quanah＠zimbra.com 01 Oct '09

01 Oct '09

--On Thursday, October 01, 2009 3:19 AM +0000 j(a)telepaths.org wrote: > > --Apple-Mail-6--965444022 > Content-Transfer-Encoding: 7bit > Content-Type: text/plain; > charset=us-ascii; > format=flowed Have you set the loglevel on all servers to include "sync" level logging? --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6319) opened files
by hyc＠symas.com 01 Oct '09

01 Oct '09

M8R-njfnr31(a)mailinator.com wrote: > Full_Name: Miki Manó > Version: 2.4.18 > OS: debian linux > URL: > Submission from: (NULL) (91.120.159.49) > > > hello > > i'm running slapd server with 100 clients over ssl. i see the server don't > close > the file descriptors always and the limit of open files is exceeded. how could > i > help your work to find this bug? > > file limit is reached too with following options: > > idletimeout 60 > writetimeout 60 > > the creation time of file descriptors is older than 60s. Please provide the output from "lsof" on the slapd process. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

ITS#6312 published: SECURITY: VC++: stdio.h(358) : error C3163: '_vsnprintf': attributes inconsistent with previous declaration
by openldap-its＠OpenLDAP.org 01 Oct '09

01 Oct '09

Private ITS#6312 is now publicly readable URL: http://www.OpenLDAP.org/its/?findid=6312

1 0

ITS#6313 published: SECURITY: VC++: stdio.h(358) : error C3163: '_vsnprintf': attributes inconsistent with previous declaration
by openldap-its＠OpenLDAP.org 01 Oct '09

01 Oct '09

Private ITS#6313 is now publicly readable URL: http://www.OpenLDAP.org/its/?findid=6313

1 0

ITS#6314 published: SECURITY: VC++: stdio.h(358) : error C3163: '_vsnprintf': attributes inconsistent with previous declaration
by openldap-its＠OpenLDAP.org 01 Oct '09

01 Oct '09

Private ITS#6314 is now publicly readable URL: http://www.OpenLDAP.org/its/?findid=6314

1 0

ITS#6315 published: SECURITY: VC++: stdio.h(358) : error C3163: '_vsnprintf': attributes inconsistent with previous declaration
by openldap-its＠OpenLDAP.org 01 Oct '09

01 Oct '09

Private ITS#6315 is now publicly readable URL: http://www.OpenLDAP.org/its/?findid=6315

1 0

ITS#6316 published: SECURITY: VC++: stdio.h(358) : error C3163: '_vsnprintf': attributes inconsistent with previous declaration
by openldap-its＠OpenLDAP.org 01 Oct '09

01 Oct '09

Private ITS#6316 is now publicly readable URL: http://www.OpenLDAP.org/its/?findid=6316

1 0

ITS#6317 published: SECURITY: _vsnprintf
by openldap-its＠OpenLDAP.org 01 Oct '09

01 Oct '09

Private ITS#6317 is now publicly readable URL: http://www.OpenLDAP.org/its/?findid=6317

1 0

ITS#6318 published: SECURITY: vsnprintf
by openldap-its＠OpenLDAP.org 01 Oct '09

01 Oct '09

Private ITS#6318 is now publicly readable URL: http://www.OpenLDAP.org/its/?findid=6318

1 0

Re: (ITS#6217) proxycache not returning cached data
by w.van.keulen＠few.vu.nl 01 Oct '09

01 Oct '09

Howard Chu wrote: > w.van.keulen(a)few.vu.nl wrote: >> masarati(a)aero.polimi.it wrote: >>>> Here are slapd.conf and proxycache.conf as attachments. >>> >>> I just wanted to feed back before I leave for a few days. I don't see >>> anything critical in your files. You explicitly give the rootdn write >>> permissions, which is useless, but apart from this everything looks >>> fine. >>> >>> I've tested a simple setup and it works as expected. I used HEAD and >>> re24 >>> code; the latter is nearly identical to 2.4.17. I also tested with TLS. >>> >>> At this point, I'm a bit clueless. There must be something we're >>> overlooking. > >> Here is some more info, that I just discovered. >> If my filter rule is >> '(&(objectClass=sambaSamAccount)(uid=wvk))' >> the data is cached the first time, but not returned >> the second and later time. However when my filter rule is >> (&(objectClass=posixAccount)(uid=wvk))' >> it works as expected. > > Sounds like you may have already had the entry in the cache DB but with > an incompatible objectclass before, and the attempt to update it with > the new objectclass failed. This seems to be related to ITS#6242 which > was recently fixed in HEAD. Can you please retest with the latest > overlays/pcache.c? > As you requested I retested with HEAD. However the cache is still not working correctly. When my filter rule is: '(&(objectClass=posixAccount)(uid=<who>))' the named attributes are cached and correctly returned from the cache for every uid from the existing user base When my filter rule is: '(&(objectClass=sambaSamAccount)(uid=<who>))' the named attributes are cached and correctly returned from the cache for a subset of the existing user base only. For all uid's the query is cached (according to the log), but for some no data is returned. regards Jim van K -- // Set startup state to online user_pref("offline.startup_state", 2);

1 0

Re: (ITS#6320) Replication Issues
by j＠telepaths.org 01 Oct '09

01 Oct '09

--Apple-Mail-6--965444022 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii; format=flowed Forgive the typo: Our entire DIT is replicated in full, with no consistencies Where I meant: with no inconsistencies Sorry .. Jeff --Apple-Mail-6--965444022 Content-Transfer-Encoding: 7bit Content-Type: text/html; charset=us-ascii <html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Forgive the typo:<div><br></div><div><span class="Apple-style-span" style="font-family: monospace; white-space: pre; ">Our entire DIT is replicated in full, with no consistencies</span></div><div><div><br></div><div>Where I meant:</div><div><br></div><div><span class="Apple-style-span" style="font-family: monospace; white-space: pre; ">with no inconsistencies</span></div><div><div><br></div><div><div>Sorry ..</div><div><br></div><div>Jeff</div></div></div></div></body></html> --Apple-Mail-6--965444022--

1 0

(ITS#6320) Replication Issues
by j＠telepaths.org 01 Oct '09

01 Oct '09

Full_Name: Jeff Doyle Version: 2.4.17 OS: Debian/Lenny URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (68.15.14.98) Greetings, I am faced with a persistent replication issue for which I am seeking guidance. First, my infrastructure: My organization is running Debian/Lenny .... HOWEVER we have backported a future release of Debian's ''slapd'' release to be more in-line with OpenLDAP current releases (as the actual release of ''slapd'' in Lenny is 2.4.11 is too far-behind, given some of the posts I've read on your site). We are running 2.4.17. I am absolutely stumped with random replication inconsistencies that have me running in circles. We're having random issues involving records added to a provider server not being replicated to all consumers. This occurs on a very random basis. It can occur with a single record, or many. Because of the severity of this issue, I have written a pretty gnarly script to check the content of each server and when differences are found, send out an alert. This script is run via cron. I simply cannot determine why our replication fails when it does. Here is a SyncRepl stanza from a "consumer" LDAP server in question: syncrepl rid=001 provider=ldaps://10.9.8.3:636/ bindmethod=simple binddn="uid=syncrepl,dc=example,dc=com" credentials=password scope=sub filter="(objectClass=*)" schemachecking=off searchbase="dc=example,dc=com" retry="120 +" sizelimit=unlimited timeout=1 type=refreshOnly interval=00:00:07:00 BE ADVISED, I have tried variations of this stanza involving: * Changing from refreshOnly to refreshAndPersist (and subsequently removing the 'interval' parameter, as it is not needed with refreshAndPersist) * Trying longer and shorter intervals, when still using refreshOnly * Trying repeated "retry" integers both with and without the "forever (+)" parameter * Using NO encryption, and also using StartTLS over 389 as opposed to LDAPS/636 * With and without the "filter" parameter * With and without the schemachecking parameter * With and without the timeout parameter * Tried the 'network-timeout' parameter with a variety of integers (after reading the man page) * Using SASL/Kerberos for the bind method instead of simple binds * Using the ROOTDN for the user instead of our dedicated SyncRepl user (who has Confirmed 100% read-access to the DIT and ALL of its OCs and Attributes) Assuming we just built this slave (with the stanza above) in our production environment, starting slapd with any mix of debugging modes show normal syncrepl, ber, acl-processing and other activities in progress. Everything seems fine. Our entire DIT is replicated in full, with no consistencies. Network conditions between all hosts is relatively calm. No policy issues or routing issues exist. LDAP search operations from every possible angle are tested. The network is internally wide-open to itself. Everything looks fine on both the provider and consumer. We can add records to a Provider server, and the records (or changes to existing records, such as deletions) are replicated just fine to the consumer. We can try this for any length of time without errors. Using a variety of 'delays' between operations, bind-users, etc. All conceivable permutations end in success. OK, now ... wait any specified amount of time. A day or two, maybe a week (during which all of the servers are doing mostly READ operations). I receive an alert via email: Records have stopped replicating to one or more hosts. We don't understand why. The prescribed fix for it its to delete the slave's database and let it rebuild. But calling that solution a "Hack" doesn't quite tell you how i Really feel about it. Help. Please. This is not production-grade behavior, I think you would agree. If you want more of my slapd.conf or other details, Just ask, and I will promptly supply them. Thanks Jeff

1 0

(ITS#6312)
by neustradamus＠hotmail.com 30 Sep '09

30 Sep '09

--_add76be3-02e1-47b1-b978-259bce59b479_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable You must change in configure and \include\PORTABLE.H : #define vsnprintf _vsnprintf with #if _MSC_VER < 1500 // 1500 =3D=3D VC++ 9.0 #define vsnprintf _vsnprintf #endif or #if _MSC_VER < 1500 #define vsnprintf _vsnprintf #endif Sorry for issues ITS#6313 at ITS#6318 but I have an error : "The system encountered a fatal error can't get info in find_id The last error code was: No such file or directory" Thanks in advance=2C Regards=2C Neustradamus =0A= _________________________________________________________________=0A= In=E9dit ! Des Emotic=F4nes D=E9jant=E9es! Installez les dans votre Messeng= er !=0A= http://www.ilovemessenger.fr/Emoticones/EmoticonesDejantees.aspx= --_add76be3-02e1-47b1-b978-259bce59b479_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <style></style> </head> <body class=3D'hmmessage'> You must change in configure and \include\PORTABLE.H :<br><br>#define vsnpr= intf _vsnprintf<br><br>with<br><br>#if _MSC_VER &lt=3B 1500&nbsp=3B // 1500= =3D=3D VC++ 9.0<br>#define vsnprintf _vsnprintf<br> #endif<br><br>or<br><b= r> #if _MSC_VER &lt=3B 1500<br>#define vsnprintf _vsnprintf<br> #endif<br><= br> Sorry for issues ITS#6313 at ITS#6318 but I have an error :<br>"The sys= tem encountered a fatal error<br>can't get info in find_id<br>The last erro= r code was: No such file or directory"<br><br>Thanks in advance=2C<br><br>R= egards=2C<br><br>Neustradamus<br> <br /><hr />Souhaitez vous =A0= =AB=A0=EAtre au bureau sans y =EAtre=A0=BB=A0? <a href=3D'http://www.micros= oft.com/france/windows/bts/default.mspx' target=3D'_new'>Oui je le veux !</= a></body> </html>= --_add76be3-02e1-47b1-b978-259bce59b479_--

1 0

(ITS#6319) opened files
by M8R-njfnr31＠mailinator.com 30 Sep '09

30 Sep '09

Full_Name: Miki Manó Version: 2.4.18 OS: debian linux URL: Submission from: (NULL) (91.120.159.49) hello i'm running slapd server with 100 clients over ssl. i see the server don't close the file descriptors always and the limit of open files is exceeded. how could i help your work to find this bug? file limit is reached too with following options: idletimeout 60 writetimeout 60 the creation time of file descriptors is older than 60s.

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs