- openldap-bugs - openldap.org

Re: (ITS#5442) slapd_rq not locked before use bugfix
by rein＠basefarm.no 31 Mar '08

31 Mar '08

On Sun, 30 Mar 2008, hyc(a)symas.com wrote: > rein(a)tollevik.no wrote: >> On Sat, 29 Mar 2008, ando(a)sys-net.it wrote: >>> rein(a)basefarm.no wrote: >>>> I was seeing random failures of the test050-syncrepl-multimaster test. One of >>>> the failures was that it went into a tight loop traversing a circular runqueue >>>> it had managed to create in slapd_rq.task_list. It seems as this was caused by >>>> missing mutex locks around accesses to slapd_rq, which the patch uploaded to >>>> ftp://ftp.openldap.org/incoming/slapd_rq_lock.patch fixes. >>>> >>>> Before I applied this patch the test failed after being run a few times, with it >>>> it has now passed 100 times and is still counting. >>> locks in back-bdb/config.c should be pointless, as modifications to the >>> configuration should only occur while all threads are paused. The rest >>> makes sort of sense, but I'd leave it to Howard. > > Ignoring the ITS#5403 changes, I don't see anything here that isn't > config-related, therefore it's all running single-threaded. Now that the configuration can be changed dynamically (as this test does) I find it a bit odd that the config stuff should always be running single-threaded. But there is obviously much I don't know about the internals of slapd. > Of the "relevant" changes in syncrepl.c, I note that three out of the four > chunks of the patch are in code that is only run when using cn=config to > delete a syncrepl configuration, and test050 never performs that operation. > The remaining chunk only takes affect when adding syncrepl config, and again, > slapd is single-threaded for that. I have been experimenting a bit more with the patch, and it appears to be the locks added to add_syncrepl() that stopped the test from failing. The test uses ordinary ldapadd to add the syncrepl configuration after slapd has been started, which I would expect to indicate that it has entered the multi-threaded state. But again, it can revert to single-threaded when modifying the configuration for all I know. > I've also run test050 thru hundreds of iterations without any issue, without > these patches. If there's a problem in test050, I don't believe it's in this code. I have also run it several hundred times on other systems, it is only on one of the 4 systems I build slapd that it fails. That is, that test failed on one of the other systems with the 2.4.8 release, also with an infinite loop, but that dosn't say vary much in this context. The system where the test fail is the only multi-cpu system I have run it on, which I recon to be the reason why it only fails there. Without this patch the test fails about 20% of the times it is run on that system, the reason varies between slapd being killed due to abort or seg. fault (unknown why), or infinite loop traversing the circular slapd_rq. With the patch it has succeeded several thousand times in a row, which I account for more than just a coincident. So, as long as this patch doesn't create any deadlock I can't really see the problem with it and will stick to it in our version. I find it a bit easier to understand and verify code that requiers locking if the locks are always used, then I don't need to know the thread states that are allowed to call the functions as well to know whether a lock is needed or not.. Rein

1 0

Re: (ITS#5444) slapd seg. fault using mulitmirrored mode
by ando＠sys-net.it 31 Mar '08

31 Mar '08

ando(a)sys-net.it wrote: > mohel(a)web.de wrote: > >> The above log was generated using the test050 script just adding a line which >> enters information also to a second server. Perhaps this test would also make >> sense for further releases. > > Can you please post the modification to the script? Never mind, I could reproduce it. It seems to be a duplicate of ITS#5437. What actually happens is that if modifications are applied to all three servers, those applied to consumers do not persist, nor get replicated; on the contrary, the consumers crash later on as described in ITS#5437. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5445) syncprov double-free bugfix.
by ando＠sys-net.it 31 Mar '08

31 Mar '08

rein(a)basefarm.no wrote: > The patch at the end adds missing parenthesis around a negated flags bit test in > syncprov.c. Without them the test always fails, the entry is never duplicated > and a double-free occur when the a_nvals is free'ed in the next statement if the > same entry is sent to more than one recipient simultaneously. Applied. Please test. Thanks, p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5440) back-meta opens too many connections in case of error
by ando＠sys-net.it 31 Mar '08

31 Mar '08

ali.pouya(a)free.fr wrote: > I precise that The behaviour is not the same if the client binds with rootdn. I've committed a fix that removes dangling connections as the rootdn; I have been unable to reproduce the issue with regular users. Please provide more useful info in order to track the issue (slapd.conf, LDIF and requests). p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5444) slapd seg. fault using mulitmirrored mode
by ando＠sys-net.it 31 Mar '08

31 Mar '08

mohel(a)web.de wrote: > The above log was generated using the test050 script just adding a line which > enters information also to a second server. Perhaps this test would also make > sense for further releases. Can you please post the modification to the script? p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

(ITS#5445) syncprov double-free bugfix.
by rein＠basefarm.no 31 Mar '08

31 Mar '08

Full_Name: Rein Tollevik Version: CVS head OS: linux and solaris URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (81.93.160.250) The patch at the end adds missing parenthesis around a negated flags bit test in syncprov.c. Without them the test always fails, the entry is never duplicated and a double-free occur when the a_nvals is free'ed in the next statement if the same entry is sent to more than one recipient simultaneously. Rein Tollevik Basefarm AS Index: OpenLDAP/servers/slapd/overlays/syncprov.c diff -u OpenLDAP/servers/slapd/overlays/syncprov.c:1.9 OpenLDAP/servers/slapd/overlays/syncprov.c:1.10 --- OpenLDAP/servers/slapd/overlays/syncprov.c:1.9 Sun Mar 23 14:06:03 2008 +++ OpenLDAP/servers/slapd/overlays/syncprov.c Mon Mar 31 15:43:31 2008 @@ -2385,7 +2385,7 @@ } if ( !ap ) { - if ( !rs->sr_flags & REP_ENTRY_MODIFIABLE ) { + if ( !(rs->sr_flags & REP_ENTRY_MODIFIABLE) ) { rs->sr_entry = entry_dup( rs->sr_entry ); rs->sr_flags |= REP_ENTRY_MODIFIABLE|REP_ENTRY_MUSTBEFREED;

1 0

(ITS#5444) slapd seg. fault using mulitmirrored mode
by mohel＠web.de 31 Mar '08

31 Mar '08

Full_Name: Markus Version: 2.4.8 OS: SLES 10 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (212.185.43.218) When using multi master replication und start the tree from scratch, I'm able to ldapadd to one of the masters and all the stuff is replciated. But wenn afterwards ldapadd a new entry to one of the other servers the server crashes with the segmentation error. The following log outout is generated: ---------------------------------------------- conn=11 op=1 SEARCH RESULT tag=101 err=0 nentries=20 text= connection_get(19) connection_get(19): got connid=12 connection_read(19): checking for input on id=12 ber_get_next ber_get_next: tag 0x30 len 162 contents: ber_get_next conn=12 op=1 do_search ber_scanf fmt ({miiiib) ber: >>> dnPrettyNormal: <dc=example,dc=com> => ldap_bv2dn(dc=example,dc=com,0) <= ldap_bv2dn(dc=example,dc=com)=0 => ldap_dn2bv(272) <= ldap_dn2bv(dc=example,dc=com)=0 => ldap_dn2bv(272) <= ldap_dn2bv(dc=example,dc=com)=0 <<< dnPrettyNormal: <dc=example,dc=com>, <dc=example,dc=com> SRCH "dc=example,dc=com" 2 0 0 0 0 ber_scanf fmt (m) ber: filter: (objectClass=*) ber_scanf fmt ({M}}) ber: => get_ctrls ber_scanf fmt ({m) ber: ber_scanf fmt (m) ber: => get_ctrls: oid="1.3.6.1.4.1.4203.1.9.1.1" (noncritical) ber_scanf fmt ({i) ber: ber_scanf fmt (m) ber: ber_scanf fmt (b) ber: ber_scanf fmt (}) ber: <= get_ctrls: n=1 rc=0 err="" attrs: * + conn=12 op=1 SRCH base="dc=example,dc=com" scope=2 deref=0 filter="(objectClass=*)" conn=12 op=1 SRCH attr=* + => bdb_search bdb_dn2entry("dc=example,dc=com") search_candidates: base="dc=example,dc=com" (0x00000001) scope=2 => bdb_dn2idl("dc=example,dc=com") => bdb_equality_candidates (entryCSN) <= bdb_equality_candidates: (entryCSN) not indexed bdb_search_candidates: id=-1 first=1 last=20 bdb_search: 1 does not match filter bdb_search: 2 does not match filter bdb_search: 3 does not match filter bdb_search: 4 does not match filter bdb_search: 5 does not match filter bdb_search: 6 does not match filter bdb_search: 7 does not match filter bdb_search: 8 does not match filter bdb_search: 9 does not match filter bdb_search: 10 does not match filter bdb_search: 11 does not match filter bdb_search: 12 does not match filter bdb_search: 13 does not match filter bdb_search: 14 does not match filter bdb_search: 15 does not match filter bdb_search: 16 does not match filter bdb_search: 17 does not match filter bdb_search: 18 does not match filter bdb_search: 20 does not match filter send_ldap_result: conn=12 op=1 p=3 send_ldap_result: err=0 matched="" text="" => bdb_search bdb_dn2entry("dc=example,dc=com") search_candidates: base="dc=example,dc=com" (0x00000001) scope=2 => bdb_dn2idl("dc=example,dc=com") => bdb_presence_candidates (objectClass) bdb_search_candidates: id=-1 first=1 last=20 send_ldap_result: conn=12 op=1 p=3 send_ldap_result: err=0 matched="" text="" send_ldap_intermediate: err=0 oid=1.3.6.1.4.1.4203.1.9.1.4 len=368 send_ldap_response: msgid=2 tag=121 err=0 ber_flush2: 409 bytes to sd 19 send_ldap_result: conn=12 op=1 p=3 send_ldap_result: err=0 matched="" text="" slap_sl_malloc of 136867984 bytes failed, using ch_malloc ---------------------------------------------- It seems as the malloc requets of almost 130MB is simple to much. The above log was generated using the test050 script just adding a line which enters information also to a second server. Perhaps this test would also make sense for further releases.

1 0

Re: (ITS#5404) back-ldap fails intermittently with 80 Internal (implementation specific) error
by toby＠inf.ed.ac.uk 31 Mar '08

31 Mar '08

> Please test and report; > > servers/slapd/back-ldap/bind.c > new revision: 1.244; previous revision: 1.243 > > The patch applies straightforwardly to re23 only if the first chunk is > omitted. Been running for 3+ weeks now on various test machines with no recurrence of error, so I'd consider this resolved. Many thanks Toby

1 0

Re: (ITS#5442) slapd_rq not locked before use bugfix
by hyc＠symas.com 30 Mar '08

30 Mar '08

rein(a)tollevik.no wrote: > On Sat, 29 Mar 2008, ando(a)sys-net.it wrote: >> rein(a)basefarm.no wrote: >>> I was seeing random failures of the test050-syncrepl-multimaster test. One of >>> the failures was that it went into a tight loop traversing a circular runqueue >>> it had managed to create in slapd_rq.task_list. It seems as this was caused by >>> missing mutex locks around accesses to slapd_rq, which the patch uploaded to >>> ftp://ftp.openldap.org/incoming/slapd_rq_lock.patch fixes. >>> >>> Before I applied this patch the test failed after being run a few times, with it >>> it has now passed 100 times and is still counting. >> locks in back-bdb/config.c should be pointless, as modifications to the >> configuration should only occur while all threads are paused. The rest >> makes sort of sense, but I'd leave it to Howard. Ignoring the ITS#5403 changes, I don't see anything here that isn't config-related, therefore it's all running single-threaded. Of the "relevant" changes in syncrepl.c, I note that three out of the four chunks of the patch are in code that is only run when using cn=config to delete a syncrepl configuration, and test050 never performs that operation. The remaining chunk only takes affect when adding syncrepl config, and again, slapd is single-threaded for that. I've also run test050 thru hundreds of iterations without any issue, without these patches. If there's a problem in test050, I don't believe it's in this code. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5442) slapd_rq not locked before use bugfix
by rein＠tollevik.no 29 Mar '08

29 Mar '08

On Sat, 29 Mar 2008, ando(a)sys-net.it wrote: > rein(a)basefarm.no wrote: > >> I was seeing random failures of the test050-syncrepl-multimaster test. One of >> the failures was that it went into a tight loop traversing a circular runqueue >> it had managed to create in slapd_rq.task_list. It seems as this was caused by >> missing mutex locks around accesses to slapd_rq, which the patch uploaded to >> ftp://ftp.openldap.org/incoming/slapd_rq_lock.patch fixes. >> >> Before I applied this patch the test failed after being run a few times, with it >> it has now passed 100 times and is still counting. > > locks in back-bdb/config.c should be pointless, as modifications to the > configuration should only occur while all threads are paused. The rest > makes sort of sense, but I'd leave it to Howard. That is probably true, it looks as if the places in config.c where locks really are required already had them. My patch adds locks everywhere slapd_rq was used without them, as I don't have enough knowledge of the code to know which functions guarranteed to only be used when threads are not running. I believe that the important patch is to syncrepl.c, but I found I it best to add locks everywhere just to be on the safe side. Looking at my copy of the patch it appears that another syncrepl.c patch which I was sure I had edited out has slipped through anyhow :-(. It is the first two modifications related to ldap_get_option, please disregard them in this bugreport. They have been reported in ITS#5403. Rein

1 0

Re: (ITS#5442) slapd_rq not locked before use bugfix
by ando＠sys-net.it 29 Mar '08

29 Mar '08

rein(a)basefarm.no wrote: > I was seeing random failures of the test050-syncrepl-multimaster test. One of > the failures was that it went into a tight loop traversing a circular runqueue > it had managed to create in slapd_rq.task_list. It seems as this was caused by > missing mutex locks around accesses to slapd_rq, which the patch uploaded to > ftp://ftp.openldap.org/incoming/slapd_rq_lock.patch fixes. > > Before I applied this patch the test failed after being run a few times, with it > it has now passed 100 times and is still counting. locks in back-bdb/config.c should be pointless, as modifications to the configuration should only occur while all threads are paused. The rest makes sort of sense, but I'd leave it to Howard. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5340) REP_ENTRY_MODIFIABLE bug in dynlist
by ando＠sys-net.it 28 Mar '08

28 Mar '08

Hallvard B Furuseth wrote: > ando(a)sys-net.it wrote: > > [about REP_ENTRY_MUSTRELEASE] >> it is not >> clear what happens when a callback chain is interrupted by >> slap_null_cb() or similar, without getting to slap_send_search_entry(). >> This seems to indicate that callbacks should always provide a last >> resort means to release the resources they set; if read-only, by keeping >> track of what they sent; if modifiable, by freeing the contents of >> rs->sr_* if not NULL, setting it to NULL to prevent further cleanup. > > That sounds cumbersome, I hope slapd could take care of that somehow. Yes, it could: for example, by providing a helper like slap_entry2modifiable() that takes s SlapReply, checks the flags and does nothing if the entry is already modifiable, or copies it if not, releasing the original one if REP_ENTRY_MUSTRELEASE was set. > But I don't see how the be_release() code can work now. It sounds like > be->be_release() functions must check (how?) that the entry was created > by 'be', and otherwise pass it on to the next overlay/backend or > otherwise to entry_free(). Might involve mucking with op->o_bd and > sr_entry->e_private, I suppose. Except maybe I'm missing some existing > magic since slapd doesn't regularly crash... Yes, but that's trivial: e_private must be NULL for temporary entries, and copying the entry loses it (no one is supposed to muck with it expect the entry's creator). And the appropriate o_bd of a (non-modifiable) entry can be easily computed from the entry's DN. > be->be_release() does receive entries that were not created by 'be' or > at least not with be->be_fetch(), see openldap-devel thread 'slapd API' > in mar 2008. Might be a bug, but I'm not familiar with that code. >> Similarly, the existence of REP_ENTRY_MUSTBEFREED is not totally clear: >> in principle as soon as REP_ENTRY_MODIFYABLE is set, it should imply >> REP_ENTRY_MUSTBEFREED; the only difference in the semantics of the two >> is that REP_ENTRY_MODIFYABLE without REP_ENTRY_MUSTBEFREED implies that >> the callback that set the former will take care of freeing the entry; >> however, other callbacks may further modify it, so freeing temporary >> data should probably be left to the final handler. > > That's not my impression. MODIFIABLE would be that other modules than > the creator can modify the entry - but the creator might still be the > one who will free it. MUSTBEFREED is that the entry must be > entry_free()ed - the creator will not do it (or not do it unless that > flag is set). They clearly mean different things; my point is that as soon as an entry is modifiable it is not read-only and thus is a temporary, and thus will eventually need to be freed. So the point is who is actually going to free it. A copy might be created by an overlay after receiving a read-only entry, but the same overlay might not actually perform the copy if it receives the entry from another overlay that already copied it, or from a proxy backend. However, after the entry is copied the overlay will have no means to determined who actually created the copy. This might be an issue depending on the order cleanup handlers are called (didn't check what order they're called). My point is that temporary entries need to be freed at some point; who frees them should not be relevant... > So, if I'm getting this right... > > A backend must expect an entry to change or be freed if it sends the > entry with REP_ENTRY_<MUSTRELEASE, MUSTBEFREED or MODIFIABLE>, or if > it passes through a slap_callback.sc_response. Right now, a backend expects the entry to change if sent with MODIFIABLE; to be released if sent with MUSTBERELEASED; to be freed if sent with MUSTBEFREED. Modifications could occur to a MODIFIABLE entry when passing through a sc_callback(). Otherwise, if the entry is not MODIFIABLE, a copy will be created if the callback needs to modify the entry; after copying, the entry will be released if MUSTBERELEASED was set. When a copy is created, the callback that creates the copy must also clear the MUSTBERELEASED flag. Optionally, it may set the MUSTBEFREED flag if it doesn't intend to take care of cleaning the entry up after it's done. > back-ldif does not: it uses e_name/e_nname _after_ sending with with > REP_ENTRY_MODIFIABLE. That should be considered a bug. > Nor overlay retcode, it calls retcode_entry_response(,,,rs->sr_entry) > which makes pointers into sr_entry without checking if those flags are > set. If I'm getting that code correctly. I haven't tested. I havent' checked, but that might be a bug as well. > Others apparent problems (also not tested, I've just browsed the code): > > Overlays that obey and reset MUSTBEFREED or MUSTRELEASE, do not > necessarily clear or set MODIFIABLE when setting a new entry. > translucent does not, even though it is a careful one which > obeys both MUSTBEFREED and MUSTRELEASE. > > > I'm not sure when the code must clear the entry_related flags. > Some code which sets new entries seem to assume they are zero, > but some code which sets sr_entry=NULL does not clear the flags. > > There are sr_flags bits which are not about the entry, in particular > REP_MATCHED_MUSTBEFREED and REP_MATCHED_MUSTBEFREED. Looks like these > flags can get lost when but some code sets sr_flags = <something> > rather sr_flags |= <something> or sr_flags ~= ~<something>. It seems to me that we should provide "smart" handlers to deal with preparing sr_entry for modification, and to take care of cleaning things up as appropriate. Those helpers should then be consistently used in the code. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5443) Multiple identical attibutes break syncrepl process fatally?
by ando＠sys-net.it 28 Mar '08

28 Mar '08

marian.eichholz(a)freenet.ag wrote: > We use openldap as mail service directory with some 8 Mio objects on several > replicas. > For openldap 2.4.x we have to migrate from slurpd to syncrepl. > We got a working syncrepl provider als slurpd consumer (slapadd -q, 36 hrs) > > So I try to get a blank DB up by syncrepl only (yes, it is not at all > performant, but informative) > > The process kind of breaks after a couple of minutes and some 44.000 objects > (8.000.000 expected). Tracing it on the consumer side (-d 16384), I see > something like this after an entry: > > syncrepl_message_to_entry: rid=001 mods check (forwardto: value #4 provided more > than once) > > Indeed, the entry to come has three "forwardto:" Attributes with the same value > (and other forwardto-attributes, too). This makes no greate sense at the > application level, but until now it has been perfectly OK for the directory, and > the LDAP-API did not complain about the attribute modification, neither did the > slurpd. It is a violation of RFC 4512, section 2.2, which OpenLDAP 2.4 conforms to. > This leads to some questions and suggestions: > > - the provider does not log anything with -d 16384, no error, no nothing. Could > it do some useful logging about successful and failing replication sessions? What's -16384? Since OpenLDAP 2.3 you can use strings to identify each log subsystem (16384 == 0x4000 == "sync"). The error occurs when the consumer tries to manipulate the data it receives. The producer has nothing to do with it, since it assumes that data contained in it already passed sanity checks when they were stored. How incorrect data got stored into the producer is a totally different business, and the producer-side replication process should not muck with it. > - the consumer does not log anything that can explain, why the remaining objects > are not read, either. A bit of warning/logging could help the hopeful admin, > probably. A sync error occurred, which prevented sync'in from continuing. This error is logged by the "sync" subsystem. As far as I understand from reading the code, the error (at least, a replication error) should be logged also by the "any" subsystem, which means that as soon as any logging is enabled, you get a message logged. > - why is one problematic object lethal for the whole rest of the objects, since > future modifications keep to be incorporated? Is this lack of robustness more a > bug or a feature? If inconsistent data is received, synchronization is supposed to stop. In fact, continuing may result in an inconsistent state. The fact that the stop is caused by a real error, and the fact that fixing the error allows synchronization to recover doesn't sound like lack of robustness to me. It sounds more about wisdom. > - are identical attributes really forbidden with LDAP? RFC 4512, Section 2.2 > - what could one do, to prevent unskillful "editors" of the master node to kill > the replication processes for the whole replication cluster? Besides adding a > checking/filtering API layer, of course. Slapd has sanity checks for this. Slapadd doesn't, since it is supposed to be operated only with consistent data, as resulting from slapcat. You might have slapadd'ed inconsistent data to the producer. In the end, I don't see how this ITS involves a bug in synchronization software. The fact your producer got corrupted by inconsistent data might have been caused by a bug in the software, however your analysis does not give a clear indication of how it happened. If it happened by slapadd, then it's a known (and desired, and documented) limitation of the software. Unless you can reproduce it, I'd consider this ITS closed. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

(ITS#5443) Multiple identical attibutes break syncrepl process fatally?
by marian.eichholz＠freenet.ag 28 Mar '08

28 Mar '08

Full_Name: Marian Eichholz Version: 2.4.8 OS: Linux URL: Submission from: (NULL) (194.97.7.65) We use openldap as mail service directory with some 8 Mio objects on several replicas. For openldap 2.4.x we have to migrate from slurpd to syncrepl. We got a working syncrepl provider als slurpd consumer (slapadd -q, 36 hrs) So I try to get a blank DB up by syncrepl only (yes, it is not at all performant, but informative) The process kind of breaks after a couple of minutes and some 44.000 objects (8.000.000 expected). Tracing it on the consumer side (-d 16384), I see something like this after an entry: syncrepl_message_to_entry: rid=001 mods check (forwardto: value #4 provided more than once) Indeed, the entry to come has three "forwardto:" Attributes with the same value (and other forwardto-attributes, too). This makes no greate sense at the application level, but until now it has been perfectly OK for the directory, and the LDAP-API did not complain about the attribute modification, neither did the slurpd. This leads to some questions and suggestions: - the provider does not log anything with -d 16384, no error, no nothing. Could it do some useful logging about successful and failing replication sessions? - the consumer does not log anything that can explain, why the remaining objects are not read, either. A bit of warning/logging could help the hopeful admin, probably. - why is one problematic object lethal for the whole rest of the objects, since future modifications keep to be incorporated? Is this lack of robustness more a bug or a feature? - are identical attributes really forbidden with LDAP? - what could one do, to prevent unskillful "editors" of the master node to kill the replication processes for the whole replication cluster? Besides adding a checking/filtering API layer, of course. Thank You in advance. Please let me know, if I can provide You with something useful information about our issue(s).

1 0

(ITS#5442) slapd_rq not locked before use bugfix
by rein＠basefarm.no 28 Mar '08

28 Mar '08

Full_Name: Rein Tollevik Version: CVS head OS: linux and solaris URL: ftp://ftp.openldap.org/incoming/slapd_rq_lock.patch Submission from: (NULL) (81.93.160.250) I was seeing random failures of the test050-syncrepl-multimaster test. One of the failures was that it went into a tight loop traversing a circular runqueue it had managed to create in slapd_rq.task_list. It seems as this was caused by missing mutex locks around accesses to slapd_rq, which the patch uploaded to ftp://ftp.openldap.org/incoming/slapd_rq_lock.patch fixes. Before I applied this patch the test failed after being run a few times, with it it has now passed 100 times and is still counting. Rein Tollevik Basefarm AS

1 0

(ITS#5441) Avoid unneccessary time() syscall in auditlog overlay
by rein＠basefarm.no 28 Mar '08

28 Mar '08

Full_Name: Rein Tollevik Version: CVS head OS: linux and solaris URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (81.93.160.250) Below is a minor enhancement to the auditlog overlay, don't fetch the time() until it is known that it is needed. Which it isn't on all those search responses that passes through it.. Rein Tollevik Basefarm AS Index: OpenLDAP/servers/slapd/overlays/auditlog.c diff -u OpenLDAP/servers/slapd/overlays/auditlog.c:1.1.1.8 OpenLDAP/servers/slapd/overlays/auditlog.c:1.2 --- OpenLDAP/servers/slapd/overlays/auditlog.c:1.1.1.8 Sat Mar 22 16:48:17 2008 +++ OpenLDAP/servers/slapd/overlays/auditlog.c Wed Mar 26 22:58:55 2008 @@ -74,7 +74,7 @@ Modifications *m; struct berval *b, *who = NULL; char *what, *suffix; - long stamp = slap_get_time(); + time_t stamp; int i; if ( rs->sr_err != LDAP_SUCCESS ) return SLAP_CB_CONTINUE; @@ -125,6 +125,7 @@ return SLAP_CB_CONTINUE; } + stamp = slap_get_time(); fprintf(f, "# %s %ld %s%s%s\n", what, stamp, suffix, who ? " " : "", who ? who->bv_val : "");

1 0

Re: (ITS#5439) syncprov race condition seg. fault
by h.b.furuseth＠usit.uio.no 28 Mar '08

28 Mar '08

rein(a)basefarm.no writes: > It now looks to me as if the entire rs->sr_entry was released an > reused, (...) Could this be ITS#5340? -- Hallvard

1 0

Re: (ITS#5340) REP_ENTRY_MODIFIABLE bug in dynlist
by h.b.furuseth＠usit.uio.no 28 Mar '08

28 Mar '08

ando(a)sys-net.it wrote: [about REP_ENTRY_MUSTRELEASE] > it is not > clear what happens when a callback chain is interrupted by > slap_null_cb() or similar, without getting to slap_send_search_entry(). > This seems to indicate that callbacks should always provide a last > resort means to release the resources they set; if read-only, by keeping > track of what they sent; if modifiable, by freeing the contents of > rs->sr_* if not NULL, setting it to NULL to prevent further cleanup. That sounds cumbersome, I hope slapd could take care of that somehow. But I don't see how the be_release() code can work now. It sounds like be->be_release() functions must check (how?) that the entry was created by 'be', and otherwise pass it on to the next overlay/backend or otherwise to entry_free(). Might involve mucking with op->o_bd and sr_entry->e_private, I suppose. Except maybe I'm missing some existing magic since slapd doesn't regularly crash... be->be_release() does receive entries that were not created by 'be' or at least not with be->be_fetch(), see openldap-devel thread 'slapd API' in mar 2008. > Similarly, the existence of REP_ENTRY_MUSTBEFREED is not totally clear: > in principle as soon as REP_ENTRY_MODIFYABLE is set, it should imply > REP_ENTRY_MUSTBEFREED; the only difference in the semantics of the two > is that REP_ENTRY_MODIFYABLE without REP_ENTRY_MUSTBEFREED implies that > the callback that set the former will take care of freeing the entry; > however, other callbacks may further modify it, so freeing temporary > data should probably be left to the final handler. That's not my impression. MODIFIABLE would be that other modules than the creator can modify the entry - but the creator might still be the one who will free it. MUSTBEFREED is that the entry must be entry_free()ed - the creator will not do it (or not do it unless that flag is set). So, if I'm getting this right... A backend must expect an entry to change or be freed if it sends the entry with REP_ENTRY_<MUSTRELEASE, MUSTBEFREED or MODIFIABLE>, or if it passes through a slap_callback.sc_response. back-ldif does not: it uses e_name/e_nname _after_ sending with with REP_ENTRY_MODIFIABLE. Nor overlay retcode, it calls retcode_entry_response(,,,rs->sr_entry) which makes pointers into sr_entry without checking if those flags are set. If I'm getting that code correctly. I haven't tested. Others apparent problems (also not tested, I've just browsed the code): Overlays that obey and reset MUSTBEFREED or MUSTRELEASE, do not necessarily clear or set MODIFIABLE when setting a new entry. translucent does not, even though it is a careful one which obeys both MUSTBEFREED and MUSTRELEASE. I'm not sure when the code must clear the entry_related flags. Some code which sets new entries seem to assume they are zero, but some code which sets sr_entry=NULL does not clear the flags. There are sr_flags bits which are not about the entry, in particular REP_MATCHED_MUSTBEFREED and REP_MATCHED_MUSTBEFREED. Looks like these flags can get lost when but some code sets sr_flags = <something> rather sr_flags |= <something> or sr_flags ~= ~<something>. -- Hallvard

1 0

Re: ITS#5391 [Fwd: [Berkeley DB Announce] Berkeley DB 4.6.21.1 - Patch Announcement]
by quanah＠zimbra.com 28 Mar '08

28 Mar '08

--On Thursday, March 27, 2008 10:44 PM +0000 bdb(a)oss.oracle.com wrote: > I know you're not using BDB 4.6 with OpenLDAP 2.3, so this patch isn't > strictly relevant to this ITS. It just raises the possibility that there > is a BDB bug involved. Looking at the code for 4.2.52, the mp_fopen.c code is essentially identical to what this patch fixes, so I'd guess it could affect 2.3 releases using bdb 4.2->4.5. And I'm guessing there'll be no patch from Oracle for those much older releases. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

ITS#5391 [Fwd: [Berkeley DB Announce] Berkeley DB 4.6.21.1 - Patch Announcement]
by bdb＠oss.oracle.com 27 Mar '08

27 Mar '08

I know you're not using BDB 4.6 with OpenLDAP 2.3, so this patch isn't strictly relevant to this ITS. It just raises the possibility that there is a BDB bug involved. -------- Original Message -------- Subject: [Berkeley DB Announce] Berkeley DB 4.6.21.1 - Patch Announcement Date: Thu, 27 Mar 2008 18:03:40 -0400 From: bdb(a)oss.oracle.com Reply-To: bdb(a)oss.oracle.com Organization: Oracle Corporation To: <bdb(a)oss.oracle.com> A patch for Berkeley DB release 4.6.21 that fixes a race condition between checkpoint and DB->close() which can result in the checkpoint thread self-deadlocking (SR [#15692]) is now available. We recommend all users of Berkeley DB apply this patch so as to avoid this issue in production. An web page with all patches (currently only one) for Berkeley DB 4.6.21. http://www.oracle.com/technology/products/berkeley-db/xml/update/4.6.21/patc h.4.6.21.html A text file with the patch: http://www.oracle.com/technology/products/berkeley-db/xml/update/4.6.21/patc h.4.6.21.1 regards, the Berkeley DB Team, -greg _____________________________________________________________________ Gregory Burd greg.burd(a)oracle.com Product Manager, Berkeley DB/JE/XML Oracle Corporation _______________________________________________ BDB mailing list BDB(a)oss.oracle.com http://oss.oracle.com/mailman/listinfo/bdb -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5440) back-meta opens too many connections in case of error
by ando＠sys-net.it 26 Mar '08

26 Mar '08

ali.pouya(a)free.fr wrote: > Full_Name: Ali Pouya > Version: 2.4.8 > OS: Linux 2.6 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (145.242.11.3) > > > Hi, > I'm testing the back-meta of OpenLdap 2.4.8 in order to upgrade to this > release. > My client opens a persistent LDAP connection to slapd, binds with an ordinary > account (from the local back-bdb) and begins search operations. > > I notice that if a search operation results in an LDAP error (for example error > 32 : No Such Object), then back-meta opens a new connection to the target > directory for the next operation, leaving the "bad" connection open. > > The conn-ttl and idle-timeout parameters do not close the "bad" connections. > These connections remain there until the client ends its connection. > > This saturates the target servers with unused idle connections. > > Is this a bug or a normal behaviour ? > > I precise that The behaviour is not the same if the client binds with rootdn. 1) Can you post your slapd.conf (sanitized)? 2) What is exactly causing the error? What error is actually being returned from all targets? For back-meta, no such object has some special semantics. Best would be if you can provide a setup that allows to reproduce the critical situation with a very simple test, involving as little targets as possible, along with the LDIF required to populate each target and the operation you used. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

(ITS#5440) back-meta opens too many connections in case of error
by ali.pouya＠free.fr 26 Mar '08

26 Mar '08

Full_Name: Ali Pouya Version: 2.4.8 OS: Linux 2.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (145.242.11.3) Hi, I'm testing the back-meta of OpenLdap 2.4.8 in order to upgrade to this release. My client opens a persistent LDAP connection to slapd, binds with an ordinary account (from the local back-bdb) and begins search operations. I notice that if a search operation results in an LDAP error (for example error 32 : No Such Object), then back-meta opens a new connection to the target directory for the next operation, leaving the "bad" connection open. The conn-ttl and idle-timeout parameters do not close the "bad" connections. These connections remain there until the client ends its connection. This saturates the target servers with unused idle connections. Is this a bug or a normal behaviour ? I precise that The behaviour is not the same if the client binds with rootdn. Thanks for your help Best regards Ali

1 0

Re: (ITS#5439) back-bdb race condition
by hyc＠symas.com 26 Mar '08

26 Mar '08

Rein Tollevik wrote: > On Tue, 25 Mar 2008, hyc(a)symas.com wrote: > It now looks to me as if the entire rs->sr_entry was released an reused, > and that the bug probably is in back-bdb. It just always happened when > syncprov was running as this is a master server with mostly writes > except from the reads syncprov does. Which also means that the title on > the bug report is probably misleading :-( > > The reason for my shift of focus is some highly suspicious variables > found in the bdb_search() frame. Look at the value of idflag in bdb_search. (see the comment around 672) Could be a locking bug with ID_NOCACHE in cache.c... > First, ei->bei_e seem to be NULL every time this happen, not the same as > the "e" variable as I would except from looking at the code. Second, > e->e_id is either 0 or ei->bei_id+1, the content of *e is the (more or > less completely initialized) entry following the *ei entry in the > database, while I would expect that ei->bei_id==e->e_id and that > ei->bei_e==e. I don't think the consecutive ID numbers or entries has > anything with the bug to do though, they are probably a result of the > searches reading the entire database. > > I looks to me as if the EntryInfo referred to by ei was released somehow > when slap_send_search_entry() was running, and that the ei->bei_e passed > as rs->rs_entry was first released and then reused for the next entry. > Alternatively that the structures wasn't properly initialized and > thereby appeared free to the second thread. > > I have found a core file that adds to my suspicions, this time with two > active threads. Their stack frames are shown at the end. In this core > file the id variable in the bdb_search() frame is 9318, the second > thread is loading id==9319. The value of the "e" variable in > bdb_search() for the first thread equals the "x" variable of the > entry_decode() frame in the second thread. Since entry_decode() has > just allocated its "x" value it shouldn't have found the same value as > the other thread has unless someone released its "e" entry while it was > still using it. Why this happened I haven't figured out yet. > > The cachsesize directive has its default value, which is way below the > optimal value. The result is that the cache is always full. I don't > know if that can have any influence on the bug? I tried to lower its > value to see if that should help reproducing the problem, but not so > far. > >>> I'm currently trying to gather more information related to this bug, any >>> pointers as to what I should look for is appreciated. I'm posting this bug >>> report now in the hope that the stack frame should enlighten someone with better >>> knowledge of the code than what I have. >> Check for stack overruns, compile without optimization and make sure it's not >> a compiler optimization bug, etc. > > I'll try to run it with a version compiled with only -g and see if the > same thing should happen then. > -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5439) syncprov race condition seg. fault
by rein＠basefarm.no 26 Mar '08

26 Mar '08

On Tue, 25 Mar 2008, hyc(a)symas.com wrote: > rein(a)basefarm.no wrote: >> We have bin hit by what looks like a race condition bug in syncprov. We got >> some core dumps all showing stack frames like the one at the end. As such nasty >> bugs tends to do it have behaved OK after I restarted slapd with more debug >> output :-( (trace + stats + stats2 + sync). >> >> >> Note the a=0xBAD argument to attr_find(), which I expect is the result of some >> other thread freeing the attribute list it was called with while it was >> processing it. The rs->sr_entry->e_attrs argument passed to attr_find() as the >> original "a" argument by findpres_cb() looks like a perfectly valid structure, >> as are all the attributes found by following the a_next pointer. The list is >> terminated by an attribute with a NULL a_next value, none of the a_next values >> are 0xBAD. > > I don't believe that's the cause. Notice that arg0 in stack frame #9 is also > 0xbad, even though it is shown correctly in frames 8 and 10. Something else is > going on. I don't know how I managed to overlook the strange 0xbad argument in frame #9, and I really haven't any explanation as to why it shows that value. But if I am on the right track in my assumptions below I currently writes is off as an unexpected result of optimization. It now looks to me as if the entire rs->sr_entry was released an reused, and that the bug probably is in back-bdb. It just always happened when syncprov was running as this is a master server with mostly writes except from the reads syncprov does. Which also means that the title on the bug report is probably misleading :-( The reason for my shift of focus is some highly suspicious variables found in the bdb_search() frame. First, ei->bei_e seem to be NULL every time this happen, not the same as the "e" variable as I would except from looking at the code. Second, e->e_id is either 0 or ei->bei_id+1, the content of *e is the (more or less completely initialized) entry following the *ei entry in the database, while I would expect that ei->bei_id==e->e_id and that ei->bei_e==e. I don't think the consecutive ID numbers or entries has anything with the bug to do though, they are probably a result of the searches reading the entire database. I looks to me as if the EntryInfo referred to by ei was released somehow when slap_send_search_entry() was running, and that the ei->bei_e passed as rs->rs_entry was first released and then reused for the next entry. Alternatively that the structures wasn't properly initialized and thereby appeared free to the second thread. I have found a core file that adds to my suspicions, this time with two active threads. Their stack frames are shown at the end. In this core file the id variable in the bdb_search() frame is 9318, the second thread is loading id==9319. The value of the "e" variable in bdb_search() for the first thread equals the "x" variable of the entry_decode() frame in the second thread. Since entry_decode() has just allocated its "x" value it shouldn't have found the same value as the other thread has unless someone released its "e" entry while it was still using it. Why this happened I haven't figured out yet. The cachsesize directive has its default value, which is way below the optimal value. The result is that the cache is always full. I don't know if that can have any influence on the bug? I tried to lower its value to see if that should help reproducing the problem, but not so far. >> I'm currently trying to gather more information related to this bug, any >> pointers as to what I should look for is appreciated. I'm posting this bug >> report now in the hope that the stack frame should enlighten someone with better >> knowledge of the code than what I have. > > Check for stack overruns, compile without optimization and make sure it's not > a compiler optimization bug, etc. I'll try to run it with a version compiled with only -g and see if the same thing should happen then. Rein Acive thread when segmentation fault occured: #0 0x0807d03a in attr_find (a=0xbad, desc=0x81e8680) at attr.c:665 #1 0xb7a656f6 in findpres_cb (op=0xac05cba4, rs=0xac05cb68) at syncprov.c:546 #2 0x0808416d in slap_response_play (op=0xac05cba4, rs=0xac05cb68) at result.c:307 #3 0x0808555b in slap_send_search_entry (op=0xac05cba4, rs=0xac05cb68) at result.c:770 #4 0x080f2cdc in bdb_search (op=0xac05cba4, rs=0xac05cb68) at search.c:870 #5 0x080db72b in overlay_op_walk (op=0xac05cba4, rs=0xac05cb68, which=op_search, oi=0x8274218, on=0x8274318) at backover.c:653 #6 0x080dbcaf in over_op_func (op=0xac05cba4, rs=0xac05cb68, which=op_search) at backover.c:705 #7 0x080dbdef in over_op_search (op=0xac05cba4, rs=0xac05cb68) at backover.c:727 #8 0x080d9570 in glue_sub_search (op=0xac05cba4, rs=0xac05cb68, b0=0xac05cba4, on=0xac05cba4) at backglue.c:340 #9 0x080da131 in glue_op_search (op=0xbad, rs=0xac05cb68) at backglue.c:459 #10 0x080db6d5 in overlay_op_walk (op=0xac05cba4, rs=0xac05cb68, which=op_search, oi=0x8271860, on=0x8271a60) at backover.c:643 #11 0x080dbcaf in over_op_func (op=0xac05cba4, rs=0xac05cb68, which=op_search) at backover.c:705 #12 0x080dbdef in over_op_search (op=0xac05cba4, rs=0xac05cb68) at backover.c:727 #13 0xb7a65ff4 in syncprov_findcsn (op=0x85085b8, mode=FIND_PRESENT) at syncprov.c:700 #14 0xb7a670a0 in syncprov_op_search (op=0x85085b8, rs=0xac05e1c0) at syncprov.c:2277 #15 0x080db6d5 in overlay_op_walk (op=0x85085b8, rs=0xac05e1c0, which=op_search, oi=0x8271860, on=0x8271b60) at backover.c:643 #16 0x080dbcaf in over_op_func (op=0x85085b8, rs=0xac05e1c0, which=op_search) at backover.c:705 #17 0x080dbdef in over_op_search (op=0x85085b8, rs=0xac05e1c0) at backover.c:727 #18 0x08076554 in fe_op_search (op=0x85085b8, rs=0xac05e1c0) at search.c:368 #19 0x080770e4 in do_search (op=0x85085b8, rs=0xac05e1c0) at search.c:217 #20 0x08073e28 in connection_operation (ctx=0xac05e2b8, arg_v=0x85085b8) at connection.c:1084 #21 0x08074f14 in connection_read_thread (ctx=0xac05e2b8, argv=0x3d) at connection.c:1211 #22 0xb7fb5546 in ldap_int_thread_pool_wrapper (xpool=0x81ee240) at tpool.c:663 #23 0xb7c80371 in start_thread () from /lib/tls/libpthread.so.0 #24 0xb7c17ffe in clone () from /lib/tls/libc.so.6 Thead loading the next entry: #0 0x08123ef6 in avl_find (root=0x81e7940, data=0xae7a62e8, fcmp=0x80b8871 <attr_index_name_cmp>) at avl.c:541 #1 0x080b7852 in at_bvfind (name=0xae7a62e8) at at.c:130 #2 0x080b5fd8 in slap_bv2ad (bv=0xae7a6354, ad=0xae7a6360, text=0xae7a6364) at ad.c:189 #3 0x0807eef5 in entry_decode (eh=0xae7a63bc, e=0xae7a64a8) at entry.c:859 #4 0x0810d849 in bdb_id2entry (be=0xae8666c0, tid=0x0, locker=0xb742da7c, id=9319, e=0xae7a64a8) at id2entry.c:169 #5 0x08104ffb in bdb_cache_find_id (op=0xae866ba4, tid=0x0, id=9319, eip=0xae7a6614, flag=2, locker=0xb742da7c, lock=0xae7a65f4) at cache.c:834 #6 0x080f2682 in bdb_search (op=0xae866ba4, rs=0xae866b68) at search.c:684 #7 0x080db72b in overlay_op_walk (op=0xae866ba4, rs=0xae866b68, which=op_search, oi=0x8274218, on=0x8274318) at backover.c:653 #8 0x080dbcaf in over_op_func (op=0xae866ba4, rs=0xae866b68, which=op_search) at backover.c:705 #9 0x080dbdef in over_op_search (op=0xae866ba4, rs=0xae866b68) at backover.c:727 #10 0x080d9570 in glue_sub_search (op=0xae866ba4, rs=0xae866b68, b0=0x81e7930, on=0xae866ba4) at backglue.c:340 #11 0x080da131 in glue_op_search (op=0xfffffffa, rs=0xae866b68) at backglue.c:459 #12 0x080db6d5 in overlay_op_walk (op=0xae866ba4, rs=0xae866b68, which=op_search, oi=0x8271860, on=0x8271a60) at backover.c:643 #13 0x080dbcaf in over_op_func (op=0xae866ba4, rs=0xae866b68, which=op_search) at backover.c:705 #14 0x080dbdef in over_op_search (op=0xae866ba4, rs=0xae866b68) at backover.c:727 #15 0xb7a65ff4 in syncprov_findcsn (op=0x8508e60, mode=FIND_PRESENT) at syncprov.c:700 #16 0xb7a670a0 in syncprov_op_search (op=0x8508e60, rs=0xae8681c0) at syncprov.c:2277 #17 0x080db6d5 in overlay_op_walk (op=0x8508e60, rs=0xae8681c0, which=op_search, oi=0x8271860, on=0x8271b60) at backover.c:643 #18 0x080dbcaf in over_op_func (op=0x8508e60, rs=0xae8681c0, which=op_search) at backover.c:705 #19 0x080dbdef in over_op_search (op=0x8508e60, rs=0xae8681c0) at backover.c:727 #20 0x08076554 in fe_op_search (op=0x8508e60, rs=0xae8681c0) at search.c:368 #21 0x080770e4 in do_search (op=0x8508e60, rs=0xae8681c0) at search.c:217 #22 0x08073e28 in connection_operation (ctx=0xae8682b8, arg_v=0x8508e60) at connection.c:1084 #23 0x08074f14 in connection_read_thread (ctx=0xae8682b8, argv=0x54) at connection.c:1211 #24 0xb7fb5546 in ldap_int_thread_pool_wrapper (xpool=0x81ee240) at tpool.c:663 #25 0xb7c80371 in start_thread () from /lib/tls/libpthread.so.0 #26 0xb7c17ffe in clone () from /lib/tls/libc.so.6

1 0

Re: ACLs broken by ITS#5419
by hyc＠symas.com 25 Mar '08

25 Mar '08

Rein Tollevik wrote: > On Mon, 24 Mar 2008, Howard Chu wrote: > >> rein(a)basefarm.no wrote: >>> The change to servers/slapd/backend.c for ITS#5416 seem to have broken the >>> ability for group and set statements in access control lines to refer to >>> entries >>> outside the backend currently being operated on. >> That ability was never intended in the first place. Historically, backends in >> slapd have been treated as isolated DSAs with no connection to each other. >> They've required special mechanisms (like back-relay or slapo-glue) to be >> joined. > > Yes, I know, the change that allowed this was imo the one that made sets > and groups really useful. Our database configuration still has traces of > the workarounds the lack of this feature once forced us to make.. > > But, the latest change also removes this ability for databases subordinate > to the same common superior (i.e using the slapo-glue). If I understand > you correct it is a bug that glue'ed databases cannot refer to each other, > although I still consider it a bug (or at least a huge drawback) if this > would no longer be generally possible. Actually, looking back over CVS, it seems this ability has existed since OpenLDAP 2.0, intended or not. Will have to work up a better solution to restore that behavior. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5439) syncprov race condition seg. fault
by hyc＠symas.com 25 Mar '08

25 Mar '08

rein(a)basefarm.no wrote: > Full_Name: Rein Tollevik > Version: CVS head > OS: CentOS 4.4 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (81.93.160.250) > > > We have bin hit by what looks like a race condition bug in syncprov. We got > some core dumps all showing stack frames like the one at the end. As such nasty > bugs tends to do it have behaved OK after I restarted slapd with more debug > output :-( (trace + stats + stats2 + sync). > > The configuration is a master server with multiple bdb backend databases all > being subordinate to the same glue database where syncprov is used. One of the > backends is a syncrepl consumer from another server, the server is master for > the other backends. There are multiple consumers for the syncprov suffix, which > I assume is what causes the race condition to happen. > > Note the a=0xBAD argument to attr_find(), which I expect is the result of some > other thread freeing the attribute list it was called with while it was > processing it. The rs->sr_entry->e_attrs argument passed to attr_find() as the > original "a" argument by findpres_cb() looks like a perfectly valid structure, > as are all the attributes found by following the a_next pointer. The list is > terminated by an attribute with a NULL a_next value, none of the a_next values > are 0xBAD. I don't believe that's the cause. Notice that arg0 in stack frame #9 is also 0xbad, even though it is shown correctly in frames 8 and 10. Something else is going on. > I'm currently trying to gather more information related to this bug, any > pointers as to what I should look for is appreciated. I'm posting this bug > report now in the hope that the stack frame should enlighten someone with better > knowledge of the code than what I have. Check for stack overruns, compile without optimization and make sure it's not a compiler optimization bug, etc. > > Rein Tollevik > Basefarm AS > > #0 0x0807d03a in attr_find (a=0xbad, desc=0x81e8680) at attr.c:665 > #1 0xb7a656f6 in findpres_cb (op=0xaf068ba4, rs=0xaf068b68) at syncprov.c:546 > #2 0x0808416d in slap_response_play (op=0xaf068ba4, rs=0xaf068b68) at > result.c:307 > #3 0x0808555b in slap_send_search_entry (op=0xaf068ba4, rs=0xaf068b68) at > result.c:770 > #4 0x080f2cdc in bdb_search (op=0xaf068ba4, rs=0xaf068b68) at search.c:870 > #5 0x080db72b in overlay_op_walk (op=0xaf068ba4, rs=0xaf068b68, > which=op_search, oi=0x8274218, on=0x8274318) at backover.c:653 > #6 0x080dbcaf in over_op_func (op=0xaf068ba4, rs=0xaf068b68, which=op_search) > at backover.c:705 > #7 0x080dbdef in over_op_search (op=0xaf068ba4, rs=0xaf068b68) at > backover.c:727 > #8 0x080d9570 in glue_sub_search (op=0xaf068ba4, rs=0xaf068b68, b0=0xaf068ba4, > on=0xaf068ba4) at backglue.c:340 > #9 0x080da131 in glue_op_search (op=0xbad, rs=0xaf068b68) at backglue.c:459 > #10 0x080db6d5 in overlay_op_walk (op=0xaf068ba4, rs=0xaf068b68, > which=op_search, oi=0x8271860, on=0x8271a60) at backover.c:643 > #11 0x080dbcaf in over_op_func (op=0xaf068ba4, rs=0xaf068b68, which=op_search) > at backover.c:705 > #12 0x080dbdef in over_op_search (op=0xaf068ba4, rs=0xaf068b68) at > backover.c:727 > #13 0xb7a65ff4 in syncprov_findcsn (op=0x85c7e60, mode=FIND_PRESENT) at > syncprov.c:700 > #14 0xb7a670a0 in syncprov_op_search (op=0x85c7e60, rs=0xaf06a1c0) at > syncprov.c:2277 > #15 0x080db6d5 in overlay_op_walk (op=0x85c7e60, rs=0xaf06a1c0, which=op_search, > oi=0x8271860, on=0x8271b60) at backover.c:643 > #16 0x080dbcaf in over_op_func (op=0x85c7e60, rs=0xaf06a1c0, which=op_search) at > backover.c:705 > #17 0x080dbdef in over_op_search (op=0x85c7e60, rs=0xaf06a1c0) at > backover.c:727 > #18 0x08076554 in fe_op_search (op=0x85c7e60, rs=0xaf06a1c0) at search.c:368 > #19 0x080770e4 in do_search (op=0x85c7e60, rs=0xaf06a1c0) at search.c:217 > #20 0x08073e28 in connection_operation (ctx=0xaf06a2b8, arg_v=0x85c7e60) at > connection.c:1084 > #21 0x08074f14 in connection_read_thread (ctx=0xaf06a2b8, argv=0x59) at > connection.c:1211 > #22 0xb7fb5546 in ldap_int_thread_pool_wrapper (xpool=0x81ee240) at tpool.c:663 > #23 0xb7c80371 in start_thread () from /lib/tls/libpthread.so.0 > #24 0xb7c17ffe in clone () from /lib/tls/libc.so.6 -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5340) REP_ENTRY_MODIFIABLE bug in dynlist
by ando＠sys-net.it 25 Mar '08

25 Mar '08

h.b.furuseth(a)usit.uio.no wrote: > h.b.furuseth(a)usit.uio.no writes: >> overlays/dynlist.c lines 371-376 sets REP_ENTRY_MUSTBEFREED in rs->sr_flags >> without duplicating the entry, if REP_ENTRY_MODIFIABLE was set. >> Thus the entry gets freed twice. Breaks test044-dynlist with back-ldif. > > Actually obeying rs->sr_flags seems to be a more general problem - but I > don't know what are bugs and what are my lacking understanding of the > flags. I.e. when can an overlay change REP_ENTRY_MUSTBEFREED, > REP_ENTRY_MUSTRELEASE? Usually backends/overlays that do not set these > flags, don't seem to expect them to change either. In general, callbacks are supposed to know about the existence of rs->sr_flags and deal with them. In detail, callbacks should use mechanisms that allow them to dispose of read-only resources without counting on the fact that rs->sr_* fields will be persistent through the callback chain. So, callbacks that need to modify data should check whether that data is modifiable; if it's not, they should first copy it, and set the rs->sr_flags as appropriate. The main reason for those flags to exist is optimization, otherwise we'd always duplicate data and dispose of it when done. So back-bdb/back-hdb, which are supposed to be the high-end database implementations, will always pass read-only data to the callback chain; however, if an overlay needs to muck with that data, it will first copy it, and set the REP_ENTRY_MODIFYABLE to inform other overlays that data can be freely modified without copying it again. On its own, back-bdb/back-hdb will take care of releasing the original entry without the need to use the value of rs->sr_entry; it'll use its copy of the entry's pointer. The existence of the REP_ENTRY_MUSTRELEASE value is not totally clear to me; it seems to indicate that in some cases callbacks want to pass a read-only entry without providing further cleanup code to release it; it makes sense, since this simple operation can be done by slap_send_search_entry() instead, if all that's required; however, this means that any callback that replaces the entry must remember to release it and must muck with rs->sr_flags accordingly. However, it is not clear what happens when a callback chain is interrupted by slap_null_cb() or similar, without getting to slap_send_search_entry(). This seems to indicate that callbacks should always provide a last resort means to release the resources they set; if read-only, by keeping track of what they sent; if modifiable, by freeing the contents of rs->sr_* if not NULL, setting it to NULL to prevent further cleanup. Similarly, the existence of REP_ENTRY_MUSTBEFREED is not totally clear: in principle as soon as REP_ENTRY_MODIFYABLE is set, it should imply REP_ENTRY_MUSTBEFREED; the only difference in the semantics of the two is that REP_ENTRY_MODIFYABLE without REP_ENTRY_MUSTBEFREED implies that the callback that set the former will take care of freeing the entry; however, other callbacks may further modify it, so freeing temporary data should probably be left to the final handler. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

(ITS#5439) syncprov race condition seg. fault
by rein＠basefarm.no 25 Mar '08

25 Mar '08

Full_Name: Rein Tollevik Version: CVS head OS: CentOS 4.4 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (81.93.160.250) We have bin hit by what looks like a race condition bug in syncprov. We got some core dumps all showing stack frames like the one at the end. As such nasty bugs tends to do it have behaved OK after I restarted slapd with more debug output :-( (trace + stats + stats2 + sync). The configuration is a master server with multiple bdb backend databases all being subordinate to the same glue database where syncprov is used. One of the backends is a syncrepl consumer from another server, the server is master for the other backends. There are multiple consumers for the syncprov suffix, which I assume is what causes the race condition to happen. Note the a=0xBAD argument to attr_find(), which I expect is the result of some other thread freeing the attribute list it was called with while it was processing it. The rs->sr_entry->e_attrs argument passed to attr_find() as the original "a" argument by findpres_cb() looks like a perfectly valid structure, as are all the attributes found by following the a_next pointer. The list is terminated by an attribute with a NULL a_next value, none of the a_next values are 0xBAD. I'm currently trying to gather more information related to this bug, any pointers as to what I should look for is appreciated. I'm posting this bug report now in the hope that the stack frame should enlighten someone with better knowledge of the code than what I have. Rein Tollevik Basefarm AS #0 0x0807d03a in attr_find (a=0xbad, desc=0x81e8680) at attr.c:665 #1 0xb7a656f6 in findpres_cb (op=0xaf068ba4, rs=0xaf068b68) at syncprov.c:546 #2 0x0808416d in slap_response_play (op=0xaf068ba4, rs=0xaf068b68) at result.c:307 #3 0x0808555b in slap_send_search_entry (op=0xaf068ba4, rs=0xaf068b68) at result.c:770 #4 0x080f2cdc in bdb_search (op=0xaf068ba4, rs=0xaf068b68) at search.c:870 #5 0x080db72b in overlay_op_walk (op=0xaf068ba4, rs=0xaf068b68, which=op_search, oi=0x8274218, on=0x8274318) at backover.c:653 #6 0x080dbcaf in over_op_func (op=0xaf068ba4, rs=0xaf068b68, which=op_search) at backover.c:705 #7 0x080dbdef in over_op_search (op=0xaf068ba4, rs=0xaf068b68) at backover.c:727 #8 0x080d9570 in glue_sub_search (op=0xaf068ba4, rs=0xaf068b68, b0=0xaf068ba4, on=0xaf068ba4) at backglue.c:340 #9 0x080da131 in glue_op_search (op=0xbad, rs=0xaf068b68) at backglue.c:459 #10 0x080db6d5 in overlay_op_walk (op=0xaf068ba4, rs=0xaf068b68, which=op_search, oi=0x8271860, on=0x8271a60) at backover.c:643 #11 0x080dbcaf in over_op_func (op=0xaf068ba4, rs=0xaf068b68, which=op_search) at backover.c:705 #12 0x080dbdef in over_op_search (op=0xaf068ba4, rs=0xaf068b68) at backover.c:727 #13 0xb7a65ff4 in syncprov_findcsn (op=0x85c7e60, mode=FIND_PRESENT) at syncprov.c:700 #14 0xb7a670a0 in syncprov_op_search (op=0x85c7e60, rs=0xaf06a1c0) at syncprov.c:2277 #15 0x080db6d5 in overlay_op_walk (op=0x85c7e60, rs=0xaf06a1c0, which=op_search, oi=0x8271860, on=0x8271b60) at backover.c:643 #16 0x080dbcaf in over_op_func (op=0x85c7e60, rs=0xaf06a1c0, which=op_search) at backover.c:705 #17 0x080dbdef in over_op_search (op=0x85c7e60, rs=0xaf06a1c0) at backover.c:727 #18 0x08076554 in fe_op_search (op=0x85c7e60, rs=0xaf06a1c0) at search.c:368 #19 0x080770e4 in do_search (op=0x85c7e60, rs=0xaf06a1c0) at search.c:217 #20 0x08073e28 in connection_operation (ctx=0xaf06a2b8, arg_v=0x85c7e60) at connection.c:1084 #21 0x08074f14 in connection_read_thread (ctx=0xaf06a2b8, argv=0x59) at connection.c:1211 #22 0xb7fb5546 in ldap_int_thread_pool_wrapper (xpool=0x81ee240) at tpool.c:663 #23 0xb7c80371 in start_thread () from /lib/tls/libpthread.so.0 #24 0xb7c17ffe in clone () from /lib/tls/libc.so.6

1 0

Re: (ITS#5401) slapd crashes in mirrormode
by ali.pouya＠free.fr 25 Mar '08

25 Mar '08

Howard Chu wrote : >A patch for this is now in CVS HEAD. Please test. >I was unable to reproduce the reported crash, so your feedback is important >here. Hi Howard; We tested the HEAD extracted march 18 around 16h GMT. It's OK. Thanks a lot. The ITS can be closed. Best regards Ali

1 0

Re: ACLs broken by ITS#5419
by rein＠basefarm.no 25 Mar '08

25 Mar '08

On Mon, 24 Mar 2008, Howard Chu wrote: > rein(a)basefarm.no wrote: >> >> The change to servers/slapd/backend.c for ITS#5416 seem to have broken the >> ability for group and set statements in access control lines to refer to >> entries >> outside the backend currently being operated on. > > That ability was never intended in the first place. Historically, backends in > slapd have been treated as isolated DSAs with no connection to each other. > They've required special mechanisms (like back-relay or slapo-glue) to be > joined. Yes, I know, the change that allowed this was imo the one that made sets and groups really useful. Our database configuration still has traces of the workarounds the lack of this feature once forced us to make.. But, the latest change also removes this ability for databases subordinate to the same common superior (i.e using the slapo-glue). If I understand you correct it is a bug that glue'ed databases cannot refer to each other, although I still consider it a bug (or at least a huge drawback) if this would no longer be generally possible. Rein

1 0

Re: ACLs broken by ITS#5419
by hyc＠symas.com 25 Mar '08

25 Mar '08

rein(a)basefarm.no wrote: > Full_Name: Rein Tollevik > Version: CVS head > OS: linux, solaris > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (84.215.2.34) > > > The change to servers/slapd/backend.c for ITS#5416 seem to have broken the > ability for group and set statements in access control lines to refer to entries > outside the backend currently being operated on. That ability was never intended in the first place. Historically, backends in slapd have been treated as isolated DSAs with no connection to each other. They've required special mechanisms (like back-relay or slapo-glue) to be joined. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

ACLs broken by ITS#5419
by rein＠basefarm.no 24 Mar '08

24 Mar '08

Full_Name: Rein Tollevik Version: CVS head OS: linux, solaris URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (84.215.2.34) The change to servers/slapd/backend.c for ITS#5416 seem to have broken the ability for group and set statements in access control lines to refer to entries outside the backend currently being operated on. This is caused by op->op_bd being set back to the backend being operated on when backend_group() was called, not the backend returned by select_backend() for the group being referred to as was always done before this change was made. >From what I can see of the cases where backend_group() is called (i.e in ACLs, limits and the dynlist and dyngroup overlays) I think the correct fix is to back out this changes as far as that function is conserned. But I don't know enough of other consequences, so I haven't tried it. Rein Tollevik Basefarm AS

1 0

Re: (ITS#5434) syncprov in a glue'ed environment must search through the glue overlay
by rein＠basefarm.no 23 Mar '08

23 Mar '08

On Wed, 19 Mar 2008, Howard Chu wrote: > rein(a)basefarm.no wrote: >> >> When the syncprov overlay is stacked on top of other overlays (notably the >> glue >> overlay) it must search through the next overlay, not the original backend >> DB. >> Otherwise it will not find any entries from the subordinate databases and >> consumers will remove those entries when they enters the refresh phase of >> the >> syncrepl protocol. >> >> The attached patch fixes this problem. > > Thanks, fixed in HEAD It looks as if my fix to this bug was utterly wrong :-( It probably only works when syncprov is stacked on top of the glue overlay. not with other overlays. First, searching through the next overlay looses the overlay semantics (i.e to return SLAP_CB_CONTINUE etc.) Even worse, a seg. fault will occur if the bi_op_search() method is not implement by the next overlay. I don't know whether that is actually allowed or not, but it is not done in at least the auditlog overlay. The correct fix should (I hope...) be to use the original overinst structure as the bd_info, as the patch at the end (which is relative to the current cvs head) implements. Since o_sync_mode appear to always have been cleared before be_search() is called it should be ignored by syncprov_op_search(). Sorry for the inconvenience! Rein Index: OpenLDAP/servers/slapd/overlays/syncprov.c diff -u OpenLDAP/servers/slapd/overlays/syncprov.c:1.1.1.12 OpenLDAP/servers/slapd/overlays/syncprov.c:1.9 --- OpenLDAP/servers/slapd/overlays/syncprov.c:1.1.1.12 Sat Mar 22 16:48:20 2008 +++ OpenLDAP/servers/slapd/overlays/syncprov.c Sun Mar 23 14:06:03 2008 @@ -696,10 +696,7 @@ break; } - if ( on->on_next ) - fop.o_bd->bd_info = (BackendInfo *)on->on_next; - else - fop.o_bd->bd_info = on->on_info->oi_orig; + fop.o_bd->bd_info = (BackendInfo *)on->on_info; fop.o_bd->be_search( &fop, &frs ); fop.o_bd->bd_info = (BackendInfo *)on; @@ -1531,10 +1528,7 @@ fop.ors_filter = ⁡ cb.sc_response = playlog_cb; - if ( on->on_next ) - fop.o_bd->bd_info = (BackendInfo *)on->on_next; - else - fop.o_bd->bd_info = on->on_info->oi_orig; + fop.o_bd->bd_info = (BackendInfo *)on->on_info; for ( i=ndel; i<num; i++ ) { if ( uuids[i].bv_len == 0 ) continue;

1 0

Re: (ITS#5340) REP_ENTRY_MODIFIABLE bug in dynlist
by h.b.furuseth＠usit.uio.no 22 Mar '08

22 Mar '08

h.b.furuseth(a)usit.uio.no writes: > overlays/dynlist.c lines 371-376 sets REP_ENTRY_MUSTBEFREED in rs->sr_flags > without duplicating the entry, if REP_ENTRY_MODIFIABLE was set. > Thus the entry gets freed twice. Breaks test044-dynlist with back-ldif. Actually obeying rs->sr_flags seems to be a more general problem - but I don't know what are bugs and what are my lacking understanding of the flags. I.e. when can an overlay change REP_ENTRY_MUSTBEFREED, REP_ENTRY_MUSTRELEASE? Usually backends/overlays that do not set these flags, don't seem to expect them to change either. -- Hallvard

1 0

Re: (ITS#5437) Fix to servers/slapd/overlays/syncprov.c
by hyc＠symas.com 21 Mar '08

21 Mar '08

c.hoover(a)its.utexas.edu wrote: > Full_Name: Clyde Hoover > Version: 2.4.8 > OS: Solaris 10 SPARC > URL: > Submission from: (NULL) (128.83.93.7) > > > The bug is around line 2015 of servers/slapd/overlays/syncprov.c > > } else if ( rs->sr_type == REP_RESULT&& rs->sr_err == LDAP_SUCCESS ) { > struct berval cookie; > > // The cookie is never initialized so if SS_CHANGED is not set > // this can cause an internal assertion failure downstream from > // syncprov_done_ctrl() (depending upon what is on the stack) > // Nulling out the cookie fixes this > memset((void *)&cookie, 0, sizeof(cookie)); // Add this > if ( ss->ss_flags& SS_CHANGED ) { > slap_compose_sync_cookie( op,&cookie, ss->ss_ctxcsn, > srs->sr_state.rid, > srs->sr_state.srs->sr_state.sid ); Please provide a stack trace from such a failure. The syncprov_done_ctrl() function should never be invoked at that point without SS_CHANGED being set, since in a normal Refresh the search will hit the shortcut and never reach that code. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#5437) Fix to servers/slapd/overlays/syncprov.c
by c.hoover＠its.utexas.edu 21 Mar '08

21 Mar '08

Full_Name: Clyde Hoover Version: 2.4.8 OS: Solaris 10 SPARC URL: Submission from: (NULL) (128.83.93.7) The bug is around line 2015 of servers/slapd/overlays/syncprov.c } else if ( rs->sr_type == REP_RESULT && rs->sr_err == LDAP_SUCCESS ) { struct berval cookie; // The cookie is never initialized so if SS_CHANGED is not set // this can cause an internal assertion failure downstream from // syncprov_done_ctrl() (depending upon what is on the stack) // Nulling out the cookie fixes this memset((void *)&cookie, 0, sizeof(cookie)); // Add this if ( ss->ss_flags & SS_CHANGED ) { slap_compose_sync_cookie( op, &cookie, ss->ss_ctxcsn, srs->sr_state.rid, srs->sr_state.srs->sr_state.sid );

1 0

Re: (ITS#5432) syncrepl seg. faults if received a sync cookie with an empty csn= value
by hyc＠symas.com 21 Mar '08

21 Mar '08

Rein Tollevik wrote: > On Wed, 19 Mar 2008, Howard Chu wrote: > >>> syncprov_playlog() in syncprov.c generates such invalid cookies when >>> replaying a >>> sessionlog where no entries has been deleted. The patch also fixes this >>> bug. >> This should never happen, since the cookie is not generated if there were no >> deletes recorded. > > Well, it do happen anyhow. Using the 2.4.8 release this can easily be > reproduced with a configuration where the syncrepl provider (with > sessionlogs enabled) is on a glue backend, the consumer replicates the > entire glue subtree. Assuming both servers are in sync and stopped, > start the producer, modify an entry in the subordinate db and then start > the consumer. The producer will send a cookie with an empty csn= value > as in this sync debug output: > slapd starting > slap_queue_csn: queing 0x418001a0 20080320125116.487494Z#000000#000#000000 > slap_graduate_commit_csn: removing 0x8d6ae0 20080320125116.487494Z#000000#000#000000 > srs csn 20080320124421.642686Z#000000#000#000000 > log csn 20080320125116.487494Z#000000#000#000000 > syncprov_playlog: cookie=rid=007,csn= > syncprov_search_response: cookie=rid=007,csn=20080320125116.487494Z#000000#000#000000 > > I have also tested it using the cvs head source, and the good news is > that this was accidentally fixed (at least partly) by the ITS#5434 > change. I.e it was caused by the be_search() in syncprov_playlog() not > being propagated to the subordinate database and thereby not finding the > entry that was modified. > > The bad news is that there is still a race condition that would have the > same effect. If someone deletes the entry that was modified just before > be_search() is done it will return the same empty cookie. I was able to > reproduce this in a debugger by setting a break point before be_search(), > deleting the entry with ldapdelete and in the debugger run next over the > be_search() call before continuing. > > So, I still think it is best to keep delcsn[0].bv_val NULL until an > actual cookie has been written to it, as slap_compose_sync_cookie() will > create a cookie with an empty csn= if passed a csn array where bv_len is > zero but bv_val is non-NULL. Although an alternative and maybe more > correct fix could be to change slap_compose_sync_cookie() instead? No. Thanks for your explanation of what you observed. The actual bug is in the mods processing, when it decides that a modified entry should be treated as a delete instead. It increments ndel then but doesn't set delcsn in that case. That's what actually needs to be fixed. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: Segfault in slapd 2.4.8 - syncrepl and glue (ITS#5430)
by ghenry＠suretecsystems.com 20 Mar '08

20 Mar '08

<quote who="hyc(a)symas.com"> > Duncan Gibb wrote: >> Howard Chu wrote: >> HYC> Most notably, in template 1, you explicitly configured >> HYC> syncprov on top of glue instead of taking the default >> HYC> behavior of syncprov under glue. As documented in >> HYC> slapd.conf(5), you should only do that when you want a >> HYC> single syncprov overlay to be master of the entire tree. >> HYC> In this case, you clearly want syncprov to only master >> HYC> one portion of the tree, with the subordinate being >> HYC> taken care of by its own syncprov overlay. >> >> That's a legacy of several previous iterations of the real project. >> Ideally each box would have at most two backends - one holding >> everything for which it's a consumer, and on the relevant machines a >> second backend holding the data for which it's master, then glue, then a >> single syncprov making the whole tree available. My colleague who >> implemented the first iteration did it that way and it should have >> worked, but the syncprov-glue interactions in 2.3.x were such that that >> resulted in sections of the tree not replicating and/or disappearing in >> various ill-defined situations. > > The whole-tree approach may have some snags. > 1) it's intended to work with a single syncprov over the entire glued > tree > 2) it requires a unique serverID for each master > 3) it probably won't work to start from a completely empty DB - the > root > entry needs to exist on each server. It's probably worth while for us and others to start a new thread on -software or -technical? Gavin.

1 0

Re: Segfault in slapd 2.4.8 - syncrepl and glue (ITS#5430)
by hyc＠symas.com 20 Mar '08

20 Mar '08

Duncan Gibb wrote: > Howard Chu wrote: > HYC> Most notably, in template 1, you explicitly configured > HYC> syncprov on top of glue instead of taking the default > HYC> behavior of syncprov under glue. As documented in > HYC> slapd.conf(5), you should only do that when you want a > HYC> single syncprov overlay to be master of the entire tree. > HYC> In this case, you clearly want syncprov to only master > HYC> one portion of the tree, with the subordinate being > HYC> taken care of by its own syncprov overlay. > > That's a legacy of several previous iterations of the real project. > Ideally each box would have at most two backends - one holding > everything for which it's a consumer, and on the relevant machines a > second backend holding the data for which it's master, then glue, then a > single syncprov making the whole tree available. My colleague who > implemented the first iteration did it that way and it should have > worked, but the syncprov-glue interactions in 2.3.x were such that that > resulted in sections of the tree not replicating and/or disappearing in > various ill-defined situations. The whole-tree approach may have some snags. 1) it's intended to work with a single syncprov over the entire glued tree 2) it requires a unique serverID for each master 3) it probably won't work to start from a completely empty DB - the root entry needs to exist on each server. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: Segfault in slapd 2.4.8 - syncrepl and glue (ITS#5430)
by duncan.gibb＠siriusit.co.uk 20 Mar '08

20 Mar '08

Howard Chu wrote: HYC> I modified your test case to make it easier to run. HYC> The new test is in ftp.openldap.org:incoming/its5430.tgz.1 Looks good; thank you very much for this. HYC> There were also some errors in your config file templates, HYC> which are corrected in the above archive. HYC> Most notably, in template 1, you explicitly configured HYC> syncprov on top of glue instead of taking the default HYC> behavior of syncprov under glue. As documented in HYC> slapd.conf(5), you should only do that when you want a HYC> single syncprov overlay to be master of the entire tree. HYC> In this case, you clearly want syncprov to only master HYC> one portion of the tree, with the subordinate being HYC> taken care of by its own syncprov overlay. That's a legacy of several previous iterations of the real project. Ideally each box would have at most two backends - one holding everything for which it's a consumer, and on the relevant machines a second backend holding the data for which it's master, then glue, then a single syncprov making the whole tree available. My colleague who implemented the first iteration did it that way and it should have worked, but the syncprov-glue interactions in 2.3.x were such that that resulted in sections of the tree not replicating and/or disappearing in various ill-defined situations. Replicating each subordinate separately and gluing them together per-server was a hack; the default overlay ordering gave us visible entries of objectclass glue (and consequent missing subtrees) on some consumers, so we reversed them. It's imperfect but less imperfect than it was before ;-) We will re-architect in the light of the recent changes to syncprov and glue. It has already been pointed out that big-picture discussion is noise on this list... HYC> Not an error, but you used unique rids across all of the HYC> config files. rids only need to be unique within a single server. I know. That's expediency in the config generator - it just increments a global counter each time it processes a consumer<-->provider relation. I'll make it more elegant next time round ;-) Cheers Duncan

1 0

Re: Segfault in slapd 2.4.8 - syncrepl and glue (ITS#5430)
by ghenry＠suretecsystems.com 20 Mar '08

20 Mar '08

<quote who="hyc(a)symas.com"> > Duncan Gibb wrote: >> hyc(a)symas.com wrote: >> HYC> syncrepl.c is now updated in HEAD to allow this replication to >> occur. >> >> HYC> Using your test, I don't see any crash. >> >> Thank you. We'll re-test with current HEAD and post the outcome. > > I modified your test case to make it easier to run. The new test is in > ftp.openldap.org:incoming/its5430.tgz.1 Above tests pass here with latest HEAD (repository revision: 1.390 /repo/OpenLDAP/pkg/ldap/servers/slapd/syncrepl.c,v > > You simply extract it inside the tests/ subdirectory of the build tree. > After > you have built the OpenLDAP source and run "make test" you can cd into the > its5430 directory and run the test.sh from there. > > There were also some errors in your config file templates, which are > corrected > in the above archive. > > Most notably, in template 1, you explicitly configured syncprov on top of > glue > instead of taking the default behavior of syncprov under glue. As > documented > in slapd.conf(5), you should only do that when you want a single syncprov > overlay to be master of the entire tree. In this case, you clearly want > syncprov to only master one portion of the tree, with the subordinate > being > taken care of by its own syncprov overlay. With your incorrect > configuration, > server3 would receive redundant updates because the subordinate branch > would > be propagated twice. server3 also doesn't need syncprov module loaded, as it's a pure consumer. Thanks, Gavin.

1 0

Re: (ITS#5432) syncrepl seg. faults if received a sync cookie with an empty csn= value
by rein＠basefarm.no 20 Mar '08

20 Mar '08

On Wed, 19 Mar 2008, Howard Chu wrote: >> syncprov_playlog() in syncprov.c generates such invalid cookies when >> replaying a >> sessionlog where no entries has been deleted. The patch also fixes this >> bug. > > This should never happen, since the cookie is not generated if there were no > deletes recorded. Well, it do happen anyhow. Using the 2.4.8 release this can easily be reproduced with a configuration where the syncrepl provider (with sessionlogs enabled) is on a glue backend, the consumer replicates the entire glue subtree. Assuming both servers are in sync and stopped, start the producer, modify an entry in the subordinate db and then start the consumer. The producer will send a cookie with an empty csn= value as in this sync debug output: slapd starting slap_queue_csn: queing 0x418001a0 20080320125116.487494Z#000000#000#000000 slap_graduate_commit_csn: removing 0x8d6ae0 20080320125116.487494Z#000000#000#000000 srs csn 20080320124421.642686Z#000000#000#000000 log csn 20080320125116.487494Z#000000#000#000000 syncprov_playlog: cookie=rid=007,csn= syncprov_search_response: cookie=rid=007,csn=20080320125116.487494Z#000000#000#000000 I have also tested it using the cvs head source, and the good news is that this was accidentally fixed (at least partly) by the ITS#5434 change. I.e it was caused by the be_search() in syncprov_playlog() not being propagated to the subordinate database and thereby not finding the entry that was modified. The bad news is that there is still a race condition that would have the same effect. If someone deletes the entry that was modified just before be_search() is done it will return the same empty cookie. I was able to reproduce this in a debugger by setting a break point before be_search(), deleting the entry with ldapdelete and in the debugger run next over the be_search() call before continuing. So, I still think it is best to keep delcsn[0].bv_val NULL until an actual cookie has been written to it, as slap_compose_sync_cookie() will create a cookie with an empty csn= if passed a csn array where bv_len is zero but bv_val is non-NULL. Although an alternative and maybe more correct fix could be to change slap_compose_sync_cookie() instead? Rein

1 0

Re: Segfault in slapd 2.4.8 - syncrepl and glue (ITS#5430)
by hyc＠symas.com 20 Mar '08

20 Mar '08

Duncan Gibb wrote: > hyc(a)symas.com wrote: > HYC> syncrepl.c is now updated in HEAD to allow this replication to occur. > > HYC> Using your test, I don't see any crash. > > Thank you. We'll re-test with current HEAD and post the outcome. I modified your test case to make it easier to run. The new test is in ftp.openldap.org:incoming/its5430.tgz.1 You simply extract it inside the tests/ subdirectory of the build tree. After you have built the OpenLDAP source and run "make test" you can cd into the its5430 directory and run the test.sh from there. There were also some errors in your config file templates, which are corrected in the above archive. Most notably, in template 1, you explicitly configured syncprov on top of glue instead of taking the default behavior of syncprov under glue. As documented in slapd.conf(5), you should only do that when you want a single syncprov overlay to be master of the entire tree. In this case, you clearly want syncprov to only master one portion of the tree, with the subordinate being taken care of by its own syncprov overlay. With your incorrect configuration, server3 would receive redundant updates because the subordinate branch would be propagated twice. You also inserted the overlay directives in the middle of your database clauses, which is poor practice (and caused crashes in older OpenLDAP releases). Again, as documented, overlays are meant to be configured at the end of their respective databases. Not an error, but you used unique rids across all of the config files. rids only need to be unique within a single server. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: Segfault in slapd 2.4.8 - syncrepl and glue (ITS#5430)
by Duncan.Gibb＠SiriusIT.co.uk 20 Mar '08

20 Mar '08

hyc(a)symas.com wrote: DG> We've scripted a test which works for me against 2.4.8 and did work DG> against CVS HEAD until Howard committed Apologies for the vagueness of last night's email. For a script whose aim is to demo a crash, "works" means "reproduces the crash with no more intervention than invoking it and pressing enter when prompted". I was trying to explain that in cases where replication doesn't occur, the test can't be done, but on re-reading in daylight perhaps I failed on that score ;-) There were two commits five minutes apart. I could reproduce the crash with syncrepl.c 1.387; replication 2-->1 did not occur with 1.389; I was just about to test with 1.388 when I read your second post. HYC> syncrepl.c is now updated in HEAD to allow this replication to occur. HYC> Using your test, I don't see any crash. Thank you. We'll re-test with current HEAD and post the outcome. Cheers Duncan

1 0

Re: Segfault in slapd 2.4.8 - syncrepl and glue (ITS#5430)
by hyc＠symas.com 20 Mar '08

20 Mar '08

duncan.gibb(a)siriusit.co.uk wrote: > Duncan.Gibb(a)siriusit.co.uk wrote: > > DG> Can anyone else reproduce this, or do we need to work on more examples? > > After more experimentation, I believe this crash only happens if > replication between servers 1 and 2 (master of the main tree and > subordinate master respectively) has completed before server 3 (consumer > of 1) is started. > > We've scripted a test which works for me against 2.4.8 and did work > against CVS HEAD until Howard committed > openldap-src/servers/slapd/syncrepl.c 2008-03-19 23:26:40 +0000. That > changes the syncrepl<-->glue interaction such that the data from server > 2 is never replicated to server 1 (a pre-requisite for this crash). syncrepl.c is now updated in HEAD to allow this replication to occur. Using your test, I don't see any crash. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: Segfault in slapd 2.4.8 - syncrepl and glue (ITS#5430)
by hyc＠symas.com 20 Mar '08

20 Mar '08

duncan.gibb(a)siriusit.co.uk wrote: > Duncan.Gibb(a)siriusit.co.uk wrote: > > DG> Can anyone else reproduce this, or do we need to work on more examples? > > After more experimentation, I believe this crash only happens if > replication between servers 1 and 2 (master of the main tree and > subordinate master respectively) has completed before server 3 (consumer > of 1) is started. > > We've scripted a test which works for me against 2.4.8 and did work > against CVS HEAD until Howard committed What does "work" mean? Please use more precise terminology. Did it reproduce the crash, or did it all replicate correctly? (The latter being the only thing that I would define as "working".) > openldap-src/servers/slapd/syncrepl.c 2008-03-19 23:26:40 +0000. Use revision numbers, that's what they're for. There is nothing in CVS with the above timestamp. > That > changes the syncrepl<-->glue interaction such that the data from server > 2 is never replicated to server 1 (a pre-requisite for this crash). > Maybe our config wrongly depends on the old behaviour. I'll look at > that tomorrow. > > Test rig (3.7K) is at > > http://pastebin.siriusit.co.uk/openldap-its5430-scripted-test-2008-03-19.ta… > > tar xzf openldap-its5430-scripted-test-2008-03-19.tar.gz > cd openldap-its5430-scripted-test-2008-03-19 > ./build.sh > ./test.sh > > Optionally copy CVS openldap-src from before the above commit into the > directory before invoking build.sh. > > Probably only works on Linux. Reproducible with bdb and hdb backends > using BDB 4.4 on Debian Lenny/i386 (bare metal). Not reproducible with > the ldif backend because replication from 2 to 1 never happens. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: Segfault in slapd 2.4.8 - syncrepl and glue (ITS#5430)
by duncan.gibb＠siriusit.co.uk 20 Mar '08

20 Mar '08

Duncan.Gibb(a)siriusit.co.uk wrote: DG> Can anyone else reproduce this, or do we need to work on more examples? After more experimentation, I believe this crash only happens if replication between servers 1 and 2 (master of the main tree and subordinate master respectively) has completed before server 3 (consumer of 1) is started. We've scripted a test which works for me against 2.4.8 and did work against CVS HEAD until Howard committed openldap-src/servers/slapd/syncrepl.c 2008-03-19 23:26:40 +0000. That changes the syncrepl<-->glue interaction such that the data from server 2 is never replicated to server 1 (a pre-requisite for this crash). Maybe our config wrongly depends on the old behaviour. I'll look at that tomorrow. Test rig (3.7K) is at http://pastebin.siriusit.co.uk/openldap-its5430-scripted-test-2008-03-19.ta… tar xzf openldap-its5430-scripted-test-2008-03-19.tar.gz cd openldap-its5430-scripted-test-2008-03-19 ./build.sh ./test.sh Optionally copy CVS openldap-src from before the above commit into the directory before invoking build.sh. Probably only works on Linux. Reproducible with bdb and hdb backends using BDB 4.4 on Debian Lenny/i386 (bare metal). Not reproducible with the ldif backend because replication from 2 to 1 never happens. Cheers Duncan

1 0

Re: (ITS#5410) Patch for contrib smbk5pwd.c to use gcrypt
by pogma＠thewrittenword.com 20 Mar '08

20 Mar '08

On Wed, Mar 19, 2008 at 05:46:28PM -0700, Howard Chu wrote: >> >> This patch allows the smbk5pwd slapd module to build with gcrypt instead of >> openssl. > > In your version of nthash, is the result in binary (and therefore a call to > hexify is missing) or is it already in hex? Looks like the call to hexify is missing, it does not copy anything into the output buffer. Peter

1 0

Re: (ITS#5410) Patch for contrib smbk5pwd.c to use gcrypt
by hyc＠symas.com 20 Mar '08

20 Mar '08

pogma(a)thewrittenword.com wrote: > Full_Name: Peter O'Gorman > Version: 2.4.8 > OS: > URL: ftp://ftp.openldap.org/incoming/the-written-word-smbk5pwd-080310.patch > Submission from: (NULL) (24.76.165.223) > > > This patch allows the smbk5pwd slapd module to build with gcrypt instead of > openssl. In your version of nthash, is the result in binary (and therefore a call to hexify is missing) or is it already in hex? -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5434) syncprov in a glue'ed environment must search through the glue overlay
by hyc＠symas.com 20 Mar '08

20 Mar '08

rein(a)basefarm.no wrote: > Full_Name: Rein Tollevik > Version: 2.4.8 > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (84.215.2.34) > > > When the syncprov overlay is stacked on top of other overlays (notably the glue > overlay) it must search through the next overlay, not the original backend DB. > Otherwise it will not find any entries from the subordinate databases and > consumers will remove those entries when they enters the refresh phase of the > syncrepl protocol. > > The attached patch fixes this problem. Thanks, fixed in HEAD -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs