- openldap-bugs - openldap.org

Re: (ITS#5340) REP_ENTRY_MODIFIABLE bug in dynlist
by h.b.furuseth＠usit.uio.no 01 Apr '08

01 Apr '08

Pierangelo Masarati writes: > Hallvard B Furuseth wrote: >> But I don't see how the be_release() code can work now. It sounds like >> be->be_release() functions must check (how?) that the entry was created >> by 'be', and otherwise pass it on to the next overlay/backend or >> otherwise to entry_free(). Might involve mucking with op->o_bd and >> sr_entry->e_private, I suppose. Except maybe I'm missing some existing >> magic since slapd doesn't regularly crash... > > Yes, but that's trivial: e_private must be NULL for temporary entries, > and copying the entry loses it (no one is supposed to muck with it > expect the entry's creator). OK... > And the appropriate o_bd of a > (non-modifiable) entry can be easily computed from the entry's DN. Hm? Can only the "outermost" backend/overlay create non-modifiable entries? I think any overlay can replace a non-modifiable rs->sr_entry value, as long as it restores it on the way out (unless mustbe<freed/released>). >>> Similarly, the existence of REP_ENTRY_MUSTBEFREED is not totally clear: >>> in principle as soon as REP_ENTRY_MODIFYABLE is set, it should imply >>> REP_ENTRY_MUSTBEFREED; the only difference in the semantics of the two >> >> That's not my impression. (...) Some functions create entries on the stack an send them MODIFIABLE - but obviously without MUSTBEFREED. They use entry_clean() instead. Others set MODIFIABLE but not MUSTBEFREED on non-stack entries, and call entry_free after sending it. Or forget to free it - back-perl/search.c if LDAP_SIZELIMIT_EXCEEDED. (Not patching yet - don't have time to test it at the moment.) > (...) A copy might be created by an overlay after receiving a > read-only entry, but the same overlay might not actually perform the > copy if it receives the entry from another overlay that already copied > it, or from a proxy backend. However, after the entry is copied the > overlay will have no means to determined who actually created the copy. I don't think it matters who created a MUSTBEFREED copy: > This might be an issue depending on the order cleanup handlers are > called (didn't check what order they're called). My point is that > temporary entries need to be freed at some point; who frees them should > not be relevant... Right, someone must call entry_free() and clear rs->sr_entry (so it's not freed again), but entry_free() is not passed a backend parameter. OTOH a MUSTBERELEASED entry must be released by the right backend/overlay, and it would be strange for such an entry to be MODIFIBALE. Though I guess it could be - if the backend has no cache, and be_release just exists to clean up some data in e_private. >> Others apparent problems (also not tested, I've just browsed the code): >> Overlays that obey and reset MUSTBEFREED or MUSTRELEASE, do not >> necessarily clear or set MODIFIABLE when setting a new entry. >> (...) > > It seems to me that we should provide "smart" handlers to deal with > preparing sr_entry for modification, and to take care of cleaning things > up as appropriate. Those helpers should then be consistently used in > the code. Yup. And add some aggressive asserts - at least #ifdef LDAP_DEVEL. Don't know when I'll have time for it though. -- Hallvard

1 0

Re: (ITS#5448) CONTRIB: Java slapd
by ando＠sys-net.it 01 Apr '08

01 Apr '08

hyc(a)OpenLDAP.org wrote: > Given the increasing popularity of Java, I decided it was time that the > OpenLDAP Project had its own Java implementation of an LDAP server. It > gave me a good opportunity to explore all of the benefits of rapid > development, introspection, and other advantages of the Java system. > Indeed, drawing on my experience with the existing OpenLDAP code base, > it took only a matter of hours to create my first working server, which > I present here. Of course it's still experimental in nature. > > Invocation is similar to the regular C-based slapd, and it accepts all > of the same arguments and reads all of the same config settings. In > addition, the first commandline argument must be the name of the directory > where slapd resides, so that the java app can locate the config files, > schema, etc. > > E.g. for a typical OpenLDAP installation in /usr/local you would run > java jslapd /usr/local/libexec -h ldap://:389 > > A small shell script has been provided to simplify this invocation. Howard, many thanks for the contribution. I haven't stressed it yet intensively, but at a first glance it seems to be as performing as the latest 2.4. Will this be released in 2.4, or do we need to wait until 2.5 (what about making it 3.0?). p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5440) back-meta opens too many connections in case of error
by ali.pouya＠free.fr 01 Apr '08

01 Apr '08

Hi Pierangelo; When trying to reproduce the problem with test035 material I found a problem in my own JAVA code ! There is no such problem in back-meta :-) Sorry for misreporting and thanks for your help. The issue is to be closed. Best regards Ali Pierangelo Masarati a écrit : > ali.pouya(a)free.fr wrote: > >> I precise that The behaviour is not the same if the client binds with >> rootdn. > > I've committed a fix that removes dangling connections as the rootdn; > I have been unable to reproduce the issue with regular users. Please > provide more useful info in order to track the issue (slapd.conf, LDIF > and requests). > > p. > > > > Ing. Pierangelo Masarati > OpenLDAP Core Team > > SysNet s.r.l. > via Dossi, 8 - 27100 Pavia - ITALIA > http://www.sys-net.it > --------------------------------------- > Office: +39 02 23998309 > Mobile: +39 333 4963172 > Email: pierangelo.masarati(a)sys-net.it > --------------------------------------- > > > >

1 0

(ITS#5449) X-PIGLATIN password hashing functions
by mbackes＠symas.com 31 Mar '08

31 Mar '08

Full_Name: Matthew Backes Version: 2.x OS: any URL: http://www.symas.net/~lucca/piglatin-patch.txt Submission from: (NULL) (76.88.99.93) In order to improve the variety of security options available to LDAP administrators, I am submitting an additional password hashing method for liblutil. This patch implements the {X-PIGLATIN} hash, e.g.: $ slappasswd -s feep -h '{X-PIGLATIN}' {X-PIGLATIN}eepfay The standard with -yay variation for leading values is used, as described in http://en.wikipedia.org/wiki/Pig_latin y is considered a vowel unless it is the leading char. The patch will need additional review for EBCDIC support. This patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Matthew Backes <mbackes(a)symas.com>. I have not assigned rights and/or interest in this work to any party. diff -u -r1.107 passwd.c --- passwd.c 7 Jan 2008 23:20:06 -0000 1.107 +++ passwd.c 1 Apr 2008 03:13:13 -0000 @@ -119,6 +119,9 @@ #endif #endif +static LUTIL_PASSWD_CHK_FUNC chk_piglatin; +static LUTIL_PASSWD_HASH_FUNC hash_piglatin; + /* password hash routines */ #ifdef SLAPD_CLEARTEXT @@ -154,6 +157,7 @@ { BER_BVC("{CLEARTEXT}"), NULL, hash_clear }, #endif + { BER_BVC("{X-PIGLATIN}"), chk_piglatin, hash_piglatin }, { BER_BVNULL, NULL, NULL } }; @@ -1127,3 +1131,78 @@ } #endif +static int +chk_piglatin( + const struct berval *sc, + const struct berval * passwd, + const struct berval * cred, + const char **text ) +{ + struct berval credhash; + int rc; + + rc = hash_piglatin( sc, cred, &credhash, NULL ); + if( rc != LUTIL_PASSWD_OK ) return rc; + + rc = passwd->bv_len == credhash.bv_len + && memcmp( passwd->bv_val, + credhash.bv_val, + credhash.bv_len ) == 0 ? + LUTIL_PASSWD_ERR : LUTIL_PASSWD_OK; + + ber_memfree( credhash.bv_val ); + return rc; +} + +/* Implement {X-PIGLATIN} + * See http://en.wikipedia.org/wiki/Pig_latin + * -yay variation for leading vowels used + * y is considered a vowel unless it is the leading char + */ + +static int +hash_piglatin( + const struct berval *scheme, + const struct berval *passwd, + struct berval *hash, + const char **text ) +{ + struct berval digest; + int c, rc, first_vowel = 0; + + hash->bv_len = scheme->bv_len + passwd->bv_len + 2; + hash->bv_val = ber_memalloc( scheme->bv_len + passwd->bv_len + 4 ); + if( !hash->bv_val ) return LUTIL_PASSWD_ERR; + + for( c = 0; c < passwd->bv_len; ++c ) { + switch( passwd->bv_val[c] & -33 ) { + case 'Y': + if( c==0 ) break; + case 'A': + case 'E': + case 'I': + case 'O': + case 'U': + first_vowel = c; + goto got_vowel; + } + } + +got_vowel: + if( first_vowel ) { + char * pos = hash->bv_val; + pos += sprintf( pos, "%s%s", + scheme->bv_val, + passwd->bv_val+first_vowel ); + snprintf( pos, first_vowel+1, "%s", + passwd->bv_val ); + pos += first_vowel; + sprintf( pos, "ay" ); + } else { + sprintf( hash->bv_val, "%s%syay", + scheme->bv_val, passwd->bv_val ); + hash->bv_len += 1; + } + + return LUTIL_PASSWD_OK; +} Also available from the provided URL.

1 0

(ITS#5448) CONTRIB: Java slapd
by hyc＠OpenLDAP.org 31 Mar '08

31 Mar '08

Full_Name: Howard Chu Version: 0.0.1 OS: URL: ftp://ftp.openldap.org/incoming/jslapd.tgz Submission from: (NULL) (76.91.220.157) Submitted by: hyc Given the increasing popularity of Java, I decided it was time that the OpenLDAP Project had its own Java implementation of an LDAP server. It gave me a good opportunity to explore all of the benefits of rapid development, introspection, and other advantages of the Java system. Indeed, drawing on my experience with the existing OpenLDAP code base, it took only a matter of hours to create my first working server, which I present here. Of course it's still experimental in nature. Invocation is similar to the regular C-based slapd, and it accepts all of the same arguments and reads all of the same config settings. In addition, the first commandline argument must be the name of the directory where slapd resides, so that the java app can locate the config files, schema, etc. E.g. for a typical OpenLDAP installation in /usr/local you would run java jslapd /usr/local/libexec -h ldap://:389 A small shell script has been provided to simplify this invocation.

1 0

Re: (ITS#5444) slapd seg. fault using mulitmirrored mode
by ando＠sys-net.it 31 Mar '08

31 Mar '08

hyc(a)symas.com wrote: > ando(a)sys-net.it wrote: >> ando(a)sys-net.it wrote: >>> mohel(a)web.de wrote: >>> >>>> The above log was generated using the test050 script just adding a line which >>>> enters information also to a second server. Perhaps this test would also make >>>> sense for further releases. >>> Can you please post the modification to the script? >> Never mind, I could reproduce it. It seems to be a duplicate of >> ITS#5437. What actually happens is that if modifications are applied to >> all three servers, those applied to consumers do not persist, nor get >> replicated; on the contrary, the consumers crash later on as described >> in ITS#5437. > > #5437? What you're talking about doesn't seem to match any text in ITS#5437. Well, sorry for not providing more details, I was going to investigate it a little bit more but I had to leave. The crash to me occurs in syncprov_done_ctrl(), as indicated in ITS#5437. What happens is that syncprov_done_ctrl() with an invalid cookie (unset; clearing it in syncprov_search_response(), as suggested in ITS#5437 leads to the behavior I described (writes to consumers are simply ignored, don't persist in the consumers and are not replicated). To reproduce, add something to a consumer close to the end of test050, before killing the servers, and wait a little bit to let things proceed to the crash. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5444) slapd seg. fault using mulitmirrored mode
by hyc＠symas.com 31 Mar '08

31 Mar '08

ando(a)sys-net.it wrote: > ando(a)sys-net.it wrote: >> mohel(a)web.de wrote: >> >>> The above log was generated using the test050 script just adding a line which >>> enters information also to a second server. Perhaps this test would also make >>> sense for further releases. >> Can you please post the modification to the script? > > Never mind, I could reproduce it. It seems to be a duplicate of > ITS#5437. What actually happens is that if modifications are applied to > all three servers, those applied to consumers do not persist, nor get > replicated; on the contrary, the consumers crash later on as described > in ITS#5437. #5437? What you're talking about doesn't seem to match any text in ITS#5437. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5442) slapd_rq not locked before use bugfix
by hyc＠symas.com 31 Mar '08

31 Mar '08

Rein Tollevik wrote: > On Sun, 30 Mar 2008, hyc(a)symas.com wrote: > >> rein(a)tollevik.no wrote: >>> On Sat, 29 Mar 2008, ando(a)sys-net.it wrote: >>>> rein(a)basefarm.no wrote: > >>>>> I was seeing random failures of the test050-syncrepl-multimaster test. One of >>>>> the failures was that it went into a tight loop traversing a circular runqueue >>>>> it had managed to create in slapd_rq.task_list. It seems as this was caused by >>>>> missing mutex locks around accesses to slapd_rq, which the patch uploaded to >>>>> ftp://ftp.openldap.org/incoming/slapd_rq_lock.patch fixes. >>>>> >>>>> Before I applied this patch the test failed after being run a few times, with it >>>>> it has now passed 100 times and is still counting. >>>> locks in back-bdb/config.c should be pointless, as modifications to the >>>> configuration should only occur while all threads are paused. The rest >>>> makes sort of sense, but I'd leave it to Howard. >> Ignoring the ITS#5403 changes, I don't see anything here that isn't >> config-related, therefore it's all running single-threaded. > > Now that the configuration can be changed dynamically (as this test does) > I find it a bit odd that the config stuff should always be running > single-threaded. But there is obviously much I don't know about the > internals of slapd. This was discussed at the 2003 OpenLDAP Developers' Day. http://www.openldap.org/conf/odd-wien-2003/proceedings.html Page 7 of my slides touched on it. It's also mentioned again here http://www.openldap.org/lists/openldap-devel/200505/msg00062.html The main point is that the original config stuff assumed that it was only executing during slapd startup, and single-threaded. Putting locks around all of the potential configurable value accesses would have been too much work, so the decision was made to force slapd to be single-threaded whenever writing to the config. I'm guessing that what's happening here is just a broken assumption - suspending the thread pool doesn't in fact freeze everything in slapd. In particular, the slapd listener thread may still wake up for a select() timeout to handle the slapd runqueue. Even though no tasks submitted as a result of this will execute (because they'll just get queued into the suspended thread pool) the runqueue itself is still being manipulated. Given that explanation, your patches make sense. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5439) back-bdb race condition
by quanah＠zimbra.com 31 Mar '08

31 Mar '08

--On Monday, March 31, 2008 6:27 PM +0000 rein(a)basefarm.no wrote: > We did see segmentation faults both Friday and Monday (not very much > going on during the weekend..), but so far those appear to be related to > ITS#5445. A slapd and db_checkpoint (run out of cron) deadlock which I > hope is due to the BerkeleyDB bug mentioned in ITS#5391 has also > occurred, so I'll upgrade from 4.6.18 to 4.6.21.1 to see if that problem > goes away. Why do you run db_checkpoint out of cron? Slapd will already take care of checkpointing the database for you using the "checkpoint" directive... --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5439) back-bdb race condition
by rein＠basefarm.no 31 Mar '08

31 Mar '08

On Tue, 25 Mar 2008, Howard Chu wrote: > Rein Tollevik wrote: >> On Tue, 25 Mar 2008, hyc(a)symas.com wrote: >> It now looks to me as if the entire rs->sr_entry was released an reused, >> and that the bug probably is in back-bdb. It just always happened when >> syncprov was running as this is a master server with mostly writes >> except from the reads syncprov does. Which also means that the title on >> the bug report is probably misleading :-( >> >> The reason for my shift of focus is some highly suspicious variables >> found in the bdb_search() frame. > > Look at the value of idflag in bdb_search. (see the comment around 672) > > Could be a locking bug with ID_NOCACHE in cache.c... Yes, it looks very much as this was the problem. We have been running with the patch at the end since thursday afternoon, and so far it seem to have worked :-) Although it is a bit too early to conclude, especially as we have seen other problems as well. We did see segmentation faults both friday and monday (not very much going on during the weekend..), but so far those appear to be related to ITS#5445. A slapd and db_checkpoint (run out of cron) deadlock which I hope is due to the BerkeleyDB bug mentioned in ITS#5391 has also occurred, so I'll upgrade from 4.6.18 to 4.6.21.1 to see if that problem goes away. Now, back to this bug. There is a potential bug in bdb_cache_find_id() as it inspects and modifies the EntryInfo without it being locked. Although I don't think this was the cause here, as bdb_search() always passes a NULL eip which mean that bdb_cache_find_ndn() should have acquired a lock. The patch always locks the entryinfo though, as I believe that to be necessary for the rest of the patch to be correct. The real problem was in bdb_cache_return_entry_rw() which released ei->bei_e without making sure it was the only thread using it. I.e it should have had a write lock on the entry_db (as bdb_cache_lru_purge() makes sure to have), but instead it freed it after releasing the read lock. Instead of write locking the entry before releasing it my alternative solution is to only let the thread that loads the entry set the CACHE_ENTRY_NOT_CACHED flag, and only if no other threads found the entryinfo while it loaded the entry. All other threads clears the flag when they find they don't need to load the entry. I also use trylock to lock the entryinfo before setting or inspecting the cache flag, for two reasons. First, honoring this flag should not be important enough to wait for a lock (after all, it should be cleared anyhow if anyone else is interested in the same entry). Second (and most important), to avoid any deadlock situations, which my first attempt to fix this bug managed to create when I moved the release of the entryinfo lock to after the entry_db lock had been acquired in bdb_cache_find_id(). Rein Index: OpenLDAP/servers/slapd/back-bdb/cache.c diff -u OpenLDAP/servers/slapd/back-bdb/cache.c:1.1.1.15 OpenLDAP/servers/slapd/back-bdb/cache.c:1.2 --- OpenLDAP/servers/slapd/back-bdb/cache.c:1.1.1.15 Sat Mar 22 16:47:56 2008 +++ OpenLDAP/servers/slapd/back-bdb/cache.c Mon Mar 31 17:52:17 2008 @@ -252,16 +252,27 @@ int free = 0; ei = e->e_private; - bdb_cache_entry_db_unlock( bdb, lock ); - if ( ei ) { - bdb_cache_entryinfo_lock( ei ); + if ( ei && + ( ei->bei_state & CACHE_ENTRY_NOT_CACHED ) && + ( bdb_cache_entryinfo_trylock( ei ) == 0 ) ) { if ( ei->bei_state & CACHE_ENTRY_NOT_CACHED ) { + /* Releasing the entry can only be done when + * we know that nobody else is using it, i.e we + * should have an entry_db writelock. But the + * flag is only set by the thread that loads the + * entry, and only if no other threads has found + * it while it was working. All other threads + * clear the flag, which mean that we should be + * the only thread using the entry if the flag + * is set here. + */ ei->bei_e = NULL; ei->bei_state ^= CACHE_ENTRY_NOT_CACHED; free = 1; } bdb_cache_entryinfo_unlock( ei ); } + bdb_cache_entry_db_unlock( bdb, lock ); if ( free ) { e->e_private = NULL; bdb_entry_return( e ); @@ -854,6 +865,11 @@ /* Ok, we found the info, do we have the entry? */ if ( rc == 0 ) { + if ( !( flag & ID_LOCKED )) { + bdb_cache_entryinfo_lock ( *eip ); + flag |= ID_LOCKED; + } + if ( (*eip)->bei_state & CACHE_ENTRY_DELETED ) { rc = DB_NOTFOUND; } else { @@ -873,13 +889,13 @@ (*eip)->bei_state |= CACHE_ENTRY_LOADING; } - /* If the entry was loaded before but uncached, and we need - * it again, clear the uncached state - */ - if ( (*eip)->bei_state & CACHE_ENTRY_NOT_CACHED ) { - (*eip)->bei_state ^= CACHE_ENTRY_NOT_CACHED; - if ( flag & ID_NOCACHE ) - flag ^= ID_NOCACHE; + if ( !load ) { + /* Clear the uncached state if we are not + * loading it, i.e it is already cached or + * another thread is currently loading it. + */ + (*eip)->bei_state &= ~CACHE_ENTRY_NOT_CACHED; + flag &= ~ID_NOCACHE; } if ( flag & ID_LOCKED ) { @@ -906,10 +922,14 @@ #endif ep = NULL; bdb_cache_lru_link( bdb, *eip ); - if ( flag & ID_NOCACHE ) { - bdb_cache_entryinfo_lock( *eip ); - (*eip)->bei_state |= CACHE_ENTRY_NOT_CACHED; - bdb_cache_entryinfo_unlock( *eip ); + if ( ( flag & ID_NOCACHE ) && + ( bdb_cache_entryinfo_trylock ( *eip ) == 0 ) ) { + /* Set the cached state only if no other thread + * found the info while we was loading the entry. + */ + if ( (*eip)->bei_finders == 1 ) + (*eip)->bei_state |= CACHE_ENTRY_NOT_CACHED; + bdb_cache_entryinfo_unlock ( *eip ); } } if ( rc == 0 ) {

1 0

Re: (ITS#5444) slapd seg. fault using mulitmirrored mode
by mohelonline＠googlemail.com 31 Mar '08

31 Mar '08

------=_Part_28055_26658090.1206987058866 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Thanks for the fast reply. I just added a few lines after the 20 seconds wait for the master replcaition: ---------------------------------- echo "Using ldapadd to populate consumer ..." $LDAPADD -D "$MANAGERDN" -H $URI2 -w $PASSWD -f $LDIFORDERED2 \ >> $TESTOUT 2>&1 SLEEP=20 echo "Waiting $SLEEP seconds for syncrepl to receive changes..." sleep $SLEEP ---------------------------------- These lines just add one additional ldapuser. After an additional 20seconds pause I check if the changes (form server with URI2) are replicated. But always the slapd prcoess dies after the ldapadd command. I'll give your above suggestion a try and give feedback afterwards. On Mon, Mar 31, 2008 at 6:04 PM, Pierangelo Masarati <ando(a)sys-net.it> wrote: > ando(a)sys-net.it wrote: > > mohel(a)web.de wrote: > > > >> The above log was generated using the test050 script just adding a line > which > >> enters information also to a second server. Perhaps this test would > also make > >> sense for further releases. > > > > Can you please post the modification to the script? > > Never mind, I could reproduce it. It seems to be a duplicate of > ITS#5437. What actually happens is that if modifications are applied to > all three servers, those applied to consumers do not persist, nor get > replicated; on the contrary, the consumers crash later on as described > in ITS#5437. > > p. > > > > Ing. Pierangelo Masarati > OpenLDAP Core Team > > SysNet s.r.l. > via Dossi, 8 - 27100 Pavia - ITALIA > http://www.sys-net.it > --------------------------------------- > Office: +39 02 23998309 > Mobile: +39 333 4963172 > Email: pierangelo.masarati(a)sys-net.it > --------------------------------------- > > > ------=_Part_28055_26658090.1206987058866 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Thanks for the fast reply.<br><br>I just added a few lines after the 20 seconds wait for the master replcaition:<br>------------------------------<div id="1eob" class="ArwC7c ckChnd">----<br>echo "Using ldapadd to populate consumer ..."<br> $LDAPADD -D "$MANAGERDN" -H $URI2 -w $PASSWD -f $LDIFORDERED2 \<br>         >> $TESTOUT 2>&1<br><br>SLEEP=20<br>echo "Waiting $SLEEP seconds for syncrepl to receive changes..."<br>sleep $SLEEP<br> ----------------------------------<br><br>These lines just add one additional ldapuser. After an additional 20seconds pause I check if the changes (form server with URI2) are replicated.<br><br>But always the slapd prcoess dies after the ldapadd command.<br> <br>I'll give your above suggestion a try and give feedback afterwards.</div><br><br><div class="gmail_quote">On Mon, Mar 31, 2008 at 6:04 PM, Pierangelo Masarati <<a href="mailto:ando@sys-net.it">ando(a)sys-net.it</a>> wrote:<br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="Ih2E3d"><a href="mailto:ando@sys-net.it">ando(a)sys-net.it</a> wrote:<br> > <a href="mailto:mohel@web.de">mohel(a)web.de</a> wrote:<br> ><br> >> The above log was generated using the test050 script just adding a line which<br> >> enters information also to a second server. Perhaps this test would also make<br> >> sense for further releases.<br> ><br> > Can you please post the modification to the script?<br> <br> </div>Never mind, I could reproduce it.  It seems to be a duplicate of<br> ITS#5437.  What actually happens is that if modifications are applied to<br> all three servers, those applied to consumers do not persist, nor get<br> replicated; on the contrary, the consumers crash later on as described<br> in ITS#5437.<br> <div><div></div><div class="Wj3C7c"><br> p.<br> <br> <br> <br> Ing. Pierangelo Masarati<br> OpenLDAP Core Team<br> <br> SysNet s.r.l.<br> via Dossi, 8 - 27100 Pavia - ITALIA<br> <a href="http://www.sys-net.it" target="_blank">http://www.sys-net.it</a><br> ---------------------------------------<br> Office:  +39 02 23998309<br> Mobile:  +39 333 4963172<br> Email:   <a href="mailto:pierangelo.masarati@sys-net.it">pierangelo.masarati(a)sys-net.it</a><br> ---------------------------------------<br> <br> <br> </div></div></blockquote></div><br> ------=_Part_28055_26658090.1206987058866--

1 0

Re: (ITS#5442) slapd_rq not locked before use bugfix
by rein＠basefarm.no 31 Mar '08

31 Mar '08

On Sun, 30 Mar 2008, hyc(a)symas.com wrote: > rein(a)tollevik.no wrote: >> On Sat, 29 Mar 2008, ando(a)sys-net.it wrote: >>> rein(a)basefarm.no wrote: >>>> I was seeing random failures of the test050-syncrepl-multimaster test. One of >>>> the failures was that it went into a tight loop traversing a circular runqueue >>>> it had managed to create in slapd_rq.task_list. It seems as this was caused by >>>> missing mutex locks around accesses to slapd_rq, which the patch uploaded to >>>> ftp://ftp.openldap.org/incoming/slapd_rq_lock.patch fixes. >>>> >>>> Before I applied this patch the test failed after being run a few times, with it >>>> it has now passed 100 times and is still counting. >>> locks in back-bdb/config.c should be pointless, as modifications to the >>> configuration should only occur while all threads are paused. The rest >>> makes sort of sense, but I'd leave it to Howard. > > Ignoring the ITS#5403 changes, I don't see anything here that isn't > config-related, therefore it's all running single-threaded. Now that the configuration can be changed dynamically (as this test does) I find it a bit odd that the config stuff should always be running single-threaded. But there is obviously much I don't know about the internals of slapd. > Of the "relevant" changes in syncrepl.c, I note that three out of the four > chunks of the patch are in code that is only run when using cn=config to > delete a syncrepl configuration, and test050 never performs that operation. > The remaining chunk only takes affect when adding syncrepl config, and again, > slapd is single-threaded for that. I have been experimenting a bit more with the patch, and it appears to be the locks added to add_syncrepl() that stopped the test from failing. The test uses ordinary ldapadd to add the syncrepl configuration after slapd has been started, which I would expect to indicate that it has entered the multi-threaded state. But again, it can revert to single-threaded when modifying the configuration for all I know. > I've also run test050 thru hundreds of iterations without any issue, without > these patches. If there's a problem in test050, I don't believe it's in this code. I have also run it several hundred times on other systems, it is only on one of the 4 systems I build slapd that it fails. That is, that test failed on one of the other systems with the 2.4.8 release, also with an infinite loop, but that dosn't say vary much in this context. The system where the test fail is the only multi-cpu system I have run it on, which I recon to be the reason why it only fails there. Without this patch the test fails about 20% of the times it is run on that system, the reason varies between slapd being killed due to abort or seg. fault (unknown why), or infinite loop traversing the circular slapd_rq. With the patch it has succeeded several thousand times in a row, which I account for more than just a coincident. So, as long as this patch doesn't create any deadlock I can't really see the problem with it and will stick to it in our version. I find it a bit easier to understand and verify code that requiers locking if the locks are always used, then I don't need to know the thread states that are allowed to call the functions as well to know whether a lock is needed or not.. Rein

1 0

Re: (ITS#5444) slapd seg. fault using mulitmirrored mode
by ando＠sys-net.it 31 Mar '08

31 Mar '08

ando(a)sys-net.it wrote: > mohel(a)web.de wrote: > >> The above log was generated using the test050 script just adding a line which >> enters information also to a second server. Perhaps this test would also make >> sense for further releases. > > Can you please post the modification to the script? Never mind, I could reproduce it. It seems to be a duplicate of ITS#5437. What actually happens is that if modifications are applied to all three servers, those applied to consumers do not persist, nor get replicated; on the contrary, the consumers crash later on as described in ITS#5437. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5445) syncprov double-free bugfix.
by ando＠sys-net.it 31 Mar '08

31 Mar '08

rein(a)basefarm.no wrote: > The patch at the end adds missing parenthesis around a negated flags bit test in > syncprov.c. Without them the test always fails, the entry is never duplicated > and a double-free occur when the a_nvals is free'ed in the next statement if the > same entry is sent to more than one recipient simultaneously. Applied. Please test. Thanks, p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5440) back-meta opens too many connections in case of error
by ando＠sys-net.it 31 Mar '08

31 Mar '08

ali.pouya(a)free.fr wrote: > I precise that The behaviour is not the same if the client binds with rootdn. I've committed a fix that removes dangling connections as the rootdn; I have been unable to reproduce the issue with regular users. Please provide more useful info in order to track the issue (slapd.conf, LDIF and requests). p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5444) slapd seg. fault using mulitmirrored mode
by ando＠sys-net.it 31 Mar '08

31 Mar '08

mohel(a)web.de wrote: > The above log was generated using the test050 script just adding a line which > enters information also to a second server. Perhaps this test would also make > sense for further releases. Can you please post the modification to the script? p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

(ITS#5445) syncprov double-free bugfix.
by rein＠basefarm.no 31 Mar '08

31 Mar '08

Full_Name: Rein Tollevik Version: CVS head OS: linux and solaris URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (81.93.160.250) The patch at the end adds missing parenthesis around a negated flags bit test in syncprov.c. Without them the test always fails, the entry is never duplicated and a double-free occur when the a_nvals is free'ed in the next statement if the same entry is sent to more than one recipient simultaneously. Rein Tollevik Basefarm AS Index: OpenLDAP/servers/slapd/overlays/syncprov.c diff -u OpenLDAP/servers/slapd/overlays/syncprov.c:1.9 OpenLDAP/servers/slapd/overlays/syncprov.c:1.10 --- OpenLDAP/servers/slapd/overlays/syncprov.c:1.9 Sun Mar 23 14:06:03 2008 +++ OpenLDAP/servers/slapd/overlays/syncprov.c Mon Mar 31 15:43:31 2008 @@ -2385,7 +2385,7 @@ } if ( !ap ) { - if ( !rs->sr_flags & REP_ENTRY_MODIFIABLE ) { + if ( !(rs->sr_flags & REP_ENTRY_MODIFIABLE) ) { rs->sr_entry = entry_dup( rs->sr_entry ); rs->sr_flags |= REP_ENTRY_MODIFIABLE|REP_ENTRY_MUSTBEFREED;

1 0

(ITS#5444) slapd seg. fault using mulitmirrored mode
by mohel＠web.de 31 Mar '08

31 Mar '08

Full_Name: Markus Version: 2.4.8 OS: SLES 10 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (212.185.43.218) When using multi master replication und start the tree from scratch, I'm able to ldapadd to one of the masters and all the stuff is replciated. But wenn afterwards ldapadd a new entry to one of the other servers the server crashes with the segmentation error. The following log outout is generated: ---------------------------------------------- conn=11 op=1 SEARCH RESULT tag=101 err=0 nentries=20 text= connection_get(19) connection_get(19): got connid=12 connection_read(19): checking for input on id=12 ber_get_next ber_get_next: tag 0x30 len 162 contents: ber_get_next conn=12 op=1 do_search ber_scanf fmt ({miiiib) ber: >>> dnPrettyNormal: <dc=example,dc=com> => ldap_bv2dn(dc=example,dc=com,0) <= ldap_bv2dn(dc=example,dc=com)=0 => ldap_dn2bv(272) <= ldap_dn2bv(dc=example,dc=com)=0 => ldap_dn2bv(272) <= ldap_dn2bv(dc=example,dc=com)=0 <<< dnPrettyNormal: <dc=example,dc=com>, <dc=example,dc=com> SRCH "dc=example,dc=com" 2 0 0 0 0 ber_scanf fmt (m) ber: filter: (objectClass=*) ber_scanf fmt ({M}}) ber: => get_ctrls ber_scanf fmt ({m) ber: ber_scanf fmt (m) ber: => get_ctrls: oid="1.3.6.1.4.1.4203.1.9.1.1" (noncritical) ber_scanf fmt ({i) ber: ber_scanf fmt (m) ber: ber_scanf fmt (b) ber: ber_scanf fmt (}) ber: <= get_ctrls: n=1 rc=0 err="" attrs: * + conn=12 op=1 SRCH base="dc=example,dc=com" scope=2 deref=0 filter="(objectClass=*)" conn=12 op=1 SRCH attr=* + => bdb_search bdb_dn2entry("dc=example,dc=com") search_candidates: base="dc=example,dc=com" (0x00000001) scope=2 => bdb_dn2idl("dc=example,dc=com") => bdb_equality_candidates (entryCSN) <= bdb_equality_candidates: (entryCSN) not indexed bdb_search_candidates: id=-1 first=1 last=20 bdb_search: 1 does not match filter bdb_search: 2 does not match filter bdb_search: 3 does not match filter bdb_search: 4 does not match filter bdb_search: 5 does not match filter bdb_search: 6 does not match filter bdb_search: 7 does not match filter bdb_search: 8 does not match filter bdb_search: 9 does not match filter bdb_search: 10 does not match filter bdb_search: 11 does not match filter bdb_search: 12 does not match filter bdb_search: 13 does not match filter bdb_search: 14 does not match filter bdb_search: 15 does not match filter bdb_search: 16 does not match filter bdb_search: 17 does not match filter bdb_search: 18 does not match filter bdb_search: 20 does not match filter send_ldap_result: conn=12 op=1 p=3 send_ldap_result: err=0 matched="" text="" => bdb_search bdb_dn2entry("dc=example,dc=com") search_candidates: base="dc=example,dc=com" (0x00000001) scope=2 => bdb_dn2idl("dc=example,dc=com") => bdb_presence_candidates (objectClass) bdb_search_candidates: id=-1 first=1 last=20 send_ldap_result: conn=12 op=1 p=3 send_ldap_result: err=0 matched="" text="" send_ldap_intermediate: err=0 oid=1.3.6.1.4.1.4203.1.9.1.4 len=368 send_ldap_response: msgid=2 tag=121 err=0 ber_flush2: 409 bytes to sd 19 send_ldap_result: conn=12 op=1 p=3 send_ldap_result: err=0 matched="" text="" slap_sl_malloc of 136867984 bytes failed, using ch_malloc ---------------------------------------------- It seems as the malloc requets of almost 130MB is simple to much. The above log was generated using the test050 script just adding a line which enters information also to a second server. Perhaps this test would also make sense for further releases.

1 0

Re: (ITS#5404) back-ldap fails intermittently with 80 Internal (implementation specific) error
by toby＠inf.ed.ac.uk 31 Mar '08

31 Mar '08

> Please test and report; > > servers/slapd/back-ldap/bind.c > new revision: 1.244; previous revision: 1.243 > > The patch applies straightforwardly to re23 only if the first chunk is > omitted. Been running for 3+ weeks now on various test machines with no recurrence of error, so I'd consider this resolved. Many thanks Toby

1 0

Re: (ITS#5442) slapd_rq not locked before use bugfix
by hyc＠symas.com 30 Mar '08

30 Mar '08

rein(a)tollevik.no wrote: > On Sat, 29 Mar 2008, ando(a)sys-net.it wrote: >> rein(a)basefarm.no wrote: >>> I was seeing random failures of the test050-syncrepl-multimaster test. One of >>> the failures was that it went into a tight loop traversing a circular runqueue >>> it had managed to create in slapd_rq.task_list. It seems as this was caused by >>> missing mutex locks around accesses to slapd_rq, which the patch uploaded to >>> ftp://ftp.openldap.org/incoming/slapd_rq_lock.patch fixes. >>> >>> Before I applied this patch the test failed after being run a few times, with it >>> it has now passed 100 times and is still counting. >> locks in back-bdb/config.c should be pointless, as modifications to the >> configuration should only occur while all threads are paused. The rest >> makes sort of sense, but I'd leave it to Howard. Ignoring the ITS#5403 changes, I don't see anything here that isn't config-related, therefore it's all running single-threaded. Of the "relevant" changes in syncrepl.c, I note that three out of the four chunks of the patch are in code that is only run when using cn=config to delete a syncrepl configuration, and test050 never performs that operation. The remaining chunk only takes affect when adding syncrepl config, and again, slapd is single-threaded for that. I've also run test050 thru hundreds of iterations without any issue, without these patches. If there's a problem in test050, I don't believe it's in this code. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5442) slapd_rq not locked before use bugfix
by rein＠tollevik.no 29 Mar '08

29 Mar '08

On Sat, 29 Mar 2008, ando(a)sys-net.it wrote: > rein(a)basefarm.no wrote: > >> I was seeing random failures of the test050-syncrepl-multimaster test. One of >> the failures was that it went into a tight loop traversing a circular runqueue >> it had managed to create in slapd_rq.task_list. It seems as this was caused by >> missing mutex locks around accesses to slapd_rq, which the patch uploaded to >> ftp://ftp.openldap.org/incoming/slapd_rq_lock.patch fixes. >> >> Before I applied this patch the test failed after being run a few times, with it >> it has now passed 100 times and is still counting. > > locks in back-bdb/config.c should be pointless, as modifications to the > configuration should only occur while all threads are paused. The rest > makes sort of sense, but I'd leave it to Howard. That is probably true, it looks as if the places in config.c where locks really are required already had them. My patch adds locks everywhere slapd_rq was used without them, as I don't have enough knowledge of the code to know which functions guarranteed to only be used when threads are not running. I believe that the important patch is to syncrepl.c, but I found I it best to add locks everywhere just to be on the safe side. Looking at my copy of the patch it appears that another syncrepl.c patch which I was sure I had edited out has slipped through anyhow :-(. It is the first two modifications related to ldap_get_option, please disregard them in this bugreport. They have been reported in ITS#5403. Rein

1 0

Re: (ITS#5442) slapd_rq not locked before use bugfix
by ando＠sys-net.it 29 Mar '08

29 Mar '08

rein(a)basefarm.no wrote: > I was seeing random failures of the test050-syncrepl-multimaster test. One of > the failures was that it went into a tight loop traversing a circular runqueue > it had managed to create in slapd_rq.task_list. It seems as this was caused by > missing mutex locks around accesses to slapd_rq, which the patch uploaded to > ftp://ftp.openldap.org/incoming/slapd_rq_lock.patch fixes. > > Before I applied this patch the test failed after being run a few times, with it > it has now passed 100 times and is still counting. locks in back-bdb/config.c should be pointless, as modifications to the configuration should only occur while all threads are paused. The rest makes sort of sense, but I'd leave it to Howard. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5340) REP_ENTRY_MODIFIABLE bug in dynlist
by ando＠sys-net.it 28 Mar '08

28 Mar '08

Hallvard B Furuseth wrote: > ando(a)sys-net.it wrote: > > [about REP_ENTRY_MUSTRELEASE] >> it is not >> clear what happens when a callback chain is interrupted by >> slap_null_cb() or similar, without getting to slap_send_search_entry(). >> This seems to indicate that callbacks should always provide a last >> resort means to release the resources they set; if read-only, by keeping >> track of what they sent; if modifiable, by freeing the contents of >> rs->sr_* if not NULL, setting it to NULL to prevent further cleanup. > > That sounds cumbersome, I hope slapd could take care of that somehow. Yes, it could: for example, by providing a helper like slap_entry2modifiable() that takes s SlapReply, checks the flags and does nothing if the entry is already modifiable, or copies it if not, releasing the original one if REP_ENTRY_MUSTRELEASE was set. > But I don't see how the be_release() code can work now. It sounds like > be->be_release() functions must check (how?) that the entry was created > by 'be', and otherwise pass it on to the next overlay/backend or > otherwise to entry_free(). Might involve mucking with op->o_bd and > sr_entry->e_private, I suppose. Except maybe I'm missing some existing > magic since slapd doesn't regularly crash... Yes, but that's trivial: e_private must be NULL for temporary entries, and copying the entry loses it (no one is supposed to muck with it expect the entry's creator). And the appropriate o_bd of a (non-modifiable) entry can be easily computed from the entry's DN. > be->be_release() does receive entries that were not created by 'be' or > at least not with be->be_fetch(), see openldap-devel thread 'slapd API' > in mar 2008. Might be a bug, but I'm not familiar with that code. >> Similarly, the existence of REP_ENTRY_MUSTBEFREED is not totally clear: >> in principle as soon as REP_ENTRY_MODIFYABLE is set, it should imply >> REP_ENTRY_MUSTBEFREED; the only difference in the semantics of the two >> is that REP_ENTRY_MODIFYABLE without REP_ENTRY_MUSTBEFREED implies that >> the callback that set the former will take care of freeing the entry; >> however, other callbacks may further modify it, so freeing temporary >> data should probably be left to the final handler. > > That's not my impression. MODIFIABLE would be that other modules than > the creator can modify the entry - but the creator might still be the > one who will free it. MUSTBEFREED is that the entry must be > entry_free()ed - the creator will not do it (or not do it unless that > flag is set). They clearly mean different things; my point is that as soon as an entry is modifiable it is not read-only and thus is a temporary, and thus will eventually need to be freed. So the point is who is actually going to free it. A copy might be created by an overlay after receiving a read-only entry, but the same overlay might not actually perform the copy if it receives the entry from another overlay that already copied it, or from a proxy backend. However, after the entry is copied the overlay will have no means to determined who actually created the copy. This might be an issue depending on the order cleanup handlers are called (didn't check what order they're called). My point is that temporary entries need to be freed at some point; who frees them should not be relevant... > So, if I'm getting this right... > > A backend must expect an entry to change or be freed if it sends the > entry with REP_ENTRY_<MUSTRELEASE, MUSTBEFREED or MODIFIABLE>, or if > it passes through a slap_callback.sc_response. Right now, a backend expects the entry to change if sent with MODIFIABLE; to be released if sent with MUSTBERELEASED; to be freed if sent with MUSTBEFREED. Modifications could occur to a MODIFIABLE entry when passing through a sc_callback(). Otherwise, if the entry is not MODIFIABLE, a copy will be created if the callback needs to modify the entry; after copying, the entry will be released if MUSTBERELEASED was set. When a copy is created, the callback that creates the copy must also clear the MUSTBERELEASED flag. Optionally, it may set the MUSTBEFREED flag if it doesn't intend to take care of cleaning the entry up after it's done. > back-ldif does not: it uses e_name/e_nname _after_ sending with with > REP_ENTRY_MODIFIABLE. That should be considered a bug. > Nor overlay retcode, it calls retcode_entry_response(,,,rs->sr_entry) > which makes pointers into sr_entry without checking if those flags are > set. If I'm getting that code correctly. I haven't tested. I havent' checked, but that might be a bug as well. > Others apparent problems (also not tested, I've just browsed the code): > > Overlays that obey and reset MUSTBEFREED or MUSTRELEASE, do not > necessarily clear or set MODIFIABLE when setting a new entry. > translucent does not, even though it is a careful one which > obeys both MUSTBEFREED and MUSTRELEASE. > > > I'm not sure when the code must clear the entry_related flags. > Some code which sets new entries seem to assume they are zero, > but some code which sets sr_entry=NULL does not clear the flags. > > There are sr_flags bits which are not about the entry, in particular > REP_MATCHED_MUSTBEFREED and REP_MATCHED_MUSTBEFREED. Looks like these > flags can get lost when but some code sets sr_flags = <something> > rather sr_flags |= <something> or sr_flags ~= ~<something>. It seems to me that we should provide "smart" handlers to deal with preparing sr_entry for modification, and to take care of cleaning things up as appropriate. Those helpers should then be consistently used in the code. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5443) Multiple identical attibutes break syncrepl process fatally?
by ando＠sys-net.it 28 Mar '08

28 Mar '08

marian.eichholz(a)freenet.ag wrote: > We use openldap as mail service directory with some 8 Mio objects on several > replicas. > For openldap 2.4.x we have to migrate from slurpd to syncrepl. > We got a working syncrepl provider als slurpd consumer (slapadd -q, 36 hrs) > > So I try to get a blank DB up by syncrepl only (yes, it is not at all > performant, but informative) > > The process kind of breaks after a couple of minutes and some 44.000 objects > (8.000.000 expected). Tracing it on the consumer side (-d 16384), I see > something like this after an entry: > > syncrepl_message_to_entry: rid=001 mods check (forwardto: value #4 provided more > than once) > > Indeed, the entry to come has three "forwardto:" Attributes with the same value > (and other forwardto-attributes, too). This makes no greate sense at the > application level, but until now it has been perfectly OK for the directory, and > the LDAP-API did not complain about the attribute modification, neither did the > slurpd. It is a violation of RFC 4512, section 2.2, which OpenLDAP 2.4 conforms to. > This leads to some questions and suggestions: > > - the provider does not log anything with -d 16384, no error, no nothing. Could > it do some useful logging about successful and failing replication sessions? What's -16384? Since OpenLDAP 2.3 you can use strings to identify each log subsystem (16384 == 0x4000 == "sync"). The error occurs when the consumer tries to manipulate the data it receives. The producer has nothing to do with it, since it assumes that data contained in it already passed sanity checks when they were stored. How incorrect data got stored into the producer is a totally different business, and the producer-side replication process should not muck with it. > - the consumer does not log anything that can explain, why the remaining objects > are not read, either. A bit of warning/logging could help the hopeful admin, > probably. A sync error occurred, which prevented sync'in from continuing. This error is logged by the "sync" subsystem. As far as I understand from reading the code, the error (at least, a replication error) should be logged also by the "any" subsystem, which means that as soon as any logging is enabled, you get a message logged. > - why is one problematic object lethal for the whole rest of the objects, since > future modifications keep to be incorporated? Is this lack of robustness more a > bug or a feature? If inconsistent data is received, synchronization is supposed to stop. In fact, continuing may result in an inconsistent state. The fact that the stop is caused by a real error, and the fact that fixing the error allows synchronization to recover doesn't sound like lack of robustness to me. It sounds more about wisdom. > - are identical attributes really forbidden with LDAP? RFC 4512, Section 2.2 > - what could one do, to prevent unskillful "editors" of the master node to kill > the replication processes for the whole replication cluster? Besides adding a > checking/filtering API layer, of course. Slapd has sanity checks for this. Slapadd doesn't, since it is supposed to be operated only with consistent data, as resulting from slapcat. You might have slapadd'ed inconsistent data to the producer. In the end, I don't see how this ITS involves a bug in synchronization software. The fact your producer got corrupted by inconsistent data might have been caused by a bug in the software, however your analysis does not give a clear indication of how it happened. If it happened by slapadd, then it's a known (and desired, and documented) limitation of the software. Unless you can reproduce it, I'd consider this ITS closed. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

(ITS#5443) Multiple identical attibutes break syncrepl process fatally?
by marian.eichholz＠freenet.ag 28 Mar '08

28 Mar '08

Full_Name: Marian Eichholz Version: 2.4.8 OS: Linux URL: Submission from: (NULL) (194.97.7.65) We use openldap as mail service directory with some 8 Mio objects on several replicas. For openldap 2.4.x we have to migrate from slurpd to syncrepl. We got a working syncrepl provider als slurpd consumer (slapadd -q, 36 hrs) So I try to get a blank DB up by syncrepl only (yes, it is not at all performant, but informative) The process kind of breaks after a couple of minutes and some 44.000 objects (8.000.000 expected). Tracing it on the consumer side (-d 16384), I see something like this after an entry: syncrepl_message_to_entry: rid=001 mods check (forwardto: value #4 provided more than once) Indeed, the entry to come has three "forwardto:" Attributes with the same value (and other forwardto-attributes, too). This makes no greate sense at the application level, but until now it has been perfectly OK for the directory, and the LDAP-API did not complain about the attribute modification, neither did the slurpd. This leads to some questions and suggestions: - the provider does not log anything with -d 16384, no error, no nothing. Could it do some useful logging about successful and failing replication sessions? - the consumer does not log anything that can explain, why the remaining objects are not read, either. A bit of warning/logging could help the hopeful admin, probably. - why is one problematic object lethal for the whole rest of the objects, since future modifications keep to be incorporated? Is this lack of robustness more a bug or a feature? - are identical attributes really forbidden with LDAP? - what could one do, to prevent unskillful "editors" of the master node to kill the replication processes for the whole replication cluster? Besides adding a checking/filtering API layer, of course. Thank You in advance. Please let me know, if I can provide You with something useful information about our issue(s).

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs