openldap-bugs

openldap-bugs@openldap.org

1 participants
20965 discussions

Re: (ITS#5457) bdb_add() does not always release entries
by hyc＠symas.com 09 Apr '08

09 Apr '08

h.b.furuseth(a)usit.uio.no wrote: > Full_Name: Hallvard B Furuseth > Version: HEAD, 2.3, 2.4 > OS: > URL: > Submission from: (NULL) (129.240.6.233) > Submitted by: hallvard > > > back-bdb/add.c has several code paths where the parent entry > "p" is not released - it does "goto return_results;" without > first doing > > if ( p&& p !=&slap_entry_root ) > bdb_unlocked_cache_return_entry_r(&bdb->bi_cache, p ); > p = NULL; Releasing entries is somewhat irrelevant, since all locks are released when the transaction commits. Note that this function is a no-op in proto-bdb.h. All of those statements are just relics from the first entry cache design, before we switched to using BDB locks for everything. Probably could stand to do some cleanup. The diffs would be quite drastic though. Again, this may be something that's best deferred until 2.3 is dropped. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#5457) bdb_add() does not always release entries
by h.b.furuseth＠usit.uio.no 09 Apr '08

09 Apr '08

Full_Name: Hallvard B Furuseth Version: HEAD, 2.3, 2.4 OS: URL: Submission from: (NULL) (129.240.6.233) Submitted by: hallvard back-bdb/add.c has several code paths where the parent entry "p" is not released - it does "goto return_results;" without first doing if ( p && p != &slap_entry_root ) bdb_unlocked_cache_return_entry_r( &bdb->bi_cache, p ); p = NULL; The code looks like it expects that to be done below the return_results lablel, since it sets p to NULL before the goto. I don't know how the bdb cache code works though. Also I imagine the 'if ( !bvmatch( &pdn, &p->e_nname ) )' branch should check that p != &slap_entry_root before releasing p. Not sure when the parent needs to be freed. By ITS#3671 entries should be released before doing network I/O - but maybe the parent must be held until the slap_graduate_commit_csn()? I don't know what that does. Is it bdb_cache_add(,,op->ora_e,,,) or the following TXN_COMMIT which makes it necessary to bdb_cache_return_entry_r the entry? HEAD and RE24 only do bdb_cache_return_entry_r( bdb, oe, &lock ); if rs->sr_err == LDAP_SUCCESS at the return_results label, but rs->sr_err can change after the bdb_cache_add(). (A bit hard to read since the entry is referred to as both 'oe', op->ora_e and op->oq_add.rs_e. Not sure what the 'oe' variable is used for anyway.)

1 0

ITS#5378 delta-sync issues
by hyc＠symas.com 09 Apr '08

09 Apr '08

A fix is in HEAD. delta-syncrepl will now fallback to regular refresh in additional cases - LDAP_ALREADY_EXISTS, which may occur for old Adds, LDAP_NO_SUCH_OBJECT, for cases already covered in ITS#5376, LDAP_NO_SUCH_ATTRIBUTE, for old Modifies, and LDAP_TYPE_OR_VALUE_EXISTS also for old Modifies. Pretty sure that covers all of the possible out-of-sync cases. Please test. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5409) paged results invalid sizelimit error
by hyc＠symas.com 08 Apr '08

08 Apr '08

Thanks for the test program. I was able to compile it with JDK 1.5 and reproduce the error in both 2.3.41 and in HEAD. This is now fixed in RE23 and HEAD. Phoebe Shao wrote: > when connection pooling is off: > > if page size = 1000 > test 1 failed at the second iteration > test 2 failed at the third iteration > > if page size = 999 > test 1 always works > test 2 failed at the second iteration > > > I have not tried other tests where expected result is other number. > > > phoebe > > > > ----- "Howard Chu"<hyc(a)symas.com> wrote: > >> pshao(a)zimbra.com wrote: >>> ------=_Part_3887_1760018525.1205731478084 Content-Type: >> text/plain; >>> charset=utf-8 Content-Transfer-Encoding: 7bit >>> >>> >>> I did more testing and found there is more to the bug, page size is >> not the >>> only factor. The number of results in the next search also >> matters. >>> However, in all cases, if JNDI connection pooling is disabled, then >> it >>> always works. >>> >>> Test description: >>> >>> create 10000 account: user-1 to user-10000 search size limit = 5000 >>> >>> test 1: step 1. search with filter (uid=*user*): expect >>> SizeLimitExceededException step 2. search with filter >> (uid=*user-9999*): >>> expect 1 entry >>> >>> test 2: step 1. search with filter (uid=*user*): expect >>> SizeLimitExceededException step 2. search with filter >> (uid=*user-10*): >>> expect 112 entries >> Please be more descriptive, re: intermittent failure - how many >> iterations of >> a test a required (on average) before the problem shows up? >> >> -- >> -- Howard Chu >> Chief Architect, Symas Corp. http://www.symas.com >> Director, Highland Sun http://highlandsun.com/hyc/ >> Chief Architect, OpenLDAP http://www.openldap.org/project/ > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5451) syncprov_checkpoint deadlock bugfix?
by h.b.furuseth＠usit.uio.no 07 Apr '08

07 Apr '08

Can you post your master and slave slapd.conf files, after removing passwords? -- Hallvard

1 0

Re: (ITS#5456) inaccurate interpretation of LDIF in back perl
by Uwe.Werler＠o3si.de 07 Apr '08

07 Apr '08

Hello Howard, in my last message (see below) the question was not wether is an entry starting with a blank line or not or what is a correct ldif. My question was if back_perl is only accepting "ldif-like" entries starting with dn-spec. If I return an entry like this: version: 1 dn: ou=foo,dc=bar ou: foo objectClass: organizationalUnit objectClass: top description: users from foo or with SEP after version-spec version: 1 dn: ou=foo,dc=bar ou: foo objectClass: organizationalUnit objectClass: top description: users from foo then the debug output from slapd is: "str2entry: entry -1 has no dn" That means that back_perl doesn't accept ldif files but only "ldif-like" entries without version-spec. If this is intended than I suggest to explain this in the man page or at least in the example shipped with back_perl. Thank You for Your attention. Regards Uwe Am Sonntag, den 06.04.2008, 12:41 -0700 schrieb Howard Chu: > Please do not email me directly unless you are a paying client of my company. > This thread started on the ITS. It should remain on the ITS. > > I already explained to you what the RFC spec says about valid LDIF. Valid LDIF > does not start with an empty line. There is nothing further to explain. > > Uwe Werler wrote: > > Dear Howard, > > > > after some testing again I determined that back_perl doesn't accept > > ldif_entries which are starting with an empty line. Then slapd tells me > > "str2entry: entry -1 has no dn". > > > > Is this behavior correct? If so it will be good to explain in the man > > page or the example that an entry should start with a dn line and > > shouldn't contain any empty lines nor the version information at the > > first line. > > > > I think that my mistake was to use sample code I've found like this: > > > > sub search > > { > > > > my $this = shift; > > my($base, $scope, $deref, $sizeLim, $timeLim, $filterStr, $attrOnly, @attrs ) = @_; > > if (! defined($this->{'LDAP'})) { > > $ldap = Net::LDAP->new( 'localhost' ) ; > > $this->{'LDAP'} = $ldap; > > } > > $mesg=$this->{'LDAP'}->search(base=>$base, > > scope=>$scope, > > deref=>$deref, > > sizelimit =>$sizeLim, > > timelimit =>$timeLim, > > filter =>$filterStr, > > attrs=> @attrs ); > > > > my @list = (); > > > > > > foreach $entry ($mesg->all_entries) { > > my $data; > > $LDIF = new IO::Scalar \$data; > > $dump=Net::LDAP::LDIF->new($LDIF,"w"); > > $dump->write($entry); > > $LDIF->close; > > push @list,$data; > > } > > return ( 0 , @list); > > > > } > > > > Because the entries returned by Net::LDAP::LDIF are starting with a leading SEP it won't work as is in back_perl. > > > > Thank You very much for Your attention. > > > > Regards Uwe > > > > > >

1 0

Re: (ITS#5451) syncprov_checkpoint deadlock bugfix?
by rein＠basefarm.no 07 Apr '08

07 Apr '08

rein(a)basefarm.no skrev: >> We have seen deadlock/hang situations in our master server today, which looks >> like a deadlock caused by the si_csn_rwlock lock being held while >> syncprov_checkpoint() is running. The patch at the end should (I hope) fix >> this. > > Hm, I looks as if I was a bit to quick here, it just hung got again :-(. > And this time without anyone competing for this lock, only the lock on > the glue suffix entry. I wonder if it can be the upgrade from db 4.6.18 > to 4.6.21.1 I dit yesterday that is the real problem. I'll try to > downgrade and see if that helps. And it didn't :-( The problem seem to be that something readlocks our glue suffix entry before forgetting about the lock. Which quickly causes the entire server to deadlock when the writelock required to update the contextCSN in the suffix entry locks out all the readers. So far the problem seem to be triggered by someone attempting to modify an entry in a subordinate syncrepl consumer backend that results in a referral to the backend master. But I haven't had very much time to look into this problem yet, so I'm still on very thin ice here. I'll return with a new ITS when I have found out more. We are currently running with a workaround that simply grants the writelocks on the glue suffix entry without actually doing it. As the glue entry is the only entry in that database it should be pretty safe, and a potential corruption of the database is not any big problem. > Please put this case on hold. Sorry! It currently looks as this patch addresses the symptom and not the real problem, although I'm not sure what could happen if a checkpoint is triggered while the suffix entry is locked by another thread. I do believe it should be considered if not a bug so an enhancement, as holding locks for as short time as possible is always a good thing. You'll have to choose whether to close this ITS or use the patch. Rein

1 0

Re: (ITS#5454) syncrepl refreshAndPersist stops receiving
by hyc＠symas.com 07 Apr '08

07 Apr '08

Rein Tollevik wrote: > Howard Chu skrev: >> How about just moving the enable to after the runqueue manipulation is >> done? > > That would still leave a glitch in the window open, unless the initial > trylock is changed to a regular lock as your checking message suggests. > > But, doing that could result in all the threads ending up waiting for > the lock if do_syncrepl of a refresh only replication is configured to > run too often. I assume the trylock is there to avoid this? Yes, but I don't think that's a problem any more. If the scheduled task is still running when the next interval hits, the daemon task will just reschedule; it won't fire off a new thread for the task. >> It also occurs to me that we probably don't even need to manipulate the >> slapd runqueue in persist mode, when si->si_conn is already set. I.e., >> in that case we can only have gotten here because of a listener event, >> and not because of a runqueue schedule. > > That is probably true. Right, my patch avoids that now. Having looked at it again I think replacing the trylock with a regular lock is fine. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5454) syncrepl refreshAndPersist stops receiving
by rein＠basefarm.no 07 Apr '08

07 Apr '08

Howard Chu skrev: > rein(a)basefarm.no wrote: >> Hm, I hope I have found the race condition that causes this :-) I'm now >> running with the patch at the end to see if that solves it, only time >> will tell.. And we haven't seen any replication problems since this was implemented friday afternoon :-). But it is still a bit too early to conclude that it has been fixed. >> The race is that between the time selecting on the syncrepl socket is >> enabled by the call to connection_client_enable() and the release of the >> si_mutex a new message may arrive. If so, the next call to do_syncrepl >> may fail in its attempt to trylock the mutex and no-one will re-enable >> selecting on it again. My patch delays enabling of the socket until the >> mutex has been released, which looks safe to me. Or can the access to >> si->si_conn without a lock be a problem? > > How about just moving the enable to after the runqueue manipulation is > done? That would still leave a glitch in the window open, unless the initial trylock is changed to a regular lock as your checking message suggests. But, doing that could result in all the threads ending up waiting for the lock if do_syncrepl of a refresh only replication is configured to run too often. I assume the trylock is there to avoid this? What about starting out with deferring the next scheduled call, as a scheduled call is not wanted until the first has completed anyhow? And a rescedule is already being done before do_syncrepl finishes. Deferring should not be necessary if rtask->next_sched.tv_sec is zero, i.e it has already been deferred. That would remove these calls for all the cases when do_syncrepl is called as a result of a new message on a persistent syncrepl connection. > Just need to be sure that do_syncrepl() isn't entered again before > si->si_conn gets initialized. I'm not quite sure what you mean here. As I see it, the problem is that do_syncrepl() must not be called after si->si_conn is enabled and the lock is still held. > It also occurs to me that we probably don't even need to manipulate the > slapd runqueue in persist mode, when si->si_conn is already set. I.e., > in that case we can only have gotten here because of a listener event, > and not because of a runqueue schedule. That is probably true. Rein

1 0

Re: (ITS#5456) inaccurate interpretation of LDIF in back perl
by h.b.furuseth＠usit.uio.no 07 Apr '08

07 Apr '08

I wrote: > You two seem to be saying the same thing: An entry should end with a > line terminator, but not with several line terminators (which give > blank lines). Duh, sorry, I didn't see you said it _begins_ with a newline. Anyway, Net::LDAP::LDIF is doing the right thing: When you write multiple entries to the same Net::LDAP::LDIF object, it puts an empty line between them. -- Hallvard

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs