- openldap-bugs - openldap.org

ITS#5378 delta-sync issues
by hyc＠symas.com 09 Apr '08

09 Apr '08

A fix is in HEAD. delta-syncrepl will now fallback to regular refresh in additional cases - LDAP_ALREADY_EXISTS, which may occur for old Adds, LDAP_NO_SUCH_OBJECT, for cases already covered in ITS#5376, LDAP_NO_SUCH_ATTRIBUTE, for old Modifies, and LDAP_TYPE_OR_VALUE_EXISTS also for old Modifies. Pretty sure that covers all of the possible out-of-sync cases. Please test. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5409) paged results invalid sizelimit error
by hyc＠symas.com 09 Apr '08

09 Apr '08

Thanks for the test program. I was able to compile it with JDK 1.5 and reproduce the error in both 2.3.41 and in HEAD. This is now fixed in RE23 and HEAD. Phoebe Shao wrote: > when connection pooling is off: > > if page size = 1000 > test 1 failed at the second iteration > test 2 failed at the third iteration > > if page size = 999 > test 1 always works > test 2 failed at the second iteration > > > I have not tried other tests where expected result is other number. > > > phoebe > > > > ----- "Howard Chu"<hyc(a)symas.com> wrote: > >> pshao(a)zimbra.com wrote: >>> ------=_Part_3887_1760018525.1205731478084 Content-Type: >> text/plain; >>> charset=utf-8 Content-Transfer-Encoding: 7bit >>> >>> >>> I did more testing and found there is more to the bug, page size is >> not the >>> only factor. The number of results in the next search also >> matters. >>> However, in all cases, if JNDI connection pooling is disabled, then >> it >>> always works. >>> >>> Test description: >>> >>> create 10000 account: user-1 to user-10000 search size limit = 5000 >>> >>> test 1: step 1. search with filter (uid=*user*): expect >>> SizeLimitExceededException step 2. search with filter >> (uid=*user-9999*): >>> expect 1 entry >>> >>> test 2: step 1. search with filter (uid=*user*): expect >>> SizeLimitExceededException step 2. search with filter >> (uid=*user-10*): >>> expect 112 entries >> Please be more descriptive, re: intermittent failure - how many >> iterations of >> a test a required (on average) before the problem shows up? >> >> -- >> -- Howard Chu >> Chief Architect, Symas Corp. http://www.symas.com >> Director, Highland Sun http://highlandsun.com/hyc/ >> Chief Architect, OpenLDAP http://www.openldap.org/project/ > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5451) syncprov_checkpoint deadlock bugfix?
by h.b.furuseth＠usit.uio.no 07 Apr '08

07 Apr '08

Can you post your master and slave slapd.conf files, after removing passwords? -- Hallvard

1 0

Re: (ITS#5456) inaccurate interpretation of LDIF in back perl
by Uwe.Werler＠o3si.de 07 Apr '08

07 Apr '08

Hello Howard, in my last message (see below) the question was not wether is an entry starting with a blank line or not or what is a correct ldif. My question was if back_perl is only accepting "ldif-like" entries starting with dn-spec. If I return an entry like this: version: 1 dn: ou=foo,dc=bar ou: foo objectClass: organizationalUnit objectClass: top description: users from foo or with SEP after version-spec version: 1 dn: ou=foo,dc=bar ou: foo objectClass: organizationalUnit objectClass: top description: users from foo then the debug output from slapd is: "str2entry: entry -1 has no dn" That means that back_perl doesn't accept ldif files but only "ldif-like" entries without version-spec. If this is intended than I suggest to explain this in the man page or at least in the example shipped with back_perl. Thank You for Your attention. Regards Uwe Am Sonntag, den 06.04.2008, 12:41 -0700 schrieb Howard Chu: > Please do not email me directly unless you are a paying client of my company. > This thread started on the ITS. It should remain on the ITS. > > I already explained to you what the RFC spec says about valid LDIF. Valid LDIF > does not start with an empty line. There is nothing further to explain. > > Uwe Werler wrote: > > Dear Howard, > > > > after some testing again I determined that back_perl doesn't accept > > ldif_entries which are starting with an empty line. Then slapd tells me > > "str2entry: entry -1 has no dn". > > > > Is this behavior correct? If so it will be good to explain in the man > > page or the example that an entry should start with a dn line and > > shouldn't contain any empty lines nor the version information at the > > first line. > > > > I think that my mistake was to use sample code I've found like this: > > > > sub search > > { > > > > my $this = shift; > > my($base, $scope, $deref, $sizeLim, $timeLim, $filterStr, $attrOnly, @attrs ) = @_; > > if (! defined($this->{'LDAP'})) { > > $ldap = Net::LDAP->new( 'localhost' ) ; > > $this->{'LDAP'} = $ldap; > > } > > $mesg=$this->{'LDAP'}->search(base=>$base, > > scope=>$scope, > > deref=>$deref, > > sizelimit =>$sizeLim, > > timelimit =>$timeLim, > > filter =>$filterStr, > > attrs=> @attrs ); > > > > my @list = (); > > > > > > foreach $entry ($mesg->all_entries) { > > my $data; > > $LDIF = new IO::Scalar \$data; > > $dump=Net::LDAP::LDIF->new($LDIF,"w"); > > $dump->write($entry); > > $LDIF->close; > > push @list,$data; > > } > > return ( 0 , @list); > > > > } > > > > Because the entries returned by Net::LDAP::LDIF are starting with a leading SEP it won't work as is in back_perl. > > > > Thank You very much for Your attention. > > > > Regards Uwe > > > > > >

1 0

Re: (ITS#5451) syncprov_checkpoint deadlock bugfix?
by rein＠basefarm.no 07 Apr '08

07 Apr '08

rein(a)basefarm.no skrev: >> We have seen deadlock/hang situations in our master server today, which looks >> like a deadlock caused by the si_csn_rwlock lock being held while >> syncprov_checkpoint() is running. The patch at the end should (I hope) fix >> this. > > Hm, I looks as if I was a bit to quick here, it just hung got again :-(. > And this time without anyone competing for this lock, only the lock on > the glue suffix entry. I wonder if it can be the upgrade from db 4.6.18 > to 4.6.21.1 I dit yesterday that is the real problem. I'll try to > downgrade and see if that helps. And it didn't :-( The problem seem to be that something readlocks our glue suffix entry before forgetting about the lock. Which quickly causes the entire server to deadlock when the writelock required to update the contextCSN in the suffix entry locks out all the readers. So far the problem seem to be triggered by someone attempting to modify an entry in a subordinate syncrepl consumer backend that results in a referral to the backend master. But I haven't had very much time to look into this problem yet, so I'm still on very thin ice here. I'll return with a new ITS when I have found out more. We are currently running with a workaround that simply grants the writelocks on the glue suffix entry without actually doing it. As the glue entry is the only entry in that database it should be pretty safe, and a potential corruption of the database is not any big problem. > Please put this case on hold. Sorry! It currently looks as this patch addresses the symptom and not the real problem, although I'm not sure what could happen if a checkpoint is triggered while the suffix entry is locked by another thread. I do believe it should be considered if not a bug so an enhancement, as holding locks for as short time as possible is always a good thing. You'll have to choose whether to close this ITS or use the patch. Rein

1 0

Re: (ITS#5454) syncrepl refreshAndPersist stops receiving
by hyc＠symas.com 07 Apr '08

07 Apr '08

Rein Tollevik wrote: > Howard Chu skrev: >> How about just moving the enable to after the runqueue manipulation is >> done? > > That would still leave a glitch in the window open, unless the initial > trylock is changed to a regular lock as your checking message suggests. > > But, doing that could result in all the threads ending up waiting for > the lock if do_syncrepl of a refresh only replication is configured to > run too often. I assume the trylock is there to avoid this? Yes, but I don't think that's a problem any more. If the scheduled task is still running when the next interval hits, the daemon task will just reschedule; it won't fire off a new thread for the task. >> It also occurs to me that we probably don't even need to manipulate the >> slapd runqueue in persist mode, when si->si_conn is already set. I.e., >> in that case we can only have gotten here because of a listener event, >> and not because of a runqueue schedule. > > That is probably true. Right, my patch avoids that now. Having looked at it again I think replacing the trylock with a regular lock is fine. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5454) syncrepl refreshAndPersist stops receiving
by rein＠basefarm.no 07 Apr '08

07 Apr '08

Howard Chu skrev: > rein(a)basefarm.no wrote: >> Hm, I hope I have found the race condition that causes this :-) I'm now >> running with the patch at the end to see if that solves it, only time >> will tell.. And we haven't seen any replication problems since this was implemented friday afternoon :-). But it is still a bit too early to conclude that it has been fixed. >> The race is that between the time selecting on the syncrepl socket is >> enabled by the call to connection_client_enable() and the release of the >> si_mutex a new message may arrive. If so, the next call to do_syncrepl >> may fail in its attempt to trylock the mutex and no-one will re-enable >> selecting on it again. My patch delays enabling of the socket until the >> mutex has been released, which looks safe to me. Or can the access to >> si->si_conn without a lock be a problem? > > How about just moving the enable to after the runqueue manipulation is > done? That would still leave a glitch in the window open, unless the initial trylock is changed to a regular lock as your checking message suggests. But, doing that could result in all the threads ending up waiting for the lock if do_syncrepl of a refresh only replication is configured to run too often. I assume the trylock is there to avoid this? What about starting out with deferring the next scheduled call, as a scheduled call is not wanted until the first has completed anyhow? And a rescedule is already being done before do_syncrepl finishes. Deferring should not be necessary if rtask->next_sched.tv_sec is zero, i.e it has already been deferred. That would remove these calls for all the cases when do_syncrepl is called as a result of a new message on a persistent syncrepl connection. > Just need to be sure that do_syncrepl() isn't entered again before > si->si_conn gets initialized. I'm not quite sure what you mean here. As I see it, the problem is that do_syncrepl() must not be called after si->si_conn is enabled and the lock is still held. > It also occurs to me that we probably don't even need to manipulate the > slapd runqueue in persist mode, when si->si_conn is already set. I.e., > in that case we can only have gotten here because of a listener event, > and not because of a runqueue schedule. That is probably true. Rein

1 0

Re: (ITS#5456) inaccurate interpretation of LDIF in back perl
by h.b.furuseth＠usit.uio.no 07 Apr '08

07 Apr '08

I wrote: > You two seem to be saying the same thing: An entry should end with a > line terminator, but not with several line terminators (which give > blank lines). Duh, sorry, I didn't see you said it _begins_ with a newline. Anyway, Net::LDAP::LDIF is doing the right thing: When you write multiple entries to the same Net::LDAP::LDIF object, it puts an empty line between them. -- Hallvard

1 0

Re: (ITS#5404) back-ldap fails intermittently with 80 Internal (implementation specific) error
by ghenry＠OpenLDAP.org 06 Apr '08

06 Apr '08

toby(a)inf.ed.ac.uk wrote: >> Please test and report; >> >> servers/slapd/back-ldap/bind.c >> new revision: 1.244; previous revision: 1.243 >> >> The patch applies straightforwardly to re23 only if the first chunk is >> omitted. > > Been running for 3+ weeks now on various test machines with no > recurrence of error, so I'd consider this resolved. > > Many thanks > Toby > > > Thanks for the followup. Gavin. -- Kind Regards, Gavin Henry. OpenLDAP Engineering Team. E ghenry(a)OpenLDAP.org Community developed LDAP software. http://www.openldap.org/project/

1 0

Re: (ITS#5456) inaccurate interpretation of LDIF in back perl
by hyc＠symas.com 06 Apr '08

06 Apr '08

Please do not email me directly unless you are a paying client of my company. This thread started on the ITS. It should remain on the ITS. I already explained to you what the RFC spec says about valid LDIF. Valid LDIF does not start with an empty line. There is nothing further to explain. Uwe Werler wrote: > Dear Howard, > > after some testing again I determined that back_perl doesn't accept > ldif_entries which are starting with an empty line. Then slapd tells me > "str2entry: entry -1 has no dn". > > Is this behavior correct? If so it will be good to explain in the man > page or the example that an entry should start with a dn line and > shouldn't contain any empty lines nor the version information at the > first line. > > I think that my mistake was to use sample code I've found like this: > > sub search > { > > my $this = shift; > my($base, $scope, $deref, $sizeLim, $timeLim, $filterStr, $attrOnly, @attrs ) = @_; > if (! defined($this->{'LDAP'})) { > $ldap = Net::LDAP->new( 'localhost' ) ; > $this->{'LDAP'} = $ldap; > } > $mesg=$this->{'LDAP'}->search(base=>$base, > scope=>$scope, > deref=>$deref, > sizelimit =>$sizeLim, > timelimit =>$timeLim, > filter =>$filterStr, > attrs=> @attrs ); > > my @list = (); > > > foreach $entry ($mesg->all_entries) { > my $data; > $LDIF = new IO::Scalar \$data; > $dump=Net::LDAP::LDIF->new($LDIF,"w"); > $dump->write($entry); > $LDIF->close; > push @list,$data; > } > return ( 0 , @list); > > } > > Because the entries returned by Net::LDAP::LDIF are starting with a leading SEP it won't work as is in back_perl. > > Thank You very much for Your attention. > > Regards Uwe > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5456) inaccurate interpretation of LDIF in back perl
by h.b.furuseth＠usit.uio.no 05 Apr '08

05 Apr '08

hyc(a)symas.com writes: > uw(a)o3si.de wrote: >> The database backend "back_perl" expects LDIF entries beginning with >> "dn:" and ending with an empty line. (...) > > Wrong. (...) all records must end with LF or CR/LF. You two seem to be saying the same thing: An entry should end with a line terminator, but not with several line terminators (which give blank lines). >> After some testing with back_perl I noticed that the LDIF entries >> produced by the perl module Net::LDAP::LDIF were not acceptet because >> they are beginning with an empty line and ending only with a simple >> CR charakter. If you really mean CR (\r), that's a Net::LDAP::LDIF bug. However Net::LDAP::LDIF on my host correctly ends an entry with LF (\n). And back-perl (at least the search operation) accepts entries ending with both zero, one and two '\n's, as well as with '\r'. There was however an entry parsing bug (ITS#4852) in 2.3.32 which was fixed in 2.3.35. I don't see how it'd be related, but I suggest you upgrade anyway. Then if you still have trouble, try a simple perl script which does something like sub search { 0, "dn: o=foo\no: foo\nobjectClass: organization\n" } instead of going via a module like Net::LDAP::LDIF, so you can be quite sure the problem is what you think it is. -- Hallvard

1 0

Re: (ITS#5456) inaccurate interpretation of LDIF in back perl
by hyc＠symas.com 05 Apr '08

05 Apr '08

uw(a)o3si.de wrote: > Full_Name: Uwe Werler > Version: 2.3.32 > OS: SLES10 SP1 Kernel 2.6.16.54-0.2.3-xen > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (155.56.68.221) > > > The database backend "back_perl" expects LDIF entries beginning with "dn:" and > ending with an empty line. RFC2849 only describes that an entry is everything > except an empty line and a line not beginning (after a CR/LF) with a single > white space character. Wrong. The grammar in RFC2849 quite explicitly states: ldif-file = ldif-content / ldif-changes ldif-content = version-spec 1*(1*SEP ldif-attrval-record) ldif-changes = version-spec 1*(1*SEP ldif-change-record) ldif-attrval-record = dn-spec SEP 1*attrval-spec ldif-change-record = dn-spec SEP *control changerecord That means an LDIF file must start with 0 or 1 version-specs, then a dn-spec. And, all records must end with LF or CR/LF. > After some testing with back_perl I noticed that the LDIF entries produced by > the perl module Net::LDAP::LDIF were not acceptet because they are beginning > with an empty line and ending only with a simple CR charakter. > > This behavior breaks RFC2849. After changing LDIF.pm from that module to produce > the CR after the entry back_perl accepts this entries too. > > IMHO this is a misbehavior so I'm very glad if You'll take a look at this > issue. Apparently it Net::LDAP::LDIF is broken. Since that is not a piece of OpenLDAP Software I don't see what we can do about it. This ITS will be closed. Contact the author of Net::LDAP::LDIF to pursue this further. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#5456) inaccurate interpretation of LDIF in back perl
by uw＠o3si.de 05 Apr '08

05 Apr '08

Full_Name: Uwe Werler Version: 2.3.32 OS: SLES10 SP1 Kernel 2.6.16.54-0.2.3-xen URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (155.56.68.221) The database backend "back_perl" expects LDIF entries beginning with "dn:" and ending with an empty line. RFC2849 only describes that an entry is everything except an empty line and a line not beginning (after a CR/LF) with a single white space character. After some testing with back_perl I noticed that the LDIF entries produced by the perl module Net::LDAP::LDIF were not acceptet because they are beginning with an empty line and ending only with a simple CR charakter. This behavior breaks RFC2849. After changing LDIF.pm from that module to produce the CR after the entry back_perl accepts this entries too. IMHO this is a misbehavior so I'm very glad if You'll take a look at this issue. Thanks in advance. Regards Uwe

1 0

Re: (ITS#5454) syncrepl refreshAndPersist stops receiving
by hyc＠symas.com 05 Apr '08

05 Apr '08

rein(a)basefarm.no wrote: > Hm, I hope I have found the race condition that causes this :-) I'm now > running with the patch at the end to see if that solves it, only time will > tell.. > > The race is that between the time selecting on the syncrepl socket is > enabled by the call to connection_client_enable() and the release of the > si_mutex a new message may arrive. If so, the next call to do_syncrepl > may fail in its attempt to trylock the mutex and no-one will re-enable > selecting on it again. My patch delays enabling of the socket until the > mutex has been released, which looks safe to me. Or can the access to > si->si_conn without a lock be a problem? How about just moving the enable to after the runqueue manipulation is done? Just need to be sure that do_syncrepl() isn't entered again before si->si_conn gets initialized. It also occurs to me that we probably don't even need to manipulate the slapd runqueue in persist mode, when si->si_conn is already set. I.e., in that case we can only have gotten here because of a listener event, and not because of a runqueue schedule. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5452) seg fault in syncprov_op abandon on master
by quanah＠zimbra.com 04 Apr '08

04 Apr '08

--On Friday, April 04, 2008 1:23 PM +0200 Hallvard B Furuseth <h.b.furuseth(a)usit.uio.no> wrote: > Since it crashes on > if ( so->s_op->o_connid == op->o_connid && > so->s_op->o_msgid == op->orn_msgid ) > please try > frame 0 > print *op > print *so->s_op > and maybe > backtrace full > while you are at it. #0 0x0000002a975f21ec in syncprov_op_abandon (op=0x42002b40, rs=0x42002a50) at syncprov.c:1026 1026 if ( so->s_op->o_connid == op->o_connid && (gdb) frame 0 #0 0x0000002a975f21ec in syncprov_op_abandon (op=0x42002b40, rs=0x42002a50) at syncprov.c:1026 1026 if ( so->s_op->o_connid == op->o_connid && (gdb) print *op $1 = {o_hdr = 0x42002ac0, o_tag = 80, o_time = 0, o_tincr = 0, o_bd = 0x42002830, o_req_dn = {bv_len = 0, bv_val = 0x0}, o_req_ndn = {bv_len = 0, bv_val = 0x0}, o_request = {oq_add = { rs_e = 0x4, rs_modlist = 0x0}, oq_bind = {rb_method = 4, rb_cred = {bv_len = 0, bv_val = 0x0}, rb_edn = {bv_len = 0, bv_val = 0x0}, rb_ssf = 0, rb_tmp_mech = {bv_len = 0, bv_val = 0x0}}, oq_compare = {rs_ava = 0x4}, oq_modify = {rs_modlist = 0x4, rs_increment = 0}, oq_modrdn = {rs_newrdn = {bv_len = 4, bv_val = 0x0}, rs_nnewrdn = {bv_len = 0, bv_val = 0x0}, rs_newSup = 0x0, rs_nnewSup = 0x0, rs_deleteoldrdn = 0}, oq_search = {rs_scope = 4, rs_deref = 0, rs_slimit = 0, rs_tlimit = 0, rs_limit = 0x0, rs_attrsonly = 0, rs_attrs = 0x0, rs_filter = 0x0, rs_filterstr = {bv_len = 0, bv_val = 0x0}}, oq_abandon = {rs_msgid = 4}, oq_cancel = {rs_msgid = 4}, oq_extended = {rs_reqoid = {bv_len = 4, bv_val = 0x0}, rs_flags = 0, rs_reqdata = 0x0}, oq_pwdexop = {rs_extended = {rs_reqoid = {bv_len = 4, bv_val = 0x0}, rs_flags = 0, rs_reqdata = 0x0}, rs_old = {bv_len = 0, bv_val = 0x0}, rs_new = {bv_len = 0, bv_val = 0x0}, rs_mods = 0x0, rs_modtail = 0x0}}, o_abandon = 0, o_cancel = 0, o_groups = 0x0, o_do_not_cache = 0 '\0', o_is_auth_check = 0 '\0', o_nocaching = 0 '\0', o_delete_glue_parent = 0 '\0', o_no_schema_check = 0 '\0', o_ctrlflag = '\0' <repeats 31 times>, o_controls = 0x0, o_authz = { sai_method = 0, sai_mech = {bv_len = 0, bv_val = 0x0}, sai_dn = {bv_len = 0, bv_val = 0x0}, sai_ndn = {bv_len = 0, bv_val = 0x0}, sai_ssf = 0, sai_transport_ssf = 0, sai_tls_ssf = 0, sai_sasl_ssf = 0}, o_ber = 0x0, o_res_ber = 0x0, o_callback = 0x42002810, o_ctrls = 0x0, o_csn = {bv_len = 0, bv_val = 0x0}, o_private = 0x0, o_next = {stqe_next = 0x0}} (gdb) print *so->s_op $2 = {o_hdr = 0x11d8fd60, o_tag = 0, o_time = 0, o_tincr = 0, o_bd = 0x0, o_req_dn = {bv_len = 0, bv_val = 0x0}, o_req_ndn = {bv_len = 0, bv_val = 0x0}, o_request = {oq_add = {rs_e = 0x0, rs_modlist = 0x0}, oq_bind = {rb_method = 0, rb_cred = {bv_len = 0, bv_val = 0x0}, rb_edn = {bv_len = 0, bv_val = 0x0}, rb_ssf = 0, rb_tmp_mech = {bv_len = 0, bv_val = 0x0}}, oq_compare = {rs_ava = 0x0}, oq_modify = {rs_modlist = 0x0, rs_increment = 0}, oq_modrdn = {rs_newrdn = {bv_len = 0, bv_val = 0x0}, rs_nnewrdn = {bv_len = 0, bv_val = 0x0}, rs_newSup = 0x0, rs_nnewSup = 0x0, rs_deleteoldrdn = 0}, oq_search = {rs_scope = 0, rs_deref = 0, rs_slimit = 0, rs_tlimit = 0, rs_limit = 0x0, rs_attrsonly = 0, rs_attrs = 0x0, rs_filter = 0x0, rs_filterstr = {bv_len = 0, bv_val = 0x0}}, oq_abandon = {rs_msgid = 0}, oq_cancel = {rs_msgid = 0}, oq_extended = {rs_reqoid = {bv_len = 0, bv_val = 0x0}, rs_flags = 0, rs_reqdata = 0x0}, oq_pwdexop = {rs_extended = {rs_reqoid = {bv_len = 0, bv_val = 0x0}, rs_flags = 0, rs_reqdata = 0x0}, rs_old = {bv_len = 0, bv_val = 0x0}, rs_new = { bv_len = 0, bv_val = 0x0}, rs_mods = 0x0, rs_modtail = 0x0}}, o_abandon = 0, o_cancel = 0, o_groups = 0x0, o_do_not_cache = 0 '\0', o_is_auth_check = 0 '\0', o_nocaching = 0 '\0', o_delete_glue_parent = 0 '\0', o_no_schema_check = 0 '\0', o_ctrlflag = '\0' <repeats 31 times>, o_controls = 0x11d8fdd8, o_authz = {sai_method = 0, sai_mech = {bv_len = 0, bv_val = 0x0}, sai_dn = {bv_len = 0, bv_val = 0x0}, sai_ndn = {bv_len = 0, bv_val = 0x0}, sai_ssf = 0, sai_transport_ssf = 0, sai_tls_ssf = 0, sai_sasl_ssf = 0}, o_ber = 0x0, o_res_ber = 0x0, o_callback = 0x0, o_ctrls = 0x0, o_csn = {bv_len = 0, bv_val = 0x0}, o_private = 0x0, o_next = {stqe_next = 0x12307c00}} (gdb) backtrace full #0 0x0000002a975f21ec in syncprov_op_abandon (op=0x42002b40, rs=0x42002a50) at syncprov.c:1026 on = (slap_overinst *) 0xde1c40 si = (syncprov_info_t *) 0xde2ee0 so = (syncops *) 0x12392240 soprev = (syncops *) 0x12394bd0 #1 0x000000000049b8dd in overlay_op_walk (op=0x42002b40, rs=0x42002a50, which=op_abandon, oi=0xde1e00, on=0xde1c40) at backover.c:640 func = (BI_op_bind **) 0xde1c98 rc = 32768 #2 0x000000000049bb39 in over_op_func (op=0x42002b40, rs=0x42002a50, which=op_abandon) at backover.c:702 oi = (slap_overinfo *) 0xde1e00 on = (slap_overinst *) 0xde1c40 be = (BackendDB *) 0xddf540 db = {bd_info = 0xde1c40, be_ctrls = "\000\001\001\001\000\000\001\000\001\000\001\001\001\000\001", '\0' <repeats 17 times>, "\001", be_flags = 256, be_restrictops = 0, be_requires = 0, be_ssf_set = {sss_ssf = 0, sss_transport = 0, sss_tls = 0, sss_sasl = 0, sss_update_ssf = 0, sss_update_transport = 0, sss_update_tls = 0, sss_update_sasl = 0, sss_simple_bind = 0}, be_suffix = 0xdd7920, be_nsuffix = 0xdd7900, be_schemadn = {bv_len = 0, bv_val = 0x0}, be_schemandn = {bv_len = 0, bv_val = 0x0}, be_rootdn = {bv_len = 12, bv_val = 0xdd8410 "cn=accesslog"}, be_rootndn = {bv_len = 12, bv_val = 0xdd8490 "cn=accesslog"}, be_rootpw = {bv_len = 0, bv_val = 0x0}, be_max_deref_depth = 15, be_def_limit = { lms_t_soft = -1, lms_t_hard = 0, lms_s_soft = -1, lms_s_hard = 0, lms_s_unchecked = -1, lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x0, be_acl = 0x8b0000, be_dfltaccess = ACL_READ, be_replica = 0x0, be_replogfile = 0x0, be_update_ndn = {bv_len = 0, bv_val = 0x0}, be_update_refs = 0x0, be_pending_csn_list = 0xf35a40, be_pcl_mutex = { __m_reserved = 0, __m_count = 0, __m_owner = 0x0, __m_kind = 0, __m_lock = {__status = 0, __spinlock = 0}}, be_pcl_mutexp = 0xddf680, be_syncinfo = 0x0, be_pb = 0x0, be_cf_ocs = 0x2a972f0fe0, be_private = 0x889c80, be_next = {stqe_next = 0xde1a80}} cb = {sc_next = 0x0, sc_response = 0x49b297 <over_back_response>, sc_cleanup = 0, sc_private = 0xde1e00} rc = 32768 __PRETTY_FUNCTION__ = "over_op_func" #3 0x000000000049bca7 in over_op_abandon (op=0x42002b40, rs=0x42002a50) at backover.c:760 No locals. #4 0x0000000000450729 in fe_op_abandon (op=0x42002b40, rs=0x42002a50) at abandon.c:115 No locals. #5 0x000000000042b249 in connection_abandon (c=0x12603dd0) at connection.c:792 o = (Operation *) 0x11d8f900 next = (Operation *) 0x0 op = {o_hdr = 0x42002ac0, o_tag = 80, o_time = 0, o_tincr = 0, o_bd = 0x42002830, o_req_dn = {bv_len = 0, bv_val = 0x0}, o_req_ndn = {bv_len = 0, bv_val = 0x0}, o_request = { oq_add = {rs_e = 0x4, rs_modlist = 0x0}, oq_bind = {rb_method = 4, rb_cred = {bv_len = 0, bv_val = 0x0}, rb_edn = {bv_len = 0, bv_val = 0x0}, rb_ssf = 0, rb_tmp_mech = {bv_len = 0, bv_val = 0x0}}, oq_compare = {rs_ava = 0x4}, oq_modify = {rs_modlist = 0x4, rs_increment = 0}, oq_modrdn = {rs_newrdn = {bv_len = 4, bv_val = 0x0}, rs_nnewrdn = {bv_len = 0, bv_val = 0x0}, rs_newSup = 0x0, rs_nnewSup = 0x0, rs_deleteoldrdn = 0}, oq_search = {rs_scope = 4, rs_deref = 0, rs_slimit = 0, rs_tlimit = 0, rs_limit = 0x0, rs_attrsonly = 0, rs_attrs = 0x0, rs_filter = 0x0, rs_filterstr = {bv_len = 0, bv_val = 0x0}}, oq_abandon = {rs_msgid = 4}, oq_cancel = {rs_msgid = 4}, oq_extended = {rs_reqoid = {bv_len = 4, bv_val = 0x0}, rs_flags = 0, rs_reqdata = 0x0}, oq_pwdexop = {rs_extended = {rs_reqoid = {bv_len = 4, bv_val = 0x0}, rs_flags = 0, rs_reqdata = 0x0}, rs_old = {bv_len = 0, bv_val = 0x0}, rs_new = {bv_len = 0, bv_val = 0x0}, rs_mods = 0x0, rs_modtail = 0x0}}, o_abandon = 0, o_cancel = 0, o_groups = 0x0, o_do_not_cache = 0 '\0', o_is_auth_check = 0 '\0', o_nocaching = 0 '\0', o_delete_glue_parent = 0 '\0', o_no_schema_check = 0 '\0', o_ctrlflag = '\0' <repeats 31 times>, o_controls = 0x0, o_authz = { sai_method = 0, sai_mech = {bv_len = 0, bv_val = 0x0}, sai_dn = {bv_len = 0, bv_val = 0x0}, sai_ndn = {bv_len = 0, bv_val = 0x0}, sai_ssf = 0, sai_transport_ssf = 0, sai_tls_ssf = 0, sai_sasl_ssf = 0}, o_ber = 0x0, o_res_ber = 0x0, o_callback = 0x42002810, o_ctrls = 0x0, o_csn = {bv_len = 0, bv_val = 0x0}, o_private = 0x0, o_next = {stqe_next = 0x0}} ohdr = {oh_opid = 0, oh_connid = 583, oh_conn = 0x12603dd0, oh_msgid = 0, oh_protocol = 0, oh_tid = 0, oh_threadctx = 0x0, oh_tmpmemctx = 0x0, oh_tmpmfuncs = 0x0, oh_log_prefix = '\0' <repeats 49 times>} rs = {sr_type = REP_RESULT, sr_tag = 0, sr_msgid = 0, sr_err = 0, sr_matched = 0x0, sr_text = 0x0, sr_ref = 0x0, sr_ctrls = 0x0, sr_un = {sru_sasl = {r_sasldata = 0x0}, sru_extended = {r_rspoid = 0x0, r_rspdata = 0x0}, sru_search = {r_entry = 0x0, r_attr_flags = 0, r_operational_attrs = 0x0, r_attrs = 0x0, r_nentries = 0, r_v2ref = 0x0}}, sr_flags = 0} #6 0x000000000042b41c in connection_closing (c=0x12603dd0, why=0x4be9a0 "connection lost") at connection.c:840 sd = 38 __PRETTY_FUNCTION__ = "connection_closing" #7 0x000000000042ca0f in connection_read (s=38, cri=0x42002d90) at connection.c:1457 rc = -2 c = (Connection *) 0x12603dd0 __PRETTY_FUNCTION__ = "connection_read" #8 0x000000000042c1f8 in connection_read_thread (ctx=0x42002e10, argv=0x26) at connection.c:1254 rc = 42 cri = {op = 0x11d8f900, func = 0, arg = 0x0, nullop = 0} s = 38 #9 0x0000002a956c3bd7 in ldap_int_thread_pool_wrapper (xpool=0x88ce10) at tpool.c:478 pool = (struct ldap_int_thread_pool_s *) 0x88ce10 ctx = (ldap_int_thread_ctx_t *) 0xf375d0 ltc_key = {{ltk_key = 0x48717a, ltk_data = 0xe658c0, ltk_free = 0x486f64 <slap_sl_mem_destroy>}, {ltk_key = 0x976100, ltk_data = 0xf, ltk_free = 0x2a971e9cf9 <bdb_locker_id_free>}, {ltk_key = 0x2a971dae1f, ltk_data = 0x159be000, ltk_free = 0x2a971dadff <search_stack_free>}, {ltk_key = 0x976680, ltk_data = 0xf, ltk_free = 0x2a971e9cf9 <bdb_locker_id_free>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0} <repeats 28 times>} tid = 1107310944 i = 219 keyslot = 219 hash = 219 #10 0x0000003342606137 in ?? () No symbol table info available. #11 0x0000000000000000 in ?? () No symbol table info available. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#5455) EntryInfo state not reset if entry_db lock fails
by rein＠basefarm.no 04 Apr '08

04 Apr '08

Full_Name: Rein Tollevik Version: CVS head OS: linux and solaris URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (81.93.160.250) In bdb_cache_find_id() in servers/slapd/back-bdb/cache.c it decrements the bei_finders of the EntryInfo and resets the loading flag in bei_state. These statements are only run if locking the entry_db succeed, but as they reset state the function has altered they should always be run. I.e, move these statements and the surrounding lock calls out of the block they currently are in. Rein Tollevik Basefarm AS

1 0

Re: (ITS#5454) syncrepl refreshAndPersist stops receiving
by rein＠basefarm.no 04 Apr '08

04 Apr '08

On Thu, 3 Apr 2008, rein(a)basefarm.no wrote: > Our persistent syncrepl consumers stops receiving data after a while, with no > indication of why :-( They don't recognize restarts of the producer, so the > only way to get the replication running again is to restart the consumers. > > So far it looks to me as if the syncrepl thread has managed to return without > adding the connection socket back to slap_daemon.sd_readers. But whether that > is correct or not, and how it has managed to do it if so, I cannot tell. Hm, I hope I have found the race condition that causes this :-) I'm now running with the patch at the end to see if that solves it, only time will tell.. The race is that between the time selecting on the syncrepl socket is enabled by the call to connection_client_enable() and the release of the si_mutex a new message may arrive. If so, the next call to do_syncrepl may fail in its attempt to trylock the mutex and no-one will re-enable selecting on it again. My patch delays enabling of the socket until the mutex has been released, which looks safe to me. Or can the access to si->si_conn without a lock be a problem? Rein Index: OpenLDAP/servers/slapd/syncrepl.c diff -u OpenLDAP/servers/slapd/syncrepl.c:1.9 OpenLDAP/servers/slapd/syncrepl.c:1.10 --- OpenLDAP/servers/slapd/syncrepl.c:1.9 Fri Mar 28 14:25:55 2008 +++ OpenLDAP/servers/slapd/syncrepl.c Fri Apr 4 16:54:05 2008 @@ -1166,6 +1166,7 @@ ber_socket_t s; int i, defer = 1, fail = 0; Backend *be; + int enable_conn = 0; Debug( LDAP_DEBUG_TRACE, "=>do_syncrepl %s\n", si->si_ridtxt, 0, 0 ); @@ -1271,11 +1272,7 @@ * If we failed, tear down the connection and reschedule. */ if ( rc == LDAP_SUCCESS ) { - if ( si->si_conn ) { - connection_client_enable( si->si_conn ); - } else { - si->si_conn = connection_client_setup( s, do_syncrepl, arg ); - } + enable_conn = 1; } else if ( si->si_conn ) { dostop = 1; } @@ -1342,6 +1339,14 @@ ldap_pvt_thread_mutex_unlock( &slapd_rq.rq_mutex ); ldap_pvt_thread_mutex_unlock( &si->si_mutex ); + if ( enable_conn ) { + if ( si->si_conn ) { + connection_client_enable( si->si_conn ); + } else { + si->si_conn = connection_client_setup( s, do_syncrepl, arg ); + } + } + if ( rc ) { if ( fail == RETRYNUM_TAIL ) { Debug( LDAP_DEBUG_ANY,

1 0

Re: (ITS#5452) seg fault in syncprov_op abandon on master
by h.b.furuseth＠usit.uio.no 04 Apr '08

04 Apr '08

Since it crashes on if ( so->s_op->o_connid == op->o_connid && so->s_op->o_msgid == op->orn_msgid ) please try frame 0 print *op print *so->s_op and maybe backtrace full while you are at it. -- Hallvard

1 0

Re: (ITS#5454) syncrepl refreshAndPersist stops receiving
by rein＠basefarm.no 04 Apr '08

04 Apr '08

On Fri, 4 Apr 2008, Howard Chu wrote: > rein(a)basefarm.no wrote: >> So far it looks to me as if the syncrepl thread has managed to return >> without >> adding the connection socket back to slap_daemon.sd_readers. But whether >> that >> is correct or not, and how it has managed to do it if so, I cannot tell. > > Since you're working with HEAD ... Were there any write operations on > cn=config just before the hang? (Thinking it may be connected to the > SYNC_PAUSED handling at line 1303, except any subsequent request would wake > up the listener anyway.) No, the config backend isn't even configured into the server. I have also tried without the auditlog just to make sure that wasn't involved, but it happend anyhow. We have also added a linux server on one of the sites to see if the problem is related to solaris. So far it hasn't stopped receiving updates, but is has only run since last evening so it is far too early to conclude anything. Rein

1 0

Re: (ITS#5454) syncrepl refreshAndPersist stops receiving
by hyc＠symas.com 04 Apr '08

04 Apr '08

rein(a)basefarm.no wrote: > So far it looks to me as if the syncrepl thread has managed to return without > adding the connection socket back to slap_daemon.sd_readers. But whether that > is correct or not, and how it has managed to do it if so, I cannot tell. Since you're working with HEAD ... Were there any write operations on cn=config just before the hang? (Thinking it may be connected to the SYNC_PAUSED handling at line 1303, except any subsequent request would wake up the listener anyway.) > Any pointers as to what can be wrong are highly appreciated. I have a core file > from a server that has stopped receiving if there is anything I should look for > there. > > Rein Tollevik > Basefarm AS > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5444) slapd seg. fault using mulitmirrored mode
by ando＠sys-net.it 04 Apr '08

04 Apr '08

hyc(a)symas.com wrote: > OK, I've reproduced the crash and patched it in HEAD, and added a consumer > write to the test050 script. Please test. Works for me. Thanks, p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5452) seg fault in syncprov_op abandon on master
by quanah＠zimbra.com 04 Apr '08

04 Apr '08

--On Thursday, April 03, 2008 6:41 PM +0000 quanah(a)zimbra.com wrote: > Full_Name: Quanah Gibson-Mount > Version: 2.3.41 > OS: Linux 2.6 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (24.23.156.219) > > > Core was generated by `/opt/zimbra/openldap/libexec/slapd -l LOCAL0 -4 -u > zimbra -h ldap://'. also: (gdb) print so $1 = (syncops *) 0x12392240 (gdb) print *so $2 = {s_next = 0x12393510, s_base = {bv_len = 12, bv_val = 0x1238dd50 "cn=accesslog"}, s_eid = 1, s_op = 0x11d8fc00, s_rid = 100, s_filterstr = {bv_len = 0, bv_val = 0x0}, s_flags = 1, s_inuse = 1, s_res = 0x0, s_restail = 0x0, s_qtask = 0x0, s_mutex = {__m_reserved = 0, __m_count = 0, __m_owner = 0x0, __m_kind = 0, __m_lock = {__status = 0, __spinlock = 0}}} from frame 0. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5450) slapd: schema_init.c:2215: integerIndexer: Assertion `i > 0' failed.
by hyc＠symas.com 04 Apr '08

04 Apr '08

quanah(a)zimbra.com wrote: > --On Tuesday, April 01, 2008 6:24 PM +0000 > sahullameerbasha.nagore(a)etrade.com wrote: > >> Full_Name: ameer basha >> Version: 2.4.8 >> OS: Red Hat linux As3 >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (198.93.33.254) >> >> >> I am getting the following error while adding the data into the ldap >> through siteminder gui. >> >> slapd: schema_init.c:2215: integerIndexer: Assertion `i> 0' failed. >> >> Do you have any idea why this error is getting > > > <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=474161> > > may also be related to this bug report. It includes reproduction data, > although the assert in this case triggered on the stringIndexer. Thanks for the data, that helped. Fixed in HEAD. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#5454) syncrepl refreshAndPersist stops receiving
by rein＠basefarm.no 04 Apr '08

04 Apr '08

Full_Name: Rein Tollevik Version: CVS head OS: solaris8 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (84.215.2.34) Our persistent syncrepl consumers stops receiving data after a while, with no indication of why :-( They don't recognize restarts of the producer, so the only way to get the replication running again is to restart the consumers. The consumers have a single bdb backend database that is replicated from the producer, and uses the auditlog overlay on this backend. There are 4 of them, running in pairs as load-balanced search servers on two sites. The two servers in each pair are identical configured, and they are all running 64bit solaris8 if that matters. There are two master servers, one on each site, with a more complicated configuration. They replicate subordinate backend datebases between each other, the consumers replicate the glue suffix of these backends. These servers are running linux in 32- and 64bit mode, and I have not seen the same type of replication stops between these master servers. Using netstat it shows that the send queue is full on the producer side, and the receive window is empty on the consumer, which to me looks as if the consumer has stopped reading from the provider. I have used a debugger to look at the servers after they have stopped receiving, and the syncrepl task is sitting at the end of slapd_rq.task_list, with next_sched.tv_sec==0 (which mean it will not be scheduled normally?) The syncrepl is configured with "retry=60 +". The si_conn of the syncinfo_t looks normal, and its c_sd socket is on slap_daemon.sd_actives fd_set, but not on sd_readers nor sd_writers. And none of the threads are running any syncrepl code. Looking at "slapd -d sync" output and the auditlogs it seem to stop receiving in the middle of a burst of updates (although that could be a coinsident). The other slave in the pairs continues to receive updates, so I assume this is a consumer side problem. So far it looks to me as if the syncrepl thread has managed to return without adding the connection socket back to slap_daemon.sd_readers. But whether that is correct or not, and how it has managed to do it if so, I cannot tell. Any pointers as to what can be wrong are highly appreciated. I have a core file from a server that has stopped receiving if there is anything I should look for there. Rein Tollevik Basefarm AS

1 0

Re: (ITS#5453) slapd binding error at startup
by hyc＠symas.com 03 Apr '08

03 Apr '08

m_hannini(a)yahoo.com wrote: > Full_Name: Mohammed > Version: 2.3.37 > OS: AIX 5.3 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (12.146.102.46) > > > Hi, > I installed openldap 2.3.37 on AIX 5.3. When I start slapd, I get this error: > daemon: bind(7) failed errno=67 (Address already in use) > slapd starting > > This is even though there is nothing running on port 389 but the ldap server > itself. > # netstat -an |grep 389 > tcp4 0 0 *.389 *.* LISTEN > > > Any ideas? There is no indication of a bug in OpenLDAP software here. This ITS will be closed. Software usage questions should be directed to the openldap-software mailing list. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5450) slapd: schema_init.c:2215: integerIndexer: Assertion `i > 0' failed.
by quanah＠zimbra.com 03 Apr '08

03 Apr '08

--On Tuesday, April 01, 2008 6:24 PM +0000 sahullameerbasha.nagore(a)etrade.com wrote: > Full_Name: ameer basha > Version: 2.4.8 > OS: Red Hat linux As3 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (198.93.33.254) > > > I am getting the following error while adding the data into the ldap > through siteminder gui. > > slapd: schema_init.c:2215: integerIndexer: Assertion `i > 0' failed. > > Do you have any idea why this error is getting <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=474161> may also be related to this bug report. It includes reproduction data, although the assert in this case triggered on the stringIndexer. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#5453) slapd binding error at startup
by m_hannini＠yahoo.com 03 Apr '08

03 Apr '08

Full_Name: Mohammed Version: 2.3.37 OS: AIX 5.3 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (12.146.102.46) Hi, I installed openldap 2.3.37 on AIX 5.3. When I start slapd, I get this error: daemon: bind(7) failed errno=67 (Address already in use) slapd starting This is even though there is nothing running on port 389 but the ldap server itself. # netstat -an |grep 389 tcp4 0 0 *.389 *.* LISTEN Any ideas? Thanks

1 0

(ITS#5452) seg fault in syncprov_op abandon on master
by quanah＠zimbra.com 03 Apr '08

03 Apr '08

Full_Name: Quanah Gibson-Mount Version: 2.3.41 OS: Linux 2.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (24.23.156.219) Core was generated by `/opt/zimbra/openldap/libexec/slapd -l LOCAL0 -4 -u zimbra -h ldap://'. Program terminated with signal 11, Segmentation fault. #0 0x0000002a975f21ec in syncprov_op_abandon (op=0x42002b40, rs=0x42002a50) at syncprov.c:1026 1026 if ( so->s_op->o_connid == op->o_connid && (gdb) bt #0 0x0000002a975f21ec in syncprov_op_abandon (op=0x42002b40, rs=0x42002a50) at syncprov.c:1026 #1 0x000000000049b8dd in overlay_op_walk (op=0x42002b40, rs=0x42002a50, which=op_abandon, oi=0xde1e00, on=0xde1c40) at backover.c:640 #2 0x000000000049bb39 in over_op_func (op=0x42002b40, rs=0x42002a50, which=op_abandon) at backover.c:702 #3 0x000000000049bca7 in over_op_abandon (op=0x42002b40, rs=0x42002a50) at backover.c:760 #4 0x0000000000450729 in fe_op_abandon (op=0x42002b40, rs=0x42002a50) at abandon.c:115 #5 0x000000000042b249 in connection_abandon (c=0x12603dd0) at connection.c:792 #6 0x000000000042b41c in connection_closing (c=0x12603dd0, why=0x4be9a0 "connection lost") at connection.c:840 #7 0x000000000042ca0f in connection_read (s=38, cri=0x42002d90) at connection.c:1457 #8 0x000000000042c1f8 in connection_read_thread (ctx=0x42002e10, argv=0x26) at connection.c:1254 #9 0x0000002a956c3bd7 in ldap_int_thread_pool_wrapper (xpool=0x88ce10) at tpool.c:478 #10 0x0000003342606137 in ?? () #11 0x0000000000000000 in ?? () (gdb) frame 0 #0 0x0000002a975f21ec in syncprov_op_abandon (op=0x42002b40, rs=0x42002a50) at syncprov.c:1026 1026 if ( so->s_op->o_connid == op->o_connid && (gdb) l 1021 syncops *so, *soprev; 1022 1023 ldap_pvt_thread_mutex_lock( &si->si_ops_mutex ); 1024 for ( so=si->si_ops, soprev = (syncops *)&si->si_ops; so; 1025 soprev=so, so=so->s_next ) { 1026 if ( so->s_op->o_connid == op->o_connid && 1027 so->s_op->o_msgid == op->orn_msgid ) { 1028 so->s_op->o_abandon = 1; 1029 soprev->s_next = so->s_next; 1030 break; (gdb)

1 0

Re: (ITS#5340) REP_ENTRY_MODIFIABLE bug in dynlist
by h.b.furuseth＠usit.uio.no 03 Apr '08

03 Apr '08

ando(a)sys-net.it writes: > That's why I'm not so happy with MUSTRELEASE. Seems to be required though. cvs log for slap.h revision 1.678. But it does look to me like SlapReply needs a BackendDB *e_owner member. I haven't observed buggy behavior related to that (or tried to provoke any), but it looks hairy for a backend with a be_release() function to make sure to intercept every entry from elsewhere and record previous owner info in e_private or callbacks. > OK, makes sense. But perhaps creating entries on the stack could be > deprecated now that entries are pooled and reused by calling > entry_alloc() entry_free(). This is just food for thoughts, we could as > well leave all those flags 'round, if we can streamline their handling. I think we might as well leave them. Since most are for saving memory, might even want to expand them eventually. (E.g. separate flags for _which_ parts of an entry are modifiable. Modifiable attribute list, DNs, values _in_ attribute list, etc.) For that matter, looking at entry_free/entry_alloc, I don't see how that pooling saves more than it costs. It saves 1 malloc for the Entry structure, but not the mallocs for the DN and attribute list. And it costs two global mutex lock/unlocks per entry, in entry_alloc() and entry_free(). An entry on the stack avoids both malloc and locks. Anyway, I guess one conclusion of all this is that the code really is as sloppy as it looked, there are no magic saving throws in there. (Except with finding the backend for be_release if I'm still missing something about that.) -- Hallvard

1 0

Re: (ITS#5444) slapd seg. fault using mulitmirrored mode
by hyc＠symas.com 03 Apr '08

03 Apr '08

Pierangelo Masarati wrote: > hyc(a)symas.com wrote: >> ando(a)sys-net.it wrote: >>> ando(a)sys-net.it wrote: >>>> mohel(a)web.de wrote: >>>> >>>>> The above log was generated using the test050 script just adding a line which >>>>> enters information also to a second server. Perhaps this test would also make >>>>> sense for further releases. >>>> Can you please post the modification to the script? >>> Never mind, I could reproduce it. It seems to be a duplicate of >>> ITS#5437. What actually happens is that if modifications are applied to >>> all three servers, those applied to consumers do not persist, nor get >>> replicated; on the contrary, the consumers crash later on as described >>> in ITS#5437. >> #5437? What you're talking about doesn't seem to match any text in ITS#5437. > > Well, sorry for not providing more details, I was going to investigate > it a little bit more but I had to leave. The crash to me occurs in > syncprov_done_ctrl(), as indicated in ITS#5437. What happens is that > syncprov_done_ctrl() with an invalid cookie (unset; clearing it in > syncprov_search_response(), as suggested in ITS#5437 leads to the > behavior I described (writes to consumers are simply ignored, don't > persist in the consumers and are not replicated). To reproduce, add > something to a consumer close to the end of test050, before killing the > servers, and wait a little bit to let things proceed to the crash. OK, I've reproduced the crash and patched it in HEAD, and added a consumer write to the test050 script. Please test. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

ITS#5437
by ando＠sys-net.it 02 Apr '08

02 Apr '08

The issue pointed out in this thread <http://www.openldap.org/lists/openldap-software/200804/msg00036.html> could be related. ITS#5444 could be related as well. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5369) GSSAPI support for client library
by mimir＠samba.org 02 Apr '08

02 Apr '08

--XsQoSWH+UP9D9v3l Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Kurt, On Mon, Mar 03, 2008 at 02:50:49PM -0800, Kurt Zeilenga wrote: > In your rights statement, you claim this patch is only derived from =20 > OpenLDAP Software. Is this correct? Are there are works from which =20 > your work is derived? That is, did you copy any material from any other= =20 > work (e.g., Samba)? Please document any in the rights statement and,=20 > when complete, update the ITS (by email reply-all) indicating so. > > The rights statement needs to be amended to include a statement that the= =20 > modifications are not subject to Stefan's employer. And then Stefan=20 > needs to send a copy of both statements from his email address to=20 > openldap-its(a)openldap.org with a subject of "ITS#5369 IPR statements"=20 > confirming they are correct. I've cc'ed Stefan on this note. Have you received Stefan's copyright statement ? cheers, --=20 Rafal Szczesniak Samba Team member http://www.samba.org Likewise Software http://www.likewisesoftware.com --XsQoSWH+UP9D9v3l Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFH81vZHvdfyv3qiKkRAhI2AKCWKZOTtjG4F+eSqYiAMN6MlCtCdwCgoCOb Ir40KCUW32l9HAkhiqyYCxA= =coUh -----END PGP SIGNATURE----- --XsQoSWH+UP9D9v3l--

1 0

Re: (ITS#5440) back-meta opens too many connections in case of error
by ali.pouya＠free.fr 02 Apr '08

02 Apr '08

Pierangelo Masarati <ando(a)sys-net.it> wrote : > > issue I observed does not show up any longer. Probably, as an outcome > of this ITS, we should change the default to "yes". > Yes. I agree with you. Because in all my configurations I have to set : pseudoroot-bind-defer yes Thanks for the improvement. Best Regards Ali

1 0

Re: (ITS#5451) syncprov_checkpoint deadlock bugfix?
by rein＠basefarm.no 02 Apr '08

02 Apr '08

On Tue, 1 Apr 2008, rein(a)basefarm.no wrote: > Full_Name: Rein Tollevik > Version: CVS head > OS: linux and solaris > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (81.93.160.250) > > > We have seen deadlock/hang situations in our master server today, which looks > like a deadlock caused by the si_csn_rwlock lock being held while > syncprov_checkpoint() is running. The patch at the end should (I hope) fix > this. Hm, I looks as if I was a bit to quick here, it just hung got again :-(. And this time without anyone competing for this lock, only the lock on the glue suffix entry. I wonder if it can be the upgrade from db 4.6.18 to 4.6.21.1 I dit yesterday that is the real problem. I'll try to downgrade and see if that helps. Please put this case on hold. Sorry! Rein

1 0

(ITS#5451) syncprov_checkpoint deadlock bugfix?
by rein＠basefarm.no 01 Apr '08

01 Apr '08

Full_Name: Rein Tollevik Version: CVS head OS: linux and solaris URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (81.93.160.250) We have seen deadlock/hang situations in our master server today, which looks like a deadlock caused by the si_csn_rwlock lock being held while syncprov_checkpoint() is running. The patch at the end should (I hope) fix this. The scenario is a server with a the syncprov overlay on a glue database with multiple bdb backends as subordinate. One of them is a syncrepl consumer, although I don't know if that matters. A modify operation on the syncrepl consumer causes a checkpoint of the contextCSN. The resulting modify on the glue backend hangs in bdb_cache_modify() awaiting the bdb_cache_entry_db_relock() to return a write lock on the glue suffix entry. At the same time is a modify on one of the other backends hanging in syncprov_op_response() waiting for a lock on the si_csn_rwlock which the first thread holds. The remaining threads quickly blocks waiting for a read lock on the glue suffix entry (the most common search base). I recon the read locks are queued to give priority to the thread waiting for the write lock. What I haven't figured out is who that holds the lock on the glue suffix entry in the first place... Anyway, holding the si_csn_rwlock while the contextCSN is written to the database should not be necessary. But I don't know what consequences it whould have if more than one thread should end up checkpointing competing values, so I have added a new flag to the syncprov_info structure to prevent simultaneons checkpoints from occuring rather than using locks to do it. Rein Tollevik Basefarm AS Index: OpenLDAP/servers/slapd/overlays/syncprov.c diff -u OpenLDAP/servers/slapd/overlays/syncprov.c:1.1.1.13 OpenLDAP/servers/slapd/overlays/syncprov.c:1.12 --- OpenLDAP/servers/slapd/overlays/syncprov.c:1.1.1.13 Mon Mar 31 17:29:48 2008 +++ OpenLDAP/servers/slapd/overlays/syncprov.c Tue Apr 1 21:54:05 2008 @@ -128,6 +128,7 @@ int si_numops; /* number of ops since last checkpoint */ int si_nopres; /* Skip present phase */ int si_usehint; /* use reload hint */ + int si_check; /* are we checkpointing? */ time_t si_chklast; /* time of last checkpoint */ Avlnode *si_mods; /* entries being modified */ sessionlog *si_logs; @@ -1308,8 +1309,10 @@ slap_callback cb = {0}; BackendDB be; + ldap_pvt_thread_rdwr_rlock( &si->si_csn_rwlock ); mod.sml_numvals = si->si_numcsns; mod.sml_values = si->si_ctxcsn; + ldap_pvt_thread_rdwr_runlock( &si->si_csn_rwlock ); mod.sml_nvalues = NULL; mod.sml_desc = slap_schema.si_ad_contextCSN; mod.sml_op = LDAP_MOD_REPLACE; @@ -1335,6 +1338,8 @@ if ( mod.sml_next != NULL ) { slap_mods_free( mod.sml_next, 1 ); } + + si->si_check = 0; } static void @@ -1612,7 +1617,7 @@ } si->si_numops++; - if ( si->si_chkops || si->si_chktime ) { + if ( !si->si_check && (si->si_chkops || si->si_chktime) ) { if ( si->si_chkops && si->si_numops >= si->si_chkops ) { do_check = 1; si->si_numops = 0; @@ -1627,12 +1632,14 @@ } } } + + if ( do_check ) + si->si_check = 1; + ldap_pvt_thread_rdwr_wunlock( &si->si_csn_rwlock ); if ( do_check ) { - ldap_pvt_thread_rdwr_rlock( &si->si_csn_rwlock ); syncprov_checkpoint( op, rs, on ); - ldap_pvt_thread_rdwr_runlock( &si->si_csn_rwlock ); } opc->sctxcsn.bv_len = maxcsn.bv_len;

1 0

(ITS#5450) slapd: schema_init.c:2215: integerIndexer: Assertion `i > 0' failed.
by sahullameerbasha.nagore＠etrade.com 01 Apr '08

01 Apr '08

Full_Name: ameer basha Version: 2.4.8 OS: Red Hat linux As3 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (198.93.33.254) I am getting the following error while adding the data into the ldap through siteminder gui. slapd: schema_init.c:2215: integerIndexer: Assertion `i > 0' failed. Do you have any idea why this error is getting

1 0

Re: (ITS#5340) REP_ENTRY_MODIFIABLE bug in dynlist
by ando＠sys-net.it 01 Apr '08

01 Apr '08

Hallvard B Furuseth wrote: > Pierangelo Masarati writes: >> Hallvard B Furuseth wrote: >>> But I don't see how the be_release() code can work now. It sounds like >>> be->be_release() functions must check (how?) that the entry was created >>> by 'be', and otherwise pass it on to the next overlay/backend or >>> otherwise to entry_free(). Might involve mucking with op->o_bd and >>> sr_entry->e_private, I suppose. Except maybe I'm missing some existing >>> magic since slapd doesn't regularly crash... >> Yes, but that's trivial: e_private must be NULL for temporary entries, >> and copying the entry loses it (no one is supposed to muck with it >> expect the entry's creator). > > OK... > >> And the appropriate o_bd of a >> (non-modifiable) entry can be easily computed from the entry's DN. > > Hm? Can only the "outermost" backend/overlay create non-modifiable > entries? > > I think any overlay can replace a non-modifiable rs->sr_entry value, as > long as it restores it on the way out (unless mustbe<freed/released>). That's why I'm not so happy with MUSTRELEASE. >>>> Similarly, the existence of REP_ENTRY_MUSTBEFREED is not totally clear: >>>> in principle as soon as REP_ENTRY_MODIFYABLE is set, it should imply >>>> REP_ENTRY_MUSTBEFREED; the only difference in the semantics of the two >>> That's not my impression. (...) > > Some functions create entries on the stack an send them MODIFIABLE - but > obviously without MUSTBEFREED. They use entry_clean() instead. > > Others set MODIFIABLE but not MUSTBEFREED on non-stack entries, and call > entry_free after sending it. Or forget to free it - back-perl/search.c > if LDAP_SIZELIMIT_EXCEEDED. (Not patching yet - don't have time to test > it at the moment.) OK, makes sense. But perhaps creating entries on the stack could be deprecated now that entries are pooled and reused by calling entry_alloc() entry_free(). This is just food for thoughts, we could as well leave all those flags 'round, if we can streamline their handling. >> (...) A copy might be created by an overlay after receiving a >> read-only entry, but the same overlay might not actually perform the >> copy if it receives the entry from another overlay that already copied >> it, or from a proxy backend. However, after the entry is copied the >> overlay will have no means to determined who actually created the copy. > > I don't think it matters who created a MUSTBEFREED copy: right. >> This might be an issue depending on the order cleanup handlers are >> called (didn't check what order they're called). My point is that >> temporary entries need to be freed at some point; who frees them should >> not be relevant... > > Right, someone must call entry_free() and clear rs->sr_entry (so it's > not freed again), but entry_free() is not passed a backend parameter. > > OTOH a MUSTBERELEASED entry must be released by the right > backend/overlay, and it would be strange for such an entry to be > MODIFIBALE. Though I guess it could be - if the backend has no cache, > and be_release just exists to clean up some data in e_private. > >>> Others apparent problems (also not tested, I've just browsed the code): >>> Overlays that obey and reset MUSTBEFREED or MUSTRELEASE, do not >>> necessarily clear or set MODIFIABLE when setting a new entry. >>> (...) >> It seems to me that we should provide "smart" handlers to deal with >> preparing sr_entry for modification, and to take care of cleaning things >> up as appropriate. Those helpers should then be consistently used in >> the code. > > Yup. And add some aggressive asserts - at least #ifdef LDAP_DEVEL. > Don't know when I'll have time for it though. Making room in my todo list... p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5448) CONTRIB: Java slapd
by Kurt＠OpenLDAP.org 01 Apr '08

01 Apr '08

On Mar 31, 2008, at 8:01 PM, hyc(a)OpenLDAP.org wrote: > Full_Name: Howard Chu > Version: 0.0.1 > OS: > URL: ftp://ftp.openldap.org/incoming/jslapd.tgz > Submission from: (NULL) (76.91.220.157) > Submitted by: hyc > > > Given the increasing popularity of Java, I decided it was time that > the > OpenLDAP Project had its own Java implementation of an LDAP server. It > gave me a good opportunity to explore all of the benefits of rapid > development, introspection, and other advantages of the Java system. > Indeed, drawing on my experience with the existing OpenLDAP code base, > it took only a matter of hours to create my first working server, > which > I present here. Of course it's still experimental in nature. > > Invocation is similar to the regular C-based slapd, and it accepts all > of the same arguments and reads all of the same config settings. In > addition, the first commandline argument must be the name of the > directory > where slapd resides, so that the java app can locate the config files, > schema, etc. > > E.g. for a typical OpenLDAP installation in /usr/local you would run > java jslapd /usr/local/libexec -h ldap://:389 > > A small shell script has been provided to simplify this invocation. > In my initial testing, I found jslapd to have performance on par with slapd(8)! And the memory footprint doesn't appear all that much larger. Good work! -- Kurt

1 0

Re: (ITS#5440) back-meta opens too many connections in case of error
by ando＠sys-net.it 01 Apr '08

01 Apr '08

Ali Pouya wrote: > Hi Pierangelo; > > When trying to reproduce the problem with test035 material I found a > problem in my own JAVA code ! > There is no such problem in back-meta :-) > Sorry for misreporting and thanks for your help. OK, no problem. I note that the issue I fixed in HEAD is basically related to not using pseudoroot-bind-defer yes The default is "no", basically to preserve the original behavior. The original behavior consisted in always binding to all targets when bound as the rootdn. When "pseudoroot-bind-defer" is set to "yes", the bind only occurs locally at the proxy, and is propagated to the targets only when they actually need to be used. The reason is that binding as the rootdn might only be needed to operate on a subset of the targets, so there is no need to add extra traffic to those outside this set. When "pseudoroot-bind-defer" is set to "yes", this no longer occurs, and the issue I observed does not show up any longer. Probably, as an outcome of this ITS, we should change the default to "yes". p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5340) REP_ENTRY_MODIFIABLE bug in dynlist
by h.b.furuseth＠usit.uio.no 01 Apr '08

01 Apr '08

Pierangelo Masarati writes: > Hallvard B Furuseth wrote: >> But I don't see how the be_release() code can work now. It sounds like >> be->be_release() functions must check (how?) that the entry was created >> by 'be', and otherwise pass it on to the next overlay/backend or >> otherwise to entry_free(). Might involve mucking with op->o_bd and >> sr_entry->e_private, I suppose. Except maybe I'm missing some existing >> magic since slapd doesn't regularly crash... > > Yes, but that's trivial: e_private must be NULL for temporary entries, > and copying the entry loses it (no one is supposed to muck with it > expect the entry's creator). OK... > And the appropriate o_bd of a > (non-modifiable) entry can be easily computed from the entry's DN. Hm? Can only the "outermost" backend/overlay create non-modifiable entries? I think any overlay can replace a non-modifiable rs->sr_entry value, as long as it restores it on the way out (unless mustbe<freed/released>). >>> Similarly, the existence of REP_ENTRY_MUSTBEFREED is not totally clear: >>> in principle as soon as REP_ENTRY_MODIFYABLE is set, it should imply >>> REP_ENTRY_MUSTBEFREED; the only difference in the semantics of the two >> >> That's not my impression. (...) Some functions create entries on the stack an send them MODIFIABLE - but obviously without MUSTBEFREED. They use entry_clean() instead. Others set MODIFIABLE but not MUSTBEFREED on non-stack entries, and call entry_free after sending it. Or forget to free it - back-perl/search.c if LDAP_SIZELIMIT_EXCEEDED. (Not patching yet - don't have time to test it at the moment.) > (...) A copy might be created by an overlay after receiving a > read-only entry, but the same overlay might not actually perform the > copy if it receives the entry from another overlay that already copied > it, or from a proxy backend. However, after the entry is copied the > overlay will have no means to determined who actually created the copy. I don't think it matters who created a MUSTBEFREED copy: > This might be an issue depending on the order cleanup handlers are > called (didn't check what order they're called). My point is that > temporary entries need to be freed at some point; who frees them should > not be relevant... Right, someone must call entry_free() and clear rs->sr_entry (so it's not freed again), but entry_free() is not passed a backend parameter. OTOH a MUSTBERELEASED entry must be released by the right backend/overlay, and it would be strange for such an entry to be MODIFIBALE. Though I guess it could be - if the backend has no cache, and be_release just exists to clean up some data in e_private. >> Others apparent problems (also not tested, I've just browsed the code): >> Overlays that obey and reset MUSTBEFREED or MUSTRELEASE, do not >> necessarily clear or set MODIFIABLE when setting a new entry. >> (...) > > It seems to me that we should provide "smart" handlers to deal with > preparing sr_entry for modification, and to take care of cleaning things > up as appropriate. Those helpers should then be consistently used in > the code. Yup. And add some aggressive asserts - at least #ifdef LDAP_DEVEL. Don't know when I'll have time for it though. -- Hallvard

1 0

Re: (ITS#5448) CONTRIB: Java slapd
by ando＠sys-net.it 01 Apr '08

01 Apr '08

hyc(a)OpenLDAP.org wrote: > Given the increasing popularity of Java, I decided it was time that the > OpenLDAP Project had its own Java implementation of an LDAP server. It > gave me a good opportunity to explore all of the benefits of rapid > development, introspection, and other advantages of the Java system. > Indeed, drawing on my experience with the existing OpenLDAP code base, > it took only a matter of hours to create my first working server, which > I present here. Of course it's still experimental in nature. > > Invocation is similar to the regular C-based slapd, and it accepts all > of the same arguments and reads all of the same config settings. In > addition, the first commandline argument must be the name of the directory > where slapd resides, so that the java app can locate the config files, > schema, etc. > > E.g. for a typical OpenLDAP installation in /usr/local you would run > java jslapd /usr/local/libexec -h ldap://:389 > > A small shell script has been provided to simplify this invocation. Howard, many thanks for the contribution. I haven't stressed it yet intensively, but at a first glance it seems to be as performing as the latest 2.4. Will this be released in 2.4, or do we need to wait until 2.5 (what about making it 3.0?). p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5440) back-meta opens too many connections in case of error
by ali.pouya＠free.fr 01 Apr '08

01 Apr '08

Hi Pierangelo; When trying to reproduce the problem with test035 material I found a problem in my own JAVA code ! There is no such problem in back-meta :-) Sorry for misreporting and thanks for your help. The issue is to be closed. Best regards Ali Pierangelo Masarati a écrit : > ali.pouya(a)free.fr wrote: > >> I precise that The behaviour is not the same if the client binds with >> rootdn. > > I've committed a fix that removes dangling connections as the rootdn; > I have been unable to reproduce the issue with regular users. Please > provide more useful info in order to track the issue (slapd.conf, LDIF > and requests). > > p. > > > > Ing. Pierangelo Masarati > OpenLDAP Core Team > > SysNet s.r.l. > via Dossi, 8 - 27100 Pavia - ITALIA > http://www.sys-net.it > --------------------------------------- > Office: +39 02 23998309 > Mobile: +39 333 4963172 > Email: pierangelo.masarati(a)sys-net.it > --------------------------------------- > > > >

1 0

(ITS#5449) X-PIGLATIN password hashing functions
by mbackes＠symas.com 01 Apr '08

01 Apr '08

Full_Name: Matthew Backes Version: 2.x OS: any URL: http://www.symas.net/~lucca/piglatin-patch.txt Submission from: (NULL) (76.88.99.93) In order to improve the variety of security options available to LDAP administrators, I am submitting an additional password hashing method for liblutil. This patch implements the {X-PIGLATIN} hash, e.g.: $ slappasswd -s feep -h '{X-PIGLATIN}' {X-PIGLATIN}eepfay The standard with -yay variation for leading values is used, as described in http://en.wikipedia.org/wiki/Pig_latin y is considered a vowel unless it is the leading char. The patch will need additional review for EBCDIC support. This patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Matthew Backes <mbackes(a)symas.com>. I have not assigned rights and/or interest in this work to any party. diff -u -r1.107 passwd.c --- passwd.c 7 Jan 2008 23:20:06 -0000 1.107 +++ passwd.c 1 Apr 2008 03:13:13 -0000 @@ -119,6 +119,9 @@ #endif #endif +static LUTIL_PASSWD_CHK_FUNC chk_piglatin; +static LUTIL_PASSWD_HASH_FUNC hash_piglatin; + /* password hash routines */ #ifdef SLAPD_CLEARTEXT @@ -154,6 +157,7 @@ { BER_BVC("{CLEARTEXT}"), NULL, hash_clear }, #endif + { BER_BVC("{X-PIGLATIN}"), chk_piglatin, hash_piglatin }, { BER_BVNULL, NULL, NULL } }; @@ -1127,3 +1131,78 @@ } #endif +static int +chk_piglatin( + const struct berval *sc, + const struct berval * passwd, + const struct berval * cred, + const char **text ) +{ + struct berval credhash; + int rc; + + rc = hash_piglatin( sc, cred, &credhash, NULL ); + if( rc != LUTIL_PASSWD_OK ) return rc; + + rc = passwd->bv_len == credhash.bv_len + && memcmp( passwd->bv_val, + credhash.bv_val, + credhash.bv_len ) == 0 ? + LUTIL_PASSWD_ERR : LUTIL_PASSWD_OK; + + ber_memfree( credhash.bv_val ); + return rc; +} + +/* Implement {X-PIGLATIN} + * See http://en.wikipedia.org/wiki/Pig_latin + * -yay variation for leading vowels used + * y is considered a vowel unless it is the leading char + */ + +static int +hash_piglatin( + const struct berval *scheme, + const struct berval *passwd, + struct berval *hash, + const char **text ) +{ + struct berval digest; + int c, rc, first_vowel = 0; + + hash->bv_len = scheme->bv_len + passwd->bv_len + 2; + hash->bv_val = ber_memalloc( scheme->bv_len + passwd->bv_len + 4 ); + if( !hash->bv_val ) return LUTIL_PASSWD_ERR; + + for( c = 0; c < passwd->bv_len; ++c ) { + switch( passwd->bv_val[c] & -33 ) { + case 'Y': + if( c==0 ) break; + case 'A': + case 'E': + case 'I': + case 'O': + case 'U': + first_vowel = c; + goto got_vowel; + } + } + +got_vowel: + if( first_vowel ) { + char * pos = hash->bv_val; + pos += sprintf( pos, "%s%s", + scheme->bv_val, + passwd->bv_val+first_vowel ); + snprintf( pos, first_vowel+1, "%s", + passwd->bv_val ); + pos += first_vowel; + sprintf( pos, "ay" ); + } else { + sprintf( hash->bv_val, "%s%syay", + scheme->bv_val, passwd->bv_val ); + hash->bv_len += 1; + } + + return LUTIL_PASSWD_OK; +} Also available from the provided URL.

1 0

(ITS#5448) CONTRIB: Java slapd
by hyc＠OpenLDAP.org 01 Apr '08

01 Apr '08

Full_Name: Howard Chu Version: 0.0.1 OS: URL: ftp://ftp.openldap.org/incoming/jslapd.tgz Submission from: (NULL) (76.91.220.157) Submitted by: hyc Given the increasing popularity of Java, I decided it was time that the OpenLDAP Project had its own Java implementation of an LDAP server. It gave me a good opportunity to explore all of the benefits of rapid development, introspection, and other advantages of the Java system. Indeed, drawing on my experience with the existing OpenLDAP code base, it took only a matter of hours to create my first working server, which I present here. Of course it's still experimental in nature. Invocation is similar to the regular C-based slapd, and it accepts all of the same arguments and reads all of the same config settings. In addition, the first commandline argument must be the name of the directory where slapd resides, so that the java app can locate the config files, schema, etc. E.g. for a typical OpenLDAP installation in /usr/local you would run java jslapd /usr/local/libexec -h ldap://:389 A small shell script has been provided to simplify this invocation.

1 0

Re: (ITS#5444) slapd seg. fault using mulitmirrored mode
by ando＠sys-net.it 01 Apr '08

01 Apr '08

hyc(a)symas.com wrote: > ando(a)sys-net.it wrote: >> ando(a)sys-net.it wrote: >>> mohel(a)web.de wrote: >>> >>>> The above log was generated using the test050 script just adding a line which >>>> enters information also to a second server. Perhaps this test would also make >>>> sense for further releases. >>> Can you please post the modification to the script? >> Never mind, I could reproduce it. It seems to be a duplicate of >> ITS#5437. What actually happens is that if modifications are applied to >> all three servers, those applied to consumers do not persist, nor get >> replicated; on the contrary, the consumers crash later on as described >> in ITS#5437. > > #5437? What you're talking about doesn't seem to match any text in ITS#5437. Well, sorry for not providing more details, I was going to investigate it a little bit more but I had to leave. The crash to me occurs in syncprov_done_ctrl(), as indicated in ITS#5437. What happens is that syncprov_done_ctrl() with an invalid cookie (unset; clearing it in syncprov_search_response(), as suggested in ITS#5437 leads to the behavior I described (writes to consumers are simply ignored, don't persist in the consumers and are not replicated). To reproduce, add something to a consumer close to the end of test050, before killing the servers, and wait a little bit to let things proceed to the crash. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5444) slapd seg. fault using mulitmirrored mode
by hyc＠symas.com 31 Mar '08

31 Mar '08

ando(a)sys-net.it wrote: > ando(a)sys-net.it wrote: >> mohel(a)web.de wrote: >> >>> The above log was generated using the test050 script just adding a line which >>> enters information also to a second server. Perhaps this test would also make >>> sense for further releases. >> Can you please post the modification to the script? > > Never mind, I could reproduce it. It seems to be a duplicate of > ITS#5437. What actually happens is that if modifications are applied to > all three servers, those applied to consumers do not persist, nor get > replicated; on the contrary, the consumers crash later on as described > in ITS#5437. #5437? What you're talking about doesn't seem to match any text in ITS#5437. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5442) slapd_rq not locked before use bugfix
by hyc＠symas.com 31 Mar '08

31 Mar '08

Rein Tollevik wrote: > On Sun, 30 Mar 2008, hyc(a)symas.com wrote: > >> rein(a)tollevik.no wrote: >>> On Sat, 29 Mar 2008, ando(a)sys-net.it wrote: >>>> rein(a)basefarm.no wrote: > >>>>> I was seeing random failures of the test050-syncrepl-multimaster test. One of >>>>> the failures was that it went into a tight loop traversing a circular runqueue >>>>> it had managed to create in slapd_rq.task_list. It seems as this was caused by >>>>> missing mutex locks around accesses to slapd_rq, which the patch uploaded to >>>>> ftp://ftp.openldap.org/incoming/slapd_rq_lock.patch fixes. >>>>> >>>>> Before I applied this patch the test failed after being run a few times, with it >>>>> it has now passed 100 times and is still counting. >>>> locks in back-bdb/config.c should be pointless, as modifications to the >>>> configuration should only occur while all threads are paused. The rest >>>> makes sort of sense, but I'd leave it to Howard. >> Ignoring the ITS#5403 changes, I don't see anything here that isn't >> config-related, therefore it's all running single-threaded. > > Now that the configuration can be changed dynamically (as this test does) > I find it a bit odd that the config stuff should always be running > single-threaded. But there is obviously much I don't know about the > internals of slapd. This was discussed at the 2003 OpenLDAP Developers' Day. http://www.openldap.org/conf/odd-wien-2003/proceedings.html Page 7 of my slides touched on it. It's also mentioned again here http://www.openldap.org/lists/openldap-devel/200505/msg00062.html The main point is that the original config stuff assumed that it was only executing during slapd startup, and single-threaded. Putting locks around all of the potential configurable value accesses would have been too much work, so the decision was made to force slapd to be single-threaded whenever writing to the config. I'm guessing that what's happening here is just a broken assumption - suspending the thread pool doesn't in fact freeze everything in slapd. In particular, the slapd listener thread may still wake up for a select() timeout to handle the slapd runqueue. Even though no tasks submitted as a result of this will execute (because they'll just get queued into the suspended thread pool) the runqueue itself is still being manipulated. Given that explanation, your patches make sense. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5439) back-bdb race condition
by quanah＠zimbra.com 31 Mar '08

31 Mar '08

--On Monday, March 31, 2008 6:27 PM +0000 rein(a)basefarm.no wrote: > We did see segmentation faults both Friday and Monday (not very much > going on during the weekend..), but so far those appear to be related to > ITS#5445. A slapd and db_checkpoint (run out of cron) deadlock which I > hope is due to the BerkeleyDB bug mentioned in ITS#5391 has also > occurred, so I'll upgrade from 4.6.18 to 4.6.21.1 to see if that problem > goes away. Why do you run db_checkpoint out of cron? Slapd will already take care of checkpointing the database for you using the "checkpoint" directive... --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5439) back-bdb race condition
by rein＠basefarm.no 31 Mar '08

31 Mar '08

On Tue, 25 Mar 2008, Howard Chu wrote: > Rein Tollevik wrote: >> On Tue, 25 Mar 2008, hyc(a)symas.com wrote: >> It now looks to me as if the entire rs->sr_entry was released an reused, >> and that the bug probably is in back-bdb. It just always happened when >> syncprov was running as this is a master server with mostly writes >> except from the reads syncprov does. Which also means that the title on >> the bug report is probably misleading :-( >> >> The reason for my shift of focus is some highly suspicious variables >> found in the bdb_search() frame. > > Look at the value of idflag in bdb_search. (see the comment around 672) > > Could be a locking bug with ID_NOCACHE in cache.c... Yes, it looks very much as this was the problem. We have been running with the patch at the end since thursday afternoon, and so far it seem to have worked :-) Although it is a bit too early to conclude, especially as we have seen other problems as well. We did see segmentation faults both friday and monday (not very much going on during the weekend..), but so far those appear to be related to ITS#5445. A slapd and db_checkpoint (run out of cron) deadlock which I hope is due to the BerkeleyDB bug mentioned in ITS#5391 has also occurred, so I'll upgrade from 4.6.18 to 4.6.21.1 to see if that problem goes away. Now, back to this bug. There is a potential bug in bdb_cache_find_id() as it inspects and modifies the EntryInfo without it being locked. Although I don't think this was the cause here, as bdb_search() always passes a NULL eip which mean that bdb_cache_find_ndn() should have acquired a lock. The patch always locks the entryinfo though, as I believe that to be necessary for the rest of the patch to be correct. The real problem was in bdb_cache_return_entry_rw() which released ei->bei_e without making sure it was the only thread using it. I.e it should have had a write lock on the entry_db (as bdb_cache_lru_purge() makes sure to have), but instead it freed it after releasing the read lock. Instead of write locking the entry before releasing it my alternative solution is to only let the thread that loads the entry set the CACHE_ENTRY_NOT_CACHED flag, and only if no other threads found the entryinfo while it loaded the entry. All other threads clears the flag when they find they don't need to load the entry. I also use trylock to lock the entryinfo before setting or inspecting the cache flag, for two reasons. First, honoring this flag should not be important enough to wait for a lock (after all, it should be cleared anyhow if anyone else is interested in the same entry). Second (and most important), to avoid any deadlock situations, which my first attempt to fix this bug managed to create when I moved the release of the entryinfo lock to after the entry_db lock had been acquired in bdb_cache_find_id(). Rein Index: OpenLDAP/servers/slapd/back-bdb/cache.c diff -u OpenLDAP/servers/slapd/back-bdb/cache.c:1.1.1.15 OpenLDAP/servers/slapd/back-bdb/cache.c:1.2 --- OpenLDAP/servers/slapd/back-bdb/cache.c:1.1.1.15 Sat Mar 22 16:47:56 2008 +++ OpenLDAP/servers/slapd/back-bdb/cache.c Mon Mar 31 17:52:17 2008 @@ -252,16 +252,27 @@ int free = 0; ei = e->e_private; - bdb_cache_entry_db_unlock( bdb, lock ); - if ( ei ) { - bdb_cache_entryinfo_lock( ei ); + if ( ei && + ( ei->bei_state & CACHE_ENTRY_NOT_CACHED ) && + ( bdb_cache_entryinfo_trylock( ei ) == 0 ) ) { if ( ei->bei_state & CACHE_ENTRY_NOT_CACHED ) { + /* Releasing the entry can only be done when + * we know that nobody else is using it, i.e we + * should have an entry_db writelock. But the + * flag is only set by the thread that loads the + * entry, and only if no other threads has found + * it while it was working. All other threads + * clear the flag, which mean that we should be + * the only thread using the entry if the flag + * is set here. + */ ei->bei_e = NULL; ei->bei_state ^= CACHE_ENTRY_NOT_CACHED; free = 1; } bdb_cache_entryinfo_unlock( ei ); } + bdb_cache_entry_db_unlock( bdb, lock ); if ( free ) { e->e_private = NULL; bdb_entry_return( e ); @@ -854,6 +865,11 @@ /* Ok, we found the info, do we have the entry? */ if ( rc == 0 ) { + if ( !( flag & ID_LOCKED )) { + bdb_cache_entryinfo_lock ( *eip ); + flag |= ID_LOCKED; + } + if ( (*eip)->bei_state & CACHE_ENTRY_DELETED ) { rc = DB_NOTFOUND; } else { @@ -873,13 +889,13 @@ (*eip)->bei_state |= CACHE_ENTRY_LOADING; } - /* If the entry was loaded before but uncached, and we need - * it again, clear the uncached state - */ - if ( (*eip)->bei_state & CACHE_ENTRY_NOT_CACHED ) { - (*eip)->bei_state ^= CACHE_ENTRY_NOT_CACHED; - if ( flag & ID_NOCACHE ) - flag ^= ID_NOCACHE; + if ( !load ) { + /* Clear the uncached state if we are not + * loading it, i.e it is already cached or + * another thread is currently loading it. + */ + (*eip)->bei_state &= ~CACHE_ENTRY_NOT_CACHED; + flag &= ~ID_NOCACHE; } if ( flag & ID_LOCKED ) { @@ -906,10 +922,14 @@ #endif ep = NULL; bdb_cache_lru_link( bdb, *eip ); - if ( flag & ID_NOCACHE ) { - bdb_cache_entryinfo_lock( *eip ); - (*eip)->bei_state |= CACHE_ENTRY_NOT_CACHED; - bdb_cache_entryinfo_unlock( *eip ); + if ( ( flag & ID_NOCACHE ) && + ( bdb_cache_entryinfo_trylock ( *eip ) == 0 ) ) { + /* Set the cached state only if no other thread + * found the info while we was loading the entry. + */ + if ( (*eip)->bei_finders == 1 ) + (*eip)->bei_state |= CACHE_ENTRY_NOT_CACHED; + bdb_cache_entryinfo_unlock ( *eip ); } } if ( rc == 0 ) {

1 0

Re: (ITS#5444) slapd seg. fault using mulitmirrored mode
by mohelonline＠googlemail.com 31 Mar '08

31 Mar '08

------=_Part_28055_26658090.1206987058866 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Thanks for the fast reply. I just added a few lines after the 20 seconds wait for the master replcaition: ---------------------------------- echo "Using ldapadd to populate consumer ..." $LDAPADD -D "$MANAGERDN" -H $URI2 -w $PASSWD -f $LDIFORDERED2 \ >> $TESTOUT 2>&1 SLEEP=20 echo "Waiting $SLEEP seconds for syncrepl to receive changes..." sleep $SLEEP ---------------------------------- These lines just add one additional ldapuser. After an additional 20seconds pause I check if the changes (form server with URI2) are replicated. But always the slapd prcoess dies after the ldapadd command. I'll give your above suggestion a try and give feedback afterwards. On Mon, Mar 31, 2008 at 6:04 PM, Pierangelo Masarati <ando(a)sys-net.it> wrote: > ando(a)sys-net.it wrote: > > mohel(a)web.de wrote: > > > >> The above log was generated using the test050 script just adding a line > which > >> enters information also to a second server. Perhaps this test would > also make > >> sense for further releases. > > > > Can you please post the modification to the script? > > Never mind, I could reproduce it. It seems to be a duplicate of > ITS#5437. What actually happens is that if modifications are applied to > all three servers, those applied to consumers do not persist, nor get > replicated; on the contrary, the consumers crash later on as described > in ITS#5437. > > p. > > > > Ing. Pierangelo Masarati > OpenLDAP Core Team > > SysNet s.r.l. > via Dossi, 8 - 27100 Pavia - ITALIA > http://www.sys-net.it > --------------------------------------- > Office: +39 02 23998309 > Mobile: +39 333 4963172 > Email: pierangelo.masarati(a)sys-net.it > --------------------------------------- > > > ------=_Part_28055_26658090.1206987058866 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Thanks for the fast reply.<br><br>I just added a few lines after the 20 seconds wait for the master replcaition:<br>------------------------------<div id="1eob" class="ArwC7c ckChnd">----<br>echo "Using ldapadd to populate consumer ..."<br> $LDAPADD -D "$MANAGERDN" -H $URI2 -w $PASSWD -f $LDIFORDERED2 \<br>         >> $TESTOUT 2>&1<br><br>SLEEP=20<br>echo "Waiting $SLEEP seconds for syncrepl to receive changes..."<br>sleep $SLEEP<br> ----------------------------------<br><br>These lines just add one additional ldapuser. After an additional 20seconds pause I check if the changes (form server with URI2) are replicated.<br><br>But always the slapd prcoess dies after the ldapadd command.<br> <br>I'll give your above suggestion a try and give feedback afterwards.</div><br><br><div class="gmail_quote">On Mon, Mar 31, 2008 at 6:04 PM, Pierangelo Masarati <<a href="mailto:ando@sys-net.it">ando(a)sys-net.it</a>> wrote:<br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="Ih2E3d"><a href="mailto:ando@sys-net.it">ando(a)sys-net.it</a> wrote:<br> > <a href="mailto:mohel@web.de">mohel(a)web.de</a> wrote:<br> ><br> >> The above log was generated using the test050 script just adding a line which<br> >> enters information also to a second server. Perhaps this test would also make<br> >> sense for further releases.<br> ><br> > Can you please post the modification to the script?<br> <br> </div>Never mind, I could reproduce it.  It seems to be a duplicate of<br> ITS#5437.  What actually happens is that if modifications are applied to<br> all three servers, those applied to consumers do not persist, nor get<br> replicated; on the contrary, the consumers crash later on as described<br> in ITS#5437.<br> <div><div></div><div class="Wj3C7c"><br> p.<br> <br> <br> <br> Ing. Pierangelo Masarati<br> OpenLDAP Core Team<br> <br> SysNet s.r.l.<br> via Dossi, 8 - 27100 Pavia - ITALIA<br> <a href="http://www.sys-net.it" target="_blank">http://www.sys-net.it</a><br> ---------------------------------------<br> Office:  +39 02 23998309<br> Mobile:  +39 333 4963172<br> Email:   <a href="mailto:pierangelo.masarati@sys-net.it">pierangelo.masarati(a)sys-net.it</a><br> ---------------------------------------<br> <br> <br> </div></div></blockquote></div><br> ------=_Part_28055_26658090.1206987058866--

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs