openldap-devel August 2009

openldap-devel@openldap.org

17 participants
37 discussions

OL 2.3 vs OL 2.4 scalability issues
by Quanah Gibson-Mount 07 Aug '09

07 Aug '09

In line with checking the perf issues between 2.3 and 2.4, I've started looking at how well each release scales. The previous benchmarks were taken from saturating the server with 48 slamd clients starting with 1 thread, and then increasing the number of threads per client across each iteration. I.e., iteration 1 has 48 connections doing queries. Iteration 2 has 96 connections doing queries, etc. For the scalability test, I switched to 9 slamd clients, incrementing 1 thread per iteration. What I see as a result is that 2.3 scales better, and keeps substantially more consistent results as the number of clients increases, whereas 2.4 gets consistently worse as the number of clients increases. I.e., 2.4 does not scale particularly well. The test ends after 3 consecutive non-improving increments. OL 2.3: Job Iterations 20090805192650-64030272-1 14028.304 Stopped because the maximum duration had been reached 20090805192650-64030272-2 18210.740 Stopped because the maximum duration had been reached 20090805192650-64030272-3 19405.395 Stopped because the maximum duration had been reached 20090805192650-64030272-4 20729.611 Stopped because the maximum duration had been reached 20090805192650-64030272-5 21265.811 Stopped because the maximum duration had been reached 20090805192650-64030272-6 21481.113 Stopped because the maximum duration had been reached 20090805192650-64030272-7 21621.295 Stopped because the maximum duration had been reached 20090805192650-64030272-8 21670.312 Stopped because the maximum duration had been reached 20090805192650-64030272-9 21665.673 Stopped because the maximum duration had been reached 20090805192650-64030272-10 21627.581 Stopped because the maximum duration had been reached 20090805192650-64030272-11 21581.896 Stopped because the maximum duration had been reached OL 2.4: Job Iterations 20090805154205-07048070-1 13382.487 Stopped because the maximum duration had been reached 20090805154205-07048070-2 18201.523 Stopped because the maximum duration had been reached 20090805154205-07048070-3 17137.449 Stopped because the maximum duration had been reached 20090805154205-07048070-4 16470.136 Stopped because the maximum duration had been reached 20090805154205-07048070-5 16127.879 Stopped because the maximum duration had been reached And again, it is painfully clear how much slower OL 2.4 is than OL 2.3 as the number of clients increases. OL 2.4 maxes out at 18 querying clients. OL 2.3 maxed out at 81 querying clients, and although it started to decline after that point, the decline was very gradual compared to OL 2.4. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

2 3

Re: slapd/acl.c: using "filter=()" in an acl does not trigger an overlay's search callback
by masarati＠aero.polimi.it 06 Aug '09

06 Aug '09

> Daniel schrieb: >> masarati(a)aero.polimi.it schrieb: >>>> What kind of alternative solutions exist or do you see? >>> >>> I'd rather define a new ordering matching rule for time-valued attrs >>> that >>> checks how a given value compares with "now"; something like >>> "greaterThanNow" and "lessThanNow". This would allow to do something >>> like >>> >>> access to attrs=validnow val/greaterThanNow=19700101000000Z >>> >>> or >>> >>> access to filter=(validnow:greaterThanNow:=19700101000000Z) >>> >>> where 19700101000000Z (the epoch) is used as a placeholder for the >>> asserted value, which would be ignored. > After some experiments with MRs I think that I have not understood your > suggestion the right way or your suggestion is misleading... > > In case the two new MRs "earlierThanNow" and "laterThanNow" are new > ORDERING MRs they have to be compared using lt- or gt-operators, haven't > they? No, you need to use an extensible filter, which is sort of a generalized assertion. What you use is "attr:mr:=value". Then, it's the purpose of "mr", your custom matching rule, to perform something that is equivalent to an ordering match. Your attributes do not need to have anything special about matching rules. They could even have no matching rule defined. All they need is to have "generalizedTime" syntax, so that your custom matching rule can be applied. > This leads me to the question why are there two MRs needed if the > operator could be used to tell the difference... (e.g. earlierThanNow: > "<" and laterThanNow: ">")? Because you need two separate assertions for earlierThanNow and laterThanNow. > Another question concerns the schema-definition of the two dead-end > attributes "validNotBefore" and "validNotAfter": In my current > definition both are of type "generalizedTime", EQUALITY is > "generalizedTimeMatch" but what should I use for ORDERING? See above. Nothing is needed about equality. > From the results of my MR-tests I think I should/could: > 1. define ORDERING using "generalizedTimeOrderingMatch" (to make > standard LE/GE ldapsearch filtering using "<=" and ">=" work out of the > box) That makes sense, although it would be orthogonal to using the earlierThanNow/laterThanNow rules through extensible filters. > 2. use the two additional MRs on-demand (replacing the above "default" > ORDERING by the new ones), e.g.: > (validNotBefore:earlierThanNow:<epoch) or/and > (validNotBefore:earlierThanNow:>epoch) The above string representations do not conform to LDAP search filters. See RFC 4515 for a definition of extensible match. > Would this be possible or ok? What alternatives do you possibly see or > suggest? Stick with the original plan, implement it correctly. p.

2 1

slapd/acl.c: using "filter=()" in an acl does not trigger an overlay's search callback
by Daniel 05 Aug '09

05 Aug '09

Hi, my experimental overlay intercepts search requests and modifies the filter-expressions (e.g. removes/expand AND|OR). Therefore the overlay modifies "struct Filter op->ors_filter" on the fly, handles memory (op->o_tmpmemctx/filter_free) and so on. It works fine, currently no memleaks (valgrind) etc and I would like contribute it (once it's finished... ;-) ) To get the maximum flexibility out of the current prototyp I'm looking after a solution to have my overlay intercept the following "search", too: access to filter=(description=editMe) by self write by * none The ACL's filter seems not to "fire" a search request because in acl.c lines 704 to 709 "test_filter(NULL, e, a->acl_filter)" get's called directly. This results in bypassing the overlay's search callback. Was this for performance reasons? Are there any chances to replace "test_filter()" by some kind of slapd_internal_search() in a way to enable overlay stack processing, for this internal search? Any sort of advice is very welcome - I just want to experiment with this, but currently I'm lost, regarding the right approach. I think I've seen some kind of internal search regarding sasl-regexp expansion. When using a filter-expressions in a sasl-regexp some sort of internal search gets triggered, because my overlay reacts correctly on it: slapd/saslauthz.c lines 1805 - 1835. Possibly something similar could be done in acl.c, too? What's your opinion? Thanks a lot! Daniel P.S.: slapd version: OpenLDAP-CVS

3 5

OL2.3 vs OL2.4 perf issues
by Quanah Gibson-Mount 05 Aug '09

05 Aug '09

I've been spending some time perf testing OL 2.4 in relation to OL 2.3. Unfortunately, RE24 is noticeably slower than 2.3 was. Results of simple auth testing with slamd show: OL 2.3: 21,745 auths/second OL 2.4: 15,733 auths/second So OL 2.4 is 6,000 auths/second (aka 12,000 searches/second) slower than 2.3. I.e., 27% slower. Howard committed a patch that slightly helps some situations, and Hallvard has a rewrite of part of the lber library that I've been testing that he'll commit soon. That helps somewhat: OL 2.4 with howard and hallvard's patches: 17,086 auths/second. That still leaves us over 4,500 auths/second (or 9000 searches/second) slower than RE2.3. I.e., 21.5% slower. Which is quite a substantial gap. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

5 12

ber_ptrlen() should return ber_len_t
by masarati＠aero.polimi.it 04 Aug '09

04 Aug '09

In fact, it makes no sense that ber_ptr < ber_buf; if it happens, the BerElement is corrupted, and an assertion failure may be more appropriate. Right now, no test is done, and the (trivial) function could be replaced with a macro... Since this is part of the private API, its change should be harmless. Also, it should be renamed ber_int_ptrlen(). p.

2 2

certificate list parsing issues
by masarati＠aero.polimi.it 04 Aug '09

04 Aug '09

I recently hit a pretty long certificate list with what appears to be crap past the end of its valid portion. I have no indication about how this was generated, but it is supposed to be in production within a CA, initially using a release of OpenLDAP without detailed CL validation in place (remember this was released in 2.4). I'm not posting this to the ITS because it's data I'm not allowed to disclose. To make a long story short, I got the CL in LDIF format; I could convert it to DER and have openssl crl play with it. Apparently, openssl crl recognizes it and deals with its contents correctly, but our CL validator fails because when it expects to be at the end there is still stuff to be parsed (some 40KB of what appears to be garbage). Howard found a small issue in CL validation and fixed it (schema_init.c 1.459 -> 1.460), but nevertheless the issue remains. Howard also discovered that regenerating the CL in DER form using openssl clr would yield a shorter certificate that passes OpenLDAP's validator. I'm raising it here because we need to understand how important it is for us to be able to deal with broken CL, and how broken we can accept them to be. In this case, the CL looks fine until the end, with garbage at the end. This could be tolerated. Or, we could just ignore any type of error, as soon as we don't need to deal with its contents (slapd is merely acting as a container, and needs not know whether it's containing good or bad data). This latter argument may be not valid as soon as our slapd takes over as much certificate handling as possible, performing certificate validation internally rather than delegating it to some external package (I understand Howard would probably like to follow that path, eventually). Unless there is strong opposition, I'd relax the last check about being at the end of the CL, in order to accept CL with this type of brokenness, possibly logging about the issue. p.

4 6

Re: commit: ldap/servers/slapd/overlays pcache.c
by Howard Chu 03 Aug '09

03 Aug '09

ando(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/servers/slapd/overlays > > Modified Files: > pcache.c 1.172 -> 1.173 > > Log Message: > count query answerability; allow configuration of private database We really need to provide a better way to configure private databases; be_config() will be obsolete (along with slapd.conf). -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel August 2009