OL 2.3 vs OL 2.4 scalability issues
by Quanah Gibson-Mount
In line with checking the perf issues between 2.3 and 2.4, I've started
looking at how well each release scales. The previous benchmarks were
taken from saturating the server with 48 slamd clients starting with 1
thread, and then increasing the number of threads per client across each
iteration. I.e., iteration 1 has 48 connections doing queries. Iteration
2 has 96 connections doing queries, etc.
For the scalability test, I switched to 9 slamd clients, incrementing 1
thread per iteration. What I see as a result is that 2.3 scales better,
and keeps substantially more consistent results as the number of clients
increases, whereas 2.4 gets consistently worse as the number of clients
increases. I.e., 2.4 does not scale particularly well. The test ends
after 3 consecutive non-improving increments.
OL 2.3:
Job Iterations
20090805192650-64030272-1 14028.304 Stopped because the maximum
duration had been reached
20090805192650-64030272-2 18210.740 Stopped because the maximum
duration had been reached
20090805192650-64030272-3 19405.395 Stopped because the maximum
duration had been reached
20090805192650-64030272-4 20729.611 Stopped because the maximum
duration had been reached
20090805192650-64030272-5 21265.811 Stopped because the maximum
duration had been reached
20090805192650-64030272-6 21481.113 Stopped because the maximum
duration had been reached
20090805192650-64030272-7 21621.295 Stopped because the maximum
duration had been reached
20090805192650-64030272-8 21670.312 Stopped because the maximum
duration had been reached
20090805192650-64030272-9 21665.673 Stopped because the maximum
duration had been reached
20090805192650-64030272-10 21627.581 Stopped because the maximum
duration had been reached
20090805192650-64030272-11 21581.896 Stopped because the maximum
duration had been reached
OL 2.4:
Job Iterations
20090805154205-07048070-1 13382.487 Stopped because the maximum
duration had been reached
20090805154205-07048070-2 18201.523 Stopped because the maximum
duration had been reached
20090805154205-07048070-3 17137.449 Stopped because the maximum
duration had been reached
20090805154205-07048070-4 16470.136 Stopped because the maximum
duration had been reached
20090805154205-07048070-5 16127.879 Stopped because the maximum
duration had been reached
And again, it is painfully clear how much slower OL 2.4 is than OL 2.3 as
the number of clients increases. OL 2.4 maxes out at 18 querying clients.
OL 2.3 maxed out at 81 querying clients, and although it started to decline
after that point, the decline was very gradual compared to OL 2.4.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
13 years, 7 months
Re: slapd/acl.c: using "filter=()" in an acl does not trigger an overlay's search callback
by masarati＠aero.polimi.it
> Daniel schrieb:
>> masarati(a)aero.polimi.it schrieb:
>>>> What kind of alternative solutions exist or do you see?
>>>
>>> I'd rather define a new ordering matching rule for time-valued attrs
>>> that
>>> checks how a given value compares with "now"; something like
>>> "greaterThanNow" and "lessThanNow". This would allow to do something
>>> like
>>>
>>> access to attrs=validnow val/greaterThanNow=19700101000000Z
>>>
>>> or
>>>
>>> access to filter=(validnow:greaterThanNow:=19700101000000Z)
>>>
>>> where 19700101000000Z (the epoch) is used as a placeholder for the
>>> asserted value, which would be ignored.
> After some experiments with MRs I think that I have not understood your
> suggestion the right way or your suggestion is misleading...
>
> In case the two new MRs "earlierThanNow" and "laterThanNow" are new
> ORDERING MRs they have to be compared using lt- or gt-operators, haven't
> they?
No, you need to use an extensible filter, which is sort of a generalized
assertion. What you use is "attr:mr:=value". Then, it's the purpose of
"mr", your custom matching rule, to perform something that is equivalent
to an ordering match. Your attributes do not need to have anything
special about matching rules. They could even have no matching rule
defined. All they need is to have "generalizedTime" syntax, so that your
custom matching rule can be applied.
> This leads me to the question why are there two MRs needed if the
> operator could be used to tell the difference... (e.g. earlierThanNow:
> "<" and laterThanNow: ">")?
Because you need two separate assertions for earlierThanNow and laterThanNow.
> Another question concerns the schema-definition of the two dead-end
> attributes "validNotBefore" and "validNotAfter": In my current
> definition both are of type "generalizedTime", EQUALITY is
> "generalizedTimeMatch" but what should I use for ORDERING?
See above. Nothing is needed about equality.
> From the results of my MR-tests I think I should/could:
> 1. define ORDERING using "generalizedTimeOrderingMatch" (to make
> standard LE/GE ldapsearch filtering using "<=" and ">=" work out of the
> box)
That makes sense, although it would be orthogonal to using the
earlierThanNow/laterThanNow rules through extensible filters.
> 2. use the two additional MRs on-demand (replacing the above "default"
> ORDERING by the new ones), e.g.:
> (validNotBefore:earlierThanNow:<epoch) or/and
> (validNotBefore:earlierThanNow:>epoch)
The above string representations do not conform to LDAP search filters.
See RFC 4515 for a definition of extensible match.
> Would this be possible or ok? What alternatives do you possibly see or
> suggest?
Stick with the original plan, implement it correctly.
p.
13 years, 7 months
slapd/acl.c: using "filter=()" in an acl does not trigger an overlay's search callback
by Daniel
Hi,
my experimental overlay intercepts search requests and modifies the
filter-expressions (e.g. removes/expand AND|OR). Therefore the overlay
modifies "struct Filter op->ors_filter" on the fly, handles memory
(op->o_tmpmemctx/filter_free) and so on. It works fine, currently no
memleaks (valgrind) etc and I would like contribute it (once it's
finished... ;-) )
To get the maximum flexibility out of the current prototyp I'm looking
after a solution to have my overlay intercept the following "search", too:
access to filter=(description=editMe)
by self write
by * none
The ACL's filter seems not to "fire" a search request because in acl.c
lines 704 to 709 "test_filter(NULL, e, a->acl_filter)" get's called
directly. This results in bypassing the overlay's search callback. Was
this for performance reasons?
Are there any chances to replace "test_filter()" by some kind of
slapd_internal_search() in a way to enable overlay stack processing, for
this internal search? Any sort of advice is very welcome - I just want
to experiment with this, but currently I'm lost, regarding the right
approach.
I think I've seen some kind of internal search regarding sasl-regexp
expansion. When using a filter-expressions in a sasl-regexp some sort of
internal search gets triggered, because my overlay reacts correctly on
it: slapd/saslauthz.c lines 1805 - 1835. Possibly something similar
could be done in acl.c, too? What's your opinion?
Thanks a lot!
Daniel
P.S.: slapd version: OpenLDAP-CVS
13 years, 7 months
OL2.3 vs OL2.4 perf issues
by Quanah Gibson-Mount
I've been spending some time perf testing OL 2.4 in relation to OL 2.3.
Unfortunately, RE24 is noticeably slower than 2.3 was. Results of simple
auth testing with slamd show:
OL 2.3: 21,745 auths/second
OL 2.4: 15,733 auths/second
So OL 2.4 is 6,000 auths/second (aka 12,000 searches/second) slower than
2.3. I.e., 27% slower.
Howard committed a patch that slightly helps some situations, and Hallvard
has a rewrite of part of the lber library that I've been testing that he'll
commit soon. That helps somewhat:
OL 2.4 with howard and hallvard's patches: 17,086 auths/second.
That still leaves us over 4,500 auths/second (or 9000 searches/second)
slower than RE2.3. I.e., 21.5% slower. Which is quite a substantial gap.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
13 years, 7 months
ber_ptrlen() should return ber_len_t
by masarati＠aero.polimi.it
In fact, it makes no sense that ber_ptr < ber_buf; if it happens, the
BerElement is corrupted, and an assertion failure may be more appropriate.
Right now, no test is done, and the (trivial) function could be replaced
with a macro... Since this is part of the private API, its change should
be harmless. Also, it should be renamed ber_int_ptrlen().
p.
13 years, 7 months
certificate list parsing issues
by masarati＠aero.polimi.it
I recently hit a pretty long certificate list with what appears to be crap
past the end of its valid portion. I have no indication about how this
was generated, but it is supposed to be in production within a CA,
initially using a release of OpenLDAP without detailed CL validation in
place (remember this was released in 2.4). I'm not posting this to the
ITS because it's data I'm not allowed to disclose.
To make a long story short, I got the CL in LDIF format; I could convert
it to DER and have openssl crl play with it. Apparently, openssl crl
recognizes it and deals with its contents correctly, but our CL validator
fails because when it expects to be at the end there is still stuff to be
parsed (some 40KB of what appears to be garbage). Howard found a small
issue in CL validation and fixed it (schema_init.c 1.459 -> 1.460), but
nevertheless the issue remains. Howard also discovered that regenerating
the CL in DER form using openssl clr would yield a shorter certificate
that passes OpenLDAP's validator.
I'm raising it here because we need to understand how important it is for
us to be able to deal with broken CL, and how broken we can accept them to
be. In this case, the CL looks fine until the end, with garbage at the
end. This could be tolerated. Or, we could just ignore any type of
error, as soon as we don't need to deal with its contents (slapd is merely
acting as a container, and needs not know whether it's containing good or
bad data). This latter argument may be not valid as soon as our slapd
takes over as much certificate handling as possible, performing
certificate validation internally rather than delegating it to some
external package (I understand Howard would probably like to follow that
path, eventually).
Unless there is strong opposition, I'd relax the last check about being at
the end of the CL, in order to accept CL with this type of brokenness,
possibly logging about the issue.
p.
13 years, 7 months
