2:11 p.m.
I've noticed a bit of a performance degradation between the 2.4.39 release
and 2.4.44, when doing searches that require ACL processing (anonymous or
authenticated user). To test, I set up my server with 2.4.39, freshly load
the DB, reboot, and then do a query of each type to ensure its data set is
fully loaded in memory, and then get timing on the subsequent searches.
The server is idle other than my searches. Overall, there's about a 30
second consistent degradation in search results, regardless of whether or
not ACLs are involved. It works out to about a 15% degredation for
authenticated queries, and a 32% degredation for rootdn searches.
Results are as follows:
2.4.39 anonymous search (only returns some data, subject to ACL processing):
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com
/dev/null
real 3m25.871s
user 1m43.872s
sys 0m45.254s
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com
/dev/null
real 3m22.730s
user 1m39.575s
sys 0m26.400s
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com
/dev/null
real 3m23.506s
user 1m41.919s
sys 0m25.968s
2.4.44 anonymous search (only returns some data, subject to ACL processing):
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com
/dev/null
real 3m54.841s
user 1m36.754s
sys 0m44.509s
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com
/dev/null
real 3m56.609s
user 1m37.875s
sys 0m48.151s
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com
/dev/null
real 3m53.275s
user 1m36.293s
sys 0m45.131s
2.4.39 authenticated search (returns all data, subject to ACL processing):
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D
uid=zimbra,cn=admins,cn=zimbra -w zimbra /dev/null
real 3m23.057s
user 1m59.358s
sys 0m35.809s
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D
uid=zimbra,cn=admins,cn=zimbra -w zimbra /dev/null
real 3m23.478s
user 2m0.236s
sys 0m26.972s
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D
uid=zimbra,cn=admins,cn=zimbra -w zimbra /dev/null
real 3m22.965s
user 1m58.245s
sys 0m28.030s
2.4.44 authenticated search (returns all data, subject to ACL processing):
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D
uid=zimbra,cn=admins,cn=zimbra -w zimbra /dev/null
real 3m57.967s
user 1m56.335s
sys 0m49.483s
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D
uid=zimbra,cn=admins,cn=zimbra -w zimbra /dev/null
real 3m57.405s
user 1m55.415s
sys 0m50.772s
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D
uid=zimbra,cn=admins,cn=zimbra -w zimbra /dev/null
real 3m57.717s
user 1m57.427s
sys 0m41.401s
2.4.39 rootdn search (no ACL processing):
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D
cn=config -w zimbra /dev/null
real 1m20.718s
user 1m14.501s
sys 0m6.164s
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D
cn=config -w zimbra /dev/null
real 1m19.569s
user 1m13.557s
sys 0m5.935s
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D
cn=config -w zimbra /dev/null
real 1m20.488s
user 1m14.514s
sys 0m5.917s
2.4.44 rootdn search (no ACL processing):
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D
cn=config -w zimbra /dev/null
real 1m57.199s
user 1m10.855s
sys 0m14.161s
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D
cn=config -w zimbra /dev/null
real 1m58.030s
user 1m10.797s
sys 0m14.970s
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D
cn=config -w zimbra /dev/null
real 1m57.754s
user 1m10.069s
sys 0m15.336s
--Quanah
--
Quanah Gibson-Mount
Platform Architect, Manager Release Engineering
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
A division of Synacor, Inc
2:20 p.m.
--On Monday, February 29, 2016 2:11 PM -0800 Quanah Gibson-Mount
<quanah(a)zimbra.com> wrote:
I've noticed a bit of a performance degradation between the
2.4.39
release and 2.4.44, when doing searches that require ACL processing
(anonymous or authenticated user).
That should be, when doing searches, period. ;)
--Quanah
--
Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
A division of Synacor, Inc
2:11 p.m.
--On Monday, February 29, 2016 2:20 PM -0800 Quanah Gibson-Mount
<quanah(a)zimbra.com> wrote:
--On Monday, February 29, 2016 2:11 PM -0800 Quanah Gibson-Mount
<quanah(a)zimbra.com> wrote:
> I've noticed a bit of a performance degradation between the 2.4.39
> release and 2.4.44, when doing searches that require ACL processing
> (anonymous or authenticated user).
That should be, when doing searches, period. ;)
This was introduced in 2.4.44 with the fixes for ITS8266. The new
olcDbRtxnSize attribute defaults to 10000. However, the more extreme the
ratio between the batch size and the db size, the more perf is
significantly impacted. Dropping down to an approximate 300,000 DB, we see
the following.
0: 9.145s, 9.450s, 9.272s, 9.129s (no batch size)
100: 1m31.463s, 1m31.850s, 1m30.230s, 1m29.850s (3000:1)
1000: 17.910s, 17.778s, 17.784s, 17.556s (300:1)
10000: 10.067s, 9.947s, 9.919s, 9.960s (30:1)
100000: 9.437s, 9.324s, 9.272s, 9.346s (3:1)
So apparently, you want the ration to be no more than 30:1, otherwise
perfomance degrades substantially.
--Quanah
--
Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
A division of Synacor, Inc
2:32 a.m.
Quanah Gibson-Mount wrote:
--On Monday, February 29, 2016 2:20 PM -0800 Quanah Gibson-Mount
<quanah(a)zimbra.com> wrote:
> --On Monday, February 29, 2016 2:11 PM -0800 Quanah Gibson-Mount
> <quanah(a)zimbra.com> wrote:
>
>> I've noticed a bit of a performance degradation between the 2.4.39
>> release and 2.4.44, when doing searches that require ACL processing
>> (anonymous or authenticated user).
>
> That should be, when doing searches, period. ;)
This was introduced in 2.4.44 with the fixes for ITS8266. The new
olcDbRtxnSize attribute defaults to 10000. However, the more extreme the
ratio between the batch size and the db size, the more perf is significantly
impacted. Dropping down to an approximate 300,000 DB, we see the following.
0: 9.145s, 9.450s, 9.272s, 9.129s (no batch size)
100: 1m31.463s, 1m31.850s, 1m30.230s, 1m29.850s (3000:1)
1000: 17.910s, 17.778s, 17.784s, 17.556s (300:1)
10000: 10.067s, 9.947s, 9.919s, 9.960s (30:1)
100000: 9.437s, 9.324s, 9.272s, 9.346s (3:1)
So apparently, you want the ration to be no more than 30:1, otherwise
perfomance degrades substantially.
On a 1M entry database I see no performance difference between using the
default rtxnsize (10000) and setting it to zero. Verified using both head and
RE24.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
3:02 p.m.
--On Saturday, March 12, 2016 10:32 AM +0000 Howard Chu <hyc(a)symas.com>
wrote:
> So apparently, you want the ration to be no more than 30:1,
otherwise
> perfomance degrades substantially.
On a 1M entry database I see no performance difference between using the
default rtxnsize (10000) and setting it to zero. Verified using both head
and RE24.
I see the differences I've noted with at least 3 different DBs of varying
size now. I've dropped the 300kish one on ada.
--Quanah
--
Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
A division of Synacor, Inc
5:31 a.m.
Quanah Gibson-Mount wrote:
--On Saturday, March 12, 2016 10:32 AM +0000 Howard Chu
<hyc(a)symas.com> wrote:
>> So apparently, you want the ration to be no more than 30:1, otherwise
>> perfomance degrades substantially.
>
> On a 1M entry database I see no performance difference between using the
> default rtxnsize (10000) and setting it to zero. Verified using both head
> and RE24.
I see the differences I've noted with at least 3 different DBs of varying size
now. I've dropped the 300kish one on ada.
With your 300k entry DB and config I see a full subtree search take about
10.4s with rtxnsize = 0, vs 11.7s with rtxnsize = 10000. Measurable certainly,
but not really noticeable.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
9:15 a.m.
--On Tuesday, March 15, 2016 1:31 PM +0000 Howard Chu <hyc(a)symas.com> wrote:
> I see the differences I've noted with at least 3 different
DBs of
> varying size now. I've dropped the 300kish one on ada.
With your 300k entry DB and config I see a full subtree search take about
10.4s with rtxnsize = 0, vs 11.7s with rtxnsize = 10000. Measurable
certainly, but not really noticeable.
As I noted, a 30:1 ratio seems to work ok. Now, if you go back to my
original mail, the problem is the default of 10k is clearly not sufficient
for any medium+ sized db. I think a mistake was made in setting the
default in 2.4.44 to anything other than zero. This is clearly an item
that needs tuning PER database, and should not have a default that creates
a severe performance penalty on databases > 300000 entries. As you can see
from my other emails, the further you get from the 30:1 ratio, the more
significant the penalty in performance. It would be interesting to know if
there are further ways to optimize the code so that the penalty is not so
severe. I.e., the next step would be to set the rtxnsize to 1000 with the
300k database and see if there are further optimizations to be had.
--Quanah
--
Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
A division of Synacor, Inc
2637
days inactive
2652
days old
6 comments
2 participants
participants (2)
-
Howard Chu -
Quanah Gibson-Mount