I've noticed a bit of a performance degradation between the 2.4.39 release and 2.4.44, when doing searches that require ACL processing (anonymous or authenticated user). To test, I set up my server with 2.4.39, freshly load the DB, reboot, and then do a query of each type to ensure its data set is fully loaded in memory, and then get timing on the subsequent searches. The server is idle other than my searches. Overall, there's about a 30 second consistent degradation in search results, regardless of whether or not ACLs are involved. It works out to about a 15% degredation for authenticated queries, and a 32% degredation for rootdn searches.
Results are as follows:
2.4.39 anonymous search (only returns some data, subject to ACL processing):
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com
/dev/null
real 3m25.871s user 1m43.872s sys 0m45.254s
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com
/dev/null
real 3m22.730s user 1m39.575s sys 0m26.400s
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com
/dev/null
real 3m23.506s user 1m41.919s sys 0m25.968s
2.4.44 anonymous search (only returns some data, subject to ACL processing):
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com
/dev/null
real 3m54.841s user 1m36.754s sys 0m44.509s zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com
/dev/null
real 3m56.609s user 1m37.875s sys 0m48.151s zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com
/dev/null
real 3m53.275s user 1m36.293s sys 0m45.131s
2.4.39 authenticated search (returns all data, subject to ACL processing):
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D uid=zimbra,cn=admins,cn=zimbra -w zimbra >/dev/null
real 3m23.057s user 1m59.358s sys 0m35.809s zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D uid=zimbra,cn=admins,cn=zimbra -w zimbra >/dev/null
real 3m23.478s user 2m0.236s sys 0m26.972s zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D uid=zimbra,cn=admins,cn=zimbra -w zimbra >/dev/null
real 3m22.965s user 1m58.245s sys 0m28.030s
2.4.44 authenticated search (returns all data, subject to ACL processing):
zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D uid=zimbra,cn=admins,cn=zimbra -w zimbra >/dev/null
real 3m57.967s user 1m56.335s sys 0m49.483s zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D uid=zimbra,cn=admins,cn=zimbra -w zimbra >/dev/null
real 3m57.405s user 1m55.415s sys 0m50.772s zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D uid=zimbra,cn=admins,cn=zimbra -w zimbra >/dev/null
real 3m57.717s user 1m57.427s sys 0m41.401s
2.4.39 rootdn search (no ACL processing): zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D cn=config -w zimbra >/dev/null
real 1m20.718s user 1m14.501s sys 0m6.164s zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D cn=config -w zimbra >/dev/null
real 1m19.569s user 1m13.557s sys 0m5.935s zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D cn=config -w zimbra >/dev/null
real 1m20.488s user 1m14.514s sys 0m5.917s
2.4.44 rootdn search (no ACL processing): zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D cn=config -w zimbra >/dev/null
real 1m57.199s user 1m10.855s sys 0m14.161s zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D cn=config -w zimbra >/dev/null
real 1m58.030s user 1m10.797s sys 0m14.970s zimbra@zre-ldap001:~$ time ldapsearch -x -h zre-ldap001.eng.zimbra.com -D cn=config -w zimbra >/dev/null
real 1m57.754s user 1m10.069s sys 0m15.336s
--Quanah
--
Quanah Gibson-Mount Platform Architect, Manager Release Engineering Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
--On Monday, February 29, 2016 2:11 PM -0800 Quanah Gibson-Mount quanah@zimbra.com wrote:
I've noticed a bit of a performance degradation between the 2.4.39 release and 2.4.44, when doing searches that require ACL processing (anonymous or authenticated user).
That should be, when doing searches, period. ;)
--Quanah
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
--On Monday, February 29, 2016 2:20 PM -0800 Quanah Gibson-Mount quanah@zimbra.com wrote:
--On Monday, February 29, 2016 2:11 PM -0800 Quanah Gibson-Mount quanah@zimbra.com wrote:
I've noticed a bit of a performance degradation between the 2.4.39 release and 2.4.44, when doing searches that require ACL processing (anonymous or authenticated user).
That should be, when doing searches, period. ;)
This was introduced in 2.4.44 with the fixes for ITS8266. The new olcDbRtxnSize attribute defaults to 10000. However, the more extreme the ratio between the batch size and the db size, the more perf is significantly impacted. Dropping down to an approximate 300,000 DB, we see the following.
0: 9.145s, 9.450s, 9.272s, 9.129s (no batch size) 100: 1m31.463s, 1m31.850s, 1m30.230s, 1m29.850s (3000:1) 1000: 17.910s, 17.778s, 17.784s, 17.556s (300:1) 10000: 10.067s, 9.947s, 9.919s, 9.960s (30:1) 100000: 9.437s, 9.324s, 9.272s, 9.346s (3:1)
So apparently, you want the ration to be no more than 30:1, otherwise perfomance degrades substantially.
--Quanah
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
Quanah Gibson-Mount wrote:
--On Monday, February 29, 2016 2:20 PM -0800 Quanah Gibson-Mount quanah@zimbra.com wrote:
--On Monday, February 29, 2016 2:11 PM -0800 Quanah Gibson-Mount quanah@zimbra.com wrote:
I've noticed a bit of a performance degradation between the 2.4.39 release and 2.4.44, when doing searches that require ACL processing (anonymous or authenticated user).
That should be, when doing searches, period. ;)
This was introduced in 2.4.44 with the fixes for ITS8266. The new olcDbRtxnSize attribute defaults to 10000. However, the more extreme the ratio between the batch size and the db size, the more perf is significantly impacted. Dropping down to an approximate 300,000 DB, we see the following.
0: 9.145s, 9.450s, 9.272s, 9.129s (no batch size) 100: 1m31.463s, 1m31.850s, 1m30.230s, 1m29.850s (3000:1) 1000: 17.910s, 17.778s, 17.784s, 17.556s (300:1) 10000: 10.067s, 9.947s, 9.919s, 9.960s (30:1) 100000: 9.437s, 9.324s, 9.272s, 9.346s (3:1)
So apparently, you want the ration to be no more than 30:1, otherwise perfomance degrades substantially.
On a 1M entry database I see no performance difference between using the default rtxnsize (10000) and setting it to zero. Verified using both head and RE24.
--On Saturday, March 12, 2016 10:32 AM +0000 Howard Chu hyc@symas.com wrote:
So apparently, you want the ration to be no more than 30:1, otherwise perfomance degrades substantially.
On a 1M entry database I see no performance difference between using the default rtxnsize (10000) and setting it to zero. Verified using both head and RE24.
I see the differences I've noted with at least 3 different DBs of varying size now. I've dropped the 300kish one on ada.
--Quanah
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
Quanah Gibson-Mount wrote:
--On Saturday, March 12, 2016 10:32 AM +0000 Howard Chu hyc@symas.com wrote:
So apparently, you want the ration to be no more than 30:1, otherwise perfomance degrades substantially.
On a 1M entry database I see no performance difference between using the default rtxnsize (10000) and setting it to zero. Verified using both head and RE24.
I see the differences I've noted with at least 3 different DBs of varying size now. I've dropped the 300kish one on ada.
With your 300k entry DB and config I see a full subtree search take about 10.4s with rtxnsize = 0, vs 11.7s with rtxnsize = 10000. Measurable certainly, but not really noticeable.
--On Tuesday, March 15, 2016 1:31 PM +0000 Howard Chu hyc@symas.com wrote:
I see the differences I've noted with at least 3 different DBs of varying size now. I've dropped the 300kish one on ada.
With your 300k entry DB and config I see a full subtree search take about 10.4s with rtxnsize = 0, vs 11.7s with rtxnsize = 10000. Measurable certainly, but not really noticeable.
As I noted, a 30:1 ratio seems to work ok. Now, if you go back to my original mail, the problem is the default of 10k is clearly not sufficient for any medium+ sized db. I think a mistake was made in setting the default in 2.4.44 to anything other than zero. This is clearly an item that needs tuning PER database, and should not have a default that creates a severe performance penalty on databases > 300000 entries. As you can see from my other emails, the further you get from the 30:1 ratio, the more significant the penalty in performance. It would be interesting to know if there are further ways to optimize the code so that the penalty is not so severe. I.e., the next step would be to set the rtxnsize to 1000 with the 300k database and see if there are further optimizations to be had.
--Quanah
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
-
Howard Chu -
Quanah Gibson-Mount