On 27/05/16 07:37, "Howard Chu" hyc@symas.com wrote:
Jürg Bircher wrote:
Hello
Improved handling for large number of databases
If interested let me know how to contribute.
Looks interesting, yes. I assume you have profiled the code before and after the suggested changes, please provide your profiling results.
Please read the Developer Guidelines. http://www.openldap.org/devel/contributing.html
Access newly opened database from another transaction
Sounds like an oddball case. Applications should open all their DBIs from a single thread and not start any other threads/transactions until all setup is completed.
Yes it is a simpler way to open all the databases at startup. However if the environment contains many databases which are not necessarily opened during the life time of the application it is an advantage to open them lazily. Any additional open database generates more overhead especially when using begin and abort.
lmdb improved (renew, reset)
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10] open databases in [0.31975] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10 of 10] open databases in [0.20350] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 100] open databases in [0.25845] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [100 of 100] open databases in [0.29663] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 1000] open databases in [0.28590] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1000 of 1000] open databases in [0.42897] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10000] open databases in [0.30004] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10000 of 10000] open databases in [1.68870] seconds
lmdb improved (begin, commit)
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10] open databases in [0.36538] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10 of 10] open databases in [0.35923] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 100] open databases in [0.34858] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [100 of 100] open databases in [0.39294] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 1000] open databases in [0.40222] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1000 of 1000] open databases in [0.54752] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10000] open databases in [0.78595] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10000 of 10000] open databases in [2.32414] seconds
lmdb original (renew, reset)
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10] open databases in [0.18597] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10 of 10] open databases in [0.21572] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 100] open databases in [0.24173] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [100 of 100] open databases in [0.46497] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 1000] open databases in [0.27127] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1000 of 1000] open databases in [4.18579] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10000] open databases in [0.30128] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10000 of 10000] open databases in [49.35048] seconds
lmdb original (begin, commit)
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10] open databases in [0.45135] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10 of 10] open databases in [0.38990] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 100] open databases in [0.47890] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [100 of 100] open databases in [0.69917] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 1000] open databases in [1.84908] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1000 of 1000] open databases in [5.88098] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10000] open databases in [22.12491] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10000 of 10000] open databases in [74.53854] seconds
Hope it is useful!
Thanks.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/