openldap-devel

openldap-devel@openldap.org

1420 discussions

slapd frontend contention
by Howard Chu 16 Aug '13

16 Aug '13

Over the past couple years I've noticed that slapd's frontend was a bottleneck in our benchmarks on larger systems. But I hadn't quantified the extent of the problem. This week I stumbled across mutrace, which is a very handy tool that measures how much time is spent in mutexes. Running against back-mdb with the test008 setup, using -l 100 on the tester showed some pretty interesting data. This is a run where slapd was configured with listener-threads 2 on my quadcore laptop: The #1 contended mutex is in the thread pool, pool->ltp_mutex. #2 and #4 are the socket-table mutexes. (With the default of 1 listener thread they're correspondingly higher in the chart.) #3 is slap_op_mutex which is used to ensure unique timestamps in each Operation structure. #5 is the LMDB writer mutex. #6 is the attr_alloc mutex, which we should probably consider eliminating but nothing after that is very significant. 7,8,9 are connection writer mutexes, and #10 is the pending CSN list mutex. ##### mutrace: 0.2 successfully initialized for process slapd (PID: 10868). ^C mutrace: Showing statistics for process slapd (PID: 10868). mutrace: 717 mutexes used. Mutex #471 (0x0x10bae48) first referenced by: mutrace.c:1448 pthread_mutex_init() tpool.c:217 ldap_pvt_thread_pool_init() Mutex #488 (0x0x9100f0) first referenced by: mutrace.c:1448 pthread_mutex_init() daemon.c:2906 slapd_daemon() Mutex #497 (0x0x9107e0) first referenced by: mutrace.c:1448 pthread_mutex_init() init.c:92 slap_init() main.c:787 main() libc-start.c:258 __libc_start_main() Mutex #487 (0x0x9100a0) first referenced by: mutrace.c:1448 pthread_mutex_init() daemon.c:1629 slapd_daemon_init() Mutex #635 (0x0x7ffe2d78c040) first referenced by: mutrace.c:1448 pthread_mutex_init() mdb.c:3795 mdb_env_setup_locks() mdb.c:3906 mdb_env_open() init.c:174 mdb_db_open() backend.c:224 backend_startup_one() backend.c:330 backend_startup() init.c:219 slap_startup() main.c:992 main() libc-start.c:258 __libc_start_main() Mutex #495 (0x0x910720) first referenced by: mutrace.c:1448 pthread_mutex_init() attr.c:708 attr_init() entry.c:95 entry_init() init.c:117 slap_init() main.c:787 main() libc-start.c:258 __libc_start_main() Mutex #113 (0x0x7ffe2d59c248) first referenced by: mutrace.c:1448 pthread_mutex_init() connection.c:421 connection_init() Mutex #98 (0x0x7ffe2d59b6c8) first referenced by: mutrace.c:1448 pthread_mutex_init() connection.c:421 connection_init() Mutex #105 (0x0x7ffe2d59bc88) first referenced by: mutrace.c:1448 pthread_mutex_init() connection.c:421 connection_init() Mutex #4 (0x0x116cf20) first referenced by: mutrace.c:1448 pthread_mutex_init() backend.c:624 backend_db_init() bconfig.c:1661 config_generic() config.c:345 config_set_vals() config.c:419 config_add_vals() config.c:783 read_config_file() bconfig.c:4439 read_config() main.c:793 main() libc-start.c:258 __libc_start_main() mutrace: Showing 10 mutexes in order of (write) contention count: Mutex # Locked Changed Cont. cont.Time[ms] tot.Time[ms] avg.Time[ms] 471 231744 178161 41572 14511.248 634.339 0.003 488 108544 83353 9455 2449.662 356.085 0.003 497 102537 97750 8429 5156.696 245.420 0.002 487 89990 73426 7462 2477.159 318.022 0.004 635 2401 2263 1562 5838.290 905.338 0.377 495 23079 2596 205 65.932 74.804 0.003 113 44029 7438 165 95.214 1260.856 0.029 98 30427 5647 146 79.943 1100.263 0.036 105 18433 4504 119 66.397 965.679 0.052 4 4800 3540 117 30.774 12.467 0.003 ... ... ... ... ... ... ... mutrace: Note that rwlocks are shown as two lines: write locks then read locks. mutrace: 45 condition variables used. Condvar #43 (0x0x10bae70) first referenced by: mutrace.c:1744 pthread_cond_init() tpool.c:220 ldap_pvt_thread_pool_init() mutrace: Showing 1 condition variables in order of wait contention count: Cond # Waits Signals Cont. tot.Time[ms] cont.Time[ms] avg.Time[ms] 43 18389 106667 17090 64222.152 10010.642 0.544 ... ... ... ... ... ... ... mutrace: Total runtime is 10582.976 ms. mutrace: Results for SMP with 4 processors. mutrace: WARNING: 2 internal hash collisions detected. Results might not be as reliable as they could be. mutrace: Try to increase --hash-size=, which is currently at 67137. ##### The total runtime is relatively arbitrary, based on how quickly I typed ^C after the slapd-tester program finished. But notice that the condition variable is contended for through almost the entire runtime. This is also in the threadpool, pool->ltp_cond. I've now written a patch to allow a configurable number of workqueues in a threadpool. Here's the same slapd-tester workload against slapd with queues=4. ##### mutrace -d --hash-size=67137 ../servers/slapd/slapd -f /tmp/testr/slapd.1.conf -h ldap://:9011 -s0 -d0 mutrace: 0.2 successfully initialized for process slapd (PID: 10191). ^C mutrace: Showing statistics for process slapd (PID: 10191). mutrace: 724 mutexes used. Mutex #535 (0x0x90f7e0) first referenced by: mutrace.c:1448 pthread_mutex_init() init.c:92 slap_init() [0x65742f706d742f00] ??() ??:0 Mutex #528 (0x0x90f0f0) first referenced by: mutrace.c:1448 pthread_mutex_init() daemon.c:2906 slapd_daemon() Mutex #527 (0x0x90f0a0) first referenced by: mutrace.c:1448 pthread_mutex_init() daemon.c:1629 slapd_daemon_init() Mutex #691 (0x0xfd8eb0) first referenced by: mutrace.c:1448 pthread_mutex_init() tpool.c:247 ldap_pvt_thread_pool_init0() Mutex #692 (0x0xfd8f78) first referenced by: mutrace.c:1448 pthread_mutex_init() tpool.c:247 ldap_pvt_thread_pool_init0() Mutex #694 (0x0xfd9108) first referenced by: mutrace.c:1448 pthread_mutex_init() tpool.c:247 ldap_pvt_thread_pool_init0() Mutex #693 (0x0xfd9040) first referenced by: mutrace.c:1448 pthread_mutex_init() tpool.c:247 ldap_pvt_thread_pool_init0() Mutex #530 (0x0x7fa641167040) first referenced by: mutrace.c:1448 pthread_mutex_init() mdb.c:3795 mdb_env_setup_locks() mdb.c:3906 mdb_env_open() init.c:174 mdb_db_open() backend.c:224 backend_startup_one() backend.c:330 backend_startup() init.c:220 slap_startup() libc-start.c:258 __libc_start_main() Mutex #533 (0x0x90f720) first referenced by: mutrace.c:1448 pthread_mutex_init() attr.c:708 attr_init() entry.c:95 entry_init() init.c:117 slap_init() [0x65742f706d742f00] ??() ??:0 Mutex #592 (0x0x7fa640f76108) first referenced by: mutrace.c:1448 pthread_mutex_init() connection.c:421 connection_init() mutrace: Showing 10 mutexes in order of (write) contention count: Mutex # Locked Changed Cont. cont.Time[ms] tot.Time[ms] avg.Time[ms] 535 102537 97558 11075 6417.798 247.221 0.002 528 119100 89722 8905 2224.464 373.760 0.003 527 105147 81593 7472 2074.638 339.454 0.003 691 63260 47195 2924 680.201 190.002 0.003 692 60573 47065 2537 504.543 180.968 0.003 694 55821 42992 2296 548.873 171.009 0.003 693 54335 39828 2234 644.864 188.289 0.003 530 2401 2068 1437 6222.018 865.117 0.360 533 23079 2464 246 87.519 66.836 0.003 592 34036 5594 236 91.546 1306.007 0.038 ... ... ... ... ... ... ... mutrace: Note that rwlocks are shown as two lines: write locks then read locks. mutrace: 51 condition variables used. Condvar #45 (0x0xfd8fa0) first referenced by: mutrace.c:1744 pthread_cond_init() tpool.c:250 ldap_pvt_thread_pool_init0() Condvar #49 (0x0xfd9130) first referenced by: mutrace.c:1744 pthread_cond_init() tpool.c:250 ldap_pvt_thread_pool_init0() Condvar #47 (0x0xfd9068) first referenced by: mutrace.c:1744 pthread_cond_init() tpool.c:250 ldap_pvt_thread_pool_init0() Condvar #43 (0x0xfd8ed8) first referenced by: mutrace.c:1744 pthread_cond_init() tpool.c:250 ldap_pvt_thread_pool_init0() mutrace: Showing 4 condition variables in order of wait contention count: Cond # Waits Signals Cont. tot.Time[ms] cont.Time[ms] avg.Time[ms] 45 5580 27496 3692 13306.017 792.845 0.142 49 3798 26011 2333 13349.744 658.738 0.173 47 4364 24985 2204 14182.993 1001.988 0.230 43 3929 29665 2102 12901.160 1186.375 0.302 ... ... ... ... ... ... ... mutrace: Total runtime is 23920.267 ms. mutrace: Results for SMP with 4 processors. ##### Now the #1 contended mutex is the slap_op_time mutex. #2 and #3 are the socket table mutexes. #4,5,6,7 are the threadpool workqueue mutexes. The important thing to note is that while the total time all 4 of these mutexes are held adds up to about the same as the single mutex in the previous run, the total contended time is less than 1/6th of the previous run. I.e., splitting 1 mutex into 4 reduced our overhead by a factor of 6. For the condvar, the total contended time of the 4 cvs is 2.7x less than the previous run. The code is currently available on my mutex branch on ada.openldap.org. The pool-pause feature hasn't yet been tested. When I've finished that I'll merge to master for RE25. I believe this patch finally gets the frontend out of LMDB's way. I've also looked at restructuring the writer mutexes, but that may prove to be a deadend, based on the signalling requirements we have. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

questions about mdb interface
by Roman Gershman 02 Aug '13

02 Aug '13

Hi, First of all, I wanted to thank you for developing and contributing to community this state of the art library! I have couple of questions about the API. I did not 1. I've noticed that MDB_val contains "void* mv_data" and not "const void*". Is there a reason for requiring non const pointers in this data structure? I am asking because the c++ functions in my code that wrap lmdb API - they receive "const char* key" argument that I am either forced to forcefully const_cast the argument into void* or to strdup just to pass it into the API. For put methods, it seems that at some point MDB must copy the data from mv_data to its storage so mv_data may be const ptr in this case. For get methods, MDB just changes the value of the pointer to point to its internal memory buffer and the clients should not change the contents of the buffer (I think) so again it can be const. Am I wrong? 2. I do not completely understand how to use lmdb in multi-threaded environment . Is it correct to say that there can be only one open write transaction per MDB_env? In addition each thread can have at most one open transaction and this transaction is thread-local? 3. Is it possible to reuse dbi handle for all the transactions and all threads? In other words, is it possible at initialization step (when opening the database) to do the following: ... mdb_env_open(env, "some_path", MDB_WRITEMAP | MDB_NOSYNC , 0664); mdb_txn_begin(env, NULL, 0, &txn); mdb_dbi_open(txn, NULL, 0, &dbi); mdb_txn_commit(txn); // needed to keep dbi valid? and then to use value of dbi for all the subsequent operations on that db? Thanks! -- Best regards, Roman

1 0

RE24 Testing call #1 (OpenLDAP 2.4.36)
by Quanah Gibson-Mount 31 Jul '13

31 Jul '13

Current RE24 is ready for testing for the 2.4.36 release. Thanks! --Quanah -- Quanah Gibson-Mount Lead Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

2 2

Clusters and scaling, again
by Howard Chu 24 Jul '13

24 Jul '13

It's been a little over 4 years since we worked with MySQL using their NDBCluster engine to develop back-ndb, in the hopes of developing a backend that could leverage horizontal scaling systems. After Oracle acquired MySQL (via acquisition of Sun) they terminated the project. NDB had some promise, although reports from the field indicate that folks felt it was too difficult to configure and deploy. It appears that NDB has not gained much traction anywhere in the past few years, and with Oracle cutting off our access to MySQL/NDB developers, back-ndb was effectively killed. In OpenLDAP 2.5 we'll be trying again, this time using HyperDex. http://hyperdex.org. HyperDex has a number of desirable properties - trivial deployment, automatic self-balancing of data, guaranteed data durability, configurable fault tolerance... There is an extension for it providing distributed transaction support as well (which I haven't looked into yet). It's also quite fast, much faster than any of the other commonly known NoSQL systems out there. It's also reliable by design, unlike systems like MongoDB (whose only consistency is that it consistently loses data...) I've been playing with it off and on for the past several months. I've also written an LMDB backend for it, available on github. https://github.com/hyc/HyperDex/tree/lmdb So using HyperDex for an OpenLDAP backend has some greater appeal, because it would be OpenLDAP code at both ends of the stack. If you're interested in working with NoSQL and large scale databases with OpenLDAP you should definitely take a look at this. I suspect it will be easier to write a slapd backend for this API than for NDB, particularly since it's free of the rigid column layouts that NDB/MySQL required. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Please test back-mdb in main
by Quanah Gibson-Mount 18 Jul '13

18 Jul '13

The work on large transactions, nested transactions, and reader tables in lmdb has come to a point where we'd like to confirm that people do not find issues when running slapd-mdb before rolling it into RE24. If people could download and build current master of OpenLDAP and do whatever tests against slapd-mdb you can think of in addition to the test suite, particularly across multiple OSes, that would be much appreciated. Thanks, Quanah -- Quanah Gibson-Mount Lead Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

LMDB dead process detection
by Howard Chu 18 Jul '13

18 Jul '13

There's been a long-running discussion about the need to have APIs in liblmdb for displaying the reader table and clearing out stale slots. Quite a few open questions on the topic: 1) What should the API look like for examining the table? My initial instinct is to provide an iterator function that returns info about the next slot each time it's called. Not sure that this is necessary or most convenient though. Another possibility is just a one-shot function that walks the table itself and dumps the output as a formatted string to stdout, stderr, or a custom output callback. 2) What should APIs look like for clearing out a stale slot? Should it just be implicit inside the library, with no externally visible API? I.e., should the library periodically check on its own, with no outside intervention? Or should there be an API that lets a user explicitly request a particular slot to be freed? This latter sounds pretty dangerous, since freeing a slot that's actually still in use would allow a reader's view of the DB to be corrupted. 3) What approach should be used for automatic detection of stale slots? Currently we record the process ID and thread ID of a reader in the table. It's not clear to me that the thread ID has anything more than informational value. Since we register a per-thread destructor for slots, exiting threads should never be leaving stale slots in the first place. I'm also not sure that there are good APIs for an outside caller to determine the liveness of a given thread ID. The process ID is also prone to wraparound; it's still very common for Linux systems to use 15 bit process IDs. So just checking that a pid is still alive doesn't guarantee that it's the same process that was using an LMDB environment at any point in time. We have two main approaches to work around this latter issue: A) set a byte range lock for every process attached to the environment. This is what slapd's alock.c already does, which is used with BDB- and LDBM- based backends. This is fairly portable code, and has the desirable property that file locks automatically go away when a process exits. But: a) On Windows, the OS can take several minutes to clean up the locks of an exited process. So just checking for presence of a lock could erroneously consider a process to be alive long after it had actually died. b) file lock syscalls are fairly slow to execute. If we are checking liveness frequently, there will be a noticeable performance hit. Their performance also degrades exponentially with the number of processes locking concurrently, and degrades further still if networked filesystems are involved. c) This approach won't tell us if a process is in Zombie state. B) check process ID and process start time. This appears to be a fairly reliable approach, and reasonably fast, but there is no POSIX standard API for obtaining this process information. Methods for obtaining the info are fairly well documented across a variety of platforms (AIX, HPUX, multiple BSDs, Linux, Solaris, etc.) but they are all different. It appears that we can implement this compactly for each of the systems, but it means carrying around a dozen or so different implementations. Also, assuming we want to support shared LMDB access across NFS (as discussed in an earlier thread), it seems we're going to have to use a lock-based solution anyway, since process IDs won't be meaningful across host boundaries. We can implement approach (A) fairly easily, with no major repercussions. For (B) we would need to add a field to the reader table records to store the process start time. (Thus a lockfile format change.) (note: performance of fcntl locks vs checking process start time was measured with some simple code on my laptop running Linux. These functions are all highly OS-dependent, so the perf ratios may vary quite a lot from system to system.) The relative performance may not even be an issue in general, since we would only need to trigger a scan if a writer actually finds that some reader txn is preventing it from using free pages from the freeDB. Most of the time this wouldn't be happening. But if there were a legitimate long running read txn (e.g., for mdb_env_copy) we may find ourselves checking fairly often. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

BerkeleyDB support EOL?
by Gavin Henry 15 Jul '13

15 Jul '13

Hi all, Hope every one is good? What's the timescale for dropping bdb and defaulting to lmdb? Kurt, what ever happened to that Google server? Thanks. -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry(a)suretec.co.uk Open Source. Open Solutions(tm). http://www.suretecsystems.com/ Suretec Systems is a limited company registered in Scotland. Registered number: SC258005. Registered office: 24 Cormack Park, Rothienorman, Inverurie, Aberdeenshire, AB51 8GL. Subject to disclaimer at http://www.suretecgroup.com/disclaimer.html Do you know we have our own VoIP provider called SureVoIP? See http://www.surevoip.co.uk Did you see our API? http://www.surevoip.co.uk/api

2 2

LMDB proposed changes
by Howard Chu 12 Jul '13

12 Jul '13

Summarizing some discussions from IRC... The hardcoded limit on the size of the dirty page list in a transaction is a problem, there should not be limits on the effective size of a transaction. The plan is to change LMDB's disk page format to include the txnID in the page header. This way, when the dirty page list gets full we can flush it to disk without losing track of which pages were dirtied. Then if a subsequent access in the same txn revisits one of these pages, when we read it back from the DB we'll know that it came from the current txn and doesn't need to be copied again before making further modifications. The P_DIRTY bit in the page header will no longer be needed - if the txnID matches, the page can be used directly. If not, the page is clean and a new page must be allocated before writing. For WRITEMAP mode the dirty page list can be completely eliminated, the only reason we keep it now is to know which pages' P_DIRTY bit we need to clear at commit time. Increasing the size of the page header by 8 bytes is a bit annoying, this will require a full slapcat/slapadd reload of existing back-mdb databases. It would be nice if we can avoid this but I don't see how. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 1

NFS-shared LMDB?
by Howard Chu 10 Jul '13

10 Jul '13

It occurs to me that there is the potential to support an interesting use case with LMDB when the database resides on remote shared storage. In the context of slapd, you could run multiple read-only slapds concurrent with a single read-write slapd on a single database. The current liblmdb would need a couple small modifications to make this safe - an option to use fcntl(LOCK) when obtaining a reader slot, and an msync() when writing to a reader slot, to force reader lock table changes back to the server before progressing on a read txn. With an appropriate sharding director (like the feature recently added to back-meta) you could arrange so that each slapd instance serves reads for a distinct portion of the overall database. Then each host's memory would be caching a distinct set of data, maximizing cache effectiveness. The DB size could then grow arbitrarily large, and you simply add more machines/RAM/slapds as needed to keep serving from cache. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

3 8

Time fo another RE24 testing call (OpenLDAP 2.4.36)?
by Michael Ströder 04 Jul '13

04 Jul '13

HI! What are plans to cut a 2.4.36 release? Given the fact that there are important changes in RE24 for back-mdb what's the plan to release them? Ciao, Michael.

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel