openldap-devel

openldap-devel@openldap.org

1416 discussions

Fwd: Re: (ITS#7713) Segmentation fault if the pagesize of the Operating system is not equal to 4096.
by Howard Chu 02 Oct '13

02 Oct '13

Fixing this will either require adding a bunch of ugly code, or changing the on-disk format again. Opinions? Currently the page in-use offsets mp_lower and mp_upper range from [PAGEHDRSZ to pagesize]. IMO this was a stupid choice, carried over from the original btree code. It should instead have ranged from [0 to pagesize-PAGEHDRSZ] and then we'd have no issue right now. Adjusting this would require only a few minor tweaks to the code, but would require a full dump/reload of existing databases. -------- Original Message -------- Subject: Re: (ITS#7713) Segmentation fault if the pagesize of the Operating system is not equal to 4096. Date: Tue, 1 Oct 2013 07:16:11 GMT From: hyc(a)symas.com To: openldap-its(a)openldap.org sumantk2(a)linux.vnet.ibm.com wrote: > Full_Name: sumanth k > Version: 2.4.35 to any recent version with mdb support > OS: Linux - ppc64 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (122.248.161.59) > > > The pagesize of Linux for x86 and s390x architecture is 4096. Whereas in Powerpc > <ppc64> , the pagesize is by default 65536. So when the pagesize is not equal to > 4096, the segmentation fault occurs. So tried to compile the powerpc64 kernel > with page size of 4096 and the problem disappears and runs smoothly. But by > default the powerpc64 architecture runs with 65536 pagesize . So there is some > problem in mdb_env_open2() function in mdb.c when the pagesize is not equal to > 4096. Thanks for the report. I believe you should be able to instead change the definition of MDB_PAGESIZE to 65536, instead of forcing your machine to use 4096 byte pages. There are other problems though; we use an unsigned short for page offsets. I'm not sure the assert that you tripped will succeed in this case. > > These are my observations : > > > Compiled the source with -O0 optimization . > /home/openldap-2.4.36/tests/../servers/slapd/slapd -s0 -f > /home/openldap-2.4.36/tests/testrun/slapd.1.conf -h ldap://localhost:9011/ -d > 0x4105 > > ./scripts/test000-rootdse: line 31: 19059 Aborted (core dumped) > $SLAPD -f $CONF1 -h $URI1 -d $LVL $TIMING > $LOG1 2>&1 > > Core file : > bash-4.2# cat .gdbinit > b mdb_db_open > b mdb_env_open > b mdb_env_open2 > b mdb_txn_begin > b mdb_txn_renew0 > b mdb_dbi_open > b mdb_cursor_init > b mdb_cursor_set > b mdb_page_search > b mdb_page_get > b mdb_page_search_root > r -s0 -f /home/openldap-2.4.36/tests/testrun/slapd.1.conf -h > ldap://localhost:9011/ -d 0x4105 > > > # Core was generated by `/home/openldap-2.4.36/tests/../servers/slapd/slapd -s0 > -f /home/openldap-2.4.36'. > Program terminated with signal 6, Aborted. > #0 0x00001fffff7adb70 in .raise () from /lib64/libc.so.6 > Missing separate debuginfos, use: debuginfo-install > cyrus-sasl-lib-2.1.26-8.mcp8_0.2.ppc64 cyrus-sasl-md5-2.1.26-8.mcp8_0.2.ppc64 > glibc-2.17-4.mcp8_0.6.ppc64 keyutils-libs-1.5.5-4.mcp8_0.2.ppc64 > krb5-libs-1.11.3-1.mcp8_0.2.ppc64 libcom_err-1.42.7-2.mcp8_0.1.ppc64 > libdb-5.3.21-8.mcp8_0.4.ppc64 libselinux-2.1.13-15.mcp8_0.3.ppc64 > nss-softokn-freebl-3.14.3-1.mcp8_0.2.ppc64 openssl-libs-1.0.1e-4.mcp8_0.1.ppc64 > pcre-8.32-7.mcp8_0.2.ppc64 zlib-1.2.7-10.mcp8_0.1.ppc64 > (gdb) where > #0 0x00001fffff7adb70 in .raise () from /lib64/libc.so.6 > #1 0x00001fffff7afb64 in .abort () from /lib64/libc.so.6 > #2 0x00001fffff7a455c in .__assert_fail_base () from /lib64/libc.so.6 > #3 0x00001fffff7a464c in .__assert_fail () from /lib64/libc.so.6 > #4 0x00000000101498cc in mdb_node_add (mc=0x3fffe89a7d40, indx=0, > key=0x3fffe89a7d20, data=0x3fffe89a7d30, pgno=0, flags=2) at > ./../../../libraries/liblmdb/mdb.c:6160 > #5 0x000000001014882c in mdb_cursor_put (mc=0x3fffe89a7d40, key=0x3fffe89a7d20, > data=0x3fffe89a7d30, flags=2) at ./../../../libraries/liblmdb/mdb.c:5877 > #6 0x00000000101516a8 in mdb_dbi_open (txn=0x1000f675980, name=0x1027b9c8 > "ad2i", flags=262152, dbi=0x1fffff0800a0) at > ./../../../libraries/liblmdb/mdb.c:7902 > #7 0x0000000010139cd4 in mdb_db_open (be=0x1000f4f83a0, cr=0x3fffe89a81c0) at > init.c:207 > #8 0x0000000010050e34 in backend_startup_one (be=0x1000f4f83a0, > cr=0x3fffe89a81c0) at backend.c:224 > #9 0x0000000010051588 in backend_startup (be=0x1000f4f83a0) at backend.c:325 > #10 0x0000000010089a7c in slap_startup (be=0x0) at init.c:219 > #11 0x000000001000a9c8 in main (argc=8, argv=0x3fffe89a8958) at main.c:991 > > > > Here is the error message : > > #/home/openldap-2.4.36/tests/../servers/slapd/slapd -s0 -f > /home/openldap-2.4.36/tests/testrun/slapd.1.conf -h ldap://localhost:9011/ -d > 0x4105 > (...some messages...) > 524a2fe1 mdb_db_open: database "o=OpenLDAP Project,l=Internet": > dbenv_open(/home/openldap-2.4.36/tests/testrun/db.1.a). > slapd: ./../../../libraries/liblmdb/mdb.c:6160: mdb_node_add: Assertion > `mp->mp_pb.pb.pb_upper >= mp->mp_pb.pb.pb_lower' failed. < === fails in assert() > ; > Aborted (core dumped) > > > > Some of my observation : > > in this file libraries/liblmdb/mdb.c > in X86 : rc = mdb_cursor_set(&mc, &key, &data, MDB_SET, &exact); the value of > rc = MD_SUCCESS, but for ppc64 it is MDB_NOTFOUND. This is due to the fact that > md_root != 2 > > The value of md_root=2 for env->me_metas[1]->mm_dbs in X86 , but some huge value > in ppc64. The value of md_pad = 4096 in x86 and 65536 in ppc64 . > > in x86: > > Breakpoint 2, mdb_txn_begin (env=0x9c5050, parent=0x0, flags=0, > ret=0x7fffffffdf98) at ./../../../libraries/liblmdb/mdb.c:2219 > 2219 int rc, size, tsize = sizeof(MDB_txn); > (gdb) p env->me_metas[1]->mm_dbs > $1 = {{md_pad = 4096, md_flags = 8, md_depth = 0, md_branch_pages = 0, > md_leaf_pages = 0, md_overflow_pages = 0, md_entries = 0, md_root = > 18446744073709551615}, {md_pad = 0, md_flags = 0, > md_depth = 1, md_branch_pages = 0, md_leaf_pages = 1, md_overflow_pages = 0, > md_entries = 4, md_root = 2}} > (gdb) p env->me_metas[0]->mm_dbs > $2 = {{md_pad = 4096, md_flags = 8, md_depth = 0, md_branch_pages = 0, > md_leaf_pages = 0, md_overflow_pages = 0, md_entries = 0, md_root = > 18446744073709551615}, {md_pad = 0, md_flags = 0, > md_depth = 0, md_branch_pages = 0, md_leaf_pages = 0, md_overflow_pages = 0, > md_entries = 0, md_root = 18446744073709551615}} > > > in ppc64: > > Breakpoint 2, mdb_txn_begin (env=0x10493ef0, parent=0x0, flags=0, > ret=0x3fffffffe590) at ./../../../libraries/liblmdb/mdb.c:2219 > 2219 int rc, size, tsize = sizeof(MDB_txn); > (gdb) p env->me_metas[0]->mm_dbs > $1 = {{md_pad = 65536, md_flags = 8, md_depth = 0, md_branch_pages = 0, > md_leaf_pages = 0, md_overflow_pages = 0, md_entries = 0, md_root = > 18446744073709551615}, {md_pad = 0, md_flags = 0, > md_depth = 0, md_branch_pages = 0, md_leaf_pages = 0, md_overflow_pages = 0, > md_entries = 0, md_root = 18446744073709551615}} > (gdb) p env->me_metas[1]->mm_dbs > $2 = {{md_pad = 65536, md_flags = 8, md_depth = 0, md_branch_pages = 0, > md_leaf_pages = 0, md_overflow_pages = 0, md_entries = 0, md_root = > 18446744073709551615}, {md_pad = 0, md_flags = 0, > md_depth = 0, md_branch_pages = 0, md_leaf_pages = 0, md_overflow_pages = 0, > md_entries = 0, md_root = 18446744073709551615}} > > > > > # From further investigation , the value of env->me_metas[1] is initialized in > mdb_env_open2() at : > > p = (MDB_page *)env->me_map; > env->me_metas[0] = METADATA(p); > env->me_metas[1] = (MDB_meta *)((char *)env->me_metas[0] + meta.mm_psize); > > Here the meta.mm_psize in ppc64 is 65536 , hence there is some problem.. If the > value of meta.mm_psize is 4096 , then everything works fine. > > > As i dont have deep knowledge in openldap ,some help is needed. > > Thank you, > Sumanth K > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

5 5

Re: mdb_drop in lmdb
by Howard Chu 14 Sep '13

14 Sep '13

Conrad Steenberg wrote: > [resending to the list, doesn't seem to have gotten through] > > Hi there, > > I'm in the process of creating a binding for lmdb for the Crack > language. So far everything works really well, except that I'm not sure > about the workings of mdb_drop() to clear all the items from a database. > > Is there an example usage somewhere that I can look at? The definitive example of how to use LMDB is OpenLDAP slapd back-mdb. mdb_drop is used in back-mdb/tools.c. You can also take a look at the dict_lmdb.c driver in Postfix 2.11 for another example. > Basically for an already open db handle inside an MDB_env I do > _tmpTxn = mdb_txn_begin(_env, NULL, 0) > mdb_drop(_tmpTxn, _dbi, 0) > mdb_txn_commit(_tmpTxn) > mdb_reader_check(_env) > > However, when the db is opened again with mdb_dbi_open() I sometimes get > MDB_BAD_RSLOT: Invalid reuse of reader locktable slot That seems a bit odd, mdb_dbi_open() cannot return MDB_BAD_RSLOT. > I'm sure I'm missing a step somewhere! Any ideas? Might help to see your actual code that causes this situation. > > Thanks! > Conrad > > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

LMDB write performance on Linux
by Howard Chu 11 Sep '13

11 Sep '13

Quanah noted a few times in IRC that LMDB write performance was a lot slower after upgrading from Linux 2.6.x to 3.x. I've also observed some major slowdowns on an Ubuntu 3.5.0-23 kernel whenever the dirty page flusher is active. Apparently this is a known issue introduced in the 3.0 kernel series http://lwn.net/Articles/486311/ I'm told that this has been fixed as of Linux 3.10. Unfortunately our most recent round of benchmarks has all been run on our 3.5.0-23 system. The SLAMD results we obtained for LMDB Modify throughput (tested back in February this year) are only about 60% of the speed we expected. http://lkml.indiana.edu/hypermail/linux/kernel/1309.1/01585.html -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

next release
by Howard Chu 09 Sep '13

09 Sep '13

A lot of my recent commits are actually intended for OpenLDAP 2.5. A few of the recent TLS-related changes added to the libldap API, so one way or another they will require a library version bump. The question is whether these changes should go into the next 2.4 release: channel binding support OpenSSL elliptic curve support logging tls version/cipher info It seems that given the heightened awareness of encryption matters due to recent revelations about NSA data gathering, it might be a good idea to include these in 2.4 as well. Any opinions? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 5

EC_KEY undefined in master (was: openldap.git branch master updated)
by Hallvard Breien Furuseth 08 Sep '13

08 Sep '13

openldap-commit2devel(a)OpenLDAP.org writes: > commit e631ce808ed56119e61321463d06db7999ba5a08 > Author: Howard Chu <hyc(a)openldap.org> > Date: Sat Sep 7 09:47:19 2013 -0700 > > ITS#7595 Add Elliptic Curve support for OpenSSL tls_o.c:326: error: 'EC_KEY' undeclared. /usr/include/openssl/ssl.h has #define SSL_OP_SINGLE_ECDH_USE 0x00080000L. "grep -w EC_KEY /usr/include/openssl/*.h" finds nothing. OPENSSL_VERSION_NUMBER = 0x10000003L. -- Hallvard

2 1

Re: openldap.git branch master updated. 34f832faee9f215dfdb61de52506f2905258b147
by Quanah Gibson-Mount 17 Aug '13

17 Aug '13

--On Saturday, August 17, 2013 2:06 AM +0000 openldap-commit2devel(a)OpenLDAP.org wrote: > commit f230503b108b1c1fd2dec072b0ad5d6785da4968 > Author: Howard Chu <hyc(a)openldap.org> > Date: Fri Aug 16 18:56:31 2013 -0700 > > Tweak mutex acquisition The mutex block around the CSN bits here was to fix ITS#5709. Are we sure moving it further down won't resurrect 5709? --Quanah -- Quanah Gibson-Mount Lead Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

2 1

slapd frontend contention
by Howard Chu 16 Aug '13

16 Aug '13

Over the past couple years I've noticed that slapd's frontend was a bottleneck in our benchmarks on larger systems. But I hadn't quantified the extent of the problem. This week I stumbled across mutrace, which is a very handy tool that measures how much time is spent in mutexes. Running against back-mdb with the test008 setup, using -l 100 on the tester showed some pretty interesting data. This is a run where slapd was configured with listener-threads 2 on my quadcore laptop: The #1 contended mutex is in the thread pool, pool->ltp_mutex. #2 and #4 are the socket-table mutexes. (With the default of 1 listener thread they're correspondingly higher in the chart.) #3 is slap_op_mutex which is used to ensure unique timestamps in each Operation structure. #5 is the LMDB writer mutex. #6 is the attr_alloc mutex, which we should probably consider eliminating but nothing after that is very significant. 7,8,9 are connection writer mutexes, and #10 is the pending CSN list mutex. ##### mutrace: 0.2 successfully initialized for process slapd (PID: 10868). ^C mutrace: Showing statistics for process slapd (PID: 10868). mutrace: 717 mutexes used. Mutex #471 (0x0x10bae48) first referenced by: mutrace.c:1448 pthread_mutex_init() tpool.c:217 ldap_pvt_thread_pool_init() Mutex #488 (0x0x9100f0) first referenced by: mutrace.c:1448 pthread_mutex_init() daemon.c:2906 slapd_daemon() Mutex #497 (0x0x9107e0) first referenced by: mutrace.c:1448 pthread_mutex_init() init.c:92 slap_init() main.c:787 main() libc-start.c:258 __libc_start_main() Mutex #487 (0x0x9100a0) first referenced by: mutrace.c:1448 pthread_mutex_init() daemon.c:1629 slapd_daemon_init() Mutex #635 (0x0x7ffe2d78c040) first referenced by: mutrace.c:1448 pthread_mutex_init() mdb.c:3795 mdb_env_setup_locks() mdb.c:3906 mdb_env_open() init.c:174 mdb_db_open() backend.c:224 backend_startup_one() backend.c:330 backend_startup() init.c:219 slap_startup() main.c:992 main() libc-start.c:258 __libc_start_main() Mutex #495 (0x0x910720) first referenced by: mutrace.c:1448 pthread_mutex_init() attr.c:708 attr_init() entry.c:95 entry_init() init.c:117 slap_init() main.c:787 main() libc-start.c:258 __libc_start_main() Mutex #113 (0x0x7ffe2d59c248) first referenced by: mutrace.c:1448 pthread_mutex_init() connection.c:421 connection_init() Mutex #98 (0x0x7ffe2d59b6c8) first referenced by: mutrace.c:1448 pthread_mutex_init() connection.c:421 connection_init() Mutex #105 (0x0x7ffe2d59bc88) first referenced by: mutrace.c:1448 pthread_mutex_init() connection.c:421 connection_init() Mutex #4 (0x0x116cf20) first referenced by: mutrace.c:1448 pthread_mutex_init() backend.c:624 backend_db_init() bconfig.c:1661 config_generic() config.c:345 config_set_vals() config.c:419 config_add_vals() config.c:783 read_config_file() bconfig.c:4439 read_config() main.c:793 main() libc-start.c:258 __libc_start_main() mutrace: Showing 10 mutexes in order of (write) contention count: Mutex # Locked Changed Cont. cont.Time[ms] tot.Time[ms] avg.Time[ms] 471 231744 178161 41572 14511.248 634.339 0.003 488 108544 83353 9455 2449.662 356.085 0.003 497 102537 97750 8429 5156.696 245.420 0.002 487 89990 73426 7462 2477.159 318.022 0.004 635 2401 2263 1562 5838.290 905.338 0.377 495 23079 2596 205 65.932 74.804 0.003 113 44029 7438 165 95.214 1260.856 0.029 98 30427 5647 146 79.943 1100.263 0.036 105 18433 4504 119 66.397 965.679 0.052 4 4800 3540 117 30.774 12.467 0.003 ... ... ... ... ... ... ... mutrace: Note that rwlocks are shown as two lines: write locks then read locks. mutrace: 45 condition variables used. Condvar #43 (0x0x10bae70) first referenced by: mutrace.c:1744 pthread_cond_init() tpool.c:220 ldap_pvt_thread_pool_init() mutrace: Showing 1 condition variables in order of wait contention count: Cond # Waits Signals Cont. tot.Time[ms] cont.Time[ms] avg.Time[ms] 43 18389 106667 17090 64222.152 10010.642 0.544 ... ... ... ... ... ... ... mutrace: Total runtime is 10582.976 ms. mutrace: Results for SMP with 4 processors. mutrace: WARNING: 2 internal hash collisions detected. Results might not be as reliable as they could be. mutrace: Try to increase --hash-size=, which is currently at 67137. ##### The total runtime is relatively arbitrary, based on how quickly I typed ^C after the slapd-tester program finished. But notice that the condition variable is contended for through almost the entire runtime. This is also in the threadpool, pool->ltp_cond. I've now written a patch to allow a configurable number of workqueues in a threadpool. Here's the same slapd-tester workload against slapd with queues=4. ##### mutrace -d --hash-size=67137 ../servers/slapd/slapd -f /tmp/testr/slapd.1.conf -h ldap://:9011 -s0 -d0 mutrace: 0.2 successfully initialized for process slapd (PID: 10191). ^C mutrace: Showing statistics for process slapd (PID: 10191). mutrace: 724 mutexes used. Mutex #535 (0x0x90f7e0) first referenced by: mutrace.c:1448 pthread_mutex_init() init.c:92 slap_init() [0x65742f706d742f00] ??() ??:0 Mutex #528 (0x0x90f0f0) first referenced by: mutrace.c:1448 pthread_mutex_init() daemon.c:2906 slapd_daemon() Mutex #527 (0x0x90f0a0) first referenced by: mutrace.c:1448 pthread_mutex_init() daemon.c:1629 slapd_daemon_init() Mutex #691 (0x0xfd8eb0) first referenced by: mutrace.c:1448 pthread_mutex_init() tpool.c:247 ldap_pvt_thread_pool_init0() Mutex #692 (0x0xfd8f78) first referenced by: mutrace.c:1448 pthread_mutex_init() tpool.c:247 ldap_pvt_thread_pool_init0() Mutex #694 (0x0xfd9108) first referenced by: mutrace.c:1448 pthread_mutex_init() tpool.c:247 ldap_pvt_thread_pool_init0() Mutex #693 (0x0xfd9040) first referenced by: mutrace.c:1448 pthread_mutex_init() tpool.c:247 ldap_pvt_thread_pool_init0() Mutex #530 (0x0x7fa641167040) first referenced by: mutrace.c:1448 pthread_mutex_init() mdb.c:3795 mdb_env_setup_locks() mdb.c:3906 mdb_env_open() init.c:174 mdb_db_open() backend.c:224 backend_startup_one() backend.c:330 backend_startup() init.c:220 slap_startup() libc-start.c:258 __libc_start_main() Mutex #533 (0x0x90f720) first referenced by: mutrace.c:1448 pthread_mutex_init() attr.c:708 attr_init() entry.c:95 entry_init() init.c:117 slap_init() [0x65742f706d742f00] ??() ??:0 Mutex #592 (0x0x7fa640f76108) first referenced by: mutrace.c:1448 pthread_mutex_init() connection.c:421 connection_init() mutrace: Showing 10 mutexes in order of (write) contention count: Mutex # Locked Changed Cont. cont.Time[ms] tot.Time[ms] avg.Time[ms] 535 102537 97558 11075 6417.798 247.221 0.002 528 119100 89722 8905 2224.464 373.760 0.003 527 105147 81593 7472 2074.638 339.454 0.003 691 63260 47195 2924 680.201 190.002 0.003 692 60573 47065 2537 504.543 180.968 0.003 694 55821 42992 2296 548.873 171.009 0.003 693 54335 39828 2234 644.864 188.289 0.003 530 2401 2068 1437 6222.018 865.117 0.360 533 23079 2464 246 87.519 66.836 0.003 592 34036 5594 236 91.546 1306.007 0.038 ... ... ... ... ... ... ... mutrace: Note that rwlocks are shown as two lines: write locks then read locks. mutrace: 51 condition variables used. Condvar #45 (0x0xfd8fa0) first referenced by: mutrace.c:1744 pthread_cond_init() tpool.c:250 ldap_pvt_thread_pool_init0() Condvar #49 (0x0xfd9130) first referenced by: mutrace.c:1744 pthread_cond_init() tpool.c:250 ldap_pvt_thread_pool_init0() Condvar #47 (0x0xfd9068) first referenced by: mutrace.c:1744 pthread_cond_init() tpool.c:250 ldap_pvt_thread_pool_init0() Condvar #43 (0x0xfd8ed8) first referenced by: mutrace.c:1744 pthread_cond_init() tpool.c:250 ldap_pvt_thread_pool_init0() mutrace: Showing 4 condition variables in order of wait contention count: Cond # Waits Signals Cont. tot.Time[ms] cont.Time[ms] avg.Time[ms] 45 5580 27496 3692 13306.017 792.845 0.142 49 3798 26011 2333 13349.744 658.738 0.173 47 4364 24985 2204 14182.993 1001.988 0.230 43 3929 29665 2102 12901.160 1186.375 0.302 ... ... ... ... ... ... ... mutrace: Total runtime is 23920.267 ms. mutrace: Results for SMP with 4 processors. ##### Now the #1 contended mutex is the slap_op_time mutex. #2 and #3 are the socket table mutexes. #4,5,6,7 are the threadpool workqueue mutexes. The important thing to note is that while the total time all 4 of these mutexes are held adds up to about the same as the single mutex in the previous run, the total contended time is less than 1/6th of the previous run. I.e., splitting 1 mutex into 4 reduced our overhead by a factor of 6. For the condvar, the total contended time of the 4 cvs is 2.7x less than the previous run. The code is currently available on my mutex branch on ada.openldap.org. The pool-pause feature hasn't yet been tested. When I've finished that I'll merge to master for RE25. I believe this patch finally gets the frontend out of LMDB's way. I've also looked at restructuring the writer mutexes, but that may prove to be a deadend, based on the signalling requirements we have. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

questions about mdb interface
by Roman Gershman 02 Aug '13

02 Aug '13

Hi, First of all, I wanted to thank you for developing and contributing to community this state of the art library! I have couple of questions about the API. I did not 1. I've noticed that MDB_val contains "void* mv_data" and not "const void*". Is there a reason for requiring non const pointers in this data structure? I am asking because the c++ functions in my code that wrap lmdb API - they receive "const char* key" argument that I am either forced to forcefully const_cast the argument into void* or to strdup just to pass it into the API. For put methods, it seems that at some point MDB must copy the data from mv_data to its storage so mv_data may be const ptr in this case. For get methods, MDB just changes the value of the pointer to point to its internal memory buffer and the clients should not change the contents of the buffer (I think) so again it can be const. Am I wrong? 2. I do not completely understand how to use lmdb in multi-threaded environment . Is it correct to say that there can be only one open write transaction per MDB_env? In addition each thread can have at most one open transaction and this transaction is thread-local? 3. Is it possible to reuse dbi handle for all the transactions and all threads? In other words, is it possible at initialization step (when opening the database) to do the following: ... mdb_env_open(env, "some_path", MDB_WRITEMAP | MDB_NOSYNC , 0664); mdb_txn_begin(env, NULL, 0, &txn); mdb_dbi_open(txn, NULL, 0, &dbi); mdb_txn_commit(txn); // needed to keep dbi valid? and then to use value of dbi for all the subsequent operations on that db? Thanks! -- Best regards, Roman

1 0

RE24 Testing call #1 (OpenLDAP 2.4.36)
by Quanah Gibson-Mount 31 Jul '13

31 Jul '13

Current RE24 is ready for testing for the 2.4.36 release. Thanks! --Quanah -- Quanah Gibson-Mount Lead Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

2 2

Clusters and scaling, again
by Howard Chu 24 Jul '13

24 Jul '13

It's been a little over 4 years since we worked with MySQL using their NDBCluster engine to develop back-ndb, in the hopes of developing a backend that could leverage horizontal scaling systems. After Oracle acquired MySQL (via acquisition of Sun) they terminated the project. NDB had some promise, although reports from the field indicate that folks felt it was too difficult to configure and deploy. It appears that NDB has not gained much traction anywhere in the past few years, and with Oracle cutting off our access to MySQL/NDB developers, back-ndb was effectively killed. In OpenLDAP 2.5 we'll be trying again, this time using HyperDex. http://hyperdex.org. HyperDex has a number of desirable properties - trivial deployment, automatic self-balancing of data, guaranteed data durability, configurable fault tolerance... There is an extension for it providing distributed transaction support as well (which I haven't looked into yet). It's also quite fast, much faster than any of the other commonly known NoSQL systems out there. It's also reliable by design, unlike systems like MongoDB (whose only consistency is that it consistently loses data...) I've been playing with it off and on for the past several months. I've also written an LMDB backend for it, available on github. https://github.com/hyc/HyperDex/tree/lmdb So using HyperDex for an OpenLDAP backend has some greater appeal, because it would be OpenLDAP code at both ends of the stack. If you're interested in working with NoSQL and large scale databases with OpenLDAP you should definitely take a look at this. I suspect it will be easier to write a slapd backend for this API than for NDB, particularly since it's free of the rigid column layouts that NDB/MySQL required. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel