openldap-bugs January 2013

openldap-bugs@openldap.org

24 participants
98 discussions

Re: (ITS#7505) liblmdb can fail with MDB_PAGE_FULL on a delete
by sruo＠vmware.com 31 Jan '13

31 Jan '13

Hi Howard: Our test finished last night and the news is GOOD. We did not see MDB_PAGE_FULL error using patch 7505. At the same time, the other test ended with MDB_PAGE_FULL running older drop. Thank you very much for the fix. --Sung

1 0

(ITS#7515) Nested liblmdb transaction bugs
by h.b.furuseth＠usit.uio.no 31 Jan '13

31 Jan '13

Full_Name: Hallvard B Furuseth Version: mdb.master 27aaecc744955d08d2bfe7a3ca786d742267c5bd OS: Linux x86_64 URL: Submission from: (NULL) (193.69.163.163) Submitted by: hallvard Here are some crashes with nested liblmdb transactions. The ./bug test program takes the following commands: R Remove old database file, if any. [,],} Begin/commit/abort txn - nested if already in a txn. O Open main DB. c,o Create/open named DB "db". p,g,d Put/get/del {key="foo\0": data="bar\0"} to last opened DB. $ ./bug R[Op[d # Put an item, delete it in a child txn Bus Error Replace mdb_page_get() with the version below, and it works. But: $ ./bug R[Op[d]p # As above, then put the item again in top txn bug: mdb.c:3786: mdb_node_search: Assertion `nkeys > 0' failed. #3 0x0000003911c2bae0 in __assert_fail () from /lib64/libc.so.6 #4 0x0000000000406c49 in mdb_node_search (mc=0x7fffffffde80, key=0x7fffffffe070, exactp=0x7fffffffc934) at mdb.c:3786 #5 0x0000000000408846 in mdb_cursor_set (mc=0x7fffffffde80, key=0x7fffffffe070, data=0x7fffffffc920, op=MDB_SET, exactp=0x7fffffffc934) at mdb.c:4472 #6 0x0000000000409c7a in mdb_cursor_put (mc=0x7fffffffde80, key=0x7fffffffe070, data=0x7fffffffe060, flags=0) at mdb.c:4879 #7 0x00000000004109b4 in mdb_put (txn=0x615110, dbi=1, key=0x7fffffffe070, data=0x7fffffffe060, flags=0) at mdb.c:6852 #8 0x00000000004016fb in main (argc=2, argv=0x7fffffffe1e8) at bug.c:39 Successful child txns can create broken output: $ ./bug R[O[p]] # Put an item in a child txn. Success, but: $ ../mdb_stat -n -a -f test.mdb; du test.mdb db_stat reports only zeroes, but the DB is 12 K. $ ./bug R[Op][[d]][g] # Put, delete in child, then Get incorrectly succeeds $ ../mdb_stat -n -a -f test.mdb <prints freelist status, then coredump> $ ./bug R[Op][d][g] # The get fails as expected if the del is not in a child 40: mdb_get(txn, dbi, &key, &rdata): MDB_NOTFOUND: No matching key/data pair found Can't use a DBI from an aborted child txn, unlike aborted main txn: $ ./bug R[cp] # Create a named DB and put an item there $ ./bug [[o}g] # Open that DB in a child, abort, use the DBI 40: mdb_get(txn, dbi, &key, &rdata): Invalid argument $ ./bug [[o]g] # Success: Commit instead of abort the child $ ./bug [o}[g] # Success: Use the DBI from an aborted main txn static int mdb_page_get(MDB_txn *txn, pgno_t pgno, MDB_page **ret) { MDB_page *p = NULL; if (!((txn->mt_flags & MDB_TXN_RDONLY) | (txn->mt_env->me_flags & MDB_WRITEMAP))) { MDB_txn *tx2 = txn; do { MDB_ID2L dl = tx2->mt_u.dirty_list; if (dl[0].mid) { unsigned x = mdb_mid2l_search(dl, pgno); if (x <= dl[0].mid && dl[x].mid == pgno) { p = dl[x].mptr; goto done; } } } while ((tx2 = tx2->mt_parent) != NULL); } if (pgno < txn->mt_next_pgno) { p = (MDB_page *)(txn->mt_env->me_map + txn->mt_env->me_psize * pgno); } else { DPRINTF("page %zu not found", pgno); assert(p != NULL); } done: *ret = p; return (p != NULL) ? MDB_SUCCESS : MDB_PAGE_NOTFOUND; } bug.c: /* Commands: * R Remove old database file, if any. * [,],} Begin/commit/abort txn - nested if already in a txn. * O Open main DB. * c,o Create/open named DB "db". * p,g,d Put/get/del {key="foo\0": data="bar\0"} to last opened DB. */ #include <lmdb.h> #include <stdio.h> #include <stdlib.h> #include <string.h> int main(int argc, char **argv) { char c, *cmd = argc < 2 ? "" : argv[1]; int rc, ti = 0; MDB_txn *txn, *txs[9] = {NULL}; /* The loop maintains txn==txs[ti] */ MDB_val key = {4, "foo"}, data = {4, "bar"}, rdata; MDB_env *env; MDB_dbi dbi; # define envname "test.mdb" # define E(e) { rc = (e); if (rc) { fprintf(stderr, "%d: %s: %s\n", \ __LINE__, #e, mdb_strerror(rc)); abort(); } } if (*cmd == 'R') { cmd++; remove(envname); } E(mdb_env_create(&env)); E(mdb_env_set_maxdbs(env, 1)); E(mdb_env_open(env, envname, MDB_NOSYNC|MDB_NOSUBDIR, 0666)); # define C(ch, expr) case ch: E(expr); break while (txn=txs[ti], c=*cmd++) switch (c) { C('[', mdb_txn_begin(env, txn, 0, &txs[++ti])); C(']', mdb_txn_commit(txs[ti--])); C('}', (mdb_txn_abort(txs[ti--]), 0)); C('O', mdb_dbi_open(txn, NULL, 0, &dbi)); C('c', mdb_dbi_open(txn, "db", MDB_CREATE, &dbi)); C('o', mdb_dbi_open(txn, "db", 0, &dbi)); C('p', mdb_put(txn, dbi, &key, &data, 0)); C('g', mdb_get(txn, dbi, &key, &rdata)); C('d', mdb_del(txn, dbi, &key, NULL)); } if (txn) E(mdb_txn_commit(txs[1])); mdb_env_close(env); return 0; }

1 0

(ITS#7514) Optimize OpenLDAP with paged results
by quanah＠OpenLDAP.org 30 Jan '13

30 Jan '13

Full_Name: Quanah Gibson-Mount Version: 2.4.33 OS: Linux 2.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (74.196.25.250) Currently, searches of back-mdb using paged results are extremely slow. Up to 300 times slower than when not using paged results: Without paged results: time ldapsearch -LLL -x -H ldap://zre-ldap002.eng.vmware.com -b "ou=people,dc=xxxx,dc=com" -s sub -D uid=zimbra,cn=admins,cn=zimbra -w zimbra '(objectClass=zimbraAccount)' uid real 0m0.004s user 0m0.010s sys 0m0.000s With paged results: zimbra@zre-ldap002:~$ time ldapsearch -LLL -x -H ldap://zre-ldap002.eng.vmware.com -b "ou=people,dc=xxxxx,dc=com" -s sub -D uid=zimbra,cn=admins,cn=zimbra -w zimbra '(objectClass=zimbraAccount)' uid -E pr=1000 real 0m1.204s user 0m0.010s sys 0m0.000s Using paged results is fairly standard in Java clients, to prevent them from hitting OOME's based on the way in which results are returned into a single object.

1 0

(ITS#7513) added TCP keepalive support to back_meta
by tedcheng＠symas.com 30 Jan '13

30 Jan '13

Full_Name: Ted C. Cheng Version: Head OS: Linux/Solaris URL: https://dl.dropbox.com/u/94235048/back_meta_tcp_keepalive.patch Submission from: (NULL) (76.174.253.197) Please find in the link a patch against HEAD for TCP keepalive support for back_meta.

1 0

(ITS#7512) MDB page leaks
by h.b.furuseth＠usit.uio.no 30 Jan '13

30 Jan '13

Full_Name: Hallvard B Furuseth Version: mdb.master 27aaecc744955d08d2bfe7a3ca786d742267c5bd OS: Linux x86_64 URL: Submission from: (NULL) (193.69.163.163) Submitted by: hallvard If the malloc at mdb_page_malloc() line 1348 fails, me_pgfirst has already been incremented and those pages can be lost. I'll fix that one. Not this, however: If a child transaction aborts, it loses pages it used up: It does not give them back to me_pghead, nor to the parent's mt_free_pgs. It does not rewind me_pgfirst/me_last to their state at txn_begin. This program illustrates. The DB grows to 107 pages. mdb_stat can see 10 of them, same as if everything was done in the main txn. MainDB will have max 1 entry while running. #include <lmdb.h> #include <stdio.h> #include <stdlib.h> int main(void) { int rc, i; MDB_val key = { sizeof(i), &i }, data = { 0, "" }; MDB_env *env; MDB_txn *txn, *ctxn; MDB_dbi dbi; # define dbname "test.mdb" # define E(e) { rc = (e); if (rc) { fprintf(stderr, "%d: %s: %s\n", \ __LINE__, #e, mdb_strerror(rc)); return rc; } } E(mdb_env_create(&env)); E(mdb_env_open(env, dbname, MDB_NOSYNC|MDB_NOSUBDIR, 0666)); for (i = 0; i < 100; ) { E(mdb_txn_begin(env, NULL, 0, &txn)); if (!i) { E(mdb_dbi_open(txn, NULL, MDB_CREATE, &dbi)); } else { E(mdb_txn_begin(env, txn, 0, &ctxn)); E(mdb_put(ctxn, dbi, &key, &data, 0)); mdb_txn_abort(ctxn); E(mdb_del(txn, dbi, &key, NULL)); } i++; E(mdb_put(txn, dbi, &key, &data, 0)); E(mdb_txn_commit(txn)); if (i % 20 == 0) system("du " dbname); } mdb_env_close(env); return 0; }

1 0

Re: (ITS#7505) liblmdb can fail with MDB_PAGE_FULL on a delete
by hyc＠symas.com 29 Jan '13

29 Jan '13

Sung Ruo wrote: > Hi Howard: > Our test program has been running for about a day now and the fix is holding up good still. > We used to reproduce this in about 1~2 days. So I'll let it runs for another day. > > Will update again sometime tomorrow. That's good news, thanks for the followup. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7505) liblmdb can fail with MDB_PAGE_FULL on a delete
by quanah＠zimbra.com 29 Jan '13

29 Jan '13

--On Tuesday, January 29, 2013 5:36 PM +0000 sruo(a)vmware.com wrote: > Hi Howard: > Our test program has been running for about a day now and the fix is > holding up good still. We used to reproduce this in about 1~2 days. So > I'll let it runs for another day. > > Will update again sometime tomorrow. Hi Sung, I don't know if you saw, but Howard made a tweak to the 7505 commit yesterday. If you could try current mdb.master from git, that would be helpful. It also has a commit addressing ITS#7511. Thanks, Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#7505) liblmdb can fail with MDB_PAGE_FULL on a delete
by sruo＠vmware.com 29 Jan '13

29 Jan '13

Hi Howard: Our test program has been running for about a day now and the fix is holding up good still. We used to reproduce this in about 1~2 days. So I'll let it runs for another day. Will update again sometime tomorrow. Thanks. --Sung

1 0

(ITS#7511) EINVAL in internal mdb_cursor_del()
by h.b.furuseth＠usit.uio.no 28 Jan '13

28 Jan '13

Full_Name: Hallvard B Furuseth Version: master, d235778c71a50da91741b8b17c572a44b143831b OS: Linux x86_64 URL: Submission from: (NULL) (193.69.163.163) Submitted by: hallvard mdb_cursor_put() can return MDB_SUCCESS with mc.mc_flags == 0. mdb_cursor_del() requires mc.mc_flags & C_INITIALIZED. mdb_txn_commit() can call _del() which fails after _put() succeeds, at the /* was completely used up */ case. This breaks slapadd -l frontier.ldif (without -q) at ada.openldap.org, at "uid=smithhehehe,ou=people,dc=frontiernet,dc=net" (line=527182). Config ~hallvard/frontiercf.ldif. Or to trigger it a lot quicker, insert assert(rc || (mc->mc_flags & C_INITIALIZED)); before 'done:' in mdb_cursor_put(). I'm guessing that is required, since ldbm.h says _put() positions the cursor at the newly inserted item. Or is that only when _inserting_ a new item, not _modifying_ an existing one? The mdb_cursor_del() check for C_INITIALIZED is incorrect, it is evaluated as if ((!mc->mc_flags) & C_INITIALIZED). Maybe that's why this has not been caught before. Same with the test in mdb_cursor_get case MDB_GET_CURRENT.

1 0

(ITS#7510) Group ACL's not cached
by tioteath＠gmail.com 28 Jan '13

28 Jan '13

Full_Name: Tio Teath Version: 2.4.33 OS: Debian GNU Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (178.120.33.222) For some reason group ACL's not cached by slapd during single connection. This behavior lead to major efficiency decrease, especially when groups provided by ldap-proxy. Each ACL produces separate request to the back-end database. Not sure if ACL cached at all, but I tend to think this is inappropriate behavior.

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs January 2013