openldap-bugs September 2015

openldap-bugs@openldap.org

25 participants
81 discussions

Re: (ITS#8259) LMDB utilities print usage incorrectly
by hyc＠symas.com 30 Sep '15

30 Sep '15

jpmens(a)gmail.com wrote: > Full_Name: Jan-Piet Mens > Version: LMDB 0.9.16 > OS: OSX, Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (2a02:908:b41:38e0:241a:be7c:787f:620) > > > LMDB's utilities (mdb_stat, mdb_copy, etc) incorrectly specify that dbpath > (envpath) comes before options. > > $ ./mdb_stat jp -a > usage: ./mdb_stat dbpath [-V] [-n] [-e] [-r[r]] [-f[f[f]]] [-a|-s subdb] > > $ ./mdb_stat -a jp > Status of Main DB > ... > > This also requires a documentation update in the man pages. Fixed in mdb.master and mdb.RE/0.9 -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8259) LMDB utilities print usage incorrectly
by jpmens＠gmail.com 30 Sep '15

30 Sep '15

Full_Name: Jan-Piet Mens Version: LMDB 0.9.16 OS: OSX, Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2a02:908:b41:38e0:241a:be7c:787f:620) LMDB's utilities (mdb_stat, mdb_copy, etc) incorrectly specify that dbpath (envpath) comes before options. $ ./mdb_stat jp -a usage: ./mdb_stat dbpath [-V] [-n] [-e] [-r[r]] [-f[f[f]]] [-a|-s subdb] $ ./mdb_stat -a jp Status of Main DB ... This also requires a documentation update in the man pages.

1 0

Re: (ITS#8258) LMDB cursor_del followed by MDB_NEXT can trigger assert crash
by hyc＠symas.com 29 Sep '15

29 Sep '15

steven.lang(a)hgst.com wrote: > Full_Name: Steven Lang > Version: LMDB 0.9.16 > OS: Ubuntu 14.04 > URL: ftp://ftp.openldap.org/incoming/steven-lang-150929.c > Submission from: (NULL) (199.255.44.5) > > > While doing some testing using randomized key and data lengths and random puts > and deletes, I found a situation in which mdb_cursor_del(...) followed by > mdb_cursor_get(..., MDB_NEXT) causes an error: > > mdb.c:5726: Assertion 'IS_LEAF(mp)' failed in mdb_cursor_next() > Aborted > > After a little debugging I was able to isolate what the cause was and produce a > simple program to reproduce the issue; during the re-balance following a delete, > it would try to move a node from the first child of the root page to the second. > If the node it moves has a key longer than the second node in the root page, and > the new key doesn't fit, it will split the root page. As a result, by deleting a > node, the tree becomes deeper. > > However, the cursor only checks and compensates for the tree becoming shallower. > So now the cursor was pointing to a branch page rather than a leaf page. Any > further cursor operations fail. (Trying to do a MDB_SET, MDB_SET_RANGE, etc will > silently fail at this point, while MDB_NEXT and possibly MDB_PREV will assert.) > > The latest head with the fix for ITS#8221 changes the behavior slightly, due to > the different arrangement of the tree with the less aggressive merging. However, > it still fails. > > Attached is a program which fills a DB, then deletes keys until it asserts. Thanks for the report. Our alternate approach to solve #8221 was to check the page fill factor including the (usually omitted) key #0 size; we didn't go that way because it was slightly more processing. But given this bug, it seems that's the only way to go. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8258) LMDB cursor_del followed by MDB_NEXT can trigger assert crash
by steven.lang＠hgst.com 29 Sep '15

29 Sep '15

Full_Name: Steven Lang Version: LMDB 0.9.16 OS: Ubuntu 14.04 URL: ftp://ftp.openldap.org/incoming/steven-lang-150929.c Submission from: (NULL) (199.255.44.5) While doing some testing using randomized key and data lengths and random puts and deletes, I found a situation in which mdb_cursor_del(...) followed by mdb_cursor_get(..., MDB_NEXT) causes an error: mdb.c:5726: Assertion 'IS_LEAF(mp)' failed in mdb_cursor_next() Aborted After a little debugging I was able to isolate what the cause was and produce a simple program to reproduce the issue; during the re-balance following a delete, it would try to move a node from the first child of the root page to the second. If the node it moves has a key longer than the second node in the root page, and the new key doesn't fit, it will split the root page. As a result, by deleting a node, the tree becomes deeper. However, the cursor only checks and compensates for the tree becoming shallower. So now the cursor was pointing to a branch page rather than a leaf page. Any further cursor operations fail. (Trying to do a MDB_SET, MDB_SET_RANGE, etc will silently fail at this point, while MDB_NEXT and possibly MDB_PREV will assert.) The latest head with the fix for ITS#8221 changes the behavior slightly, due to the different arrangement of the tree with the less aggressive merging. However, it still fails. Attached is a program which fills a DB, then deletes keys until it asserts.

1 0

Re: (ITS#8257) test061-syncreplication-initiation seg faults
by hyc＠symas.com 29 Sep '15

29 Sep '15

michael(a)stroeder.com wrote: > Full_Name: > Version: openSUSE Linux > OS: RE24 branch > URL: > Submission from: (NULL) (84.178.28.66) > > > The current re24 branch seg faults: Fixed on mdb.RE/0.9. Closing this ITS, use #8221. > >>>>>> Starting test061-syncreplication-initiation for mdb... > Running defines.sh > Initializing server configurations > Starting provider slapd on ldap://localhost:9011/ > Starting forward1 slapd on ldap://localhost:9013/ > Starting consumer slapd on ldap://localhost:9012/ > Adding schema on ldap://localhost:9011/ > Adding backend module on ldap://localhost:9011/... > Adding schema on ldap://localhost:9012/ > Adding backend module on ldap://localhost:9012/... > Adding schema on ldap://localhost:9013/ > Adding backend module on ldap://localhost:9013/... > Adding database configuration on ldap://localhost:9011/ > Populating provider on ldap://localhost:9011/ > Adding database configuration on ldap://localhost:9013/ > Adding database configuration on ldap://localhost:9012/ > Using ldapsearch to check that ldap://localhost:9013/ received database... > Using ldapsearch to check that ldap://localhost:9012/ received database... > Running 1 of 1 syncrepl initiation race tests... > Stopping forwarders for add test > Using ldapadd to add 10 entries on provider > Starting forwarders again > Using ldapadd to add 10 more entries on provider > Checking replication to ldap://localhost:9013/ > Checking replication to ldap://localhost:9012/ > Stopping forwarders for add/delete test > Using ldapadd to add 10 entries on provider > Using ldapdelete to delete 10 entries on provider > Starting forwarders again > Using ldapadd to add 10 more entries on provider > Using ldapdelete to delete 10 more entries on provider > Checking replication to ldap://localhost:9013/ > Checking replication to ldap://localhost:9012/ > Stopping forwarders for delete test > Using ldapdelete to delete entries on provider > Starting forwarders again > Using ldapdelete to delete 10 more entries on provider > ldap_result: Can't contact LDAP server (-1) > ldapdelete failed (255)! >>>>>> test061-syncreplication-initiation failed for mdb > (exit 1) > Makefile:309: recipe for target 'mdb-mod' failed > make[2]: *** [mdb-mod] Error 1 > make[2]: Leaving directy %2'/usr/src/michael/openldap-git/re24/openldap/tests' > Makefile:279: recipe for target 'test' failed > make[1]: *** [test] Error 2 > make[1]: Leaving directory '/usr/src/michael/openldap-git/re24/openldap/tests' > Makefile:291: recipe for target 'test' failed > make: *** [test] Error 2 > michael@nb2:/usr/src/michael/openldap-git/re24/openldap> find -name core > ./tests/testrun/prov/core > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8257) test061-syncreplication-initiation seg faults
by hyc＠symas.com 29 Sep '15

29 Sep '15

michael(a)stroeder.com wrote: > Full_Name: > Version: openSUSE Linux > OS: RE24 branch > URL: > Submission from: (NULL) (84.178.28.66) > > > The current re24 branch seg faults: Thanks, this seems to be a problem with the ITS#8221 patch. Looking into it. > >>>>>> Starting test061-syncreplication-initiation for mdb... > Running defines.sh > Initializing server configurations > Starting provider slapd on ldap://localhost:9011/ > Starting forward1 slapd on ldap://localhost:9013/ > Starting consumer slapd on ldap://localhost:9012/ > Adding schema on ldap://localhost:9011/ > Adding backend module on ldap://localhost:9011/... > Adding schema on ldap://localhost:9012/ > Adding backend module on ldap://localhost:9012/... > Adding schema on ldap://localhost:9013/ > Adding backend module on ldap://localhost:9013/... > Adding database configuration on ldap://localhost:9011/ > Populating provider on ldap://localhost:9011/ > Adding database configuration on ldap://localhost:9013/ > Adding database configuration on ldap://localhost:9012/ > Using ldapsearch to check that ldap://localhost:9013/ received database... > Using ldapsearch to check that ldap://localhost:9012/ received database... > Running 1 of 1 syncrepl initiation race tests... > Stopping forwarders for add test > Using ldapadd to add 10 entries on provider > Starting forwarders again > Using ldapadd to add 10 more entries on provider > Checking replication to ldap://localhost:9013/ > Checking replication to ldap://localhost:9012/ > Stopping forwarders for add/delete test > Using ldapadd to add 10 entries on provider > Using ldapdelete to delete 10 entries on provider > Starting forwarders again > Using ldapadd to add 10 more entries on provider > Using ldapdelete to delete 10 more entries on provider > Checking replication to ldap://localhost:9013/ > Checking replication to ldap://localhost:9012/ > Stopping forwarders for delete test > Using ldapdelete to delete entries on provider > Starting forwarders again > Using ldapdelete to delete 10 more entries on provider > ldap_result: Can't contact LDAP server (-1) > ldapdelete failed (255)! >>>>>> test061-syncreplication-initiation failed for mdb > (exit 1) > Makefile:309: recipe for target 'mdb-mod' failed > make[2]: *** [mdb-mod] Error 1 > make[2]: Leaving directy %2'/usr/src/michael/openldap-git/re24/openldap/tests' > Makefile:279: recipe for target 'test' failed > make[1]: *** [test] Error 2 > make[1]: Leaving directory '/usr/src/michael/openldap-git/re24/openldap/tests' > Makefile:291: recipe for target 'test' failed > make: *** [test] Error 2 > michael@nb2:/usr/src/michael/openldap-git/re24/openldap> find -name core > ./tests/testrun/prov/core > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8257) test061-syncreplication-initiation seg faults
by michael＠stroeder.com 29 Sep '15

29 Sep '15

Full_Name: Version: openSUSE Linux OS: RE24 branch URL: Submission from: (NULL) (84.178.28.66) The current re24 branch seg faults: >>>>> Starting test061-syncreplication-initiation for mdb... Running defines.sh Initializing server configurations Starting provider slapd on ldap://localhost:9011/ Starting forward1 slapd on ldap://localhost:9013/ Starting consumer slapd on ldap://localhost:9012/ Adding schema on ldap://localhost:9011/ Adding backend module on ldap://localhost:9011/... Adding schema on ldap://localhost:9012/ Adding backend module on ldap://localhost:9012/... Adding schema on ldap://localhost:9013/ Adding backend module on ldap://localhost:9013/... Adding database configuration on ldap://localhost:9011/ Populating provider on ldap://localhost:9011/ Adding database configuration on ldap://localhost:9013/ Adding database configuration on ldap://localhost:9012/ Using ldapsearch to check that ldap://localhost:9013/ received database... Using ldapsearch to check that ldap://localhost:9012/ received database... Running 1 of 1 syncrepl initiation race tests... Stopping forwarders for add test Using ldapadd to add 10 entries on provider Starting forwarders again Using ldapadd to add 10 more entries on provider Checking replication to ldap://localhost:9013/ Checking replication to ldap://localhost:9012/ Stopping forwarders for add/delete test Using ldapadd to add 10 entries on provider Using ldapdelete to delete 10 entries on provider Starting forwarders again Using ldapadd to add 10 more entries on provider Using ldapdelete to delete 10 more entries on provider Checking replication to ldap://localhost:9013/ Checking replication to ldap://localhost:9012/ Stopping forwarders for delete test Using ldapdelete to delete entries on provider Starting forwarders again Using ldapdelete to delete 10 more entries on provider ldap_result: Can't contact LDAP server (-1) ldapdelete failed (255)! >>>>> test061-syncreplication-initiation failed for mdb (exit 1) Makefile:309: recipe for target 'mdb-mod' failed make[2]: *** [mdb-mod] Error 1 make[2]: Leaving directy %2'/usr/src/michael/openldap-git/re24/openldap/tests' Makefile:279: recipe for target 'test' failed make[1]: *** [test] Error 2 make[1]: Leaving directory '/usr/src/michael/openldap-git/re24/openldap/tests' Makefile:291: recipe for target 'test' failed make: *** [test] Error 2 michael@nb2:/usr/src/michael/openldap-git/re24/openldap> find -name core ./tests/testrun/prov/core

1 0

Re: (ITS#8221) getting MDB_PAGE_FULL using mdb_cursor_del
by sergej.jurecko＠gmail.com 26 Sep '15

26 Sep '15

> Fixed now in mdb.master, please test and followup with results. Thanks. Yes this fixed it. Thank you. Sergej

1 0

Re: (ITS#8221) getting MDB_PAGE_FULL using mdb_cursor_del
by hyc＠symas.com 26 Sep '15

26 Sep '15

hyc(a)symas.com wrote: > sergej.jurecko(a)gmail.com wrote: >> Full_Name: Sergej Jure&#269;ko >> Version: lmdb 0.9.16 >> OS: osx >> URL: ftp://ftp.openldap.org/incoming/delerror.zip >> Submission from: (NULL) (193.189.172.218) >> >> >> Code and lmdb file is uploaded to ftp. Code is also here: >> https://gist.github.com/SergejJurecko/68979ff6460806581ad5 >> >> If you run the code on the uploaded lmdb file, it will result in MDB_PAGE_FULL >> error out of mdb_cursor_del. >> >> I use dupsorts a lot in my app. Every value in the dupsort has 2 64bit integers >> and 1 u8 in the beginning, which are used in the custom comparison function to >> sort the values. >> >> During normal operation values are added to the end of the dupsort and >> eventually a cleanup operation occurs that deletes unused values. >> >> The example app tries to delete the first value in the dupsort. But it will fail >> the same if it tries to delete the next one instead. > > Thanks for the code and data, it shows the problem pretty clearly. > Unfortunately the solution is less obvious at the moment, still working on it. Fixed now in mdb.master, please test and followup with results. Thanks. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8221) getting MDB_PAGE_FULL using mdb_cursor_del
by hyc＠symas.com 26 Sep '15

26 Sep '15

sergej.jurecko(a)gmail.com wrote: > Full_Name: Sergej Jure&#269;ko > Version: lmdb 0.9.16 > OS: osx > URL: ftp://ftp.openldap.org/incoming/delerror.zip > Submission from: (NULL) (193.189.172.218) > > > Code and lmdb file is uploaded to ftp. Code is also here: > https://gist.github.com/SergejJurecko/68979ff6460806581ad5 > > If you run the code on the uploaded lmdb file, it will result in MDB_PAGE_FULL > error out of mdb_cursor_del. > > I use dupsorts a lot in my app. Every value in the dupsort has 2 64bit integers > and 1 u8 in the beginning, which are used in the custom comparison function to > sort the values. > > During normal operation values are added to the end of the dupsort and > eventually a cleanup operation occurs that deletes unused values. > > The example app tries to delete the first value in the dupsort. But it will fail > the same if it tries to delete the next one instead. Thanks for the code and data, it shows the problem pretty clearly. Unfortunately the solution is less obvious at the moment, still working on it. The situation is this: deleting the record causes the leaf page to become too empty, so it gets merged with a neighboring page - this is no problem. The merge causes its parent branch page to also become too empty, so that is also merged with a neighboring branch page. This is where the problem arises: In a branch page, the key for slot 0 is always empty - it's never stored. So we have two branch pages, each with two nodes, and node 0 of each is essentially zero length. But when merging, we need to store a valid key. We go from page 1, slot 0; page 1, slot 1; and page 2, slot 0; page 2, slot 1 to page 1, slot 0, slot 1, slot 2, slot 3 In this case, we have to retrieve the key corresponding to page 2 slot 0 and store it in page 1 slot 2. In this DB this key is very large, and storing it leaves no room for the last key in slot 3. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs September 2015