(ITS#8259) LMDB utilities print usage incorrectly
by jpmens＠gmail.com
Full_Name: Jan-Piet Mens
Version: LMDB 0.9.16
OS: OSX, Linux
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (2a02:908:b41:38e0:241a:be7c:787f:620)
LMDB's utilities (mdb_stat, mdb_copy, etc) incorrectly specify that dbpath
(envpath) comes before options.
$ ./mdb_stat jp -a
usage: ./mdb_stat dbpath [-V] [-n] [-e] [-r[r]] [-f[f[f]]] [-a|-s subdb]
$ ./mdb_stat -a jp
Status of Main DB
...
This also requires a documentation update in the man pages.
8 years, 2 months
Re: (ITS#8258) LMDB cursor_del followed by MDB_NEXT can trigger assert crash
by hyc＠symas.com
steven.lang(a)hgst.com wrote:
> Full_Name: Steven Lang
> Version: LMDB 0.9.16
> OS: Ubuntu 14.04
> URL: ftp://ftp.openldap.org/incoming/steven-lang-150929.c
> Submission from: (NULL) (199.255.44.5)
>
>
> While doing some testing using randomized key and data lengths and random puts
> and deletes, I found a situation in which mdb_cursor_del(...) followed by
> mdb_cursor_get(..., MDB_NEXT) causes an error:
>
> mdb.c:5726: Assertion 'IS_LEAF(mp)' failed in mdb_cursor_next()
> Aborted
>
> After a little debugging I was able to isolate what the cause was and produce a
> simple program to reproduce the issue; during the re-balance following a delete,
> it would try to move a node from the first child of the root page to the second.
> If the node it moves has a key longer than the second node in the root page, and
> the new key doesn't fit, it will split the root page. As a result, by deleting a
> node, the tree becomes deeper.
>
> However, the cursor only checks and compensates for the tree becoming shallower.
> So now the cursor was pointing to a branch page rather than a leaf page. Any
> further cursor operations fail. (Trying to do a MDB_SET, MDB_SET_RANGE, etc will
> silently fail at this point, while MDB_NEXT and possibly MDB_PREV will assert.)
>
> The latest head with the fix for ITS#8221 changes the behavior slightly, due to
> the different arrangement of the tree with the less aggressive merging. However,
> it still fails.
>
> Attached is a program which fills a DB, then deletes keys until it asserts.
Thanks for the report. Our alternate approach to solve #8221 was to check the
page fill factor including the (usually omitted) key #0 size; we didn't go
that way because it was slightly more processing. But given this bug, it seems
that's the only way to go.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
8 years, 2 months
(ITS#8258) LMDB cursor_del followed by MDB_NEXT can trigger assert crash
by steven.lang＠hgst.com
Full_Name: Steven Lang
Version: LMDB 0.9.16
OS: Ubuntu 14.04
URL: ftp://ftp.openldap.org/incoming/steven-lang-150929.c
Submission from: (NULL) (199.255.44.5)
While doing some testing using randomized key and data lengths and random puts
and deletes, I found a situation in which mdb_cursor_del(...) followed by
mdb_cursor_get(..., MDB_NEXT) causes an error:
mdb.c:5726: Assertion 'IS_LEAF(mp)' failed in mdb_cursor_next()
Aborted
After a little debugging I was able to isolate what the cause was and produce a
simple program to reproduce the issue; during the re-balance following a delete,
it would try to move a node from the first child of the root page to the second.
If the node it moves has a key longer than the second node in the root page, and
the new key doesn't fit, it will split the root page. As a result, by deleting a
node, the tree becomes deeper.
However, the cursor only checks and compensates for the tree becoming shallower.
So now the cursor was pointing to a branch page rather than a leaf page. Any
further cursor operations fail. (Trying to do a MDB_SET, MDB_SET_RANGE, etc will
silently fail at this point, while MDB_NEXT and possibly MDB_PREV will assert.)
The latest head with the fix for ITS#8221 changes the behavior slightly, due to
the different arrangement of the tree with the less aggressive merging. However,
it still fails.
Attached is a program which fills a DB, then deletes keys until it asserts.
8 years, 2 months
Re: (ITS#8257) test061-syncreplication-initiation seg faults
by hyc＠symas.com
michael(a)stroeder.com wrote:
> Full_Name:
> Version: openSUSE Linux
> OS: RE24 branch
> URL:
> Submission from: (NULL) (84.178.28.66)
>
>
> The current re24 branch seg faults:
Fixed on mdb.RE/0.9. Closing this ITS, use #8221.
>
>>>>>> Starting test061-syncreplication-initiation for mdb...
> Running defines.sh
> Initializing server configurations
> Starting provider slapd on ldap://localhost:9011/
> Starting forward1 slapd on ldap://localhost:9013/
> Starting consumer slapd on ldap://localhost:9012/
> Adding schema on ldap://localhost:9011/
> Adding backend module on ldap://localhost:9011/...
> Adding schema on ldap://localhost:9012/
> Adding backend module on ldap://localhost:9012/...
> Adding schema on ldap://localhost:9013/
> Adding backend module on ldap://localhost:9013/...
> Adding database configuration on ldap://localhost:9011/
> Populating provider on ldap://localhost:9011/
> Adding database configuration on ldap://localhost:9013/
> Adding database configuration on ldap://localhost:9012/
> Using ldapsearch to check that ldap://localhost:9013/ received database...
> Using ldapsearch to check that ldap://localhost:9012/ received database...
> Running 1 of 1 syncrepl initiation race tests...
> Stopping forwarders for add test
> Using ldapadd to add 10 entries on provider
> Starting forwarders again
> Using ldapadd to add 10 more entries on provider
> Checking replication to ldap://localhost:9013/
> Checking replication to ldap://localhost:9012/
> Stopping forwarders for add/delete test
> Using ldapadd to add 10 entries on provider
> Using ldapdelete to delete 10 entries on provider
> Starting forwarders again
> Using ldapadd to add 10 more entries on provider
> Using ldapdelete to delete 10 more entries on provider
> Checking replication to ldap://localhost:9013/
> Checking replication to ldap://localhost:9012/
> Stopping forwarders for delete test
> Using ldapdelete to delete entries on provider
> Starting forwarders again
> Using ldapdelete to delete 10 more entries on provider
> ldap_result: Can't contact LDAP server (-1)
> ldapdelete failed (255)!
>>>>>> test061-syncreplication-initiation failed for mdb
> (exit 1)
> Makefile:309: recipe for target 'mdb-mod' failed
> make[2]: *** [mdb-mod] Error 1
> make[2]: Leaving directy %2'/usr/src/michael/openldap-git/re24/openldap/tests'
> Makefile:279: recipe for target 'test' failed
> make[1]: *** [test] Error 2
> make[1]: Leaving directory '/usr/src/michael/openldap-git/re24/openldap/tests'
> Makefile:291: recipe for target 'test' failed
> make: *** [test] Error 2
> michael@nb2:/usr/src/michael/openldap-git/re24/openldap> find -name core
> ./tests/testrun/prov/core
>
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
8 years, 2 months
Re: (ITS#8257) test061-syncreplication-initiation seg faults
by hyc＠symas.com
michael(a)stroeder.com wrote:
> Full_Name:
> Version: openSUSE Linux
> OS: RE24 branch
> URL:
> Submission from: (NULL) (84.178.28.66)
>
>
> The current re24 branch seg faults:
Thanks, this seems to be a problem with the ITS#8221 patch. Looking into it.
>
>>>>>> Starting test061-syncreplication-initiation for mdb...
> Running defines.sh
> Initializing server configurations
> Starting provider slapd on ldap://localhost:9011/
> Starting forward1 slapd on ldap://localhost:9013/
> Starting consumer slapd on ldap://localhost:9012/
> Adding schema on ldap://localhost:9011/
> Adding backend module on ldap://localhost:9011/...
> Adding schema on ldap://localhost:9012/
> Adding backend module on ldap://localhost:9012/...
> Adding schema on ldap://localhost:9013/
> Adding backend module on ldap://localhost:9013/...
> Adding database configuration on ldap://localhost:9011/
> Populating provider on ldap://localhost:9011/
> Adding database configuration on ldap://localhost:9013/
> Adding database configuration on ldap://localhost:9012/
> Using ldapsearch to check that ldap://localhost:9013/ received database...
> Using ldapsearch to check that ldap://localhost:9012/ received database...
> Running 1 of 1 syncrepl initiation race tests...
> Stopping forwarders for add test
> Using ldapadd to add 10 entries on provider
> Starting forwarders again
> Using ldapadd to add 10 more entries on provider
> Checking replication to ldap://localhost:9013/
> Checking replication to ldap://localhost:9012/
> Stopping forwarders for add/delete test
> Using ldapadd to add 10 entries on provider
> Using ldapdelete to delete 10 entries on provider
> Starting forwarders again
> Using ldapadd to add 10 more entries on provider
> Using ldapdelete to delete 10 more entries on provider
> Checking replication to ldap://localhost:9013/
> Checking replication to ldap://localhost:9012/
> Stopping forwarders for delete test
> Using ldapdelete to delete entries on provider
> Starting forwarders again
> Using ldapdelete to delete 10 more entries on provider
> ldap_result: Can't contact LDAP server (-1)
> ldapdelete failed (255)!
>>>>>> test061-syncreplication-initiation failed for mdb
> (exit 1)
> Makefile:309: recipe for target 'mdb-mod' failed
> make[2]: *** [mdb-mod] Error 1
> make[2]: Leaving directy %2'/usr/src/michael/openldap-git/re24/openldap/tests'
> Makefile:279: recipe for target 'test' failed
> make[1]: *** [test] Error 2
> make[1]: Leaving directory '/usr/src/michael/openldap-git/re24/openldap/tests'
> Makefile:291: recipe for target 'test' failed
> make: *** [test] Error 2
> michael@nb2:/usr/src/michael/openldap-git/re24/openldap> find -name core
> ./tests/testrun/prov/core
>
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
8 years, 2 months
(ITS#8257) test061-syncreplication-initiation seg faults
by michael＠stroeder.com
Full_Name:
Version: openSUSE Linux
OS: RE24 branch
URL:
Submission from: (NULL) (84.178.28.66)
The current re24 branch seg faults:
>>>>> Starting test061-syncreplication-initiation for mdb...
Running defines.sh
Initializing server configurations
Starting provider slapd on ldap://localhost:9011/
Starting forward1 slapd on ldap://localhost:9013/
Starting consumer slapd on ldap://localhost:9012/
Adding schema on ldap://localhost:9011/
Adding backend module on ldap://localhost:9011/...
Adding schema on ldap://localhost:9012/
Adding backend module on ldap://localhost:9012/...
Adding schema on ldap://localhost:9013/
Adding backend module on ldap://localhost:9013/...
Adding database configuration on ldap://localhost:9011/
Populating provider on ldap://localhost:9011/
Adding database configuration on ldap://localhost:9013/
Adding database configuration on ldap://localhost:9012/
Using ldapsearch to check that ldap://localhost:9013/ received database...
Using ldapsearch to check that ldap://localhost:9012/ received database...
Running 1 of 1 syncrepl initiation race tests...
Stopping forwarders for add test
Using ldapadd to add 10 entries on provider
Starting forwarders again
Using ldapadd to add 10 more entries on provider
Checking replication to ldap://localhost:9013/
Checking replication to ldap://localhost:9012/
Stopping forwarders for add/delete test
Using ldapadd to add 10 entries on provider
Using ldapdelete to delete 10 entries on provider
Starting forwarders again
Using ldapadd to add 10 more entries on provider
Using ldapdelete to delete 10 more entries on provider
Checking replication to ldap://localhost:9013/
Checking replication to ldap://localhost:9012/
Stopping forwarders for delete test
Using ldapdelete to delete entries on provider
Starting forwarders again
Using ldapdelete to delete 10 more entries on provider
ldap_result: Can't contact LDAP server (-1)
ldapdelete failed (255)!
>>>>> test061-syncreplication-initiation failed for mdb
(exit 1)
Makefile:309: recipe for target 'mdb-mod' failed
make[2]: *** [mdb-mod] Error 1
make[2]: Leaving directy %2'/usr/src/michael/openldap-git/re24/openldap/tests'
Makefile:279: recipe for target 'test' failed
make[1]: *** [test] Error 2
make[1]: Leaving directory '/usr/src/michael/openldap-git/re24/openldap/tests'
Makefile:291: recipe for target 'test' failed
make: *** [test] Error 2
michael@nb2:/usr/src/michael/openldap-git/re24/openldap> find -name core
./tests/testrun/prov/core
8 years, 2 months
Re: (ITS#8221) getting MDB_PAGE_FULL using mdb_cursor_del
by hyc＠symas.com
hyc(a)symas.com wrote:
> sergej.jurecko(a)gmail.com wrote:
>> Full_Name: Sergej Jurečko
>> Version: lmdb 0.9.16
>> OS: osx
>> URL: ftp://ftp.openldap.org/incoming/delerror.zip
>> Submission from: (NULL) (193.189.172.218)
>>
>>
>> Code and lmdb file is uploaded to ftp. Code is also here:
>> https://gist.github.com/SergejJurecko/68979ff6460806581ad5
>>
>> If you run the code on the uploaded lmdb file, it will result in MDB_PAGE_FULL
>> error out of mdb_cursor_del.
>>
>> I use dupsorts a lot in my app. Every value in the dupsort has 2 64bit integers
>> and 1 u8 in the beginning, which are used in the custom comparison function to
>> sort the values.
>>
>> During normal operation values are added to the end of the dupsort and
>> eventually a cleanup operation occurs that deletes unused values.
>>
>> The example app tries to delete the first value in the dupsort. But it will fail
>> the same if it tries to delete the next one instead.
>
> Thanks for the code and data, it shows the problem pretty clearly.
> Unfortunately the solution is less obvious at the moment, still working on it.
Fixed now in mdb.master, please test and followup with results. Thanks.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
8 years, 2 months
Re: (ITS#8221) getting MDB_PAGE_FULL using mdb_cursor_del
by hyc＠symas.com
sergej.jurecko(a)gmail.com wrote:
> Full_Name: Sergej Jurečko
> Version: lmdb 0.9.16
> OS: osx
> URL: ftp://ftp.openldap.org/incoming/delerror.zip
> Submission from: (NULL) (193.189.172.218)
>
>
> Code and lmdb file is uploaded to ftp. Code is also here:
> https://gist.github.com/SergejJurecko/68979ff6460806581ad5
>
> If you run the code on the uploaded lmdb file, it will result in MDB_PAGE_FULL
> error out of mdb_cursor_del.
>
> I use dupsorts a lot in my app. Every value in the dupsort has 2 64bit integers
> and 1 u8 in the beginning, which are used in the custom comparison function to
> sort the values.
>
> During normal operation values are added to the end of the dupsort and
> eventually a cleanup operation occurs that deletes unused values.
>
> The example app tries to delete the first value in the dupsort. But it will fail
> the same if it tries to delete the next one instead.
Thanks for the code and data, it shows the problem pretty clearly.
Unfortunately the solution is less obvious at the moment, still working on it.
The situation is this: deleting the record causes the leaf page to become too
empty, so it gets merged with a neighboring page - this is no problem. The
merge causes its parent branch page to also become too empty, so that is also
merged with a neighboring branch page. This is where the problem arises:
In a branch page, the key for slot 0 is always empty - it's never stored. So
we have two branch pages, each with two nodes, and node 0 of each is
essentially zero length. But when merging, we need to store a valid key.
We go from
page 1, slot 0; page 1, slot 1; and page 2, slot 0; page 2, slot 1
to
page 1, slot 0, slot 1, slot 2, slot 3
In this case, we have to retrieve the key corresponding to page 2 slot 0 and
store it in page 1 slot 2. In this DB this key is very large, and storing it
leaves no room for the last key in slot 3.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
8 years, 2 months
