March 2014 - openldap-bugs

Re: (ITS#7829) MDB mdb_cursor_del causes records to be skipped

by hyc＠symas.com

Hallvard Breien Furuseth wrote: > Sorry, forgot the crash info. Thanks, confirmed. In a nested rebalance, doing a page merge with the neighbor on the left was discarding the rebalanced state. Fixed now in mdb.master > > mdb.c:7419: Assertion 'NUMKEYS(mc->mc_pg[ptop]) > 1' failed in mdb_rebalance() > > (gdb) p *mc->mc_pg[ptop] > $1 = { > mp_p = {p_pgno = 24, p_next = 0x18}, mp_pad = 0, mp_flags = 17, > mp_pb = {pb = {pb_lower = 18, pb_upper = 1016}, pb_pages = 66584594}, > mp_ptrs = {1016} > } > (gdb) info locals > node = 0x0 > rc = -268443168 > ptop = 0 > minkeys = 1 > mn = { > mc_next = 0x7ffff5ce0010, mc_backup = 0x7fffe41013a0, mc_xcursor = 0x0, > mc_txn = 0x7fffe42c9700, mc_dbi = 6, mc_db = 0x7fffe42c9890, > mc_dbx = 0x7fffe42c3fb0, mc_dbflag = 0x7fffe42c3fb0 "\002", > mc_snum = 2, mc_top = 1, mc_flags = 65, > mc_pg = {0x7fffe4102840, 0x7fffe8109670, 0x2efffe300, 0x7fffefffe0c0, 0x7fffefffe0a0, 0x7fffefffe2c0, > 0x7fffefffe2c0, 0x53e7cd, 0x7fffefffe1c0, 0x7fffe40019f0, 0x7fffe41013a0, 0x7fffefffe2c0, 0x30c3669909, > 0xfbad8001, 0x7fffefffe2c0, 0x7fffefffe2c0, 0x7fffefffe2c0, 0x7fffefffe130, 0x7fffefffe2d0, 0x7fffefffe180, > 0x49c7ca, 0x7fffefffe140, 0x7fffefffe1dc, 0x1100000000, 0x0, 0x7fffefffe2d0, 0x7fffe4101528, 0x7fffefffe180, 0x1, > 0x7fffe81099d4, 0x7fffefffe260, 0x100000008}, > mc_ki = {5416, 58384, 32767, 0, 39380, 59408, 32767, 0, 39364, 59408, 32767, 0, 39400, 59408, 32767, 0, 39332, > 59408, 32767, 0, 68, 0, 0, 0, 15369, 50024, 48, 0, 57792, 61439, 32767, 0} > } > oldki = 0 > __FUNCTION__ = "mdb_rebalance" > (gdb) p *mc > $2 = { > mc_next = 0x0, mc_backup = 0x0, mc_xcursor = 0x7fffe4101528, > mc_txn = 0x7fffe42c9700, mc_dbi = 6, mc_db = 0x7fffe42c9890, > mc_dbx = 0x7fffe42c3fb0, mc_dbflag = 0x7fffe42cb3e6 "\t\n\n", > mc_snum = 2, mc_top = 1, mc_flags = 65, > mc_pg = {0x7fffe4102840, 0x7fffe8109670, 0x0, 0x40, 0x44, 0x0, 0x6a206e616d726568, 0x6f65703d756f2c72, > 0x653d63642c656c70, 0x642c656c706d6178, 0x302e006d6f633d63, 0x30235a3632353631, 0x2d1, 0x7fffe4000338, 0x81, > 0x7fffe4102c40, 0x7fffe42c7860, 0x20, 0x64, 0x0, 0x4, 0x1a, 0x19, 0x10, 0xe, 0xd, 0xb, 0xa, 0x9, 0x140, 0x84, > 0x0}, > mc_ki = {0, 5, 0, 0, 22160, 63214, 32767, 0, 32769, 0, 0, 0, 64592, 70, 0, 0, 22160, 63214, 32767, 0, 32775, 0, 0, > 0, 63424, 70, 0, 0, 22160, 63214, 32767, 0} > } > > Backtrace: > #2 mdb_assert_fail () > #3 mdb_rebalance (mc=0x7fffe41013a0) at mdb.c:7419 > #4 mdb_cursor_del0 (mc=0x7fffe41013a0) at mdb.c:7496 > #5 mdb_cursor_del (mc=0x7fffe41013a0, flags=0) at mdb.c:6360 > #6 mdb_idl_delete_keys (be, cursor=0x7fffe41013a0, keys, id=7) at idl.c:608 > #7 indexer (op=0x7fffe4000950, txn, ai, ad=0x838180, atname=0x837fc8, vals=0x7fffe40019f0, id=7, opid=2, mask=1814) at index.c:258 > #8 index_at_values (op=0x7fffe4000950, txn=0x7fffe42c9700, ad, type=0x837f60, tags=0x8381a0, vals=0x7fffe40019f0, id=7, opid=2) at index.c:337 > #9 mdb_index_values (op, txn, desc, vals, id, opid) at index.c:386 > #10 mdb_index_entry (op=0x7fffe4000950, txn=0x7fffe42c9700, opid=2, e=0x7fffe40017d0) at index.c:558 > #11 mdb_delete (op=0x7fffe4000950, rs=0x7fffefffe910) at delete.c:347 > #12 overlay_op_walk (op=0x7fffe4000950, rs=0x7fffefffe910, which=op_delete, oi=0x7fffe81036a0, on=0x0) at backover.c:671 > #13 over_op_func (op=0x7fffe4000950, rs, which) at backover.c:723 > #14 fe_op_delete (op=0x7fffe4000950, rs=0x7fffefffe910) at delete.c:174 > #15 do_delete (op=0x7fffe4000950, rs=0x7fffefffe910) at delete.c:95 > #16 connection_operation (ctx=0x7fffefffea70, arg_v=0x7fffe4000950) at connection.c:1155 > #17 connection_read_thread (ctx=0x7fffefffea70, argv) at connection.c:1291 > #18 ldap_int_thread_pool_wrapper (xpool=0x83b3b0) at tpool.c:688 > > Test output: > Running ldapadd to build slapd config database... > Running ldapadd to build slapd database... > Search the entire database... > Running ldapmodify to add a member... > Re-search the entire database... > Running ldapmodify to rename a member... > Re-search the entire database... > Running ldapmodify to rename a group... > Re-search the entire database... > Running ldapmodify to add self... > Re-search the entire database... > Running ldapdelete to remove a member... > Re-search the entire database... > Running ldapdelete to remove a group... > Re-search the entire database... > Adding groups with MAY member type schemas... > > Log ends with: > 533898e1 => index_entry_del( 7, "cn=Jessica Rabbit,ou=People,dc=example,dc=com" ) > 533898e1 mdb_idl_delete_keys: 7 [860433ad] > 533898e1 mdb_idl_delete_keys: 7 [00000000] > 533898e1 mdb_idl_delete_keys: 7 [aaacba45] > 533898e1 mdb_idl_delete_keys: 7 [e10d2617] > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

9 years, 1 month

1
0
0 / 0

Re: (ITS#7829) MDB mdb_cursor_del causes records to be skipped

by h.b.furuseth＠usit.uio.no

Sorry, forgot the crash info. mdb.c:7419: Assertion 'NUMKEYS(mc->mc_pg[ptop]) > 1' failed in mdb_rebalance() (gdb) p *mc->mc_pg[ptop] $1 = { mp_p = {p_pgno = 24, p_next = 0x18}, mp_pad = 0, mp_flags = 17, mp_pb = {pb = {pb_lower = 18, pb_upper = 1016}, pb_pages = 66584594}, mp_ptrs = {1016} } (gdb) info locals node = 0x0 rc = -268443168 ptop = 0 minkeys = 1 mn = { mc_next = 0x7ffff5ce0010, mc_backup = 0x7fffe41013a0, mc_xcursor = 0x0, mc_txn = 0x7fffe42c9700, mc_dbi = 6, mc_db = 0x7fffe42c9890, mc_dbx = 0x7fffe42c3fb0, mc_dbflag = 0x7fffe42c3fb0 "\002", mc_snum = 2, mc_top = 1, mc_flags = 65, mc_pg = {0x7fffe4102840, 0x7fffe8109670, 0x2efffe300, 0x7fffefffe0c0, 0x7fffefffe0a0, 0x7fffefffe2c0, 0x7fffefffe2c0, 0x53e7cd, 0x7fffefffe1c0, 0x7fffe40019f0, 0x7fffe41013a0, 0x7fffefffe2c0, 0x30c3669909, 0xfbad8001, 0x7fffefffe2c0, 0x7fffefffe2c0, 0x7fffefffe2c0, 0x7fffefffe130, 0x7fffefffe2d0, 0x7fffefffe180, 0x49c7ca, 0x7fffefffe140, 0x7fffefffe1dc, 0x1100000000, 0x0, 0x7fffefffe2d0, 0x7fffe4101528, 0x7fffefffe180, 0x1, 0x7fffe81099d4, 0x7fffefffe260, 0x100000008}, mc_ki = {5416, 58384, 32767, 0, 39380, 59408, 32767, 0, 39364, 59408, 32767, 0, 39400, 59408, 32767, 0, 39332, 59408, 32767, 0, 68, 0, 0, 0, 15369, 50024, 48, 0, 57792, 61439, 32767, 0} } oldki = 0 __FUNCTION__ = "mdb_rebalance" (gdb) p *mc $2 = { mc_next = 0x0, mc_backup = 0x0, mc_xcursor = 0x7fffe4101528, mc_txn = 0x7fffe42c9700, mc_dbi = 6, mc_db = 0x7fffe42c9890, mc_dbx = 0x7fffe42c3fb0, mc_dbflag = 0x7fffe42cb3e6 "\t\n\n", mc_snum = 2, mc_top = 1, mc_flags = 65, mc_pg = {0x7fffe4102840, 0x7fffe8109670, 0x0, 0x40, 0x44, 0x0, 0x6a206e616d726568, 0x6f65703d756f2c72, 0x653d63642c656c70, 0x642c656c706d6178, 0x302e006d6f633d63, 0x30235a3632353631, 0x2d1, 0x7fffe4000338, 0x81, 0x7fffe4102c40, 0x7fffe42c7860, 0x20, 0x64, 0x0, 0x4, 0x1a, 0x19, 0x10, 0xe, 0xd, 0xb, 0xa, 0x9, 0x140, 0x84, 0x0}, mc_ki = {0, 5, 0, 0, 22160, 63214, 32767, 0, 32769, 0, 0, 0, 64592, 70, 0, 0, 22160, 63214, 32767, 0, 32775, 0, 0, 0, 63424, 70, 0, 0, 22160, 63214, 32767, 0} } Backtrace: #2 mdb_assert_fail () #3 mdb_rebalance (mc=0x7fffe41013a0) at mdb.c:7419 #4 mdb_cursor_del0 (mc=0x7fffe41013a0) at mdb.c:7496 #5 mdb_cursor_del (mc=0x7fffe41013a0, flags=0) at mdb.c:6360 #6 mdb_idl_delete_keys (be, cursor=0x7fffe41013a0, keys, id=7) at idl.c:608 #7 indexer (op=0x7fffe4000950, txn, ai, ad=0x838180, atname=0x837fc8, vals=0x7fffe40019f0, id=7, opid=2, mask=1814) at index.c:258 #8 index_at_values (op=0x7fffe4000950, txn=0x7fffe42c9700, ad, type=0x837f60, tags=0x8381a0, vals=0x7fffe40019f0, id=7, opid=2) at index.c:337 #9 mdb_index_values (op, txn, desc, vals, id, opid) at index.c:386 #10 mdb_index_entry (op=0x7fffe4000950, txn=0x7fffe42c9700, opid=2, e=0x7fffe40017d0) at index.c:558 #11 mdb_delete (op=0x7fffe4000950, rs=0x7fffefffe910) at delete.c:347 #12 overlay_op_walk (op=0x7fffe4000950, rs=0x7fffefffe910, which=op_delete, oi=0x7fffe81036a0, on=0x0) at backover.c:671 #13 over_op_func (op=0x7fffe4000950, rs, which) at backover.c:723 #14 fe_op_delete (op=0x7fffe4000950, rs=0x7fffefffe910) at delete.c:174 #15 do_delete (op=0x7fffe4000950, rs=0x7fffefffe910) at delete.c:95 #16 connection_operation (ctx=0x7fffefffea70, arg_v=0x7fffe4000950) at connection.c:1155 #17 connection_read_thread (ctx=0x7fffefffea70, argv) at connection.c:1291 #18 ldap_int_thread_pool_wrapper (xpool=0x83b3b0) at tpool.c:688 Test output: Running ldapadd to build slapd config database... Running ldapadd to build slapd database... Search the entire database... Running ldapmodify to add a member... Re-search the entire database... Running ldapmodify to rename a member... Re-search the entire database... Running ldapmodify to rename a group... Re-search the entire database... Running ldapmodify to add self... Re-search the entire database... Running ldapdelete to remove a member... Re-search the entire database... Running ldapdelete to remove a group... Re-search the entire database... Adding groups with MAY member type schemas... Log ends with: 533898e1 => index_entry_del( 7, "cn=Jessica Rabbit,ou=People,dc=example,dc=com" ) 533898e1 mdb_idl_delete_keys: 7 [860433ad] 533898e1 mdb_idl_delete_keys: 7 [00000000] 533898e1 mdb_idl_delete_keys: 7 [aaacba45] 533898e1 mdb_idl_delete_keys: 7 [e10d2617] -- Hallvard

9 years, 1 month

1
0
0 / 0

Re: (ITS#7829) MDB mdb_cursor_del causes records to be skipped

by h.b.furuseth＠usit.uio.no

hyc(a)symas.com writes: > Thanks for the report. Fixed now in mdb.master. That fix breaks test052-memberof with pagesize 1024. Unrealistic, but it's not a MAXKEYSIZE issue so it should not have made a difference. Tested with RE24 9e22d8f2cd5ed10cd1dd22acbd729a49e5ecc1ab + mdb.master~ or mdb.master ac3acc121855e4297d597a4d71e9863e1e1e2450 + this patch: diff --git a/libraries/liblmdb/lmdb.h b/libraries/liblmdb/lmdb.h index 2ebd43f..3f4a167 100644 --- a/libraries/liblmdb/lmdb.h +++ b/libraries/liblmdb/lmdb.h @@ -411,4 +411,4 @@ typedef enum MDB_cursor_op { /** Too big key/data, key is empty, or wrong DUPFIXED size */ -#define MDB_BAD_VALSIZE (-30781) -#define MDB_LAST_ERRCODE MDB_BAD_VALSIZE +#define MDB_BAD_VALSIZE (abort(),-30781) /* we don't hit MDB_MAXKEYSIZE */ +#define MDB_LAST_ERRCODE (-30781) /** @} */ diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c index 7e29058..ad9bfc6 100644 --- a/libraries/liblmdb/mdb.c +++ b/libraries/liblmdb/mdb.c @@ -405,3 +405,3 @@ static txnid_t mdb_debug_start; #ifndef MDB_MAXKEYSIZE -#define MDB_MAXKEYSIZE 511 +#define MDB_MAXKEYSIZE 400 /* max-max is 446 for psize=1024 on 64-bit hosts */ #endif @@ -3536,3 +3536,3 @@ mdb_env_open2(MDB_env *env) newenv = 1; - env->me_psize = env->me_os_psize; + env->me_psize = 1024; if (env->me_psize > MAX_PAGESIZE) -- Hallvard

9 years, 1 month

1
0
0 / 0

Re: (ITS#7829) MDB mdb_cursor_del causes records to be skipped

by hyc＠symas.com

armon.dadgar(a)gmail.com wrote: > Full_Name: Armon Dadgar > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (50.78.97.10) > > > I have an issue where when iterating over data and deleting keys, the cursor > will incorrectly skip rows, and then returns an early NOTFOUND even though more > records still exist. > > I have a gist demonstrating this: https://gist.github.com/armon/9825666 > > I am using small Go wrapper around LMDB, version 0.9.11. > > The expected behavior is to numbers 1..91 printed. However, when I run it, > it gets to 16 and returns no further rows. > > Thanks for the report. Fixed now in mdb.master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

9 years, 2 months

1
0
0 / 0

Re: (ITS#7825) LMDB: misleading error message

by timur.kristof＠gmail.com

On Fri Mar 28 2014 18:36:11 GMT+0100 (CET), Howard Chu wrote: > timur.kristof(a)gmail.com wrote: > > > Here's a small test case which will demonstrate it: > > http://pastebin.com/hHwD1srf > > As an aside, this is some pretty horrible code. Why are you using > MDB_GET_CURRENT after all of your cursor_get calls? It truly is horrendous. I wrote it in an attempt to (roughly) reproduce what one of my examples did in javascript. Then when I discovered what the issue really was, I forgot to clean it up before pastebinning. I hope it's still okay as a testcase for this bugreport :) > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ > -- Timur Sent from my Jolla

9 years, 2 months

1
0
0 / 0

Re: (ITS#7825) LMDB: misleading error message

by hyc＠symas.com

timur.kristof(a)gmail.com wrote: > Here's a small test case which will demonstrate it: > http://pastebin.com/hHwD1srf As an aside, this is some pretty horrible code. Why are you using MDB_GET_CURRENT after all of your cursor_get calls? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

9 years, 2 months

1
0
0 / 0

(ITS#7830) MDB Size grown a lot during massive updates and concurrent read access

by antonin.meunier＠cgi.com

Full_Name: Antonin Meunier Version: 2.4.39 OS: Red Hat 6.5 URL: Submission from: (NULL) (195.6.127.108) Ldap is set to a maximum MDB database size of 80GB. Ldap uses 20GB of effective Size. There are approximately 60 index , 1500000 peoples and 4500000 groups. The LDAP is accessed in read / write by about 20 tomcat. In parallel to this, update treatment (inject ldif) are started performing lots of massive upates (750 000, 500 000 groups). When processing the massives updates, the MDB size increases to the 80max and therefore the treatment plant updates. If I stop all tomcat, the problem no longer occurs. So it seems that the problem comes from locks that are placed on the pages of indexes and data by readers and force to create new pages for changes with conservation of all intermediate pages. After that, the only way to find a "normal" size is to export the ldap , delete it and re-import the backup. The LDAP database is then again 20GB

9 years, 2 months

1
0
0 / 0

Re: (ITS#7828) LMDB crash in mdb_cursor_put(MDB_CURRENT)

by sbn＠tbricks.com

Works after reverting commit (checked f537429) Thanks!

9 years, 2 months

1
0
0 / 0

(ITS#7829) MDB mdb_cursor_del causes records to be skipped

by armon.dadgar＠gmail.com

Full_Name: Armon Dadgar Version: OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (50.78.97.10) I have an issue where when iterating over data and deleting keys, the cursor will incorrectly skip rows, and then returns an early NOTFOUND even though more records still exist. I have a gist demonstrating this: https://gist.github.com/armon/9825666 I am using small Go wrapper around LMDB, version 0.9.11. The expected behavior is to numbers 1..91 printed. However, when I run it, it gets to 16 and returns no further rows.

9 years, 2 months

1
0
0 / 0

Re: (ITS#7828) LMDB crash in mdb_cursor_put(MDB_CURRENT)

by hyc＠symas.com

h.b.furuseth(a)usit.uio.no wrote: > Doc bug, fixed recently: MDB_CURRENT must not change the data size. Wrong. This is a regression due to ITS#7793. That commit has now been reverted. > Also with DUPSORT the data item must sort in the same place as the > previous one, that needs to be documented too. I'll add this note to the doc. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

9 years, 2 months

1
0
0 / 0

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs March 2014