openldap-bugs

openldap-bugs@openldap.org

1 participants
20967 discussions

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by markus＠greenrobot.de 23 Oct '19

23 Oct '19

On 21.10.19 17:14, Howard Chu wrote: > I believe this was fixed by commit > 7edf504106c61639a89b9a4e5987242598196932 in mdb.master. I can not confirm that this works. This is the stack trace where MDB_TXN_FULL is still returned with latest mdb.master (note: line numbers shown here are off by 60 compared to mdb.master): mdb_rpage_get mdb.c:6196 mdb_page_get mdb.c:6378 mdb_page_search_lowest mdb.c:6492 mdb_node_move mdb.c:8842 mdb_rebalance mdb.c:9366 mdb_page_merge mdb.c:9166 mdb_rebalance mdb.c:9373 mdb_cursor_del0 mdb.c:9426 mdb_cursor_del mdb.c:811 Code segment: if (tl[0].mid >= MDB_TRPAGE_MAX) return MDB_TXN_FULL; Debugger shows tl[0].mid to be 4095. Hope that helps. Markus

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by hyc＠symas.com 21 Oct '19

21 Oct '19

markus(a)greenrobot.de wrote: > Full_Name: Markus Junginger > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (77.189.91.168) > > > We have a 1GB LMDB file with 7M K/V entries. With MDB_VL32, we always get a > MDB_TXN_FULL error for a transaction that is removing 2M entries (probably fails > at a lower count, we haven't measured that). Without MDB_VL32 it works fine. > > Another observation: > Once the transaction fails with MDB_TXN_FULL, the data file has grown to 1.5GB. > Without MDB_VL32, the data file stays consistent at 1 GB even if all 7M entries > are deleted in a single transaction. > > Expected behavior: > No MDB_TXN_FULL error, no data file growth. > > > I believe this was fixed by commit 7edf504106c61639a89b9a4e5987242598196932 in mdb.master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9099) LMDB: range delete
by hyc＠symas.com 21 Oct '19

21 Oct '19

opensource(a)gmx-topmail.de wrote: > Full_Name: Markus > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (91.64.106.220) > > > We built a database with LMDB and are very pleased with it in general. The one > area giving our users headaches regularly are bulk deletes. But some context > first: > > Like many others, we use a hierarchical key notation. Some simplified examples: > "user/00001", "user/00002", "user/00003", ...; "usergroup/00001", > "usergroup/00002", .... > > Or, let's say data by timestamp: "sensordata/32489732", "sensordata/32489733", > ... > > Use case like "remove all users" or "remove all sensordata entries from time x > to y" are usual. Those (transactional) operations call mdb_cursor_del() on each > entry. This runs fine until we have LOTS of entries to delete. At a magnitude of > 100,000s of entries, two things happen: 1) the time it takes to delete gets > quite noticeable and 2) the 32 bit (MDB_VL32) will run into MDB_TXN_FULL (see > also #8813). > > Given the key structure, those operations operate on consecutive keys. Because > of that, can we make this more efficient (and get rid MDB_TXN_FULL on the fly)? > > A high number of individual deletes involves lots of page merging (and a lot of > dirty pages). If we had a specialized "range delete" this could likely be > avoided. The signature of the proposed functionality could look like this: The number of pages dirtied will be the same, whether you batch or delete one by one. As such, this won't solve the MDB_TXN_FULL problem. Also, as of commit 7edf504106c61639a89b9a4e5987242598196932 this problem should have already been removed for MDB_VL32. > int mdb_cursor_del_range(MDB_cursor *mc, MDB_val *key_start, MDB_val *key_stop, > unsigned int flags) > > Given key_start and key_stop, this new function could decide to prune entire > leaf pages, or even entire sub branches. This should result in a significant > efficiency/performance gain. The challenge I'm seeing here is balancing the tree > though. Right, rebalancing would become quite a pain. A similar problem exists for PUT_MULTIPLE which is why it just internally loops over one item at a time. > What do you think? I'm not sure if OpenLdap would profit here, but I'm quite > certain that LMDB as a general database would. I'm also open to contribute if > you are willing to give some guidance. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9099) LMDB: range delete
by opensource＠gmx-topmail.de 21 Oct '19

21 Oct '19

Full_Name: Markus Version: OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (91.64.106.220) We built a database with LMDB and are very pleased with it in general. The one area giving our users headaches regularly are bulk deletes. But some context first: Like many others, we use a hierarchical key notation. Some simplified examples: "user/00001", "user/00002", "user/00003", ...; "usergroup/00001", "usergroup/00002", .... Or, let's say data by timestamp: "sensordata/32489732", "sensordata/32489733", ... Use case like "remove all users" or "remove all sensordata entries from time x to y" are usual. Those (transactional) operations call mdb_cursor_del() on each entry. This runs fine until we have LOTS of entries to delete. At a magnitude of 100,000s of entries, two things happen: 1) the time it takes to delete gets quite noticeable and 2) the 32 bit (MDB_VL32) will run into MDB_TXN_FULL (see also #8813). Given the key structure, those operations operate on consecutive keys. Because of that, can we make this more efficient (and get rid MDB_TXN_FULL on the fly)? A high number of individual deletes involves lots of page merging (and a lot of dirty pages). If we had a specialized "range delete" this could likely be avoided. The signature of the proposed functionality could look like this: int mdb_cursor_del_range(MDB_cursor *mc, MDB_val *key_start, MDB_val *key_stop, unsigned int flags) Given key_start and key_stop, this new function could decide to prune entire leaf pages, or even entire sub branches. This should result in a significant efficiency/performance gain. The challenge I'm seeing here is balancing the tree though. What do you think? I'm not sure if OpenLdap would profit here, but I'm quite certain that LMDB as a general database would. I'm also open to contribute if you are willing to give some guidance. Thanks for your consideration, Markus

1 0

Re: (ITS#9096)
by priit＠ww.ee 18 Oct '19

18 Oct '19

--000000000000c70a5505952b7d50 Content-Type: text/plain; charset="UTF-8" Also a remark I must add, this only happens in one of my 4 Manjaro installations, which is also the only one running on AMD system (Ryzen 1700 + ASRock AB350M PRO4 motherboard). --000000000000c70a5505952b7d50 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div>Also a remark I must add, this only = happens in one of my 4 Manjaro installations, which is also the only one ru= nning on AMD system (Ryzen 1700 + <span class=3D"gmail-il">ASRock</span> AB= 350M PRO4 motherboard).<br></div><br></div></div> --000000000000c70a5505952b7d50--

1 0

Re: (ITS#9098) assert fails in meta_back_search in some cases after reconnect
by quanah＠symas.com 16 Oct '19

16 Oct '19

--On Wednesday, October 16, 2019 8:37 PM +0000 maxime.besson(a)worteks.com wrote: > Full_Name: Maxime Besson > Version: 2.4.47 > OS: Debian Jessie > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (2a01:cb00:802:8400:2cbe:3c60:fca6:e50b) I suggest upgrading to 2.4.48 to pick up this fix: Fixed slapd-meta assertion when network interface goes down (ITS#8841) --Quaanh -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9098) assert fails in meta_back_search in some cases after reconnect
by maxime.besson＠worteks.com 16 Oct '19

16 Oct '19

Full_Name: Maxime Besson Version: 2.4.47 OS: Debian Jessie URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2a01:cb00:802:8400:2cbe:3c60:fca6:e50b) I am running a meta-directory with the following DB configuration. version 2.4.47, LTB build on Ubuntu 16.04 dn: olcDatabase={1}meta,cn=config objectClass: olcMetaConfig objectClass: olcDatabaseConfig objectClass: olcConfig objectClass: top olcDatabase: {1}meta olcSuffix: dc=com olcAccess: {0}to * by * read olcRootDN: cn=admin,dc=com dn: olcMetaSub={0}uri,olcDatabase={1}meta,cn=config objectClass: olcMetaTargetConfig objectClass: olcConfig objectClass: top olcMetaSub: {0}uri olcDbURI: ldap://1.2.3.4/dc=example,dc=com olcDbIDAssertBind: mode=legacy flags=non-prescriptive,proxy-authz-non-critical bindmethod=simple binddn="cn=admin,dc=example,dc=com" credentials="XXXXX" olcDbTimeout: 5 olcDbNetworkTimeout: 3 olcDbNretries: never olcDbRebindAsUser: true ... (There are 8 backends in total) Timeouts were added in order to avoid blocking OpenLDAP completely when one server becomes completely unavailable. However, since I added them, the slapd process started crashing every now and then (from a couple hours to a couple of days), usually during small network interruptions that affect all backends: I see plenty of reconnect logs shortly before the crashes. The crash is always immediately preceded by the following log message: meta_search_dobind_init[{i}]: retrying URI="{url}" DN="{DN}" {i} is never the same, and {url} and {DN} are the correct settings for backend i. The crash itself is an ABRT at the following assert in back-meta/search.c: 1957 assert( candidates[ i ].sr_msgid >= 0 1958 || candidates[ i ].sr_msgid == META_MSGID_CONNECTING ); I have analyzed several core dumps, and found that every single time slapd crashes, sr_msgid has a value of -1 (META_MSGID_IGNORE), which indeed causes the assert to fail. I found that candidates[i]->sr_flags has a value of 3 (META_CANDIDATE + META_BINDING) And the msc_mscflags in mc->mc_conns[ i ] are * 0x100081 for all connections before the one that triggers the crash * 0x100010 for the candidate that crashes the server * 0x100080 for all connections after it I am having trouble reproducing this in a test environment, but it happens regularly in production, I have tried changing the timeouts, adding a non-default bind timeout , and disabling retries (they were originally allowed) but the crashes keep happening. Note that disabling retries (olcDbNretries: never) still seems to lead to retries in meta_search_dobind_init, since the log message is still there. I cannot share the core dumps due to the sensitive information inside them. However I would gladly extract more information from them if it can help solving this.

1 0

Re: (ITS#9097) lmdb: premature free of env->me_txn0
by hyc＠symas.com 16 Oct '19

16 Oct '19

Christopher Zimmermann wrote: > Hi again, > > I did some further debugging and now I'm afraid I am indeed finishing a > transaction twice. I do call txn_commit() and after that I call > txn_abort(). But the MDB_MAP_FULL event happens during mdb_txn_commit(): > > (gdb) frame 0 > #0 mdb_page_alloc (mc=0x7f7ffffbd580, num=1, mp=0x7f7ffffbd228) at mdb.c:2286 > 2286 rc = MDB_MAP_FULL; > (gdb) bt > #0 mdb_page_alloc (mc=0x7f7ffffbd580, num=1, mp=0x7f7ffffbd228) at mdb.c:2286 > #1 0x00000fb6fb6aa239 in mdb_page_touch (mc=0x7f7ffffbd580) at mdb.c:2428 > #2 0x00000fb6fb6a9db3 in mdb_page_search (mc=0x7f7ffffbd580, key=0x0, flags=5) at mdb.c:5627 > #3 0x00000fb6fb69e585 in mdb_freelist_save (txn=0xfb6bfb3ae00) at mdb.c:3087 > #4 0x00000fb6fb69c2a4 in mdb_txn_commit (txn=0xfb6bfb3ae00) at mdb.c:3612 > > mdb_txn_commit() will return MDB_MAP_FULL. > > It may be worth adding MDB_MAP_FULL and MDB_BAD_TXN to the possible > error codes of mdb_txn_commit(), or in general provide some guidance on > how to clean up a txn after some failure. The doc for commit and abort is explicit. A txn must not be used again after calling commit or abort. There is nothing to clean up, the txn is always gone when commit or abort returns. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9097) lmdb: premature free of env->me_txn0
by madroach＠gmerlin.de 16 Oct '19

16 Oct '19

Hi again, I did some further debugging and now I'm afraid I am indeed finishing a transaction twice. I do call txn_commit() and after that I call txn_abort(). But the MDB_MAP_FULL event happens during mdb_txn_commit(): (gdb) frame 0 #0 mdb_page_alloc (mc=0x7f7ffffbd580, num=1, mp=0x7f7ffffbd228) at mdb.c:2286 2286 rc = MDB_MAP_FULL; (gdb) bt #0 mdb_page_alloc (mc=0x7f7ffffbd580, num=1, mp=0x7f7ffffbd228) at mdb.c:2286 #1 0x00000fb6fb6aa239 in mdb_page_touch (mc=0x7f7ffffbd580) at mdb.c:2428 #2 0x00000fb6fb6a9db3 in mdb_page_search (mc=0x7f7ffffbd580, key=0x0, flags=5) at mdb.c:5627 #3 0x00000fb6fb69e585 in mdb_freelist_save (txn=0xfb6bfb3ae00) at mdb.c:3087 #4 0x00000fb6fb69c2a4 in mdb_txn_commit (txn=0xfb6bfb3ae00) at mdb.c:3612 mdb_txn_commit() will return MDB_MAP_FULL. It may be worth adding MDB_MAP_FULL and MDB_BAD_TXN to the possible error codes of mdb_txn_commit(), or in general provide some guidance on how to clean up a txn after some failure. Christopher -- http://gmerlin.de OpenPGP: http://gmerlin.de/christopher.pub CB07 DA40 B0B6 571D 35E2 0DEF 87E2 92A7 13E5 DEE1

1 0

Re: (ITS#9097) lmdb: premature free of env->me_txn0
by madroach＠gmerlin.de 15 Oct '19

15 Oct '19

On Wed, Oct 16, 2019 at 01:59:51AM +0100, Howard Chu wrote: > christopher(a)gmerlin.de wrote: > > Full_Name: Christopher Zimmermann > > Version: lmdb 0.9.24 > > OS: OpenBSD > > URL: ftp://ftp.openldap.org/incoming/ > > Submission from: (NULL) (85.212.180.240) > > > > > > Hi, > > > > I can reliably hit a Bus error on OpenBSD. > > This is triggered by OpenBSDs malloc/free junking [1] and a use-after-free bug > > in lmdb. > > > > Steps to reproduce: > > - begin a read/write transaction (getting env->me_txn0) > > - fill the environment > > -> returns MDB_MAP_FULL > > -> sets txn->mt_flags |= MDB_TXN_ERROR; (This is also env->me_txn0 !) > > -> calls mdb_txn_abort > ... > > - abort the transaction (again) with mdb_abort() > > This is a bug in your code, you can't call txn_abort twice. This is > already documented. Closing this ITS. Hi, thanks for having a look. I could not find any documentation about when one must / must not call mdb_txn_abort. But in any case I do _not_ call txn_abort() twice. The problem rather seems to be that lmdb _implicitely_ frees invalid transactions. And my code then aborts them again _explicitely_. So mdb_abort was indeed called twice. Once in the MDB_TXN_ERROR internally in lmdb case and once from my code. On second thought my fix won't fix the same problem for errored child transactions. What is necessary seems to be either documenting on which error conditions the user needs to call txn_abort() and on which transactions (child vs parent, too). Or (what I would prefer) let the user clean up the transaction. In case this is indeed already documented I would appreciate a pointer to the location of this documentation. Thanks, Christopher -- http://gmerlin.de OpenPGP: http://gmerlin.de/christopher.pub CB07 DA40 B0B6 571D 35E2 0DEF 87E2 92A7 13E5 DEE1

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs