- openldap-bugs - openldap.org

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by hyc＠symas.com 21 Oct '19

21 Oct '19

markus(a)greenrobot.de wrote: > Full_Name: Markus Junginger > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (77.189.91.168) > > > We have a 1GB LMDB file with 7M K/V entries. With MDB_VL32, we always get a > MDB_TXN_FULL error for a transaction that is removing 2M entries (probably fails > at a lower count, we haven't measured that). Without MDB_VL32 it works fine. > > Another observation: > Once the transaction fails with MDB_TXN_FULL, the data file has grown to 1.5GB. > Without MDB_VL32, the data file stays consistent at 1 GB even if all 7M entries > are deleted in a single transaction. > > Expected behavior: > No MDB_TXN_FULL error, no data file growth. > > > I believe this was fixed by commit 7edf504106c61639a89b9a4e5987242598196932 in mdb.master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9099) LMDB: range delete
by hyc＠symas.com 21 Oct '19

21 Oct '19

opensource(a)gmx-topmail.de wrote: > Full_Name: Markus > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (91.64.106.220) > > > We built a database with LMDB and are very pleased with it in general. The one > area giving our users headaches regularly are bulk deletes. But some context > first: > > Like many others, we use a hierarchical key notation. Some simplified examples: > "user/00001", "user/00002", "user/00003", ...; "usergroup/00001", > "usergroup/00002", .... > > Or, let's say data by timestamp: "sensordata/32489732", "sensordata/32489733", > ... > > Use case like "remove all users" or "remove all sensordata entries from time x > to y" are usual. Those (transactional) operations call mdb_cursor_del() on each > entry. This runs fine until we have LOTS of entries to delete. At a magnitude of > 100,000s of entries, two things happen: 1) the time it takes to delete gets > quite noticeable and 2) the 32 bit (MDB_VL32) will run into MDB_TXN_FULL (see > also #8813). > > Given the key structure, those operations operate on consecutive keys. Because > of that, can we make this more efficient (and get rid MDB_TXN_FULL on the fly)? > > A high number of individual deletes involves lots of page merging (and a lot of > dirty pages). If we had a specialized "range delete" this could likely be > avoided. The signature of the proposed functionality could look like this: The number of pages dirtied will be the same, whether you batch or delete one by one. As such, this won't solve the MDB_TXN_FULL problem. Also, as of commit 7edf504106c61639a89b9a4e5987242598196932 this problem should have already been removed for MDB_VL32. > int mdb_cursor_del_range(MDB_cursor *mc, MDB_val *key_start, MDB_val *key_stop, > unsigned int flags) > > Given key_start and key_stop, this new function could decide to prune entire > leaf pages, or even entire sub branches. This should result in a significant > efficiency/performance gain. The challenge I'm seeing here is balancing the tree > though. Right, rebalancing would become quite a pain. A similar problem exists for PUT_MULTIPLE which is why it just internally loops over one item at a time. > What do you think? I'm not sure if OpenLdap would profit here, but I'm quite > certain that LMDB as a general database would. I'm also open to contribute if > you are willing to give some guidance. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9099) LMDB: range delete
by opensource＠gmx-topmail.de 21 Oct '19

21 Oct '19

Full_Name: Markus Version: OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (91.64.106.220) We built a database with LMDB and are very pleased with it in general. The one area giving our users headaches regularly are bulk deletes. But some context first: Like many others, we use a hierarchical key notation. Some simplified examples: "user/00001", "user/00002", "user/00003", ...; "usergroup/00001", "usergroup/00002", .... Or, let's say data by timestamp: "sensordata/32489732", "sensordata/32489733", ... Use case like "remove all users" or "remove all sensordata entries from time x to y" are usual. Those (transactional) operations call mdb_cursor_del() on each entry. This runs fine until we have LOTS of entries to delete. At a magnitude of 100,000s of entries, two things happen: 1) the time it takes to delete gets quite noticeable and 2) the 32 bit (MDB_VL32) will run into MDB_TXN_FULL (see also #8813). Given the key structure, those operations operate on consecutive keys. Because of that, can we make this more efficient (and get rid MDB_TXN_FULL on the fly)? A high number of individual deletes involves lots of page merging (and a lot of dirty pages). If we had a specialized "range delete" this could likely be avoided. The signature of the proposed functionality could look like this: int mdb_cursor_del_range(MDB_cursor *mc, MDB_val *key_start, MDB_val *key_stop, unsigned int flags) Given key_start and key_stop, this new function could decide to prune entire leaf pages, or even entire sub branches. This should result in a significant efficiency/performance gain. The challenge I'm seeing here is balancing the tree though. What do you think? I'm not sure if OpenLdap would profit here, but I'm quite certain that LMDB as a general database would. I'm also open to contribute if you are willing to give some guidance. Thanks for your consideration, Markus

1 0

Re: (ITS#9096)
by priit＠ww.ee 18 Oct '19

18 Oct '19

--000000000000c70a5505952b7d50 Content-Type: text/plain; charset="UTF-8" Also a remark I must add, this only happens in one of my 4 Manjaro installations, which is also the only one running on AMD system (Ryzen 1700 + ASRock AB350M PRO4 motherboard). --000000000000c70a5505952b7d50 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div>Also a remark I must add, this only = happens in one of my 4 Manjaro installations, which is also the only one ru= nning on AMD system (Ryzen 1700 + <span class=3D"gmail-il">ASRock</span> AB= 350M PRO4 motherboard).<br></div><br></div></div> --000000000000c70a5505952b7d50--

1 0

Re: (ITS#9098) assert fails in meta_back_search in some cases after reconnect
by quanah＠symas.com 16 Oct '19

16 Oct '19

--On Wednesday, October 16, 2019 8:37 PM +0000 maxime.besson(a)worteks.com wrote: > Full_Name: Maxime Besson > Version: 2.4.47 > OS: Debian Jessie > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (2a01:cb00:802:8400:2cbe:3c60:fca6:e50b) I suggest upgrading to 2.4.48 to pick up this fix: Fixed slapd-meta assertion when network interface goes down (ITS#8841) --Quaanh -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9098) assert fails in meta_back_search in some cases after reconnect
by maxime.besson＠worteks.com 16 Oct '19

16 Oct '19

Full_Name: Maxime Besson Version: 2.4.47 OS: Debian Jessie URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2a01:cb00:802:8400:2cbe:3c60:fca6:e50b) I am running a meta-directory with the following DB configuration. version 2.4.47, LTB build on Ubuntu 16.04 dn: olcDatabase={1}meta,cn=config objectClass: olcMetaConfig objectClass: olcDatabaseConfig objectClass: olcConfig objectClass: top olcDatabase: {1}meta olcSuffix: dc=com olcAccess: {0}to * by * read olcRootDN: cn=admin,dc=com dn: olcMetaSub={0}uri,olcDatabase={1}meta,cn=config objectClass: olcMetaTargetConfig objectClass: olcConfig objectClass: top olcMetaSub: {0}uri olcDbURI: ldap://1.2.3.4/dc=example,dc=com olcDbIDAssertBind: mode=legacy flags=non-prescriptive,proxy-authz-non-critical bindmethod=simple binddn="cn=admin,dc=example,dc=com" credentials="XXXXX" olcDbTimeout: 5 olcDbNetworkTimeout: 3 olcDbNretries: never olcDbRebindAsUser: true ... (There are 8 backends in total) Timeouts were added in order to avoid blocking OpenLDAP completely when one server becomes completely unavailable. However, since I added them, the slapd process started crashing every now and then (from a couple hours to a couple of days), usually during small network interruptions that affect all backends: I see plenty of reconnect logs shortly before the crashes. The crash is always immediately preceded by the following log message: meta_search_dobind_init[{i}]: retrying URI="{url}" DN="{DN}" {i} is never the same, and {url} and {DN} are the correct settings for backend i. The crash itself is an ABRT at the following assert in back-meta/search.c: 1957 assert( candidates[ i ].sr_msgid >= 0 1958 || candidates[ i ].sr_msgid == META_MSGID_CONNECTING ); I have analyzed several core dumps, and found that every single time slapd crashes, sr_msgid has a value of -1 (META_MSGID_IGNORE), which indeed causes the assert to fail. I found that candidates[i]->sr_flags has a value of 3 (META_CANDIDATE + META_BINDING) And the msc_mscflags in mc->mc_conns[ i ] are * 0x100081 for all connections before the one that triggers the crash * 0x100010 for the candidate that crashes the server * 0x100080 for all connections after it I am having trouble reproducing this in a test environment, but it happens regularly in production, I have tried changing the timeouts, adding a non-default bind timeout , and disabling retries (they were originally allowed) but the crashes keep happening. Note that disabling retries (olcDbNretries: never) still seems to lead to retries in meta_search_dobind_init, since the log message is still there. I cannot share the core dumps due to the sensitive information inside them. However I would gladly extract more information from them if it can help solving this.

1 0

Re: (ITS#9097) lmdb: premature free of env->me_txn0
by hyc＠symas.com 16 Oct '19

16 Oct '19

Christopher Zimmermann wrote: > Hi again, > > I did some further debugging and now I'm afraid I am indeed finishing a > transaction twice. I do call txn_commit() and after that I call > txn_abort(). But the MDB_MAP_FULL event happens during mdb_txn_commit(): > > (gdb) frame 0 > #0 mdb_page_alloc (mc=0x7f7ffffbd580, num=1, mp=0x7f7ffffbd228) at mdb.c:2286 > 2286 rc = MDB_MAP_FULL; > (gdb) bt > #0 mdb_page_alloc (mc=0x7f7ffffbd580, num=1, mp=0x7f7ffffbd228) at mdb.c:2286 > #1 0x00000fb6fb6aa239 in mdb_page_touch (mc=0x7f7ffffbd580) at mdb.c:2428 > #2 0x00000fb6fb6a9db3 in mdb_page_search (mc=0x7f7ffffbd580, key=0x0, flags=5) at mdb.c:5627 > #3 0x00000fb6fb69e585 in mdb_freelist_save (txn=0xfb6bfb3ae00) at mdb.c:3087 > #4 0x00000fb6fb69c2a4 in mdb_txn_commit (txn=0xfb6bfb3ae00) at mdb.c:3612 > > mdb_txn_commit() will return MDB_MAP_FULL. > > It may be worth adding MDB_MAP_FULL and MDB_BAD_TXN to the possible > error codes of mdb_txn_commit(), or in general provide some guidance on > how to clean up a txn after some failure. The doc for commit and abort is explicit. A txn must not be used again after calling commit or abort. There is nothing to clean up, the txn is always gone when commit or abort returns. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9097) lmdb: premature free of env->me_txn0
by madroach＠gmerlin.de 16 Oct '19

16 Oct '19

Hi again, I did some further debugging and now I'm afraid I am indeed finishing a transaction twice. I do call txn_commit() and after that I call txn_abort(). But the MDB_MAP_FULL event happens during mdb_txn_commit(): (gdb) frame 0 #0 mdb_page_alloc (mc=0x7f7ffffbd580, num=1, mp=0x7f7ffffbd228) at mdb.c:2286 2286 rc = MDB_MAP_FULL; (gdb) bt #0 mdb_page_alloc (mc=0x7f7ffffbd580, num=1, mp=0x7f7ffffbd228) at mdb.c:2286 #1 0x00000fb6fb6aa239 in mdb_page_touch (mc=0x7f7ffffbd580) at mdb.c:2428 #2 0x00000fb6fb6a9db3 in mdb_page_search (mc=0x7f7ffffbd580, key=0x0, flags=5) at mdb.c:5627 #3 0x00000fb6fb69e585 in mdb_freelist_save (txn=0xfb6bfb3ae00) at mdb.c:3087 #4 0x00000fb6fb69c2a4 in mdb_txn_commit (txn=0xfb6bfb3ae00) at mdb.c:3612 mdb_txn_commit() will return MDB_MAP_FULL. It may be worth adding MDB_MAP_FULL and MDB_BAD_TXN to the possible error codes of mdb_txn_commit(), or in general provide some guidance on how to clean up a txn after some failure. Christopher -- http://gmerlin.de OpenPGP: http://gmerlin.de/christopher.pub CB07 DA40 B0B6 571D 35E2 0DEF 87E2 92A7 13E5 DEE1

1 0

Re: (ITS#9097) lmdb: premature free of env->me_txn0
by madroach＠gmerlin.de 15 Oct '19

15 Oct '19

On Wed, Oct 16, 2019 at 01:59:51AM +0100, Howard Chu wrote: > christopher(a)gmerlin.de wrote: > > Full_Name: Christopher Zimmermann > > Version: lmdb 0.9.24 > > OS: OpenBSD > > URL: ftp://ftp.openldap.org/incoming/ > > Submission from: (NULL) (85.212.180.240) > > > > > > Hi, > > > > I can reliably hit a Bus error on OpenBSD. > > This is triggered by OpenBSDs malloc/free junking [1] and a use-after-free bug > > in lmdb. > > > > Steps to reproduce: > > - begin a read/write transaction (getting env->me_txn0) > > - fill the environment > > -> returns MDB_MAP_FULL > > -> sets txn->mt_flags |= MDB_TXN_ERROR; (This is also env->me_txn0 !) > > -> calls mdb_txn_abort > ... > > - abort the transaction (again) with mdb_abort() > > This is a bug in your code, you can't call txn_abort twice. This is > already documented. Closing this ITS. Hi, thanks for having a look. I could not find any documentation about when one must / must not call mdb_txn_abort. But in any case I do _not_ call txn_abort() twice. The problem rather seems to be that lmdb _implicitely_ frees invalid transactions. And my code then aborts them again _explicitely_. So mdb_abort was indeed called twice. Once in the MDB_TXN_ERROR internally in lmdb case and once from my code. On second thought my fix won't fix the same problem for errored child transactions. What is necessary seems to be either documenting on which error conditions the user needs to call txn_abort() and on which transactions (child vs parent, too). Or (what I would prefer) let the user clean up the transaction. In case this is indeed already documented I would appreciate a pointer to the location of this documentation. Thanks, Christopher -- http://gmerlin.de OpenPGP: http://gmerlin.de/christopher.pub CB07 DA40 B0B6 571D 35E2 0DEF 87E2 92A7 13E5 DEE1

1 0

Re: (ITS#9097) lmdb: premature free of env->me_txn0
by hyc＠symas.com 15 Oct '19

15 Oct '19

christopher(a)gmerlin.de wrote: > Full_Name: Christopher Zimmermann > Version: lmdb 0.9.24 > OS: OpenBSD > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (85.212.180.240) > > > Hi, > > I can reliably hit a Bus error on OpenBSD. > This is triggered by OpenBSDs malloc/free junking [1] and a use-after-free bug > in lmdb. > > Steps to reproduce: > - begin a read/write transaction (getting env->me_txn0) > - fill the environment > -> returns MDB_MAP_FULL > -> sets txn->mt_flags |= MDB_TXN_ERROR; (This is also env->me_txn0 !) > -> calls mdb_txn_abort ... > - abort the transaction (again) with mdb_abort() This is a bug in your code, you can't call txn_abort twice. This is already documented. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9097) lmdb: premature free of env->me_txn0
by christopher＠gmerlin.de 15 Oct '19

15 Oct '19

Full_Name: Christopher Zimmermann Version: lmdb 0.9.24 OS: OpenBSD URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (85.212.180.240) Hi, I can reliably hit a Bus error on OpenBSD. This is triggered by OpenBSDs malloc/free junking [1] and a use-after-free bug in lmdb. Steps to reproduce: - begin a read/write transaction (getting env->me_txn0) - fill the environment -> returns MDB_MAP_FULL -> sets txn->mt_flags |= MDB_TXN_ERROR; (This is also env->me_txn0 !) -> calls mdb_txn_abort -> calls mdb_txn_end(txn, MDB_END_ABORT|MDB_END_SLOT|MDB_END_FREE): mdb_txn_end tries not to free env->me_txn0: } else if (!F_ISSET(txn->mt_flags, MDB_TXN_FINISHED)) { [...] txn->mt_flags = MDB_TXN_FINISHED; if (!txn->mt_parent) { [...] mdb.c:3020 mode = 0; /* txn == env->me_txn0, do not free() it */ [...] } [...] } if (mode & MDB_END_FREE) free(txn); this prevents the free only for unfinished transactions. Unfinished transactions are now finished. - abort the transaction (again) with mdb_abort() -> calls mdb_txn_end(txn, MDB_END_ABORT|MDB_END_SLOT|MDB_END_FREE): since the env->me_txn0 detection is skipped on MDB_TXN_FINISHED transactions the transaction is freed, the memory will get "junked". - begin a read/write transaction (getting now invalid env->me_txn0) -> calls mdb_txn_renew0(env->me_txn0) MDB_env *env = txn->mt_env; /* txn->mt_env is now invalid */ MDB_txninfo *ti = env->me_txns; /* triggers bus error */ Please make the protection against freeing of me_txn0 more robust. Thanks, Christopher Zimmermann [1] https://man.openbsd.org/malloc#j

1 0

Re: (ITS#9091) mdb attribute get mixed up in slapadd continue (-c) mode
by maxime.besson＠worteks.com 14 Oct '19

14 Oct '19

On 10/14/19 8:47 PM, Howard Chu wrote: > > I'd also note that using -c has always been pretty risky. You would be better off using -j > in situations like this, as show below: > Thanks for the fix and tip!

1 0

Re: (ITS#9096)
by priit＠ww.ee 14 Oct '19

14 Oct '19

--0000000000009f14730594e762f7 Content-Type: text/plain; charset="UTF-8" [deemon@Zen ~]$ pkg-config --modversion lmdb 0.9.24 could this be the version number you are looking for? - Priit On Mon, 14 Oct 2019 at 20:27, Priit Oorn <priit(a)ww.ee> wrote: > Don't know the version or how to find out, but I zipped it for you so you > can maybe find out yourself? :-) > Or give me some hints how to find out with what command? > > - Priit > --0000000000009f14730594e762f7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">[deemon@Zen ~]$ pkg-config --modversion lmdb<br>0.9.24<br>= <div><div dir=3D"ltr" class=3D"gmail_signature" data-smartmail=3D"gmail_sig= nature"><br></div><div class=3D"gmail_signature" data-smartmail=3D"gmail_si= gnature">could this be the version number you are looking for?<br></div><di= v dir=3D"ltr" class=3D"gmail_signature" data-smartmail=3D"gmail_signature">= <br>- Priit</div></div><br></div><br><div class=3D"gmail_quote"><div dir=3D= "ltr" class=3D"gmail_attr">On Mon, 14 Oct 2019 at 20:27, Priit Oorn <<a = href=3D"mailto:priit@ww.ee">priit(a)ww.ee</a>> wrote:<br></div><blockquote= class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px so= lid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div>Don't know= the version or how to find out, but I zipped it for you so you can maybe f= ind out yourself? :-)</div><div>Or give me some hints how to find out with = what command?<br></div><div><div><div><div dir=3D"ltr"><br>- Priit</div></d= iv></div></div></div> </blockquote></div> --0000000000009f14730594e762f7--

1 0

Re: (ITS#9091) mdb attribute get mixed up in slapadd continue (-c) mode
by hyc＠symas.com 14 Oct '19

14 Oct '19

Howard Chu wrote: > maxime.besson(a)worteks.com wrote: >> Full_Name: Maxime Besson >> Version: 2.4.48 >> OS: Linux >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (77.193.139.162) >> >> >> I am attempting to implement the following disaster recovery process: >> >> * rm all previous data >> * run a configuration script (puppet) to recreate a bare-bones LDAP server and >> DIT >> * restore a backed-up slapcat dump on top of the freshly installed OpenLDAP >> server, ignoring duplicates already inserted by Puppet >> >> But it seems that when skipping over the existing objects, something goes wrong >> and causes attributes to get mixed up. But only when certain attributes are >> present in the existing objects. Here is how to reproduce: > > Thanks for the report and simple test case. This is now fixed in git master. I'd also note that using -c has always been pretty risky. You would be better off using -j in situations like this, as show below: >> >> puppet_init.ldif >> === >> dn: dc=example,dc=com >> objectClass: domain >> dc: example >> === >> >> backup.ldif >> === >> dn: dc=example,dc=com >> objectClass: domain >> dc: example >> contextCSN: 20190909094705.796552Z#000000#001#000000 >> >> dn: uid=ttully,dc=example,dc=com >> objectClass: inetOrgPerson >> uid: ttully >> userPassword:: c2Nob29uZXI= >> facsimileTelephoneNumber: +1 408 555 0111 >> givenName: Torrey >> cn: Torrey Tully >> telephoneNumber: +1 408 555 2274 >> sn: Tully >> roomNumber: 3924 >> mail: ttully(a)example.com >> l: Sunnyvale >> ou: Human Resources >> ou: People >> === >> >> When running the following commands: >> >> === >> rm -f /tmp/*.mdb >> slapadd -f slapd.conf puppet_init.ldif >> slapadd -f slapd.conf -c backup.ldif >> === slapadd -f slapd.conf -l puppet_init.ldif slapadd -f slapd.conf -l backup.ldif -j 5 -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9091) mdb attribute get mixed up in slapadd continue (-c) mode
by hyc＠symas.com 14 Oct '19

14 Oct '19

maxime.besson(a)worteks.com wrote: > Full_Name: Maxime Besson > Version: 2.4.48 > OS: Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (77.193.139.162) > > > I am attempting to implement the following disaster recovery process: > > * rm all previous data > * run a configuration script (puppet) to recreate a bare-bones LDAP server and > DIT > * restore a backed-up slapcat dump on top of the freshly installed OpenLDAP > server, ignoring duplicates already inserted by Puppet > > But it seems that when skipping over the existing objects, something goes wrong > and causes attributes to get mixed up. But only when certain attributes are > present in the existing objects. Here is how to reproduce: Thanks for the report and simple test case. This is now fixed in git master. > > > slapd.conf: > === > include /etc/ldap/schema/core.schema > include /etc/ldap/schema/cosine.schema > include /etc/ldap/schema/inetorgperson.schema > moduleload back_mdb > database mdb > maxsize 1073741824 > suffix "dc=example,dc=com" > directory /tmp > === > > > puppet_init.ldif > === > dn: dc=example,dc=com > objectClass: domain > dc: example > === > > backup.ldif > === > dn: dc=example,dc=com > objectClass: domain > dc: example > contextCSN: 20190909094705.796552Z#000000#001#000000 > > dn: uid=ttully,dc=example,dc=com > objectClass: inetOrgPerson > uid: ttully > userPassword:: c2Nob29uZXI= > facsimileTelephoneNumber: +1 408 555 0111 > givenName: Torrey > cn: Torrey Tully > telephoneNumber: +1 408 555 2274 > sn: Tully > roomNumber: 3924 > mail: ttully(a)example.com > l: Sunnyvale > ou: Human Resources > ou: People > === > > When running the following commands: > > === > rm -f /tmp/*.mdb > slapadd -f slapd.conf puppet_init.ldif > slapadd -f slapd.conf -c backup.ldif > === > > The redundant root object in backup.ldif gets skipped as -c should do, but the > attributes from my "ttully" user (and every following object in the ldif dump) > end up all mixed up: > > > === > slapcat -f slapd.conf > ... > dn: uid=ttully,dc=example,dc=com > objectClass: inetOrgPerson > userPassword:: dHR1bGx5 > facsimileTelephoneNumber: schooner > givenName: +1 408 555 0111 > cn: Torrey > telephoneNumber: Torrey Tully > sn: +1 408 555 2274 > roomNumber: Tully > mail: 3924 > l: ttully(a)example.com > ou: Sunnyvale > ou: Human Resources > ou: People > ... > === > > > This mixup does NOT happen if: > > * I import backup.ldif into an empty database > * OR I remove "contextCSN" from the backup ldif > * OR I use BDB instead of MDB > > So it seems that my issue is caused by a combination of MDB, skipping existing > entries, and having special attributes (contextCSN) in the skipped objects. > > I was able to reproduce this behavior: > * On Debian Buster (OpenLDAP 2.4.47) > * On RHEL7 + LTB project RPMs 2.4.48 > * Using a git snapshot (3be82f40d5cd4ca050e10859ecb961f28c807c41) with no > particular config options > * Using cn=config rather than slapd.conf > * On a real production system, rather than the simplified version presented > here. > * Using another "special" attribute such as pwdAccountLockedTime instead of > "contextCSN" > > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9096) tons of crashing and core dumps after every boot
by hyc＠symas.com 14 Oct '19

14 Oct '19

priit(a)ww.ee wrote: > Full_Name: Priit Oorn > Version: no idea > OS: Manjaro KDE stable > URL: https://bugsfiles.kde.org/attachment.cgi?id=123175 > Submission from: (NULL) (2001:7d0:8240:e600:7a6:c9cf:974a:2e6e) > > > STEPS TO REPRODUCE > 1. reboot system > 2. journalctl -p3 > > OBSERVED RESULT > KDE baloo_file and baloo_file_extr serial crashing due to LMDB crashing. > > EXPECTED RESULT > no services crashing obviously. > > SOFTWARE/OS VERSIONS > Linux/KDE Plasma: Manjaro Linux KDE stable > KDE Plasma Version: 5.16.5 > KDE Frameworks Version: 5.62.0 > Qt Version: 5.13.1 > LBDM version: no idea... that comes with Manjaro KDE stable default install + > all updates. the crashing thing has been there forever though over several > versions. > > reported the bug to KDE initially, but they say it's your fault... :P > https://bugs.kde.org/show_bug.cgi?id=412925 The stack traces you provided there https://bugsfiles.kde.org/attachment.cgi?id=123175 would indicate a crash in LMDB, but without providing version info we have nothing to go on. Nor do we have any insight into how KDE integrated LMDB and whether they used its API correctly. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9096) tons of crashing and core dumps after every boot
by priit＠ww.ee 14 Oct '19

14 Oct '19

Full_Name: Priit Oorn Version: no idea OS: Manjaro KDE stable URL: https://bugsfiles.kde.org/attachment.cgi?id=123175 Submission from: (NULL) (2001:7d0:8240:e600:7a6:c9cf:974a:2e6e) STEPS TO REPRODUCE 1. reboot system 2. journalctl -p3 OBSERVED RESULT KDE baloo_file and baloo_file_extr serial crashing due to LMDB crashing. EXPECTED RESULT no services crashing obviously. SOFTWARE/OS VERSIONS Linux/KDE Plasma: Manjaro Linux KDE stable KDE Plasma Version: 5.16.5 KDE Frameworks Version: 5.62.0 Qt Version: 5.13.1 LBDM version: no idea... that comes with Manjaro KDE stable default install + all updates. the crashing thing has been there forever though over several versions. reported the bug to KDE initially, but they say it's your fault... :P https://bugs.kde.org/show_bug.cgi?id=412925

1 0

Re: (ITS#9018) dynlist don't close connection
by quanah＠symas.com 14 Oct '19

14 Oct '19

Decoded we have: "I have changed back-meta to the back-ldap, and it resolved this problem." --On Tuesday, September 24, 2019 11:59 AM +0000 I.Harbuz(a)A1.by wrote: -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9095) back-mdb slapindex missing records
by hyc＠openldap.org 11 Oct '19

11 Oct '19

Full_Name: Howard Chu Version: 2.4 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (84.203.26.174) Submitted by: hyc In back-mdb the entry reindexer operates in batches of 500 entries at a time. (I.e., it commits one LMDB txn for every 500 entries.) But a final txn_commit is missing in tool_entry_close() to flush out the last few entries that were operated on.

1 0

Re: (ITS#9092) LDAP server out of memory
by quanah＠symas.com 10 Oct '19

10 Oct '19

--On Wednesday, October 9, 2019 9:38 AM +0000 gihanridal8(a)gmail.com wrote: > Full_Name: Gihan De Silva > Version: 2.4.44 > OS: 7.5 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (123.231.119.138) Hello, The OpenLDAP bug tracking system is for bug reports only. Usage questions should be directed to the OpenLDAP Technical discussion list: <https://www.openldap.org/lists/mm/listinfo/openldap-technical> If you have an urgent support need, then I suggest contacting one of the listed 3rd party support providers at: <https://www.openldap.org/support/> This ITS will be closed. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

RE: (ITS#9088) Concurrency not seen in OpenLDAP 2.4.44
by jrajalakshmi＠juniper.net 10 Oct '19

10 Oct '19

SGksDQpXZSB3aWxsIHNlbmQgdGhlIHNhbXBsZSBjb2RlIHNob3J0bHkuDQoNClRoYW5rcywNClJh amkNCg0KDQpKdW5pcGVyIEJ1c2luZXNzIFVzZSBPbmx5DQoNCi0tLS0tT3JpZ2luYWwgTWVzc2Fn ZS0tLS0tDQpGcm9tOiBPbmTFmWVqIEt1em7DrWsgPG9uZHJhQG1pc3RvdGViZS5uZXQ+IA0KU2Vu dDogVHVlc2RheSwgT2N0b2JlciA4LCAyMDE5IDEwOjQ1IFBNDQpUbzogUmFqYWxha3NobWkgSmF5 YXJhbWFuIDxqcmFqYWxha3NobWlAanVuaXBlci5uZXQ+DQpDYzogb3BlbmxkYXAtaXRzQE9wZW5M REFQLm9yZw0KU3ViamVjdDogUmU6IChJVFMjOTA4OCkgQ29uY3VycmVuY3kgbm90IHNlZW4gaW4g T3BlbkxEQVAgMi40LjQ0DQoNCk9uIFR1ZSwgT2N0IDA4LCAyMDE5IGF0IDEyOjU2OjA0UE0gKzAw MDAsIGpyYWphbGFrc2htaUBqdW5pcGVyLm5ldCB3cm90ZToNCj4gSGksDQo+IA0KPiBUaGFua3Mg Zm9yIHRoZSByZXBseS4gVGhlcmUgaXMgYW4gaW1wbGVtZW50YXRpb24gYWxyZWFkeSBpbiBvdXIg cHJvZHVjdC4NCg0KSGksDQphcyBtZW50aW9uZWQgcHJldmlvdXNseSwgcGxlYXNlIHByb3ZpZGUg YSBzYW1wbGUgQyBwcm9ncmFtIHRoYXQgc2hvd3Mgd2hhdCdzIGhhcHBlbmluZyBhbmQgcHJlZmVy YWJseSBvbmUgdGhhdCBhbHNvIGJlaGF2ZXMgYmV0dGVyIHdpdGggMi40LjIzIGZyb20geW91ciBw ZXJzcGVjdGl2ZSBzbyB3ZSBjYW4gaGVscCB5b3UgaW52ZXN0aWdhdGUuIEkgZG9uJ3QgYmVsaWV2 ZSBwYWNrZXQgY2FwdHVyZXMgd291bGQgaGVscCB1cyBoZXJlIGF0IHRoaXMgcG9pbnQuDQoNCj4g VG8gc3VtbWFyaXplIG9uIHRoZSBiZWhhdmlvciBvYnNlcnZlZA0KPiANCj4gU2NlbmFyaW8jMToN Cj4gDQo+IFNpbmdsZSBMREFQIHNlc3Npb24gd2l0aCBMREFQIEJpbmQgZG9lcyBhdXRoZW50aWNh dGlvbiBpbiBTZXF1ZW50aWFsIA0KPiBPcmRlci4gRHVlIHRvIHNlcXVlbnRpYWwgZXhlY3V0aW9u IG9mIE9wZW5MREFQIFNlYXJjaCBBUEkgYWZ0ZXIgDQo+IHZlcnNpb24gMi40LjI0IHdlIG9ic2Vy dmUgcGFja2V0IGRyb3BzIGluIExEQVAgQXV0aGVudGljYXRpb24gUmVxdWVzdC4NCj4gVGhpcyBp cyByZXN1bHRpbmcgaW4gTG93IHBlcmZvcm1hbmNlIG9mIG91ciBwcm9kdWN0Lg0KDQpQcmV2aW91 c2x5IHlvdSBtZW50aW9uZWQgb25seSBzZWFyY2hlcywgd2hpY2ggc2hvdWxkIHdvcmsgb3ZlciB0 aGUgc2FtZSBjb25uZWN0aW9uIGRlcGVuZGluZyBob3cgeW91IGRvIHRoaW5ncy4gTm90IHN1cmUg aG93IGJpbmRzIGZpdCBpbiB0aGUgcGljdHVyZSBoZXJlPyBKdXN0IGEgbm90ZSB0aGF0IHlvdSBj YW4gb25seSBoYXZlIGNlcnRhaW4gdHlwZXMgb2Ygb3BlcmF0aW9ucyBwZW5kaW5nIGF0IHRoZSBz YW1lIHRpbWUgYW5kIGJpbmRzIGFyZSBub3Qgb25lIG9mIHRoZW0uDQoNCkFsc28gZG9uJ3Qga25v dyB3aGF0IHlvdSBtZWFuIGJ5IHBhY2tldCBkcm9wcyBoZXJlLiBJcyB0aGF0IHBhY2tldCBkcm9w cyBpbiB5b3VyIHByb2R1Y3Qgb3IgcGFja2V0IGRyb3BzIG9uIHRoZSBMREFQIHNlc3Npb24/DQoN Cj4gU2NlbmFyaW8gIzI6ICh0aGlzIGlzIGEgc2FtcGxlIHRlc3QgY2FzZSB0cmllZCB0byBjb21w YXJlIHRoZSBiZWhhdmlvciANCj4gd2l0aCBTY2VuYXJpbyMxKQ0KPiANCj4gVGhlcmUgYXJlIHNh eSAxMCBMREFQIGluc3RhbmNlcyBjb25maWd1cmVkLCBhbGwgcG9pbnRpbmcgdG8gdGhlIHNhbWUg DQo+IExEQVAgc2VydmVyLiBUaGUgcmVxdWVzdC1yZXNwb25zZSBpcyBoYW5kbGVkIHByb3Blcmx5 LCBwZXJmb3JtYW5jZSBpcyANCj4gaW1wcm92ZWQgd2hlbiBjb21wYXJlZCB0byBzY2VuYXJpbyMx LiBCdXQgdGhlIGRyYXdiYWNrIGlzIHRoZSBtdWx0aXBsZSANCj4gdGhyZWFkcyBmb3IgZXZlcnkg TERBUCBjbGllbnQgc2Vzc2lvbiwgdXRpbGl6aW5nIG1vcmUgbWVtb3J5LiBUaGlzIA0KPiBkZXNp Z24gbWF5IG5vdCBiZSByZWNvbW1lbmRlZCBmb3Igb3VyIHByb2R1Y3QuDQoNCllvdSBjYW4gaGF2 ZSBhIHNpbmdsZSB0aHJlYWQgdGFraW5nIGNhcmUgb2YgdGhlIHNlc3Npb24gKG9yIGhvd2V2ZXIg bWFueSBzZXNzaW9ucyB5b3Ugd2FudCkgaWYgeW91IHVzZSBhIG5vbi1ibG9ja2luZyBzb2NrZXQg Zm9yIHRoZSB1bmRlcmx5aW5nDQpjb25uZWN0aW9uKHMpIChhbmQgdXNlIGxkYXBfcmVzdWx0IGFu ZCBmcmllbmRzKSB3aGljaCBpcyB3aGF0IHRoZSBhc3luY2hyb25vdXMgQVBJIGhhcyBiZWVuIGRl c2lnbmVkIGZvci4NCg0KPiBQbC4gcmVwbHkgYmFjayBmb3IgYW55IGZ1cnRoZXIgcXVlcmllcyAv IGNsYXJpZmljYXRpb25zLiBUaGUgcGFja2V0IA0KPiBjYXB0dXJlcyBmaWxlIHNpemUgaXMgYmln ICg+Nk1CKSwgY2FuIHRyeSB0byBzaGFyZSB0aGUgcmVsZXZhbnQgDQo+IHJlcXVlc3QtcmVzcG9u c2UgZGV0YWlscy4uLi4uLg0KDQpSZWdhcmRzLA0KDQotLQ0KT25kxZllaiBLdXpuw61rDQpTZW5p b3IgU29mdHdhcmUgRW5naW5lZXINClN5bWFzIENvcnBvcmF0aW9uICAgICAgICAgICAgICAgICAg ICAgICBodHRwczovL3VybGRlZmVuc2UuY29tL3YzL19faHR0cDovL3d3dy5zeW1hcy5jb21fXzsh OFdvQTZSakM4MWMhVDNyV05tZTYzRlJFcmJzZ3I4RDVvaV9WVU9HWEY3YVFXSTZFckswMWp0elVM Nm9tcmhXSXFkay1ySDllUGxCald6UXUkIA0KUGFja2FnZWQsIGNlcnRpZmllZCwgYW5kIHN1cHBv cnRlZCBMREFQIHNvbHV0aW9ucyBwb3dlcmVkIGJ5IE9wZW5MREFQDQo=

1 0

(ITS#9094) LDAP server out of memory
by gihanridal8＠gmail.com 09 Oct '19

09 Oct '19

Full_Name: Gihan De Silva Version: 2.4.44 OS: 7.5 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (123.231.119.138) Dear Sir/ Madam, We are having two OpenLDAP servers on production with master master data synchronization. We are facing an incident where both servers consume memory upto 99% and LDAP server2 leads to not responding state. As a workaround we have restart both OpenLDAP nodes. However after few days the memory goes high at its peak. There are about 100k+ requests for one LDAP server per day. where both servers are identical in terms of 16 GB RAM, 4 CPUs, CentOS7 OS and LDAP server 2.4.44. We didnt observe any suspiciuos errors in the logs as well. Appreciate your valued consultation and recomendations on this incident for a RCA and a solution. Thank You Regards, Gihan.

1 0

(ITS#9093) LDAP server out of memory
by gihanridal8＠gmail.com 09 Oct '19

09 Oct '19

Full_Name: Gihan De Silva Version: 2.4.44 OS: 7.5 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (123.231.119.138) Dear Sir/ Madam, We are having two OpenLDAP servers on production with master master data synchronization. We are facing an incident where both servers consume memory upto 99% and LDAP server2 leads to not responding state. As a workaround we have restart both OpenLDAP nodes. However after few days the memory goes high at its peak. There are about 100k+ requests for one LDAP server per day. where both servers are identical in terms of 16 GB RAM, 4 CPUs, CentOS7 OS and LDAP server 2.4.44. We didnt observe any suspiciuos errors in the logs as well. Appreciate your valued consultation and recomendations on this incident for a RCA and a solution. Thank You Regards, Gihan.

1 0

(ITS#9092) LDAP server out of memory
by gihanridal8＠gmail.com 09 Oct '19

09 Oct '19

Full_Name: Gihan De Silva Version: 2.4.44 OS: 7.5 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (123.231.119.138) Dear Sir/ Madam, We are having two OpenLDAP servers on production with master master data synchronization. We are facing an incident where both servers consume memory upto 99% and LDAP server2 leads to not responding state. As a workaround we have restart both OpenLDAP nodes. However after few days the memory goes high at its peak. There are about 100k+ requests for one LDAP server per day. where both servers are identical in terms of 16 GB RAM, 4 CPUs, CentOS7 OS and LDAP server 2.4.44. We didnt observe any suspiciuos errors in the logs as well. Appreciate your valued consultation and recomendations on this incident for a RCA and a solution. Thank You Regards, Gihan.

1 0

(ITS#9091) mdb attribute get mixed up in slapadd continue (-c) mode
by maxime.besson＠worteks.com 08 Oct '19

08 Oct '19

Full_Name: Maxime Besson Version: 2.4.48 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (77.193.139.162) I am attempting to implement the following disaster recovery process: * rm all previous data * run a configuration script (puppet) to recreate a bare-bones LDAP server and DIT * restore a backed-up slapcat dump on top of the freshly installed OpenLDAP server, ignoring duplicates already inserted by Puppet But it seems that when skipping over the existing objects, something goes wrong and causes attributes to get mixed up. But only when certain attributes are present in the existing objects. Here is how to reproduce: slapd.conf: === include /etc/ldap/schema/core.schema include /etc/ldap/schema/cosine.schema include /etc/ldap/schema/inetorgperson.schema moduleload back_mdb database mdb maxsize 1073741824 suffix "dc=example,dc=com" directory /tmp === puppet_init.ldif === dn: dc=example,dc=com objectClass: domain dc: example === backup.ldif === dn: dc=example,dc=com objectClass: domain dc: example contextCSN: 20190909094705.796552Z#000000#001#000000 dn: uid=ttully,dc=example,dc=com objectClass: inetOrgPerson uid: ttully userPassword:: c2Nob29uZXI= facsimileTelephoneNumber: +1 408 555 0111 givenName: Torrey cn: Torrey Tully telephoneNumber: +1 408 555 2274 sn: Tully roomNumber: 3924 mail: ttully(a)example.com l: Sunnyvale ou: Human Resources ou: People === When running the following commands: === rm -f /tmp/*.mdb slapadd -f slapd.conf puppet_init.ldif slapadd -f slapd.conf -c backup.ldif === The redundant root object in backup.ldif gets skipped as -c should do, but the attributes from my "ttully" user (and every following object in the ldif dump) end up all mixed up: === slapcat -f slapd.conf ... dn: uid=ttully,dc=example,dc=com objectClass: inetOrgPerson userPassword:: dHR1bGx5 facsimileTelephoneNumber: schooner givenName: +1 408 555 0111 cn: Torrey telephoneNumber: Torrey Tully sn: +1 408 555 2274 roomNumber: Tully mail: 3924 l: ttully(a)example.com ou: Sunnyvale ou: Human Resources ou: People ... === This mixup does NOT happen if: * I import backup.ldif into an empty database * OR I remove "contextCSN" from the backup ldif * OR I use BDB instead of MDB So it seems that my issue is caused by a combination of MDB, skipping existing entries, and having special attributes (contextCSN) in the skipped objects. I was able to reproduce this behavior: * On Debian Buster (OpenLDAP 2.4.47) * On RHEL7 + LTB project RPMs 2.4.48 * Using a git snapshot (3be82f40d5cd4ca050e10859ecb961f28c807c41) with no particular config options * Using cn=config rather than slapd.conf * On a real production system, rather than the simplified version presented here. * Using another "special" attribute such as pwdAccountLockedTime instead of "contextCSN"

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs