openldap-bugs October 2019

openldap-bugs@openldap.org

20 participants
54 discussions

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by markus＠greenrobot.de 24 Oct '19

24 Oct '19

On 24.10.19 15:03, Markus Junginger wrote: > per transaction MDB_TRPAGE_SIZE bytes is allocated I guess, I was too quick here. It's MDB_TRPAGE_SIZE * 16. Also, I guess, txn->mt_rpages are just pointers to in-memory cached pages (or chunks?), so the memory consumption of those is much higher. The docs mention "chunks of 16 pages"; does that refer to mt_rpages? If so, a single mt_rpages pointer can actually hold on to 16 * 4K = 64 KB? Thus a MDB_TRPAGE_SIZE 4096 can go up to 256 MB memory? That changes the picture of course... PS.: let me express the irony once more that "removing data" is the most costly operation here ;-)

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by markus＠greenrobot.de 24 Oct '19

24 Oct '19

On 24.10.19 03:49, Howard Chu wrote: > You're free to define MDB_TRPAGE_MAX to a larger value. It just means > you increase the chance of overrunning the 2GB available address space. > There's no magic, you can't fit every 64bit database workload into only > a 32bit address space. When your transactions are too large, the normal > thing to do is commit more often so they don't grow so large. I think you meant MDB_TRPAGE_SIZE? At least that seemed to work, while MDB_TRPAGE_MAX ended up in another MDB_TXN_FULL (tl[0].mid < MDB_TRPAGE_SIZE check failed). Doubling MDB_TRPAGE_SIZE also doubled the threshold of the object count where it starts failing: using 8192 it was able to remove 4M entries while the limit with 4096 was 2M entries. MDB_TRPAGE_SIZE is only used to malloc txn->mt_rpages (and some checks), as far as I can tell. To make a reasonable decision here, could you please confirm: 1. txn->mt_rpages is RAM only and has no impact on mmaped sections (and/or file format). 2. RAM consumption is defined by number of transactions only: per transaction MDB_TRPAGE_SIZE bytes is allocated. Other than that it does not increase memory consumption. 3. There is no other disadvantage other than higher memory consumption per transaction as written in the previous point. In that case, I think I'll go with 16K (or even 32K). Does not seem too much for an transaction, given that it quadruples the amount of data allowed to be processed. 32 bit devices are not servers, so I do not expect a high number of concurrent transactions anyway. Thanks! Markus

1 0

(ITS#9100) Issue with domainScope controlValue implementation
by philip.brusten＠kuleuven.be 24 Oct '19

24 Oct '19

Full_Name: Philip Brusten Version: 2.4.47 OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2a02:2c40:100:b210::1:c3bc) Hi we are experiencing issues with MS LDAP clients that connect to our OpenLDAP servers since v2.4.47, because of the fix for issue ITS#8840 That fix was based on (old?) documentation of MS from https://msdn.microsoft.com/en-us/library/aa366979%28v=vs.85%29.aspx The is no current version of this document. At https://docs.microsoft.com/en-us/windows/win32/api/_ldap/ there is a link to Lightweigt Directory Access Protocol (https://docs.microsoft.com/en-us/windows/desktop/ldap), but it gives a 404. MS is the owner of that LDAPControl LDAP_SERVER_DOMAIN_SCOPE_OID 1.2.840.113556.1.4.1339. According to the RFC4511 [1] the controlValue is defined by its specification: > The controlValue may contain information associated with the > controlType. Its format is defined by the specification of the > control. Implementations MUST be prepared to handle arbitrary > contents of the controlValue octet string, including zero bytes. It > is absent only if there is no value information that is associated > with a control of its type. When a controlValue is defined in terms > of ASN.1, and BER-encoded according to Section 5.1, it also follows > the extensibility rules in Section 4. The latest specification can be found at: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/ba5f20… > "When sending this control to the DC, the controlValue field of the Control structure is omitted." We observed that whenever MS clients are sending the domainScope control they add an octet string as the controlValue with bytes "04 00". But this causes the OpenLDAP code if ( !BER_BVISNULL( &ctrl->ldctl_value )) { to evaluate to true, which returns the LDAP_PROTOCOL_ERROR in OpenLDAP. The LDAP Extended Control spec of MS states [2]: > If the controlValue field contains data that is not in conformance with the > specification of the control, including the case where the controlValue field contains > data and the specification of the control states that the controlValue field is omitted, > then if the control is marked critical the server returns the error > unavailableCriticalExtension / ERROR_INVALID_PARAMETER. If the controlValue field is > incorrect but the control is not marked critical, the server ignores the control. So the server should only return an error when the control is marked critical. Otherwise it should *ignore* the control. Is it possible to relax the validation of the LDAP control at [3]? Since there is no value associated with this control, it should not matter if the controlValue is missing or an empty octet string {0,NULL}... The ldapsearch code sets the controlValue to [4]: > c[i].ldctl_value.bv_val = NULL; > c[i].ldctl_value.bv_len = 0; We observed that the controlValue is than missing via Wireshark. So we see a difference on the line for empty octet strings, but should it matter? Wireshark trace of MS LDAP client with domainScope control: https://i.imgur.com/gwBjLMr.png Wireshark trace of ldapsearch with domainScope control: https://i.imgur.com/2RbFkIY.png Regards, Philip [1] https://tools.ietf.org/html/rfc4511#section-4.1.11 [2] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/3c5e87… [3] https://github.com/openldap/openldap/blob/master/servers/slapd/controls.c#L… [4] https://github.com/openldap/openldap/blob/master/clients/tools/ldapsearch.c…

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by hyc＠symas.com 23 Oct '19

23 Oct '19

markus(a)greenrobot.de wrote: > On 21.10.19 17:14, Howard Chu wrote: >> I believe this was fixed by commit=20 >> 7edf504106c61639a89b9a4e5987242598196932 in mdb.master.=20 >=20 > I can not confirm that this works. >=20 > This is the stack trace where MDB_TXN_FULL is still returned with lates= t=20 > mdb.master (note: line numbers shown here are off by 60 compared to=20 > mdb.master): >=20 > mdb_rpage_get mdb.c:6196 > mdb_page_get mdb.c:6378 > mdb_page_search_lowest mdb.c:6492 > mdb_node_move mdb.c:8842 > mdb_rebalance mdb.c:9366 > mdb_page_merge mdb.c:9166 > mdb_rebalance mdb.c:9373 > mdb_cursor_del0 mdb.c:9426 > mdb_cursor_del mdb.c:811 >=20 >=20 > Code segment: >=20 > if (tl[0].mid >=3D MDB_TRPAGE_MAX) > =C3=82=C2=A0=C3=82=C2=A0=C3=82=C2=A0 return MDB_TXN_FULL; >=20 > Debugger shows tl[0].mid to be 4095. You're free to define MDB_TRPAGE_MAX to a larger value. It just means you increase the chance of overrunning the 2GB available address space. There's no magic, you can't fit every 64bit database workload into only a 32bit address space. When your transactions are too large, the normal thing to do is commit more often so they don't grow so large. --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by markus＠greenrobot.de 23 Oct '19

23 Oct '19

On 21.10.19 17:14, Howard Chu wrote: > I believe this was fixed by commit > 7edf504106c61639a89b9a4e5987242598196932 in mdb.master. I can not confirm that this works. This is the stack trace where MDB_TXN_FULL is still returned with latest mdb.master (note: line numbers shown here are off by 60 compared to mdb.master): mdb_rpage_get mdb.c:6196 mdb_page_get mdb.c:6378 mdb_page_search_lowest mdb.c:6492 mdb_node_move mdb.c:8842 mdb_rebalance mdb.c:9366 mdb_page_merge mdb.c:9166 mdb_rebalance mdb.c:9373 mdb_cursor_del0 mdb.c:9426 mdb_cursor_del mdb.c:811 Code segment: if (tl[0].mid >= MDB_TRPAGE_MAX) return MDB_TXN_FULL; Debugger shows tl[0].mid to be 4095. Hope that helps. Markus

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by hyc＠symas.com 21 Oct '19

21 Oct '19

markus(a)greenrobot.de wrote: > Full_Name: Markus Junginger > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (77.189.91.168) > > > We have a 1GB LMDB file with 7M K/V entries. With MDB_VL32, we always get a > MDB_TXN_FULL error for a transaction that is removing 2M entries (probably fails > at a lower count, we haven't measured that). Without MDB_VL32 it works fine. > > Another observation: > Once the transaction fails with MDB_TXN_FULL, the data file has grown to 1.5GB. > Without MDB_VL32, the data file stays consistent at 1 GB even if all 7M entries > are deleted in a single transaction. > > Expected behavior: > No MDB_TXN_FULL error, no data file growth. > > > I believe this was fixed by commit 7edf504106c61639a89b9a4e5987242598196932 in mdb.master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9099) LMDB: range delete
by hyc＠symas.com 21 Oct '19

21 Oct '19

opensource(a)gmx-topmail.de wrote: > Full_Name: Markus > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (91.64.106.220) > > > We built a database with LMDB and are very pleased with it in general. The one > area giving our users headaches regularly are bulk deletes. But some context > first: > > Like many others, we use a hierarchical key notation. Some simplified examples: > "user/00001", "user/00002", "user/00003", ...; "usergroup/00001", > "usergroup/00002", .... > > Or, let's say data by timestamp: "sensordata/32489732", "sensordata/32489733", > ... > > Use case like "remove all users" or "remove all sensordata entries from time x > to y" are usual. Those (transactional) operations call mdb_cursor_del() on each > entry. This runs fine until we have LOTS of entries to delete. At a magnitude of > 100,000s of entries, two things happen: 1) the time it takes to delete gets > quite noticeable and 2) the 32 bit (MDB_VL32) will run into MDB_TXN_FULL (see > also #8813). > > Given the key structure, those operations operate on consecutive keys. Because > of that, can we make this more efficient (and get rid MDB_TXN_FULL on the fly)? > > A high number of individual deletes involves lots of page merging (and a lot of > dirty pages). If we had a specialized "range delete" this could likely be > avoided. The signature of the proposed functionality could look like this: The number of pages dirtied will be the same, whether you batch or delete one by one. As such, this won't solve the MDB_TXN_FULL problem. Also, as of commit 7edf504106c61639a89b9a4e5987242598196932 this problem should have already been removed for MDB_VL32. > int mdb_cursor_del_range(MDB_cursor *mc, MDB_val *key_start, MDB_val *key_stop, > unsigned int flags) > > Given key_start and key_stop, this new function could decide to prune entire > leaf pages, or even entire sub branches. This should result in a significant > efficiency/performance gain. The challenge I'm seeing here is balancing the tree > though. Right, rebalancing would become quite a pain. A similar problem exists for PUT_MULTIPLE which is why it just internally loops over one item at a time. > What do you think? I'm not sure if OpenLdap would profit here, but I'm quite > certain that LMDB as a general database would. I'm also open to contribute if > you are willing to give some guidance. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9099) LMDB: range delete
by opensource＠gmx-topmail.de 21 Oct '19

21 Oct '19

Full_Name: Markus Version: OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (91.64.106.220) We built a database with LMDB and are very pleased with it in general. The one area giving our users headaches regularly are bulk deletes. But some context first: Like many others, we use a hierarchical key notation. Some simplified examples: "user/00001", "user/00002", "user/00003", ...; "usergroup/00001", "usergroup/00002", .... Or, let's say data by timestamp: "sensordata/32489732", "sensordata/32489733", ... Use case like "remove all users" or "remove all sensordata entries from time x to y" are usual. Those (transactional) operations call mdb_cursor_del() on each entry. This runs fine until we have LOTS of entries to delete. At a magnitude of 100,000s of entries, two things happen: 1) the time it takes to delete gets quite noticeable and 2) the 32 bit (MDB_VL32) will run into MDB_TXN_FULL (see also #8813). Given the key structure, those operations operate on consecutive keys. Because of that, can we make this more efficient (and get rid MDB_TXN_FULL on the fly)? A high number of individual deletes involves lots of page merging (and a lot of dirty pages). If we had a specialized "range delete" this could likely be avoided. The signature of the proposed functionality could look like this: int mdb_cursor_del_range(MDB_cursor *mc, MDB_val *key_start, MDB_val *key_stop, unsigned int flags) Given key_start and key_stop, this new function could decide to prune entire leaf pages, or even entire sub branches. This should result in a significant efficiency/performance gain. The challenge I'm seeing here is balancing the tree though. What do you think? I'm not sure if OpenLdap would profit here, but I'm quite certain that LMDB as a general database would. I'm also open to contribute if you are willing to give some guidance. Thanks for your consideration, Markus

1 0

Re: (ITS#9096)
by priit＠ww.ee 18 Oct '19

18 Oct '19

--000000000000c70a5505952b7d50 Content-Type: text/plain; charset="UTF-8" Also a remark I must add, this only happens in one of my 4 Manjaro installations, which is also the only one running on AMD system (Ryzen 1700 + ASRock AB350M PRO4 motherboard). --000000000000c70a5505952b7d50 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div>Also a remark I must add, this only = happens in one of my 4 Manjaro installations, which is also the only one ru= nning on AMD system (Ryzen 1700 + <span class=3D"gmail-il">ASRock</span> AB= 350M PRO4 motherboard).<br></div><br></div></div> --000000000000c70a5505952b7d50--

1 0

Re: (ITS#9098) assert fails in meta_back_search in some cases after reconnect
by quanah＠symas.com 16 Oct '19

16 Oct '19

--On Wednesday, October 16, 2019 8:37 PM +0000 maxime.besson(a)worteks.com wrote: > Full_Name: Maxime Besson > Version: 2.4.47 > OS: Debian Jessie > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (2a01:cb00:802:8400:2cbe:3c60:fca6:e50b) I suggest upgrading to 2.4.48 to pick up this fix: Fixed slapd-meta assertion when network interface goes down (ITS#8841) --Quaanh -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs October 2019