- openldap-bugs - openldap.org

(ITS#9110) new feature
by ydgdsnn＠163.com 28 Oct '19

28 Oct '19

Full_Name: Nannan Song Version: 2.4.44 OS: SUSE URL: Submission from: (NULL) (221.226.97.96) When LDAP is used to manage user and user group information, openldap only supports the configuration of the plain text password of the read-only user in the '/etc/ldap.conf/'. The password of the read-only user only supports plain text storage. so there is a security issue that the authentication credential file is readable to all users. Now we hope ldap can support the feature that using the encrypted text to save password for read only user.

1 0

(ITS#9109) new feature
by ydgdsnn＠163.com 28 Oct '19

28 Oct '19

Full_Name: Nannan Song Version: 2.4.44 OS: SUSE URL: Submission from: (NULL) (221.226.97.96) When LDAP is used to manage user and user group information, openldap only supports the configuration of the plain text password of the read-only user in the '/etc/ldap.conf/'. The password of the read-only user only supports plain text storage. so there is a security issue that the authentication credential file is readable to all users. Now we hope ldap can support the feature that using the encrypted text to save password for read only user.

1 0

(ITS#9108) new feature
by ydgdsnn＠163.com 28 Oct '19

28 Oct '19

Full_Name: Nannan Song Version: 2.4.44 OS: SUSE URL: Submission from: (NULL) (221.226.97.96) When LDAP is used to manage user and user group information, openldap only supports the configuration of the plain text password of the read-only user in the '/etc/ldap.conf/'. The password of the read-only user only supports plain text storage. so there is a security issue that the authentication credential file is readable to all users. Now we hope ldap can support the feature that using the encrypted text to save password for read only user.

1 0

(ITS#9107) new feature
by ydgdsnn＠163.com 28 Oct '19

28 Oct '19

Full_Name: Nannan Song Version: 2.4.44 OS: SUSE URL: Submission from: (NULL) (221.226.97.96) When LDAP is used to manage user and user group information, openldap only supports the configuration of the plain text password of the read-only user in the '/etc/ldap.conf/'. The password of the read-only user only supports plain text storage. so there is a security issue that the authentication credential file is readable to all users. Now we hope ldap can support the feature that using the encrypted text to save password for read only user.

1 0

(ITS#9102) Update sasl-secprops ssf wording
by quanah＠openldap.org 25 Oct '19

25 Oct '19

Full_Name: Quanah Gibson-Mount Version: 2.4.48 OS: N/A URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.154.120) The slapd.conf(5)/slapd-config(5) man pages contain this wording for the sasl-secprops keyword: 0 (zero) implies no protection, 1 implies integrity protection only, 56 allows DES or other weak ciphers, 112 allows triple DES and other strong ciphers, 128 allows RC4, Blowfish and other modern strong ciphers. The default is 0. I think it's been quite a while since RC4 was considered a "modern strong cipher".

1 0

Re: (ITS#9101) man pages don't reflect some global options are actually global/database
by ryan＠nardis.ca 25 Oct '19

25 Oct '19

On Fri, Oct 25, 2019 at 09:15:33PM +0000, quanah(a)symas.com wrote: >Correction, slapd-config(5) handles this via discussion of the frontend >database. Sort of - at least it has that distinction, but the three attrs you mentioned are still documented only in the GLOBAL CONFIGURATION OPTIONS section and not in the GLOBAL DATABASE OPTIONS section, while they actually are accepted on both, as you noted.

1 0

Re: (ITS#9101) man pages don't reflect some global options are actually global/database
by quanah＠symas.com 25 Oct '19

25 Oct '19

--On Friday, October 25, 2019 10:09 PM +0000 quanah(a)openldap.org wrote: > Full_Name: Quanah Gibson-Mount > Version: 2.4.48 > OS: N/A > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (47.208.154.120) > > > The slapd.conf(5)/slapd-config(5) man pages list the security/olcSecurity > configuration option as a "GLOBAL" only config option. Correction, slapd-config(5) handles this via discussion of the frontend database. It's slapd.conf(5) where this is a mess (doesn't have any differentiation). --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9101) man pages don't reflect some global options are actually global/database
by quanah＠openldap.org 25 Oct '19

25 Oct '19

Full_Name: Quanah Gibson-Mount Version: 2.4.48 OS: N/A URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.154.120) The slapd.conf(5)/slapd-config(5) man pages list the security/olcSecurity configuration option as a "GLOBAL" only config option. However, inspection of the code reveals that it is in fact global OR a general database option. The man pages should be updated to reflect this fact. The same issue exists for the sizelimit and timelimit directives as well. May need to add a new structure to the manual pages to cover these 3 items, as they appear to be the only things that can exist as both global and per-db items. I.e., we should cover what happens if they are set both globally and per-db (what takes precedence, etc).

1 0

Re: (ITS#9100) Issue with domainScope controlValue implementation
by hyc＠symas.com 25 Oct '19

25 Oct '19

philip.brusten(a)kuleuven.be wrote: > Full_Name: Philip Brusten > Version: 2.4.47 > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (2a02:2c40:100:b210::1:c3bc) > > > Hi > > we are experiencing issues with MS LDAP clients that connect to our OpenLDAP > servers since v2.4.47, because of the fix for issue ITS#8840 > > That fix was based on (old?) documentation of MS from > https://msdn.microsoft.com/en-us/library/aa366979%28v=vs.85%29.aspx > The is no current version of this document. The above URL redirects me to https://docs.microsoft.com/en-us/previous-versions/windows/desktop/ldap/lda… Our implementation conforms to the above spec. > At > https://docs.microsoft.com/en-us/windows/win32/api/_ldap/ there is a link to > Lightweigt Directory Access Protocol > (https://docs.microsoft.com/en-us/windows/desktop/ldap), but it gives a 404. > > MS is the owner of that LDAPControl LDAP_SERVER_DOMAIN_SCOPE_OID > 1.2.840.113556.1.4.1339. According to the RFC4511 [1] the controlValue is > defined by its specification: > >> The controlValue may contain information associated with the >> controlType. Its format is defined by the specification of the >> control. Implementations MUST be prepared to handle arbitrary >> contents of the controlValue octet string, including zero bytes. It >> is absent only if there is no value information that is associated >> with a control of its type. When a controlValue is defined in terms >> of ASN.1, and BER-encoded according to Section 5.1, it also follows >> the extensibility rules in Section 4. > > The latest specification can be found at: > https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/ba5f20… > >> "When sending this control to the DC, the controlValue field of the Control > structure is omitted." > > We observed that whenever MS clients are sending the domainScope control they > add an octet string as the controlValue with bytes "04 00". But this causes the > OpenLDAP code > if ( !BER_BVISNULL( &ctrl->ldctl_value )) { > to evaluate to true, which returns the LDAP_PROTOCOL_ERROR in OpenLDAP. This is the correct response to a malformed message according to RFC4511. > > The LDAP Extended Control spec of MS states [2]: >> If the controlValue field contains data that is not in conformance with the >> specification of the control, including the case where the controlValue field > contains >> data and the specification of the control states that the controlValue field > is omitted, >> then if the control is marked critical the server returns the error >> unavailableCriticalExtension / ERROR_INVALID_PARAMETER. If the controlValue > field is >> incorrect but the control is not marked critical, the server ignores the > control. MS seems to be inventing their own protocol here since RFC4511 contains no such language. > So the server should only return an error when the control is marked critical. > Otherwise it should *ignore* the control. > > Is it possible to relax the validation of the LDAP control at [3]? Since there > is no value associated with this control, it should not matter if the > controlValue is missing or an empty octet string {0,NULL}... > > The ldapsearch code sets the controlValue to [4]: >> c[i].ldctl_value.bv_val = NULL; >> c[i].ldctl_value.bv_len = 0; > We observed that the controlValue is than missing via Wireshark. So we see a > difference on the line for empty octet strings, but should it matter? Yes, in ASN.1 "empty value" and "absent value" are not the same thing. E.g., you can store zero-length strings as attribute values. Those values are "present" even though they are empty. MS is playing fast and loose with their specifications and implementation. This is at the very least a doc bug in their control specification, if it's not just a bug in their client libraries. You should at least open a bug report with them so that it's on the record somewhere (even if they ultimately ignore it). We can alter our parser to relax this check, I guess. Along with a comment explaining that this is done for compatibility with MS's broken clients. This is not the first time we've run into MS spec and implementation disagreeing with each other... > > Wireshark trace of MS LDAP client with domainScope control: > https://i.imgur.com/gwBjLMr.png > Wireshark trace of ldapsearch with domainScope control: > https://i.imgur.com/2RbFkIY.png > > Regards, > Philip > > > [1] https://tools.ietf.org/html/rfc4511#section-4.1.11 > [2] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/3c5e87… > [3] https://github.com/openldap/openldap/blob/master/servers/slapd/controls.c#L… > [4] https://github.com/openldap/openldap/blob/master/clients/tools/ldapsearch.c… > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by markus＠greenrobot.de 24 Oct '19

24 Oct '19

On 24.10.19 15:03, Markus Junginger wrote: > per transaction MDB_TRPAGE_SIZE bytes is allocated I guess, I was too quick here. It's MDB_TRPAGE_SIZE * 16. Also, I guess, txn->mt_rpages are just pointers to in-memory cached pages (or chunks?), so the memory consumption of those is much higher. The docs mention "chunks of 16 pages"; does that refer to mt_rpages? If so, a single mt_rpages pointer can actually hold on to 16 * 4K = 64 KB? Thus a MDB_TRPAGE_SIZE 4096 can go up to 256 MB memory? That changes the picture of course... PS.: let me express the irony once more that "removing data" is the most costly operation here ;-)

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by markus＠greenrobot.de 24 Oct '19

24 Oct '19

On 24.10.19 03:49, Howard Chu wrote: > You're free to define MDB_TRPAGE_MAX to a larger value. It just means > you increase the chance of overrunning the 2GB available address space. > There's no magic, you can't fit every 64bit database workload into only > a 32bit address space. When your transactions are too large, the normal > thing to do is commit more often so they don't grow so large. I think you meant MDB_TRPAGE_SIZE? At least that seemed to work, while MDB_TRPAGE_MAX ended up in another MDB_TXN_FULL (tl[0].mid < MDB_TRPAGE_SIZE check failed). Doubling MDB_TRPAGE_SIZE also doubled the threshold of the object count where it starts failing: using 8192 it was able to remove 4M entries while the limit with 4096 was 2M entries. MDB_TRPAGE_SIZE is only used to malloc txn->mt_rpages (and some checks), as far as I can tell. To make a reasonable decision here, could you please confirm: 1. txn->mt_rpages is RAM only and has no impact on mmaped sections (and/or file format). 2. RAM consumption is defined by number of transactions only: per transaction MDB_TRPAGE_SIZE bytes is allocated. Other than that it does not increase memory consumption. 3. There is no other disadvantage other than higher memory consumption per transaction as written in the previous point. In that case, I think I'll go with 16K (or even 32K). Does not seem too much for an transaction, given that it quadruples the amount of data allowed to be processed. 32 bit devices are not servers, so I do not expect a high number of concurrent transactions anyway. Thanks! Markus

1 0

(ITS#9100) Issue with domainScope controlValue implementation
by philip.brusten＠kuleuven.be 24 Oct '19

24 Oct '19

Full_Name: Philip Brusten Version: 2.4.47 OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2a02:2c40:100:b210::1:c3bc) Hi we are experiencing issues with MS LDAP clients that connect to our OpenLDAP servers since v2.4.47, because of the fix for issue ITS#8840 That fix was based on (old?) documentation of MS from https://msdn.microsoft.com/en-us/library/aa366979%28v=vs.85%29.aspx The is no current version of this document. At https://docs.microsoft.com/en-us/windows/win32/api/_ldap/ there is a link to Lightweigt Directory Access Protocol (https://docs.microsoft.com/en-us/windows/desktop/ldap), but it gives a 404. MS is the owner of that LDAPControl LDAP_SERVER_DOMAIN_SCOPE_OID 1.2.840.113556.1.4.1339. According to the RFC4511 [1] the controlValue is defined by its specification: > The controlValue may contain information associated with the > controlType. Its format is defined by the specification of the > control. Implementations MUST be prepared to handle arbitrary > contents of the controlValue octet string, including zero bytes. It > is absent only if there is no value information that is associated > with a control of its type. When a controlValue is defined in terms > of ASN.1, and BER-encoded according to Section 5.1, it also follows > the extensibility rules in Section 4. The latest specification can be found at: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/ba5f20… > "When sending this control to the DC, the controlValue field of the Control structure is omitted." We observed that whenever MS clients are sending the domainScope control they add an octet string as the controlValue with bytes "04 00". But this causes the OpenLDAP code if ( !BER_BVISNULL( &ctrl->ldctl_value )) { to evaluate to true, which returns the LDAP_PROTOCOL_ERROR in OpenLDAP. The LDAP Extended Control spec of MS states [2]: > If the controlValue field contains data that is not in conformance with the > specification of the control, including the case where the controlValue field contains > data and the specification of the control states that the controlValue field is omitted, > then if the control is marked critical the server returns the error > unavailableCriticalExtension / ERROR_INVALID_PARAMETER. If the controlValue field is > incorrect but the control is not marked critical, the server ignores the control. So the server should only return an error when the control is marked critical. Otherwise it should *ignore* the control. Is it possible to relax the validation of the LDAP control at [3]? Since there is no value associated with this control, it should not matter if the controlValue is missing or an empty octet string {0,NULL}... The ldapsearch code sets the controlValue to [4]: > c[i].ldctl_value.bv_val = NULL; > c[i].ldctl_value.bv_len = 0; We observed that the controlValue is than missing via Wireshark. So we see a difference on the line for empty octet strings, but should it matter? Wireshark trace of MS LDAP client with domainScope control: https://i.imgur.com/gwBjLMr.png Wireshark trace of ldapsearch with domainScope control: https://i.imgur.com/2RbFkIY.png Regards, Philip [1] https://tools.ietf.org/html/rfc4511#section-4.1.11 [2] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/3c5e87… [3] https://github.com/openldap/openldap/blob/master/servers/slapd/controls.c#L… [4] https://github.com/openldap/openldap/blob/master/clients/tools/ldapsearch.c…

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by hyc＠symas.com 24 Oct '19

24 Oct '19

markus(a)greenrobot.de wrote: > On 21.10.19 17:14, Howard Chu wrote: >> I believe this was fixed by commit=20 >> 7edf504106c61639a89b9a4e5987242598196932 in mdb.master.=20 >=20 > I can not confirm that this works. >=20 > This is the stack trace where MDB_TXN_FULL is still returned with lates= t=20 > mdb.master (note: line numbers shown here are off by 60 compared to=20 > mdb.master): >=20 > mdb_rpage_get mdb.c:6196 > mdb_page_get mdb.c:6378 > mdb_page_search_lowest mdb.c:6492 > mdb_node_move mdb.c:8842 > mdb_rebalance mdb.c:9366 > mdb_page_merge mdb.c:9166 > mdb_rebalance mdb.c:9373 > mdb_cursor_del0 mdb.c:9426 > mdb_cursor_del mdb.c:811 >=20 >=20 > Code segment: >=20 > if (tl[0].mid >=3D MDB_TRPAGE_MAX) > =C3=82=C2=A0=C3=82=C2=A0=C3=82=C2=A0 return MDB_TXN_FULL; >=20 > Debugger shows tl[0].mid to be 4095. You're free to define MDB_TRPAGE_MAX to a larger value. It just means you increase the chance of overrunning the 2GB available address space. There's no magic, you can't fit every 64bit database workload into only a 32bit address space. When your transactions are too large, the normal thing to do is commit more often so they don't grow so large. --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by markus＠greenrobot.de 23 Oct '19

23 Oct '19

On 21.10.19 17:14, Howard Chu wrote: > I believe this was fixed by commit > 7edf504106c61639a89b9a4e5987242598196932 in mdb.master. I can not confirm that this works. This is the stack trace where MDB_TXN_FULL is still returned with latest mdb.master (note: line numbers shown here are off by 60 compared to mdb.master): mdb_rpage_get mdb.c:6196 mdb_page_get mdb.c:6378 mdb_page_search_lowest mdb.c:6492 mdb_node_move mdb.c:8842 mdb_rebalance mdb.c:9366 mdb_page_merge mdb.c:9166 mdb_rebalance mdb.c:9373 mdb_cursor_del0 mdb.c:9426 mdb_cursor_del mdb.c:811 Code segment: if (tl[0].mid >= MDB_TRPAGE_MAX) return MDB_TXN_FULL; Debugger shows tl[0].mid to be 4095. Hope that helps. Markus

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by hyc＠symas.com 21 Oct '19

21 Oct '19

markus(a)greenrobot.de wrote: > Full_Name: Markus Junginger > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (77.189.91.168) > > > We have a 1GB LMDB file with 7M K/V entries. With MDB_VL32, we always get a > MDB_TXN_FULL error for a transaction that is removing 2M entries (probably fails > at a lower count, we haven't measured that). Without MDB_VL32 it works fine. > > Another observation: > Once the transaction fails with MDB_TXN_FULL, the data file has grown to 1.5GB. > Without MDB_VL32, the data file stays consistent at 1 GB even if all 7M entries > are deleted in a single transaction. > > Expected behavior: > No MDB_TXN_FULL error, no data file growth. > > > I believe this was fixed by commit 7edf504106c61639a89b9a4e5987242598196932 in mdb.master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9099) LMDB: range delete
by hyc＠symas.com 21 Oct '19

21 Oct '19

opensource(a)gmx-topmail.de wrote: > Full_Name: Markus > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (91.64.106.220) > > > We built a database with LMDB and are very pleased with it in general. The one > area giving our users headaches regularly are bulk deletes. But some context > first: > > Like many others, we use a hierarchical key notation. Some simplified examples: > "user/00001", "user/00002", "user/00003", ...; "usergroup/00001", > "usergroup/00002", .... > > Or, let's say data by timestamp: "sensordata/32489732", "sensordata/32489733", > ... > > Use case like "remove all users" or "remove all sensordata entries from time x > to y" are usual. Those (transactional) operations call mdb_cursor_del() on each > entry. This runs fine until we have LOTS of entries to delete. At a magnitude of > 100,000s of entries, two things happen: 1) the time it takes to delete gets > quite noticeable and 2) the 32 bit (MDB_VL32) will run into MDB_TXN_FULL (see > also #8813). > > Given the key structure, those operations operate on consecutive keys. Because > of that, can we make this more efficient (and get rid MDB_TXN_FULL on the fly)? > > A high number of individual deletes involves lots of page merging (and a lot of > dirty pages). If we had a specialized "range delete" this could likely be > avoided. The signature of the proposed functionality could look like this: The number of pages dirtied will be the same, whether you batch or delete one by one. As such, this won't solve the MDB_TXN_FULL problem. Also, as of commit 7edf504106c61639a89b9a4e5987242598196932 this problem should have already been removed for MDB_VL32. > int mdb_cursor_del_range(MDB_cursor *mc, MDB_val *key_start, MDB_val *key_stop, > unsigned int flags) > > Given key_start and key_stop, this new function could decide to prune entire > leaf pages, or even entire sub branches. This should result in a significant > efficiency/performance gain. The challenge I'm seeing here is balancing the tree > though. Right, rebalancing would become quite a pain. A similar problem exists for PUT_MULTIPLE which is why it just internally loops over one item at a time. > What do you think? I'm not sure if OpenLdap would profit here, but I'm quite > certain that LMDB as a general database would. I'm also open to contribute if > you are willing to give some guidance. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9099) LMDB: range delete
by opensource＠gmx-topmail.de 21 Oct '19

21 Oct '19

Full_Name: Markus Version: OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (91.64.106.220) We built a database with LMDB and are very pleased with it in general. The one area giving our users headaches regularly are bulk deletes. But some context first: Like many others, we use a hierarchical key notation. Some simplified examples: "user/00001", "user/00002", "user/00003", ...; "usergroup/00001", "usergroup/00002", .... Or, let's say data by timestamp: "sensordata/32489732", "sensordata/32489733", ... Use case like "remove all users" or "remove all sensordata entries from time x to y" are usual. Those (transactional) operations call mdb_cursor_del() on each entry. This runs fine until we have LOTS of entries to delete. At a magnitude of 100,000s of entries, two things happen: 1) the time it takes to delete gets quite noticeable and 2) the 32 bit (MDB_VL32) will run into MDB_TXN_FULL (see also #8813). Given the key structure, those operations operate on consecutive keys. Because of that, can we make this more efficient (and get rid MDB_TXN_FULL on the fly)? A high number of individual deletes involves lots of page merging (and a lot of dirty pages). If we had a specialized "range delete" this could likely be avoided. The signature of the proposed functionality could look like this: int mdb_cursor_del_range(MDB_cursor *mc, MDB_val *key_start, MDB_val *key_stop, unsigned int flags) Given key_start and key_stop, this new function could decide to prune entire leaf pages, or even entire sub branches. This should result in a significant efficiency/performance gain. The challenge I'm seeing here is balancing the tree though. What do you think? I'm not sure if OpenLdap would profit here, but I'm quite certain that LMDB as a general database would. I'm also open to contribute if you are willing to give some guidance. Thanks for your consideration, Markus

1 0

Re: (ITS#9096)
by priit＠ww.ee 18 Oct '19

18 Oct '19

--000000000000c70a5505952b7d50 Content-Type: text/plain; charset="UTF-8" Also a remark I must add, this only happens in one of my 4 Manjaro installations, which is also the only one running on AMD system (Ryzen 1700 + ASRock AB350M PRO4 motherboard). --000000000000c70a5505952b7d50 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div>Also a remark I must add, this only = happens in one of my 4 Manjaro installations, which is also the only one ru= nning on AMD system (Ryzen 1700 + <span class=3D"gmail-il">ASRock</span> AB= 350M PRO4 motherboard).<br></div><br></div></div> --000000000000c70a5505952b7d50--

1 0

Re: (ITS#9098) assert fails in meta_back_search in some cases after reconnect
by quanah＠symas.com 16 Oct '19

16 Oct '19

--On Wednesday, October 16, 2019 8:37 PM +0000 maxime.besson(a)worteks.com wrote: > Full_Name: Maxime Besson > Version: 2.4.47 > OS: Debian Jessie > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (2a01:cb00:802:8400:2cbe:3c60:fca6:e50b) I suggest upgrading to 2.4.48 to pick up this fix: Fixed slapd-meta assertion when network interface goes down (ITS#8841) --Quaanh -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9098) assert fails in meta_back_search in some cases after reconnect
by maxime.besson＠worteks.com 16 Oct '19

16 Oct '19

Full_Name: Maxime Besson Version: 2.4.47 OS: Debian Jessie URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2a01:cb00:802:8400:2cbe:3c60:fca6:e50b) I am running a meta-directory with the following DB configuration. version 2.4.47, LTB build on Ubuntu 16.04 dn: olcDatabase={1}meta,cn=config objectClass: olcMetaConfig objectClass: olcDatabaseConfig objectClass: olcConfig objectClass: top olcDatabase: {1}meta olcSuffix: dc=com olcAccess: {0}to * by * read olcRootDN: cn=admin,dc=com dn: olcMetaSub={0}uri,olcDatabase={1}meta,cn=config objectClass: olcMetaTargetConfig objectClass: olcConfig objectClass: top olcMetaSub: {0}uri olcDbURI: ldap://1.2.3.4/dc=example,dc=com olcDbIDAssertBind: mode=legacy flags=non-prescriptive,proxy-authz-non-critical bindmethod=simple binddn="cn=admin,dc=example,dc=com" credentials="XXXXX" olcDbTimeout: 5 olcDbNetworkTimeout: 3 olcDbNretries: never olcDbRebindAsUser: true ... (There are 8 backends in total) Timeouts were added in order to avoid blocking OpenLDAP completely when one server becomes completely unavailable. However, since I added them, the slapd process started crashing every now and then (from a couple hours to a couple of days), usually during small network interruptions that affect all backends: I see plenty of reconnect logs shortly before the crashes. The crash is always immediately preceded by the following log message: meta_search_dobind_init[{i}]: retrying URI="{url}" DN="{DN}" {i} is never the same, and {url} and {DN} are the correct settings for backend i. The crash itself is an ABRT at the following assert in back-meta/search.c: 1957 assert( candidates[ i ].sr_msgid >= 0 1958 || candidates[ i ].sr_msgid == META_MSGID_CONNECTING ); I have analyzed several core dumps, and found that every single time slapd crashes, sr_msgid has a value of -1 (META_MSGID_IGNORE), which indeed causes the assert to fail. I found that candidates[i]->sr_flags has a value of 3 (META_CANDIDATE + META_BINDING) And the msc_mscflags in mc->mc_conns[ i ] are * 0x100081 for all connections before the one that triggers the crash * 0x100010 for the candidate that crashes the server * 0x100080 for all connections after it I am having trouble reproducing this in a test environment, but it happens regularly in production, I have tried changing the timeouts, adding a non-default bind timeout , and disabling retries (they were originally allowed) but the crashes keep happening. Note that disabling retries (olcDbNretries: never) still seems to lead to retries in meta_search_dobind_init, since the log message is still there. I cannot share the core dumps due to the sensitive information inside them. However I would gladly extract more information from them if it can help solving this.

1 0

Re: (ITS#9097) lmdb: premature free of env->me_txn0
by hyc＠symas.com 16 Oct '19

16 Oct '19

Christopher Zimmermann wrote: > Hi again, > > I did some further debugging and now I'm afraid I am indeed finishing a > transaction twice. I do call txn_commit() and after that I call > txn_abort(). But the MDB_MAP_FULL event happens during mdb_txn_commit(): > > (gdb) frame 0 > #0 mdb_page_alloc (mc=0x7f7ffffbd580, num=1, mp=0x7f7ffffbd228) at mdb.c:2286 > 2286 rc = MDB_MAP_FULL; > (gdb) bt > #0 mdb_page_alloc (mc=0x7f7ffffbd580, num=1, mp=0x7f7ffffbd228) at mdb.c:2286 > #1 0x00000fb6fb6aa239 in mdb_page_touch (mc=0x7f7ffffbd580) at mdb.c:2428 > #2 0x00000fb6fb6a9db3 in mdb_page_search (mc=0x7f7ffffbd580, key=0x0, flags=5) at mdb.c:5627 > #3 0x00000fb6fb69e585 in mdb_freelist_save (txn=0xfb6bfb3ae00) at mdb.c:3087 > #4 0x00000fb6fb69c2a4 in mdb_txn_commit (txn=0xfb6bfb3ae00) at mdb.c:3612 > > mdb_txn_commit() will return MDB_MAP_FULL. > > It may be worth adding MDB_MAP_FULL and MDB_BAD_TXN to the possible > error codes of mdb_txn_commit(), or in general provide some guidance on > how to clean up a txn after some failure. The doc for commit and abort is explicit. A txn must not be used again after calling commit or abort. There is nothing to clean up, the txn is always gone when commit or abort returns. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9097) lmdb: premature free of env->me_txn0
by madroach＠gmerlin.de 16 Oct '19

16 Oct '19

Hi again, I did some further debugging and now I'm afraid I am indeed finishing a transaction twice. I do call txn_commit() and after that I call txn_abort(). But the MDB_MAP_FULL event happens during mdb_txn_commit(): (gdb) frame 0 #0 mdb_page_alloc (mc=0x7f7ffffbd580, num=1, mp=0x7f7ffffbd228) at mdb.c:2286 2286 rc = MDB_MAP_FULL; (gdb) bt #0 mdb_page_alloc (mc=0x7f7ffffbd580, num=1, mp=0x7f7ffffbd228) at mdb.c:2286 #1 0x00000fb6fb6aa239 in mdb_page_touch (mc=0x7f7ffffbd580) at mdb.c:2428 #2 0x00000fb6fb6a9db3 in mdb_page_search (mc=0x7f7ffffbd580, key=0x0, flags=5) at mdb.c:5627 #3 0x00000fb6fb69e585 in mdb_freelist_save (txn=0xfb6bfb3ae00) at mdb.c:3087 #4 0x00000fb6fb69c2a4 in mdb_txn_commit (txn=0xfb6bfb3ae00) at mdb.c:3612 mdb_txn_commit() will return MDB_MAP_FULL. It may be worth adding MDB_MAP_FULL and MDB_BAD_TXN to the possible error codes of mdb_txn_commit(), or in general provide some guidance on how to clean up a txn after some failure. Christopher -- http://gmerlin.de OpenPGP: http://gmerlin.de/christopher.pub CB07 DA40 B0B6 571D 35E2 0DEF 87E2 92A7 13E5 DEE1

1 0

Re: (ITS#9097) lmdb: premature free of env->me_txn0
by madroach＠gmerlin.de 16 Oct '19

16 Oct '19

On Wed, Oct 16, 2019 at 01:59:51AM +0100, Howard Chu wrote: > christopher(a)gmerlin.de wrote: > > Full_Name: Christopher Zimmermann > > Version: lmdb 0.9.24 > > OS: OpenBSD > > URL: ftp://ftp.openldap.org/incoming/ > > Submission from: (NULL) (85.212.180.240) > > > > > > Hi, > > > > I can reliably hit a Bus error on OpenBSD. > > This is triggered by OpenBSDs malloc/free junking [1] and a use-after-free bug > > in lmdb. > > > > Steps to reproduce: > > - begin a read/write transaction (getting env->me_txn0) > > - fill the environment > > -> returns MDB_MAP_FULL > > -> sets txn->mt_flags |= MDB_TXN_ERROR; (This is also env->me_txn0 !) > > -> calls mdb_txn_abort > ... > > - abort the transaction (again) with mdb_abort() > > This is a bug in your code, you can't call txn_abort twice. This is > already documented. Closing this ITS. Hi, thanks for having a look. I could not find any documentation about when one must / must not call mdb_txn_abort. But in any case I do _not_ call txn_abort() twice. The problem rather seems to be that lmdb _implicitely_ frees invalid transactions. And my code then aborts them again _explicitely_. So mdb_abort was indeed called twice. Once in the MDB_TXN_ERROR internally in lmdb case and once from my code. On second thought my fix won't fix the same problem for errored child transactions. What is necessary seems to be either documenting on which error conditions the user needs to call txn_abort() and on which transactions (child vs parent, too). Or (what I would prefer) let the user clean up the transaction. In case this is indeed already documented I would appreciate a pointer to the location of this documentation. Thanks, Christopher -- http://gmerlin.de OpenPGP: http://gmerlin.de/christopher.pub CB07 DA40 B0B6 571D 35E2 0DEF 87E2 92A7 13E5 DEE1

1 0

Re: (ITS#9097) lmdb: premature free of env->me_txn0
by hyc＠symas.com 16 Oct '19

16 Oct '19

christopher(a)gmerlin.de wrote: > Full_Name: Christopher Zimmermann > Version: lmdb 0.9.24 > OS: OpenBSD > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (85.212.180.240) > > > Hi, > > I can reliably hit a Bus error on OpenBSD. > This is triggered by OpenBSDs malloc/free junking [1] and a use-after-free bug > in lmdb. > > Steps to reproduce: > - begin a read/write transaction (getting env->me_txn0) > - fill the environment > -> returns MDB_MAP_FULL > -> sets txn->mt_flags |= MDB_TXN_ERROR; (This is also env->me_txn0 !) > -> calls mdb_txn_abort ... > - abort the transaction (again) with mdb_abort() This is a bug in your code, you can't call txn_abort twice. This is already documented. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9097) lmdb: premature free of env->me_txn0
by christopher＠gmerlin.de 16 Oct '19

16 Oct '19

Full_Name: Christopher Zimmermann Version: lmdb 0.9.24 OS: OpenBSD URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (85.212.180.240) Hi, I can reliably hit a Bus error on OpenBSD. This is triggered by OpenBSDs malloc/free junking [1] and a use-after-free bug in lmdb. Steps to reproduce: - begin a read/write transaction (getting env->me_txn0) - fill the environment -> returns MDB_MAP_FULL -> sets txn->mt_flags |= MDB_TXN_ERROR; (This is also env->me_txn0 !) -> calls mdb_txn_abort -> calls mdb_txn_end(txn, MDB_END_ABORT|MDB_END_SLOT|MDB_END_FREE): mdb_txn_end tries not to free env->me_txn0: } else if (!F_ISSET(txn->mt_flags, MDB_TXN_FINISHED)) { [...] txn->mt_flags = MDB_TXN_FINISHED; if (!txn->mt_parent) { [...] mdb.c:3020 mode = 0; /* txn == env->me_txn0, do not free() it */ [...] } [...] } if (mode & MDB_END_FREE) free(txn); this prevents the free only for unfinished transactions. Unfinished transactions are now finished. - abort the transaction (again) with mdb_abort() -> calls mdb_txn_end(txn, MDB_END_ABORT|MDB_END_SLOT|MDB_END_FREE): since the env->me_txn0 detection is skipped on MDB_TXN_FINISHED transactions the transaction is freed, the memory will get "junked". - begin a read/write transaction (getting now invalid env->me_txn0) -> calls mdb_txn_renew0(env->me_txn0) MDB_env *env = txn->mt_env; /* txn->mt_env is now invalid */ MDB_txninfo *ti = env->me_txns; /* triggers bus error */ Please make the protection against freeing of me_txn0 more robust. Thanks, Christopher Zimmermann [1] https://man.openbsd.org/malloc#j

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs