openldap-bugs

openldap-bugs@openldap.org

1 participants
21065 discussions

Re: (ITS#9101) man pages don't reflect some global options are actually global/database
by ryan＠nardis.ca 25 Oct '19

25 Oct '19

On Fri, Oct 25, 2019 at 09:15:33PM +0000, quanah(a)symas.com wrote: >Correction, slapd-config(5) handles this via discussion of the frontend >database. Sort of - at least it has that distinction, but the three attrs you mentioned are still documented only in the GLOBAL CONFIGURATION OPTIONS section and not in the GLOBAL DATABASE OPTIONS section, while they actually are accepted on both, as you noted.

1 0

Re: (ITS#9101) man pages don't reflect some global options are actually global/database
by quanah＠symas.com 25 Oct '19

25 Oct '19

--On Friday, October 25, 2019 10:09 PM +0000 quanah(a)openldap.org wrote: > Full_Name: Quanah Gibson-Mount > Version: 2.4.48 > OS: N/A > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (47.208.154.120) > > > The slapd.conf(5)/slapd-config(5) man pages list the security/olcSecurity > configuration option as a "GLOBAL" only config option. Correction, slapd-config(5) handles this via discussion of the frontend database. It's slapd.conf(5) where this is a mess (doesn't have any differentiation). --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9101) man pages don't reflect some global options are actually global/database
by quanah＠openldap.org 25 Oct '19

25 Oct '19

Full_Name: Quanah Gibson-Mount Version: 2.4.48 OS: N/A URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.154.120) The slapd.conf(5)/slapd-config(5) man pages list the security/olcSecurity configuration option as a "GLOBAL" only config option. However, inspection of the code reveals that it is in fact global OR a general database option. The man pages should be updated to reflect this fact. The same issue exists for the sizelimit and timelimit directives as well. May need to add a new structure to the manual pages to cover these 3 items, as they appear to be the only things that can exist as both global and per-db items. I.e., we should cover what happens if they are set both globally and per-db (what takes precedence, etc).

1 0

Re: (ITS#9100) Issue with domainScope controlValue implementation
by hyc＠symas.com 25 Oct '19

25 Oct '19

philip.brusten(a)kuleuven.be wrote: > Full_Name: Philip Brusten > Version: 2.4.47 > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (2a02:2c40:100:b210::1:c3bc) > > > Hi > > we are experiencing issues with MS LDAP clients that connect to our OpenLDAP > servers since v2.4.47, because of the fix for issue ITS#8840 > > That fix was based on (old?) documentation of MS from > https://msdn.microsoft.com/en-us/library/aa366979%28v=vs.85%29.aspx > The is no current version of this document. The above URL redirects me to https://docs.microsoft.com/en-us/previous-versions/windows/desktop/ldap/lda… Our implementation conforms to the above spec. > At > https://docs.microsoft.com/en-us/windows/win32/api/_ldap/ there is a link to > Lightweigt Directory Access Protocol > (https://docs.microsoft.com/en-us/windows/desktop/ldap), but it gives a 404. > > MS is the owner of that LDAPControl LDAP_SERVER_DOMAIN_SCOPE_OID > 1.2.840.113556.1.4.1339. According to the RFC4511 [1] the controlValue is > defined by its specification: > >> The controlValue may contain information associated with the >> controlType. Its format is defined by the specification of the >> control. Implementations MUST be prepared to handle arbitrary >> contents of the controlValue octet string, including zero bytes. It >> is absent only if there is no value information that is associated >> with a control of its type. When a controlValue is defined in terms >> of ASN.1, and BER-encoded according to Section 5.1, it also follows >> the extensibility rules in Section 4. > > The latest specification can be found at: > https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/ba5f20… > >> "When sending this control to the DC, the controlValue field of the Control > structure is omitted." > > We observed that whenever MS clients are sending the domainScope control they > add an octet string as the controlValue with bytes "04 00". But this causes the > OpenLDAP code > if ( !BER_BVISNULL( &ctrl->ldctl_value )) { > to evaluate to true, which returns the LDAP_PROTOCOL_ERROR in OpenLDAP. This is the correct response to a malformed message according to RFC4511. > > The LDAP Extended Control spec of MS states [2]: >> If the controlValue field contains data that is not in conformance with the >> specification of the control, including the case where the controlValue field > contains >> data and the specification of the control states that the controlValue field > is omitted, >> then if the control is marked critical the server returns the error >> unavailableCriticalExtension / ERROR_INVALID_PARAMETER. If the controlValue > field is >> incorrect but the control is not marked critical, the server ignores the > control. MS seems to be inventing their own protocol here since RFC4511 contains no such language. > So the server should only return an error when the control is marked critical. > Otherwise it should *ignore* the control. > > Is it possible to relax the validation of the LDAP control at [3]? Since there > is no value associated with this control, it should not matter if the > controlValue is missing or an empty octet string {0,NULL}... > > The ldapsearch code sets the controlValue to [4]: >> c[i].ldctl_value.bv_val = NULL; >> c[i].ldctl_value.bv_len = 0; > We observed that the controlValue is than missing via Wireshark. So we see a > difference on the line for empty octet strings, but should it matter? Yes, in ASN.1 "empty value" and "absent value" are not the same thing. E.g., you can store zero-length strings as attribute values. Those values are "present" even though they are empty. MS is playing fast and loose with their specifications and implementation. This is at the very least a doc bug in their control specification, if it's not just a bug in their client libraries. You should at least open a bug report with them so that it's on the record somewhere (even if they ultimately ignore it). We can alter our parser to relax this check, I guess. Along with a comment explaining that this is done for compatibility with MS's broken clients. This is not the first time we've run into MS spec and implementation disagreeing with each other... > > Wireshark trace of MS LDAP client with domainScope control: > https://i.imgur.com/gwBjLMr.png > Wireshark trace of ldapsearch with domainScope control: > https://i.imgur.com/2RbFkIY.png > > Regards, > Philip > > > [1] https://tools.ietf.org/html/rfc4511#section-4.1.11 > [2] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/3c5e87… > [3] https://github.com/openldap/openldap/blob/master/servers/slapd/controls.c#L… > [4] https://github.com/openldap/openldap/blob/master/clients/tools/ldapsearch.c… > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by markus＠greenrobot.de 24 Oct '19

24 Oct '19

On 24.10.19 15:03, Markus Junginger wrote: > per transaction MDB_TRPAGE_SIZE bytes is allocated I guess, I was too quick here. It's MDB_TRPAGE_SIZE * 16. Also, I guess, txn->mt_rpages are just pointers to in-memory cached pages (or chunks?), so the memory consumption of those is much higher. The docs mention "chunks of 16 pages"; does that refer to mt_rpages? If so, a single mt_rpages pointer can actually hold on to 16 * 4K = 64 KB? Thus a MDB_TRPAGE_SIZE 4096 can go up to 256 MB memory? That changes the picture of course... PS.: let me express the irony once more that "removing data" is the most costly operation here ;-)

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by markus＠greenrobot.de 24 Oct '19

24 Oct '19

On 24.10.19 03:49, Howard Chu wrote: > You're free to define MDB_TRPAGE_MAX to a larger value. It just means > you increase the chance of overrunning the 2GB available address space. > There's no magic, you can't fit every 64bit database workload into only > a 32bit address space. When your transactions are too large, the normal > thing to do is commit more often so they don't grow so large. I think you meant MDB_TRPAGE_SIZE? At least that seemed to work, while MDB_TRPAGE_MAX ended up in another MDB_TXN_FULL (tl[0].mid < MDB_TRPAGE_SIZE check failed). Doubling MDB_TRPAGE_SIZE also doubled the threshold of the object count where it starts failing: using 8192 it was able to remove 4M entries while the limit with 4096 was 2M entries. MDB_TRPAGE_SIZE is only used to malloc txn->mt_rpages (and some checks), as far as I can tell. To make a reasonable decision here, could you please confirm: 1. txn->mt_rpages is RAM only and has no impact on mmaped sections (and/or file format). 2. RAM consumption is defined by number of transactions only: per transaction MDB_TRPAGE_SIZE bytes is allocated. Other than that it does not increase memory consumption. 3. There is no other disadvantage other than higher memory consumption per transaction as written in the previous point. In that case, I think I'll go with 16K (or even 32K). Does not seem too much for an transaction, given that it quadruples the amount of data allowed to be processed. 32 bit devices are not servers, so I do not expect a high number of concurrent transactions anyway. Thanks! Markus

1 0

(ITS#9100) Issue with domainScope controlValue implementation
by philip.brusten＠kuleuven.be 24 Oct '19

24 Oct '19

Full_Name: Philip Brusten Version: 2.4.47 OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2a02:2c40:100:b210::1:c3bc) Hi we are experiencing issues with MS LDAP clients that connect to our OpenLDAP servers since v2.4.47, because of the fix for issue ITS#8840 That fix was based on (old?) documentation of MS from https://msdn.microsoft.com/en-us/library/aa366979%28v=vs.85%29.aspx The is no current version of this document. At https://docs.microsoft.com/en-us/windows/win32/api/_ldap/ there is a link to Lightweigt Directory Access Protocol (https://docs.microsoft.com/en-us/windows/desktop/ldap), but it gives a 404. MS is the owner of that LDAPControl LDAP_SERVER_DOMAIN_SCOPE_OID 1.2.840.113556.1.4.1339. According to the RFC4511 [1] the controlValue is defined by its specification: > The controlValue may contain information associated with the > controlType. Its format is defined by the specification of the > control. Implementations MUST be prepared to handle arbitrary > contents of the controlValue octet string, including zero bytes. It > is absent only if there is no value information that is associated > with a control of its type. When a controlValue is defined in terms > of ASN.1, and BER-encoded according to Section 5.1, it also follows > the extensibility rules in Section 4. The latest specification can be found at: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/ba5f20… > "When sending this control to the DC, the controlValue field of the Control structure is omitted." We observed that whenever MS clients are sending the domainScope control they add an octet string as the controlValue with bytes "04 00". But this causes the OpenLDAP code if ( !BER_BVISNULL( &ctrl->ldctl_value )) { to evaluate to true, which returns the LDAP_PROTOCOL_ERROR in OpenLDAP. The LDAP Extended Control spec of MS states [2]: > If the controlValue field contains data that is not in conformance with the > specification of the control, including the case where the controlValue field contains > data and the specification of the control states that the controlValue field is omitted, > then if the control is marked critical the server returns the error > unavailableCriticalExtension / ERROR_INVALID_PARAMETER. If the controlValue field is > incorrect but the control is not marked critical, the server ignores the control. So the server should only return an error when the control is marked critical. Otherwise it should *ignore* the control. Is it possible to relax the validation of the LDAP control at [3]? Since there is no value associated with this control, it should not matter if the controlValue is missing or an empty octet string {0,NULL}... The ldapsearch code sets the controlValue to [4]: > c[i].ldctl_value.bv_val = NULL; > c[i].ldctl_value.bv_len = 0; We observed that the controlValue is than missing via Wireshark. So we see a difference on the line for empty octet strings, but should it matter? Wireshark trace of MS LDAP client with domainScope control: https://i.imgur.com/gwBjLMr.png Wireshark trace of ldapsearch with domainScope control: https://i.imgur.com/2RbFkIY.png Regards, Philip [1] https://tools.ietf.org/html/rfc4511#section-4.1.11 [2] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/3c5e87… [3] https://github.com/openldap/openldap/blob/master/servers/slapd/controls.c#L… [4] https://github.com/openldap/openldap/blob/master/clients/tools/ldapsearch.c…

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by hyc＠symas.com 23 Oct '19

23 Oct '19

markus(a)greenrobot.de wrote: > On 21.10.19 17:14, Howard Chu wrote: >> I believe this was fixed by commit=20 >> 7edf504106c61639a89b9a4e5987242598196932 in mdb.master.=20 >=20 > I can not confirm that this works. >=20 > This is the stack trace where MDB_TXN_FULL is still returned with lates= t=20 > mdb.master (note: line numbers shown here are off by 60 compared to=20 > mdb.master): >=20 > mdb_rpage_get mdb.c:6196 > mdb_page_get mdb.c:6378 > mdb_page_search_lowest mdb.c:6492 > mdb_node_move mdb.c:8842 > mdb_rebalance mdb.c:9366 > mdb_page_merge mdb.c:9166 > mdb_rebalance mdb.c:9373 > mdb_cursor_del0 mdb.c:9426 > mdb_cursor_del mdb.c:811 >=20 >=20 > Code segment: >=20 > if (tl[0].mid >=3D MDB_TRPAGE_MAX) > =C3=82=C2=A0=C3=82=C2=A0=C3=82=C2=A0 return MDB_TXN_FULL; >=20 > Debugger shows tl[0].mid to be 4095. You're free to define MDB_TRPAGE_MAX to a larger value. It just means you increase the chance of overrunning the 2GB available address space. There's no magic, you can't fit every 64bit database workload into only a 32bit address space. When your transactions are too large, the normal thing to do is commit more often so they don't grow so large. --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by markus＠greenrobot.de 23 Oct '19

23 Oct '19

On 21.10.19 17:14, Howard Chu wrote: > I believe this was fixed by commit > 7edf504106c61639a89b9a4e5987242598196932 in mdb.master. I can not confirm that this works. This is the stack trace where MDB_TXN_FULL is still returned with latest mdb.master (note: line numbers shown here are off by 60 compared to mdb.master): mdb_rpage_get mdb.c:6196 mdb_page_get mdb.c:6378 mdb_page_search_lowest mdb.c:6492 mdb_node_move mdb.c:8842 mdb_rebalance mdb.c:9366 mdb_page_merge mdb.c:9166 mdb_rebalance mdb.c:9373 mdb_cursor_del0 mdb.c:9426 mdb_cursor_del mdb.c:811 Code segment: if (tl[0].mid >= MDB_TRPAGE_MAX) return MDB_TXN_FULL; Debugger shows tl[0].mid to be 4095. Hope that helps. Markus

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by hyc＠symas.com 21 Oct '19

21 Oct '19

markus(a)greenrobot.de wrote: > Full_Name: Markus Junginger > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (77.189.91.168) > > > We have a 1GB LMDB file with 7M K/V entries. With MDB_VL32, we always get a > MDB_TXN_FULL error for a transaction that is removing 2M entries (probably fails > at a lower count, we haven't measured that). Without MDB_VL32 it works fine. > > Another observation: > Once the transaction fails with MDB_TXN_FULL, the data file has grown to 1.5GB. > Without MDB_VL32, the data file stays consistent at 1 GB even if all 7M entries > are deleted in a single transaction. > > Expected behavior: > No MDB_TXN_FULL error, no data file growth. > > > I believe this was fixed by commit 7edf504106c61639a89b9a4e5987242598196932 in mdb.master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs