- openldap-bugs - openldap.org

Re: (ITS#9120) Searches including generalizedTime attributes with index
by hyc＠symas.com 26 Nov '19

26 Nov '19

Howard Chu wrote: > markus.widmer(a)daasi.de wrote: >> We could reproduce this with 2.4.42, 2.4.44 and 2.4.48. We hope you can >> reproduce this as well to see what is happening here. > > Yes, can reproduce this. The function that converts a component-wise time > into a timet is referencing time since 1970. The date in 1956 would yield a > negative timet value but the fields are unsigned ints, so instead it's treated > as 17,000 years in the future. We can probably change this to handle signed > timestamps but need to consider this further. > I believe the best way forward would be to allow signed values, and also to switch our epoch reference from 1970-01-01 to 0000-01-01 (i.e., use Gregorian Proleptic calendar). Year zero would be 1 BCE in this calendar, and anything earlier would be a negative year. Changing these functions will require regenerating any indices. Looks like something we'll rewrite for 2.5 but leave 2.4 alone. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9120) Searches including generalizedTime attributes with index
by hyc＠symas.com 25 Nov '19

25 Nov '19

markus.widmer(a)daasi.de wrote: > Full_Name: Markus Widmer > Version: 2.4.48 > OS: CentOS 7 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (37.24.13.226) > > > Hi! > > we have recognized a strange behaviour for searches which include an attribute > with syntax 1.3.6.1.4.1.1466.115.121.1.24 and the matching rules > generalizedTimeMatch and generalizedTimeOrderingMatch set. If the timestamps in > the objects have values before 1970, then a search for > "<=" or ">=" only seams to works if: > - The attribute is not part of the index > - The attribute is part of the index but "objectClass" is not > > Reproduction: > We used two entries with the attribute "schacExpiryDate" from the > objectClass "schacEntryMetadata" but this can be reproduced with other > attributes as well: > > attributetype ( 1.3.6.1.4.1.25178:1:2:17 > NAME 'schacExpiryDate' > DESC 'Date from which the set of data is to be considered invalid (format > YYYYMMDDhhmmssZ)' > EQUALITY generalizedTimeMatch > ORDERING generalizedTimeOrderingMatch > SINGLE-VALUE > SYNTAX 1.3.6.1.4.1.1466.115.121.1.24 ) > > We used two entries: > > user.1: > schacExpiryDate: 19711007093230.514Z > > user.2: > schacExpiryDate: 19561007093230.514Z > > First try: we used this index configuration: > > index sn eq > index schacExpiryDate eq > index objectClass eq > > Search filter 1 was: (schacExpiryDate<=19800101120000Z) > Result 1 was: user.1 > > Search filter 2 was: (schacExpiryDate>=19010101120000Z) > Result 2 was: user.2 > > > Second try: we used this index configuration: > > index sn eq > #index schacExpiryDate eq > index objectClass eq > > Search filter 1 was: (schacExpiryDate<=19800101120000Z) > Result 1 was: user.1 and user.2 > > Search filter 2 was: (schacExpiryDate>=19010101120000Z) > Result 2 was: user.2 and user.2 > > > Third try: we used this index configuration: > > index sn eq > index schacExpiryDate eq > #index objectClass eq > > Search filter 1 was: (schacExpiryDate<=19800101120000Z) > Result 1 was: user.1 and user.2 > > Search filter 2 was: (schacExpiryDate>=19010101120000Z) > Result 2 was: user.1 and user.2 > > > We could reproduce this with 2.4.42, 2.4.44 and 2.4.48. We hope you can > reproduce this as well to see what is happening here. Yes, can reproduce this. The function that converts a component-wise time into a timet is referencing time since 1970. The date in 1956 would yield a negative timet value but the fields are unsigned ints, so instead it's treated as 17,000 years in the future. We can probably change this to handle signed timestamps but need to consider this further. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9122) slapadd with dry run fails with back-mdb
by quanah＠openldap.org 25 Nov '19

25 Nov '19

Full_Name: Quanah Gibson-Mount Version: 2.4.48 OS: N/A URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.143.26) When testing an slapadd import with back-mdb as the database backend, it errors out incorrectly with a unknown attribute error: slapadd -F /tmp/slapd.d -l slapd-config-config.ldif -n 0 -u 5ddc186e PROXIED attributeDescription "DC" inserted. 5ddc186e <= str2entry: str2ad(olcDbNoSync): attribute type undefined slapadd: could not parse entry (line=1620) _################### 99.10% eta none elapsed none spd 18.7 M/s However the actual LDIF file is just fine: slapadd -F /tmp/slapd.d -l slapd-config.ldif -n 0 _#################### 100.00% eta none elapsed none fast! Closing DB... I'm guessing that on a dry-run, it doesn't actually fully load the modules defined in moduleload to get their schema definitions, which results in a false failure instead of a success.

1 0

(ITS#9121) dynlist enhancements
by quanah＠openldap.org 20 Nov '19

20 Nov '19

Full_Name: Quanah Gibson-Mount Version: 2.4 OS: N/A URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.143.26) There are some enhancements to dynlist that would make it more useful/versatile: a) Be able to do (unique)member= searches to find all groups a given group member belongs to b) be able to generate dynamic (is)memberOf values on user entries for the groups they belong to

1 0

(ITS#9120) Searches including generalizedTime attributes with index
by markus.widmer＠daasi.de 19 Nov '19

19 Nov '19

Full_Name: Markus Widmer Version: 2.4.48 OS: CentOS 7 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (37.24.13.226) Hi! we have recognized a strange behaviour for searches which include an attribute with syntax 1.3.6.1.4.1.1466.115.121.1.24 and the matching rules generalizedTimeMatch and generalizedTimeOrderingMatch set. If the timestamps in the objects have values before 1970, then a search for "<=" or ">=" only seams to works if: - The attribute is not part of the index - The attribute is part of the index but "objectClass" is not Reproduction: We used two entries with the attribute "schacExpiryDate" from the objectClass "schacEntryMetadata" but this can be reproduced with other attributes as well: attributetype ( 1.3.6.1.4.1.25178:1:2:17 NAME 'schacExpiryDate' DESC 'Date from which the set of data is to be considered invalid (format YYYYMMDDhhmmssZ)' EQUALITY generalizedTimeMatch ORDERING generalizedTimeOrderingMatch SINGLE-VALUE SYNTAX 1.3.6.1.4.1.1466.115.121.1.24 ) We used two entries: user.1: schacExpiryDate: 19711007093230.514Z user.2: schacExpiryDate: 19561007093230.514Z First try: we used this index configuration: index sn eq index schacExpiryDate eq index objectClass eq Search filter 1 was: (schacExpiryDate<=19800101120000Z) Result 1 was: user.1 Search filter 2 was: (schacExpiryDate>=19010101120000Z) Result 2 was: user.2 Second try: we used this index configuration: index sn eq #index schacExpiryDate eq index objectClass eq Search filter 1 was: (schacExpiryDate<=19800101120000Z) Result 1 was: user.1 and user.2 Search filter 2 was: (schacExpiryDate>=19010101120000Z) Result 2 was: user.2 and user.2 Third try: we used this index configuration: index sn eq index schacExpiryDate eq #index objectClass eq Search filter 1 was: (schacExpiryDate<=19800101120000Z) Result 1 was: user.1 and user.2 Search filter 2 was: (schacExpiryDate>=19010101120000Z) Result 2 was: user.1 and user.2 We could reproduce this with 2.4.42, 2.4.44 and 2.4.48. We hope you can reproduce this as well to see what is happening here. Thank you very much for doing a great job!

1 0

Re: (ITS#9119) back-monitor can miscount monitorOpCompleted
by hyc＠symas.com 15 Nov '19

15 Nov '19

quanah(a)openldap.org wrote: > Full_Name: Quanah Gibson-Mount > Version: 2.4.43 > OS: N/A > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (47.208.143.26) > > > When querying the monitor backend for monitorOpCompleted, the results are > inconsistent. The code that calculates this value is also inefficient, as there > is already a global counter of completed operations, but back-monitor doesn't > use this and instead queries all the per-operation counters and then sums them > together for the result, causing unnecessary work. Fixed in git master. > > These are the results of a series of queries: > > monitorOpCompleted: 326801 > monitorOpCompleted: 326733 > monitorOpCompleted: 326830 > monitorOpCompleted: 326830 > monitorOpCompleted: 326752 > monitorOpCompleted: 326855 > monitorOpCompleted: 326777 Unable to reproduce. Please test the patch and report back if the problem still shows up. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9118) Add support for MAP_NOSYNC
by hyc＠symas.com 15 Nov '19

15 Nov '19

quanah(a)openldap.org wrote: > Full_Name: Quanah Gibson-Mount > Version: 2.4.48 > OS: FreeBSD 11 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (47.208.143.26) > > > To improve back-mdb performance on LMDB when dbnosync is set, it would be useful > to support MAP_NOSYNC as documented at > http://nixdoc.net/man-pages/FreeBSD/mmap.2.html > > Added in mdb.master -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9119) back-monitor can miscount monitorOpCompleted
by quanah＠openldap.org 15 Nov '19

15 Nov '19

Full_Name: Quanah Gibson-Mount Version: 2.4.43 OS: N/A URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.143.26) When querying the monitor backend for monitorOpCompleted, the results are inconsistent. The code that calculates this value is also inefficient, as there is already a global counter of completed operations, but back-monitor doesn't use this and instead queries all the per-operation counters and then sums them together for the result, causing unnecessary work. These are the results of a series of queries: monitorOpCompleted: 326801 monitorOpCompleted: 326733 monitorOpCompleted: 326830 monitorOpCompleted: 326830 monitorOpCompleted: 326752 monitorOpCompleted: 326855 monitorOpCompleted: 326777

1 0

(ITS#9118) Add support for MAP_NOSYNC
by quanah＠openldap.org 15 Nov '19

15 Nov '19

Full_Name: Quanah Gibson-Mount Version: 2.4.48 OS: FreeBSD 11 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.143.26) To improve back-mdb performance on LMDB when dbnosync is set, it would be useful to support MAP_NOSYNC as documented at http://nixdoc.net/man-pages/FreeBSD/mmap.2.html

1 0

Re: (ITS#9117) LMDB incorrect value for mapsize [Docs]
by hyc＠symas.com 14 Nov '19

14 Nov '19

progr-d(a)yandex.ru wrote: > No, I mean that in the code it is 1MB, and in the documentation - 10MB. Repeating myself just this once: Yes, this is known. The point is that yo= u're not supposed to trust the default value, you're supposed to always set th= e size explicitly. Closing this ITS. > 14 =D0=BD=D0=BE=D1=8F=D0=B1. 2019 =D0=B3., 5:05 +0300, Howard Chu <hyc@= symas.com>, =D0=BF=D0=B8=D1=81=D0=B0=D0=BB: >> progr-d(a)yandex.ru wrote: >>> Full_Name: Denis >>> Version: lmdb 0.9.24 >>> OS: macOS >>> URL: ftp://ftp.openldap.org/incoming/ >>> Submission from: (NULL) (80.211.15.98) >>> >>> >>> Incorrect value in documentation or code for DEFAULT_MAPSIZE: in code= is >>> 1048576, but in doc: 10485760. >>> >>> Code: http://www.lmdb.tech/doc/group__internal.html#ga506f893519db205= 966f7988c03c920f5 >>> Doc (see method description):=C2=A0http://www.lmdb.tech/doc/group__md= b.html#gaa2506ec8dab3d969b0e609cd82e619e5 >> >> Yes, this is known. The point is to force you to explicitly set a sane= value, therefore this >> will never be changed. >> >> -- >> -- Howard Chu >> CTO, Symas Corp. http://www.symas.com >> Director, Highland Sun http://highlandsun.com/hyc/ >> Chief Architect, OpenLDAP http://www.openldap.org/project/ --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9117) LMDB incorrect value for mapsize [Docs]
by progr-d＠yandex.ru 14 Nov '19

14 Nov '19

--5dccf546_46e87ccd_19a8 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline No, I mean that in the code it is 1MB, and in the documentation - 10MB. 14 =D0=BD=D0=BE=D1=8F=D0=B1. 2019 =D0=B3., 5:05 +0300, Howard Chu <hyc=40= symas.com>, =D0=BF=D0=B8=D1=81=D0=B0=D0=BB: > progr-d=40yandex.ru wrote: > > =46ull=5FName: Denis > > Version: lmdb 0.9.24 > > OS: macOS > > URL: ftp://ftp.openldap.org/incoming/ > > Submission from: (NULL) (80.211.15.98) > > > > > > Incorrect value in documentation or code for DE=46AULT=5FMAPSIZE: in = code is > > 1048576, but in doc: 10485760. > > > > Code: http://www.lmdb.tech/doc/group=5F=5Finternal.html=23ga506f89351= 9db205966f7988c03c920f5 > > Doc (see method description):=C2=A0http://www.lmdb.tech/doc/group=5F=5F= mdb.html=23gaa2506ec8dab3d969b0e609cd82e619e5 > > Yes, this is known. The point is to force you to explicitly set a sane = value, therefore this > will never be changed. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ --5dccf546_46e87ccd_19a8 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline <html xmlns=3D=22http://www.w3.org/1999/xhtml=22> <head> <title></title> </head> <body> <div name=3D=22messageBodySection=22> <div dir=3D=22auto=22>No, I mean that in the code it is 1MB, and in the d= ocumentation - 10MB.</div> </div> <div name=3D=22messageReplySection=22>14 =D0=BD=D0=BE=D1=8F=D0=B1. 2019 =D0= =B3., 5:05 +0300, Howard Chu <hyc=40symas.com>, =D0=BF=D0=B8=D1=81=D0= =B0=D0=BB:<br /> <blockquote type=3D=22cite=22 class=3D=22spark=5Fquote=22 style=3D=22marg= in: 5px 5px; padding-left: 10px; border-left: thin solid =231abc9c;=22>pr= ogr-d=40yandex.ru wrote:<br /> <blockquote type=3D=22cite=22 class=3D=22spark=5Fquote=22 style=3D=22marg= in: 5px 5px; padding-left: 10px; border-left: thin solid =23e67e22;=22>=46= ull=5FName: Denis<br /> Version: lmdb 0.9.24<br /> OS: macOS<br /> URL: ftp://ftp.openldap.org/incoming/<br /> Submission from: (NULL) (80.211.15.98)<br /> <br /> <br /> Incorrect value in documentation or code for DE=46AULT=5FMAPSIZE: in code= is<br /> 1048576, but in doc: 10485760.<br /> <br /> Code: http://www.lmdb.tech/doc/group=5F=5Finternal.html=23ga506f893519db2= 05966f7988c03c920f5<br /> Doc (see method description):&=23160;http://www.lmdb.tech/doc/group=5F=5F= mdb.html=23gaa2506ec8dab3d969b0e609cd82e619e5<br /></blockquote> <br /> Yes, this is known. The point is to force you to explicitly set a sane va= lue, therefore this<br /> will never be changed.<br /> <br /> --<br /> -- Howard Chu<br /> CTO, Symas Corp. http://www.symas.com<br /> Director, Highland Sun http://highlandsun.com/hyc/<br /> Chief Architect, OpenLDAP http://www.openldap.org/project/<br /></blockqu= ote> </div> </body> </html> --5dccf546_46e87ccd_19a8--

1 0

Re: (ITS#9117) LMDB incorrect value for mapsize [Docs]
by hyc＠symas.com 14 Nov '19

14 Nov '19

progr-d(a)yandex.ru wrote: > Full_Name: Denis > Version: lmdb 0.9.24 > OS: macOS > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (80.211.15.98) >=20 >=20 > Incorrect value in documentation or code for DEFAULT_MAPSIZE: in code i= s > 1048576, but in doc: 10485760. >=20 > Code: http://www.lmdb.tech/doc/group__internal.html#ga506f893519db20596= 6f7988c03c920f5 > Doc (see method description):=C2=A0http://www.lmdb.tech/doc/group__mdb.= html#gaa2506ec8dab3d969b0e609cd82e619e5 Yes, this is known. The point is to force you to explicitly set a sane va= lue, therefore this will never be changed. --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9117) LMDB incorrect value for mapsize [Docs]
by progr-d＠yandex.ru 13 Nov '19

13 Nov '19

Full_Name: Denis Version: lmdb 0.9.24 OS: macOS URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (80.211.15.98) Incorrect value in documentation or code for DEFAULT_MAPSIZE: in code is 1048576, but in doc: 10485760. Code: http://www.lmdb.tech/doc/group__internal.html#ga506f893519db205966f7988c03c… Doc (see method description): http://www.lmdb.tech/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5

1 0

Re: (ITS#8575) Argon2 Password hashing contrib module
by ondra＠mistotebe.net 12 Nov '19

12 Nov '19

On Mon, Nov 11, 2019 at 05:48:05PM +0000, ondra(a)mistotebe.net wrote: > An implementation using libsodium is now available at > https://github.com/mistotebe/openldap/tree/its8575-argon > > Not configurable yet as to what parameters are chosen when a plaintext > password is being hashed, however. That branch now supports parameters being passed in at module load time. It won't help slappasswd as that one doesn't know how to pass parameters to modules but that is a different issue. -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

1 0

Re: (ITS#8575) Argon2 Password hashing contrib module
by ondra＠mistotebe.net 11 Nov '19

11 Nov '19

On Thu, Jan 31, 2019 at 03:20:22PM +0000, simon(a)slevermann.de wrote: > Hi, > > I have essentially given up on doing this, because I no longer work for > the employer that had me work on this, and at the time I did, I never > got the thing done. The code itself worked when I tried it, but it has > the caveat of not being configurable. I never quite found out how to > properly implement configuration of a module that isn't an overlay, so I > never got that done. Feel free to adjust the existing code, it should be > adaptable to libsodium relatively easily. An implementation using libsodium is now available at https://github.com/mistotebe/openldap/tree/its8575-argon Not configurable yet as to what parameters are chosen when a plaintext password is being hashed, however. Regards, -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

1 0

Re: (ITS#9116) LMDB: Bogus BAD_DBI result on read-only txn
by hyc＠symas.com 10 Nov '19

10 Nov '19

hyc(a)openldap.org wrote: > Full_Name: Howard Chu > Version: 0.9.24 > OS: Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (84.203.24.208) > Submitted by: hyc > > > There are some cases when renewing a cursor in a read-only txn that may return > MDB_BAD_DBI if the DBI has gone stale. This error is spurious, the check is only > supposed to be done in writable txns (see ITS#7825). A patch was pushed to mdb.master, but it is apparently not actually needed. The actual problem was caused by reusing a readtxn that had been left open after its original env had been closed and re-opened. This is a misuse of the API, all txns must be closed before closing the env. Ignore this issue. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9116) LMDB: Bogus BAD_DBI result on read-only txn
by hyc＠openldap.org 10 Nov '19

10 Nov '19

Full_Name: Howard Chu Version: 0.9.24 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (84.203.24.208) Submitted by: hyc There are some cases when renewing a cursor in a read-only txn that may return MDB_BAD_DBI if the DBI has gone stale. This error is spurious, the check is only supposed to be done in writable txns (see ITS#7825).

1 0

Re: (ITS#9115) It seems like the parameters about tlsCipherSuite not work
by quanah＠symas.com 08 Nov '19

08 Nov '19

--On Wednesday, November 6, 2019 8:14 AM +0000 bjmoya(a)cn.ibm.com wrote: > Full_Name: nancy.mo > Version: 2.4.46 > OS: redhat7 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (129.42.208.182) Hello, The ITS system is for bug reports, not help requests. Please redirect your question to the openldap-technical list for further assistance. <https://www.openldap.org/lists/mm/listinfo/openldap-technical> Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#9114) pagedResultsControl not available in openldap search response
by quanah＠symas.com 08 Nov '19

08 Nov '19

--On Monday, November 4, 2019 11:52 AM +0000 prashanthmadduri(a)gmail.com wrote: > Full_Name: Prashanth Madduri > Version: 2.4.40 > OS: Windows > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (103.6.33.5) > > > Hi Team, > > I have huge data in OpenLDAP server and retrieve data using pagination. I > am using LDAPJS client library search functionality with paging to > retrieve the data. However as per my observation the response is not > returning pagedResultsControl in response. Hello, The ITS system is for bug reports, not help requests. Please redirect your question to the openldap-technical list for further assistance. <https://www.openldap.org/lists/mm/listinfo/openldap-technical> Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#9113) i can not find memberOf.la file
by quanah＠symas.com 08 Nov '19

08 Nov '19

--On Tuesday, October 29, 2019 2:47 AM +0000 machao0605(a)qq.com wrote: > Full_Name: ma > Version: 2.4.44 > OS: centos 7.5 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (119.253.39.18) Hello, The ITS system is for bug reports, not help requests. Please redirect your question to the openldap-technical list for further assistance. <https://www.openldap.org/lists/mm/listinfo/openldap-technical> Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9115) It seems like the parameters about tlsCipherSuite not work
by bjmoya＠cn.ibm.com 06 Nov '19

06 Nov '19

Full_Name: nancy.mo Version: 2.4.46 OS: redhat7 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (129.42.208.182) Hi, I set the parameter about cipher suite in client(ldap.conf) and server (slapd.conf) and restart the service, the tcp/ip log, find the cipher not changed. In ldap.conf: TLS_CIPHER_SUITE ALL:!TLSv1.3 In slapd.conf: TLSCipherSuite !TLSv1.3 openssl provide those cipher suites: [root@ ~]# openssl ciphers -v 'TLSv1.3' TLS_AES_256_GCM_SHA384 TLSv1.3 Kx=any Au=any Enc=AESGCM(256) Mac=AEAD TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 Kx=any Au=any Enc=CHACHA20/POLY1305(256) Mac=AEAD TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD when openldap worked as a client, it send 4 cipher suites to server in TLS1.3 client hello. Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff) Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff) Cipher Suite: TLS_CHACHA20_POLY1305_SHA256 (0x1303) Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302) When openldap worked as a server, it used TLS_AES_256_GCM_SHA384 to connect in TLS server hello. And when i set one specific cipher in client, TLS_CIPHER_SUITE TLS_CHACHA20_POLY1305_SHA256 It also send same four suites in client hello. Could you help me to have a look? thanks.

1 0

(ITS#9114) pagedResultsControl not available in openldap search response
by prashanthmadduri＠gmail.com 04 Nov '19

04 Nov '19

Full_Name: Prashanth Madduri Version: 2.4.40 OS: Windows URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (103.6.33.5) Hi Team, I have huge data in OpenLDAP server and retrieve data using pagination. I am using LDAPJS client library search functionality with paging to retrieve the data. However as per my observation the response is not returning pagedResultsControl in response. Please assist. Thanks in advance. Regards, Prashanth

1 0

(ITS#9113) i can not find memberOf.la file
by machao0605＠qq.com 29 Oct '19

29 Oct '19

Full_Name: ma Version: 2.4.44 OS: centos 7.5 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (119.253.39.18) Dear develops: I hava some trouble use openldap . I use openldap-2.4.44.tgz package. when i install this software . I can not find olcModulePath. i see /usr/local/etc/openldap/slapd.ldif it show this dir is /usr/local/libexec/openldap but i can not find it . i need a file name is memberOf.la . so could you help me to find it ? thanks ! This is me install openldap step: 1../configure --with-tls=openssl --enable-syslog --enable-module -- enable-debug CPPFLAGS="-I/usr/local/bdb_5.2.42/include" LDFLAGS="- L/usr/local/bdb_5.2.42/lib -Wl,-rpath,/usr/local/bdb_5.2.42/lib" 2.make depend 3.make 4.make install

1 0

(ITS#9112) Fixup socket error handling
by hyc＠openldap.org 29 Oct '19

29 Oct '19

Full_Name: Howard Chu Version: 2.4 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (84.203.24.208) Submitted by: hyc During connection setup, there are a couple of places where we perform operations on a socket and log an error if it fails, but otherwise keep going. It turns out, if the error is EBADF, eventually this gets back to the event loop, which always does a clean shutdown on any socket error. We should instead stop init'ing the socket, and not hand it over to the event loop. This situation only arose because of a long-standing bug in 3rd party code that was double-closing an fd. https://github.com/heimdal/heimdal/issues/431 . In normal situations, none of this can ever occur. Regardless, tracking this here and committing the debug code we used to track it down, in case we ever need it again in the future.

1 0

Re: (ITS#9107) new feature
by quanah＠symas.com 28 Oct '19

28 Oct '19

--On Monday, October 28, 2019 2:33 AM +0000 ydgdsnn(a)163.com wrote: > Full_Name: Nannan Song I would ask in the future that you not spam the bug system with 8 copies of the same report. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#9100) Issue with domainScope controlValue implementation
by hyc＠symas.com 28 Oct '19

28 Oct '19

Philip Brusten wrote: >=20 > On 25/10/2019 18:41, Howard Chu wrote: >> philip.brusten(a)kuleuven.be wrote: >>> That fix was based on (old?) documentation of MS from >>> https://msdn.microsoft.com/en-us/library/aa366979%28v=3Dvs.85%29.aspx >>> The is no current version of this document. >> The above URL redirects me to https://docs.microsoft.com/en-us/previou= s-versions/windows/desktop/ldap/ldap-server-domain-scope-oid >> >> Our implementation conforms to the above spec. > Correct, but the URL path contains "previous-versions", hence I was loo= king for the current version, which does not exist. >> This is the correct response to a malformed message according to RFC45= 11. >=20 > Could you please quote the RFC4511 on this?=C2=A0 This is not 100% clea= r to me... This is simply the definition of the protocolError result code. RFC 4511 = Appendix A, section A.2. >>> We observed that the controlValue is than missing via Wireshark. So w= e see a >>> difference on the line for empty octet strings, but should it matter? >> Yes, in ASN.1 "empty value" and "absent value" are not the same thing.= E.g., you can >> store zero-length strings as attribute values. Those values are "prese= nt" even though >> they are empty. >=20 > IMHO Microsoft and OpenLDAP are doing the same thing on a protocol leve= l. You are both setting the controlValue to a struct {0,NULL}. >=20 > However with Microsoft, this is still translated to the network level, = whereas with OpenLDAP this controlValue is omitted. >=20 > The "value" of the "controlValue" struct is however still NULL. So it's= not clear to me why it's translated to empty, not null. (is there an som= ewhere an > assumption that translates zero-length octet string to non null values?= ) See the ASN.1 specification of controls, RFC 4511 section 4.1.11. The val= ue must be absent if there is no value information that is associated with a control of its= type. >> MS is playing fast and loose with their specifications and implementat= ion. This is >> at the very least a doc bug in their control specification, if it's no= t just a bug >> in their client libraries. You should at least open a bug report with = them so that >> it's on the record somewhere (even if they ultimately ignore it). >> >> We can alter our parser to relax this check, I guess. Along with a com= ment explaining >> that this is done for compatibility with MS's broken clients. This is = not the first >> time we've run into MS spec and implementation disagreeing with each o= ther... >=20 > We will submit the issue to MS. But in the case of this domainScope con= trol, it should not matter if the value is empty or missing, only the con= trolType is > relevant. Could Postel's law be applied in this case? As I already said, we can relax this check. But considering that Microsoft are the authors of both the spec and their= implementation of this control, and the two don't agree, and this is not the only instance = of such occurrences (the pagedResults control also comes to mind) you have to wonder - are th= ey just too stupid and incompetent to implement their own spec, or have they broken things i= ntentionally, to prevent their clients from working with non-Microsoft servers... --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9100) Issue with domainScope controlValue implementation
by philip.brusten＠kuleuven.be 28 Oct '19

28 Oct '19

On 25/10/2019 18:41, Howard Chu wrote: > philip.brusten(a)kuleuven.be wrote: >> That fix was based on (old?) documentation of MS from >> https://msdn.microsoft.com/en-us/library/aa366979%28v=vs.85%29.aspx >> The is no current version of this document. > The above URL redirects me to https://docs.microsoft.com/en-us/previous-versions/windows/desktop/ldap/lda… > > Our implementation conforms to the above spec. Correct, but the URL path contains "previous-versions", hence I was looking for the current version, which does not exist. > This is the correct response to a malformed message according to RFC4511. Could you please quote the RFC4511 on this? This is not 100% clear to me... >> The LDAP Extended Control spec of MS states [2]: >>> If the controlValue field contains data that is not in conformance with the >>> specification of the control, including the case where the controlValue field contains data and the specification of the control states that the controlValue field is omitted, then if the control is marked critical the server returns the error unavailableCriticalExtension / ERROR_INVALID_PARAMETER. If the controlValue field is incorrect but the control is not marked critical, the server ignores the control. > MS seems to be inventing their own protocol here since RFC4511 contains no such language. >> So the server should only return an error when the control is marked critical. >> Otherwise it should *ignore* the control. >> >> Is it possible to relax the validation of the LDAP control at [3]? Since there >> is no value associated with this control, it should not matter if the >> controlValue is missing or an empty octet string {0,NULL}... >> >> The ldapsearch code sets the controlValue to [4]: >>> c[i].ldctl_value.bv_val = NULL; >>> c[i].ldctl_value.bv_len = 0; >> We observed that the controlValue is than missing via Wireshark. So we see a >> difference on the line for empty octet strings, but should it matter? > Yes, in ASN.1 "empty value" and "absent value" are not the same thing. E.g., you can > store zero-length strings as attribute values. Those values are "present" even though > they are empty. IMHO Microsoft and OpenLDAP are doing the same thing on a protocol level. You are both setting the controlValue to a struct {0,NULL}. However with Microsoft, this is still translated to the network level, whereas with OpenLDAP this controlValue is omitted. The "value" of the "controlValue" struct is however still NULL. So it's not clear to me why it's translated to empty, not null. (is there an somewhere an assumption that translates zero-length octet string to non null values?) > MS is playing fast and loose with their specifications and implementation. This is > at the very least a doc bug in their control specification, if it's not just a bug > in their client libraries. You should at least open a bug report with them so that > it's on the record somewhere (even if they ultimately ignore it). > > We can alter our parser to relax this check, I guess. Along with a comment explaining > that this is done for compatibility with MS's broken clients. This is not the first > time we've run into MS spec and implementation disagreeing with each other... We will submit the issue to MS. But in the case of this domainScope control, it should not matter if the value is empty or missing, only the controlType is relevant. Could Postel's law be applied in this case? Regards, Philip

1 0

ITS#9106 published: SECURITY: new feature
by openldap-its＠OpenLDAP.org 28 Oct '19

28 Oct '19

Private ITS#9106 is now publicly readable URL: http://www.OpenLDAP.org/its/?findid=9106

1 0

ITS#9105 published: SECURITY: new feature
by openldap-its＠OpenLDAP.org 28 Oct '19

28 Oct '19

Private ITS#9105 is now publicly readable URL: http://www.OpenLDAP.org/its/?findid=9105

1 0

ITS#9104 published: SECURITY: new feature
by openldap-its＠OpenLDAP.org 28 Oct '19

28 Oct '19

Private ITS#9104 is now publicly readable URL: http://www.OpenLDAP.org/its/?findid=9104

1 0

ITS#9103 published: SECURITY: new feature
by openldap-its＠OpenLDAP.org 28 Oct '19

28 Oct '19

Private ITS#9103 is now publicly readable URL: http://www.OpenLDAP.org/its/?findid=9103

1 0

Re: (ITS#9107) new feature
by hyc＠symas.com 28 Oct '19

28 Oct '19

ydgdsnn(a)163.com wrote: > Full_Name: Nannan Song > Version: 2.4.44 > OS: SUSE > URL: > Submission from: (NULL) (221.226.97.96) > > > When LDAP is used to manage user and user group information, openldap only > supports the configuration of the plain text password of the read-only user in > the '/etc/ldap.conf/'. The password of the read-only user only supports plain > text storage. so there is a security issue that the authentication credential > file is readable to all users. > Now we hope ldap can support the feature that using the encrypted text to save > password for read only user. We saw this the first time, no need to resubmit it 10 times. Supposing you could put an encrypted password into ldap.conf - where would you put the key for decrypting the password, so that the software can use it? When LDAP is *correctly* used to manage user and group information, the credentials used to contact the LDAP server belong to a low-privilege account, so that theft of those credentials is of minimal harm. And they are used by a single authentication daemon (like nslcd in the nss-pam-ldapd package) and as such never appear in any world-readable files. Closing this ITS and all the other copies of it. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9111) new feature:using the encrypted text to save password for read only user
by ydgdsnn＠163.com 28 Oct '19

28 Oct '19

Full_Name: Nannan Song Version: 2.4.44 OS: SUSE URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (221.226.97.96) When LDAP is used to manage user and user group information, openldap only supports the configuration of the plain text password of the read-only user in the '/etc/ldap.conf/'. The password of the read-only user only supports plain text storage. so there is a security issue that the authentication credential file is readable to all users. Now we hope ldap can support the feature that using the encrypted text to save password for read only user.

1 0

(ITS#9110) new feature
by ydgdsnn＠163.com 28 Oct '19

28 Oct '19

Full_Name: Nannan Song Version: 2.4.44 OS: SUSE URL: Submission from: (NULL) (221.226.97.96) When LDAP is used to manage user and user group information, openldap only supports the configuration of the plain text password of the read-only user in the '/etc/ldap.conf/'. The password of the read-only user only supports plain text storage. so there is a security issue that the authentication credential file is readable to all users. Now we hope ldap can support the feature that using the encrypted text to save password for read only user.

1 0

(ITS#9109) new feature
by ydgdsnn＠163.com 28 Oct '19

28 Oct '19

Full_Name: Nannan Song Version: 2.4.44 OS: SUSE URL: Submission from: (NULL) (221.226.97.96) When LDAP is used to manage user and user group information, openldap only supports the configuration of the plain text password of the read-only user in the '/etc/ldap.conf/'. The password of the read-only user only supports plain text storage. so there is a security issue that the authentication credential file is readable to all users. Now we hope ldap can support the feature that using the encrypted text to save password for read only user.

1 0

(ITS#9108) new feature
by ydgdsnn＠163.com 28 Oct '19

28 Oct '19

Full_Name: Nannan Song Version: 2.4.44 OS: SUSE URL: Submission from: (NULL) (221.226.97.96) When LDAP is used to manage user and user group information, openldap only supports the configuration of the plain text password of the read-only user in the '/etc/ldap.conf/'. The password of the read-only user only supports plain text storage. so there is a security issue that the authentication credential file is readable to all users. Now we hope ldap can support the feature that using the encrypted text to save password for read only user.

1 0

(ITS#9107) new feature
by ydgdsnn＠163.com 28 Oct '19

28 Oct '19

Full_Name: Nannan Song Version: 2.4.44 OS: SUSE URL: Submission from: (NULL) (221.226.97.96) When LDAP is used to manage user and user group information, openldap only supports the configuration of the plain text password of the read-only user in the '/etc/ldap.conf/'. The password of the read-only user only supports plain text storage. so there is a security issue that the authentication credential file is readable to all users. Now we hope ldap can support the feature that using the encrypted text to save password for read only user.

1 0

(ITS#9102) Update sasl-secprops ssf wording
by quanah＠openldap.org 25 Oct '19

25 Oct '19

Full_Name: Quanah Gibson-Mount Version: 2.4.48 OS: N/A URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.154.120) The slapd.conf(5)/slapd-config(5) man pages contain this wording for the sasl-secprops keyword: 0 (zero) implies no protection, 1 implies integrity protection only, 56 allows DES or other weak ciphers, 112 allows triple DES and other strong ciphers, 128 allows RC4, Blowfish and other modern strong ciphers. The default is 0. I think it's been quite a while since RC4 was considered a "modern strong cipher".

1 0

Re: (ITS#9101) man pages don't reflect some global options are actually global/database
by ryan＠nardis.ca 25 Oct '19

25 Oct '19

On Fri, Oct 25, 2019 at 09:15:33PM +0000, quanah(a)symas.com wrote: >Correction, slapd-config(5) handles this via discussion of the frontend >database. Sort of - at least it has that distinction, but the three attrs you mentioned are still documented only in the GLOBAL CONFIGURATION OPTIONS section and not in the GLOBAL DATABASE OPTIONS section, while they actually are accepted on both, as you noted.

1 0

Re: (ITS#9101) man pages don't reflect some global options are actually global/database
by quanah＠symas.com 25 Oct '19

25 Oct '19

--On Friday, October 25, 2019 10:09 PM +0000 quanah(a)openldap.org wrote: > Full_Name: Quanah Gibson-Mount > Version: 2.4.48 > OS: N/A > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (47.208.154.120) > > > The slapd.conf(5)/slapd-config(5) man pages list the security/olcSecurity > configuration option as a "GLOBAL" only config option. Correction, slapd-config(5) handles this via discussion of the frontend database. It's slapd.conf(5) where this is a mess (doesn't have any differentiation). --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9101) man pages don't reflect some global options are actually global/database
by quanah＠openldap.org 25 Oct '19

25 Oct '19

Full_Name: Quanah Gibson-Mount Version: 2.4.48 OS: N/A URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.154.120) The slapd.conf(5)/slapd-config(5) man pages list the security/olcSecurity configuration option as a "GLOBAL" only config option. However, inspection of the code reveals that it is in fact global OR a general database option. The man pages should be updated to reflect this fact. The same issue exists for the sizelimit and timelimit directives as well. May need to add a new structure to the manual pages to cover these 3 items, as they appear to be the only things that can exist as both global and per-db items. I.e., we should cover what happens if they are set both globally and per-db (what takes precedence, etc).

1 0

Re: (ITS#9100) Issue with domainScope controlValue implementation
by hyc＠symas.com 25 Oct '19

25 Oct '19

philip.brusten(a)kuleuven.be wrote: > Full_Name: Philip Brusten > Version: 2.4.47 > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (2a02:2c40:100:b210::1:c3bc) > > > Hi > > we are experiencing issues with MS LDAP clients that connect to our OpenLDAP > servers since v2.4.47, because of the fix for issue ITS#8840 > > That fix was based on (old?) documentation of MS from > https://msdn.microsoft.com/en-us/library/aa366979%28v=vs.85%29.aspx > The is no current version of this document. The above URL redirects me to https://docs.microsoft.com/en-us/previous-versions/windows/desktop/ldap/lda… Our implementation conforms to the above spec. > At > https://docs.microsoft.com/en-us/windows/win32/api/_ldap/ there is a link to > Lightweigt Directory Access Protocol > (https://docs.microsoft.com/en-us/windows/desktop/ldap), but it gives a 404. > > MS is the owner of that LDAPControl LDAP_SERVER_DOMAIN_SCOPE_OID > 1.2.840.113556.1.4.1339. According to the RFC4511 [1] the controlValue is > defined by its specification: > >> The controlValue may contain information associated with the >> controlType. Its format is defined by the specification of the >> control. Implementations MUST be prepared to handle arbitrary >> contents of the controlValue octet string, including zero bytes. It >> is absent only if there is no value information that is associated >> with a control of its type. When a controlValue is defined in terms >> of ASN.1, and BER-encoded according to Section 5.1, it also follows >> the extensibility rules in Section 4. > > The latest specification can be found at: > https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/ba5f20… > >> "When sending this control to the DC, the controlValue field of the Control > structure is omitted." > > We observed that whenever MS clients are sending the domainScope control they > add an octet string as the controlValue with bytes "04 00". But this causes the > OpenLDAP code > if ( !BER_BVISNULL( &ctrl->ldctl_value )) { > to evaluate to true, which returns the LDAP_PROTOCOL_ERROR in OpenLDAP. This is the correct response to a malformed message according to RFC4511. > > The LDAP Extended Control spec of MS states [2]: >> If the controlValue field contains data that is not in conformance with the >> specification of the control, including the case where the controlValue field > contains >> data and the specification of the control states that the controlValue field > is omitted, >> then if the control is marked critical the server returns the error >> unavailableCriticalExtension / ERROR_INVALID_PARAMETER. If the controlValue > field is >> incorrect but the control is not marked critical, the server ignores the > control. MS seems to be inventing their own protocol here since RFC4511 contains no such language. > So the server should only return an error when the control is marked critical. > Otherwise it should *ignore* the control. > > Is it possible to relax the validation of the LDAP control at [3]? Since there > is no value associated with this control, it should not matter if the > controlValue is missing or an empty octet string {0,NULL}... > > The ldapsearch code sets the controlValue to [4]: >> c[i].ldctl_value.bv_val = NULL; >> c[i].ldctl_value.bv_len = 0; > We observed that the controlValue is than missing via Wireshark. So we see a > difference on the line for empty octet strings, but should it matter? Yes, in ASN.1 "empty value" and "absent value" are not the same thing. E.g., you can store zero-length strings as attribute values. Those values are "present" even though they are empty. MS is playing fast and loose with their specifications and implementation. This is at the very least a doc bug in their control specification, if it's not just a bug in their client libraries. You should at least open a bug report with them so that it's on the record somewhere (even if they ultimately ignore it). We can alter our parser to relax this check, I guess. Along with a comment explaining that this is done for compatibility with MS's broken clients. This is not the first time we've run into MS spec and implementation disagreeing with each other... > > Wireshark trace of MS LDAP client with domainScope control: > https://i.imgur.com/gwBjLMr.png > Wireshark trace of ldapsearch with domainScope control: > https://i.imgur.com/2RbFkIY.png > > Regards, > Philip > > > [1] https://tools.ietf.org/html/rfc4511#section-4.1.11 > [2] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/3c5e87… > [3] https://github.com/openldap/openldap/blob/master/servers/slapd/controls.c#L… > [4] https://github.com/openldap/openldap/blob/master/clients/tools/ldapsearch.c… > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by markus＠greenrobot.de 24 Oct '19

24 Oct '19

On 24.10.19 15:03, Markus Junginger wrote: > per transaction MDB_TRPAGE_SIZE bytes is allocated I guess, I was too quick here. It's MDB_TRPAGE_SIZE * 16. Also, I guess, txn->mt_rpages are just pointers to in-memory cached pages (or chunks?), so the memory consumption of those is much higher. The docs mention "chunks of 16 pages"; does that refer to mt_rpages? If so, a single mt_rpages pointer can actually hold on to 16 * 4K = 64 KB? Thus a MDB_TRPAGE_SIZE 4096 can go up to 256 MB memory? That changes the picture of course... PS.: let me express the irony once more that "removing data" is the most costly operation here ;-)

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by markus＠greenrobot.de 24 Oct '19

24 Oct '19

On 24.10.19 03:49, Howard Chu wrote: > You're free to define MDB_TRPAGE_MAX to a larger value. It just means > you increase the chance of overrunning the 2GB available address space. > There's no magic, you can't fit every 64bit database workload into only > a 32bit address space. When your transactions are too large, the normal > thing to do is commit more often so they don't grow so large. I think you meant MDB_TRPAGE_SIZE? At least that seemed to work, while MDB_TRPAGE_MAX ended up in another MDB_TXN_FULL (tl[0].mid < MDB_TRPAGE_SIZE check failed). Doubling MDB_TRPAGE_SIZE also doubled the threshold of the object count where it starts failing: using 8192 it was able to remove 4M entries while the limit with 4096 was 2M entries. MDB_TRPAGE_SIZE is only used to malloc txn->mt_rpages (and some checks), as far as I can tell. To make a reasonable decision here, could you please confirm: 1. txn->mt_rpages is RAM only and has no impact on mmaped sections (and/or file format). 2. RAM consumption is defined by number of transactions only: per transaction MDB_TRPAGE_SIZE bytes is allocated. Other than that it does not increase memory consumption. 3. There is no other disadvantage other than higher memory consumption per transaction as written in the previous point. In that case, I think I'll go with 16K (or even 32K). Does not seem too much for an transaction, given that it quadruples the amount of data allowed to be processed. 32 bit devices are not servers, so I do not expect a high number of concurrent transactions anyway. Thanks! Markus

1 0

(ITS#9100) Issue with domainScope controlValue implementation
by philip.brusten＠kuleuven.be 24 Oct '19

24 Oct '19

Full_Name: Philip Brusten Version: 2.4.47 OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2a02:2c40:100:b210::1:c3bc) Hi we are experiencing issues with MS LDAP clients that connect to our OpenLDAP servers since v2.4.47, because of the fix for issue ITS#8840 That fix was based on (old?) documentation of MS from https://msdn.microsoft.com/en-us/library/aa366979%28v=vs.85%29.aspx The is no current version of this document. At https://docs.microsoft.com/en-us/windows/win32/api/_ldap/ there is a link to Lightweigt Directory Access Protocol (https://docs.microsoft.com/en-us/windows/desktop/ldap), but it gives a 404. MS is the owner of that LDAPControl LDAP_SERVER_DOMAIN_SCOPE_OID 1.2.840.113556.1.4.1339. According to the RFC4511 [1] the controlValue is defined by its specification: > The controlValue may contain information associated with the > controlType. Its format is defined by the specification of the > control. Implementations MUST be prepared to handle arbitrary > contents of the controlValue octet string, including zero bytes. It > is absent only if there is no value information that is associated > with a control of its type. When a controlValue is defined in terms > of ASN.1, and BER-encoded according to Section 5.1, it also follows > the extensibility rules in Section 4. The latest specification can be found at: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/ba5f20… > "When sending this control to the DC, the controlValue field of the Control structure is omitted." We observed that whenever MS clients are sending the domainScope control they add an octet string as the controlValue with bytes "04 00". But this causes the OpenLDAP code if ( !BER_BVISNULL( &ctrl->ldctl_value )) { to evaluate to true, which returns the LDAP_PROTOCOL_ERROR in OpenLDAP. The LDAP Extended Control spec of MS states [2]: > If the controlValue field contains data that is not in conformance with the > specification of the control, including the case where the controlValue field contains > data and the specification of the control states that the controlValue field is omitted, > then if the control is marked critical the server returns the error > unavailableCriticalExtension / ERROR_INVALID_PARAMETER. If the controlValue field is > incorrect but the control is not marked critical, the server ignores the control. So the server should only return an error when the control is marked critical. Otherwise it should *ignore* the control. Is it possible to relax the validation of the LDAP control at [3]? Since there is no value associated with this control, it should not matter if the controlValue is missing or an empty octet string {0,NULL}... The ldapsearch code sets the controlValue to [4]: > c[i].ldctl_value.bv_val = NULL; > c[i].ldctl_value.bv_len = 0; We observed that the controlValue is than missing via Wireshark. So we see a difference on the line for empty octet strings, but should it matter? Wireshark trace of MS LDAP client with domainScope control: https://i.imgur.com/gwBjLMr.png Wireshark trace of ldapsearch with domainScope control: https://i.imgur.com/2RbFkIY.png Regards, Philip [1] https://tools.ietf.org/html/rfc4511#section-4.1.11 [2] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/3c5e87… [3] https://github.com/openldap/openldap/blob/master/servers/slapd/controls.c#L… [4] https://github.com/openldap/openldap/blob/master/clients/tools/ldapsearch.c…

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by hyc＠symas.com 24 Oct '19

24 Oct '19

markus(a)greenrobot.de wrote: > On 21.10.19 17:14, Howard Chu wrote: >> I believe this was fixed by commit=20 >> 7edf504106c61639a89b9a4e5987242598196932 in mdb.master.=20 >=20 > I can not confirm that this works. >=20 > This is the stack trace where MDB_TXN_FULL is still returned with lates= t=20 > mdb.master (note: line numbers shown here are off by 60 compared to=20 > mdb.master): >=20 > mdb_rpage_get mdb.c:6196 > mdb_page_get mdb.c:6378 > mdb_page_search_lowest mdb.c:6492 > mdb_node_move mdb.c:8842 > mdb_rebalance mdb.c:9366 > mdb_page_merge mdb.c:9166 > mdb_rebalance mdb.c:9373 > mdb_cursor_del0 mdb.c:9426 > mdb_cursor_del mdb.c:811 >=20 >=20 > Code segment: >=20 > if (tl[0].mid >=3D MDB_TRPAGE_MAX) > =C3=82=C2=A0=C3=82=C2=A0=C3=82=C2=A0 return MDB_TXN_FULL; >=20 > Debugger shows tl[0].mid to be 4095. You're free to define MDB_TRPAGE_MAX to a larger value. It just means you increase the chance of overrunning the 2GB available address space. There's no magic, you can't fit every 64bit database workload into only a 32bit address space. When your transactions are too large, the normal thing to do is commit more often so they don't grow so large. --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by markus＠greenrobot.de 23 Oct '19

23 Oct '19

On 21.10.19 17:14, Howard Chu wrote: > I believe this was fixed by commit > 7edf504106c61639a89b9a4e5987242598196932 in mdb.master. I can not confirm that this works. This is the stack trace where MDB_TXN_FULL is still returned with latest mdb.master (note: line numbers shown here are off by 60 compared to mdb.master): mdb_rpage_get mdb.c:6196 mdb_page_get mdb.c:6378 mdb_page_search_lowest mdb.c:6492 mdb_node_move mdb.c:8842 mdb_rebalance mdb.c:9366 mdb_page_merge mdb.c:9166 mdb_rebalance mdb.c:9373 mdb_cursor_del0 mdb.c:9426 mdb_cursor_del mdb.c:811 Code segment: if (tl[0].mid >= MDB_TRPAGE_MAX) return MDB_TXN_FULL; Debugger shows tl[0].mid to be 4095. Hope that helps. Markus

1 0

Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
by hyc＠symas.com 21 Oct '19

21 Oct '19

markus(a)greenrobot.de wrote: > Full_Name: Markus Junginger > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (77.189.91.168) > > > We have a 1GB LMDB file with 7M K/V entries. With MDB_VL32, we always get a > MDB_TXN_FULL error for a transaction that is removing 2M entries (probably fails > at a lower count, we haven't measured that). Without MDB_VL32 it works fine. > > Another observation: > Once the transaction fails with MDB_TXN_FULL, the data file has grown to 1.5GB. > Without MDB_VL32, the data file stays consistent at 1 GB even if all 7M entries > are deleted in a single transaction. > > Expected behavior: > No MDB_TXN_FULL error, no data file growth. > > > I believe this was fixed by commit 7edf504106c61639a89b9a4e5987242598196932 in mdb.master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9099) LMDB: range delete
by hyc＠symas.com 21 Oct '19

21 Oct '19

opensource(a)gmx-topmail.de wrote: > Full_Name: Markus > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (91.64.106.220) > > > We built a database with LMDB and are very pleased with it in general. The one > area giving our users headaches regularly are bulk deletes. But some context > first: > > Like many others, we use a hierarchical key notation. Some simplified examples: > "user/00001", "user/00002", "user/00003", ...; "usergroup/00001", > "usergroup/00002", .... > > Or, let's say data by timestamp: "sensordata/32489732", "sensordata/32489733", > ... > > Use case like "remove all users" or "remove all sensordata entries from time x > to y" are usual. Those (transactional) operations call mdb_cursor_del() on each > entry. This runs fine until we have LOTS of entries to delete. At a magnitude of > 100,000s of entries, two things happen: 1) the time it takes to delete gets > quite noticeable and 2) the 32 bit (MDB_VL32) will run into MDB_TXN_FULL (see > also #8813). > > Given the key structure, those operations operate on consecutive keys. Because > of that, can we make this more efficient (and get rid MDB_TXN_FULL on the fly)? > > A high number of individual deletes involves lots of page merging (and a lot of > dirty pages). If we had a specialized "range delete" this could likely be > avoided. The signature of the proposed functionality could look like this: The number of pages dirtied will be the same, whether you batch or delete one by one. As such, this won't solve the MDB_TXN_FULL problem. Also, as of commit 7edf504106c61639a89b9a4e5987242598196932 this problem should have already been removed for MDB_VL32. > int mdb_cursor_del_range(MDB_cursor *mc, MDB_val *key_start, MDB_val *key_stop, > unsigned int flags) > > Given key_start and key_stop, this new function could decide to prune entire > leaf pages, or even entire sub branches. This should result in a significant > efficiency/performance gain. The challenge I'm seeing here is balancing the tree > though. Right, rebalancing would become quite a pain. A similar problem exists for PUT_MULTIPLE which is why it just internally loops over one item at a time. > What do you think? I'm not sure if OpenLdap would profit here, but I'm quite > certain that LMDB as a general database would. I'm also open to contribute if > you are willing to give some guidance. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9099) LMDB: range delete
by opensource＠gmx-topmail.de 21 Oct '19

21 Oct '19

Full_Name: Markus Version: OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (91.64.106.220) We built a database with LMDB and are very pleased with it in general. The one area giving our users headaches regularly are bulk deletes. But some context first: Like many others, we use a hierarchical key notation. Some simplified examples: "user/00001", "user/00002", "user/00003", ...; "usergroup/00001", "usergroup/00002", .... Or, let's say data by timestamp: "sensordata/32489732", "sensordata/32489733", ... Use case like "remove all users" or "remove all sensordata entries from time x to y" are usual. Those (transactional) operations call mdb_cursor_del() on each entry. This runs fine until we have LOTS of entries to delete. At a magnitude of 100,000s of entries, two things happen: 1) the time it takes to delete gets quite noticeable and 2) the 32 bit (MDB_VL32) will run into MDB_TXN_FULL (see also #8813). Given the key structure, those operations operate on consecutive keys. Because of that, can we make this more efficient (and get rid MDB_TXN_FULL on the fly)? A high number of individual deletes involves lots of page merging (and a lot of dirty pages). If we had a specialized "range delete" this could likely be avoided. The signature of the proposed functionality could look like this: int mdb_cursor_del_range(MDB_cursor *mc, MDB_val *key_start, MDB_val *key_stop, unsigned int flags) Given key_start and key_stop, this new function could decide to prune entire leaf pages, or even entire sub branches. This should result in a significant efficiency/performance gain. The challenge I'm seeing here is balancing the tree though. What do you think? I'm not sure if OpenLdap would profit here, but I'm quite certain that LMDB as a general database would. I'm also open to contribute if you are willing to give some guidance. Thanks for your consideration, Markus

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs