- openldap-bugs - openldap.org

Re: (ITS#7985) Recursive values
by andrew.findlay＠skills-1st.co.uk 19 Nov '14

19 Nov '14

On Tue, Nov 18, 2014 at 07:48:28PM +0000, michael(a)stroeder.com wrote: > > Same problems (recursion) with slapcat - > > In this case I rather suspect something's wrong with your LDAP management > client application. I'd check the slapd logs covering the period the entries > were created. Good point. The critical thing to note in the slapcat output is that the entryUUID values are all different. I had missed that first time around but it clearly indicates that those entries actually exist. Andrew -- ----------------------------------------------------------------------- | From Andrew Findlay, Skills 1st Ltd | | Consultant in large-scale systems, networks, and directory services | | http://www.skills-1st.co.uk/ +44 1628 782565 | -----------------------------------------------------------------------

1 0

Re: (ITS#7985) Recursive values
by michael＠stroeder.com 18 Nov '14

18 Nov '14

belykh.o(a)gmail.com wrote: > Same problems (recursion) with slapcat - In this case I rather suspect something's wrong with your LDAP management client application. I'd check the slapd logs covering the period the entries were created. Ciao, Michael.

1 0

Re: (ITS#7985) Recursive values
by belykh.o＠gmail.com 18 Nov '14

18 Nov '14

--f46d04428fb47234e2050826ed02 Content-Type: text/plain; charset=UTF-8 Database reindexing: no effect. Overlays: overlay syncprov Same problems (recursion) with slapcat - dn: telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=2020,ou=phones ,ou=accounts,dc=... telephoneNumber: 1000 telephoneNumberAccessCode: 8864 objectClass: telephoneNumberAccount userPassword:: ... structuralObjectClass: telephoneNumberAccount entryUUID: bc8a4014-0355-1034-8cad-1351d00c5bbd creatorsName: cn=ldroot,dc=... createTimestamp: 20141118100223Z entryCSN: 20141118100223.423286Z#000000#000#000000 modifiersName: cn=ldroot,dc=... modifyTimestamp: 20141118100223Z dn: telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,telephone Number=2020,ou=phones,ou=accounts,dc=... telephoneNumber: 1000 telephoneNumberAccessCode: 8864 objectClass: telephoneNumberAccount userPassword:: ... structuralObjectClass: telephoneNumberAccount entryUUID: bc8d2860-0355-1034-8cae-1351d00c5bbd creatorsName: cn=ldroot,dc=... createTimestamp: 20141118100223Z entryCSN: 20141118100223.442503Z#000000#000#000000 modifiersName: cn=ldroot,dc=... modifyTimestamp: 20141118100223Z dn: telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,telephone Number=1000,telephoneNumber=2020,ou=phones,ou=accounts,dc=... telephoneNumber: 1000 telephoneNumberAccessCode: 8864 objectClass: telephoneNumberAccount userPassword:: ... structuralObjectClass: telephoneNumberAccount entryUUID: bc914c06-0355-1034-8caf-1351d00c5bbd creatorsName: cn=ldroot,dc=... createTimestamp: 20141118100223Z entryCSN: 20141118100223.469632Z#000000#000#000000 modifiersName: cn=ldroot,dc=... modifyTimestamp: 20141118100223Z dn: telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,telephone Number=1000,telephoneNumber=1000,telephoneNumber=2020,ou=phones,ou=accounts ,dc=... telephoneNumber: 1000 telephoneNumberAccessCode: 8864 objectClass: telephoneNumberAccount userPassword:: ... structuralObjectClass: telephoneNumberAccount entryUUID: bc9429a8-0355-1034-8cb0-1351d00c5bbd creatorsName: cn=ldroot,dc=... createTimestamp: 20141118100223Z On 18 November 2014 17:47, <openldap-its(a)openldap.org> wrote: > > *** THIS IS AN AUTOMATICALLY GENERATED REPLY *** > > Thanks for your report to the OpenLDAP Issue Tracking System. Your > report has been assigned the tracking number ITS#7985. > > One of our support engineers will look at your report in due course. > Note that this may take some time because our support engineers > are volunteers. They only work on OpenLDAP when they have spare > time. > > If you need to provide additional information in regards to your > issue report, you may do so by replying to this message. Note that > any mail sent to openldap-its(a)openldap.org with (ITS#7985) > in the subject will automatically be attached to the issue report. > > mailto:openldap-its@openldap.org?subject=(ITS#7985) > > You may follow the progress of this report by loading the following > URL in a web browser: > http://www.OpenLDAP.org/its/index.cgi?findid=7985 > > Please remember to retain your issue tracking number (ITS#7985) > on any further messages you send to us regarding this report. If > you don't then you'll just waste our time and yours because we > won't be able to properly track the report. > > Please note that the Issue Tracking System is not intended to > be used to seek help in the proper use of OpenLDAP Software. > Such requests will be closed. > > OpenLDAP Software is user supported. > http://www.OpenLDAP.org/support/ > > -------------- > Copyright 1998-2007 The OpenLDAP Foundation, All Rights Reserved. > > --f46d04428fb47234e2050826ed02 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Database reindexing: no effect.</div><div><br></div><= div>Overlays:</div><div>overlay syncprov</div><div><br></div><div>Same prob= lems (recursion) with slapcat -=C2=A0</div><div><br style=3D"color:rgb(0,0,= 0);font-family:Helvetica;font-size:medium;text-align:-webkit-auto"><div sty= le=3D"color:rgb(0,0,0);font-family:Helvetica;font-size:medium;text-align:-w= ebkit-auto"><div>dn: telephoneNumber=3D1000,telephoneNumber=3D1000,telephon= eNumber=3D2020,ou=3Dphones</div><div>=C2=A0,ou=3Daccounts,dc=3D...</div><di= v>telephoneNumber: 1000</div><div>telephoneNumberAccessCode: 8864</div><div= >objectClass: telephoneNumberAccount</div><div>userPassword:: ...</div><div= >structuralObjectClass: telephoneNumberAccount</div><div>entryUUID: bc8a401= 4-0355-1034-8cad-1351d00c5bbd</div><div>creatorsName: cn=3Dldroot,dc=3D...<= /div><div>createTimestamp: 20141118100223Z</div><div>entryCSN: 201411181002= 23.423286Z#000000#000#000000</div><div>modifiersName: cn=3Dldroot,dc=3D...<= /div><div>modifyTimestamp: 20141118100223Z</div><div><br></div><div>dn: tel= ephoneNumber=3D1000,telephoneNumber=3D1000,telephoneNumber=3D1000,telephone= </div><div>=C2=A0Number=3D2020,ou=3Dphones,ou=3Daccounts,dc=3D...</div><div= >telephoneNumber: 1000</div><div>telephoneNumberAccessCode: 8864</div><div>= objectClass: telephoneNumberAccount</div><div>userPassword:: ...</div><div>= structuralObjectClass: telephoneNumberAccount</div><div>entryUUID: bc8d2860= -0355-1034-8cae-1351d00c5bbd</div><div>creatorsName: cn=3Dldroot,dc=3D...</= div><div>createTimestamp: 20141118100223Z</div><div>entryCSN: 2014111810022= 3.442503Z#000000#000#000000</div><div>modifiersName: cn=3Dldroot,dc=3D...</= div><div>modifyTimestamp: 20141118100223Z</div><div><br></div><div>dn: tele= phoneNumber=3D1000,telephoneNumber=3D1000,telephoneNumber=3D1000,telephone<= /div><div>=C2=A0Number=3D1000,telephoneNumber=3D2020,ou=3Dphones,ou=3Daccou= nts,dc=3D...</div><div>telephoneNumber: 1000</div><div>telephoneNumberAcces= sCode: 8864</div><div>objectClass: telephoneNumberAccount</div><div>userPas= sword:: ...</div><div>structuralObjectClass: telephoneNumberAccount</div><d= iv>entryUUID: bc914c06-0355-1034-8caf-1351d00c5bbd</div><div>creatorsName: = cn=3Dldroot,dc=3D...</div><div>createTimestamp: 20141118100223Z</div><div>e= ntryCSN: 20141118100223.469632Z#000000#000#000000</div><div>modifiersName: = cn=3Dldroot,dc=3D...</div><div>modifyTimestamp: 20141118100223Z</div><div><= br></div><div>dn: telephoneNumber=3D1000,telephoneNumber=3D1000,telephoneNu= mber=3D1000,telephone</div><div>=C2=A0Number=3D1000,telephoneNumber=3D1000,= telephoneNumber=3D2020,ou=3Dphones,ou=3Daccounts</div><div>=C2=A0,dc=3D...<= /div><div>telephoneNumber: 1000</div><div>telephoneNumberAccessCode: 8864</= div><div>objectClass: telephoneNumberAccount</div><div>userPassword:: ...</= div><div>structuralObjectClass: telephoneNumberAccount</div><div>entryUUID:= bc9429a8-0355-1034-8cb0-1351d00c5bbd</div><div>creatorsName: cn=3Dldroot,d= c=3D...</div><div>createTimestamp: 20141118100223Z</div><div><br></div></di= v></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On = 18 November 2014 17:47, <span dir=3D"ltr"><<a href=3D"mailto:openldap-i= ts(a)openldap.org" target=3D"_blank">openldap-its(a)openldap.org</a>></span>= wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bor= der-left:1px #ccc solid;padding-left:1ex"><br> *** THIS IS AN AUTOMATICALLY GENERATED REPLY ***<br> <br> Thanks for your report to the OpenLDAP Issue Tracking System.=C2=A0 Your<br= > report has been assigned the tracking number ITS#7985.<br> <br> One of our support engineers will look at your report in due course.<br> Note that this may take some time because our support engineers<br> are volunteers.=C2=A0 They only work on OpenLDAP when they have spare<br> time.<br> <br> If you need to provide additional information in regards to your<br> issue report, you may do so by replying to this message.=C2=A0 Note that<br= > any mail sent to <a href=3D"mailto:openldap-its@openldap.org">openldap-its@= openldap.org</a> with (ITS#7985)<br> in the subject will automatically be attached to the issue report.<br> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 mailto:<a href=3D"mailto:openldap-its@openldap.= org">openldap-its(a)openldap.org</a>?subject=3D(ITS#7985)<br> <br> You may follow the progress of this report by loading the following<br> URL in a web browser:<br> =C2=A0 =C2=A0 <a href=3D"http://www.OpenLDAP.org/its/index.cgi?findid=3D798= 5" target=3D"_blank">http://www.OpenLDAP.org/its/index.cgi?findid=3D7985</a= ><br> <br> Please remember to retain your issue tracking number (ITS#7985)<br> on any further messages you send to us regarding this report.=C2=A0 If<br> you don't then you'll just waste our time and yours because we<br> won't be able to properly track the report.<br> <br> Please note that the Issue Tracking System is not intended to<br> be used to seek help in the proper use of OpenLDAP Software.<br> Such requests will be closed.<br> <br> OpenLDAP Software is user supported.<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"http://www.OpenLDAP.org/support/" ta= rget=3D"_blank">http://www.OpenLDAP.org/support/</a><br> <br> --------------<br> Copyright 1998-2007 The OpenLDAP Foundation, All Rights Reserved.<br> <br> </blockquote></div><br></div> --f46d04428fb47234e2050826ed02--

1 0

Re: (ITS#7985) Recursive values
by michael＠stroeder.com 18 Nov '14

18 Nov '14

belykh.o(a)gmail.com wrote: > Error details: request returns recursive values on some leaves. What does the slapcat output look like? Also which overlays are configured? slapo-rwm? Ciao, Michael.

1 0

Re: (ITS#7985) Recursive values
by andrew.findlay＠skills-1st.co.uk 18 Nov '14

18 Nov '14

On Tue, Nov 18, 2014 at 11:47:10AM +0000, belykh.o(a)gmail.com wrote: > Error details: request returns recursive values on some leaves. Some sensitive > values replaced with '…' Please check: That is certainly an odd one. You are going to have to supply a lot more information before the developers will consider this a usable bug report, but before getting into that I suggest you stop the server and use slapindex to re-build all the indexes. If you have been modifying the configuration after loading data it is possible that the index data is inconsistent. Andrew -- ----------------------------------------------------------------------- | From Andrew Findlay, Skills 1st Ltd | | Consultant in large-scale systems, networks, and directory services | | http://www.skills-1st.co.uk/ +44 1628 782565 | -----------------------------------------------------------------------

1 0

(ITS#7985) Recursive values
by belykh.o＠gmail.com 18 Nov '14

18 Nov '14

Full_Name: Oleg Belykh Version: 2.4.40 OS: FreeBSD URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (37.99.40.12) We are testing latest OpenLDAP 2.4.40 with mdb (FreeBSD 10) with our custom schema and structure. Error details: request returns recursive values on some leaves. Some sensitive values replaced with 'âŠ' Please check: custom schema: # Telephone Attributes attributetype ( 1.3.6.1.4.1.4203.666.6273.2.1 NAME 'telephoneNumberAccessCode' DESC 'Access code for telephoneNumber services' EQUALITY caseIgnoreIA5Match SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 ) attributetype ( 1.3.6.1.4.1.4203.666.6273.2.2 NAME 'faxDeliveryMailbox' EQUALITY caseIgnoreIA5Match SUBSTR caseIgnoreIA5SubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.26{256} ) attributetype ( 1.3.6.1.4.1.4203.666.6273.2.3 NAME 'voiceDeliveryMailbox' DESC 'Voice Mailbox' EQUALITY caseIgnoreIA5Match SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 ) attributetype ( 1.3.6.1.4.1.4203.666.6273.2.4 NAME 'phoneGroupName' DESC 'Telephone Group Name'D0D EQUALITY caseIgnoreIA5Match SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 ) objectclass ( 1.3.6.1.4.1.4203.666.6273.2.100 NAME 'telephoneNumberAccount' DESC 'Telephone account' SUP top STRUCTURAL MUST ( telephoneNumber ) MAY ( userPassword $ telephoneNumberAccessCode $ macAddress $ faxDeliveryMailbox ) ) ldapsearch results: root@sw:/lib/ldap # ldapsearch -H 'ldapi://%2fvar%2frun%2fopenldap%2fldapi/' -W -b 'dc=âŠ' -D 'cn=ldroot,dc=âŠ' Enter LDAP Password: # extended LDIF # # LDAPv3 # base <dc=âŠ> with scope subtree # filter: (objectclass=*) # requesting: ALL # # âŠ dn: dc=âŠ objectClass: dcObject objectClass: organization objectClass: top dc: ... o: ... # accounts, âŠ dn: ou=accounts,dc=âŠ objectClass: top objectClass: organizationalUnit ou: accounts # persons, accounts, âŠ dn: ou=persons,ou=accounts,dc=âŠ objectClass: organizationalUnit ou: persons # kerberos, accounts, âŠ dn: ou=kerberos,ou=accounts,dc=âŠ objectClass: organizaonalalUnit ou: kerberos # mails, accounts, âŠ dn: ou=mails,ou=accounts,dc=âŠ objectClass: organizationalUnit ou: mails # phones, accounts, âŠ dn: ou=phones,ou=accounts,dc=âŠ objectClass: organizationalUnit ou: phones # groups, âŠ dn: ou=groups,dc=âŠ objectClass: top objectClass: organizationalUnit ou: groups # userGroups, groups, âŠ dn: ou=userGroups,ou=groups,dc=âŠ objectClass: organizationalUnit ou: usergroups # phoneGroups, groups, âŠ dn: ou=phoneGroups,ou=groups,dc>2E2Š objectClass: organizationalUnit ou: phonegroups # computers, âŠ dn: ou=computers,dc=âŠ objectClass: top objectClass: organizationalUnit ou: computers # services, âŠ dn: ou=services,dc=âŠ objectClass: top objectClass: organizationalUnit ou: services # manager, accounts, âŠ dn: uid=manager,ou=accounts,dc=âŠ objectClass: account objectClass: simpleSecurityObject uid: manager userPassword:: ... # freeswitch, accounts, âŠ dn: uid=freeswitch,ou=accounts,dc=âŠ objectClass: account objectClass: simpleSecurityObject uid: freeswitch userPassword:: ... # admins, userGroups, groups, âŠ dn: cn=admins,ou=userGroups,ou=groups,dc=âŠ objectClass: posixGroup cn: admins gidNumber: 10000 description: Group account memberUid: ... # users, userGroups, groups, âŠ dn: cn=users,ou=userGroups,ou=groups,dc=âŠ objectClass: posixGroup cn: users gidNumber: 10001 description: Group account # ..., persons, accounts, âŠ dn: uid=...,ou=persons,ou=accounts,dc=2%2Š objectClass: posixAccount objectClass: top objectClass: inetOrgPerson gidNumber: 10000 givenName: ... initials: v sn: .. displayName: ... uid: ... homeDirectory: /dev/null loginShell: /bin/sh cn: ... uidNumber: 20107 userPassword:: ... telephoneNumber: 2020 ( !!!! ) # 1000, phones, accounts, âŠ dn: telephoneNumber=1000,ou=phones,ou=accounts,dc=âŠ telephoneNumber: 1000 telephoneNumberAccessCode: 8864 objectClass: telephoneNumberAccount userPassword:: âŠ. # 2020, phones, accounts, âŠ dn: telephoneNumber=2020,ou=phones,ou=accounts,dc=âŠ telephoneNumber: 2020 telephoneNumberAccessCode: 8864 objectClass: telephoneNumberAccount userPassword:: âŠ. # 1000, 2020, phones, accounts, âŠ dn: telephoneNumber=1000,telephonumumber=2020,ou=phones,ou=accounts,dc=... telephoneNumber: 1000 telephoneNumberAccessCode: 8864 objectClass: telephoneNumberAccount userPassword:: âŠ. # 1000, 1000, 2020, phones, accounts, âŠ dn: telephoneNumber=1000,telephoneNumber=0000,telephoneNumber=2020,ou=phones,o u=accounts,dc=âŠ telephoneNumber: 1000 telephoneNumberAccessCode: 8864 objectClass: telephoneNumberAccount userPassword:: âŠ. # 1000, 1000, 1000, 2020, phones, accounts, âŠ dn: telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,telephoneNu mber=2020,ou=phones,ou=accounts,dc=âŠ telephoneNumber: 1000 telephoneNumberAccessCode: 8864 objectClass: telephoneNumberAccount userPassword:: âŠ. # 1000, 1000, 1000, 1000, 2020, phones, accounts, â080Š dn: telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,telephoneNu mber=1000,telephoneNumber=2020,ou=phones,ou=accounts,dc=âŠ telephoneNumber: 1000 telephoneNumberAccessCode: 8864 objectClass: telephoneNumberAccount userPassword:: âŠ. # 1000, 1000, 1000% 1 1000, 1000, 2020, phones, accounts, âŠ dn: telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,telephoneNu mber=1000,telephoneNumber=1000,telephoneNumber=2020,ou=phones,ou=accounts,dc=... telephoneNumber: 1000 telephoneNumberAccessCode:86864 objectClass: telephoneNumberAccount userPassword:: âŠ. # 1000, 1000, 1000, 1000, 1000, 1000, 2020, phones, accounts, âŠ dn: telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,telephoneNu mber=1000,telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=2020,ou=p hones,ou=accounts,dc=âŠ telephoneNumber: 1000 telephoneNumberAccessCode: 8864 objectClass: telephoneNumberAccount userPassword:: âŠ. # 1000, 1000, 1000, 1000, 1000, 1000, 1000, 2020, phones, accounts, âŠ dn: telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,telephoneNu mber=1000,telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,tele phoneNumber=2020,ou=phones,ou=accounts,dc=âŠ telephoneNumber: 1000 telephoneNumberAccessCode: 64%0 objectClass: telephoneNumberAccount userPassword:: âŠ. # 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 2020, phones, accounts, time. kz dn: telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,telephoneNu mber=1000,telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,tele phoneNumber=1000,telephoneNumber=2020,ou=phones,ou=accounts,dc=âŠ telephoneNumber: 1000 telephoneNumberAccessCode: 8864 objectClass: telephoneNumberAccount userPassword:: âŠ. # 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 2020, phones, accounts, âŠ dn: telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,telephoneNu mber=1000,telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,tele phoneNumber=1000,telephoneNumber=1000,telephoneNumber=2020,ou=phones,ou=accou nts,dc=âŠ telephoneNumber: 1000 telephoneNumberAccessCode: 8864 objectClass: telephoneNumberAccount userPassword:: âŠ. # 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000!01000, 2020, phones, acc ounts, âŠ dn: telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,telephoneNu mber=1000,telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,tele phoneNumber=1000,telephoneNumber=1000,telephoneNumber=1000,telephoneNumber=20 20,ou=phones,ou=accounts,dc=âŠ telephoneNumber: 1000 telephoneNumberAccessCode: 8864 objectClass: telephoneNumberAccount userPassword:: âŠ. if you need screenshots from some ldap management utils, please mail me.

1 0

Re: (ITS#7969) LMDB: Globally shared fields of meta-data are not 'volatile'.
by h.b.furuseth＠usit.uio.no 15 Nov '14

15 Nov '14

__sync_synchronize() needs an #ifdef around it. #ifdef __GNUC__, or something more general? Not sure what to do for the #else. We could mostly avoid using mm_txnid when the lockfile is initialized and use mti_txnid instead (e.g. in pick_meta). And penalize MDB_NOLOCK with some sync call in the #else. -- Hallvard

1 0

(ITS#7982) Add tls cipher suite, protocol to ldapsearch debug logging
by quanah＠openldap.org 14 Nov '14

14 Nov '14

Full_Name: Quanah Gibson-Mount Version: 2.4.40 OS: Linux 2.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (76.14.70.124) Support for logging the TLS protocol and cipher suite being used was added for slapd connections in ITS#7683. However, there's no way from the client side (for those who don't have access to the LDAP server) to see what protocol and cipher suite were negotiated. This can be useful information to have on hand.

1 0

(ITS#7981) Feature request: Crypt scheme in pwdPolicy
by michael＠stroeder.com 14 Nov '14

14 Nov '14

Full_Name: Version: OS: URL: Submission from: (NULL) (79.219.125.95) It would be handy if the password storage/crypt scheme could be specified in a pwdPolicy entry. LDAP Modify Password Ext.Op. should use this information instead of global configuration olcPasswordHash. Rationale: There might be in one database several different account types needed with pwdPolicySubentry pointing to separate pwdPolicy entries. Example: - Normal account with strongly hashed password for direct LDAP simple bind - Clear-text userPassword for WLAN authenticated through RADIUS server Ideally this should be standardized when the ldapext WG is revived. ;-)

1 0

(ITS#7980) LMDB added support for user context in compare function
by rouzier＠gmail.com 13 Nov '14

13 Nov '14

Full_Name: James Rouzier Version: 2.4 OS: GNU/Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (192.222.129.11) https://github.com/rouzier/openldap/commit/33f1dc8be4a8503e78930f130ff9c8e5… I added support for user supplied context for lmdb in the compare functions. This make it much easier to integrate with scripting languages. -- The attached patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by James Rouzier <rouzier(a)gmail.com>. I have not assigned rights and/or interest in this work to any party. I, James Rouzier, hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice. --

1 0

Re: (ITS#7979) mozNSS does not process TLS_PROTOCOL_MIN
by hyc＠symas.com 13 Nov '14

13 Nov '14

Mark Reynolds wrote: > > On 11/12/2014 04:56 PM, Howard Chu wrote: >> mreynolds(a)redhat.com wrote: >>> Full_Name: Mark Reynolds >>> Version: 2.4.40 >>> OS: Fedora 20 >>> URL: ftp://ftp.openldap.org/incoming/mark-reynolds-141112.patch >>> Submission from: (NULL) (174.60.44.17) >>> >>> >>> Currently there is no check for TLS_PROTOCOL_MIN in the mozNSS code. >>> mozNSS >>> defaults to SSLv3/TLS1.0 which is no longer considered secure. If a >>> client only >>> supports TLSv1.1 and up, the openldap ldapsearch will fail to connect >>> over SSL. >>> >>> ldapsearch -H "ldaps://localhost.localdomain:636" -b "" -s base >>> objectclass=* >>> >>> or >>> >>> LDAPTLS_PROTOCOL_MIN=3.2 ldapsearch -H >>> "ldaps://localhost.localdomain:636" -b "" >>> -s base objectclass=* >>> >>> The fix is to grab the supported version range from NSS, adjust the >>> minimum >>> range if TLS_PROTOCOL_MIN is set, and then set the NSS default range >>> with the >>> min and max versions. >> >> Thanks for the patch. I'm concerned because I see you adding MozNSS >> constants (SSL_LIBRARY_VERSION_TLS_1_2) in code that expects libldap >> values (LDAP_OPT_X_TLS_PROTOCOL_TLS1_2). I haven't checked; they may >> well be identical values. But please make sure, and add a comment to >> that effect, so that it's clear that setting lt_protocol_min is >> actually doing what's expected. > Thanks for the feedback Howard. Yes, the SSL versions are the same in > NSS & openldap. I have uploaded a new patch with the requested > comments: mark-reynolds-141113.patch Thanks, committed to master. > On a side note, we are pushing the NSS team to update the NSS API to > provide the SSL version to version string mapping. So we will be able > to remove the hardcoded map(pvers) in openldap once this get addressed. Great. Nice to see they're finally addressing their usability issues. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7979) mozNSS does not process TLS_PROTOCOL_MIN
by mreynolds＠redhat.com 13 Nov '14

13 Nov '14

On 11/12/2014 04:56 PM, Howard Chu wrote: > mreynolds(a)redhat.com wrote: >> Full_Name: Mark Reynolds >> Version: 2.4.40 >> OS: Fedora 20 >> URL: ftp://ftp.openldap.org/incoming/mark-reynolds-141112.patch >> Submission from: (NULL) (174.60.44.17) >> >> >> Currently there is no check for TLS_PROTOCOL_MIN in the mozNSS code. >> mozNSS >> defaults to SSLv3/TLS1.0 which is no longer considered secure. If a >> client only >> supports TLSv1.1 and up, the openldap ldapsearch will fail to connect >> over SSL. >> >> ldapsearch -H "ldaps://localhost.localdomain:636" -b "" -s base >> objectclass=* >> >> or >> >> LDAPTLS_PROTOCOL_MIN=3.2 ldapsearch -H >> "ldaps://localhost.localdomain:636" -b "" >> -s base objectclass=* >> >> The fix is to grab the supported version range from NSS, adjust the >> minimum >> range if TLS_PROTOCOL_MIN is set, and then set the NSS default range >> with the >> min and max versions. > > Thanks for the patch. I'm concerned because I see you adding MozNSS > constants (SSL_LIBRARY_VERSION_TLS_1_2) in code that expects libldap > values (LDAP_OPT_X_TLS_PROTOCOL_TLS1_2). I haven't checked; they may > well be identical values. But please make sure, and add a comment to > that effect, so that it's clear that setting lt_protocol_min is > actually doing what's expected. Thanks for the feedback Howard. Yes, the SSL versions are the same in NSS & openldap. I have uploaded a new patch with the requested comments: mark-reynolds-141113.patch On a side note, we are pushing the NSS team to update the NSS API to provide the SSL version to version string mapping. So we will be able to remove the hardcoded map(pvers) in openldap once this get addressed. Regards, Mark >> >> Also updated the NSS version string map table to support up to TLSv1.3 >

1 0

Re: (ITS#7979) mozNSS does not process TLS_PROTOCOL_MIN
by hyc＠symas.com 12 Nov '14

12 Nov '14

mreynolds(a)redhat.com wrote: > Full_Name: Mark Reynolds > Version: 2.4.40 > OS: Fedora 20 > URL: ftp://ftp.openldap.org/incoming/mark-reynolds-141112.patch > Submission from: (NULL) (174.60.44.17) > > > Currently there is no check for TLS_PROTOCOL_MIN in the mozNSS code. mozNSS > defaults to SSLv3/TLS1.0 which is no longer considered secure. If a client only > supports TLSv1.1 and up, the openldap ldapsearch will fail to connect over SSL. > > ldapsearch -H "ldaps://localhost.localdomain:636" -b "" -s base objectclass=* > > or > > LDAPTLS_PROTOCOL_MIN=3.2 ldapsearch -H "ldaps://localhost.localdomain:636" -b "" > -s base objectclass=* > > The fix is to grab the supported version range from NSS, adjust the minimum > range if TLS_PROTOCOL_MIN is set, and then set the NSS default range with the > min and max versions. Thanks for the patch. I'm concerned because I see you adding MozNSS constants (SSL_LIBRARY_VERSION_TLS_1_2) in code that expects libldap values (LDAP_OPT_X_TLS_PROTOCOL_TLS1_2). I haven't checked; they may well be identical values. But please make sure, and add a comment to that effect, so that it's clear that setting lt_protocol_min is actually doing what's expected. > > Also updated the NSS version string map table to support up to TLSv1.3 -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7977) Supported PBKDF2-SHA256 and PBKDF2-SHA512
by hyc＠symas.com 12 Nov '14

12 Nov '14

Looking over this patch https://www.osstech.co.jp/download/hamano/openldap-pbkdf2_nettle.patch You've added a new contributor: @@ -97,3 +99,5 @@ top-level directory of the distribution or, alternatively, at # ACKNOWLEDGEMENT This work was initially developed by HAMANO Tsukasa <hamano(a)osstech.co.jp> +Contributor: +Luca Bruno(lucab) We cannot accept 3rd party submissions; Luca will have to submit any relevant patches directly to us, along with a corresponding IPR statement as outlined in our Contributors guidelines. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7979) mozNSS does not process TLS_PROTOCOL_MIN
by mreynolds＠redhat.com 12 Nov '14

12 Nov '14

Full_Name: Mark Reynolds Version: 2.4.40 OS: Fedora 20 URL: ftp://ftp.openldap.org/incoming/mark-reynolds-141112.patch Submission from: (NULL) (174.60.44.17) Currently there is no check for TLS_PROTOCOL_MIN in the mozNSS code. mozNSS defaults to SSLv3/TLS1.0 which is no longer considered secure. If a client only supports TLSv1.1 and up, the openldap ldapsearch will fail to connect over SSL. ldapsearch -H "ldaps://localhost.localdomain:636" -b "" -s base objectclass=* or LDAPTLS_PROTOCOL_MIN=3.2 ldapsearch -H "ldaps://localhost.localdomain:636" -b "" -s base objectclass=* The fix is to grab the supported version range from NSS, adjust the minimum range if TLS_PROTOCOL_MIN is set, and then set the NSS default range with the min and max versions. Also updated the NSS version string map table to support up to TLSv1.3

1 0

Re: (ITS#7832) Proposing ppolicy extended module for OpenLDAP
by dcoutadeur＠linagora.com 12 Nov '14

12 Nov '14

Hi, The ppm module is still alive, and is evolving. Last version is 1.2, and source code is now available on Github : https://github.com/davidcoutadeur/ppm There are also plans to package it. If anybody is interrested, I am always glad to hear about comments, ideas, improvements,... Sincerely, David

1 0

Re: (ITS#7977) Supported PBKDF2-SHA256 and PBKDF2-SHA512
by hamano＠osstech.co.jp 06 Nov '14

06 Nov '14

Hi, Please merge the additional patch: https://www.osstech.co.jp/download/hamano/openldap-pbkdf2_nettle.patch This patch include nettle support and fix a issue. https://github.com/hamano/openldap-pbkdf2/pull/4 https://github.com/hamano/openldap-pbkdf2/pull/3 Thank you. At Wed, 05 Nov 2014 11:57:33 +0000, Howard Chu wrote: > > Tsukasa HAMANO wrote: > > Hi, Howard > > > > At Wed, 05 Nov 2014 09:32:43 +0000, > > Howard Chu wrote: > >> > >> Any particular reason you've decreased the iterations from 60000 to 10000? > >> > > > > It was too slow when stretching 60000 on powerless server. > > My tiny VM needed over 1sec if iterate 60000 by PBKDF2-SHA512. > > RFC recommends more than 1000 iterations, it would be safe enough 10000 iterations. > > FYI: http://security.stackexchange.com/questions/3959/recommended-of-iterations-… > > OK. I've committed it without any changes, thanks for the patch. > > > It is desirable to be able to change the operator, but slapasswd does > > not read slapd.conf so I was stuck. > > I'm planning to change slappasswd that accept iteration count in the future. > > Thank you. > > > > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ -- Open Source Solution Technology Corporation HAMANO Tsukasa <hamano(a)osstech.co.jp> fingerprint = 2285 2111 6D34 3816 3C2E A5B9 16BE D101 6069 BE55

1 0

Re: (ITS#7978) OpenLDAP 2.4 fails to build with LibreSSL
by hyc＠symas.com 05 Nov '14

05 Nov '14

Spil Oss wrote: > Hi Howard, > > Thanks for the pointer. --enable-lmpasswd was indeed enabled in the > FreeBSD port. Notifying maintainer of port to switch it off and > provided a patch for the port. > Hope the patch I created for OpenLDAP is usable after all! Deprecated > code in a function that should not be used, would it not be better to > remove it completely? (or is that violating the RFCs?) Very likely we should remove it. Will queue that up for 2.5. 2.4 is end-of-life and feature-frozen so nothing will be added or removed from it. > Kind regards, > > Bernard. > > On Wed, Nov 5, 2014 at 5:48 PM, Howard Chu <hyc(a)symas.com> wrote: >> spil.oss(a)gmail.com wrote: >>> >>> Full_Name: Bernard Spil >>> Version: 2.4.40 >>> OS: FreeBSD 10.1-RC2 >>> URL: ftp://ftp.openldap.org/incoming/ >>> Submission from: (NULL) (185.9.255.20) >>> >>> >>> When compiling OpenLDAP against the LibreSSL OpenSSL fork, compilation >>> fails >>> because deprecated types and functions are used. These types and functions >>> have >>> been marked deprecated by OpenSSL since 2002 and moved from des.h to >>> des_old.h. >>> LibreSSL removed these deprecated types and functions in April 2014 see >>> >>> https://github.com/libressl-portable/openbsd/commit/e0d211052a6946b9f8af112… >>> >>>> From the make output: >> >> >> It appears you're compiling with the old LANMAN hash support. Nobody should >> be using LANMAN any more, it's trivially insecure. I'm inclined to ignore >> this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7978) OpenLDAP 2.4 fails to build with LibreSSL
by spil.oss＠gmail.com 05 Nov '14

05 Nov '14

Hi Howard, Thanks for the pointer. --enable-lmpasswd was indeed enabled in the FreeBSD port. Notifying maintainer of port to switch it off and provided a patch for the port. Hope the patch I created for OpenLDAP is usable after all! Deprecated code in a function that should not be used, would it not be better to remove it completely? (or is that violating the RFCs?) Kind regards, Bernard. On Wed, Nov 5, 2014 at 5:48 PM, Howard Chu <hyc(a)symas.com> wrote: > spil.oss(a)gmail.com wrote: >> >> Full_Name: Bernard Spil >> Version: 2.4.40 >> OS: FreeBSD 10.1-RC2 >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (185.9.255.20) >> >> >> When compiling OpenLDAP against the LibreSSL OpenSSL fork, compilation >> fails >> because deprecated types and functions are used. These types and functions >> have >> been marked deprecated by OpenSSL since 2002 and moved from des.h to >> des_old.h. >> LibreSSL removed these deprecated types and functions in April 2014 see >> >> https://github.com/libressl-portable/openbsd/commit/e0d211052a6946b9f8af112… >> >>> From the make output: > > > It appears you're compiling with the old LANMAN hash support. Nobody should > be using LANMAN any more, it's trivially insecure. I'm inclined to ignore > this ITS. >> >> >> --- passwd.o --- >> passwd.c:41:9: error: unknown type name 'des_cblock'; did you mean >> 'DES_cblock'? >> typedef des_cblock des_key; >> ^~~~~~~~~~ >> DES_cblock >> /usr/local/include/openssl/des.h:73:23: note: 'DES_cblock' declared here >> typedef unsigned char DES_cblock[8]; >> ^ >> passwd.c:42:9: erro3A3A unknown type name 'des_cblock'; did you mean >> 'DES_cblock'? >> typedef des_cblock des_data_block; >> ^~~~~~~~~~ >> DES_cblock >> /usr/local/include/openssl/des.h:73:23: note: 'DES_cblock' declared here >> typedef unsigned char DES_cblock[8]; >> ^ >> passwd.c:43:9: error: unknown type name 'des_key_schedule'; did you mean >> 'DES_key_schedule'? >> typedef des_key_schedule des_context; >> ^~~~~~~~~~~~~~~~ >> DES_key_schedule >> /usr/local/include/openssl/des.h:87:7: note: 'DESeyey_schedule' declared >> here >> } DES_key_schedule; >> ^ >> passwd.c:670:5: warning: implicit declaration of function >> 'des_set_odd_parity' >> is invalid in C99 [-Wimplicit-function-declaration] >> des_set_odd_parity( key ); >> ^ >> passwd.c:867:2: warningA imimplicit declaration of function >> 'des_set_key_unchecked' is invalid in C99 >> [-Wimplicit-function-declaration] >> des_set_key_unchecked( &key, schedule ); >> ^ >> passwd.c:868:2: warning: implicit declaration of function >> 'des_ecb_encrypt' is >> invalid in C99 [-Wimplicit-function-declaration] >> des_ecb_encrypt( &StdText, &PasswordHash1, schedule , DES_ENCRYPT >> ); >> ^ >> 3 warnings and 3 errors generated. >> *** [passwd.o] Error code 1 >> >> make[4]: stopped in >> /usr/ports/net/openldap24-server/work/openldap-2.4.40/libraries/liblutil >> 1 error >> >> make[4]: stopped in >> /usr/ports/net/openldap24-server/work/openldap-2.4.40/libraries/liblutil >> *** [all-common] Error code 2 >> >> make[3]: stopped in >> /usr/ports/net/openldap24-server/work/openldap-2.4.40/librars%s >> 1 error >> >> make[3]: stopped in >> /usr/ports/net/openldap24-server/work/openldap-2.4.40/libraries >> *** [all-common] Error code 2 >> >> make[2]: stopped in /usr/ports/net/openldap24-server/work/openldap-2.4.40 >> 1 error >> >> make[2]: stopped in /u%2/ports/net/openldap24-server/work/openldap-2.4.40 >> ===> Compilation failed unexpectedly. >> Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure >> to >> the maintainer. >> *** Error code 1 >> >> Stop. >> >> >> > > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7978) OpenLDAP 2.4 fails to build with LibreSSL
by bernard＠bachfreund.nl 05 Nov '14

05 Nov '14

--=_6538b474f9e6aa431d332f9d976b1b83 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed Patch to upgrade to the current DES_* types and functions, at least compiles with this. On 2014-11-05 17:48, Howard Chu wrote: > spil.oss(a)gmail.com wrote: >> Full_Name: Bernard Spil >> Version: 2.4.40 >> OS: FreeBSD 10.1-RC2 >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (185.9.255.20) >> >> >> When compiling OpenLDAP against the LibreSSL OpenSSL fork, compilation >> fails >> because deprecated types and functions are used. These types and >> functions have >> been marked deprecated by OpenSSL since 2002 and moved from des.h to >> des_old.h. >> LibreSSL removed these deprecated types and functions in April 2014 >> see >> https://github.com/libressl-portable/openbsd/commit/e0d211052a6946b9f8af112… >> >>> From the make output: > > It appears you're compiling with the old LANMAN hash support. Nobody > should be using LANMAN any more, it's trivially insecure. I'm inclined > to ignore this ITS. >> >> --- passwd.o --- >> passwd.c:41:9: error: unknown type name 'des_cblock'; did you mean >> 'DES_cblock'? >> typedef des_cblock des_key; >> ^~~~~~~~~~ >> DES_cblock >> /usr/local/include/openssl/des.h:73:23: note: 'DES_cblock' declared >> here >> typedef unsigned char DES_cblock[8]; >> ^ >> passwd.c:42:9: erro3A3A unknown type name 'des_cblock'; did you mean >> 'DES_cblock'? >> typedef des_cblock des_data_block; >> ^~~~~~~~~~ >> DES_cblock >> /usr/local/include/openssl/des.h:73:23: note: 'DES_cblock' declared >> here >> typedef unsigned char DES_cblock[8]; >> ^ >> passwd.c:43:9: error: unknown type name 'des_key_schedule'; did you >> mean >> 'DES_key_schedule'? >> typedef des_key_schedule des_context; >> ^~~~~~~~~~~~~~~~ >> DES_key_schedule >> /usr/local/include/openssl/des.h:87:7: note: 'DESeyey_schedule' >> declared here >> } DES_key_schedule; >> ^ >> passwd.c:670:5: warning: implicit declaration of function >> 'des_set_odd_parity' >> is invalid in C99 [-Wimplicit-function-declaration] >> des_set_odd_parity( key ); >> ^ >> passwd.c:867:2: warningA imimplicit declaration of function >> 'des_set_key_unchecked' is invalid in C99 >> [-Wimplicit-function-declaration] >> des_set_key_unchecked( &key, schedule ); >> ^ >> passwd.c:868:2: warning: implicit declaration of function >> 'des_ecb_encrypt' is >> invalid in C99 [-Wimplicit-function-declaration] >> des_ecb_encrypt( &StdText, &PasswordHash1, schedule , >> DES_ENCRYPT ); >> ^ >> 3 warnings and 3 errors generated. >> *** [passwd.o] Error code 1 >> >> make[4]: stopped in >> /usr/ports/net/openldap24-server/work/openldap-2.4.40/libraries/liblutil >> 1 error >> >> make[4]: stopped in >> /usr/ports/net/openldap24-server/work/openldap-2.4.40/libraries/liblutil >> *** [all-common] Error code 2 >> >> make[3]: stopped in >> /usr/ports/net/openldap24-server/work/openldap-2.4.40/librars%s >> 1 error >> >> make[3]: stopped in >> /usr/ports/net/openldap24-server/work/openldap-2.4.40/libraries >> *** [all-common] Error code 2 >> >> make[2]: stopped in >> /usr/ports/net/openldap24-server/work/openldap-2.4.40 >> 1 error >> >> make[2]: stopped in >> /u%2/ports/net/openldap24-server/work/openldap-2.4.40 >> ===> Compilation failed unexpectedly. >> Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the >> failure to >> the maintainer. >> *** Error code 1 >> >> Stop. >> >> >> --=_6538b474f9e6aa431d332f9d976b1b83 Content-Transfer-Encoding: base64 Content-Type: text/x-diff; name=patch-libraries_liblutil_passwd.c Content-Disposition: attachment; filename=patch-libraries_liblutil_passwd.c; size=5977 LS0tIGxpYnJhcmllcy9saWJsdXRpbC9wYXNzd2QuYy5vcmlnCTIwMTQtMDktMTkgMDM6NDg6NDku MDAwMDAwMDAwICswMjAwCisrKyBsaWJyYXJpZXMvbGlibHV0aWwvcGFzc3dkLmMJMjAxNC0xMS0w NSAxOTo1NzoxMC44MDc1NTUwMjUgKzAxMDAKQEAgLTM4LDExICszOCwxMSBAQAogIwlpbmNsdWRl IDxvcGVuc3NsL2Rlcy5oPgogCiAKLXR5cGVkZWYgZGVzX2NibG9jayBkZXNfa2V5OwotdHlwZWRl ZiBkZXNfY2Jsb2NrIGRlc19kYXRhX2Jsb2NrOwotdHlwZWRlZiBkZXNfa2V5X3NjaGVkdWxlIGRl c19jb250ZXh0OwotI2RlZmluZSBkZXNfZmFpbGVkKGVuY3J5cHRlZCkgMAotI2RlZmluZSBkZXNf ZmluaXNoKGtleSwgc2NoZWR1bGUpIAordHlwZWRlZiBERVNfY2Jsb2NrIERFU19rZXk7Cit0eXBl ZGVmIERFU19jYmxvY2sgREVTX2RhdGFfYmxvY2s7Cit0eXBlZGVmIERFU19rZXlfc2NoZWR1bGUg REVTX2NvbnRleHQ7CisjZGVmaW5lIERFU19mYWlsZWQoZW5jcnlwdGVkKSAwCisjZGVmaW5lIERF U19maW5pc2goa2V5LCBzY2hlZHVsZSkgCiAKICNlbGlmIGRlZmluZWQoSEFWRV9NT1pOU1MpCiAv KgpAQCAtNTMsOSArNTMsOSBAQAogKi8KICNkZWZpbmUgUFJPVFlQRVNfSCAxCiAjCWluY2x1ZGUg PG5zcy9wazExcHViLmg+Ci10eXBlZGVmIFBLMTFTeW1LZXkgKmRlc19rZXk7Ci10eXBlZGVmIHVu c2lnbmVkIGNoYXIgZGVzX2RhdGFfYmxvY2tbOF07Ci10eXBlZGVmIFBLMTFDb250ZXh0ICpkZXNf Y29udGV4dFsxXTsKK3R5cGVkZWYgUEsxMVN5bUtleSAqREVTX2tleTsKK3R5cGVkZWYgdW5zaWdu ZWQgY2hhciBERVNfZGF0YV9ibG9ja1s4XTsKK3R5cGVkZWYgUEsxMUNvbnRleHQgKkRFU19jb250 ZXh0WzFdOwogI2RlZmluZSBERVNfRU5DUllQVCBDS0FfRU5DUllQVAogCiAjZW5kaWYKQEAgLTY2 NCwxMCArNjY0LDEwIEBACiAgKiBhYnN0cmFjdCBhd2F5IHNldHRpbmcgdGhlIHBhcml0eS4KICAq Lwogc3RhdGljIHZvaWQKLWRlc19zZXRfa2V5X2FuZF9wYXJpdHkoIGRlc19rZXkgKmtleSwgdW5z aWduZWQgY2hhciAqa2V5RGF0YSkKK0RFU19zZXRfa2V5X2FuZF9wYXJpdHkoIERFU19rZXkgKmtl eSwgdW5zaWduZWQgY2hhciAqa2V5RGF0YSkKIHsKICAgICBtZW1jcHkoa2V5LCBrZXlEYXRhLCA4 KTsKLSAgICBkZXNfc2V0X29kZF9wYXJpdHkoIGtleSApOworICAgIERFU19zZXRfb2RkX3Bhcml0 eSgga2V5ICk7CiB9CiAKIApAQCAtNjc3LDcgKzY3Nyw3IEBACiAgKiBpbXBsZW1lbnQgTW96TlNT IHdyYXBwZXJzIGZvciB0aGUgb3BlblNTTCBjYWxscyAKICAqLwogc3RhdGljIHZvaWQKLWRlc19z ZXRfa2V5X2FuZF9wYXJpdHkoIGRlc19rZXkgKmtleSwgdW5zaWduZWQgY2hhciAqa2V5RGF0YSkK K0RFU19zZXRfa2V5X2FuZF9wYXJpdHkoIERFU19rZXkgKmtleSwgdW5zaWduZWQgY2hhciAqa2V5 RGF0YSkKIHsKICAgICBTRUNJdGVtIGtleURhdGFJdGVtOwogICAgIFBLMTFTbG90SW5mbyAqc2xv dDsKQEAgLTY5OSw3ICs2OTksNyBAQAogfQogCiBzdGF0aWMgdm9pZAotZGVzX3NldF9rZXlfdW5j aGVja2VkKCBkZXNfa2V5ICprZXksIGRlc19jb250ZXh0IGN0eHQgKQorREVTX3NldF9rZXlfdW5j aGVja2VkKCBERVNfa2V5ICprZXksIERFU19jb250ZXh0IGN0eHQgKQogewogICAgIGN0eHRbMF0g PSBOVUxMOwogCkBAIC03MTIsMzcgKzcxMiwzNyBAQAogfQogCiBzdGF0aWMgdm9pZAotZGVzX2Vj Yl9lbmNyeXB0KCBkZXNfZGF0YV9ibG9jayAqcGxhaW4sIGRlc19kYXRhX2Jsb2NrICplbmNyeXB0 ZWQsIAotCQkJZGVzX2NvbnRleHQgY3R4dCwgaW50IG9wKQorREVTX2VjYl9lbmNyeXB0KCBERVNf ZGF0YV9ibG9jayAqcGxhaW4sIERFU19kYXRhX2Jsb2NrICplbmNyeXB0ZWQsIAorCQkJREVTX2Nv bnRleHQgY3R4dCwgaW50IG9wKQogewogICAgIFNFQ1N0YXR1cyBydjsKICAgICBpbnQgc2l6ZTsK IAogICAgIGlmIChjdHh0WzBdID09IE5VTEwpIHsKIAkvKiBuZWVkIHRvIGZhaWwgaGVyZS4uLiAg Ki8KLQltZW1zZXQoZW5jcnlwdGVkLCAwLCBzaXplb2YoZGVzX2RhdGFfYmxvY2spKTsKKwltZW1z ZXQoZW5jcnlwdGVkLCAwLCBzaXplb2YoREVTX2RhdGFfYmxvY2spKTsKIAlyZXR1cm47CiAgICAg fQogICAgIHJ2ID0gUEsxMV9DaXBoZXJPcChjdHh0WzBdLCAodW5zaWduZWQgY2hhciAqKSZlbmNy eXB0ZWRbMF0sIAotCQkJJnNpemUsIHNpemVvZihkZXNfZGF0YV9ibG9jayksCi0JCQkodW5zaWdu ZWQgY2hhciAqKSZwbGFpblswXSwgc2l6ZW9mKGRlc19kYXRhX2Jsb2NrKSk7CisJCQkmc2l6ZSwg c2l6ZW9mKERFU19kYXRhX2Jsb2NrKSwKKwkJCSh1bnNpZ25lZCBjaGFyICopJnBsYWluWzBdLCBz aXplb2YoREVTX2RhdGFfYmxvY2spKTsKICAgICBpZiAocnYgIT0gU0VDU3VjY2VzcykgewogCS8q IHNpZ25hbCBmYWlsdXJlICovCi0JbWVtc2V0KGVuY3J5cHRlZCwgMCwgc2l6ZW9mKGRlc19kYXRh X2Jsb2NrKSk7CisJbWVtc2V0KGVuY3J5cHRlZCwgMCwgc2l6ZW9mKERFU19kYXRhX2Jsb2NrKSk7 CiAJcmV0dXJuOwogICAgIH0KICAgICByZXR1cm47CiB9CiAKIHN0YXRpYyBpbnQKLWRlc19mYWls ZWQoZGVzX2RhdGFfYmxvY2sgKmVuY3J5cHRlZCkKK0RFU19mYWlsZWQoREVTX2RhdGFfYmxvY2sg KmVuY3J5cHRlZCkKIHsKLSAgIHN0YXRpYyBjb25zdCBkZXNfZGF0YV9ibG9jayB6ZXJvID0geyAw IH07CisgICBzdGF0aWMgY29uc3QgREVTX2RhdGFfYmxvY2sgemVybyA9IHsgMCB9OwogICAgcmV0 dXJuIG1lbWNtcChlbmNyeXB0ZWQsIHplcm8sIHNpemVvZih6ZXJvKSkgPT0gMDsKIH0KIAogc3Rh dGljIHZvaWQKLWRlc19maW5pc2goZGVzX2tleSAqa2V5LCBkZXNfY29udGV4dCBjdHh0KQorREVT X2ZpbmlzaChERVNfa2V5ICprZXksIERFU19jb250ZXh0IGN0eHQpCiB7CiAgICAgIGlmICgqa2V5 KSB7CiAJUEsxMV9GcmVlU3ltS2V5KCprZXkpOwpAQCAtODE3LDcgKzgxNyw3IEBACiAKIHN0YXRp YyB2b2lkIGxtUGFzc3dkX3RvX2tleSgKIAljb25zdCBjaGFyICpsbVBhc3N3ZCwKLQlkZXNfa2V5 ICprZXkpCisJREVTX2tleSAqa2V5KQogewogCWNvbnN0IHVuc2lnbmVkIGNoYXIgKmxwdyA9IChj b25zdCB1bnNpZ25lZCBjaGFyICopIGxtUGFzc3dkOwogCXVuc2lnbmVkIGNoYXIga1s4XTsKQEAg LTgzMiw3ICs4MzIsNyBAQAogCWtbNl0gPSAoKGxwd1s1XSAmIDB4M0YpIDw8IDIpIHwgKGxwd1s2 XSA+PiA2KTsKIAlrWzddID0gKChscHdbNl0gJiAweDdGKSA8PCAxKTsKIAkJCi0JZGVzX3NldF9r ZXlfYW5kX3Bhcml0eSgga2V5LCBrICk7CisJREVTX3NldF9rZXlfYW5kX3Bhcml0eSgga2V5LCBr ICk7CiB9CQogCiBzdGF0aWMgaW50IGNoa19sYW5tYW4oCkBAIC04NDMsMTAgKzg0MywxMCBAQAog ewogCWJlcl9sZW5fdCBpOwogCWNoYXIgVWNhc2VQYXNzd29yZFsxNV07Ci0JZGVzX2tleSBrZXk7 Ci0JZGVzX2NvbnRleHQgc2NoZWR1bGU7Ci0JZGVzX2RhdGFfYmxvY2sgU3RkVGV4dCA9ICJLR1Mh QCMkJSI7Ci0JZGVzX2RhdGFfYmxvY2sgUGFzc3dvcmRIYXNoMSwgUGFzc3dvcmRIYXNoMjsKKwlE RVNfa2V5IGtleTsKKwlERVNfY29udGV4dCBzY2hlZHVsZTsKKwlERVNfZGF0YV9ibG9jayBTdGRU ZXh0ID0gIktHUyFAIyQlIjsKKwlERVNfZGF0YV9ibG9jayBQYXNzd29yZEhhc2gxLCBQYXNzd29y ZEhhc2gyOwogCWNoYXIgUGFzc3dvcmRIYXNoWzMzXSwgc3RvcmVkUGFzc3dvcmRIYXNoWzMzXTsK IAkKIAlmb3IoIGk9MDsgaTxjcmVkLT5idl9sZW47IGkrKykgewpAQCAtODY0LDIxICs4NjQsMjEg QEAKIAlsZGFwX3B2dF9zdHIydXBwZXIoIFVjYXNlUGFzc3dvcmQgKTsKIAkKIAlsbVBhc3N3ZF90 b19rZXkoIFVjYXNlUGFzc3dvcmQsICZrZXkgKTsKLQlkZXNfc2V0X2tleV91bmNoZWNrZWQoICZr ZXksIHNjaGVkdWxlICk7Ci0JZGVzX2VjYl9lbmNyeXB0KCAmU3RkVGV4dCwgJlBhc3N3b3JkSGFz aDEsIHNjaGVkdWxlICwgREVTX0VOQ1JZUFQgKTsKKwlERVNfc2V0X2tleV91bmNoZWNrZWQoICZr ZXksICZzY2hlZHVsZSApOworCURFU19lY2JfZW5jcnlwdCggJlN0ZFRleHQsICZQYXNzd29yZEhh c2gxLCAmc2NoZWR1bGUgLCBERVNfRU5DUllQVCApOwogCi0JaWYgKGRlc19mYWlsZWQoJlBhc3N3 b3JkSGFzaDEpKSB7CisJaWYgKERFU19mYWlsZWQoJlBhc3N3b3JkSGFzaDEpKSB7CiAJICAgIHJl dHVybiBMVVRJTF9QQVNTV0RfRVJSOwogCX0KIAkKIAlsbVBhc3N3ZF90b19rZXkoICZVY2FzZVBh c3N3b3JkWzddLCAma2V5ICk7Ci0JZGVzX3NldF9rZXlfdW5jaGVja2VkKCAma2V5LCBzY2hlZHVs ZSApOwotCWRlc19lY2JfZW5jcnlwdCggJlN0ZFRleHQsICZQYXNzd29yZEhhc2gyLCBzY2hlZHVs ZSAsIERFU19FTkNSWVBUICk7Ci0JaWYgKGRlc19mYWlsZWQoJlBhc3N3b3JkSGFzaDIpKSB7CisJ REVTX3NldF9rZXlfdW5jaGVja2VkKCAma2V5LCAmc2NoZWR1bGUgKTsKKwlERVNfZWNiX2VuY3J5 cHQoICZTdGRUZXh0LCAmUGFzc3dvcmRIYXNoMiwgJnNjaGVkdWxlICwgREVTX0VOQ1JZUFQgKTsK KwlpZiAoREVTX2ZhaWxlZCgmUGFzc3dvcmRIYXNoMikpIHsKIAkgICAgcmV0dXJuIExVVElMX1BB U1NXRF9FUlI7CiAJfQogCi0JZGVzX2ZpbmlzaCggJmtleSwgc2NoZWR1bGUgKTsKKwlERVNfZmlu aXNoKCAma2V5LCBzY2hlZHVsZSApOwogCQogCXNwcmludGYoIFBhc3N3b3JkSGFzaCwgIiUwMngl MDJ4JTAyeCUwMnglMDJ4JTAyeCUwMnglMDJ4JTAyeCUwMnglMDJ4JTAyeCUwMnglMDJ4JTAyeCUw MngiLCAKIAkJUGFzc3dvcmRIYXNoMVswXSxQYXNzd29yZEhhc2gxWzFdLFBhc3N3b3JkSGFzaDFb Ml0sUGFzc3dvcmRIYXNoMVszXSwKQEAgLTExMzksMTAgKzExMzksMTAgQEAKIAogCWJlcl9sZW5f dCBpOwogCWNoYXIgVWNhc2VQYXNzd29yZFsxNV07Ci0JZGVzX2tleSBrZXk7Ci0JZGVzX2NvbnRl eHQgc2NoZWR1bGU7Ci0JZGVzX2RhdGFfYmxvY2sgU3RkVGV4dCA9ICJLR1MhQCMkJSI7Ci0JZGVz X2RhdGFfYmxvY2sgUGFzc3dvcmRIYXNoMSwgUGFzc3dvcmRIYXNoMjsKKwlERVNfa2V5IGtleTsK KwlERVNfY29udGV4dCBzY2hlZHVsZTsKKwlERVNfZGF0YV9ibG9jayBTdGRUZXh0ID0gIktHUyFA IyQlIjsKKwlERVNfZGF0YV9ibG9jayBQYXNzd29yZEhhc2gxLCBQYXNzd29yZEhhc2gyOwogCWNo YXIgUGFzc3dvcmRIYXNoWzMzXTsKIAkKIAlmb3IoIGk9MDsgaTxwYXNzd2QtPmJ2X2xlbjsgaSsr KSB7CkBAIC0xMTYwLDEyICsxMTYwLDEyIEBACiAJbGRhcF9wdnRfc3RyMnVwcGVyKCBVY2FzZVBh c3N3b3JkICk7CiAJCiAJbG1QYXNzd2RfdG9fa2V5KCBVY2FzZVBhc3N3b3JkLCAma2V5ICk7Ci0J ZGVzX3NldF9rZXlfdW5jaGVja2VkKCAma2V5LCBzY2hlZHVsZSApOwotCWRlc19lY2JfZW5jcnlw dCggJlN0ZFRleHQsICZQYXNzd29yZEhhc2gxLCBzY2hlZHVsZSAsIERFU19FTkNSWVBUICk7CisJ REVTX3NldF9rZXlfdW5jaGVja2VkKCAma2V5LCAmc2NoZWR1bGUgKTsKKwlERVNfZWNiX2VuY3J5 cHQoICZTdGRUZXh0LCAmUGFzc3dvcmRIYXNoMSwgJnNjaGVkdWxlICwgREVTX0VOQ1JZUFQgKTsK IAkKIAlsbVBhc3N3ZF90b19rZXkoICZVY2FzZVBhc3N3b3JkWzddLCAma2V5ICk7Ci0JZGVzX3Nl dF9rZXlfdW5jaGVja2VkKCAma2V5LCBzY2hlZHVsZSApOwotCWRlc19lY2JfZW5jcnlwdCggJlN0 ZFRleHQsICZQYXNzd29yZEhhc2gyLCBzY2hlZHVsZSAsIERFU19FTkNSWVBUICk7CisJREVTX3Nl dF9rZXlfdW5jaGVja2VkKCAma2V5LCAmc2NoZWR1bGUgKTsKKwlERVNfZWNiX2VuY3J5cHQoICZT dGRUZXh0LCAmUGFzc3dvcmRIYXNoMiwgJnNjaGVkdWxlICwgREVTX0VOQ1JZUFQgKTsKIAkKIAlz cHJpbnRmKCBQYXNzd29yZEhhc2gsICIlMDJ4JTAyeCUwMnglMDJ4JTAyeCUwMnglMDJ4JTAyeCUw MnglMDJ4JTAyeCUwMnglMDJ4JTAyeCUwMnglMDJ4IiwgCiAJCVBhc3N3b3JkSGFzaDFbMF0sUGFz c3dvcmRIYXNoMVsxXSxQYXNzd29yZEhhc2gxWzJdLFBhc3N3b3JkSGFzaDFbM10sCg== --=_6538b474f9e6aa431d332f9d976b1b83--

1 0

Re: (ITS#7978) OpenLDAP 2.4 fails to build with LibreSSL
by hyc＠symas.com 05 Nov '14

05 Nov '14

spil.oss(a)gmail.com wrote: > Full_Name: Bernard Spil > Version: 2.4.40 > OS: FreeBSD 10.1-RC2 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (185.9.255.20) > > > When compiling OpenLDAP against the LibreSSL OpenSSL fork, compilation fails > because deprecated types and functions are used. These types and functions have > been marked deprecated by OpenSSL since 2002 and moved from des.h to des_old.h. > LibreSSL removed these deprecated types and functions in April 2014 see > https://github.com/libressl-portable/openbsd/commit/e0d211052a6946b9f8af112… > >>From the make output: It appears you're compiling with the old LANMAN hash support. Nobody should be using LANMAN any more, it's trivially insecure. I'm inclined to ignore this ITS. > > --- passwd.o --- > passwd.c:41:9: error: unknown type name 'des_cblock'; did you mean > 'DES_cblock'? > typedef des_cblock des_key; > ^~~~~~~~~~ > DES_cblock > /usr/local/include/openssl/des.h:73:23: note: 'DES_cblock' declared here > typedef unsigned char DES_cblock[8]; > ^ > passwd.c:42:9: erro3A3A unknown type name 'des_cblock'; did you mean > 'DES_cblock'? > typedef des_cblock des_data_block; > ^~~~~~~~~~ > DES_cblock > /usr/local/include/openssl/des.h:73:23: note: 'DES_cblock' declared here > typedef unsigned char DES_cblock[8]; > ^ > passwd.c:43:9: error: unknown type name 'des_key_schedule'; did you mean > 'DES_key_schedule'? > typedef des_key_schedule des_context; > ^~~~~~~~~~~~~~~~ > DES_key_schedule > /usr/local/include/openssl/des.h:87:7: note: 'DESeyey_schedule' declared here > } DES_key_schedule; > ^ > passwd.c:670:5: warning: implicit declaration of function 'des_set_odd_parity' > is invalid in C99 [-Wimplicit-function-declaration] > des_set_odd_parity( key ); > ^ > passwd.c:867:2: warningA imimplicit declaration of function > 'des_set_key_unchecked' is invalid in C99 [-Wimplicit-function-declaration] > des_set_key_unchecked( &key, schedule ); > ^ > passwd.c:868:2: warning: implicit declaration of function 'des_ecb_encrypt' is > invalid in C99 [-Wimplicit-function-declaration] > des_ecb_encrypt( &StdText, &PasswordHash1, schedule , DES_ENCRYPT ); > ^ > 3 warnings and 3 errors generated. > *** [passwd.o] Error code 1 > > make[4]: stopped in /usr/ports/net/openldap24-server/work/openldap-2.4.40/libraries/liblutil > 1 error > > make[4]: stopped in /usr/ports/net/openldap24-server/work/openldap-2.4.40/libraries/liblutil > *** [all-common] Error code 2 > > make[3]: stopped in /usr/ports/net/openldap24-server/work/openldap-2.4.40/librars%s > 1 error > > make[3]: stopped in /usr/ports/net/openldap24-server/work/openldap-2.4.40/libraries > *** [all-common] Error code 2 > > make[2]: stopped in /usr/ports/net/openldap24-server/work/openldap-2.4.40 > 1 error > > make[2]: stopped in /u%2/ports/net/openldap24-server/work/openldap-2.4.40 > ===> Compilation failed unexpectedly. > Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to > the maintainer. > *** Error code 1 > > Stop. > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7978) OpenLDAP 2.4 fails to build with LibreSSL
by spil.oss＠gmail.com 05 Nov '14

05 Nov '14

Full_Name: Bernard Spil Version: 2.4.40 OS: FreeBSD 10.1-RC2 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (185.9.255.20) When compiling OpenLDAP against the LibreSSL OpenSSL fork, compilation fails because deprecated types and functions are used. These types and functions have been marked deprecated by OpenSSL since 2002 and moved from des.h to des_old.h. LibreSSL removed these deprecated types and functions in April 2014 see https://github.com/libressl-portable/openbsd/commit/e0d211052a6946b9f8af112… >From the make output: --- passwd.o --- passwd.c:41:9: error: unknown type name 'des_cblock'; did you mean 'DES_cblock'? typedef des_cblock des_key; ^~~~~~~~~~ DES_cblock /usr/local/include/openssl/des.h:73:23: note: 'DES_cblock' declared here typedef unsigned char DES_cblock[8]; ^ passwd.c:42:9: erro3A3A unknown type name 'des_cblock'; did you mean 'DES_cblock'? typedef des_cblock des_data_block; ^~~~~~~~~~ DES_cblock /usr/local/include/openssl/des.h:73:23: note: 'DES_cblock' declared here typedef unsigned char DES_cblock[8]; ^ passwd.c:43:9: error: unknown type name 'des_key_schedule'; did you mean 'DES_key_schedule'? typedef des_key_schedule des_context; ^~~~~~~~~~~~~~~~ DES_key_schedule /usr/local/include/openssl/des.h:87:7: note: 'DESeyey_schedule' declared here } DES_key_schedule; ^ passwd.c:670:5: warning: implicit declaration of function 'des_set_odd_parity' is invalid in C99 [-Wimplicit-function-declaration] des_set_odd_parity( key ); ^ passwd.c:867:2: warningA imimplicit declaration of function 'des_set_key_unchecked' is invalid in C99 [-Wimplicit-function-declaration] des_set_key_unchecked( &key, schedule ); ^ passwd.c:868:2: warning: implicit declaration of function 'des_ecb_encrypt' is invalid in C99 [-Wimplicit-function-declaration] des_ecb_encrypt( &StdText, &PasswordHash1, schedule , DES_ENCRYPT ); ^ 3 warnings and 3 errors generated. *** [passwd.o] Error code 1 make[4]: stopped in /usr/ports/net/openldap24-server/work/openldap-2.4.40/libraries/liblutil 1 error make[4]: stopped in /usr/ports/net/openldap24-server/work/openldap-2.4.40/libraries/liblutil *** [all-common] Error code 2 make[3]: stopped in /usr/ports/net/openldap24-server/work/openldap-2.4.40/librars%s 1 error make[3]: stopped in /usr/ports/net/openldap24-server/work/openldap-2.4.40/libraries *** [all-common] Error code 2 make[2]: stopped in /usr/ports/net/openldap24-server/work/openldap-2.4.40 1 error make[2]: stopped in /u%2/ports/net/openldap24-server/work/openldap-2.4.40 ===> Compilation failed unexpectedly. Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to the maintainer. *** Error code 1 Stop.

1 0

Re: (ITS#7977) Supported PBKDF2-SHA256 and PBKDF2-SHA512
by hyc＠symas.com 05 Nov '14

05 Nov '14

Tsukasa HAMANO wrote: > Hi, Howard > > At Wed, 05 Nov 2014 09:32:43 +0000, > Howard Chu wrote: >> >> Any particular reason you've decreased the iterations from 60000 to 10000? >> > > It was too slow when stretching 60000 on powerless server. > My tiny VM needed over 1sec if iterate 60000 by PBKDF2-SHA512. > RFC recommends more than 1000 iterations, it would be safe enough 10000 iterations. > FYI: http://security.stackexchange.com/questions/3959/recommended-of-iterations-… OK. I've committed it without any changes, thanks for the patch. > It is desirable to be able to change the operator, but slapasswd does > not read slapd.conf so I was stuck. > I'm planning to change slappasswd that accept iteration count in the future. > Thank you. > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7977) Supported PBKDF2-SHA256 and PBKDF2-SHA512
by hamano＠osstech.co.jp 05 Nov '14

05 Nov '14

Hi, Howard At Wed, 05 Nov 2014 09:32:43 +0000, Howard Chu wrote: > > Any particular reason you've decreased the iterations from 60000 to 10000? > It was too slow when stretching 60000 on powerless server. My tiny VM needed over 1sec if iterate 60000 by PBKDF2-SHA512. RFC recommends more than 1000 iterations, it would be safe enough 10000 iterations. FYI: http://security.stackexchange.com/questions/3959/recommended-of-iterations-… It is desirable to be able to change the operator, but slapasswd does not read slapd.conf so I was stuck. I'm planning to change slappasswd that accept iteration count in the future. Thank you. -- Open Source Solution Technology Corporation HAMANO Tsukasa <hamano(a)osstech.co.jp> fingerprint = 2285 2111 6D34 3816 3C2E A5B9 16BE D101 6069 BE55

1 0

Re: (ITS#7977) Supported PBKDF2-SHA256 and PBKDF2-SHA512
by hyc＠symas.com 05 Nov '14

05 Nov '14

hamano(a)osstech.co.jp wrote: > Full_Name: HAMANO Tsukasa > Version: git master > OS: GNU/Linux > URL: https://www.osstech.co.jp/download/hamano/openldap-pbkdf2_sha2.patch > Submission from: (NULL) (240b:10:2640:bf0:426c:8fff:fe50:23a8) > > > I've supported new schemes PBKDF2-SHA256 and PBKDF2-SHA512. > Thank you. Any particular reason you've decreased the iterations from 60000 to 10000? > > -- > The attached patch file is derived from OpenLDAP Software. All of the > modifications to OpenLDAP Software represented in the following > patch(es) were developed by HAMANO Tsukasa <hamano(a)osstech.co.jp>. I > have not assigned rights and/or interest in this work to any party. > > Copyright 2014 HAMANO Tsukasa <hamano(a)osstech.co.jp> > Redistribution and use in source and binary forms, with or without > modification, e e permitted only as authorized by the OpenLDAP Public > License. > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7977) Supported PBKDF2-SHA256 and PBKDF2-SHA512
by hamano＠osstech.co.jp 05 Nov '14

05 Nov '14

Full_Name: HAMANO Tsukasa Version: git master OS: GNU/Linux URL: https://www.osstech.co.jp/download/hamano/openldap-pbkdf2_sha2.patch Submission from: (NULL) (240b:10:2640:bf0:426c:8fff:fe50:23a8) I've supported new schemes PBKDF2-SHA256 and PBKDF2-SHA512. Thank you. -- The attached patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by HAMANO Tsukasa <hamano(a)osstech.co.jp>. I have not assigned rights and/or interest in this work to any party. Copyright 2014 HAMANO Tsukasa <hamano(a)osstech.co.jp> Redistribution and use in source and binary forms, with or without modification, e e permitted only as authorized by the OpenLDAP Public License.

1 0

Re: (ITS#7976) Bug in syncrepl.c short-circuits overlay stack cleanup
by hyc＠symas.com 31 Oct '14

31 Oct '14

mwarren(a)symas.com wrote: > Full_Name: Mark Warren > Version: 2.4.32 > OS: RHEL > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (72.35.133.119) > > > > via Howard: > > "The bug is syncrepl.c lines 2036-2038. Returning LDAP_SUCCESS here > short-circuits the overlay stack and *doesn't* call the cleanup handlers." > > Note: The line numbers are skewed as we're working with an older version. On further analysis, it appears the patch is not needed in current code. It seems the problem (an incoming mod whose entryCSN matches the existing entryCSN of the target entry) was already avoided by the patch for ITS#7427, which was released in 2.4.34. Definitely unable to reproduce the issue in 2.4.40. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7976) Bug in syncrepl.c short-circuits overlay stack cleanup
by mwarren＠symas.com 29 Oct '14

29 Oct '14

Full_Name: Mark Warren Version: 2.4.32 OS: RHEL URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (72.35.133.119) via Howard: "The bug is syncrepl.c lines 2036-2038. Returning LDAP_SUCCESS here short-circuits the overlay stack and *doesn't* call the cleanup handlers." Note: The line numbers are skewed as we're working with an older version. Best Regards, Mark

1 0

Re: (ITS#7975) MDB search scope one and filter returns search base
by hyc＠symas.com 29 Oct '14

29 Oct '14

requate(a)univention.de wrote: > Full_Name: Arvid Requate > Version: 2.4.40 > OS: Debian / UCS > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (82.198.197.8) > > > With an mdb backend the command > > ldapsearch -xLLL -b dc=ar40i1,dc=qa -s one objectClass=domain dn > > returns the base DN. With a bdb backend it doesn't. > > The crucial point seems to be the filter in this case, which matches the base > object only, and none of it's children. If I change the filter to > objectclass=top (or leave it away) then only the children are returned, just > like the behaviour of the bdb backend. Fixed now in git master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7975) MDB search scope one and filter returns search base
by requate＠univention.de 29 Oct '14

29 Oct '14

Full_Name: Arvid Requate Version: 2.4.40 OS: Debian / UCS URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (82.198.197.8) With an mdb backend the command ldapsearch -xLLL -b dc=ar40i1,dc=qa -s one objectClass=domain dn returns the base DN. With a bdb backend it doesn't. The crucial point seems to be the filter in this case, which matches the base object only, and none of it's children. If I change the filter to objectclass=top (or leave it away) then only the children are returned, just like the behaviour of the bdb backend.

1 0

Re: (ITS#7974) LDBM's "laggard reader" flaw still present, in continue of ITS#7904
by leo＠yuriev.ru 22 Oct '14

22 Oct '14

I assume ITS#7830 is the same issue.

1 0

Re: (ITS#7830) MDB Size grown a lot during massive updates and concurrent read access
by leo＠yuriev.ru 22 Oct '14

22 Oct '14

Seems to will be fixed by ITS#7974.

1 0

Re: (ITS#7974) LDBM's "laggard reader" flaw still present, in continue of ITS#7904
by leo＠yuriev.ru 22 Oct '14

22 Oct '14

This is a multi-part message in MIME format. --------------060806060106060808080102 Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit The attached files is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Peter-Service LLC, Moscow, Russia. Peter-Service LLC has not assigned rights and/or interest in this work to any party. I, Leonid Yuriev am authorized by Peter-Service LLC, my employer, to release this work under the following terms. Peter-Service LLC hereby places the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice. --------------060806060106060808080102 Content-Type: text/x-patch; name="0001-lmdb-ITS-7974-a-reading-lag-for-dreamcatcher.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0001-lmdb-ITS-7974-a-reading-lag-for-dreamcatcher.patch" >From a30ece9b236b7217481c086fd27b133bd3404317 Mon Sep 17 00:00:00 2001 From: Leo Yuriev <leo(a)yuriev.ru> Date: Tue, 21 Oct 2014 15:34:22 +0400 Subject: [PATCH 1/2] lmdb: ITS#7974 a reading lag for dreamcatcher. --- libraries/liblmdb/lmdb.h | 11 +++++++++++ libraries/liblmdb/mdb.c | 20 ++++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/libraries/liblmdb/lmdb.h b/libraries/liblmdb/lmdb.h index a3ca62e..82eff14 100644 --- a/libraries/liblmdb/lmdb.h +++ b/libraries/liblmdb/lmdb.h @@ -1571,6 +1571,17 @@ void mdb_env_set_oomkiller(MDB_env *env, MDB_oomkiller_func *oomkiller); * @return A #MDB_oomkiller_func function or NULL if disabled. */ MDB_oomkiller_func* mdb_env_get_oomkiller(MDB_env *env); + + /** @brief Returns a reading lag. + * + * Returns an information for estimate how much given read-only + * transaction is lagging relative the to actual head. + * + * @param[in] txn A transaction handle returned by #mdb_txn_begin() + * @param[out] percent Percentage of page allocation in the database. + * @return Number of transactions committed after the given was started for read, or -1 on failure. + */ +int mdb_txn_straggler(MDB_txn *txnm, int *percent); /** @} */ #ifdef __cplusplus diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c index e60d83d..a417c9b 100644 --- a/libraries/liblmdb/mdb.c +++ b/libraries/liblmdb/mdb.c @@ -2823,6 +2823,26 @@ mdb_dbis_update(MDB_txn *txn, int keep) env->me_numdbs = n; } +int +mdb_txn_straggler(MDB_txn *txn, int *percent) +{ + MDB_env *env; + MDB_meta *meta; + txnid_t lag; + + if (! txn || ! txn->mt_u.reader) + return -1; + + env = txn->mt_env; + meta = env->me_metas[ mdb_env_pick_meta(env) ]; + if (percent) { + long cent = env->me_maxpg / 100; + *percent = (meta->mm_last_pg + cent / 2 + 1) / (cent ? cent : 1); + } + lag = meta->mm_txnid - txn->mt_u.reader->mr_txnid; + return (0 > (int) lag) ? ~0u >> 1: lag; +} + /** Common code for #mdb_txn_reset() and #mdb_txn_abort(). * May be called twice for readonly txns: First reset it, then abort. * @param[in] txn the transaction handle to reset -- 2.1.0 --------------060806060106060808080102 Content-Type: text/x-patch; name="0002-slapd-ITS-7974-dreamcatcher-feature.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-slapd-ITS-7974-dreamcatcher-feature.patch" >From 133cec8eadc93fe3083d110e58259ba2a067908c Mon Sep 17 00:00:00 2001 From: Leo Yuriev <leo(a)yuriev.ru> Date: Tue, 21 Oct 2014 15:38:28 +0400 Subject: [PATCH 2/2] slapd: ITS#7974 dreamcatcher feature. --- servers/slapd/back-mdb/config.c | 46 +++++++++++++++++++++++++++++- servers/slapd/back-mdb/search.c | 62 ++++++++++++++++++++++------------------- 2 files changed, 79 insertions(+), 29 deletions(-) diff --git a/servers/slapd/back-mdb/config.c b/servers/slapd/back-mdb/config.c index b54da49..65034b1 100644 --- a/servers/slapd/back-mdb/config.c +++ b/servers/slapd/back-mdb/config.c @@ -39,7 +39,8 @@ enum { MDB_MAXREADERS, MDB_MAXSIZE, MDB_MODE, - MDB_SSTACK + MDB_SSTACK, + MDB_DREAMCATCHER }; static ConfigTable mdbcfg[] = { @@ -74,6 +75,10 @@ static ConfigTable mdbcfg[] = { mdb_cf_gen, "( OLcfgDbAt:12.2 NAME 'olcDbMaxSize' " "DESC 'Maximum size of DB in bytes' " "SYNTAX OMsInteger SINGLE-VALUE )", NULL, NULL }, + { "dreamcatcher", "lag> <percentage", 3, 3, 0, ARG_MAGIC|MDB_DREAMCATCHER, + mdb_cf_gen, "( OLcfgDbAt:12.4 NAME 'olcDbDreamcatcher' " + "DESC 'Dreamcatcher to avoids withhold of reclaiming' " + "SYNTAX OMsDirectoryString SINGLE-VALUE )",NULL, NULL }, { "mode", "mode", 2, 2, 0, ARG_MAGIC|MDB_MODE, mdb_cf_gen, "( OLcfgDbAt:0.3 NAME 'olcDbMode' " "DESC 'Unix permissions of database files' " @@ -319,6 +324,23 @@ mdb_cf_gen( ConfigArgs *c ) } break; + case MDB_DREAMCATCHER: + if ( mdb->mi_renew_lag ) { + char buf[64]; + struct berval bv; + bv.bv_len = snprintf( buf, sizeof(buf), "%ld %ld", + (long) mdb->mi_renew_lag, (long) mdb->mi_renew_percent ); + if ( bv.bv_len > 0 && bv.bv_len < sizeof(buf) ) { + bv.bv_val = buf; + value_add_one( &c->rvalue_vals, &bv ); + } else { + rc = 1; + } + } else { + rc = 1; + } + break; + case MDB_DIRECTORY: if ( mdb->mi_dbenv_home ) { c->value_string = ch_strdup( mdb->mi_dbenv_home ); @@ -389,6 +411,10 @@ mdb_cf_gen( ConfigArgs *c ) } mdb->mi_txn_cp = 0; break; + case MDB_DREAMCATCHER: + mdb->mi_renew_lag = 0; + mdb->mi_renew_percent = 0; + break; case MDB_DIRECTORY: mdb->mi_flags |= MDB_RE_OPEN; ch_free( mdb->mi_dbenv_home ); @@ -574,6 +600,24 @@ mdb_cf_gen( ConfigArgs *c ) } } break; + case MDB_DREAMCATCHER: { + long l; + if ( lutil_atolx( &l, c->argv[1], 0 ) != 0 || l < 1 ) { + fprintf( stderr, "%s: " + "invalid lag \"%s\" in \"dreamcatcher\".\n", + c->log, c->argv[1] ); + return 1; + } + mdb->mi_renew_lag = l; + if ( lutil_atolx( &l, c->argv[2], 0 ) != 0 || l < 0 || l > 100 ) { + fprintf( stderr, "%s: " + "invalid percentage \"%s\" in \"dreamcatcher\".\n", + c->log, c->argv[2] ); + return 1; + } + mdb->mi_renew_percent = l; + } break; + case MDB_DIRECTORY: { FILE *f; char *ptr, *testpath; diff --git a/servers/slapd/back-mdb/search.c b/servers/slapd/back-mdb/search.c index 98b5f25..381cb80 100644 --- a/servers/slapd/back-mdb/search.c +++ b/servers/slapd/back-mdb/search.c @@ -340,22 +340,29 @@ typedef struct ww_ctx { * case return an LDAP_BUSY error - let the client know this search * couldn't succeed, but might succeed on a retry. */ + +static void +mdb_befree( Operation *op, ww_ctx *ww ) +{ + assert(! ww->flag); + if ( ww->mcd ) { + MDB_val key, data; + mdb_cursor_get( ww->mcd, &key, &data, MDB_GET_CURRENT ); + memcpy( &ww->key, key.mv_data, sizeof(ID) ); + ww->data.mv_size = data.mv_size; + ww->data.mv_data = op->o_tmpalloc( data.mv_size, op->o_tmpmemctx ); + memcpy(ww->data.mv_data, data.mv_data, data.mv_size); + } + mdb_txn_reset( ww->txn ); + ww->flag = 1; +} + static void mdb_writewait( Operation *op, slap_callback *sc ) { ww_ctx *ww = sc->sc_private; - if ( !ww->flag ) { - if ( ww->mcd ) { - MDB_val key, data; - mdb_cursor_get( ww->mcd, &key, &data, MDB_GET_CURRENT ); - memcpy( &ww->key, key.mv_data, sizeof(ID) ); - ww->data.mv_size = data.mv_size; - ww->data.mv_data = op->o_tmpalloc( data.mv_size, op->o_tmpmemctx ); - memcpy(ww->data.mv_data, data.mv_data, data.mv_size); - } - mdb_txn_reset( ww->txn ); - ww->flag = 1; - } + if ( !ww->flag ) + mdb_befree( op, ww ); } static int @@ -796,7 +803,6 @@ loop_begin: goto done; } - if ( nsubs < ncand ) { unsigned i; /* Is this entry in the candidate list? */ @@ -1034,14 +1040,6 @@ notfound: ber_bvarray_free( erefs ); rs->sr_ref = NULL; - if ( wwctx.flag ) { - rs->sr_err = mdb_waitfixup( op, &wwctx, mci, mcd ); - if ( rs->sr_err ) { - send_ldap_result( op, rs ); - goto done; - } - } - goto loop_continue; } @@ -1096,13 +1094,6 @@ notfound: } goto done; } - if ( wwctx.flag ) { - rs->sr_err = mdb_waitfixup( op, &wwctx, mci, mcd ); - if ( rs->sr_err ) { - send_ldap_result( op, rs ); - goto done; - } - } } } else { @@ -1113,6 +1104,21 @@ notfound: } loop_continue: + if ( ! wwctx.flag && mdb->mi_renew_lag ) { + int percentage, lag = mdb_txn_straggler( ltid, &percentage ); + if ( lag >= mdb->mi_renew_lag && percentage >= mdb->mi_renew_percent ) { + Debug( LDAP_DEBUG_TRACE, "dreamcather: lag %d, percentage %u%%\n", lag, percentage, 0 ); + mdb_befree( op, &wwctx ); + } + } + if ( wwctx.flag ) { + rs->sr_err = mdb_waitfixup( op, &wwctx, mci, mcd ); + if ( rs->sr_err ) { + send_ldap_result( op, rs ); + goto done; + } + } + if( e != NULL ) { if ( e != base ) mdb_entry_return( op, e ); -- 2.1.0 --------------060806060106060808080102--

1 0

Re: (ITS#7974) LDBM's "laggard reader" flaw still present, in continue of ITS#7904
by leo＠yuriev.ru 22 Oct '14

22 Oct '14

This is a multi-part message in MIME format. --------------000000040604020102000407 Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit The attached files is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Peter-Service LLC, Moscow, Russia. Peter-Service LLC has not assigned rights and/or interest in this work to any party. I, Leonid Yuriev am authorized by Peter-Service LLC, my employer, to release this work under the following terms. Peter-Service LLC hereby places the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice. --------------000000040604020102000407 Content-Type: text/x-patch; name="0001-lmdb-ITS-7974-oomkiller-feature.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0001-lmdb-ITS-7974-oomkiller-feature.patch" >From 85fce95eaa0e71ee43625ccc202c173f7d4acb4a Mon Sep 17 00:00:00 2001 From: Leo Yuriev <leo(a)yuriev.ru> Date: Tue, 21 Oct 2014 19:25:32 +0400 Subject: [PATCH 1/2] lmdb: ITS#7974 oomkiller feature. --- libraries/liblmdb/lmdb.h | 34 +++++++++++++++++ libraries/liblmdb/mdb.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 126 insertions(+), 3 deletions(-) diff --git a/libraries/liblmdb/lmdb.h b/libraries/liblmdb/lmdb.h index bdbb0b9..a3ca62e 100644 --- a/libraries/liblmdb/lmdb.h +++ b/libraries/liblmdb/lmdb.h @@ -1537,6 +1537,40 @@ int mdb_reader_list(MDB_env *env, MDB_msg_func *func, void *ctx); * @return 0 on success, non-zero on failure. */ int mdb_reader_check(MDB_env *env, int *dead); + + /** @brief A callback function for killing a laggard readers, + * called in case of MDB_MAP_FULL error. + * + * @param[in] env An environment handle returned by #mdb_env_create(). + * @param[in] pid pid of the reader process. + * @param[in] thread_id thread_id of the reader thread. + * @param[in] txn Transaction number on which stalled. + * @return -1 on failure (reader is not killed), + * 0 on a race condition (no such reader), + * 1 on success (reader was killed), + * >1 on success (reader was SURE killed). + */ +typedef int (MDB_oomkiller_func)(MDB_env *env, int pid, void* thread_id, size_t txn); + + /** @brief Set the oomkiller callback. + * + * Callback will be called only on out-of-pages case for killing + * a laggard readers to allowing reclaiming of freeDB. + * + * @param[in] env An environment handle returned by #mdb_env_create(). + * @param[in] oomkiller A #MDB_oomkiller_func function or NULL to disable. + */ +void mdb_env_set_oomkiller(MDB_env *env, MDB_oomkiller_func *oomkiller); + + /** @brief Get the current oomkiller callback. + * + * Callback will be called only on out-of-pages case for killing + * a laggard readers to allowing reclaiming of freeDB. + * + * @param[in] env An environment handle returned by #mdb_env_create(). + * @return A #MDB_oomkiller_func function or NULL if disabled. + */ +MDB_oomkiller_func* mdb_env_get_oomkiller(MDB_env *env); /** @} */ #ifdef __cplusplus diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c index 6cc3433..e60d83d 100644 --- a/libraries/liblmdb/mdb.c +++ b/libraries/liblmdb/mdb.c @@ -1145,6 +1145,7 @@ struct MDB_env { #endif void *me_userctx; /**< User-settable context */ MDB_assert_func *me_assert_func; /**< Callback for assertion failures */ + MDB_oomkiller_func *me_oomkiller; /**< Callback for killing laggard readers */ }; /** Nested transaction */ @@ -1900,6 +1901,77 @@ mdb_find_oldest(MDB_txn *txn) return oldest; } +static txnid_t +mdb_laggard_reader(MDB_env *env, int *laggard) +{ + txnid_t tail = 0; + if (laggard) + *laggard = -1; + if (env->me_txns->mti_txnid > 1) { + int i; + MDB_reader *r = env->me_txns->mti_readers; + + tail = env->me_txns->mti_txnid - 1; + for (i = env->me_txns->mti_numreaders; --i >= 0; ) { + if (r[i].mr_pid) { + txnid_t mr = r[i].mr_txnid; + if (tail > mr) { + tail = mr; + if (laggard) + *laggard = i; + } + } + } + } + + return tail; +} + +static int +mdb_oomkill_laggard(MDB_env *env) +{ + int dead, idx; + txnid_t tail = mdb_laggard_reader(env, &idx); + if (idx < 0) + return 0; + + for(;;) { + MDB_reader *r; + MDB_THR_T tid; + pid_t pid; + int rc; + + if (mdb_reader_check(env, &dead)) + break; + + if (dead && tail < mdb_laggard_reader(env, NULL)) + return 1; + + if (!env->me_oomkiller) + break; + + r = &env->me_txns->mti_readers[ idx ]; + pid = r->mr_pid; + tid = r->mr_tid; + if (r->mr_txnid != tail || pid <= 0) + continue; + + rc = env->me_oomkiller(env, pid, (void*) tid, tail); + if (rc < 0) + break; + + if (rc) { + r->mr_txnid = (txnid_t)-1; + if (rc > 1) { + r->mr_tid = 0; + r->mr_pid = 0; + } + } + } + + return tail < mdb_laggard_reader(env, NULL); +} + /** Add a page to the txn's dirty list */ static void mdb_page_dirty(MDB_txn *txn, MDB_page *mp) @@ -1978,6 +2050,7 @@ mdb_page_alloc(MDB_cursor *mc, int num, MDB_page **mp) goto fail; } +oomkill_retry:; for (op = MDB_FIRST;; op = MDB_NEXT) { MDB_val key, data; MDB_node *leaf; @@ -2073,9 +2146,11 @@ mdb_page_alloc(MDB_cursor *mc, int num, MDB_page **mp) i = 0; pgno = txn->mt_next_pgno; if (pgno + num >= env->me_maxpg) { - DPUTS("DB size maxed out"); - rc = MDB_MAP_FULL; - goto fail; + DPUTS("DB size maxed out"); + if (mdb_oomkill_laggard(env)) + goto oomkill_retry; + rc = MDB_MAP_FULL; + goto fail; } search_done: @@ -9403,4 +9478,18 @@ mdb_reader_check(MDB_env *env, int *dead) *dead = count; return MDB_SUCCESS; } + +void +mdb_env_set_oomkiller(MDB_env *env, MDB_oomkiller_func *oomkiller) +{ + if (env) + env->me_oomkiller = oomkiller; +} + +MDB_oomkiller_func* +mdb_env_get_oomkiller(MDB_env *env) +{ + return env ? env->me_oomkiller : NULL; +} + /** @} */ -- 2.1.0 --------------000000040604020102000407 Content-Type: text/x-patch; name="0002-slapd-ITS-7974-oomkiller-feature.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-slapd-ITS-7974-oomkiller-feature.patch" >From caf076698e093077fa44e490797847c3187d485b Mon Sep 17 00:00:00 2001 From: Leo Yuriev <leo(a)yuriev.ru> Date: Tue, 21 Oct 2014 19:49:05 +0400 Subject: [PATCH 2/2] slapd: ITS#7974 oomkiller feature. --- servers/slapd/back-mdb/back-mdb.h | 3 +++ servers/slapd/back-mdb/config.c | 40 ++++++++++++++++++++++++++++++++++---- servers/slapd/back-mdb/init.c | 2 ++ servers/slapd/back-mdb/proto-mdb.h | 2 ++ 4 files changed, 43 insertions(+), 4 deletions(-) diff --git a/servers/slapd/back-mdb/back-mdb.h b/servers/slapd/back-mdb/back-mdb.h index 9d5d4b1..be82da8 100644 --- a/servers/slapd/back-mdb/back-mdb.h +++ b/servers/slapd/back-mdb/back-mdb.h @@ -81,6 +81,9 @@ struct mdb_info { uint32_t mi_txn_cp_kbyte; struct re_s *mi_txn_cp_task; struct re_s *mi_index_task; + uint32_t mi_renew_lag; + uint32_t mi_renew_percent; + int mi_oomkill; mdb_monitor_t mi_monitor; diff --git a/servers/slapd/back-mdb/config.c b/servers/slapd/back-mdb/config.c index 5b402c5..b54da49 100644 --- a/servers/slapd/back-mdb/config.c +++ b/servers/slapd/back-mdb/config.c @@ -106,6 +106,8 @@ static slap_verbmasks mdb_envflags[] = { { BER_BVC("writemap"), MDB_WRITEMAP }, { BER_BVC("mapasync"), MDB_MAPASYNC }, { BER_BVC("nordahead"), MDB_NORDAHEAD }, +#define MDB_OOMKILL (MDB_NOMEMINIT << 4) + { BER_BVC("oomkill"), MDB_OOMKILL }, { BER_BVNULL, 0 } }; @@ -123,6 +125,23 @@ mdb_checkpoint( void *ctx, void *arg ) return NULL; } +/* perform killing a laggard readers */ +int +mdb_oomkiller(MDB_env *env, int pid, void* thread_id, size_t txn) +{ + if ( pid != getpid() ) { + if ( kill( pid, SIGKILL ) == 0 ) { + Debug( LDAP_DEBUG_ANY, "oomkiller: SIGKILL to pid %i\n", pid, 0, 0 ); + sched_yield(); + return 2; + } + if ( errno == ESRCH ) + return 0; + Debug( LDAP_DEBUG_ANY, "oomkiller: SIGKILL to pid %i: %s\n", pid, strerror(errno), 0 ); + } + return -1; +} + /* reindex entries on the fly */ static void * mdb_online_index( void *ctx, void *arg ) @@ -313,12 +332,15 @@ mdb_cf_gen( ConfigArgs *c ) c->value_int = 1; break; - case MDB_ENVFLAGS: - if ( mdb->mi_dbenv_flags ) { - mask_to_verbs( mdb_envflags, mdb->mi_dbenv_flags, &c->rvalue_vals ); - } + case MDB_ENVFLAGS: { + long flags = mdb->mi_dbenv_flags; + if ( mdb->mi_oomkill ) + flags |= MDB_OOMKILL; + if ( flags ) + mask_to_verbs( mdb_envflags, flags, &c->rvalue_vals ); if ( !c->rvalue_vals ) rc = 1; break; + } case MDB_INDEX: mdb_attr_index_unparse( mdb, &c->rvalue_vals ); @@ -380,6 +402,8 @@ mdb_cf_gen( ConfigArgs *c ) break; case MDB_ENVFLAGS: + mdb->mi_oomkill = 0; + mdb_env_set_oomkiller( mdb->mi_dbenv, NULL ); if ( c->valx == -1 ) { int i; for ( i=0; mdb_envflags[i].mask; i++) { @@ -596,6 +620,14 @@ mdb_cf_gen( ConfigArgs *c ) for ( i=1; i<c->argc; i++ ) { j = verb_to_mask( c->argv[i], mdb_envflags ); if ( mdb_envflags[j].mask ) { + + if ( MDB_OOMKILL == mdb_envflags[j].mask ) { + mdb->mi_oomkill = 1; + if ( mdb->mi_flags & MDB_IS_OPEN ) + mdb_env_set_oomkiller( mdb->mi_dbenv, mdb_oomkiller ); + break; + } + if ( mdb->mi_flags & MDB_IS_OPEN ) rc = mdb_env_set_flags( mdb->mi_dbenv, mdb_envflags[j].mask, 1 ); else diff --git a/servers/slapd/back-mdb/init.c b/servers/slapd/back-mdb/init.c index 1c5ab83..c7c09d4 100644 --- a/servers/slapd/back-mdb/init.c +++ b/servers/slapd/back-mdb/init.c @@ -150,6 +150,8 @@ mdb_db_open( BackendDB *be, ConfigReply *cr ) goto fail; } + mdb_env_set_oomkiller( mdb->mi_dbenv, mdb->mi_oomkill ? mdb_oomkiller : NULL); + #ifdef HAVE_EBCDIC strcpy( path, mdb->mi_dbenv_home ); __atoe( path ); diff --git a/servers/slapd/back-mdb/proto-mdb.h b/servers/slapd/back-mdb/proto-mdb.h index b6b8d7c..3ec986e 100644 --- a/servers/slapd/back-mdb/proto-mdb.h +++ b/servers/slapd/back-mdb/proto-mdb.h @@ -20,6 +20,8 @@ LDAP_BEGIN_DECL #define MDB_UCTYPE "MDB" +MDB_oomkiller_func mdb_oomkiller; + /* * attr.c */ -- 2.1.0 --------------000000040604020102000407--

1 0

LDBM's "laggard reader" flaw still present, in continue of ITS#7904
by leo＠yuriev.ru 22 Oct '14

22 Oct '14

Full_Name: Leonid Yuriev Version: 2.4.40 OS: RHEL7 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (31.130.36.33) Currently there is flaw that does not allow using OpenLDAP + LMDB in projects with high rate of updates (add/modify/delete). The root of these problems is that LMDB cannot reclaim freed pages by a presence of a "laggard reader", or in other words if they are still referenced by an active read. It should be noted, that withholding of reclaiming while the high update rate, burns free pages very quickly. Fix of the ITS#7904 significantly improves the situation, but does not solve all the problems completely. Firstly, seemingly innocuous use of something like a "mdb_stat -efff | less" can lead to the MDB_MAP_FULL and paralyze update. Second, ITS#7904 affects the syncrepl only partially. Approximately half of the "long read" operations occur without sending data to the network. Therefore, in many cases get MDB_MAP_FULL easily enough. This leads to a chain of problems and in some cases makes the replication impossible. To solve these problems, I made two simple improvements. 1) OOMKiller feature just a fuse likely Linux kernel oomkiller. In generally, in case of MDB_MAP_FULL will send the SIGKILL to a laggard reader, but not to self. On success will retry to reclaim and continue. Engaged by envflags oomkill. 2) Dreamcatcher feature really, it has caught and forced vanish our nightmares with syncrepl & MDB MAP_FULL ;) Based on ITS#7904 fix. In generally, renew read-txt when the lag from last txn is greater than a configured threshold and the percentage of pages allocated is greater than the configured value. Engaged by dreamcatcher lag percentage. Two patchsets will be attached soon.

1 0

(ITS#7973) LDAP not seeing Berkeley DB
by jfha73＠gmail.com 20 Oct '14

20 Oct '14

Full_Name: Jorge F. Hernandez Version: 2.4 OS: Mac OS X 10.10 URL: Submission from: (NULL) (74.8.80.62) Hey guys, I'm trying to build OpenLDAP from source on a Mac OS X 10.10 using Berkeley DB 4.6, but when it checks for it it says it had version 1.0, I tried building Berkeley DB using --prefix/usr and leaving it as /usr/cacal and LDAP doesn't see it. I was able to do the same on Mac OS X 10.9 so I guess it may be something related to the OS recognition (Yosemite - 10.10) just came out. Thanks.

1 0

Re: (ITS#7972) zgrim
by hyc＠symas.com 19 Oct '14

19 Oct '14

zzgrim(a)gmail.com wrote: > Full_Name: zgrim > Version: 2.4.40 > OS: Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (79.115.186.237) > > > Hello, thank you for lmdb. > I'm using lmdb via a Go wrapper and under some not-yet-clear circumstances, > under high concurrency stress testing, > I sometimes get "double free or corruption" SIGABRT related to > mdb_dbis_update(). > The setup is using the NOTLS and RDONLY flags on a 64bit 3.17.0 Arch Linux > system. > If I protect the free() inside mdb_dbis_update() (in mdb.c) with a mutex (code > inline below), the symptom disappears. > I'm sorry I can't provide a simple test case, but in my code, without the > mutex, launching some tens of test clients simultaneously trigger the > corruption fairly quick, i.e. a few minutes or even seconds. Sounds like you have not read the mdb_dbi_open() doc closely. The code in mdb_dbis_update() only triggers if you've opened new DBs in a transaction. The doc for mdb_dbi_open() specifically says you cannot call it from multiple concurrent transactions. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7958) LMDB: LIFO-reclaiming, write-performance improvement & bugfixes
by leo＠yuriev.ru 19 Oct '14

19 Oct '14

>>> We are using the combination: >>> envflags writemap nosync lifo >>> checkpoint 0 1 >>> >>> If the checkpoint is set in seconds, it gives us the assurance >>> consistent state database on disk. >>> However, without this patch meta-pages can be written by the kernel >>> before the data. >>> >>> In fact, for a full guarantee in case of death slapd process, >>> meta-page should be written explicitly. > > > No, the DB can never go inconsistent due to a process crashing - the pages > in OS cache are always correct. It can only go inconsistent if the OS > crashes and a proper sync has not occurred. Yes, Howard, you are right. But apparently I need to be more precise. Talking about "death" of slapd, I meant all the reasons, including power off. For example, a power-off case: - The main power is turned off and the system switches to the UPS. - Given the notice, OS starts an emergency stop processes. - For some reason (does not have enough time to stop) slapd receives SIGKILL. - OS tries to write mmap-region of the DB-file and begins with the lower address. - let the meta-pages has written completely, but for the rest of the data is not enough battery power. - Now DB is completely destroyed on the disk. To avoid this, the meta-pages should not be included in the rw-mapped region, and should be written explicity after a data pages. >>>>> commit 8ddd63161aeb2689822d1a8d27385d62e4e341ae >>>>> Author: Leo Yuriev <leo(a)yuriev.ru> >>>>> Date: 2014-09-19 22:47:19 +0400 >>>>> >>>>> BUGFIX - lmdb: properly sync meta-pages in mdb_sync_env(). >>>>> >>>>> Meta-pages may be updated during data-syncing in mdb_sync_env(), >>>>> in this case database would be inconsistent. >>>>> >>>>> Check-and-retry if lead txn-id changed during flushing data in >>>>> mdb_sync_env(). > > Fundamentally, you are trying to make an inherently unsafe configuration > "safer", but it's impossible. Assume you have mlock'd the meta pages into > memory, so the OS never flushes them itself any more, and you're running > with NOSYNC. That means, within 3 transactions, the data pages on disk will > be out of sync with the meta pages on disk. If the OS crashes at that point, > the entire DB will be lost. Not a problem. I had explained above - we should write meta-pages explicitly after the data sync. But also we should not perform reclaiming ahead of the last checkpoint. > The only way to make this mode of operation somewhat safe is to defer > reclaiming pages for even longer. E.g., instead of halting at current_txnid > - 3, halt at current_txnid - 22, in which case the data pointed to by the > on-disk meta pages cannot get obsolete until 20 transactions have occurred. > > Note that in combination with your LIFO patch, it's pretty much guaranteed > that the on-disk meta pages will be useless after only 2 un-sync'd > transactions. Yes, Howard, you are right. But I think there is confusion in the discussion because of mixing of LIFO-feature and changes for checkpoints consistency in a NOSYNC and WRITEMAP+NOSYNC modes. For a "NOSYNC + checkpoints" topic I will submit a separate ITS (like a 'volaile' related 7969,7970,7971). My opinion - it is a flaw, and no reason to don't fix it. Continuing the conversation about checkpoints in a LIFO context. I saw the problem, that you specified, and thinking over its solution, but have not yet found "golden ratio". And since we are having a serious problem with syncrepl, then I put off this task with an excuse "LIFO-patch not does worse than it was." In general, we should do not reclaim anything ahead of the txn, that is synced to the disk (let this be named a R-rule). To do so we need a second field like mti_txnid, but which will be update only at the end of mdb_env_write_meta(). Finally we should start search in mdb_find_oldest() from value of this new field instead of the current txn number. This seems to will be work fine. However, I stopped on a reasoning - about the purpose of the checkpoints, about design LMDB as a product, about the expectations of the user and the necessary configuration parameters: - checkpoints are needed ONLY in nosync modes; - if the user does NOT activate the checkpoints, he do not care about consistency; - but if it is turned on, we MUST provide consistency on the checkpoints; - otherwise a checkpoints feature is thoughtless and should be REMOVED. Therefore implementation of checkpoints & reclaiming should be updated to conform to the "R-rule", that noted above. >From this point of view a LIFO-feature also should be refined, but nevertheless can be very useful. - SYNC mode = takes a benefit from storage with write-back cache (assume powered by battery). - ASYNC/NOSYNC without checkpoint = significant reduction of committed/dirty pages and thereby much less write-iops. - ASYNC/NOSYNC with checkpoint = seems to same as a SYNC case. Total of all the above - I think first we need to fix a reclaiming or delete the checkpoints, and then I will complete LIFO. > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ Thank for conversation. Leonid.

1 0

(ITS#7972) zgrim
by zzgrim＠gmail.com 19 Oct '14

19 Oct '14

Full_Name: zgrim Version: 2.4.40 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (79.115.186.237) Hello, thank you for lmdb. I'm using lmdb via a Go wrapper and under some not-yet-clear circumstances, under high concurrency stress testing, I sometimes get "double free or corruption" SIGABRT related to mdb_dbis_update(). The setup is using the NOTLS and RDONLY flags on a 64bit 3.17.0 Arch Linux system. If I protect the free() inside mdb_dbis_update() (in mdb.c) with a mutex (code inline below), the symptom disappears. I'm sorry I can't provide a simple test case, but in my code, without the mutex, launching some tens of test clients simultaneously trigger the corruption fairly quick, i.e. a few minutes or even seconds. I tried older releases of lmdb as well as the latest git tree, 9a8eb95674c7b500cfe5f44d03493ff76c9fc0c1 of mdb.master, the bahaviour is the same. I also tried several approaches in the higher level code, like either txn reset and renew and a pool of txns or simplified the code to bare bones to just abort after each transaction and remake new txns, it does not matter, the corruption happens the same. If the problem should lie in the higher level code, I'd welcome your insights on possible culprits, I'm very new to lmdb. Thanks. against latest git mentioned above i applied something like, @@ -2769,19 +2769,24 @@ MDB_dbi n = txn->mt_numdbs; MDB_env *env = txn->mt_env; unsigned char *tdbflags = txn->mt_dbflags; + int rc = 0, rx = 0; for (i = n; --i >= 2;) { if (tdbflags[i] & DB_NEW) { if (keep) { env->me_dbflags[i] = txn->mt_dbs[i].md_flags | MDB_VALID; } else { - char *ptr = env->me_dbxs[i].md_name.mv_data; - if (ptr) { + if (env->me_dbxs[i].md_name.mv_data != NULL) { + mdb_mutex_t *wmutex = MDB_MUTEX(env, w); + rc = LOCK_MUTEX(rx, env, wmutex); + + free(env->me_dbxs[i].md_name.mv_data); env->me_dbxs[i].md_name.mv_data = NULL; env->me_dbxs[i].md_name.mv_size = 0; env->me_dbflags[i] = 0; env->me_dbiseqs[i]++; - free(ptr); + + if (!rc) UNLOCK_MUTEX(wmutex); } } }

1 0

Re: (ITS#7958) LMDB: LIFO-reclaiming, write-performance improvement & bugfixes
by hyc＠symas.com 18 Oct '14

18 Oct '14

Леонид Юрьев wrote: > 2014-10-03 3:13 GMT+04:00 Howard Chu <hyc(a)symas.com>: > 2014-10-04 0:04 GMT+04:00 Леонид Юрьев <leo(a)yuriev.ru>: >> 2014-10-03 3:13 GMT+04:00 Howard Chu <hyc(a)symas.com>: >>>> commit 841059330fd44769e93eb4b937c3ce42654fad6f >>>> Author: Leo Yuriev <leo(a)yuriev.ru> >>>> Date: 2014-09-20 07:16:15 +0400 >>>> >>>> BUGFIX - lmdb: lock meta-pages in writemap-mode to avoid unexpected >>>> write, >>>> before the data pages would be synchronized. >>>> >>>> Without locking the meta-pages may be writen by OS before other >>>> data, >>>> in this case database would be inconsistent. >>> >>> >>> Seems unnecessary. Won't happen by default; could happen with MDB_NOSYNC but >>> that risk is already documented. >> >> We are using the combination: >> envflags writemap nosync lifo >> checkpoint 0 1 >> >> If the checkpoint is set in seconds, it gives us the assurance >> consistent state database on disk. >> However, without this patch meta-pages can be written by the kernel >> before the data. >> >> In fact, for a full guarantee in case of death slapd process, >> meta-page should be written explicitly. No, the DB can never go inconsistent due to a process crashing - the pages in OS cache are always correct. It can only go inconsistent if the OS crashes and a proper sync has not occurred. >> But it requires a lot of changes and I do not do that. >>>> commit 8ddd63161aeb2689822d1a8d27385d62e4e341ae >>>> Author: Leo Yuriev <leo(a)yuriev.ru> >>>> Date: 2014-09-19 22:47:19 +0400 >>>> >>>> BUGFIX - lmdb: properly sync meta-pages in mdb_sync_env(). >>>> >>>> Meta-pages may be updated during data-syncing in mdb_sync_env(), >>>> in this case database would be inconsistent. >>>> >>>> Check-and-retry if lead txn-id changed during flushing data in >>>> mdb_sync_env(). Fundamentally, you are trying to make an inherently unsafe configuration "safer", but it's impossible. Assume you have mlock'd the meta pages into memory, so the OS never flushes them itself any more, and you're running with NOSYNC. That means, within 3 transactions, the data pages on disk will be out of sync with the meta pages on disk. If the OS crashes at that point, the entire DB will be lost. The only way to make this mode of operation somewhat safe is to defer reclaiming pages for even longer. E.g., instead of halting at current_txnid - 3, halt at current_txnid - 22, in which case the data pointed to by the on-disk meta pages cannot get obsolete until 20 transactions have occurred. Note that in combination with your LIFO patch, it's pretty much guaranteed that the on-disk meta pages will be useless after only 2 un-sync'd transactions. >>> Probably could simplify this, just obtain the write mutex unconditionally, >>> then there's no need to loop or retry. But also, this depends on MDB_NOLOCK >>> - if that's set, then do no locking at all. >> >> I did so for reasons of performance and less a lock retention time. >> >> Retries will be if there an intensive flow of changes. >> In this case it will be a lot of updated pages, the record which will >> take some time. >> >> However, in subsequent iterations (if a transactions had committed >> while there was a record), >> the modified pages will be much fewer, and the sync will be quick. >> >> Thus (and it was seen in tests) even when a substantial amount of the >> transactions, >> usually only two iterations of the cycle, >> without locking and flow of changes are not suspended. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7969) LMDB: Globally shared fields of meta-data are not 'volatile'.
by leo＠yuriev.ru 18 Oct '14

18 Oct '14

Hi, Howard. I also noticed a problem with the whilespace, so that is also posted patches on github's gist and gave references in each of three ITS. https://gist.github.com/leo-yuriev/dcedd5e5b2a09c605108 https://gist.github.com/leo-yuriev/fc00313b8035cddba6a7 https://gist.github.com/leo-yuriev/6f1e1444c9e29bd94885 > > Committed to mdb.master. The whitespace is wrong in all of the patches you > submitted, had to apply them manually. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7971) LMDB: Uncarefully appointment when beginning a readonly txn.
by hyc＠symas.com 17 Oct '14

17 Oct '14

Committed to mdb.master > commit ff0bde6a95bb8ab2355324b65f451216d5ef2a0e > Author: Leo Yuriev <leo(a)yuriev.ru> > Date: 2014-10-18 02:00:37 +0400 > > ITS#7971 for LMDB: clarification in mdb_txn_renew0(). > > diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c > index 83a15c1..1e766af 100644 > --- a/libraries/liblmdb/mdb.c > +++ b/libraries/liblmdb/mdb.c > @@ -2507,15 +2507,16 @@ mdb_txn_renew0(MDB_txn *txn) > UNLOCK_MUTEX_R(env); > return MDB_READERS_FULL; > } > - ti->mti_readers[i].mr_pid = pid; > - ti->mti_readers[i].mr_tid = tid; > + r = &ti->mti_readers[i]; > + r->mr_txnid = (txnid_t)-1; > + r->mr_tid = tid; > + r->mr_pid = pid; /* should be written last, see ITS#. */ > if (i == nr) > ti->mti_numreaders = ++nr; > /* Save numreaders for un-mutexed mdb_env_close() */ > env->me_numreaders = nr; > UNLOCK_MUTEX_R(env); > > - r = &ti->mti_readers[i]; > new_notls = (env->me_flags & MDB_NOTLS); > if (!new_notls && (rc=pthread_setspecific(env->me_txkey, r))) { > r->mr_pid = 0; > > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7970) LMDB: Critical Heisenbug - Inconsistent reading & SIGSEGV due to the race condition.
by hyc＠symas.com 17 Oct '14

17 Oct '14

Committed to mdb.master > commit 6028662665c09f4d34b751bfd9b09223f6cb2433 > Author: Leo Yuriev <leo(a)yuriev.ru> > Date: 2014-10-17 23:29:56 +0400 > > ITS#7970 for LMDB: the Heisenbug. > > diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c > index 3286ffb..83a15c1 100644 > --- a/libraries/liblmdb/mdb.c > +++ b/libraries/liblmdb/mdb.c > @@ -2522,7 +2522,12 @@ mdb_txn_renew0(MDB_txn *txn) > return rc; > } > } > - txn->mt_txnid = r->mr_txnid = ti->mti_txnid; > + > + do /* LY: Retry on a race, ITS#7970. */ > + r->mr_txnid = ti->mti_txnid; > + while(r->mr_txnid != ti->mti_txnid); > + > + txn->mt_txnid = r->mr_txnid; > txn->mt_u.reader = r; > meta = env->me_metas[txn->mt_txnid & 1]; > } > > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7969) LMDB: Globally shared fields of meta-data are not 'volatile'.
by hyc＠symas.com 17 Oct '14

17 Oct '14

leo(a)yuriev.ru wrote: > The attached files is derived from OpenLDAP Software. All of the modifications > to OpenLDAP Software represented in the following patch(es) were developed by > Peter-Service LLC, Moscow, Russia. Peter-Service LLC has not assigned rights > and/or interest in this work to any party. I, Leonid Yuriev am authorized by > Peter-Service LLC, my employer, to release this work under the following terms. > > Peter-Service LLC hereby places the following modifications to OpenLDAP Software > (and only these modifications) into the public domain. Hence, these > modifications may be freely used and/or redistributed for any purpose with or > without attribution and/or other notice. Committed to mdb.master. The whitespace is wrong in all of the patches you submitted, had to apply them manually. > > commit 5e83ea8a76f81dac8d5a9dcd74d5b10a01330102 > Author: Leo Yuriev <leo(a)yuriev.ru> > Date: 2014-09-12 01:32:13 +0400 > > ITS#7969 for LMDB: volatile & __synchronize(). > > diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c > index 6cc3433..3286ffb 100644 > --- a/libraries/liblmdb/mdb.c > +++ b/libraries/liblmdb/mdb.c > @@ -580,11 +580,11 @@ typedef struct MDB_rxbody { > * started from so we can avoid overwriting any data used in that > * particular version. > */ > - txnid_t mrb_txnid; > + volatile txnid_t mrb_txnid; > /** The process ID of the process owning this reader txn. */ > - MDB_PID_T mrb_pid; > + volatile MDB_PID_T mrb_pid; > /** The thread ID of the thread owning this txn. */ > - MDB_THR_T mrb_tid; > + volatile MDB_THR_T mrb_tid; > } MDB_rxbody; > > /** The actual reader record, with cacheline padding. */ > @@ -632,12 +632,12 @@ typedef struct MDB_txbody { > * This is recorded here only for convenience; the value can always > * be determined by reading the main database meta pages. > */ > - txnid_t mtb_txnid; > + volatile txnid_t mtb_txnid; > /** The number of slots that have been used in the reader table. > * This always records the maximum count, it is not decremented > * when readers release their slots. > */ > - unsigned mtb_numreaders; > + volatile unsigned mtb_numreaders; > } MDB_txbody; > > /** The actual reader table definition. */ > @@ -908,7 +908,7 @@ typedef struct MDB_meta { > /** Any persistent environment flags. @ref mdb_env */ > #define mm_flags mm_dbs[0].md_flags > pgno_t mm_last_pg; /**< last used page in file */ > - txnid_t mm_txnid; /**< txnid that committed this page */ > + volatile txnid_t mm_txnid; /**< txnid that committed this page */ > } MDB_meta; > > /** Buffer for a stack-allocated meta page. > @@ -3559,6 +3559,10 @@ mdb_env_write_meta(MDB_txn *txn) > mp->mm_dbs[0] = txn->mt_dbs[0]; > mp->mm_dbs[1] = txn->mt_dbs[1]; > mp->mm_last_pg = txn->mt_next_pgno - 1; > +#if !(defined(_MSC_VER) || defined(__i386__) || defined(__x86_64__)) > + /* LY: issue a memory barrier, if not x86. */ > + __sync_synchronize(); > +#endif > mp->mm_txnid = txn->mt_txnid; > if (!(env->me_flags & (MDB_NOMETASYNC|MDB_NOSYNC))) { > unsigned meta_size = env->me_psize; > > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7971) LMDB: Uncarefully appointment when beginning a readonly txn.
by leo＠yuriev.ru 17 Oct '14

17 Oct '14

The attached files is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Peter-Service LLC, Moscow, Russia. Peter-Service LLC has not assigned rights and/or interest in this work to any party. I, Leonid Yuriev am authorized by Peter-Service LLC, my employer, to release this work under the following terms. Peter-Service LLC hereby places the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice. https://gist.github.com/leo-yuriev/6f1e1444c9e29bd94885

1 0

Re: (ITS#7970) LMDB: Critical Heisenbug - Inconsistent reading & SIGSEGV due to the race condition.
by leo＠yuriev.ru 17 Oct '14

17 Oct '14

The attached files is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Peter-Service LLC, Moscow, Russia. Peter-Service LLC has not assigned rights and/or interest in this work to any party. I, Leonid Yuriev am authorized by Peter-Service LLC, my employer, to release this work under the following terms. Peter-Service LLC hereby places the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice. https://gist.github.com/leo-yuriev/fc00313b8035cddba6a7

1 0

Re: (ITS#7969) LMDB: Globally shared fields of meta-data are not 'volatile'.
by leo＠yuriev.ru 17 Oct '14

17 Oct '14

The attached files is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Peter-Service LLC, Moscow, Russia. Peter-Service LLC has not assigned rights and/or interest in this work to any party. I, Leonid Yuriev am authorized by Peter-Service LLC, my employer, to release this work under the following terms. Peter-Service LLC hereby places the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice. https://gist.github.com/leo-yuriev/dcedd5e5b2a09c605108

1 0

Re: (ITS#7971) LMDB: Uncarefully appointment when beginning a readonly txn.
by leo＠yuriev.ru 17 Oct '14

17 Oct '14

The attached files is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Peter-Service LLC, Moscow, Russia. Peter-Service LLC has not assigned rights and/or interest in this work to any party. I, Leonid Yuriev am authorized by Peter-Service LLC, my employer, to release this work under the following terms. Peter-Service LLC hereby places the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice. commit ff0bde6a95bb8ab2355324b65f451216d5ef2a0e Author: Leo Yuriev <leo(a)yuriev.ru> Date: 2014-10-18 02:00:37 +0400 ITS#7971 for LMDB: clarification in mdb_txn_renew0(). diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c index 83a15c1..1e766af 100644 --- a/libraries/liblmdb/mdb.c +++ b/libraries/liblmdb/mdb.c @@ -2507,15 +2507,16 @@ mdb_txn_renew0(MDB_txn *txn) UNLOCK_MUTEX_R(env); return MDB_READERS_FULL; } - ti->mti_readers[i].mr_pid = pid; - ti->mti_readers[i].mr_tid = tid; + r = &ti->mti_readers[i]; + r->mr_txnid = (txnid_t)-1; + r->mr_tid = tid; + r->mr_pid = pid; /* should be written last, see ITS#. */ if (i == nr) ti->mti_numreaders = ++nr; /* Save numreaders for un-mutexed mdb_env_close() */ env->me_numreaders = nr; UNLOCK_MUTEX_R(env); - r = &ti->mti_readers[i]; new_notls = (env->me_flags & MDB_NOTLS); if (!new_notls && (rc=pthread_setspecific(env->me_txkey, r))) { r->mr_pid = 0;

1 0

(ITS#7971) LMDB: Uncarefully appointment when beginning a readonly txn.
by leo＠yuriev.ru 17 Oct '14

17 Oct '14

Full_Name: Leonid Yuriev Version: 2.4.40 OS: RHEL7 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (31.130.36.33) In continue of ITS#7969 and ITS#7970. An additional pages may be allocated unreasonable from map. Also the MDB_MAP_FULL error may returned while the space is really available. As we can see on lines 2500-2550 of mdb.c the field mr_txnid updated after the mr_pid, which is also is a flag of that reader's slot is occupied and other fields are valid, e.g. it is: ti->mti_readers[i].mr_pid = pid; /* setup reader's pid, and indicate that other fields is valid. / ti->mti_readers[i].mr_tid = tid; /* setup reader's thread id. */ ... /* do a little bit of something. / r->mr_txnid = ti->mti_txnid; / * update a "detent" for reclaiming. */ The problem is that not all cases, the value mr_txnid correctly before being updated, but will be counted immediately after setup pid in reader's slot. Valid value for mr_txnid can be any number not less than the last transaction to now. We can easily verify that the MRC may be incorrect adding mdb_eassert(env, r->mr_txnid >= ti->mti_txnid) after updating mti_numreaders. diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c index 262373e..f2ed653 100644 --- a/libraries/liblmdb/mdb.c +++ b/libraries/liblmdb/mdb.c @@ -2522,6 +2522,7 @@ mdb_txn_renew0(MDB_txn *txn) return rc; } %%7} + mdb_eassert(env, r->mr_txnid >= ti->mti_txnid); do /* ITS# */ r->mr_txnid = ti->mti_txnid; And then by 'make test' got: #2 0x0000000000403722 in mdb_assert_fail (env=env@entry=0x134e010, expr_txt=expr_txt@entry=0x40fd14 "r->mr_txnid >= ti->mti_txnid", func=func@entry=0x41067f <__FUNCTION__.7150> "mdb_txn_renew0", line=line@entry=2525, file=0x40fd08 "mdb.c") at mdb.c:1347 buf = "mdb.c:2525: Assertion 'r->mr_txnid >= ti->mti_txnid' failed in mdb_txn_renew0()\000\000\000\000\000\000\000\000\000 \025\063\337\273\177\000\000\360\360\316\367\377\177\000\000\340\360\316\367\377\177\000\000\236,]\242\000\000\000\000x\n@\000\000\000\000\000\377\377\377\377\000\000\000\000\002\000\000\000\000\000\000\000\320\016C3C357\336\273\177\000\000\310\344\062\337\273\177\000\000\001\000\000\000\000\000\000\000\003", '\000' <repeats 15 times>, "M\000\000\000\000\000\000\000\003\000\000\000\000\000\000\000\016\000\000\000\000\000\000\000"... #3 0x0000000000403de8 in mdb_txn_renew0 (txn=txn@entry=0x134e210) at mdb.c:2525 r = 0x7fbbdf32b080 env = 0x134e010 ti = 0x7fbbdf32b000 meta = <optimised out> i = <optimised out> nr = <optimised out> x = <optimised out> rc = <optimised out> new_notls = 0 __FUNCTION__ = "mdb_txn_renew0" #4 0x00000000004045b8 in mdb_txn_begin (env=0x134e010, parent=parent@entry=0x0, flags=flags@entry=131072, ret=ret@entry=0x7ffff7cef2f0) at mdb.c:2706 txn = 0x134e210 ntxn = <optimised out> rc = <optimised out> size = <optimised out> tsize = 136 There is a possibility that the writer will call mdb_page_alloc() while the value of mr is wrong. In such case it is possible then reclaiming couldn't be done and a new page will be allocated from the map. But also possible the MDB_MAP_FULL error. Thus, because of this bug additional pages may be allocated unreasonable, and the MDB_MAP_FULL error may happen while the space is available.

1 0

Re: (ITS#7970) LMDB: Critical Heisenbug - Inconsistent reading & SIGSEGV due to the race condition.
by leo＠yuriev.ru 17 Oct '14

17 Oct '14

The fix is really simple. The patch is attached below, but previously volatile-fix of ITS is required. Also, the simple testcase (make test), with an extra yields and the couple of paranoid asserts, is: diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c index 3286ffb..2558d6f 100644 --- a/libraries/liblmdb/mdb.c +++ b/libraries/liblmdb/mdb.c @@ -2018,6 +2018,8 @@ mdb_page_alloc(MDB_cursor *mc, int num, MDB_page **mp) if (oldest <= last) { if (!found_old) { oldest = mdb_find_oldest(txn); + /* LY: catch heisenbug. */ + mdb_tassert(txn, oldest >= env->me_pgoldest); env->me_pgoldest = oldest; found_old = 1; } @@ -2034,6 +2036,8 @@ mdb_page_alloc(MDB_cursor *mc, int num, MDB_page **mp) if (oldest <= last) { if (!found_old) { oldest = mdb_find_oldest(txn); + /* LY: catch heisenbug. */ + mdb_tassert(txn, oldest >= env->me_pgoldest); env->me_pgoldest = oldest; found_old = 1; } @@ -2522,7 +2526,16 @@ mdb_txn_renew0(MDB_txn *txn) return rc; } } - txn->mt_txnid = r->mr_txnid = ti->mti_txnid; + do { + r->mr_txnid = ti->mti_txnid; + sched_yield(); + sched_yield(); + sched_yield(); + } while(r->mr_txnid != ti->mti_txnid); + sched_yield(); + sched_yield(); + sched_yield(); + txn->mt_txnid = r->mr_txnid; txn->mt_u.reader = r; meta = env->me_metas[txn->mt_txnid & 1]; } --- The attached files is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Peter-Service LLC, Moscow, Russia. Peter-Service LLC has not assigned rights and/or interest in this work to any party. I, Leonid Yuriev am authorized by Peter-Service LLC, my employer, to release this work under the following terms. Peter-Service LLC hereby places the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice. commit 6028662665c09f4d34b751bfd9b09223f6cb2433 Author: Leo Yuriev <leo(a)yuriev.ru> Date: 2014-10-17 23:29:56 +0400 ITS#7970 for LMDB: the Heisenbug. diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c index 3286ffb..83a15c1 100644 --- a/libraries/liblmdb/mdb.c +++ b/libraries/liblmdb/mdb.c @@ -2522,7 +2522,12 @@ mdb_txn_renew0(MDB_txn *txn) return rc; } } - txn->mt_txnid = r->mr_txnid = ti->mti_txnid; + + do /* LY: Retry on a race, ITS#7970. */ + r->mr_txnid = ti->mti_txnid; + while(r->mr_txnid != ti->mti_txnid); + + txn->mt_txnid = r->mr_txnid; txn->mt_u.reader = r; meta = env->me_metas[txn->mt_txnid & 1]; }

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs