- openldap-bugs - openldap.org

Re: (ITS#9098) assert fails in meta_back_search in some cases after reconnect
by clement.oudot＠worteks.com 17 Feb '20

17 Feb '20

Hello, we just has the same crash with OpenLDAP 2.4.49. I think this issue should be reopened as it was not fixed. The core dump given by Maxime can be used to dig into the bug. We can provide other information if needed. -- Clément Oudot | Identity Solutions Manager clement.oudot(a)worteks.com Worteks | https://www.worteks.com

1 0

Re: (ITS#9173) Openldap configuration
by quanah＠symas.com 17 Feb '20

17 Feb '20

--On Monday, February 17, 2020 4:28 PM +0000 biswas.raju(a)gmail.com wrote: > Full_Name: Raju Biswas > Version: 2.4.44 Hello, The ITS system is for reporting bugs, not help requests. If you need help use the openldap-technical(a)openldap.org email list, as that is what it is for. This ITS will be closed. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9173) Openldap configuration
by biswas.raju＠gmail.com 17 Feb '20

17 Feb '20

Full_Name: Raju Biswas Version: 2.4.44 OS: RHEL7.4 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (14.141.44.130) My requirement is to add the posixGroup and groupofNames object class together. To achieve this, I used the link https://devopsideas.com/openldap-linux-client-ldap-integration/ I need help on this. If you need more info kindly mail me so that I can provide more information. I added the customposixGroup and then added the group as dn: cn=server_dev,ou=graylog,ou=rgroup,dc=rad,dc=com objectclass: customposixGroup objectclass: groupOfNames cn: server_dev gidNumber: 7000 description: Server Dev Group member: uid=aron.francis,ou=People,dc=rad,dc=com User added as dn: uid=aron.francis,ou=People,dc=rad,dc=comcn: aron.francis givenName: aron.francis sn: useruid: aron.francis uidNumber: 7001gidNumber: 7000 homeDirectory: /home/aron.francis objectClass: top objectClass: posixAccount objectClass: shadowAccountobjectClass: inetOrgPerson objectClass: organizationalPerson objectClass: personloginShell: /bin/bash userPassword: {SHA}Gf3pq4puDhvQ/KIgE7c1QQixnM4= memberOf: cn=server_dev,ou=graylog,ou=rgroup,dc=rad,dc=com Configured the sssduid=7001(aron.francis(a)rad.com) gid=7000 groups=7000 Help I wanted help on why the group name is not getting displayed when I use the id command from the LDAP client. But if I use posixGroup alone and not groupOfName object class then the group name is getting displayed. I need to use both posixGroup and groupOfNames [root@rad testing_dev]# ldapsearch -H ldaps:// -x -b "dc=rad,dc=com" "uid=aron.francis" "member=uid=aron.francis,ou=People,dc=rad,dc=com" # extended LDIF # # LDAPv3 # base <dc=rad,dc=com> with scope subtree # filter: uid=aron.francis # requesting: member=uid=aron.francis,ou=People,dc=rad,dc=com # # aron.francis, People, rad.com dn: uid=aron.francis,ou=People,dc=rad,dc=com # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [root@radisys testing_dev]# ldapsearch -H ldaps:// -x -b "dc=rad,dc=com" "member=uid=aron.francis,ou=People,dc=rad,dc=com" # extended LDIF # # LDAPv3 # base <dc=rad,dc=com> with scope subtree # filter: member=uid=aron.francis,ou=People,dc=rad,dc=com # requesting: ALL # # server_dev, graylog, rgroup, rad.com dn: cn=server_dev,ou=graylog,ou=rgroup,dc=rad,dc=com objectClass: top objectClass: aposixGroup objectClass: groupOfNames cn: server_dev gidNumber: 7000 description: Server Dev Group member: uid=aron.francis,ou=People,dc=rad,dc=com # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [root@rad testing_dev]#

1 0

Re: (ITS#9017) Improving performance of commit sync in Windows
by hyc＠symas.com 17 Feb '20

17 Feb '20

hyc(a)symas.com wrote: > Kris Zyp wrote: >> Sorry to keep pestering, but just pinging about this patch again, as I still think this fix could benefit windows users. And at this point, I think I can say we >> have tested it pretty well, running on our servers for almost a year :). > > Looks like this patch is against the 0.9 release branch. I hit a bunch of conflicts > trying to apply it to mdb.master. We'll be stopping work on 0.9 soon, and getting > LMDB 1.0 out the door finally, so can you please verify that your changes will work > on mdb.master as well? I was manually applying this patch to mdb.master and see something odd in the changes to mdb_page_flush. Why did you move the MIPS-specific CACHEFLUSH invocation? > >> Thanks, >> Kris >> >> On Wed, Sep 18, 2019 at 12:56 PM Kris Zyp <kriszyp(a)gmail.com <mailto:kriszyp@gmail.com>> wrote: >> >> Checking on this again, is this still a possibility for merging into LMDB? This fix is still working great (improved performance) on our systems. >> Thanks, >> Kris >> >> On Mon, Jun 17, 2019 at 1:04 PM Kris Zyp <kriszyp(a)gmail.com <mailto:kriszyp@gmail.com>> wrote: >> >> Is this still being considered/reviewed? Let me know if there are any other changes you would like me to make. This patch has continued to yield >> significant and reliable performance improvements for us, and seems like it would be nice for this to be available for other Windows users. >> >> On Fri, May 3, 2019 at 3:52 PM Kris Zyp <kriszyp(a)gmail.com <mailto:kriszyp@gmail.com>> wrote: >> >> For the sake of putting this in the email thread (other code discussion in GitHub), here is the latest squashed commit of the proposed patch (with >> the on-demand, retained overlapped array to reduce re-malloc and opening event handles): >> https://github.com/kriszyp/node-lmdb/commit/726a9156662c703bf3d453aab75ee22… >> > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9017) Improving performance of commit sync in Windows
by hyc＠symas.com 17 Feb '20

17 Feb '20

Kris Zyp wrote: > Sorry to keep pestering, but just pinging about this patch again, as I still think this fix could benefit windows users. And at this point, I think I can say we > have tested it pretty well, running on our servers for almost a year :). Looks like this patch is against the 0.9 release branch. I hit a bunch of conflicts trying to apply it to mdb.master. We'll be stopping work on 0.9 soon, and getting LMDB 1.0 out the door finally, so can you please verify that your changes will work on mdb.master as well? > Thanks, > Kris > > On Wed, Sep 18, 2019 at 12:56 PM Kris Zyp <kriszyp(a)gmail.com <mailto:kriszyp@gmail.com>> wrote: > > Checking on this again, is this still a possibility for merging into LMDB? This fix is still working great (improved performance) on our systems. > Thanks, > Kris > > On Mon, Jun 17, 2019 at 1:04 PM Kris Zyp <kriszyp(a)gmail.com <mailto:kriszyp@gmail.com>> wrote: > > Is this still being considered/reviewed? Let me know if there are any other changes you would like me to make. This patch has continued to yield > significant and reliable performance improvements for us, and seems like it would be nice for this to be available for other Windows users. > > On Fri, May 3, 2019 at 3:52 PM Kris Zyp <kriszyp(a)gmail.com <mailto:kriszyp@gmail.com>> wrote: > > For the sake of putting this in the email thread (other code discussion in GitHub), here is the latest squashed commit of the proposed patch (with > the on-demand, retained overlapped array to reduce re-malloc and opening event handles): > https://github.com/kriszyp/node-lmdb/commit/726a9156662c703bf3d453aab75ee22… > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9171) slapd with ppolicy and pwdAccountLockedTime crashes
by tokos＠ipp.cas.cz 17 Feb '20

17 Feb '20

--000000000000d6ac46059ec636d9 Content-Type: text/plain; charset="UTF-8" Hi Ondrej, I uploaded my configuration here(ftp.openldap.org/incoming) : ftp> ls 200 PORT command successful 150 Opening ASCII mode data connection for file list -rw-r--r-- 1 ftp ftp 22099 Feb 17 13:55 backtrace-ldap.txt -rw-r--r-- 1 ftp ftp 110686 Feb 17 13:51 config-200217.ldif -rw-r--r-- 1 ftp ftp 5436 Feb 17 13:51 data-200217.ldif -rw-r--r-- 1 ftp ftp 5341 Feb 17 13:51 slapd.conf-compassinitial 226 Transfer complete ftp> pwd 257 "/incoming/9171" is the current directory Please, could you look into it? Thanks in advance. Stanislav --000000000000d6ac46059ec636d9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div>Hi Ondrej,</div><div> </div><div>=C2=A0 = I uploaded my configuration here(<a href=3D"http://ftp.openldap.org/incomin= g">ftp.openldap.org/incoming</a>)=C2=A0:</div><div> </div><div> </div><div dir=3D"ltr">ftp> ls 200 PORT command successful 150 Opening ASCII mode data connection for file list -rw-r--r-- =C2=A0=C2=A01 ftp =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0ftp =C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A022099 Feb 17 13:55 backtrace-ldap= .txt -rw-r--r-- =C2=A0=C2=A01 ftp =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0ftp =C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0110686 Feb 17 13:51 config-200217.ldif -rw-r--r-- =C2=A0=C2=A01 ftp =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0ftp =C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A05436 Feb 17 13:51 data-2002= 17.ldif -rw-r--r-- =C2=A0=C2=A01 ftp =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0ftp =C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A05341 Feb 17 13:51 slapd.con= f-compassinitial 226 Transfer complete ftp> pwd 257 "/incoming/9171" is the current directory </div><div>Please, could y= ou look into it?=C2=A0</div><div> </div><div>Thanks in=C2= =A0advance.</div><div> </sp= an></div><div>Stanislav</div><= /div></div> --000000000000d6ac46059ec636d9--

1 0

Re: (ITS#9171) slapd with ppolicy and pwdAccountLockedTime crashes
by ondra＠mistotebe.net 17 Feb '20

17 Feb '20

--eAbsdosE1cNLO4uF Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit On Fri, Feb 14, 2020 at 09:38:05AM +0000, tokos(a)ipp.cas.cz wrote: > Several logins on locked account with operational attribute pwdAccountLockedTime > ends with crash slapd. Hi Stanislav, I've tried to reproduce the issue, but everything works just fine for me with the attached configuration. Are you using any other overlays and modules apart from ppolicy? It would be best if you could attach your configuration (without passwords), backtrace and other set up needed to reproduce the issue. Thanks, -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP --eAbsdosE1cNLO4uF Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="slapd.conf" include ./re24/servers/slapd/schema/core.schema include ./re24/servers/slapd/schema/cosine.schema include ./re24/servers/slapd/schema/inetorgperson.schema include ./re24/servers/slapd/schema/ppolicy.schema include rfc2307bis.schema moduleload ./re24/servers/slapd/overlays/ppolicy.la database mdb directory ./db suffix dc=compass overlay ppolicy ppolicy_default cn=ppolicy,ou=policies,dc=compass --eAbsdosE1cNLO4uF--

1 0

(ITS#9171) slapd with ppolicy and pwdAccountLockedTime crashes
by tokos＠ipp.cas.cz 14 Feb '20

14 Feb '20

Full_Name: Stanislav Toko. Version: 2.4.45 - 2.4.49 OS: Gentoo and Ubuntu 18.04 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (147.231.37.132) Several logins on locked account with operational attribute pwdAccountLockedTime ends with crash slapd. Details are below: #1 0x00007f92f6e1a801 in __GI_abort () at abort.c:79 save_stage = 1 act = {__sigaction_handler = {sa_handler = 0x7f9290000c40, sa_sigaction = 0x7f9290000c40}, sa_mask = {__val = {0, 0, 0, 0, 0, 140267535435096, 0, 140267535434928, 140269185487312, 21474836480, 140269185472472, 0, 3086719068234460928, 140269185457428, 0, 140269185472472}}, sa_flags = -282311760, sa_restorer = 0x7f92ef2c41a3} sigs = {__val = {32, 0 <repeats 15 times>}} __cnt = <optimized out> __set = <optimized out> __cnt = <optimized out> __set = <optimized out> #2 0x00007f92f6e0a39a in __assert_fail_base (fmt=0x7f92f6f917d8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7f92ef2c41a3 "rs->sr_ctrls != NULL", file=file@entry=0x7f92ef2c43b0 "../../../../../servers/slapd/overlays/ppolicy.c", line=line@entry=912, function=function@entry=0x7f92ef2c5310 "ctrls_cleanup") at assert.c:92 str = 0x7f9290000c40 "`\032" total = 4096 #3 0x00007f92f6e0a412 in __GI___assert_fail (assertion=0x7f92ef2c41a3 "rs->sr_ctrls != NULL", file=0x7f92ef2c43b0 "../../../../../servers/slapd/overlays/ppolicy.c", line=912, function=0x7f92ef2c5310 "ctrls_cleanup") at assert.c:101 No locals. Feb 14 08:49:50 net slapd[8393]: => get_ctrls Feb 14 08:49:50 net slapd[8393]: => get_ctrls: oid="1.3.6.1.4.1.42.2.27.8.5.1" (noncritical) Feb 14 08:49:50 net slapd[8393]: <= get_ctrls: n=1 rc=0 err="" Feb 14 08:49:50 net slapd[8393]: >>> dnPrettyNormal: <uid=test,ou=People,dc=compass> Feb 14 08:49:50 net slapd[8393]: <<< dnPrettyNormal: <uid=test,ou=People,dc=compass>, <uid=test,ou=people,dc=compass> Feb 14 08:49:50 net slapd[8393]: conn=1005 op=1 BIND dn="uid=test,ou=People,dc=compass" method=128 Feb 14 08:49:50 net slapd[8393]: do_bind: version=3 dn="uid=test,ou=People,dc=compass" method=128 Feb 14 08:49:50 net slapd[8393]: => mdb_entry_get: ndn: "uid=test,ou=people,dc=compass" Feb 14 08:49:50 net slapd[8393]: => mdb_entry_get: oc: "(null)", at: "(null)" Feb 14 08:49:50 net slapd[8393]: mdb_dn2entry("uid=test,ou=people,dc=compass") Feb 14 08:49:50 net slapd[8393]: => mdb_dn2id("uid=test,ou=people,dc=compass") Feb 14 08:49:50 net slapd[8393]: <= mdb_dn2id: got id=0x6 Feb 14 08:49:50 net slapd[8393]: => mdb_entry_decode: Feb 14 08:49:50 net slapd[8393]: <= mdb_entry_decode Feb 14 08:49:50 net slapd[8393]: => mdb_entry_get: found entry: "uid=test,ou=people,dc=compass" Feb 14 08:49:50 net slapd[8393]: mdb_entry_get: rc=0 Feb 14 08:49:50 net slapd[8393]: => mdb_entry_get: ndn: "cn=ppolicy,ou=policies,dc=compass" Feb 14 08:49:50 net slapd[8393]: => mdb_entry_get: oc: "(null)", at: "(null)" Feb 14 08:49:50 net slapd[8393]: mdb_dn2entry("cn=ppolicy,ou=policies,dc=compass") Feb 14 08:49:50 net slapd[8393]: => mdb_dn2id("cn=ppolicy,ou=policies,dc=compass") Feb 14 08:49:50 net slapd[8393]: <= mdb_dn2id: got id=0x9 Feb 14 08:49:50 net slapd[8393]: => mdb_entry_decode: Feb 14 08:49:50 net slapd[8393]: <= mdb_entry_decode Feb 14 08:49:50 net slapd[8393]: => mdb_entry_get: found entry: "cn=ppolicy,ou=policies,dc=compass" Feb 14 08:49:50 net slapd[8393]: mdb_entry_get: rc=0 Feb 14 08:49:50 net slapd[8393]: send_ldap_result: conn=1005 op=1 p=3 Feb 14 08:49:50 net slapd[8393]: send_ldap_result: err=49 matched="" text="" Feb 14 08:49:50 net slapd[8393]: send_ldap_response: msgid=2 tag=97 err=49 Feb 14 08:49:50 net slapd[8393]: conn=1005 op=1 RESULT tag=97 err=49 text= Feb 14 08:49:50 net slapd[8716]: * Stopping OpenLDAP slapd Feb 14 08:49:50 net slapd[8716]: ...done. database: dn: dc=compass dc: compass objectClass: top objectClass: domain objectClass: dcObject creatorsName: cn=Manager,dc=compass o: COMPASS description: old compass domain modifiersName: cn=manager,dc=compass structuralObjectClass: domain entryUUID: 7127423c-e0e2-1039-97dc-334be5149c5f createTimestamp: 20200211062111Z entryCSN: 20200211062111.313576Z#000000#000#000000 modifyTimestamp: 20200211062111Z dn: ou=Group,dc=compass ou: Group objectClass: top objectClass: organizationalUnit structuralObjectClass: organizationalUnit entryUUID: 7b30de80-e0e4-1039-84b8-c7dff63f23d9 creatorsName: cn=manager,dc=compass createTimestamp: 20200211063547Z entryCSN: 20200211063547.147292Z#000000#001#000000 modifiersName: cn=manager,dc=compass modifyTimestamp: 20200211063547Z dn: cn=users,ou=Group,dc=compass cn: users gidNumber: 100 objectClass: posixGroup objectClass: groupOfMembers objectClass: top objectClass: sambaGroupMapping sambaSID: S-1-5-21-3927199971-2202295266-3581313430-1201 sambaGroupType: 2 memberUid: fridrich structuralObjectClass: groupOfMembers entryUUID: 7b30eff6-e0e4-1039-84b9-c7dff63f23d9 creatorsName: cn=manager,dc=compass createTimestamp: 20200211063547Z entryCSN: 20200211063547.147739Z#000000#001#000000 modifiersName: cn=manager,dc=compass modifyTimestamp: 20200211063547Z dn: ou=People,dc=compass ou: People structuralObjectClass: organizationalUnit entryUUID: 7b310586-e0e4-1039-84ba-c7dff63f23d9 creatorsName: cn=manager,dc=compass createTimestamp: 20200211063547Z objectClass: organizationalUnit objectClass: top entryCSN: 20200211111336.349173Z#000000#001#000000 modifiersName: cn=manager,dc=compass modifyTimestamp: 20200211111336Z dn: uid=test,ou=People,dc=compass cn: test test sn: test uid: test uidNumber: 1358 gidNumber: 100 homeDirectory: /compass/home/test loginShell: /bin/bash gecos: test test givenName: test displayName: test test krb5KDCFlags: 126 krb5PrincipalName: test(a)TOK.IPP.CAS.CZ mail: test(a)test.cz manager: uid=fridrich,ou=People,dc=compass structuralObjectClass: inetOrgPerson entryUUID: 6e413e62-e0e5-1039-84bb-c7dff63f23d9 creatorsName: cn=manager,dc=compass createTimestamp: 20200211064234Z shadowLastChange: 18303 userPassword:: e1NTSEF9djBSTjBYUXFRdm1LdHRLTEpLbm9nWnM5cEF1NWl2OFY= krb5KeyVersionNumber: 14 krb5Key:: MDShEzARoAMCAQGhCgQIdtwIZKEO04qiHTAboAMCAQOhFAQSVE9LLklQUC5DQVMuQ1 p0ZXN0 krb5Key:: MDShEzARoAMCAQKhCgQIdtwIZKEO04qiHTAboAMCAQOhFAQSVE9LLklQUC5DQVMuQ1 p0ZXN0 krb5Key:: MDShEzARoAMCAQOhCgQIdtwIZKEO04qiHTAboAMCAQOhFAQSVE9LLklQUC5DQVMuQ1 p0ZXN0 krb5Key:: MEShIzAhoAMCARChGgQYx7AONCytE1gay6jvNEaegLZDsFgTnaiboh0wG6ADAgEDoR QEElRPSy5JUFAuQ0FTLkNadGVzdA== krb5Key:: MDyhGzAZoAMCARehEgQQg1WsugnCKytyC1Uox/1uv6IdMBugAwIBA6EUBBJUT0suSV BQLkNBUy5DWnRlc3Q= krb5Key:: MEyhKzApoAMCARKhIgQgXXs6NrJQUZFXZy00IXNfViZYxbBTaL5WeRwlOFa1xlWiHT AboAMCAQOhFAQSVE9LLklQUC5DQVMuQ1p0ZXN0 pwdChangedTime: 20200212113320Z pwdHistory: 20200212113320Z#1.3.6.1.4.1.1466.115.121.1.40#38#{SSHA}tOXL8axp5 Op31HrMfL/cOMokrHqEfVD2 pwdFailureTime: 20200212134208.674800Z pwdFailureTime: 20200212134212.394433Z pwdFailureTime: 20200212134220.973965Z pwdFailureTime: 20200212134224.525514Z pwdFailureTime: 20200212134228.174908Z objectClass: inetOrgPerson objectClass: krb5KDCEntry objectClass: organizationalPerson objectClass: person objectClass: posixAccount objectClass: sambaSamAccount objectClass: shadowAccount objectClass: top sambaAcctFlags: [UX] sambaHomeDrive: H: sambaHomePath: \\DIPROTON\test sambaLMPassword: 2fdae2e1215a354faad3b435b51404ee sambaLogoffTime: 2147483647 sambaLogonScript: logon.bat sambaLogonTime: 2147483647 sambaNTPassword: 8355acba09c22b2b720b5528c7fd6ebf sambaPrimaryGroupSID: S-1-5-21-3927199971-2202295266-3581313430-1201 sambaPwdCanChange: 17563 sambaPwdLastSet: 1581507200 sambaPwdMustChange: 2147483647 sambaSID: S-1-5-21-3927199971-2202295266-3581313430-3716 pwdAccountLockedTime: 000001010000Z entryCSN: 20200212161607.719838Z#000000#001#000000 modifiersName: cn=manager,dc=compass modifyTimestamp: 20200212161607Z dn: ou=policies,dc=compass objectClass: top objectClass: organizationalUnit ou: policies description: Password policy structuralObjectClass: organizationalUnit entryUUID: a7264508-e1fb-1039-9fc0-6fee7d141603 creatorsName: cn=manager,dc=compass createTimestamp: 20200212155410Z entryCSN: 20200212155410.485051Z#000000#001#000000 modifiersName: cn=manager,dc=compass modifyTimestamp: 20200212155410Z dn: cn=ppolicy,ou=policies,dc=compass objectClass: pwdPolicy objectClass: device objectClass: top pwdAllowUserChange: TRUE pwdAttribute: userPassword pwdCheckQuality: 2 pwdExpireWarning: 600 pwdFailureCountInterval: 30 pwdGraceAuthNLimit: 5 pwdInHistory: 5 pwdLockout: TRUE pwdLockoutDuration: 0 pwdMaxAge: 0 pwdMaxFailure: 5 pwdMinAge: 0 pwdMinLength: 5 pwdMustChange: FALSE pwdSafeModify: FALSE structuralObjectClass: device entryUUID: ea2d7e38-e1fc-1039-9fc1-6fee7d141603 creatorsName: cn=manager,dc=compass createTimestamp: 20200212160312Z cn: ppolicy entryCSN: 20200212160530.268134Z#000000#001#000000 modifiersName: cn=Manager,dc=compass modifyTimestamp: 20200212160530Z

1 0

Re: (ITS#9168) Memory Leak in LDAP search operation
by quanah＠symas.com 13 Feb '20

13 Feb '20

--On Thursday, February 13, 2020 4:50 PM +0000 bananashake2004(a)yahoo.de wrote: > Full_Name: Stefan Koch > Version: 2.4.44 Hello, The 2.4.44 release is over 4 years old. Please use a current OpenLDAP release prior to reporting bugs. This ITS will be closed. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9168) Memory Leak in LDAP search operation
by bananashake2004＠yahoo.de 13 Feb '20

13 Feb '20

Full_Name: Stefan Koch Version: 2.4.44 OS: Debian Stretch URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2001:4dd0:f0a3:2:a44f:a486:843b:bdde) Valgrind shows a memory leak in slapd that will lead to successive increment of memory consumption at runtime. valgrind --tool=memcheck --leak-check=yes --num-callers=50 /usr/sbin/slapd -d [...] ==26332== 22,000 (80 direct, 21,920 indirect) bytes in 1 blocks are definitely lost in loss record 1,768 of 1,791 ==26332== at 0x4C2DBC5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26332== by 0x5090D24: ber_memcalloc_x (in /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.7) ==26332== by 0x15CEC2: ch_calloc (in /usr/sbin/slapd) ==26332== by 0x9D8582C: ??? ==26332== by 0x9D8E048: ??? ==26332== by 0x143E40: fe_op_search (in /usr/sbin/slapd) ==26332== by 0x143803: do_search (in /usr/sbin/slapd) ==26332== by 0x141485: ??? (in /usr/sbin/slapd) ==26332== by 0x141764: ??? (in /usr/sbin/slapd) ==26332== by 0x4E47FD9: ??? (in /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.7) ==26332== by 0x69024A3: start_thread (pthread_create.c:456) ==26332== LEAK SUMMARY: ==26332== definitely lost: 3,555 bytes in 209 blocks ==26332== indirectly lost: 22,510 bytes in 297 blocks ==26332== possibly lost: 41,688 bytes in 66 blocks ==26332== still reachable: 4,749,541 bytes in 10,436 blocks ==26332== suppressed: 0 bytes in 0 blocks This bug was already reported here: https://www.openldap.org/lists/openldap-devel/200507/msg00053.html

1 0

Re: (ITS#9167) Using OpenLDAP as Proxy
by quanah＠symas.com 13 Feb '20

13 Feb '20

--On Thursday, February 13, 2020 1:40 PM +0000 abboud.makhoul(a)emcrey.com wrote: > Full_Name: Abboud Makhoul > Version: 2.4.44 > OS: oracle linux 7.6 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (109.233.20.178) > > > We are trying to use OpenLDAP as a proxy to OUD (Oracle Unified > Directory). We have added the following file in > /etc/openldap/slapd.d/cn=config: olcDatabase={1}ldap.ldif Hello, The ITS system is for reporting bug, not usage questions. Please use the openldap-technical(a)openldap.org email list for usage questions such as this. This ITS will be closed. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9167) Using OpenLDAP as Proxy
by abboud.makhoul＠emcrey.com 13 Feb '20

13 Feb '20

Full_Name: Abboud Makhoul Version: 2.4.44 OS: oracle linux 7.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (109.233.20.178) We are trying to use OpenLDAP as a proxy to OUD (Oracle Unified Directory). We have added the following file in /etc/openldap/slapd.d/cn=config: olcDatabase={1}ldap.ldif The file contains the following: dn: olcDatabase={1}ldap,cn=config objectClass: olcDatabaseConfig objectClass: olcLDAPConfig olcDatabase: {1}ldap olcSuffix: dc=example,dc=com olcAccess: {0}to dn.base="" by * read olcAccess: {1}to dn.base="cn=oudadmin" by * read olcAccess: {2}to * by self write by users read by anonymous auth olcRootDN: cn=oudadmin,dc=example,dc=com olcRootPW: password olcDbURI: "ldap://localhost:1389" olcDbACLBind: bindmethod=simple binddn="cn=oudadmin" credentials="password" starttls=no After running the ldapadd command, we are getting the following error: Invalid syntax (21) additional info: objectClass: value #1 invalid per syntax. It seems that the objectClasses are not being recognized.

1 0

(ITS#9166) RE25 admin guide fails to build
by ryan＠openldap.org 12 Feb '20

12 Feb '20

Full_Name: Ryan Tandy Version: 2.5 OS: URL: Submission from: (NULL) (70.66.128.207) Submitted by: ryan openldap/doc/guide/admin$ make sdf -2html guide.sdf ./slapdconfig.sdf warning, para. on 531: {{ phrase not terminated make: *** [Makefile:81: guide.html] Error 8

1 0

(ITS#9165) Paged search results on FreeBSD exceed static buffer
by fjo-openldap＠ogris.de 11 Feb '20

11 Feb '20

Full_Name: Felix J. Ogris Version: 2.4.49 OS: FreeBSD 12.1 amd64 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (80.139.74.144) Hi, I observed that cookies for paged searches against a Windows based Active Directory are up to 1000 bytes in length. print_paged_results() in clients/tools/common.c uses a buf of BUFSIZ chars. BUFSIZ on FreeBSD is only 1024 - see https://svnweb.freebsd.org/base/release/12.1.0/include/stdio.h?view=markup#… Thus, ldapsearch against a Windows AD will print the cookie without a trailing newline, and may expose a security issue if querying an arbitrary LDAP server. Fixing this specific occurrence is easy (e.g. replace buf[BUFIZ] by buf[8192]). What about defining a global LDAP_BUFIZ of at least 8192, since BUFSIZ is used at several locations in the OpenLDAP source tree? BR, Felix

1 0

Re: (ITS#9164) back-monitor returning non-requested attributes
by quanah＠symas.com 06 Feb '20

06 Feb '20

--On Thursday, February 6, 2020 8:47 PM +0000 quanah(a)openldap.org wrote: > This appears to be because they (IMHO) incorrectly SUP monitoredInfo or > monitorCounter Or, thinking on it further, I think this is ok, just not what I expected. I'll close the ITS. ;) --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9164) back-monitor returning non-requested attributes
by quanah＠openldap.org 06 Feb '20

06 Feb '20

Full_Name: Quanah Gibson-Mount Version: 2.4.49 OS: N/A URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.128.236) The new additions to back-monitor to expose more data from back-mdb are returning unrequested attributes. For example: ldapsearch -x -LLL -H ldap:/// -s base -b "cn=database 1,cn=databases,cn=monitor" "(|(objectClass=monitorCounterObject)(objectClass=monitorOperation)(objectClass=monitoredObject))" monitorCounter monitorOpInitiated monitorOpCompleted monitoredInfo olmMDBPagesMax olmMDBPagesUsed olmMDBPagesFree olmMDBReadersMax olmMDBReadersUsed olmMDBEntries Will return: dn: cn=Database 1,cn=Databases,cn=Monitor monitoredInfo: mdb monitorOverlay: dynlist olmMDBPagesMax: 262144 olmMDBPagesUsed: 629 olmMDBPagesFree: 58 olmMDBReadersMax: 126 olmMDBReadersUsed: 2 olmMDBEntries: 1005 olmDbDirectory: /var/symas/openldap-data/example/ Note that I did not request monitorOverlay nor did I request olmDbDirectory This appears to be because they (IMHO) incorrectly SUP monitoredInfo or monitorCounter

1 0

Re: (ITS#9163) warning: logical not is only applied to the left hand side of bitwise operator
by hyc＠symas.com 06 Feb '20

06 Feb '20

noloader(a)gmail.com wrote: > Full_Name: Jeffrey Walton > Version: 2.4.47 > OS: OS X 10.10 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (100.18.0.145) > > > Hi Everyone. I'm building OpenSLAP 2.4.47 from the release tarball on OS X. > Clang is triggering a warning that may be of interest: Thanks for the report. Appears to be a dup of ITS#8890, which was fixed in master in 2018. Looks like this fell through the cracks for the 2.4 release. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9017) Improving performance of commit sync in Windows
by kriszyp＠gmail.com 06 Feb '20

06 Feb '20

--000000000000bdb86d059deaf350 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sorry to keep pestering, but just pinging about this patch again, as I still think this fix could benefit windows users. And at this point, I think I can say we have tested it pretty well, running on our servers for almost a year :). Thanks, Kris On Wed, Sep 18, 2019 at 12:56 PM Kris Zyp <kriszyp(a)gmail.com> wrote: > Checking on this again, is this still a possibility for merging into LMDB= ? > This fix is still working great (improved performance) on our systems. > Thanks, > Kris > > On Mon, Jun 17, 2019 at 1:04 PM Kris Zyp <kriszyp(a)gmail.com> wrote: > >> Is this still being considered/reviewed? Let me know if there are any >> other changes you would like me to make. This patch has continued to yie= ld >> significant and reliable performance improvements for us, and seems like= it >> would be nice for this to be available for other Windows users. >> >> On Fri, May 3, 2019 at 3:52 PM Kris Zyp <kriszyp(a)gmail.com> wrote: >> >>> For the sake of putting this in the email thread (other code discussion >>> in GitHub), here is the latest squashed commit of the proposed patch (w= ith >>> the on-demand, retained overlapped array to reduce re-malloc and openin= g >>> event handles): >>> https://github.com/kriszyp/node-lmdb/commit/726a9156662c703bf3d453aab75= ee222072b990f >>> >>> >>> >>> Thanks, >>> Kris >>> >>> >>> >>> *From: *Kris Zyp <kriszyp(a)gmail.com> >>> *Sent: *April 30, 2019 12:43 PM >>> *To: *Howard Chu <hyc(a)symas.com>; openldap-its(a)OpenLDAP.org >>> *Subject: *RE: (ITS#9017) Improving performance of commit sync in >>> Windows >>> >>> >>> >>> > What is the point of using writemap mode if you still need to use >>> WriteFile >>> >>> > on every individual page? >>> >>> >>> >>> As I understood from the documentation, and have observed, using >>> writemap mode is faster (and uses less temporary memory) because it doe= sn=E2=80=99t >>> require mallocs to allocate pages (docs: =E2=80=9CThis is faster and us= es fewer >>> mallocs=E2=80=9D). To be clear though, LMDB is so incredibly fast and e= fficient, >>> that in sync-mode, it takes enormous transactions before the time spent >>> allocating and creating the dirty pages with the updated b-tree is anyw= here >>> even remotely close to the time it takes to wait for disk flushing, eve= n >>> with an SSD. But the more pertinent question is efficiency, and measuri= ng >>> CPU cycles rather than time spent (efficiency is more important than ju= st >>> time spent). When I ran my tests this morning of 100 (sync) transaction= s >>> with 100 puts per transaction, times varied quite a bit, but it seemed = like >>> running with writemap enabled typically averages about 500ms of CPU and >>> with writemap disabled it typically averages around 600ms. Not a huge >>> difference, but still definitely worthwhile, I think. >>> >>> >>> >>> Caveat emptor: Measuring LMDB performance with sync interactions on >>> Windows is one of the most frustratingly erratic things to measure. It = is >>> sunny outside right now, times could be different when it starts rainin= g >>> later, but, this is what I saw this morning... >>> >>> >>> >>> > What is the performance difference between your patch using writemap, >>> and just >>> >>> > not using writemap in the first place? >>> >>> >>> >>> Running 1000 sync transactions on 3GB db with a single put per >>> transaction, without writemap map, without the patch took about 60 seco= nds. >>> And it took about 1 second with the patch with writemap mode enabled! >>> (there is no significant difference in sync times with writemap enabled= or >>> disabled with the patch.) So the difference was huge in my test. And no= t >>> only that, without the patch, the CPU usage was actually _*higher*_ >>> during that 60 seconds (close to 100% of a core) than during the execut= ion >>> with the patch for one second (close to 50%). Anyway, there are certai= nly >>> tests I have run where the differences are not as large (doing small >>> commits on large dbs accentuates the differences), but the patch always >>> seems to win. It could also be that my particular configuration causes >>> bigger differences (on an SSD drive, and maybe a more fragmented file?)= . >>> >>> >>> >>> Anyway, I added error handling for the malloc, and fixed/changed the >>> other things you suggested. Be happy to make any other changes you want= . >>> The updated patch is here: >>> >>> >>> https://github.com/kriszyp/node-lmdb/commit/25366dea9453749cf6637f43ec1= 7b9b62094acde >>> >>> >>> >>> > OVERLAPPED* ov =3D malloc((pagecount - keep) * sizeof(OVERLAPPED)); >>> >>> > Probably this ought to just be pre-allocated based on the maximum >>> number of dirty pages a txn allows. >>> >>> >>> >>> I wasn=E2=80=99t sure I understood this comment. Are you suggesting we = malloc(MDB_IDL_UM_MAX >>> * sizeof(OVERLAPPED)) for each environment, and retain it for the life = of >>> the environment? I think that is 4MB, if my math is right, which seems = like >>> a lot of memory to keep allocated (we usually have a lot of open >>> environments). If the goal is to reduce the number of mallocs, how abou= t we >>> retain the OVERLAPPED array, and only free and re-malloc if the previou= s >>> allocation wasn=E2=80=99t large enough? Then there isn=E2=80=99t unnece= ssary allocation, >>> and we only malloc when there is a bigger transaction than any previous= . I >>> put this together in a separate commit, as I wasn=E2=80=99t sure if thi= s what you >>> wanted (can squash if you prefer): >>> https://github.com/kriszyp/node-lmdb/commit/2fe68fb5269c843e2e789746a17= a4b2adefaac40 >>> >>> >>> >>> Thank you for the review! >>> >>> >>> >>> Thanks, >>> Kris >>> >>> >>> >>> *From: *Howard Chu <hyc(a)symas.com> >>> *Sent: *April 30, 2019 7:12 AM >>> *To: *kriszyp(a)gmail.com; openldap-its(a)OpenLDAP.org >>> *Subject: *Re: (ITS#9017) Improving performance of commit sync in >>> Windows >>> >>> >>> >>> kriszyp(a)gmail.com wrote: >>> >>> > Full_Name: Kristopher William Zyp >>> >>> > Version: LMDB 0.9.23 >>> >>> > OS: Windows >>> >>> > URL: >>> https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0a= b9332b7fc4ce9 >>> >>> > Submission from: (NULL) (71.199.6.148) >>> >>> > >>> >>> > >>> >>> > We have seen very poor performance on the sync of commits on large >>> databases in >>> >>> > Windows. On databases with 2GB of data, in writemap mode, the sync of >>> even small >>> >>> > commits is consistently well over 100ms (without writemap it is >>> faster, but >>> >>> > still slow). It is expected that a sync should take some time while >>> waiting for >>> >>> > disk confirmation of the writes, but more concerning is that these sy= nc >>> >>> > operations (in writemap mode) are instead dominated by nearly 100% >>> system CPU >>> >>> > utilization, so operations that requires sub-millisecond b-tree updat= e >>> >>> > operations are then dominated by very large amounts of system CPU >>> cycles during >>> >>> > the sync phase. >>> >>> > >>> >>> > I think that the fundamental problem is that FlushViewOfFile seems to >>> be an O(n) >>> >>> > operation where n is the size of the file (or map). I presume that >>> Windows is >>> >>> > scanning the entire map/file for dirty pages to flush, I'm guessing >>> because it >>> >>> > doesn't have an internal index of all the dirty pages for every >>> file/map-view in >>> >>> > the OS disk cache. Therefore, the turns into an extremely expensive, >>> CPU-bound >>> >>> > operation to find the dirty pages for (large file) and initiate their >>> writes, >>> >>> > which, of course, is contrary to the whole goal of a scalable databas= e >>> system. >>> >>> > And FlushFileBuffers is also relatively slow as well. We have >>> attempted to batch >>> >>> > as many operations into single transaction as possible, but this is >>> still a very >>> >>> > large overhead. >>> >>> > >>> >>> > The Windows docs for FlushFileBuffers itself warns about the >>> inefficiencies of >>> >>> > this function ( >>> https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi= -flushfilebuffers >>> ). >>> >>> > Which also points to the solution: it is much faster to write out the >>> dirty >>> >>> > pages with WriteFile through a sync file handle >>> (FILE_FLAG_WRITE_THROUGH). >>> >>> > >>> >>> > The associated patch >>> >>> > ( >>> https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0a= b9332b7fc4ce9 >>> ) >>> >>> > is my attempt at implementing this solution, for Windows. Fortunately= , >>> with the >>> >>> > design of LMDB, this is relatively straightforward. LMDB already >>> supports >>> >>> > writing out dirty pages with WriteFile calls. I added a write-through >>> handle for >>> >>> > sending these writes directly to disk. I then made that file-handle >>> >>> > overlapped/asynchronously, so all the writes for a commit could be >>> started in >>> >>> > overlap mode, and (at least theoretically) transfer in parallel to th= e >>> drive and >>> >>> > then used GetOverlappedResult to wait for the completion. So basicall= y >>> >>> > mdb_page_flush becomes the sync. I extended the writing of dirty page= s >>> through >>> >>> > WriteFile to writemap mode as well (for writing meta too), so that >>> WriteFile >>> >>> > with write-through can be used to flush the data without ever needing >>> to call >>> >>> > FlushViewOfFile or FlushFileBuffers. I also implemented support for >>> write >>> >>> > gathering in writemap mode where contiguous file positions infers >>> contiguous >>> >>> > memory (by tracking the starting position with wdp and writing >>> contiguous pages >>> >>> > in single operations). Sorting of the dirty list is maintained even i= n >>> writemap >>> >>> > mode for this purpose. >>> >>> >>> >>> What is the point of using writemap mode if you still need to use >>> WriteFile >>> >>> on every individual page? >>> >>> >>> >>> > The performance benefits of this patch, in my testing, are >>> considerable. Writing >>> >>> > out/syncing transactions is typically over 5x faster in writemap mode= , >>> and 2x >>> >>> > faster in standard mode. And perhaps more importantly (especially in >>> environment >>> >>> > with many threads/processes), the efficiency benefits are even larger= , >>> >>> > particularly in writemap mode, where there can be a 50-100x reduction >>> in the >>> >>> > system CPU usage by using this patch. This brings windows performance >>> with >>> >>> > sync'ed transactions in LMDB back into the range of "lightning" >>> performance :). >>> >>> >>> >>> What is the performance difference between your patch using writemap, >>> and just >>> >>> not using writemap in the first place? >>> >>> >>> >>> -- >>> >>> -- Howard Chu >>> >>> CTO, Symas Corp. http://www.symas.com >>> >>> Director, Highland Sun http://highlandsun.com/hyc/ >>> >>> Chief Architect, OpenLDAP http://www.openldap.org/project/ >>> >>> >>> >>> >>> >> --000000000000bdb86d059deaf350 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Sorry to keep pestering, but just pinging about this patch= again, as I still think this fix could benefit windows users. And at this = point, I think I can say we have tested it pretty well, running on our serv= ers for almost a year :).<div>Thanks,</div><div>Kris</div></div> <div cl= ass=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Sep 18, 2= 019 at 12:56 PM Kris Zyp <<a href=3D"mailto:kriszyp@gmail.com">kriszyp@g= mail.com</a>> wrote: </div><blockquote class=3D"gmail_quote" style=3D= "margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-le= ft:1ex"><div dir=3D"ltr">Checking on this again, is this still a possibilit= y for merging into LMDB? This fix is still working great (improved performa= nce) on our systems.<div>Thanks,</div><div>Kris</div></div> <div class= =3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Jun 17, 2019= at 1:04 PM Kris Zyp <<a href=3D"mailto:kriszyp@gmail.com" target=3D"_bl= ank">kriszyp(a)gmail.com</a>> wrote: </div><blockquote class=3D"gmail_q= uote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,2= 04);padding-left:1ex"><div dir=3D"ltr">Is this still being considered/revie= wed? Let me know if there are any other changes you would like me to make. = This patch has continued to yield significant and reliable performance impr= ovements for us, and seems like it would be nice for this to be available f= or other Windows users.</div> <div class=3D"gmail_quote"><div dir=3D"ltr= " class=3D"gmail_attr">On Fri, May 3, 2019 at 3:52 PM Kris Zyp <<a href= =3D"mailto:kriszyp@gmail.com" target=3D"_blank">kriszyp(a)gmail.com</a>> w= rote: </div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0p= x 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang= =3D"EN-CA"><div>For the sake of putting this in the = email thread (other code discussion in GitHub), here is the latest squashed= commit of the proposed patch (with the on-demand, retained overlapped arra= y to reduce re-malloc and opening event handles): <a href=3D"https://github= .com/kriszyp/node-lmdb/commit/726a9156662c703bf3d453aab75ee222072b990f" tar= get=3D"_blank">https://github.com/kriszyp/node-lmdb/commit/726a9156662c703b= f3d453aab75ee222072b990f</a>=C2=A0= Thanks, Kris</u= >=C2=A0<div style=3D"border-right:none;border-bottom:none;border= -left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">From: <a href=3D= "mailto:kriszyp@gmail.com" target=3D"_blank">Kris Zyp</a> Sent: A= pril 30, 2019 12:43 PM To: <a href=3D"mailto:hyc@symas.com" targe= t=3D"_blank">Howard Chu</a>; <a href=3D"mailto:openldap-its@OpenLDAP.org" t= arget=3D"_blank">openldap-its(a)OpenLDAP.org</a> Subject: RE: (ITS#= 9017) Improving performance of commit sync in Windows</div>=C2=A0> What is the = point of using writemap mode if you still need to use WriteFile</= u>> on every individual page?</= p>=C2=A0As = I understood from the documentation, and have observed, using writemap mode= is faster (and uses less temporary memory) because it doesn=E2=80=99t requ= ire mallocs to allocate pages (docs: =E2=80=9CThis is faster and uses fewer= mallocs=E2=80=9D). To be clear though, LMDB is so incredibly fast and effi= cient, that in sync-mode, it takes enormous transactions before the time sp= ent allocating and creating the dirty pages with the updated b-tree is anyw= here even remotely close to the time it takes to wait for disk flushing, ev= en with an SSD. But the more pertinent question is efficiency, and measurin= g CPU cycles rather than time spent (efficiency is more important than just= time spent). When I ran my tests this morning of 100 (sync) transactions w= ith 100 puts per transaction, times varied quite a bit, but it seemed like = running with writemap enabled typically averages about 500ms of CPU and wit= h writemap disabled it typically averages around 600ms. Not a huge differen= ce, but still definitely worthwhile, I think.=C2=A0Caveat emptor: Me= asuring LMDB performance with sync interactions on Windows is one of the mo= st frustratingly erratic things to measure. It is sunny outside right now, = times could be different when it starts raining later, but, this is what I = saw this morning...=C2=A0<u= >> What is the performance difference bet= ween your patch using writemap, and just> not using writemap in the first place?=C2=A0Running 1000 = sync transactions on 3GB db with a single put per transaction, without writ= emap map, without the patch took about 60 seconds. And it took about 1 seco= nd with the patch with writemap mode enabled! (there is no significant diff= erence in sync times with writemap enabled or disabled with the patch.) So = the difference was huge in my test. And not only that, without the patch, t= he CPU usage was actually _higher_ during that 60 seconds (close to = 100% of a core) than during the execution with the patch for one second (cl= ose to 50%).=C2=A0 Anyway, there are certainly tests I have run where the d= ifferences are not as large (doing small commits on large dbs accentuates t= he differences), but the patch always seems to win. It could also be that m= y particular configuration causes bigger differences (on an SSD drive, and = maybe a more fragmented file?).= =C2=A0Anyway, I added error handling = for the malloc, and fixed/changed the other things you suggested. Be happy = to make any other changes you want. The updated patch is here:</u= ><a href=3D"https://github.com/kriszyp/node-lmdb= /commit/25366dea9453749cf6637f43ec17b9b62094acde" target=3D"_blank">https:/= /github.com/kriszyp/node-lmdb/commit/25366dea9453749cf6637f43ec17b9b62094ac= de</a>=C2=A0> OVERLAPPED* ov =3D malloc<= /span>((pagecount - keep) * sizeof<s= pan style=3D"font-size:9pt;font-family:Consolas;color:rgb(36,41,46)">(OVERL= APPED));> Probably th= is ought to just be pre-allocated based on the maximum number of dirty page= s a txn allows.=C2=A0<= /span>I w= asn=E2=80=99t sure I understood this comment. Are you suggesting we = malloc(MDB_IDL_UM_MAX * sizeof(OVERLAPPED)) for each environment, and retai= n it for the life of the environment? I think that is 4MB, if my math is ri= ght, which seems like a lot of memory to keep allocated (we usually have a = lot of open environments). If the goal is to reduce the number of mallocs, = how about we retain the OVERLAPPED array, and only free and re-malloc if th= e previous allocation wasn=E2=80=99t large enough? Then there isn=E2=80=99t= unnecessary allocation, and we only malloc when there is a bigger transact= ion than any previous. I put this together in a separate commit, as I wasn= =E2=80=99t sure if this what you wanted (can squash if you prefer): <a href= =3D"https://github.com/kriszyp/node-lmdb/commit/2fe68fb5269c843e2e789746a17= a4b2adefaac40" target=3D"_blank">https://github.com/kriszyp/node-lmdb/commi= t/2fe68fb5269c843e2e789746a17a4b2adefaac40</a>=C2=A0Thank you for th= e review! <= p class=3D"MsoNormal">=C2=A0Thanks= , Kris=C2=A0<= div style=3D"border-right:none;border-bottom:none;border-left:none;border-t= op:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm"><= b>From: <a href=3D"mailto:hyc@symas.com" target=3D"_blank">Howard Chu</= a> Sent: April 30, 2019 7:12 AM To: <a href=3D"mailto:k= riszyp(a)gmail.com" target=3D"_blank">kriszyp(a)gmail.com</a>; <a href=3D"mailt= o:openldap-its@OpenLDAP.org" target=3D"_blank">openldap-its(a)OpenLDAP.org</a= > Subject: Re: (ITS#9017) Improving performance of commit sync in= Windows</div>=C2=A0= <a href=3D"mailto:kriszyp@gmail.com" target=3D"_= blank">kriszyp(a)gmail.com</a> wrote:> Full_Name: Kristopher William Zyp> Version: LMDB 0.9.23>= OS: Windows> URL: <a href=3D"h= ttps://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab9332= b7fc4ce9" target=3D"_blank">https://github.com/kriszyp/node-lmdb/commit/7ff= 525ae57684a163d32af74a0ab9332b7fc4ce9</a>> Submission from: (NULL) (71.199.6.148)> > <= u>> We have seen very poor performance on= the sync of commits on large databases in> Windows. On databases with 2GB of data, in writemap mode, the = sync of even small> commits is = consistently well over 100ms (without writemap it is faster, but<= /u>> still slow). It is expected that a sync = should take some time while waiting for> disk confirmation of the writes, but more concerning is that thes= e sync> operations (in writemap= mode) are instead dominated by nearly 100% system CPU> utilization, so operations that requires sub-milli= second b-tree update> operation= s are then dominated by very large amounts of system CPU cycles during</= u>> the sync phase.<= p class=3D"MsoNormal">> > I = think that the fundamental problem is that FlushViewOfFile seems to be an O= (n)> operation where n is the s= ize of the file (or map). I presume that Windows is> scanning the entire map/file for dirty pages to flush= , I'm guessing because it> = doesn't have an internal index of all the dirty pages for every file/ma= p-view in> the OS disk cache. T= herefore, the turns into an extremely expensive, CPU-bound</p= >> operation to find the dirty pages for (large f= ile) and initiate their writes,&gt= ; which, of course, is contrary to the whole goal of a scalable database sy= stem.> And FlushFileBuffers is = also relatively slow as well. We have attempted to batch<= p class=3D"MsoNormal">> as many operations into single transaction as po= ssible, but this is still a very&g= t; large overhead.> <= /u>> The Windows docs for FlushFileBuffers it= self warns about the inefficiencies of> this function (<a href=3D"https://docs.microsoft.com/en-us/windows= /desktop/api/fileapi/nf-fileapi-flushfilebuffers" target=3D"_blank">https:/= /docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-flushfileb= uffers</a>).> Which also points= to the solution: it is much faster to write out the dirty</p= >> pages with WriteFile through a sync file handl= e (FILE_FLAG_WRITE_THROUGH).> <= u>> The associated patch= > (<a href=3D"https://github.com/kriszyp/= node-lmdb/commit/7ff525ae57684a163d32af74a0ab9332b7fc4ce9" target=3D"_blank= ">https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab9= 332b7fc4ce9</a>)> is my attempt= at implementing this solution, for Windows. Fortunately, with the<u= >> design of LMDB, this is relatively str= aightforward. LMDB already supports> writing out dirty pages with WriteFile calls. I added a write-through= handle for> sending these writ= es directly to disk. I then made that file-handle> overlapped/asynchronously, so all the writes for a comm= it could be started in> overlap= mode, and (at least theoretically) transfer in parallel to the drive and<u= >> then used GetOverlappedResult t= o wait for the completion. So basically> mdb_page_flush becomes the sync. I extended the writing of dirty = pages through> WriteFile to wri= temap mode as well (for writing meta too), so that WriteFile<= /p>> with write-through can be used to flush the = data without ever needing to call&= gt; FlushViewOfFile or FlushFileBuffers. I also implemented support for wri= te> gathering in writemap mode = where contiguous file positions infers contiguous> memory (by tracking the starting position with wdp and = writing contiguous pages> in si= ngle operations). Sorting of the dirty list is maintained even in writemap<= u>> mode for this purpose.<= u>=C2=A0What is the point of using writemap mode if you still need to use Writ= eFileon every individual page?<= /u>=C2=A0> The performance benefits of this patch, in my testing, are co= nsiderable. Writing> out/syncin= g transactions is typically over 5x faster in writemap mode, and 2x<= u>> faster in standard mode. And perhaps = more importantly (especially in environment> with many threads/processes), the efficiency benefits are eve= n larger,> particularly in writ= emap mode, where there can be a 50-100x reduction in the<= p class=3D"MsoNormal">> system CPU usage by using this patch. This bring= s windows performance with> syn= c'ed transactions in LMDB back into the range of "lightning" = performance :).=C2=A0</u= >What is the performance difference between your= patch using writemap, and justnot= using writemap in the first place?=C2=A0-- =C2=A0=C2=A0-- Howard Chu=C2=A0 CTO, Symas Corp.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 <a href=3D"http://www.symas.com" target=3D"_blank">http:= //www.symas.com</a>=C2=A0 Director= , Highland Sun=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"http://highlandsun.com/hy= c/" target=3D"_blank">http://highlandsun.com/hyc/</a>=C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http:/= /www.openldap.org/project/" target=3D"_blank">http://www.openldap.org/proje= ct/</a>=C2=A0=C2=A0</div></div></blockquote></div> </blockquote></div> </blockquote></div> --000000000000bdb86d059deaf350--

1 0

(ITS#9163) warning: logical not is only applied to the left hand side of bitwise operator
by noloader＠gmail.com 06 Feb '20

06 Feb '20

Full_Name: Jeffrey Walton Version: 2.4.47 OS: OS X 10.10 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (100.18.0.145) Hi Everyone. I'm building OpenSLAP 2.4.47 from the release tarball on OS X. Clang is triggering a warning that may be of interest: gcc -g2 -O2 -march=native -fPIC -pthread -I../../include -I/usr/local/include -DNDEBUG -c -o common.o common.c ... common.c:2329:10: warning: logical not is only applied to the left hand side of this bitwise operator [-Wlogical-not-parentheses] if ( !tool_ctrl_response[j].mask & tool_type ) { ^ ~ common.c:2329:10: note: add parentheses after the '!' to evaluate the bitwise operator first if ( !tool_ctrl_response[j].mask & tool_type ) { ^ ( ) common.c:2329:10: note: add parentheses around left hand side expression to silence this warning if ( !tool_ctrl_response[j].mask & tool_type ) { ^ ( ) As written the statement is being evaluated as: (!tool_ctrl_response[j].mask) & tool_type It looks like this is what was intended: !(tool_ctrl_response[j].mask & tool_type) Perhaps the project can add parenthesis to clear the warning.

1 0

Re: (ITS#8988) Undefined Behavior in slapadd
by openldap-technical＠kolttonen.fi 03 Feb '20

03 Feb '20

This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463811718-1330329680-1580755187=:215607 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 8BIT Hello, On Mon, 3 Feb 2020, hyc(a)symas.com wrote: > No. The POSIX spec is *not* a part of the C spec - yet it is still valid. POSIX spec *is* a C standard that exists in addition to the ISO C standard. ISO C defines the so-called "basic language and features" of C if you can call it that. > Which simply proves the point that just because something is not defined > in the C spec does not make it invalid. The compiler guys are idiots for > taking this position. Like Jeff already said, wrongly aligned object access *is* defined by the ISO C specification. It is wrong and explicitly defined as Undefined Behaviour. Invoking UB is playing with fire, and calling compiler writers with offending names does not change that fact at all. Spending long time on comp.lang.c should be mandatory for all C programmers out there. It is shocking to invoke UB and not bother to fix it, instead blaming compiler writers and C standard writers. Best regards, Jokke Hämäläinen ---1463811718-1330329680-1580755187=:215607--

1 0

Re: (ITS#9162) dbMaxSize property does not as expected
by quanah＠symas.com 03 Feb '20

03 Feb '20

--On Monday, February 3, 2020 9:49 AM +0000 pasumarthivijaykumar(a)gmail.com wrote: > Full_Name: Vijay Kumar > Version: 2.4.48 > OS: windows > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (103.204.96.6) Please stop filing ITSes, as you clearly are working with an already broken database. This ITS will be closed. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#8988) Undefined Behavior in slapadd
by hyc＠symas.com 03 Feb '20

03 Feb '20

openldap-technical(a)kolttonen.fi wrote: > This message is in MIME format. The first part should be readable text, > while the remaining parts are likely unreadable without MIME-aware tools. > > ---1463811718-806940296-1580670508=:196090 > Content-Type: text/plain; charset=ISO-8859-15 > Content-Transfer-Encoding: 8BIT > > > Hello, > > On Sat, 8 Jun 2019, hyc(a)symas.com wrote: > >> The gcc/clang folks have their heads up their asses. They've >> deliberately misinterpreted the spec claiming undefined behaviors are >> forbidden, supposedly to enable essential optimizations. Most of which >> only apply to obscure corner cases in compiler benchmark suites, that >> nobody in the real world ever benefits from. > > I realize this thread is very old, but Jeff and the C compiler folks are > right. All C programs that invoke undefined behavior are illegal C > programs and should be fixed. > > Posix threads are well-defined by Posix standards, so calling them > "undefined behaviour" is not a valid argument. No. The POSIX spec is *not* a part of the C spec - yet it is still valid. Which simply proves the point that just because something is not defined in the C spec does not make it invalid. The compiler guys are idiots for taking this position. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9162) dbMaxSize property does not as expected
by pasumarthivijaykumar＠gmail.com 03 Feb '20

03 Feb '20

Full_Name: Vijay Kumar Version: 2.4.48 OS: windows URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (103.204.96.6) if i try to set the value from property, i dont understand from where the value 1406945XXXXXX and getting set. Please check the details mentioned. 1) olcDbMaxSize: 10485760 >mdb_stat.exe -rne "C:\testMaxSize2\mdb\data.mdb" Environment Info Map address: 0000000000000000 Map size: 140694549168128 Page size: 4096 Max pages: 34349255168 Number of pages used: 2 Last transaction ID: 0 Max readers: 126 Number of readers used: 0 Reader Table Status (no active readers) >slapd.exe -T add -F ..\slapd.d -l backupContent.ldif 5e37a371 mdb_db_open: database "o=abc.net" cannot be opened: ⁿ (87). Restore from backup! 5e37a371 backend_startup_one (type=mdb, suffix="o=abc.net"): bi_db_open failed! (87) slap_startup failed 2) olcDbMaxSize: 1048576 >slapd.exe -T add -F ..\slapd.d -l backupContent.ldif 5e37a5ef mdb_db_open: database "o=abc.net" cannot be opened: ⁿ (87). Restore from backup! 5e37a5ef backend_startup_one (type=mdb, suffix="o=abc.net"): bi_db_open failed! (87) slap_startup failed >mdb_stat.exe -rne "C:\testMaxSize2\mdb\data.mdb" Environment Info Map address: 0000000000000000 Map size: 140694539730944 Page size: 4096 Max pages: 34349252864 Number of pages used: 2 Last transaction ID: 0 Max readers: 126 Number of readers used: 0 Reader Table Status (no active readers) 3) olcDbMaxSize: 1024 >slapd.exe -T add -F ..\slapd.d -l backupContent.ldif 5e37a7fd mdb_db_open: database "o=abc.net" cannot be opened: ⁿ (87). Restore from backup! 5e37a7fd backend_startup_one (type=mdb, suffix="o=abc.net"): bi_db_open failed! (87) slap_startup failed >mdb_stat.exe -rne "C:\testMaxSize2\mdb\data.mdb" Environment Info Map address: 0000000000000000 Map size: 140694538683392 Page size: 4096 Max pages: 34349252608 Number of pages used: 2 Last transaction ID: 0 Max readers: 126 Number of readers used: 0 Reader Table Status (no active readers) 5) olcDbMaxSize 10 >slapd.exe -T add -F ..\slapd.d -l backupContent.ldif 5e37a92b mdb_db_open: database "o=abc.net" cannot be opened: ⁿ (87). Restore from backup! 5e37a92b backend_startup_one (type=mdb, suffix="o=abc.net"): bi_db_open failed! (87) slap_startup failed >mdb_stat.exe -rne "C:\testMaxSize2\mdb\data.mdb" Environment Info Map address: 0000000000000000 Map size: 140694538682378 Page size: 4096 Max pages: 34349252608 Number of pages used: 2 Last transaction ID: 0 Max readers: 126 Number of readers used: 0 Reader Table Status (no active readers) 6) olcDbMaxSize 1 >slapd.exe -T add -F ..\slapd.d -l backupContent.ldif 5e37a92b mdb_db_open: database "o=abc.net" cannot be opened: ⁿ (87). Restore from backup! 5e37a92b backend_startup_one (type=mdb, suffix="o=abc.net"): bi_db_open failed! (87) slap_startup failed >mdb_stat.exe -rne "C:\testMaxSize2\mdb\data.mdb" Environment Info Map address: 0000000000000000 Map size: 140694538682369 Page size: 4096 Max pages: 34349252608 Number of pages used: 2 Last transaction ID: 0 Max readers: 126 Number of readers used: 0 Reader Table Status (no active readers) 7) olcDbMaxSize not set >mdb_stat.exe -rne "C:\testMaxSize2\mdb\data.mdb" Environment Info Map address: 0000000000000000 Map size: 1048576 Page size: 4096 Max pages: 256 Number of pages used: 254 Last transaction ID: 104 Max readers: 126 Number of readers used: 0 Reader Table Status (no active readers)

1 0

Re: (ITS#8988) Undefined Behavior in slapadd
by openldap-technical＠kolttonen.fi 02 Feb '20

02 Feb '20

This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463811718-326687762-1580674674=:196994 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 8BIT On Sun, 2 Feb 2020, openldap-technical(a)kolttonen.fi wrote: > I realize this thread is very old, but Jeff and the C compiler folks are > right. All C programs that invoke undefined behavior are illegal C > programs and should be fixed. > > Posix threads are well-defined by Posix standards, so calling them > "undefined behaviour" is not a valid argument. To make it clear that UB in C programs is horrible, let me provide a real world example of UB: Cyrus IMAPD had an unnoticed dormant strcpy() related UB bug for several years. You see, the manual page of strcpy() says that the src and dst strings must not overlap, or else the C program invokes UB. For many years, Cyrus operated correctly even though the src and dst strings *did* overlap in one part of their database code. UB of course allows this behaviour too. The reason was because GNU libc folks had written their strcpy() implementation in such a way that breaking the contract of having non-overlapping strings did not cause any problems. Then, years later, GNU libc folks deciced to optimize (or otherwise change) their strcpy() implementation. Now the new implementation punished all UB invokers, and we saw our Cyrus mailbox database slowly but surely getting corrupted. It is indeed shocking to hear that LMDB implementation invokes UB. GCC and Clang could change their behaviour tomorrow, and LMDB could get corrupted, crash or whatever. And certainly the blame would be on LMDB code, not the compiler writers. Best Regards, Jokke Hämäläinen ---1463811718-326687762-1580674674=:196994--

1 0

Re: (ITS#8988) Undefined Behavior in slapadd
by openldap-technical＠kolttonen.fi 02 Feb '20

02 Feb '20

This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463811718-806940296-1580670508=:196090 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 8BIT Hello, On Sat, 8 Jun 2019, hyc(a)symas.com wrote: > The gcc/clang folks have their heads up their asses. They've > deliberately misinterpreted the spec claiming undefined behaviors are > forbidden, supposedly to enable essential optimizations. Most of which > only apply to obscure corner cases in compiler benchmark suites, that > nobody in the real world ever benefits from. I realize this thread is very old, but Jeff and the C compiler folks are right. All C programs that invoke undefined behavior are illegal C programs and should be fixed. Posix threads are well-defined by Posix standards, so calling them "undefined behaviour" is not a valid argument. Best regards, Jokke Hämäläinen ---1463811718-806940296-1580670508=:196090--

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs