openldap-bugs

openldap-bugs@openldap.org

1 participants
20965 discussions

RE: (ITS#8972) Performance/freezing issues using NTDLL onWindowsServer on Azure
by kriszyp＠gmail.com 08 Feb '19

08 Feb '19

--_FEED8D1F-4CE2-4DE8-A6B1-1703AFFCC4FC_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To follow up to this, after switching to the stable release in node-ldmb, a= nd running on our servers for several days, I can confirm this definitely d= oes fix our issues (and ticket can be closed). I=E2=80=99ve submitted a PR = for getting the correct stable version in node-lmdb, to ensure in this does= n=E2=80=99t happen for other NodeJS users: https://github.com/Venemo/node-l= mdb/pull/141=20 I did find a minor issue with setting up the file mapping in the release ve= rsion, but I will file that separately. And if the NTDLL mapping isn=E2=80= =99t intended to be merged into a release branch, sounds like there shouldn= =E2=80=99t be any issues for Windows users that are using LMDB releases. And to be clear, I am a happy windows user, sorting out versions hardly den= ts my enthusiasm for LMDB. And thank you again for the help. Thanks, Kris --_FEED8D1F-4CE2-4DE8-A6B1-1703AFFCC4FC_ Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8" <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta ht= tp-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta name= =3DGenerator content=3D"Microsoft Word 15 (filtered medium)"><style></style></head><body lang=3DEN-CA link=3Dblue vlink=3D"#954F72"><div cla= ss=3DWordSection1>To follow up to this, after switchin= g to the stable release in node-ldmb, and running on our servers for severa= l days, I can confirm this definitely does fix our issues (and ticket can b= e closed). I=E2=80=99ve submitted a PR for getting the correct stable versi= on in node-lmdb, to ensure in this doesn=E2=80=99t happen for other NodeJS = users: <a href=3D"https://github.com/Venemo/node-lmdb/pull/141">https://git= hub.com/Venemo/node-lmdb/pull/141</a> <o:p> <= /o:p>I did find a minor issue with setting up the = file mapping in the release version, but I will file that separately. And i= f the NTDLL mapping isn=E2=80=99t intended to be merged into a release bran= ch, sounds like there shouldn=E2=80=99t be any issues for Windows users tha= t are using LMDB releases.<o:p> </o:p>And to be clear, I am a happy windows user, sorting out v= ersions hardly dents my enthusiasm for LMDB. And thank you again for the he= lp.<o:p></o:p><o:p> </o:p>Thanks, Kris<o:p></o:p></div></body></html>= --_FEED8D1F-4CE2-4DE8-A6B1-1703AFFCC4FC_--

1 0

Re: (ITS#8974) olcAttributeOptions cannot be modified once set
by hyc＠symas.com 07 Feb '19

07 Feb '19

skoranda(a)gmail.com wrote: > Full_Name: Scott Koranda > Version: 2.4.47 > OS: Debian Linux 9.6 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (104.231.255.140) > > > Once the option olcAttributeOptions has been configured it cannot be modified. New options can be added; existing options cannot be deleted. Feel free to submit a patch adding the required support. Note the FIXME comment in slapd/bconfig.c. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8974) olcAttributeOptions cannot be modified once set
by skoranda＠gmail.com 07 Feb '19

07 Feb '19

Full_Name: Scott Koranda Version: 2.4.47 OS: Debian Linux 9.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (104.231.255.140) Once the option olcAttributeOptions has been configured it cannot be modified. To reproduce begin with the following state: #ldapsearch -LLL -Y EXTERNAL -H ldapi:/// -b cn=config -s base olcAttributeOptions 2>/dev/null dn: cn=config olcAttributeOptions: lang- app- Create an LDIF file to modify the value for olcAttributeOptions: # cat /root/olcAttributeOptions.ldif dn: cn=config changetype: modify replace: olcAttributeOptions olcAttributeOptions: lang- app- internal prior role- scope- time- Use ldapmodify to attempt to modify the configuration: # ldapmodify -Y EXTERNAL -H ldapi:/// -f /root/olcAttributeOptions.ldif SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth SASL SSF: 0 modifying entry "cn=config" ldap_modify: Other (e.g., implementation specific) error (80) The slapd process running with -d -1 emits this output: 5c5c2e01 slapd: line 0: option "lang-" is already defined 5c5c2e01 olcAttributeOptions: value #0: <olcAttributeOptions> handler exited with 1! This behavior happens regardless of whether any directory has records that include options with attributes.

1 0

Re: (ITS#8973) Error with domainScope control when using C# LDAP API
by clement.oudot＠worteks.com 06 Feb '19

06 Feb '19

Le 06/02/2019 à 12:57, hyc(a)symas.com a écrit : > hyc(a)symas.com wrote: >> clement.oudot(a)worteks.com wrote: >>> Full_Name: Clement OUDOT >>> Version: 2.4.47 >>> OS: GNU/Linux >>> URL: ftp://ftp.openldap.org/incoming/ >>> Submission from: (NULL) (81.250.130.213) >>> >>> >>> Hello, >>> >>> after upgrading from 2.4.46 to 2.4.47, this C# code does not work anymore : >> The code in controls.c is correct. The DomainScope control does not take a value, >> so the value is required to be NULL. > Microsoft spec https://msdn.microsoft.com/en-us/library/cc223348.aspx > > "When sending this control to the DC, the controlValue field of the Control structure is omitted." > Thanks a lot for your feedback, I will try to understand with the C# code is sending a value inside the control. -- Clément Oudot | Identity Solutions Manager clement.oudot(a)worteks.com Worteks | https://www.worteks.com

1 0

Re: (ITS#8973) Error with domainScope control when using C# LDAP API
by hyc＠symas.com 06 Feb '19

06 Feb '19

hyc(a)symas.com wrote: > clement.oudot(a)worteks.com wrote: >> Full_Name: Clement OUDOT >> Version: 2.4.47 >> OS: GNU/Linux >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (81.250.130.213) >> >> >> Hello, >> >> after upgrading from 2.4.46 to 2.4.47, this C# code does not work anymore : > > The code in controls.c is correct. The DomainScope control does not take a value, > so the value is required to be NULL. Microsoft spec https://msdn.microsoft.com/en-us/library/cc223348.aspx "When sending this control to the DC, the controlValue field of the Control structure is omitted." -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8973) Error with domainScope control when using C# LDAP API
by hyc＠symas.com 06 Feb '19

06 Feb '19

clement.oudot(a)worteks.com wrote: > Full_Name: Clement OUDOT > Version: 2.4.47 > OS: GNU/Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (81.250.130.213) > > > Hello, > > after upgrading from 2.4.46 to 2.4.47, this C# code does not work anymore : The code in controls.c is correct. The DomainScope control does not take a value, so the value is required to be NULL. > ---- > $auth = [System.DirectoryServices.AuthenticationTypes]::FastBind > $de = New-Object System.DirectoryServices.DirectoryEntry($DN,$userReferentiel, > $passwordReferentiel, $auth) > ---- > > The log in OpenLDAP 2.4.47 : > ---- > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 fd=17 ACCEPT from > IP=10.10.2.163:47304 (IP=0.0.0.0:390) > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=0 BIND > dn="uid=ADPassword,ou=configuration,dc=referentiel,dc=local" method=128 > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=0 BIND > dn="uid=ADPassword,ou=configuration,dc=referentiel,dc=local" mech=SIMPLE ssf=0 > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=0 RESULT tag=97 err=0 text= > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=1 SRCH > base="dc=referentiel,dc=local" scope=0 deref=0 filter="(objectClass=*)" > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=1 SRCH attr=objectClass > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=1 SEARCH RESULT tag=101 err=0 > nentries=1 text= > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=2 SRCH base="" scope=0 > deref=0 filter="(objectClass=*)" > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=2 SRCH attr=subschemaSubentry > dsServiceName namingContexts defaultNamingContext schemaNamingContext > configurationNamingContext rootDomainNamingContext supportedControl > supportedLDAPVersion supportedLDAPPolicies supportedSASLMechanisms dnsHostName > ldapServiceName serverName supportedCapabilities > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=2 SEARCH RESULT tag=101 err=0 > nentries=1 text= > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=3 SEARCH RESULT tag=101 err=2 > nentries=0 text=domainScope control value not absent > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=3 do_search: get_ctrls > failed > ---- > > The behavior change is related to the fix of ITS#8840 : > www.openldap.org/its/index.cgi/Software Bugs?id=8840 > > I did not see how to change the C# code, are we sure that the fix of ITS#8840 is > valid? > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8973) Error with domainScope control when using C# LDAP API
by clement.oudot＠worteks.com 06 Feb '19

06 Feb '19

Full_Name: Clement OUDOT Version: 2.4.47 OS: GNU/Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (81.250.130.213) Hello, after upgrading from 2.4.46 to 2.4.47, this C# code does not work anymore : ---- $auth = [System.DirectoryServices.AuthenticationTypes]::FastBind $de = New-Object System.DirectoryServices.DirectoryEntry($DN,$userReferentiel, $passwordReferentiel, $auth) ---- The log in OpenLDAP 2.4.47 : ---- Feb 5 18:21:03 anteros slapd[30762]: conn=1577 fd=17 ACCEPT from IP=10.10.2.163:47304 (IP=0.0.0.0:390) Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=0 BIND dn="uid=ADPassword,ou=configuration,dc=referentiel,dc=local" method=128 Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=0 BIND dn="uid=ADPassword,ou=configuration,dc=referentiel,dc=local" mech=SIMPLE ssf=0 Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=0 RESULT tag=97 err=0 text= Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=1 SRCH base="dc=referentiel,dc=local" scope=0 deref=0 filter="(objectClass=*)" Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=1 SRCH attr=objectClass Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=1 SEARCH RESULT tag=101 err=0 nentries=1 text= Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=2 SRCH base="" scope=0 deref=0 filter="(objectClass=*)" Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=2 SRCH attr=subschemaSubentry dsServiceName namingContexts defaultNamingContext schemaNamingContext configurationNamingContext rootDomainNamingContext supportedControl supportedLDAPVersion supportedLDAPPolicies supportedSASLMechanisms dnsHostName ldapServiceName serverName supportedCapabilities Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=2 SEARCH RESULT tag=101 err=0 nentries=1 text= Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=3 SEARCH RESULT tag=101 err=2 nentries=0 text=domainScope control value not absent Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=3 do_search: get_ctrls failed ---- The behavior change is related to the fix of ITS#8840 : www.openldap.org/its/index.cgi/Software Bugs?id=8840 I did not see how to change the C# code, are we sure that the fix of ITS#8840 is valid?

1 0

RE: (ITS#8972) Performance/freezing issues using NTDLL on WindowsServer on Azure
by kriszyp＠gmail.com 05 Feb '19

05 Feb '19

--_22B17C00-48BC-46BE-A758-3013CC9A68A3_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Oh wow, this is just a matter of me using the wrong LMDB version? I had no = idea I wasn=E2=80=99t using the release version. I=E2=80=99ve been using ht= tps://github.com/Venemo/node-lmdb, which includes LMDB directly in it, and = had assumed that the version number in the code implied a release. I can pu= t together a PR for that project to get it on the right version. Thank you so much the pointer, I appreciate! And thank you so much for the = great software, its a fantastic package! Thanks, Kris From: Howard Chu Sent: February 5, 2019 12:03 PM To: kriszyp(a)gmail.com; openldap-its(a)OpenLDAP.org Subject: Re: (ITS#8972) Performance/freezing issues using NTDLL on WindowsS= erver on Azure kriszyp(a)gmail.com wrote: > Full_Name: Kristopher William Zyp > Version: LMDB 0.9.70 > OS: Windows Server 2012 R2 > URL:=20 > Submission from: (NULL) (71.199.6.148) I guess Windows users are never happy. Note that you're using the LMDB work-in-progress code. You ought to be usin= g the LMDB 0.9 release branch, which just uses Win32. The fact is that you get optimal performance by using LMDB as originally de= signed and documented - choose as large a mapsize as possible and then forget abou= t it. Preallocation is always more efficient than incremental growth. Clearly Windows will struggle to allocate disk space to grow files and addr= ess space to grow maps while there's competing system activity occurring, and t= his problem is worse when running under a hypervisor. This is the reason we nev= er merged the NTDLL code into the release branch; we knew from the outset that= it would destroy performance. https://openldap.org/lists/openldap-technical/201511/msg00107.html As a general rule, we're opposed to adding platform-specific API options to= LMDB. The API is supposed to be uniform across all supported platforms. If you wa= nt to submit a patch for compile-time selection of this feature, I guess that cou= ld work. As it is, LMDB is clearly not going to be a preinstalled Windows DLL = so it should be being built privately for every app that uses it anyway, so th= is shouldn't cause compatibility issues. > We have experienced significant performance issues in the form of strange= pauses > or freezing of any processes that are using our LMDB databases. These pau= ses are > periodic, often every few minutes, and typically completely freeze a proc= ess for > about 5 seconds, sometimes less, sometimes as much as a minute. They ofte= n occur > when the process is completely idle (we have a sleep timer to monitor for= them), > but we also see a lot of unusually large pauses on commits too. We have n= ever > been able to reproduce any of these issues with our local machines runnin= g > Windows 10, only on the virtual Azure Windows servers. >=20 > After weeks of investigation, we finally realized that all of these stran= ge > performance issues seemed to be due to the use of NTDLL for memory mappin= g. I > swapped out the NtCreateSection/NtMapViewOfSection calls for standard Win= 32 > CreateFileMapping/MapViewOfFile calls to create the memory map, and sudde= nly > these issues went away, and performance dramatically improved. Making thi= s > switch does mean you lose the progressive file allocation functionality o= f > NtCreateSection, but this is a small price to pay for the vast performanc= e > benefit of using standard file mapping. >=20 > Anyway, my request is to simply to provide an option to use CreateFileMap= ping > API instead of NtCreateSection. I recognize that the NTDLL based APIs may= be > more convenient for many users, but an option to use the standard APIs, e= ven if > it means the entire map is pre-allocated in the file size, is well worth = it for > the gains. I also recognize that Windows isn't the first choice of LMDB, = but > performance with these standard APIs is very good. >=20 > I'd be happy to work on putting together a branch/pull request to address= this. > However, your repository is a little different than typical github reposi= tories, > so I wanted to know if that was doable, and if there was a proper way to = submit > a PR, or if it needs to be submitted through this ticket system as a patc= h. http://www.openldap.org/devel/contributing.html --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ --_22B17C00-48BC-46BE-A758-3013CC9A68A3_ Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8" <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta ht= tp-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta name= =3DGenerator content=3D"Microsoft Word 15 (filtered medium)"><style></style></head><body lang=3DEN-CA link=3Dblue vlink=3D"#954F72"><div cla= ss=3DWordSection1>Oh wow, this is just a matter of me = using the wrong LMDB version? I had no idea I wasn=E2=80=99t using the rele= ase version. I=E2=80=99ve been using <a href=3D"https://github.com/Venemo/n= ode-lmdb">https://github.com/Venemo/node-lmdb</a>, which includes LMDB dire= ctly in it, and had assumed that the version number in the code implied a r= elease. I can put together a PR for that project to get it on the right ver= sion.Thank you so much the pointer, I appreciate! = And thank you so much for the great software, its a fantastic package!<= p class=3DMsoNormal><o:p> </o:p>Thanks, Kr= is<o:p> </o:p><div style=3D'mso-element:p= ara-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm= 0cm 0cm'>From: <= /b><a href=3D"mailto:hyc@symas.com">Howard Chu</a> Sent: February= 5, 2019 12:03 PM To: <a href=3D"mailto:kriszyp@gmail.com">kriszy= p(a)gmail.com</a>; <a href=3D"mailto:openldap-its@OpenLDAP.org">openldap-its@= OpenLDAP.org</a> Subject: Re: (ITS#8972) Performance/freezing iss= ues using NTDLL on WindowsServer on Azure</div><o:= p> </o:p>kriszyp(a)gmail.com wrote:> Full_Name: Kristopher William Zyp> Version: LMDB 0.9.70> OS: Windows Server = 2012 R2> URL: > Sub= mission from: (NULL) (71.199.6.148)<o:p> </o:= p>I guess Windows users are never happy.<o:p> </o:p>Note that you're u= sing the LMDB work-in-progress code. You ought to be using theLMDB 0.9 release branch, which just uses Win32.<o:p> </o:p>The fact is that you ge= t optimal performance by using LMDB as originally designedand documented - choose as large a mapsize as possible and then for= get about it.Preallocation is always more efficien= t than incremental growth.<o:p> </o:p>Clearly Windows will struggle to allocate disk space to g= row files and addressspace to grow maps while ther= e's competing system activity occurring, and thisp= roblem is worse when running under a hypervisor. This is the reason we neve= rmerged the NTDLL code into the release branch; we= knew from the outset that itwould destroy perform= ance.<o:p> </o:p>htt= ps://openldap.org/lists/openldap-technical/201511/msg00107.html<o:p> </o:p>As a general rule, w= e're opposed to adding platform-specific API options to LMDB.The API is supposed to be uniform across all supported platfor= ms. If you want tosubmit a patch for compile-time = selection of this feature, I guess that couldwork.= As it is, LMDB is clearly not going to be a preinstalled Windows DLL so</p= >it should be being built privately for every app that= uses it anyway, so thisshouldn't cause compatibil= ity issues.<o:p> </o:p>> We have experienced significant performance issues in the form of s= trange pauses> or freezing of any processes tha= t are using our LMDB databases. These pauses are&g= t; periodic, often every few minutes, and typically completely freeze a pro= cess for> about 5 seconds, sometimes less, some= times as much as a minute. They often occur> wh= en the process is completely idle (we have a sleep timer to monitor for the= m),> but we also see a lot of unusually large p= auses on commits too. We have never> been able = to reproduce any of these issues with our local machines running> Windows 10, only on the virtual Azure Windows servers.</= p>> > After weeks of in= vestigation, we finally realized that all of these strange> performance issues seemed to be due to the use of NTDLL for me= mory mapping. I> swapped out the NtCreateSectio= n/NtMapViewOfSection calls for standard Win32> = CreateFileMapping/MapViewOfFile calls to create the memory map, and suddenl= y> these issues went away, and performance dram= atically improved. Making this> switch does mea= n you lose the progressive file allocation functionality of> NtCreateSection, but this is a small price to pay for the vas= t performance> benefit of using standard file m= apping.> > Anyway, = my request is to simply to provide an option to use CreateFileMapping<p= class=3DMsoNormal>> API instead of NtCreateSection. I recognize that th= e NTDLL based APIs may be> more convenient for = many users, but an option to use the standard APIs, even if> it means the entire map is pre-allocated in the file size, is= well worth it for> the gains. I also recognize= that Windows isn't the first choice of LMDB, but&= gt; performance with these standard APIs is very good.> > I'd be happy to work on putting toge= ther a branch/pull request to address this.> Ho= wever, your repository is a little different than typical github repositori= es,> so I wanted to know if that was doable, an= d if there was a proper way to submit> a PR, or= if it needs to be submitted through this ticket system as a patch.<o:p> </o:p>http://www.openl= dap.org/devel/contributing.html<o:p> </o:p></= p>-- =C2=A0=C2=A0-- Howard Ch= u=C2=A0 CTO, Symas Corp.=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 http://www.symas.com=C2=A0 Director, Highland Sun=C2=A0=C2=A0=C2=A0=C2=A0 http://highlan= dsun.com/hyc/ =C2=A0Chief Architect, OpenLDAP=C2= =A0 http://www.openldap.org/project/<o:p> </o= :p></div></body></html>= --_22B17C00-48BC-46BE-A758-3013CC9A68A3_--

1 0

Re: (ITS#8972) Performance/freezing issues using NTDLL on Windows Server on Azure
by hyc＠symas.com 05 Feb '19

05 Feb '19

kriszyp(a)gmail.com wrote: > Full_Name: Kristopher William Zyp > Version: LMDB 0.9.70 > OS: Windows Server 2012 R2 > URL: > Submission from: (NULL) (71.199.6.148) I guess Windows users are never happy. Note that you're using the LMDB work-in-progress code. You ought to be using the LMDB 0.9 release branch, which just uses Win32. The fact is that you get optimal performance by using LMDB as originally designed and documented - choose as large a mapsize as possible and then forget about it. Preallocation is always more efficient than incremental growth. Clearly Windows will struggle to allocate disk space to grow files and address space to grow maps while there's competing system activity occurring, and this problem is worse when running under a hypervisor. This is the reason we never merged the NTDLL code into the release branch; we knew from the outset that it would destroy performance. https://openldap.org/lists/openldap-technical/201511/msg00107.html As a general rule, we're opposed to adding platform-specific API options to LMDB. The API is supposed to be uniform across all supported platforms. If you want to submit a patch for compile-time selection of this feature, I guess that could work. As it is, LMDB is clearly not going to be a preinstalled Windows DLL so it should be being built privately for every app that uses it anyway, so this shouldn't cause compatibility issues. > We have experienced significant performance issues in the form of strange pauses > or freezing of any processes that are using our LMDB databases. These pauses are > periodic, often every few minutes, and typically completely freeze a process for > about 5 seconds, sometimes less, sometimes as much as a minute. They often occur > when the process is completely idle (we have a sleep timer to monitor for them), > but we also see a lot of unusually large pauses on commits too. We have never > been able to reproduce any of these issues with our local machines running > Windows 10, only on the virtual Azure Windows servers. > > After weeks of investigation, we finally realized that all of these strange > performance issues seemed to be due to the use of NTDLL for memory mapping. I > swapped out the NtCreateSection/NtMapViewOfSection calls for standard Win32 > CreateFileMapping/MapViewOfFile calls to create the memory map, and suddenly > these issues went away, and performance dramatically improved. Making this > switch does mean you lose the progressive file allocation functionality of > NtCreateSection, but this is a small price to pay for the vast performance > benefit of using standard file mapping. > > Anyway, my request is to simply to provide an option to use CreateFileMapping > API instead of NtCreateSection. I recognize that the NTDLL based APIs may be > more convenient for many users, but an option to use the standard APIs, even if > it means the entire map is pre-allocated in the file size, is well worth it for > the gains. I also recognize that Windows isn't the first choice of LMDB, but > performance with these standard APIs is very good. > > I'd be happy to work on putting together a branch/pull request to address this. > However, your repository is a little different than typical github repositories, > so I wanted to know if that was doable, and if there was a proper way to submit > a PR, or if it needs to be submitted through this ticket system as a patch. http://www.openldap.org/devel/contributing.html -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8972) Performance/freezing issues using NTDLL on Windows Server on Azure
by kriszyp＠gmail.com 05 Feb '19

05 Feb '19

Full_Name: Kristopher William Zyp Version: LMDB 0.9.70 OS: Windows Server 2012 R2 URL: Submission from: (NULL) (71.199.6.148) We have experienced significant performance issues in the form of strange pauses or freezing of any processes that are using our LMDB databases. These pauses are periodic, often every few minutes, and typically completely freeze a process for about 5 seconds, sometimes less, sometimes as much as a minute. They often occur when the process is completely idle (we have a sleep timer to monitor for them), but we also see a lot of unusually large pauses on commits too. We have never been able to reproduce any of these issues with our local machines running Windows 10, only on the virtual Azure Windows servers. After weeks of investigation, we finally realized that all of these strange performance issues seemed to be due to the use of NTDLL for memory mapping. I swapped out the NtCreateSection/NtMapViewOfSection calls for standard Win32 CreateFileMapping/MapViewOfFile calls to create the memory map, and suddenly these issues went away, and performance dramatically improved. Making this switch does mean you lose the progressive file allocation functionality of NtCreateSection, but this is a small price to pay for the vast performance benefit of using standard file mapping. Anyway, my request is to simply to provide an option to use CreateFileMapping API instead of NtCreateSection. I recognize that the NTDLL based APIs may be more convenient for many users, but an option to use the standard APIs, even if it means the entire map is pre-allocated in the file size, is well worth it for the gains. I also recognize that Windows isn't the first choice of LMDB, but performance with these standard APIs is very good. I'd be happy to work on putting together a branch/pull request to address this. However, your repository is a little different than typical github repositories, so I wanted to know if that was doable, and if there was a proper way to submit a PR, or if it needs to be submitted through this ticket system as a patch.

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs