openldap-bugs

openldap-bugs@openldap.org

1 participants
20967 discussions

Re: (ITS#8975) Calling mdb_env_set_mapsize on a shared DB fails sometimes
by hyc＠symas.com 10 Feb '19

10 Feb '19

kriszyp(a)gmail.com wrote: > Full_Name: Kristopher William Zyp > Version: LMDB 0.9.23 > OS: Windows Server 2012 R2, Windows 10 > URL: https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45bd32… > Submission from: (NULL) (71.199.6.148) > Thanks for the report and patch, added to mdb.RE/0.9 > Calling mdb_env_set_mapsize to increase the map size, when a DB is also in use > by other processes, when MDB_WRITEMAP is enabled (and the db file has been > opened with PAGE_READWRITE access), on Windows, will occasionally (seems like > about 1/100 attempts fails) produce an error "The requested operation cannot be > performed on a file with a user-mapped section open", or segfaults. The error > occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map that is > performed to increase the allocated file size to the map size, prior to > CreateFileMapping. > > As it turns out this is pretty easy to solve, because manually expanding the > file size is not necessary when calling CreateFileMapping with PAGE_READWRITE > access, as Windows will automatically expand the file size for us, when opened > with the page write access enabled. Of course, this means all processes must be > consistent in use of MDB_WRITEMAP, but the documentation already makes this > explicit and clear. > > I believe this can be fixed by simply adding a check for MDB_WRITEMAP in the if > statement that calls SetFilePointer: > > if (!(flags & MDB_WRITEMAP) && (SetFilePointer(env->me_fd, sizelo, &sizehi, 0) > != (DWORD)sizelo > || !SetEndOfFile(env->me_fd) > || SetFilePointer(env->me_fd, 0, NULL, 0) != 0)) > return ErrCode(); > > The attached URL has the change as a patch/diff as applied to our node package. > > I am certainly happy to just keep this change on our own branches. There may be > nuances of this that I might not be aware of, but it seems to be working great > for us and I have tested this with MDB_WRITEMAP enabled and disabled. So I > thought I would offer/suggest this change, as it seems like it is > straightforward change to improve stability. Thank you! > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8975) Calling mdb_env_set_mapsize on a shared DB fails sometimes
by kriszyp＠gmail.com 08 Feb '19

08 Feb '19

Full_Name: Kristopher William Zyp Version: LMDB 0.9.23 OS: Windows Server 2012 R2, Windows 10 URL: https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45bd32… Submission from: (NULL) (71.199.6.148) Calling mdb_env_set_mapsize to increase the map size, when a DB is also in use by other processes, when MDB_WRITEMAP is enabled (and the db file has been opened with PAGE_READWRITE access), on Windows, will occasionally (seems like about 1/100 attempts fails) produce an error "The requested operation cannot be performed on a file with a user-mapped section open", or segfaults. The error occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map that is performed to increase the allocated file size to the map size, prior to CreateFileMapping. As it turns out this is pretty easy to solve, because manually expanding the file size is not necessary when calling CreateFileMapping with PAGE_READWRITE access, as Windows will automatically expand the file size for us, when opened with the page write access enabled. Of course, this means all processes must be consistent in use of MDB_WRITEMAP, but the documentation already makes this explicit and clear. I believe this can be fixed by simply adding a check for MDB_WRITEMAP in the if statement that calls SetFilePointer: if (!(flags & MDB_WRITEMAP) && (SetFilePointer(env->me_fd, sizelo, &sizehi, 0) != (DWORD)sizelo || !SetEndOfFile(env->me_fd) || SetFilePointer(env->me_fd, 0, NULL, 0) != 0)) return ErrCode(); The attached URL has the change as a patch/diff as applied to our node package. I am certainly happy to just keep this change on our own branches. There may be nuances of this that I might not be aware of, but it seems to be working great for us and I have tested this with MDB_WRITEMAP enabled and disabled. So I thought I would offer/suggest this change, as it seems like it is straightforward change to improve stability. Thank you!

1 0

RE: (ITS#8972) Performance/freezing issues using NTDLL onWindowsServer on Azure
by kriszyp＠gmail.com 08 Feb '19

08 Feb '19

--_FEED8D1F-4CE2-4DE8-A6B1-1703AFFCC4FC_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To follow up to this, after switching to the stable release in node-ldmb, a= nd running on our servers for several days, I can confirm this definitely d= oes fix our issues (and ticket can be closed). I=E2=80=99ve submitted a PR = for getting the correct stable version in node-lmdb, to ensure in this does= n=E2=80=99t happen for other NodeJS users: https://github.com/Venemo/node-l= mdb/pull/141=20 I did find a minor issue with setting up the file mapping in the release ve= rsion, but I will file that separately. And if the NTDLL mapping isn=E2=80= =99t intended to be merged into a release branch, sounds like there shouldn= =E2=80=99t be any issues for Windows users that are using LMDB releases. And to be clear, I am a happy windows user, sorting out versions hardly den= ts my enthusiasm for LMDB. And thank you again for the help. Thanks, Kris --_FEED8D1F-4CE2-4DE8-A6B1-1703AFFCC4FC_ Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8" <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta ht= tp-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta name= =3DGenerator content=3D"Microsoft Word 15 (filtered medium)"><style></style></head><body lang=3DEN-CA link=3Dblue vlink=3D"#954F72"><div cla= ss=3DWordSection1>To follow up to this, after switchin= g to the stable release in node-ldmb, and running on our servers for severa= l days, I can confirm this definitely does fix our issues (and ticket can b= e closed). I=E2=80=99ve submitted a PR for getting the correct stable versi= on in node-lmdb, to ensure in this doesn=E2=80=99t happen for other NodeJS = users: <a href=3D"https://github.com/Venemo/node-lmdb/pull/141">https://git= hub.com/Venemo/node-lmdb/pull/141</a> <o:p> <= /o:p>I did find a minor issue with setting up the = file mapping in the release version, but I will file that separately. And i= f the NTDLL mapping isn=E2=80=99t intended to be merged into a release bran= ch, sounds like there shouldn=E2=80=99t be any issues for Windows users tha= t are using LMDB releases.<o:p> </o:p>And to be clear, I am a happy windows user, sorting out v= ersions hardly dents my enthusiasm for LMDB. And thank you again for the he= lp.<o:p></o:p><o:p> </o:p>Thanks, Kris<o:p></o:p></div></body></html>= --_FEED8D1F-4CE2-4DE8-A6B1-1703AFFCC4FC_--

1 0

Re: (ITS#8974) olcAttributeOptions cannot be modified once set
by hyc＠symas.com 07 Feb '19

07 Feb '19

skoranda(a)gmail.com wrote: > Full_Name: Scott Koranda > Version: 2.4.47 > OS: Debian Linux 9.6 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (104.231.255.140) > > > Once the option olcAttributeOptions has been configured it cannot be modified. New options can be added; existing options cannot be deleted. Feel free to submit a patch adding the required support. Note the FIXME comment in slapd/bconfig.c. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8974) olcAttributeOptions cannot be modified once set
by skoranda＠gmail.com 07 Feb '19

07 Feb '19

Full_Name: Scott Koranda Version: 2.4.47 OS: Debian Linux 9.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (104.231.255.140) Once the option olcAttributeOptions has been configured it cannot be modified. To reproduce begin with the following state: #ldapsearch -LLL -Y EXTERNAL -H ldapi:/// -b cn=config -s base olcAttributeOptions 2>/dev/null dn: cn=config olcAttributeOptions: lang- app- Create an LDIF file to modify the value for olcAttributeOptions: # cat /root/olcAttributeOptions.ldif dn: cn=config changetype: modify replace: olcAttributeOptions olcAttributeOptions: lang- app- internal prior role- scope- time- Use ldapmodify to attempt to modify the configuration: # ldapmodify -Y EXTERNAL -H ldapi:/// -f /root/olcAttributeOptions.ldif SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth SASL SSF: 0 modifying entry "cn=config" ldap_modify: Other (e.g., implementation specific) error (80) The slapd process running with -d -1 emits this output: 5c5c2e01 slapd: line 0: option "lang-" is already defined 5c5c2e01 olcAttributeOptions: value #0: <olcAttributeOptions> handler exited with 1! This behavior happens regardless of whether any directory has records that include options with attributes.

1 0

Re: (ITS#8973) Error with domainScope control when using C# LDAP API
by clement.oudot＠worteks.com 06 Feb '19

06 Feb '19

Le 06/02/2019 à 12:57, hyc(a)symas.com a écrit : > hyc(a)symas.com wrote: >> clement.oudot(a)worteks.com wrote: >>> Full_Name: Clement OUDOT >>> Version: 2.4.47 >>> OS: GNU/Linux >>> URL: ftp://ftp.openldap.org/incoming/ >>> Submission from: (NULL) (81.250.130.213) >>> >>> >>> Hello, >>> >>> after upgrading from 2.4.46 to 2.4.47, this C# code does not work anymore : >> The code in controls.c is correct. The DomainScope control does not take a value, >> so the value is required to be NULL. > Microsoft spec https://msdn.microsoft.com/en-us/library/cc223348.aspx > > "When sending this control to the DC, the controlValue field of the Control structure is omitted." > Thanks a lot for your feedback, I will try to understand with the C# code is sending a value inside the control. -- Clément Oudot | Identity Solutions Manager clement.oudot(a)worteks.com Worteks | https://www.worteks.com

1 0

Re: (ITS#8973) Error with domainScope control when using C# LDAP API
by hyc＠symas.com 06 Feb '19

06 Feb '19

hyc(a)symas.com wrote: > clement.oudot(a)worteks.com wrote: >> Full_Name: Clement OUDOT >> Version: 2.4.47 >> OS: GNU/Linux >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (81.250.130.213) >> >> >> Hello, >> >> after upgrading from 2.4.46 to 2.4.47, this C# code does not work anymore : > > The code in controls.c is correct. The DomainScope control does not take a value, > so the value is required to be NULL. Microsoft spec https://msdn.microsoft.com/en-us/library/cc223348.aspx "When sending this control to the DC, the controlValue field of the Control structure is omitted." -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8973) Error with domainScope control when using C# LDAP API
by hyc＠symas.com 06 Feb '19

06 Feb '19

clement.oudot(a)worteks.com wrote: > Full_Name: Clement OUDOT > Version: 2.4.47 > OS: GNU/Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (81.250.130.213) > > > Hello, > > after upgrading from 2.4.46 to 2.4.47, this C# code does not work anymore : The code in controls.c is correct. The DomainScope control does not take a value, so the value is required to be NULL. > ---- > $auth = [System.DirectoryServices.AuthenticationTypes]::FastBind > $de = New-Object System.DirectoryServices.DirectoryEntry($DN,$userReferentiel, > $passwordReferentiel, $auth) > ---- > > The log in OpenLDAP 2.4.47 : > ---- > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 fd=17 ACCEPT from > IP=10.10.2.163:47304 (IP=0.0.0.0:390) > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=0 BIND > dn="uid=ADPassword,ou=configuration,dc=referentiel,dc=local" method=128 > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=0 BIND > dn="uid=ADPassword,ou=configuration,dc=referentiel,dc=local" mech=SIMPLE ssf=0 > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=0 RESULT tag=97 err=0 text= > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=1 SRCH > base="dc=referentiel,dc=local" scope=0 deref=0 filter="(objectClass=*)" > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=1 SRCH attr=objectClass > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=1 SEARCH RESULT tag=101 err=0 > nentries=1 text= > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=2 SRCH base="" scope=0 > deref=0 filter="(objectClass=*)" > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=2 SRCH attr=subschemaSubentry > dsServiceName namingContexts defaultNamingContext schemaNamingContext > configurationNamingContext rootDomainNamingContext supportedControl > supportedLDAPVersion supportedLDAPPolicies supportedSASLMechanisms dnsHostName > ldapServiceName serverName supportedCapabilities > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=2 SEARCH RESULT tag=101 err=0 > nentries=1 text= > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=3 SEARCH RESULT tag=101 err=2 > nentries=0 text=domainScope control value not absent > Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=3 do_search: get_ctrls > failed > ---- > > The behavior change is related to the fix of ITS#8840 : > www.openldap.org/its/index.cgi/Software Bugs?id=8840 > > I did not see how to change the C# code, are we sure that the fix of ITS#8840 is > valid? > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8973) Error with domainScope control when using C# LDAP API
by clement.oudot＠worteks.com 06 Feb '19

06 Feb '19

Full_Name: Clement OUDOT Version: 2.4.47 OS: GNU/Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (81.250.130.213) Hello, after upgrading from 2.4.46 to 2.4.47, this C# code does not work anymore : ---- $auth = [System.DirectoryServices.AuthenticationTypes]::FastBind $de = New-Object System.DirectoryServices.DirectoryEntry($DN,$userReferentiel, $passwordReferentiel, $auth) ---- The log in OpenLDAP 2.4.47 : ---- Feb 5 18:21:03 anteros slapd[30762]: conn=1577 fd=17 ACCEPT from IP=10.10.2.163:47304 (IP=0.0.0.0:390) Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=0 BIND dn="uid=ADPassword,ou=configuration,dc=referentiel,dc=local" method=128 Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=0 BIND dn="uid=ADPassword,ou=configuration,dc=referentiel,dc=local" mech=SIMPLE ssf=0 Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=0 RESULT tag=97 err=0 text= Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=1 SRCH base="dc=referentiel,dc=local" scope=0 deref=0 filter="(objectClass=*)" Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=1 SRCH attr=objectClass Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=1 SEARCH RESULT tag=101 err=0 nentries=1 text= Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=2 SRCH base="" scope=0 deref=0 filter="(objectClass=*)" Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=2 SRCH attr=subschemaSubentry dsServiceName namingContexts defaultNamingContext schemaNamingContext configurationNamingContext rootDomainNamingContext supportedControl supportedLDAPVersion supportedLDAPPolicies supportedSASLMechanisms dnsHostName ldapServiceName serverName supportedCapabilities Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=2 SEARCH RESULT tag=101 err=0 nentries=1 text= Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=3 SEARCH RESULT tag=101 err=2 nentries=0 text=domainScope control value not absent Feb 5 18:21:03 anteros slapd[30762]: conn=1577 op=3 do_search: get_ctrls failed ---- The behavior change is related to the fix of ITS#8840 : www.openldap.org/its/index.cgi/Software Bugs?id=8840 I did not see how to change the C# code, are we sure that the fix of ITS#8840 is valid?

1 0

RE: (ITS#8972) Performance/freezing issues using NTDLL on WindowsServer on Azure
by kriszyp＠gmail.com 05 Feb '19

05 Feb '19

--_22B17C00-48BC-46BE-A758-3013CC9A68A3_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Oh wow, this is just a matter of me using the wrong LMDB version? I had no = idea I wasn=E2=80=99t using the release version. I=E2=80=99ve been using ht= tps://github.com/Venemo/node-lmdb, which includes LMDB directly in it, and = had assumed that the version number in the code implied a release. I can pu= t together a PR for that project to get it on the right version. Thank you so much the pointer, I appreciate! And thank you so much for the = great software, its a fantastic package! Thanks, Kris From: Howard Chu Sent: February 5, 2019 12:03 PM To: kriszyp(a)gmail.com; openldap-its(a)OpenLDAP.org Subject: Re: (ITS#8972) Performance/freezing issues using NTDLL on WindowsS= erver on Azure kriszyp(a)gmail.com wrote: > Full_Name: Kristopher William Zyp > Version: LMDB 0.9.70 > OS: Windows Server 2012 R2 > URL:=20 > Submission from: (NULL) (71.199.6.148) I guess Windows users are never happy. Note that you're using the LMDB work-in-progress code. You ought to be usin= g the LMDB 0.9 release branch, which just uses Win32. The fact is that you get optimal performance by using LMDB as originally de= signed and documented - choose as large a mapsize as possible and then forget abou= t it. Preallocation is always more efficient than incremental growth. Clearly Windows will struggle to allocate disk space to grow files and addr= ess space to grow maps while there's competing system activity occurring, and t= his problem is worse when running under a hypervisor. This is the reason we nev= er merged the NTDLL code into the release branch; we knew from the outset that= it would destroy performance. https://openldap.org/lists/openldap-technical/201511/msg00107.html As a general rule, we're opposed to adding platform-specific API options to= LMDB. The API is supposed to be uniform across all supported platforms. If you wa= nt to submit a patch for compile-time selection of this feature, I guess that cou= ld work. As it is, LMDB is clearly not going to be a preinstalled Windows DLL = so it should be being built privately for every app that uses it anyway, so th= is shouldn't cause compatibility issues. > We have experienced significant performance issues in the form of strange= pauses > or freezing of any processes that are using our LMDB databases. These pau= ses are > periodic, often every few minutes, and typically completely freeze a proc= ess for > about 5 seconds, sometimes less, sometimes as much as a minute. They ofte= n occur > when the process is completely idle (we have a sleep timer to monitor for= them), > but we also see a lot of unusually large pauses on commits too. We have n= ever > been able to reproduce any of these issues with our local machines runnin= g > Windows 10, only on the virtual Azure Windows servers. >=20 > After weeks of investigation, we finally realized that all of these stran= ge > performance issues seemed to be due to the use of NTDLL for memory mappin= g. I > swapped out the NtCreateSection/NtMapViewOfSection calls for standard Win= 32 > CreateFileMapping/MapViewOfFile calls to create the memory map, and sudde= nly > these issues went away, and performance dramatically improved. Making thi= s > switch does mean you lose the progressive file allocation functionality o= f > NtCreateSection, but this is a small price to pay for the vast performanc= e > benefit of using standard file mapping. >=20 > Anyway, my request is to simply to provide an option to use CreateFileMap= ping > API instead of NtCreateSection. I recognize that the NTDLL based APIs may= be > more convenient for many users, but an option to use the standard APIs, e= ven if > it means the entire map is pre-allocated in the file size, is well worth = it for > the gains. I also recognize that Windows isn't the first choice of LMDB, = but > performance with these standard APIs is very good. >=20 > I'd be happy to work on putting together a branch/pull request to address= this. > However, your repository is a little different than typical github reposi= tories, > so I wanted to know if that was doable, and if there was a proper way to = submit > a PR, or if it needs to be submitted through this ticket system as a patc= h. http://www.openldap.org/devel/contributing.html --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ --_22B17C00-48BC-46BE-A758-3013CC9A68A3_ Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8" <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta ht= tp-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta name= =3DGenerator content=3D"Microsoft Word 15 (filtered medium)"><style></style></head><body lang=3DEN-CA link=3Dblue vlink=3D"#954F72"><div cla= ss=3DWordSection1>Oh wow, this is just a matter of me = using the wrong LMDB version? I had no idea I wasn=E2=80=99t using the rele= ase version. I=E2=80=99ve been using <a href=3D"https://github.com/Venemo/n= ode-lmdb">https://github.com/Venemo/node-lmdb</a>, which includes LMDB dire= ctly in it, and had assumed that the version number in the code implied a r= elease. I can put together a PR for that project to get it on the right ver= sion.Thank you so much the pointer, I appreciate! = And thank you so much for the great software, its a fantastic package!<= p class=3DMsoNormal><o:p> </o:p>Thanks, Kr= is<o:p> </o:p><div style=3D'mso-element:p= ara-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm= 0cm 0cm'>From: <= /b><a href=3D"mailto:hyc@symas.com">Howard Chu</a> Sent: February= 5, 2019 12:03 PM To: <a href=3D"mailto:kriszyp@gmail.com">kriszy= p(a)gmail.com</a>; <a href=3D"mailto:openldap-its@OpenLDAP.org">openldap-its@= OpenLDAP.org</a> Subject: Re: (ITS#8972) Performance/freezing iss= ues using NTDLL on WindowsServer on Azure</div><o:= p> </o:p>kriszyp(a)gmail.com wrote:> Full_Name: Kristopher William Zyp> Version: LMDB 0.9.70> OS: Windows Server = 2012 R2> URL: > Sub= mission from: (NULL) (71.199.6.148)<o:p> </o:= p>I guess Windows users are never happy.<o:p> </o:p>Note that you're u= sing the LMDB work-in-progress code. You ought to be using theLMDB 0.9 release branch, which just uses Win32.<o:p> </o:p>The fact is that you ge= t optimal performance by using LMDB as originally designedand documented - choose as large a mapsize as possible and then for= get about it.Preallocation is always more efficien= t than incremental growth.<o:p> </o:p>Clearly Windows will struggle to allocate disk space to g= row files and addressspace to grow maps while ther= e's competing system activity occurring, and thisp= roblem is worse when running under a hypervisor. This is the reason we neve= rmerged the NTDLL code into the release branch; we= knew from the outset that itwould destroy perform= ance.<o:p> </o:p>htt= ps://openldap.org/lists/openldap-technical/201511/msg00107.html<o:p> </o:p>As a general rule, w= e're opposed to adding platform-specific API options to LMDB.The API is supposed to be uniform across all supported platfor= ms. If you want tosubmit a patch for compile-time = selection of this feature, I guess that couldwork.= As it is, LMDB is clearly not going to be a preinstalled Windows DLL so</p= >it should be being built privately for every app that= uses it anyway, so thisshouldn't cause compatibil= ity issues.<o:p> </o:p>> We have experienced significant performance issues in the form of s= trange pauses> or freezing of any processes tha= t are using our LMDB databases. These pauses are&g= t; periodic, often every few minutes, and typically completely freeze a pro= cess for> about 5 seconds, sometimes less, some= times as much as a minute. They often occur> wh= en the process is completely idle (we have a sleep timer to monitor for the= m),> but we also see a lot of unusually large p= auses on commits too. We have never> been able = to reproduce any of these issues with our local machines running> Windows 10, only on the virtual Azure Windows servers.</= p>> > After weeks of in= vestigation, we finally realized that all of these strange> performance issues seemed to be due to the use of NTDLL for me= mory mapping. I> swapped out the NtCreateSectio= n/NtMapViewOfSection calls for standard Win32> = CreateFileMapping/MapViewOfFile calls to create the memory map, and suddenl= y> these issues went away, and performance dram= atically improved. Making this> switch does mea= n you lose the progressive file allocation functionality of> NtCreateSection, but this is a small price to pay for the vas= t performance> benefit of using standard file m= apping.> > Anyway, = my request is to simply to provide an option to use CreateFileMapping<p= class=3DMsoNormal>> API instead of NtCreateSection. I recognize that th= e NTDLL based APIs may be> more convenient for = many users, but an option to use the standard APIs, even if> it means the entire map is pre-allocated in the file size, is= well worth it for> the gains. I also recognize= that Windows isn't the first choice of LMDB, but&= gt; performance with these standard APIs is very good.> > I'd be happy to work on putting toge= ther a branch/pull request to address this.> Ho= wever, your repository is a little different than typical github repositori= es,> so I wanted to know if that was doable, an= d if there was a proper way to submit> a PR, or= if it needs to be submitted through this ticket system as a patch.<o:p> </o:p>http://www.openl= dap.org/devel/contributing.html<o:p> </o:p></= p>-- =C2=A0=C2=A0-- Howard Ch= u=C2=A0 CTO, Symas Corp.=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 http://www.symas.com=C2=A0 Director, Highland Sun=C2=A0=C2=A0=C2=A0=C2=A0 http://highlan= dsun.com/hyc/ =C2=A0Chief Architect, OpenLDAP=C2= =A0 http://www.openldap.org/project/<o:p> </o= :p></div></body></html>= --_22B17C00-48BC-46BE-A758-3013CC9A68A3_--

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs