--_22B17C00-48BC-46BE-A758-3013CC9A68A3_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8"
Oh wow, this is just a matter of me using the wrong LMDB version? I had no = idea I wasn=E2=80=99t using the release version. I=E2=80=99ve been using ht= tps://github.com/Venemo/node-lmdb, which includes LMDB directly in it, and = had assumed that the version number in the code implied a release. I can pu= t together a PR for that project to get it on the right version. Thank you so much the pointer, I appreciate! And thank you so much for the = great software, its a fantastic package!
Thanks, Kris
From: Howard Chu Sent: February 5, 2019 12:03 PM To: kriszyp@gmail.com; openldap-its@OpenLDAP.org Subject: Re: (ITS#8972) Performance/freezing issues using NTDLL on WindowsS= erver on Azure
kriszyp@gmail.com wrote:
Full_Name: Kristopher William Zyp Version: LMDB 0.9.70 OS: Windows Server 2012 R2 URL:=20 Submission from: (NULL) (71.199.6.148)
I guess Windows users are never happy.
Note that you're using the LMDB work-in-progress code. You ought to be usin= g the LMDB 0.9 release branch, which just uses Win32.
The fact is that you get optimal performance by using LMDB as originally de= signed and documented - choose as large a mapsize as possible and then forget abou= t it. Preallocation is always more efficient than incremental growth.
Clearly Windows will struggle to allocate disk space to grow files and addr= ess space to grow maps while there's competing system activity occurring, and t= his problem is worse when running under a hypervisor. This is the reason we nev= er merged the NTDLL code into the release branch; we knew from the outset that= it would destroy performance.
https://openldap.org/lists/openldap-technical/201511/msg00107.html
As a general rule, we're opposed to adding platform-specific API options to= LMDB. The API is supposed to be uniform across all supported platforms. If you wa= nt to submit a patch for compile-time selection of this feature, I guess that cou= ld work. As it is, LMDB is clearly not going to be a preinstalled Windows DLL = so it should be being built privately for every app that uses it anyway, so th= is shouldn't cause compatibility issues.
We have experienced significant performance issues in the form of strange=
pauses
or freezing of any processes that are using our LMDB databases. These pau=
ses are
periodic, often every few minutes, and typically completely freeze a proc=
ess for
about 5 seconds, sometimes less, sometimes as much as a minute. They ofte=
n occur
when the process is completely idle (we have a sleep timer to monitor for=
them),
but we also see a lot of unusually large pauses on commits too. We have n=
ever
been able to reproduce any of these issues with our local machines runnin=
g
Windows 10, only on the virtual Azure Windows servers. =20 After weeks of investigation, we finally realized that all of these stran=
ge
performance issues seemed to be due to the use of NTDLL for memory mappin=
g. I
swapped out the NtCreateSection/NtMapViewOfSection calls for standard Win=
32
CreateFileMapping/MapViewOfFile calls to create the memory map, and sudde=
nly
these issues went away, and performance dramatically improved. Making thi=
s
switch does mean you lose the progressive file allocation functionality o=
f
NtCreateSection, but this is a small price to pay for the vast performanc=
e
benefit of using standard file mapping. =20 Anyway, my request is to simply to provide an option to use CreateFileMap=
ping
API instead of NtCreateSection. I recognize that the NTDLL based APIs may=
be
more convenient for many users, but an option to use the standard APIs, e=
ven if
it means the entire map is pre-allocated in the file size, is well worth =
it for
the gains. I also recognize that Windows isn't the first choice of LMDB, =
but
performance with these standard APIs is very good. =20 I'd be happy to work on putting together a branch/pull request to address=
this.
However, your repository is a little different than typical github reposi=
tories,
so I wanted to know if that was doable, and if there was a proper way to =
submit
a PR, or if it needs to be submitted through this ticket system as a patc=
h.
http://www.openldap.org/devel/contributing.html
--=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
--_22B17C00-48BC-46BE-A758-3013CC9A68A3_ Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8"
<html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40%22%3E<head><meta ht= tp-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta name= =3DGenerator content=3D"Microsoft Word 15 (filtered medium)"><style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 72.0pt 72.0pt 72.0pt;} div.WordSection1 {page:WordSection1;} --></style></head><body lang=3DEN-CA link=3Dblue vlink=3D"#954F72"><div cla= ss=3DWordSection1><p class=3DMsoNormal>Oh wow, this is just a matter of me = using the wrong LMDB version? I had no idea I wasn=E2=80=99t using the rele= ase version. I=E2=80=99ve been using <a href=3D"https://github.com/Venemo/n= ode-lmdb">https://github.com/Venemo/node-lmdb</a>, which includes LMDB dire= ctly in it, and had assumed that the version number in the code implied a r= elease. I can put together a PR for that project to get it on the right ver= sion.</p><p class=3DMsoNormal>Thank you so much the pointer, I appreciate! = And thank you so much for the great software, its a fantastic package!</p><= p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>Thanks,<br>Kr= is</p><p class=3DMsoNormal><o:p> </o:p></p><div style=3D'mso-element:p= ara-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm= 0cm 0cm'><p class=3DMsoNormal style=3D'border:none;padding:0cm'><b>From: <= /b><a href=3D"mailto:hyc@symas.com">Howard Chu</a><br><b>Sent: </b>February= 5, 2019 12:03 PM<br><b>To: </b><a href=3D"mailto:kriszyp@gmail.com">kriszy= p@gmail.com</a>; <a href=3D"mailto:openldap-its@OpenLDAP.org">openldap-its@= OpenLDAP.org</a><br><b>Subject: </b>Re: (ITS#8972) Performance/freezing iss= ues using NTDLL on WindowsServer on Azure</p></div><p class=3DMsoNormal><o:= p> </o:p></p><p class=3DMsoNormal>kriszyp@gmail.com wrote:</p><p class= =3DMsoNormal>> Full_Name: Kristopher William Zyp</p><p class=3DMsoNormal=
> Version: LMDB 0.9.70</p><p class=3DMsoNormal>> OS: Windows Server =
2012 R2</p><p class=3DMsoNormal>> URL: </p><p class=3DMsoNormal>> Sub= mission from: (NULL) (71.199.6.148)</p><p class=3DMsoNormal><o:p> </o:= p></p><p class=3DMsoNormal>I guess Windows users are never happy.</p><p cla= ss=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>Note that you're u= sing the LMDB work-in-progress code. You ought to be using the</p><p class= =3DMsoNormal>LMDB 0.9 release branch, which just uses Win32.</p><p class=3D= MsoNormal><o:p> </o:p></p><p class=3DMsoNormal>The fact is that you ge= t optimal performance by using LMDB as originally designed</p><p class=3DMs= oNormal>and documented - choose as large a mapsize as possible and then for= get about it.</p><p class=3DMsoNormal>Preallocation is always more efficien= t than incremental growth.</p><p class=3DMsoNormal><o:p> </o:p></p><p = class=3DMsoNormal>Clearly Windows will struggle to allocate disk space to g= row files and address</p><p class=3DMsoNormal>space to grow maps while ther= e's competing system activity occurring, and this</p><p class=3DMsoNormal>p= roblem is worse when running under a hypervisor. This is the reason we neve= r</p><p class=3DMsoNormal>merged the NTDLL code into the release branch; we= knew from the outset that it</p><p class=3DMsoNormal>would destroy perform= ance.</p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>htt= ps://openldap.org/lists/openldap-technical/201511/msg00107.html</p><p class= =3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>As a general rule, w= e're opposed to adding platform-specific API options to LMDB.</p><p class= =3DMsoNormal>The API is supposed to be uniform across all supported platfor= ms. If you want to</p><p class=3DMsoNormal>submit a patch for compile-time = selection of this feature, I guess that could</p><p class=3DMsoNormal>work.= As it is, LMDB is clearly not going to be a preinstalled Windows DLL so</p=
<p class=3DMsoNormal>it should be being built privately for every app that=
uses it anyway, so this</p><p class=3DMsoNormal>shouldn't cause compatibil= ity issues.</p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNorm= al>> We have experienced significant performance issues in the form of s= trange pauses</p><p class=3DMsoNormal>> or freezing of any processes tha= t are using our LMDB databases. These pauses are</p><p class=3DMsoNormal>&g= t; periodic, often every few minutes, and typically completely freeze a pro= cess for</p><p class=3DMsoNormal>> about 5 seconds, sometimes less, some= times as much as a minute. They often occur</p><p class=3DMsoNormal>> wh= en the process is completely idle (we have a sleep timer to monitor for the= m),</p><p class=3DMsoNormal>> but we also see a lot of unusually large p= auses on commits too. We have never</p><p class=3DMsoNormal>> been able = to reproduce any of these issues with our local machines running</p><p clas= s=3DMsoNormal>> Windows 10, only on the virtual Azure Windows servers.</= p><p class=3DMsoNormal>> </p><p class=3DMsoNormal>> After weeks of in= vestigation, we finally realized that all of these strange</p><p class=3DMs= oNormal>> performance issues seemed to be due to the use of NTDLL for me= mory mapping. I</p><p class=3DMsoNormal>> swapped out the NtCreateSectio= n/NtMapViewOfSection calls for standard Win32</p><p class=3DMsoNormal>> = CreateFileMapping/MapViewOfFile calls to create the memory map, and suddenl= y</p><p class=3DMsoNormal>> these issues went away, and performance dram= atically improved. Making this</p><p class=3DMsoNormal>> switch does mea= n you lose the progressive file allocation functionality of</p><p class=3DM= soNormal>> NtCreateSection, but this is a small price to pay for the vas= t performance</p><p class=3DMsoNormal>> benefit of using standard file m= apping.</p><p class=3DMsoNormal>> </p><p class=3DMsoNormal>> Anyway, = my request is to simply to provide an option to use CreateFileMapping</p><p= class=3DMsoNormal>> API instead of NtCreateSection. I recognize that th= e NTDLL based APIs may be</p><p class=3DMsoNormal>> more convenient for = many users, but an option to use the standard APIs, even if</p><p class=3DM= soNormal>> it means the entire map is pre-allocated in the file size, is= well worth it for</p><p class=3DMsoNormal>> the gains. I also recognize= that Windows isn't the first choice of LMDB, but</p><p class=3DMsoNormal>&= gt; performance with these standard APIs is very good.</p><p class=3DMsoNor= mal>> </p><p class=3DMsoNormal>> I'd be happy to work on putting toge= ther a branch/pull request to address this.</p><p class=3DMsoNormal>> Ho= wever, your repository is a little different than typical github repositori= es,</p><p class=3DMsoNormal>> so I wanted to know if that was doable, an= d if there was a proper way to submit</p><p class=3DMsoNormal>> a PR, or= if it needs to be submitted through this ticket system as a patch.</p><p c= lass=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>http://www.openl= dap.org/devel/contributing.html</p><p class=3DMsoNormal><o:p> </o:p></= p><p class=3DMsoNormal>-- </p><p class=3DMsoNormal>=C2=A0=C2=A0-- Howard Ch= u</p><p class=3DMsoNormal>=C2=A0 CTO, Symas Corp.=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 http://www.symas.com</p><p class=3DMso= Normal>=C2=A0 Director, Highland Sun=C2=A0=C2=A0=C2=A0=C2=A0 http://highlan= dsun.com/hyc/</p><p class=3DMsoNormal> =C2=A0Chief Architect, OpenLDAP=C2= =A0 http://www.openldap.org/project/</p><p class=3DMsoNormal><o:p> </o= :p></p></div></body></html>=
--_22B17C00-48BC-46BE-A758-3013CC9A68A3_--