--000000000000bdb86d059deaf350 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Sorry to keep pestering, but just pinging about this patch again, as I still think this fix could benefit windows users. And at this point, I think I can say we have tested it pretty well, running on our servers for almost a year :). Thanks, Kris
On Wed, Sep 18, 2019 at 12:56 PM Kris Zyp kriszyp@gmail.com wrote:
Checking on this again, is this still a possibility for merging into LMDB=
?
This fix is still working great (improved performance) on our systems. Thanks, Kris
On Mon, Jun 17, 2019 at 1:04 PM Kris Zyp kriszyp@gmail.com wrote:
Is this still being considered/reviewed? Let me know if there are any other changes you would like me to make. This patch has continued to yie=
ld
significant and reliable performance improvements for us, and seems like=
it
would be nice for this to be available for other Windows users.
On Fri, May 3, 2019 at 3:52 PM Kris Zyp kriszyp@gmail.com wrote:
For the sake of putting this in the email thread (other code discussion in GitHub), here is the latest squashed commit of the proposed patch (w=
ith
the on-demand, retained overlapped array to reduce re-malloc and openin=
g
event handles): https://github.com/kriszyp/node-lmdb/commit/726a9156662c703bf3d453aab75=
ee222072b990f
Thanks, Kris
*From: *Kris Zyp kriszyp@gmail.com *Sent: *April 30, 2019 12:43 PM *To: *Howard Chu hyc@symas.com; openldap-its@OpenLDAP.org *Subject: *RE: (ITS#9017) Improving performance of commit sync in Windows
What is the point of using writemap mode if you still need to use
WriteFile
on every individual page?
As I understood from the documentation, and have observed, using writemap mode is faster (and uses less temporary memory) because it doe=
sn=E2=80=99t
require mallocs to allocate pages (docs: =E2=80=9CThis is faster and us=
es fewer
mallocs=E2=80=9D). To be clear though, LMDB is so incredibly fast and e=
fficient,
that in sync-mode, it takes enormous transactions before the time spent allocating and creating the dirty pages with the updated b-tree is anyw=
here
even remotely close to the time it takes to wait for disk flushing, eve=
n
with an SSD. But the more pertinent question is efficiency, and measuri=
ng
CPU cycles rather than time spent (efficiency is more important than ju=
st
time spent). When I ran my tests this morning of 100 (sync) transaction=
s
with 100 puts per transaction, times varied quite a bit, but it seemed =
like
running with writemap enabled typically averages about 500ms of CPU and with writemap disabled it typically averages around 600ms. Not a huge difference, but still definitely worthwhile, I think.
Caveat emptor: Measuring LMDB performance with sync interactions on Windows is one of the most frustratingly erratic things to measure. It =
is
sunny outside right now, times could be different when it starts rainin=
g
later, but, this is what I saw this morning...
What is the performance difference between your patch using writemap,
and just
not using writemap in the first place?
Running 1000 sync transactions on 3GB db with a single put per transaction, without writemap map, without the patch took about 60 seco=
nds.
And it took about 1 second with the patch with writemap mode enabled! (there is no significant difference in sync times with writemap enabled=
or
disabled with the patch.) So the difference was huge in my test. And no=
t
only that, without the patch, the CPU usage was actually _*higher*_ during that 60 seconds (close to 100% of a core) than during the execut=
ion
with the patch for one second (close to 50%). Anyway, there are certai=
nly
tests I have run where the differences are not as large (doing small commits on large dbs accentuates the differences), but the patch always seems to win. It could also be that my particular configuration causes bigger differences (on an SSD drive, and maybe a more fragmented file?)=
.
Anyway, I added error handling for the malloc, and fixed/changed the other things you suggested. Be happy to make any other changes you want=
.
The updated patch is here:
https://github.com/kriszyp/node-lmdb/commit/25366dea9453749cf6637f43ec1=
7b9b62094acde
OVERLAPPED* ov =3D malloc((pagecount - keep) * sizeof(OVERLAPPED));
Probably this ought to just be pre-allocated based on the maximum
number of dirty pages a txn allows.
I wasn=E2=80=99t sure I understood this comment. Are you suggesting we =
malloc(MDB_IDL_UM_MAX
- sizeof(OVERLAPPED)) for each environment, and retain it for the life =
of
the environment? I think that is 4MB, if my math is right, which seems =
like
a lot of memory to keep allocated (we usually have a lot of open environments). If the goal is to reduce the number of mallocs, how abou=
t we
retain the OVERLAPPED array, and only free and re-malloc if the previou=
s
allocation wasn=E2=80=99t large enough? Then there isn=E2=80=99t unnece=
ssary allocation,
and we only malloc when there is a bigger transaction than any previous=
. I
put this together in a separate commit, as I wasn=E2=80=99t sure if thi=
s what you
wanted (can squash if you prefer): https://github.com/kriszyp/node-lmdb/commit/2fe68fb5269c843e2e789746a17=
a4b2adefaac40
Thank you for the review!
Thanks, Kris
*From: *Howard Chu hyc@symas.com *Sent: *April 30, 2019 7:12 AM *To: *kriszyp@gmail.com; openldap-its@OpenLDAP.org *Subject: *Re: (ITS#9017) Improving performance of commit sync in Windows
kriszyp@gmail.com wrote:
Full_Name: Kristopher William Zyp
Version: LMDB 0.9.23
OS: Windows
URL:
https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0a=
b9332b7fc4ce9
Submission from: (NULL) (71.199.6.148)
We have seen very poor performance on the sync of commits on large
databases in
Windows. On databases with 2GB of data, in writemap mode, the sync of
even small
commits is consistently well over 100ms (without writemap it is
faster, but
still slow). It is expected that a sync should take some time while
waiting for
disk confirmation of the writes, but more concerning is that these sy=
nc
operations (in writemap mode) are instead dominated by nearly 100%
system CPU
utilization, so operations that requires sub-millisecond b-tree updat=
e
operations are then dominated by very large amounts of system CPU
cycles during
the sync phase.
I think that the fundamental problem is that FlushViewOfFile seems to
be an O(n)
operation where n is the size of the file (or map). I presume that
Windows is
scanning the entire map/file for dirty pages to flush, I'm guessing
because it
doesn't have an internal index of all the dirty pages for every
file/map-view in
the OS disk cache. Therefore, the turns into an extremely expensive,
CPU-bound
operation to find the dirty pages for (large file) and initiate their
writes,
which, of course, is contrary to the whole goal of a scalable databas=
e
system.
And FlushFileBuffers is also relatively slow as well. We have
attempted to batch
as many operations into single transaction as possible, but this is
still a very
large overhead.
The Windows docs for FlushFileBuffers itself warns about the
inefficiencies of
this function (
https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi=
-flushfilebuffers
).
Which also points to the solution: it is much faster to write out the
dirty
pages with WriteFile through a sync file handle
(FILE_FLAG_WRITE_THROUGH).
The associated patch
(
https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0a=
b9332b7fc4ce9
)
is my attempt at implementing this solution, for Windows. Fortunately=
,
with the
design of LMDB, this is relatively straightforward. LMDB already
supports
writing out dirty pages with WriteFile calls. I added a write-through
handle for
sending these writes directly to disk. I then made that file-handle
overlapped/asynchronously, so all the writes for a commit could be
started in
overlap mode, and (at least theoretically) transfer in parallel to th=
e
drive and
then used GetOverlappedResult to wait for the completion. So basicall=
y
mdb_page_flush becomes the sync. I extended the writing of dirty page=
s
through
WriteFile to writemap mode as well (for writing meta too), so that
WriteFile
with write-through can be used to flush the data without ever needing
to call
FlushViewOfFile or FlushFileBuffers. I also implemented support for
write
gathering in writemap mode where contiguous file positions infers
contiguous
memory (by tracking the starting position with wdp and writing
contiguous pages
in single operations). Sorting of the dirty list is maintained even i=
n
writemap
mode for this purpose.
What is the point of using writemap mode if you still need to use WriteFile
on every individual page?
The performance benefits of this patch, in my testing, are
considerable. Writing
out/syncing transactions is typically over 5x faster in writemap mode=
,
and 2x
faster in standard mode. And perhaps more importantly (especially in
environment
with many threads/processes), the efficiency benefits are even larger=
,
particularly in writemap mode, where there can be a 50-100x reduction
in the
system CPU usage by using this patch. This brings windows performance
with
sync'ed transactions in LMDB back into the range of "lightning"
performance :).
What is the performance difference between your patch using writemap, and just
not using writemap in the first place?
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
--000000000000bdb86d059deaf350 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Sorry to keep pestering, but just pinging about this patch= again, as I still think this fix could benefit windows users. And at this = point, I think I can say we have tested it pretty well, running on our serv= ers for almost a year :).<div>Thanks,</div><div>Kris</div></div><br><div cl= ass=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Sep 18, 2= 019 at 12:56 PM Kris Zyp <<a href=3D"mailto:kriszyp@gmail.com">kriszyp@g= mail.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D= "margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-le= ft:1ex"><div dir=3D"ltr">Checking on this again, is this still a possibilit= y for merging into LMDB? This fix is still working great (improved performa= nce) on our systems.<div>Thanks,</div><div>Kris</div></div><br><div class= =3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Jun 17, 2019= at 1:04 PM Kris Zyp <<a href=3D"mailto:kriszyp@gmail.com" target=3D"_bl= ank">kriszyp@gmail.com</a>> wrote:<br></div><blockquote class=3D"gmail_q= uote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,2= 04);padding-left:1ex"><div dir=3D"ltr">Is this still being considered/revie= wed? Let me know if there are any other changes you would like me to make. = This patch has continued to yield significant and reliable performance impr= ovements for us, and seems like it would be nice for this to be available f= or other Windows users.</div><br><div class=3D"gmail_quote"><div dir=3D"ltr= " class=3D"gmail_attr">On Fri, May 3, 2019 at 3:52 PM Kris Zyp <<a href= =3D"mailto:kriszyp@gmail.com" target=3D"_blank">kriszyp@gmail.com</a>> w= rote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0p= x 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang= =3D"EN-CA"><div><p class=3D"MsoNormal">For the sake of putting this in the = email thread (other code discussion in GitHub), here is the latest squashed= commit of the proposed patch (with the on-demand, retained overlapped arra= y to reduce re-malloc and opening event handles): <a href=3D"https://github= .com/kriszyp/node-lmdb/commit/726a9156662c703bf3d453aab75ee222072b990f" tar= get=3D"_blank">https://github.com/kriszyp/node-lmdb/commit/726a9156662c703b= f3d453aab75ee222072b990f</a></p><p class=3D"MsoNormal"><u></u>=C2=A0<u></u>= </p><p class=3D"MsoNormal">Thanks,<br>Kris</p><p class=3D"MsoNormal"><u></u=
=C2=A0<u></u></p><div style=3D"border-right:none;border-bottom:none;border=
-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm"><p cl= ass=3D"MsoNormal" style=3D"border:none;padding:0cm"><b>From: </b><a href=3D= "mailto:kriszyp@gmail.com" target=3D"_blank">Kris Zyp</a><br><b>Sent: </b>A= pril 30, 2019 12:43 PM<br><b>To: </b><a href=3D"mailto:hyc@symas.com" targe= t=3D"_blank">Howard Chu</a>; <a href=3D"mailto:openldap-its@OpenLDAP.org" t= arget=3D"_blank">openldap-its@OpenLDAP.org</a><br><b>Subject: </b>RE: (ITS#= 9017) Improving performance of commit sync in Windows</p></div><p class=3D"= MsoNormal"><u></u>=C2=A0<u></u></p><p class=3D"MsoNormal">> What is the = point of using writemap mode if you still need to use WriteFile<u></u><u></= u></p><p class=3D"MsoNormal">> on every individual page?<u></u><u></u></= p><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><p class=3D"MsoNormal">As = I understood from the documentation, and have observed, using writemap mode= is faster (and uses less temporary memory) because it doesn=E2=80=99t requ= ire mallocs to allocate pages (docs: =E2=80=9CThis is faster and uses fewer= mallocs=E2=80=9D). To be clear though, LMDB is so incredibly fast and effi= cient, that in sync-mode, it takes enormous transactions before the time sp= ent allocating and creating the dirty pages with the updated b-tree is anyw= here even remotely close to the time it takes to wait for disk flushing, ev= en with an SSD. But the more pertinent question is efficiency, and measurin= g CPU cycles rather than time spent (efficiency is more important than just= time spent). When I ran my tests this morning of 100 (sync) transactions w= ith 100 puts per transaction, times varied quite a bit, but it seemed like = running with writemap enabled typically averages about 500ms of CPU and wit= h writemap disabled it typically averages around 600ms. Not a huge differen= ce, but still definitely worthwhile, I think.<u></u><u></u></p><p class=3D"= MsoNormal"><u></u>=C2=A0<u></u></p><p class=3D"MsoNormal">Caveat emptor: Me= asuring LMDB performance with sync interactions on Windows is one of the mo= st frustratingly erratic things to measure. It is sunny outside right now, = times could be different when it starts raining later, but, this is what I = saw this morning...<u></u><u></u></p><p class=3D"MsoNormal"><u></u>=C2=A0<u=
</u></p><p class=3D"MsoNormal">> What is the performance difference bet=
ween your patch using writemap, and just<u></u><u></u></p><p class=3D"MsoNo= rmal">> not using writemap in the first place?<u></u><u></u></p><p class= =3D"MsoNormal"><u></u>=C2=A0<u></u></p><p class=3D"MsoNormal">Running 1000 = sync transactions on 3GB db with a single put per transaction, without writ= emap map, without the patch took about 60 seconds. And it took about 1 seco= nd with the patch with writemap mode enabled! (there is no significant diff= erence in sync times with writemap enabled or disabled with the patch.) So = the difference was huge in my test. And not only that, without the patch, t= he CPU usage was actually _<i>higher</i>_ during that 60 seconds (close to = 100% of a core) than during the execution with the patch for one second (cl= ose to 50%).=C2=A0 Anyway, there are certainly tests I have run where the d= ifferences are not as large (doing small commits on large dbs accentuates t= he differences), but the patch always seems to win. It could also be that m= y particular configuration causes bigger differences (on an SSD drive, and = maybe a more fragmented file?).<u></u><u></u></p><p class=3D"MsoNormal"><u>= </u>=C2=A0<u></u></p><p class=3D"MsoNormal">Anyway, I added error handling = for the malloc, and fixed/changed the other things you suggested. Be happy = to make any other changes you want. The updated patch is here:<u></u><u></u=
</p><p class=3D"MsoNormal"><a href=3D"https://github.com/kriszyp/node-lmdb=
/commit/25366dea9453749cf6637f43ec17b9b62094acde" target=3D"_blank">https:/= /github.com/kriszyp/node-lmdb/commit/25366dea9453749cf6637f43ec17b9b62094ac= de</a><u></u><u></u></p><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><p c= lass=3D"MsoNormal">><span><span style=3D"font-size:9pt;font-family:Conso= las;color:rgb(36,41,46)"> OVERLAPPED* ov =3D </span></span><span><span styl= e=3D"font-size:9pt;font-family:Consolas;color:rgb(0,92,197)">malloc</span><= /span><span><span style=3D"font-size:9pt;font-family:Consolas;color:rgb(36,= 41,46)">((pagecount - keep) * </span></span><span><span style=3D"font-size:= 9pt;font-family:Consolas;color:rgb(215,58,73)">sizeof</span></span><span><s= pan style=3D"font-size:9pt;font-family:Consolas;color:rgb(36,41,46)">(OVERL= APPED));</span></span><span><span style=3D"font-size:9pt;font-family:Consol= as;color:rgb(36,41,46)"><u></u><u></u></span></span></p><p class=3D"MsoNorm= al"><span><span style=3D"font-size:9pt;font-family:Consolas;color:rgb(36,41= ,46)">> </span></span><span style=3D"font-size:10.5pt;font-family:"= Segoe UI",sans-serif;color:rgb(36,41,46);background:white">Probably th= is ought to just be pre-allocated based on the maximum number of dirty page= s a txn allows.</span><span style=3D"font-size:10.5pt;font-family:"Seg= oe UI",sans-serif;background:white"><u></u><u></u></span></p><p class= =3D"MsoNormal"><span style=3D"font-size:10.5pt;font-family:"Segoe UI&q= uot;,sans-serif;color:rgb(36,41,46);background:white"><u></u>=C2=A0<u></u><= /span></p><p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;font-famil= y:"Segoe UI",sans-serif;color:rgb(36,41,46);background:white">I w= asn=E2=80=99t sure I understood this comment. Are you suggesting we </span>= malloc(MDB_IDL_UM_MAX * sizeof(OVERLAPPED)) for each environment, and retai= n it for the life of the environment? I think that is 4MB, if my math is ri= ght, which seems like a lot of memory to keep allocated (we usually have a = lot of open environments). If the goal is to reduce the number of mallocs, = how about we retain the OVERLAPPED array, and only free and re-malloc if th= e previous allocation wasn=E2=80=99t large enough? Then there isn=E2=80=99t= unnecessary allocation, and we only malloc when there is a bigger transact= ion than any previous. I put this together in a separate commit, as I wasn= =E2=80=99t sure if this what you wanted (can squash if you prefer): <a href= =3D"https://github.com/kriszyp/node-lmdb/commit/2fe68fb5269c843e2e789746a17= a4b2adefaac40" target=3D"_blank">https://github.com/kriszyp/node-lmdb/commi= t/2fe68fb5269c843e2e789746a17a4b2adefaac40</a><u></u><u></u></p><p class=3D= "MsoNormal"><u></u>=C2=A0<u></u></p><p class=3D"MsoNormal">Thank you for th= e review! <span style=3D"font-size:10.5pt;font-family:"Segoe UI",= sans-serif;color:rgb(36,41,46);background:white"><u></u><u></u></span></p><= p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><p class=3D"MsoNormal">Thanks= ,<br>Kris<u></u><u></u></p><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><= div style=3D"border-right:none;border-bottom:none;border-left:none;border-t= op:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm"><p class=3D"MsoNormal"><= b>From: </b><a href=3D"mailto:hyc@symas.com" target=3D"_blank">Howard Chu</= a><br><b>Sent: </b>April 30, 2019 7:12 AM<br><b>To: </b><a href=3D"mailto:k= riszyp@gmail.com" target=3D"_blank">kriszyp@gmail.com</a>; <a href=3D"mailt= o:openldap-its@OpenLDAP.org" target=3D"_blank">openldap-its@OpenLDAP.org</a=
<br><b>Subject: </b>Re: (ITS#9017) Improving performance of commit sync in=
Windows<u></u><u></u></p></div><p class=3D"MsoNormal"><u></u>=C2=A0<u></u>= </p><p class=3D"MsoNormal"><a href=3D"mailto:kriszyp@gmail.com" target=3D"_= blank">kriszyp@gmail.com</a> wrote:<u></u><u></u></p><p class=3D"MsoNormal"=
> Full_Name: Kristopher William Zyp<u></u><u></u></p><p class=3D"MsoNor=
mal">> Version: LMDB 0.9.23<u></u><u></u></p><p class=3D"MsoNormal">>= OS: Windows<u></u><u></u></p><p class=3D"MsoNormal">> URL: <a href=3D"h= ttps://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab9332= b7fc4ce9" target=3D"_blank">https://github.com/kriszyp/node-lmdb/commit/7ff= 525ae57684a163d32af74a0ab9332b7fc4ce9</a><u></u><u></u></p><p class=3D"MsoN= ormal">> Submission from: (NULL) (71.199.6.148)<u></u><u></u></p><p clas= s=3D"MsoNormal">> <u></u><u></u></p><p class=3D"MsoNormal">> <u></u><= u></u></p><p class=3D"MsoNormal">> We have seen very poor performance on= the sync of commits on large databases in<u></u><u></u></p><p class=3D"Mso= Normal">> Windows. On databases with 2GB of data, in writemap mode, the = sync of even small<u></u><u></u></p><p class=3D"MsoNormal">> commits is = consistently well over 100ms (without writemap it is faster, but<u></u><u><= /u></p><p class=3D"MsoNormal">> still slow). It is expected that a sync = should take some time while waiting for<u></u><u></u></p><p class=3D"MsoNor= mal">> disk confirmation of the writes, but more concerning is that thes= e sync<u></u><u></u></p><p class=3D"MsoNormal">> operations (in writemap= mode) are instead dominated by nearly 100% system CPU<u></u><u></u></p><p = class=3D"MsoNormal">> utilization, so operations that requires sub-milli= second b-tree update<u></u><u></u></p><p class=3D"MsoNormal">> operation= s are then dominated by very large amounts of system CPU cycles during<u></= u><u></u></p><p class=3D"MsoNormal">> the sync phase.<u></u><u></u></p><= p class=3D"MsoNormal">> <u></u><u></u></p><p class=3D"MsoNormal">> I = think that the fundamental problem is that FlushViewOfFile seems to be an O= (n)<u></u><u></u></p><p class=3D"MsoNormal">> operation where n is the s= ize of the file (or map). I presume that Windows is<u></u><u></u></p><p cla= ss=3D"MsoNormal">> scanning the entire map/file for dirty pages to flush= , I'm guessing because it<u></u><u></u></p><p class=3D"MsoNormal">> = doesn't have an internal index of all the dirty pages for every file/ma= p-view in<u></u><u></u></p><p class=3D"MsoNormal">> the OS disk cache. T= herefore, the turns into an extremely expensive, CPU-bound<u></u><u></u></p=
<p class=3D"MsoNormal">> operation to find the dirty pages for (large f=
ile) and initiate their writes,<u></u><u></u></p><p class=3D"MsoNormal">>= ; which, of course, is contrary to the whole goal of a scalable database sy= stem.<u></u><u></u></p><p class=3D"MsoNormal">> And FlushFileBuffers is = also relatively slow as well. We have attempted to batch<u></u><u></u></p><= p class=3D"MsoNormal">> as many operations into single transaction as po= ssible, but this is still a very<u></u><u></u></p><p class=3D"MsoNormal">&g= t; large overhead.<u></u><u></u></p><p class=3D"MsoNormal">> <u></u><u><= /u></p><p class=3D"MsoNormal">> The Windows docs for FlushFileBuffers it= self warns about the inefficiencies of<u></u><u></u></p><p class=3D"MsoNorm= al">> this function (<a href=3D"https://docs.microsoft.com/en-us/windows= /desktop/api/fileapi/nf-fileapi-flushfilebuffers" target=3D"_blank">https:/= /docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-flushfileb= uffers</a>).<u></u><u></u></p><p class=3D"MsoNormal">> Which also points= to the solution: it is much faster to write out the dirty<u></u><u></u></p=
<p class=3D"MsoNormal">> pages with WriteFile through a sync file handl=
e (FILE_FLAG_WRITE_THROUGH).<u></u><u></u></p><p class=3D"MsoNormal">> <= u></u><u></u></p><p class=3D"MsoNormal">> The associated patch<u></u><u>= </u></p><p class=3D"MsoNormal">> (<a href=3D"https://github.com/kriszyp/= node-lmdb/commit/7ff525ae57684a163d32af74a0ab9332b7fc4ce9" target=3D"_blank= ">https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab9= 332b7fc4ce9</a>)<u></u><u></u></p><p class=3D"MsoNormal">> is my attempt= at implementing this solution, for Windows. Fortunately, with the<u></u><u=
</u></p><p class=3D"MsoNormal">> design of LMDB, this is relatively str=
aightforward. LMDB already supports<u></u><u></u></p><p class=3D"MsoNormal"=
> writing out dirty pages with WriteFile calls. I added a write-through=
handle for<u></u><u></u></p><p class=3D"MsoNormal">> sending these writ= es directly to disk. I then made that file-handle<u></u><u></u></p><p class= =3D"MsoNormal">> overlapped/asynchronously, so all the writes for a comm= it could be started in<u></u><u></u></p><p class=3D"MsoNormal">> overlap= mode, and (at least theoretically) transfer in parallel to the drive and<u=
</u><u></u></p><p class=3D"MsoNormal">> then used GetOverlappedResult t=
o wait for the completion. So basically<u></u><u></u></p><p class=3D"MsoNor= mal">> mdb_page_flush becomes the sync. I extended the writing of dirty = pages through<u></u><u></u></p><p class=3D"MsoNormal">> WriteFile to wri= temap mode as well (for writing meta too), so that WriteFile<u></u><u></u><= /p><p class=3D"MsoNormal">> with write-through can be used to flush the = data without ever needing to call<u></u><u></u></p><p class=3D"MsoNormal">&= gt; FlushViewOfFile or FlushFileBuffers. I also implemented support for wri= te<u></u><u></u></p><p class=3D"MsoNormal">> gathering in writemap mode = where contiguous file positions infers contiguous<u></u><u></u></p><p class= =3D"MsoNormal">> memory (by tracking the starting position with wdp and = writing contiguous pages<u></u><u></u></p><p class=3D"MsoNormal">> in si= ngle operations). Sorting of the dirty list is maintained even in writemap<= u></u><u></u></p><p class=3D"MsoNormal">> mode for this purpose.<u></u><= u></u></p><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><p class=3D"MsoNor= mal">What is the point of using writemap mode if you still need to use Writ= eFile<u></u><u></u></p><p class=3D"MsoNormal">on every individual page?<u><= /u><u></u></p><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><p class=3D"Ms= oNormal">> The performance benefits of this patch, in my testing, are co= nsiderable. Writing<u></u><u></u></p><p class=3D"MsoNormal">> out/syncin= g transactions is typically over 5x faster in writemap mode, and 2x<u></u><= u></u></p><p class=3D"MsoNormal">> faster in standard mode. And perhaps = more importantly (especially in environment<u></u><u></u></p><p class=3D"Ms= oNormal">> with many threads/processes), the efficiency benefits are eve= n larger,<u></u><u></u></p><p class=3D"MsoNormal">> particularly in writ= emap mode, where there can be a 50-100x reduction in the<u></u><u></u></p><= p class=3D"MsoNormal">> system CPU usage by using this patch. This bring= s windows performance with<u></u><u></u></p><p class=3D"MsoNormal">> syn= c'ed transactions in LMDB back into the range of "lightning" = performance :).<u></u><u></u></p><p class=3D"MsoNormal"><u></u>=C2=A0<u></u=
</p><p class=3D"MsoNormal">What is the performance difference between your=
patch using writemap, and just<u></u><u></u></p><p class=3D"MsoNormal">not= using writemap in the first place?<u></u><u></u></p><p class=3D"MsoNormal"=
<u></u>=C2=A0<u></u></p><p class=3D"MsoNormal">-- <u></u><u></u></p><p cla=
ss=3D"MsoNormal">=C2=A0=C2=A0-- Howard Chu<u></u><u></u></p><p class=3D"Mso= Normal">=C2=A0 CTO, Symas Corp.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 <a href=3D"http://www.symas.com" target=3D"_blank">http:= //www.symas.com</a><u></u><u></u></p><p class=3D"MsoNormal">=C2=A0 Director= , Highland Sun=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"http://highlandsun.com/hy= c/" target=3D"_blank">http://highlandsun.com/hyc/</a><u></u><u></u></p><p c= lass=3D"MsoNormal">=C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http:/= /www.openldap.org/project/" target=3D"_blank">http://www.openldap.org/proje= ct/</a><u></u><u></u></p><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><p = class=3D"MsoNormal"><u></u>=C2=A0<u></u></p></div></div></blockquote></div> </blockquote></div> </blockquote></div>
--000000000000bdb86d059deaf350--