openldap-bugs

openldap-bugs@openldap.org

1 participants
20966 discussions

Re: (ITS#8349) fix ppolicy issue
by hamano＠osstech.co.jp 06 May '19

06 May '19

Sorry for my late reply. https://www.osstech.co.jp/download/hamano/openldap/ppolicy_fix_pwdInHistory… The attached file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Open Source Solution Technology Corporation. Open Source Solution Technology Corporation has not assigned rights and/or interest in this work to any party. I, HAMANO Tsukasa am authorized by Open Source Solution Technology Corporation, my employer, to release this work under the following terms. Open Source Solution Technology Corporation hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice. On Fri, 08 Sep 2017 01:21:42 +0900, Quanah Gibson-Mount wrote: > > --On Tuesday, January 12, 2016 7:28 AM +0000 hamano(a)osstech.co.jp wrote: > > > Full_Name: HAMANO Tsukasa > > Version: 2.4.43 > > OS: Linux > > URL: > > https://www.osstech.co.jp/download/hamano/openldap/ppolicy_fix_pwdInHisto > > ry.patch Submission from: (NULL) (240b:10:2640:bf0:290:4cff:fe0d:f43e) > > > > > > We fixed several issue around ppolicy. > > > > 1) reduce pwdInHistory > > If set pwdInHistory to 5 then reduce pwdInHistory to 3, > > We expect to check password with three history, but ppolicy check > > password with all pwdHistory attribute. > > > > 2) reduce pwdInHistory to zero > > If set pwdInHistory to 5 then reduce pwdInHistory to 0, > > We expect that ppolicy password checking will be disbale. but the > > pwdHistory attribute are remains, so password checking is still > > enabled. > > We need to remove pwdHistory attribute. > > Hi, > > I'm working on catching up on old ITS submissions. This submission is > missing an IPR and cannot be included until it is provided. Please > see <http://www.openldap.org/devel/contributing.html> for information > on the IPR requirements. > > Thanks, > Quanah > > > -- > > Quanah Gibson-Mount > Product Architect > Symas Corporation > Packaged, certified, and supported LDAP solutions powered by OpenLDAP: > <http://www.symas.com> > -- Open Source Solution Technology Corporation HAMANO Tsukasa <hamano(a)osstech.co.jp>

1 0

Re: (ITS#8977) Make IDL sizes configurable in back-mdb
by quanah＠symas.com 06 May '19

06 May '19

--On Friday, February 15, 2019 2:37 PM +0000 hyc(a)openldap.org wrote: > Full_Name: Howard Chu > Version: 2.5 > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (80.233.42.124) > Submitted by: hyc > > > IDLs used for index slots are hardcoded to a max of 65536 elements in the > DB and twice that in memory. Sites with larger DBs tend to need larger > IDLs; if they're running with sufficient memory it ought to be possible > to configure larger IDLs. Per the documentation, this new setting appears to be a new type of "global" option that affects all MDB databases in the configuration. This new setting, per the documentation, must be configured in a "backend mdb" section. However, there appear to be some problems with the implmentation. Issue #1: "backend mdb" requires a "directory" directive Since this is a new "global" concept, "directory" should not be required (and in fact, seems problematc). Issue #2: If a "directory" statement is supplied (see above), slapd core dumps when attempting to convert this configuration to cn=config: The slapd.conf file I'm using to test is: include /usr/local/etc/openldap/schema/core.schema include /usr/local/etc/openldap/schema/cosine.schema include /usr/local/etc/openldap/schema/inetorgperson.schema modulepath /usr/local/lib64/openldap moduleload back_mdb backend mdb idlexp 32 database mdb directory /var/openldap-data suffix dc=example,dc=com index default eq index objectClass (The directory line location was moved to the "backend" section after initial testing due to issue #1). --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

RE: (ITS#9017) Improving performance of commit sync in Windows
by kriszyp＠gmail.com 03 May '19

03 May '19

--_E0C027EF-451F-4EC6-B6DE-2F6B94348BB5_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For the sake of putting this in the email thread (other code discussion in = GitHub), here is the latest squashed commit of the proposed patch (with the= on-demand, retained overlapped array to reduce re-malloc and opening event= handles): https://github.com/kriszyp/node-lmdb/commit/726a9156662c703bf3d4= 53aab75ee222072b990f Thanks, Kris From: Kris Zyp Sent: April 30, 2019 12:43 PM To: Howard Chu; openldap-its(a)OpenLDAP.org Subject: RE: (ITS#9017) Improving performance of commit sync in Windows > What is the point of using writemap mode if you still need to use WriteFi= le > on every individual page? As I understood from the documentation, and have observed, using writemap m= ode is faster (and uses less temporary memory) because it doesn=E2=80=99t r= equire mallocs to allocate pages (docs: =E2=80=9CThis is faster and uses fe= wer mallocs=E2=80=9D). To be clear though, LMDB is so incredibly fast and e= fficient, that in sync-mode, it takes enormous transactions before the time= spent allocating and creating the dirty pages with the updated b-tree is a= nywhere even remotely close to the time it takes to wait for disk flushing,= even with an SSD. But the more pertinent question is efficiency, and measu= ring CPU cycles rather than time spent (efficiency is more important than j= ust time spent). When I ran my tests this morning of 100 (sync) transaction= s with 100 puts per transaction, times varied quite a bit, but it seemed li= ke running with writemap enabled typically averages about 500ms of CPU and = with writemap disabled it typically averages around 600ms. Not a huge diffe= rence, but still definitely worthwhile, I think. Caveat emptor: Measuring LMDB performance with sync interactions on Windows= is one of the most frustratingly erratic things to measure. It is sunny ou= tside right now, times could be different when it starts raining later, but= , this is what I saw this morning... > What is the performance difference between your patch using writemap, and= just > not using writemap in the first place? Running 1000 sync transactions on 3GB db with a single put per transaction,= without writemap map, without the patch took about 60 seconds. And it took= about 1 second with the patch with writemap mode enabled! (there is no sig= nificant difference in sync times with writemap enabled or disabled with th= e patch.) So the difference was huge in my test. And not only that, without= the patch, the CPU usage was actually _higher_ during that 60 seconds (clo= se to 100% of a core) than during the execution with the patch for one seco= nd (close to 50%). =C2=A0Anyway, there are certainly tests I have run where= the differences are not as large (doing small commits on large dbs accentu= ates the differences), but the patch always seems to win. It could also be = that my particular configuration causes bigger differences (on an SSD drive= , and maybe a more fragmented file?). Anyway, I added error handling for the malloc, and fixed/changed the other = things you suggested. Be happy to make any other changes you want. The upda= ted patch is here: https://github.com/kriszyp/node-lmdb/commit/25366dea9453749cf6637f43ec17b9b= 62094acde > OVERLAPPED* ov =3D malloc((pagecount - keep) * sizeof(OVERLAPPED)); > Probably this ought to just be pre-allocated based on the maximum number = of dirty pages a txn allows. I wasn=E2=80=99t sure I understood this comment. Are you suggesting we mall= oc(MDB_IDL_UM_MAX * sizeof(OVERLAPPED)) for each environment, and retain it= for the life of the environment? I think that is 4MB, if my math is right,= which seems like a lot of memory to keep allocated (we usually have a lot = of open environments). If the goal is to reduce the number of mallocs, how = about we retain the OVERLAPPED array, and only free and re-malloc if the pr= evious allocation wasn=E2=80=99t large enough? Then there isn=E2=80=99t unn= ecessary allocation, and we only malloc when there is a bigger transaction = than any previous. I put this together in a separate commit, as I wasn=E2= =80=99t sure if this what you wanted (can squash if you prefer): https://gi= thub.com/kriszyp/node-lmdb/commit/2fe68fb5269c843e2e789746a17a4b2adefaac40 Thank you for the review!=20 Thanks, Kris From: Howard Chu Sent: April 30, 2019 7:12 AM To: kriszyp(a)gmail.com; openldap-its(a)OpenLDAP.org Subject: Re: (ITS#9017) Improving performance of commit sync in Windows kriszyp(a)gmail.com wrote: > Full_Name: Kristopher William Zyp > Version: LMDB 0.9.23 > OS: Windows > URL: https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74= a0ab9332b7fc4ce9 > Submission from: (NULL) (71.199.6.148) >=20 >=20 > We have seen very poor performance on the sync of commits on large databa= ses in > Windows. On databases with 2GB of data, in writemap mode, the sync of eve= n small > commits is consistently well over 100ms (without writemap it is faster, b= ut > still slow). It is expected that a sync should take some time while waiti= ng for > disk confirmation of the writes, but more concerning is that these sync > operations (in writemap mode) are instead dominated by nearly 100% system= CPU > utilization, so operations that requires sub-millisecond b-tree update > operations are then dominated by very large amounts of system CPU cycles = during > the sync phase. >=20 > I think that the fundamental problem is that FlushViewOfFile seems to be = an O(n) > operation where n is the size of the file (or map). I presume that Window= s is > scanning the entire map/file for dirty pages to flush, I'm guessing becau= se it > doesn't have an internal index of all the dirty pages for every file/map-= view in > the OS disk cache. Therefore, the turns into an extremely expensive, CPU-= bound > operation to find the dirty pages for (large file) and initiate their wri= tes, > which, of course, is contrary to the whole goal of a scalable database sy= stem. > And FlushFileBuffers is also relatively slow as well. We have attempted t= o batch > as many operations into single transaction as possible, but this is still= a very > large overhead. >=20 > The Windows docs for FlushFileBuffers itself warns about the inefficienci= es of > this function (https://docs.microsoft.com/en-us/windows/desktop/api/filea= pi/nf-fileapi-flushfilebuffers). > Which also points to the solution: it is much faster to write out the dir= ty > pages with WriteFile through a sync file handle (FILE_FLAG_WRITE_THROUGH)= . >=20 > The associated patch > (https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab= 9332b7fc4ce9) > is my attempt at implementing this solution, for Windows. Fortunately, wi= th the > design of LMDB, this is relatively straightforward. LMDB already supports > writing out dirty pages with WriteFile calls. I added a write-through han= dle for > sending these writes directly to disk. I then made that file-handle > overlapped/asynchronously, so all the writes for a commit could be starte= d in > overlap mode, and (at least theoretically) transfer in parallel to the dr= ive and > then used GetOverlappedResult to wait for the completion. So basically > mdb_page_flush becomes the sync. I extended the writing of dirty pages th= rough > WriteFile to writemap mode as well (for writing meta too), so that WriteF= ile > with write-through can be used to flush the data without ever needing to = call > FlushViewOfFile or FlushFileBuffers. I also implemented support for write > gathering in writemap mode where contiguous file positions infers contigu= ous > memory (by tracking the starting position with wdp and writing contiguous= pages > in single operations). Sorting of the dirty list is maintained even in wr= itemap > mode for this purpose. What is the point of using writemap mode if you still need to use WriteFile on every individual page? > The performance benefits of this patch, in my testing, are considerable. = Writing > out/syncing transactions is typically over 5x faster in writemap mode, an= d 2x > faster in standard mode. And perhaps more importantly (especially in envi= ronment > with many threads/processes), the efficiency benefits are even larger, > particularly in writemap mode, where there can be a 50-100x reduction in = the > system CPU usage by using this patch. This brings windows performance wit= h > sync'ed transactions in LMDB back into the range of "lightning" performan= ce :). What is the performance difference between your patch using writemap, and j= ust not using writemap in the first place? --=20 =C2=A0=C2=A0-- Howard Chu =C2=A0 CTO, Symas Corp.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 http://www.symas.com =C2=A0 Director, Highland Sun=C2=A0=C2=A0=C2=A0=C2=A0 http://highlandsun.co= m/hyc/ =C2=A0 Chief Architect, OpenLDAP=C2=A0 http://www.openldap.org/project/ --_E0C027EF-451F-4EC6-B6DE-2F6B94348BB5_ Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8" <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta ht= tp-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta name= =3DGenerator content=3D"Microsoft Word 15 (filtered medium)"><style></style></head><body lang=3DEN-CA link=3Dblue vlink=3D"#954F72"><div cla= ss=3DWordSection1>For the sake of putting this in the = email thread (other code discussion in GitHub), here is the latest squashed= commit of the proposed patch (with the on-demand, retained overlapped arra= y to reduce re-malloc and opening event handles): https://github.com/kriszy= p/node-lmdb/commit/726a9156662c703bf3d453aab75ee222072b990f<o:p> </o:p>Thanks, Kris<o:p> </o:p><div style=3D'mso-element:para-border-= div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><= p class=3DMsoNormal style=3D'border:none;padding:0cm'>From: <a href= =3D"mailto:kriszyp@gmail.com">Kris Zyp</a> Sent: April 30, 2019 1= 2:43 PM To: <a href=3D"mailto:hyc@symas.com">Howard Chu</a>; <a h= ref=3D"mailto:openldap-its@OpenLDAP.org">openldap-its(a)OpenLDAP.org</a> <= b>Subject: RE: (ITS#9017) Improving performance of commit sync in Windo= ws</div><o:p> </o:p>= > What is the point of using writemap mode if you still need to use Writ= eFile<o:p></o:p>> on every individual page?<o:p= ></o:p><o:p> </o:p>A= s I understood from the documentation, and have observed, using writemap mo= de is faster (and uses less temporary memory) because it doesn=E2=80=99t re= quire mallocs to allocate pages (docs: =E2=80=9CThis is faster and uses few= er mallocs=E2=80=9D). To be clear though, LMDB is so incredibly fast and ef= ficient, that in sync-mode, it takes enormous transactions before the time = spent allocating and creating the dirty pages with the updated b-tree is an= ywhere even remotely close to the time it takes to wait for disk flushing, = even with an SSD. But the more pertinent question is efficiency, and measur= ing CPU cycles rather than time spent (efficiency is more important than ju= st time spent). When I ran my tests this morning of 100 (sync) transactions= with 100 puts per transaction, times varied quite a bit, but it seemed lik= e running with writemap enabled typically averages about 500ms of CPU and w= ith writemap disabled it typically averages around 600ms. Not a huge differ= ence, but still definitely worthwhile, I think.<o:p></o:p><o:p> </o:p>Caveat emptor: Measuring = LMDB performance with sync interactions on Windows is one of the most frust= ratingly erratic things to measure. It is sunny outside right now, times co= uld be different when it starts raining later, but, this is what I saw this= morning...<o:p></o:p><o:p> </o:p>> What is the performance difference between your patch us= ing writemap, and just<o:p></o:p>> not using wr= itemap in the first place?<o:p></o:p><o:p> </= o:p>Running 1000 sync transactions on 3GB db with = a single put per transaction, without writemap map, without the patch took = about 60 seconds. And it took about 1 second with the patch with writemap m= ode enabled! (there is no significant difference in sync times with writema= p enabled or disabled with the patch.) So the difference was huge in my tes= t. And not only that, without the patch, the CPU usage was actually _hig= her_ during that 60 seconds (close to 100% of a core) than during the e= xecution with the patch for one second (close to 50%).  Anyway, there = are certainly tests I have run where the differences are not as large (doin= g small commits on large dbs accentuates the differences), but the patch al= ways seems to win. It could also be that my particular configuration causes= bigger differences (on an SSD drive, and maybe a more fragmented file?).<o= :p></o:p><o:p> </o:p>Anyway, I added error handling for the malloc, and fixed/changed the other= things you suggested. Be happy to make any other changes you want. The upd= ated patch is here:<o:p></o:p>https://github.com/k= riszyp/node-lmdb/commit/25366dea9453749cf6637f43ec17b9b62094acde<o:p></o:p>= <o:p> </o:p>><spa= n class=3Dblob-code-inner> OVERLAPPED* ov =3D <sp= an style=3D'font-size:9.0pt;font-family:Consolas;color:#005CC5'>malloc</spa= n>((pagecount - keep) * sizeof(OVERLAPPED));<s= pan class=3Dblob-code-inner><o:p></o:p><span= class=3Dblob-code-inner>> Probably this ou= ght to just be pre-allocated based on the maximum number of dirty pages a t= xn allows.<o:p></o:p><o:p> </o:p>I wasn=E2=80=99t sure I understood this comment. Are you sugg= esting we malloc(MDB_IDL_UM_MAX * sizeof(OVERLAPPED)) for each envir= onment, and retain it for the life of the environment? I think that is 4MB,= if my math is right, which seems like a lot of memory to keep allocated (w= e usually have a lot of open environments). If the goal is to reduce the nu= mber of mallocs, how about we retain the OVERLAPPED array, and only free an= d re-malloc if the previous allocation wasn=E2=80=99t large enough? Then th= ere isn=E2=80=99t unnecessary allocation, and we only malloc when there is = a bigger transaction than any previous. I put this together in a separate c= ommit, as I wasn=E2=80=99t sure if this what you wanted (can squash if you = prefer): https://github.com/kriszyp/node-lmdb/commit/2fe68fb5269c843e2e7897= 46a17a4b2adefaac40<o:p></o:p><o:p> </o:p>= Thank you for the review! <o:p>= </o:p><o:p> </o:p>Thanks, Kris<o:p></o:p><o:p> </o:p></= p><div style=3D'border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0c= m 0cm 0cm'>From: <a href=3D"mailto:hyc@symas.co= m">Howard Chu</a> Sent: April 30, 2019 7:12 AM To: <a h= ref=3D"mailto:kriszyp@gmail.com">kriszyp(a)gmail.com</a>; <a href=3D"mailto:o= penldap-its(a)OpenLDAP.org">openldap-its(a)OpenLDAP.org</a> Subject: = Re: (ITS#9017) Improving performance of commit sync in Windows<o:p></o:p></= p></div><o:p> </o:p>kris= zyp(a)gmail.com wrote:<o:p></o:p>> Full_Name: Kri= stopher William Zyp<o:p></o:p>> Version: LMDB 0= .9.23<o:p></o:p>> OS: Windows<o:p></o:p>> URL: https://github.com/kriszyp/node-lmdb/commit/7ff= 525ae57684a163d32af74a0ab9332b7fc4ce9<o:p></o:p>&g= t; Submission from: (NULL) (71.199.6.148)<o:p></o:p>> <o:p></o:p>> <o:p></o:p>> We have seen very poor performance on the sync of commits on = large databases in<o:p></o:p>> Windows. On data= bases with 2GB of data, in writemap mode, the sync of even small<o:p></o:p>= > commits is consistently well over 100ms (with= out writemap it is faster, but<o:p></o:p>> stil= l slow). It is expected that a sync should take some time while waiting for= <o:p></o:p>> disk confirmation of the writes, b= ut more concerning is that these sync<o:p></o:p>&g= t; operations (in writemap mode) are instead dominated by nearly 100% syste= m CPU<o:p></o:p>> utilization, so operations th= at requires sub-millisecond b-tree update<o:p></o:p>> operations are then dominated by very large amounts of system CPU cy= cles during<o:p></o:p>> the sync phase.<o:p></o= :p>> <o:p></o:p>> I= think that the fundamental problem is that FlushViewOfFile seems to be an = O(n)<o:p></o:p>> operation where n is the size = of the file (or map). I presume that Windows is<o:p></o:p>> scanning the entire map/file for dirty pages to flush, I'm gue= ssing because it<o:p></o:p>> doesn't have an in= ternal index of all the dirty pages for every file/map-view in<o:p></o:p></= p>> the OS disk cache. Therefore, the turns into an= extremely expensive, CPU-bound<o:p></o:p>> ope= ration to find the dirty pages for (large file) and initiate their writes,<= o:p></o:p>> which, of course, is contrary to th= e whole goal of a scalable database system.<o:p></o:p>> And FlushFileBuffers is also relatively slow as well. We have atte= mpted to batch<o:p></o:p>> as many operations i= nto single transaction as possible, but this is still a very<o:p></o:p>= > large overhead.<o:p></o:p>> <o:p></o:p>> The Windows docs for FlushF= ileBuffers itself warns about the inefficiencies of<o:p></o:p>> this function (https://docs.microsoft.com/en-us/windows/d= esktop/api/fileapi/nf-fileapi-flushfilebuffers).<o:p></o:p>> Which also points to the solution: it is much faster to write= out the dirty<o:p></o:p>> pages with WriteFile= through a sync file handle (FILE_FLAG_WRITE_THROUGH).<o:p></o:p>> <o:p></o:p>> The associated= patch<o:p></o:p>> (https://github.com/kriszyp/= node-lmdb/commit/7ff525ae57684a163d32af74a0ab9332b7fc4ce9)<o:p></o:p><p= class=3DMsoNormal>> is my attempt at implementing this solution, for Wi= ndows. Fortunately, with the<o:p></o:p>> design= of LMDB, this is relatively straightforward. LMDB already supports<o:p></o= :p>> writing out dirty pages with WriteFile cal= ls. I added a write-through handle for<o:p></o:p>&= gt; sending these writes directly to disk. I then made that file-handle<o:p= ></o:p>> overlapped/asynchronously, so all the = writes for a commit could be started in<o:p></o:p>= > overlap mode, and (at least theoretically) transfer in parallel to the= drive and<o:p></o:p>> then used GetOverlappedR= esult to wait for the completion. So basically<o:p></o:p>> mdb_page_flush becomes the sync. I extended the writing of dirt= y pages through<o:p></o:p>> WriteFile to writem= ap mode as well (for writing meta too), so that WriteFile<o:p></o:p>> with write-through can be used to flush the data wit= hout ever needing to call<o:p></o:p>> FlushView= OfFile or FlushFileBuffers. I also implemented support for write<o:p></o:p>= > gathering in writemap mode where contiguous f= ile positions infers contiguous<o:p></o:p>> mem= ory (by tracking the starting position with wdp and writing contiguous page= s<o:p></o:p>> in single operations). Sorting of= the dirty list is maintained even in writemap<o:p></o:p>> mode for this purpose.<o:p></o:p><o:p>=  </o:p>What is the point of using writemap mo= de if you still need to use WriteFile<o:p></o:p>on= every individual page?<o:p></o:p><o:p> </o:p= >> The performance benefits of this patch, in m= y testing, are considerable. Writing<o:p></o:p>&gt= ; out/syncing transactions is typically over 5x faster in writemap mode, an= d 2x<o:p></o:p>> faster in standard mode. And p= erhaps more importantly (especially in environment<o:p></o:p>> with many threads/processes), the efficiency benefits are= even larger,<o:p></o:p>> particularly in write= map mode, where there can be a 50-100x reduction in the<o:p></o:p>> system CPU usage by using this patch. This brings wind= ows performance with<o:p></o:p>> sync'ed transa= ctions in LMDB back into the range of "lightning" performance :).= <o:p></o:p><o:p> </o:p>What is the performance difference between your patch using writemap, an= d just<o:p></o:p>not using writemap in the first p= lace?<o:p></o:p><o:p> </o:p>-- <o:p></o:p>  -- Howard Chu<o:= p></o:p>  CTO, Symas Corp.   &= nbsp;       http://www.symas.com<o:p></o:p></= p>  Director, Highland Sun   &nbsp= ; http://highlandsun.com/hyc/<o:p></o:p>  Chi= ef Architect, OpenLDAP  http://www.openldap.org/project/<o:p></o:p></p= ><o:p> </o:p><o:p> = </o:p></div></body></html>= --_E0C027EF-451F-4EC6-B6DE-2F6B94348BB5_--

1 0

Re: (ITS#9014) OpenLDAP modifies user provided TLS certificate before sending it to client
by quanah＠symas.com 03 May '19

03 May '19

--On Wednesday, April 24, 2019 9:23 PM +0000 siddjain(a)live.com wrote: > --_000_MWHPR08MB2400D7AE5E8EEC3D17192FACB53C0MWHPR08MB2400namp_ > Content-Type: text/plain; charset="iso-8859-1" > Content-Transfer-Encoding: quoted-printable > > Thank you. we tried using another openldap image and that worked. so it > see= ms the problem is with the osixia docker image we were using to run > openlda= p. it is based on debian (which uses GnuTLS per your email) so > tbh we are s= urprised it would have such a bug in it. the image that > worked for us is ba= sed on alpine. > https://github.com/osixia/docker-light-baseimage/blob/stable/image/Docker > fi= le > https://github.com/tiredofit/docker-openldap/blob/master/Dockerfile > but back to your comment, how can one isolate what TLS/SSL library > OpenLDAP= is linked to in the environment you're using? Use the "ldd" command or similar to see what libraries it is linked to. I see you filed a bug against Osixia, however the bug should be filed against GnuTLS, as that's where the issue is. This ITS will be closed as it is not OpenLDAP related. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#8314) ldap_back_is_proxy_authz returned 0 - misconfigured uri? - Maybe a Bug?
by quanah＠symas.com 03 May '19

03 May '19

--On Friday, May 03, 2019 5:22 PM +0000 dominiks.mail(a)gmx.net wrote: > 2. flags=3Doverride > After I added the flags=3Doverride setting in the idassert-bind section > the problem was gone and everything works for me. I don't know if it's a > good approach but at least it works for now with our openLDAP backend > server. Prior to OpenLDAP 2.4.33 where ITS#7403 was fixed (8829115c76), back-ldap would always behave as if the "override" flag had been set if the idassert-bind directive has been configured. I.e., the previously working configuration depended upon a bug and that bug was fixed. This ITS will be closed as back-ldap is functioning as designed and documented in the slapd-ldap(5) man page. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#8314) ldap_back_is_proxy_authz returned 0 - misconfigured uri? - Maybe a Bug?
by dominiks.mail＠gmx.net 03 May '19

03 May '19

Hi, I know this topic is years old but I faced the same issue today, running the current version of OpenLDAP. The problem can also occur when using the translucent backend. While troubleshooting the issue, I found two possible workarounds for our setup. 1. acl-bind Instead of using idassert-bind I used acl-bind. The connection problem "Error: ldap_back_is_proxy_authz returned 0, misconfigured URI?" didn't happen anymore, but since I got different results with my search filters, it wasn't sufficient. 2. flags=3Doverride After I added the flags=3Doverride setting in the idassert-bind section the problem was gone and everything works for me. I don't know if it's a good approach but at least it works for now with our openLDAP backend server. =2D- Regards Dominik

1 0

(ITS#8314) ldap_back_is_proxy_authz returned 0 - misconfigured uri? - Maybe a Bug?
by dominiks.mail＠gmx.net 03 May '19

03 May '19

1 0

(ITS#9018) dynlist don't close connection
by i.harbuz＠velcom.by 02 May '19

02 May '19

Full_Name: Ihar Harbuz Version: 2.4.44 OS: RHEL 7.6 URL: http://ftp.openldap.org/incoming/ Submission from: (NULL) (128.140.241.193) Good day. I have next configuration: ######################################################################### include /etc/openldap/schema/core.schema include /etc/openldap/schema/cosine.schema include /etc/openldap/schema/duaconf.schema include /etc/openldap/schema/dyngroup.schema include /etc/openldap/schema/nis.schema pidfile /var/run/openldap/slapd.pid argsfile /var/run/openldap/slapd.args modulepath /usr/lib64/openldap moduleload back_ldap.la moduleload back_meta.la moduleload dynlist.la moduleload memberof.la moduleload deref.la overlay dynlist dynlist-attrset posixGroup labeledURI database meta suffix "dc=main,dc=company,dc=by" uri "ldap://dc1-cont.main.company.by/dc=main,dc=company,dc=by" "ldap://dc2-cont.main.company.by/" idassert-bind bindmethod=simple binddn="CN=ldapproxy,OU=ServiceAccounts,DC=MAIN,DC=company,DC=BY" credentials="XXXXXXXXXXXXXXX" mode=none idassert-authzFrom * rebind-as-user yes subordinate rewriteEngine on rewriteContext searchFilter rewriteRule "RecursiveMemberOf=(.*),dc=by" "memberOf:1.2.840.113556.1.4.1941:=%1,dc=by" ":" database meta suffix "dc=external,dc=company,dc=by" uri "ldap://edc1-cont.main.company.by/dc=external,dc=company,dc=by" "ldap://edc2-cont.main.company.by/" idassert-bind bindmethod=simple binddn="CN=ldapproxy,DC=external,DC=company,DC=by" credentials="XXXXXXXXXXXXXXXXXXXX" mode=none idassert-authzFrom * rebind-as-user yes subordinate rewriteEngine on rewriteContext searchFilter rewriteRule "RecursiveMemberOf=(.*),dc=by" "memberOf:1.2.840.113556.1.4.1941:=%1,dc=by" ":" database hdb suffix "dc=company,dc=by" directory /var/lib/ldap rootdn cn=ldapadm,dc=company,dc=by rootpw "XXXXXXXXXX" index objectClass eq,pres index ou,cn,mail,surname,givenname eq,pres,sub database monitor ##################################################################################### Any ldapsearch commands work fine if request doesn't hit into dynlist. If request hit in dynlist then it output information and hanged up. slapd -d -1 write: ber_flush2: 1441 bytes to sd 15 0000: 30 82 05 9d 02 01 02 64 82 05 96 04 4c 63 6e 3d 0......d....Lcn= 0010: 53 47 5f 4f 53 5f 53 4f 4c 41 52 49 53 2c 6f 75 SG_OS_SOLARIS,ou 0020: 3d 53 65 63 75 72 69 74 79 20 47 72 6f 75 70 73 =Security Groups ................. ldap_write: want=1441, written=1441 0000: 30 82 05 9d 02 01 02 64 82 05 96 04 4c 63 6e 3d 0......d....Lcn= 0010: 53 47 5f 4f 53 5f 53 4f 4c 41 52 49 53 2c 6f 75 SG_OS_SOLARIS,ou 0020: 3d 53 65 63 75 72 69 74 79 20 47 72 6f 75 70 73 =Security Groups ................. 5ccac7d7 <= send_search_entry: conn 1000 exit. ldap_msgfree And wait while I interrupt request on client. After it: 5ccac856 daemon: activity on 1 descriptor 5ccac856 daemon: activity on: 15r 5ccac856 daemon: read active on 15 5ccac856 daemon: epoll: listen=7 active_threads=0 tvp=NULL 5ccac856 daemon: epoll: listen=8 active_threads=0 tvp=NULL 5ccac856 connection_get(15) 5ccac856 connection_get(15): got connid=1000 5ccac856 connection_read(15): checking for input on id=1000 ber_get_next ldap_read: want=8, got=0 5ccac856 ber_get_next on fd 15 failed errno=0 (Success) 5ccac856 connection_read(15): input error=-2 id=1000, closing. 5ccac856 connection_closing: readying conn=1000 sd=15 for close 5ccac856 connection_close: deferring conn=1000 sd=15 5ccac856 daemon: activity on 1 descriptor 5ccac856 daemon: activity on: 5ccac856 daemon: epoll: listen=7 active_threads=0 tvp=NULL 5ccac856 daemon: epoll: listen=8 active_threads=0 tvp=NULL 5ccac856 connection_resched: attempting closing conn=1000 sd=15 5ccac856 connection_close: conn=1000 sd=15 5ccac856 =>meta_back_conn_destroy: fetching conn=1000 DN="cn=solaris,dc=company,dc=by" 5ccac856 =>meta_back_conn_destroy: destroying conn 1000 refcnt=0 flags=0x00000100 ldap_free_connection 1 1 ldap_send_unbind ber_flush2: 7 bytes to sd 18 0000: 30 05 02 01 04 42 00 0....B. ldap_write: want=7, written=7 0000: 30 05 02 01 04 42 00 0....B. ldap_free_connection: actually freed ldap_msgfree ldap_free_connection 1 1 ldap_send_unbind ber_flush2: 7 bytes to sd 17 0000: 30 05 02 01 04 42 00 0....B. ldap_write: want=7, written=7 0000: 30 05 02 01 04 42 00 0....B. ldap_free_connection: actually freed ldap_msgfree 5ccac856 =>meta_back_conn_destroy: fetching conn=1000 DN="cn=solaris,dc=company,dc=by" 5ccac856 daemon: removing 15 5ccac856 conn=1000 fd=15 closed (connection lost) In client side it's look like: ldapsearch -b .... ... USERPRINCIPALNAME: test1(a)MAIN.COMPANY.BY USERPRINCIPALNAME: test2(a)MAIN.COMPANY.BY ^C

1 0

RE: (ITS#9017) Improving performance of commit sync in Windows
by kriszyp＠gmail.com 30 Apr '19

30 Apr '19

--_F6B70E9A-5F12-495E-A3D8-F48F4F20717D_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" > What is the point of using writemap mode if you still need to use WriteFi= le > on every individual page? As I understood from the documentation, and have observed, using writemap m= ode is faster (and uses less temporary memory) because it doesn=E2=80=99t r= equire mallocs to allocate pages (docs: =E2=80=9CThis is faster and uses fe= wer mallocs=E2=80=9D). To be clear though, LMDB is so incredibly fast and e= fficient, that in sync-mode, it takes enormous transactions before the time= spent allocating and creating the dirty pages with the updated b-tree is a= nywhere even remotely close to the time it takes to wait for disk flushing,= even with an SSD. But the more pertinent question is efficiency, and measu= ring CPU cycles rather than time spent (efficiency is more important than j= ust time spent). When I ran my tests this morning of 100 (sync) transaction= s with 100 puts per transaction, times varied quite a bit, but it seemed li= ke running with writemap enabled typically averages about 500ms of CPU and = with writemap disabled it typically averages around 600ms. Not a huge diffe= rence, but still definitely worthwhile, I think. Caveat emptor: Measuring LMDB performance with sync interactions on Windows= is one of the most frustratingly erratic things to measure. It is sunny ou= tside right now, times could be different when it starts raining later, but= , this is what I saw this morning... > What is the performance difference between your patch using writemap, and= just > not using writemap in the first place? Running 1000 sync transactions on 3GB db with a single put per transaction,= without writemap map, without the patch took about 60 seconds. And it took= about 1 second with the patch with writemap mode enabled! (there is no sig= nificant difference in sync times with writemap enabled or disabled with th= e patch.) So the difference was huge in my test. And not only that, without= the patch, the CPU usage was actually _higher_ during that 60 seconds (clo= se to 100% of a core) than during the execution with the patch for one seco= nd (close to 50%). Anyway, there are certainly tests I have run where the = differences are not as large (doing small commits on large dbs accentuates = the differences), but the patch always seems to win. It could also be that = my particular configuration causes bigger differences (on an SSD drive, and= maybe a more fragmented file?). Anyway, I added error handling for the malloc, and fixed/changed the other = things you suggested. Be happy to make any other changes you want. The upda= ted patch is here: https://github.com/kriszyp/node-lmdb/commit/25366dea9453749cf6637f43ec17b9b= 62094acde > OVERLAPPED* ov =3D malloc((pagecount - keep) * sizeof(OVERLAPPED)); > Probably this ought to just be pre-allocated based on the maximum number = of dirty pages a txn allows. I wasn=E2=80=99t sure I understood this comment. Are you suggesting we mall= oc(MDB_IDL_UM_MAX * sizeof(OVERLAPPED)) for each environment, and retain it= for the life of the environment? I think that is 4MB, if my math is right,= which seems like a lot of memory to keep allocated (we usually have a lot = of open environments). If the goal is to reduce the number of mallocs, how = about we retain the OVERLAPPED array, and only free and re-malloc if the pr= evious allocation wasn=E2=80=99t large enough? Then there isn=E2=80=99t unn= ecessary allocation, and we only malloc when there is a bigger transaction = than any previous. I put this together in a separate commit, as I wasn=E2= =80=99t sure if this what you wanted (can squash if you prefer): https://gi= thub.com/kriszyp/node-lmdb/commit/2fe68fb5269c843e2e789746a17a4b2adefaac40 Thank you for the review!=20 Thanks, Kris From: Howard Chu Sent: April 30, 2019 7:12 AM To: kriszyp(a)gmail.com; openldap-its(a)OpenLDAP.org Subject: Re: (ITS#9017) Improving performance of commit sync in Windows kriszyp(a)gmail.com wrote: > Full_Name: Kristopher William Zyp > Version: LMDB 0.9.23 > OS: Windows > URL: https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74= a0ab9332b7fc4ce9 > Submission from: (NULL) (71.199.6.148) >=20 >=20 > We have seen very poor performance on the sync of commits on large databa= ses in > Windows. On databases with 2GB of data, in writemap mode, the sync of eve= n small > commits is consistently well over 100ms (without writemap it is faster, b= ut > still slow). It is expected that a sync should take some time while waiti= ng for > disk confirmation of the writes, but more concerning is that these sync > operations (in writemap mode) are instead dominated by nearly 100% system= CPU > utilization, so operations that requires sub-millisecond b-tree update > operations are then dominated by very large amounts of system CPU cycles = during > the sync phase. >=20 > I think that the fundamental problem is that FlushViewOfFile seems to be = an O(n) > operation where n is the size of the file (or map). I presume that Window= s is > scanning the entire map/file for dirty pages to flush, I'm guessing becau= se it > doesn't have an internal index of all the dirty pages for every file/map-= view in > the OS disk cache. Therefore, the turns into an extremely expensive, CPU-= bound > operation to find the dirty pages for (large file) and initiate their wri= tes, > which, of course, is contrary to the whole goal of a scalable database sy= stem. > And FlushFileBuffers is also relatively slow as well. We have attempted t= o batch > as many operations into single transaction as possible, but this is still= a very > large overhead. >=20 > The Windows docs for FlushFileBuffers itself warns about the inefficienci= es of > this function (https://docs.microsoft.com/en-us/windows/desktop/api/filea= pi/nf-fileapi-flushfilebuffers). > Which also points to the solution: it is much faster to write out the dir= ty > pages with WriteFile through a sync file handle (FILE_FLAG_WRITE_THROUGH)= . >=20 > The associated patch > (https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab= 9332b7fc4ce9) > is my attempt at implementing this solution, for Windows. Fortunately, wi= th the > design of LMDB, this is relatively straightforward. LMDB already supports > writing out dirty pages with WriteFile calls. I added a write-through han= dle for > sending these writes directly to disk. I then made that file-handle > overlapped/asynchronously, so all the writes for a commit could be starte= d in > overlap mode, and (at least theoretically) transfer in parallel to the dr= ive and > then used GetOverlappedResult to wait for the completion. So basically > mdb_page_flush becomes the sync. I extended the writing of dirty pages th= rough > WriteFile to writemap mode as well (for writing meta too), so that WriteF= ile > with write-through can be used to flush the data without ever needing to = call > FlushViewOfFile or FlushFileBuffers. I also implemented support for write > gathering in writemap mode where contiguous file positions infers contigu= ous > memory (by tracking the starting position with wdp and writing contiguous= pages > in single operations). Sorting of the dirty list is maintained even in wr= itemap > mode for this purpose. What is the point of using writemap mode if you still need to use WriteFile on every individual page? > The performance benefits of this patch, in my testing, are considerable. = Writing > out/syncing transactions is typically over 5x faster in writemap mode, an= d 2x > faster in standard mode. And perhaps more importantly (especially in envi= ronment > with many threads/processes), the efficiency benefits are even larger, > particularly in writemap mode, where there can be a 50-100x reduction in = the > system CPU usage by using this patch. This brings windows performance wit= h > sync'ed transactions in LMDB back into the range of "lightning" performan= ce :). What is the performance difference between your patch using writemap, and j= ust not using writemap in the first place? --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ --_F6B70E9A-5F12-495E-A3D8-F48F4F20717D_ Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8" <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta ht= tp-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta name= =3DGenerator content=3D"Microsoft Word 15 (filtered medium)"><style></style></head><body lang=3DEN-CA link=3Dblue vlink=3D"#954F72"><div cla= ss=3DWordSection1>> What is the point of using writ= emap mode if you still need to use WriteFile<o:p></o:p>> on every individual page?<o:p> </o:= p>As I understood from the documentation, and have= observed, using writemap mode is faster (and uses less temporary memory) b= ecause it doesn=E2=80=99t require mallocs to allocate pages (docs: =E2=80= =9CThis is faster and uses fewer mallocs=E2=80=9D). To be clear though, LMD= B is so incredibly fast and efficient, that in sync-mode, it takes enormous= transactions before the time spent allocating and creating the dirty pages= with the updated b-tree is anywhere even remotely close to the time it tak= es to wait for disk flushing, even with an SSD. But the more pertinent ques= tion is efficiency, and measuring CPU cycles rather than time spent (effici= ency is more important than just time spent). When I ran my tests this morn= ing of 100 (sync) transactions with 100 puts per transaction, times varied = quite a bit, but it seemed like running with writemap enabled typically ave= rages about 500ms of CPU and with writemap disabled it typically averages a= round 600ms. Not a huge difference, but still definitely worthwhile, I thin= k.<o:p> </o:p>Caveat= emptor: Measuring LMDB performance with sync interactions on Windows is on= e of the most frustratingly erratic things to measure. It is sunny outside = right now, times could be different when it starts raining later, but, this= is what I saw this morning...<o:p></o:p><o:p>&nbs= p;</o:p>> What is the performance difference be= tween your patch using writemap, and just<o:p></o:p>> not using writemap in the first place?<o:p></o:p><o:p> </o:p>Running 1000 sync transacti= ons on 3GB db with a single put per transaction, without writemap map, with= out the patch took about 60 seconds. And it took about 1 second with the pa= tch with writemap mode enabled! (there is no significant difference in sync= times with writemap enabled or disabled with the patch.) So the difference= was huge in my test. And not only that, without the patch, the CPU usage w= as actually _higher_ during that 60 seconds (close to 100% of a core= ) than during the execution with the patch for one second (close to 50%). = =C2=A0Anyway, there are certainly tests I have run where the differences ar= e not as large (doing small commits on large dbs accentuates the difference= s), but the patch always seems to win. It could also be that my particular = configuration causes bigger differences (on an SSD drive, and maybe a more = fragmented file?).<o:p> </o:p>Anyway, I added error handling for the malloc, and fixed/changed = the other things you suggested. Be happy to make any other changes you want= . The updated patch is here:<o:p></o:p>https://git= hub.com/kriszyp/node-lmdb/commit/25366dea9453749cf6637f43ec17b9b62094acde<o= :p></o:p><o:p> </o:p>> OVERLAPPED* ov =3D ma= lloc((pagecount - keep) * </span= >sizeof(OVERLAPPED));<o:p>= </o:p>= > </s= pan>Probably this ought to just be pre-alloca= ted based on the maximum number of dirty pages a txn allows.<o:p></o:p></sp= an><o:p> </o:p>= I wasn=E2=80=99t sure I und= erstood this comment. Are you suggesting we malloc(MDB_IDL_UM_MAX * = sizeof(OVERLAPPED)) for each environment, and retain it for the life of the= environment? I think that is 4MB, if my math is right, which seems like a = lot of memory to keep allocated (we usually have a lot of open environments= ). If the goal is to reduce the number of mallocs, how about we retain the = OVERLAPPED array, and only free and re-malloc if the previous allocation wa= sn=E2=80=99t large enough? Then there isn=E2=80=99t unnecessary allocation,= and we only malloc when there is a bigger transaction than any previous. I= put this together in a separate commit, as I wasn=E2=80=99t sure if this w= hat you wanted (can squash if you prefer): https://github.com/kriszyp/node-= lmdb/commit/2fe68fb5269c843e2e789746a17a4b2adefaac40<o:p> </o:p>Thank you for the review! <span= style=3D'font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#24292E;= background:white'><o:p></o:p><o:p> </o= :p>Thanks, Kris<o:p>&n= bsp;</o:p><div style=3D'mso-element:para-border-div;border:none;border-= top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'>From: <a href=3D"mailto:hyc@symas.com= ">Howard Chu</a> Sent: April 30, 2019 7:12 AM To: <a hr= ef=3D"mailto:kriszyp@gmail.com">kriszyp(a)gmail.com</a>; <a href=3D"mailto:op= enldap-its(a)OpenLDAP.org">openldap-its(a)OpenLDAP.org</a> Subject: R= e: (ITS#9017) Improving performance of commit sync in Windows</div><o:p> </o:p>kriszyp(a)gmail.co= m wrote:> Full_Name: Kristopher William Zyp= > Version: LMDB 0.9.23>= OS: Windows> URL: https://github.com/kriszyp/n= ode-lmdb/commit/7ff525ae57684a163d32af74a0ab9332b7fc4ce9> Submission from: (NULL) (71.199.6.148)&= gt; > > We have see= n very poor performance on the sync of commits on large databases in> Windows. On databases with 2GB of data, in writemap = mode, the sync of even small> commits is consis= tently well over 100ms (without writemap it is faster, but> still slow). It is expected that a sync should take some time = while waiting for> disk confirmation of the wri= tes, but more concerning is that these sync> op= erations (in writemap mode) are instead dominated by nearly 100% system CPU= > utilization, so operations that requires sub-= millisecond b-tree update> operations are then = dominated by very large amounts of system CPU cycles during> the sync phase.> > I think that the fundamental problem is that FlushViewOfFile = seems to be an O(n)> operation where n is the s= ize of the file (or map). I presume that Windows is> scanning the entire map/file for dirty pages to flush, I'm guessing b= ecause it> doesn't have an internal index of al= l the dirty pages for every file/map-view in> t= he OS disk cache. Therefore, the turns into an extremely expensive, CPU-bou= nd> operation to find the dirty pages for (larg= e file) and initiate their writes,> which, of c= ourse, is contrary to the whole goal of a scalable database system.> And FlushFileBuffers is also relatively slow as well.= We have attempted to batch> as many operations= into single transaction as possible, but this is still a very> large overhead.> > The Windows docs for FlushFileBuffers itself warns about = the inefficiencies of> this function (https://d= ocs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-flushfilebuf= fers).> Which also points to the solution: it i= s much faster to write out the dirty> pages wit= h WriteFile through a sync file handle (FILE_FLAG_WRITE_THROUGH).> > The associated patch<= p class=3DMsoNormal>> (https://github.com/kriszyp/node-lmdb/commit/7ff52= 5ae57684a163d32af74a0ab9332b7fc4ce9)> is my att= empt at implementing this solution, for Windows. Fortunately, with the<= p class=3DMsoNormal>> design of LMDB, this is relatively straightforward= . LMDB already supports> writing out dirty page= s with WriteFile calls. I added a write-through handle for> sending these writes directly to disk. I then made that file-h= andle> overlapped/asynchronously, so all the wr= ites for a commit could be started in> overlap = mode, and (at least theoretically) transfer in parallel to the drive and</p= >> then used GetOverlappedResult to wait for the co= mpletion. So basically> mdb_page_flush becomes = the sync. I extended the writing of dirty pages through> WriteFile to writemap mode as well (for writing meta too), so tha= t WriteFile> with write-through can be used to = flush the data without ever needing to call> Fl= ushViewOfFile or FlushFileBuffers. I also implemented support for write= > gathering in writemap mode where contiguous file = positions infers contiguous> memory (by trackin= g the starting position with wdp and writing contiguous pages> in single operations). Sorting of the dirty list is maint= ained even in writemap> mode for this purpose.<= /p><o:p> </o:p>What is t= he point of using writemap mode if you still need to use WriteFileon every individual page?<o:p>&nbs= p;</o:p>> The performance benefits of this patc= h, in my testing, are considerable. Writing> ou= t/syncing transactions is typically over 5x faster in writemap mode, and 2x= > faster in standard mode. And perhaps more imp= ortantly (especially in environment> with many = threads/processes), the efficiency benefits are even larger,> particularly in writemap mode, where there can be a 50-100x = reduction in the> system CPU usage by using thi= s patch. This brings windows performance with> = sync'ed transactions in LMDB back into the range of "lightning" p= erformance :).<o:p> </o:p>What is the performance difference between your patch using writemap,= and justnot using writemap in the first place?</p= ><o:p> </o:p>-- =C2=A0=C2=A0-- Howard Chu=C2=A0 C= TO, Symas Corp.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= http://www.symas.com=C2=A0 Director, Highland Sun= =C2=A0=C2=A0=C2=A0=C2=A0 http://highlandsun.com/hyc/=C2=A0 Chief Architect, OpenLDAP=C2=A0 http://www.openldap.org/project/</= p><o:p> </o:p></div></body></html>= --_F6B70E9A-5F12-495E-A3D8-F48F4F20717D_--

1 0

Re: (ITS#9017) Improving performance of commit sync in Windows
by hyc＠symas.com 30 Apr '19

30 Apr '19

kriszyp(a)gmail.com wrote: > Full_Name: Kristopher William Zyp > Version: LMDB 0.9.23 > OS: Windows > URL: https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab933… > Submission from: (NULL) (71.199.6.148) > > > We have seen very poor performance on the sync of commits on large databases in > Windows. On databases with 2GB of data, in writemap mode, the sync of even small > commits is consistently well over 100ms (without writemap it is faster, but > still slow). It is expected that a sync should take some time while waiting for > disk confirmation of the writes, but more concerning is that these sync > operations (in writemap mode) are instead dominated by nearly 100% system CPU > utilization, so operations that requires sub-millisecond b-tree update > operations are then dominated by very large amounts of system CPU cycles during > the sync phase. > > I think that the fundamental problem is that FlushViewOfFile seems to be an O(n) > operation where n is the size of the file (or map). I presume that Windows is > scanning the entire map/file for dirty pages to flush, I'm guessing because it > doesn't have an internal index of all the dirty pages for every file/map-view in > the OS disk cache. Therefore, the turns into an extremely expensive, CPU-bound > operation to find the dirty pages for (large file) and initiate their writes, > which, of course, is contrary to the whole goal of a scalable database system. > And FlushFileBuffers is also relatively slow as well. We have attempted to batch > as many operations into single transaction as possible, but this is still a very > large overhead. > > The Windows docs for FlushFileBuffers itself warns about the inefficiencies of > this function (https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-flu…). > Which also points to the solution: it is much faster to write out the dirty > pages with WriteFile through a sync file handle (FILE_FLAG_WRITE_THROUGH). > > The associated patch > (https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab933…) > is my attempt at implementing this solution, for Windows. Fortunately, with the > design of LMDB, this is relatively straightforward. LMDB already supports > writing out dirty pages with WriteFile calls. I added a write-through handle for > sending these writes directly to disk. I then made that file-handle > overlapped/asynchronously, so all the writes for a commit could be started in > overlap mode, and (at least theoretically) transfer in parallel to the drive and > then used GetOverlappedResult to wait for the completion. So basically > mdb_page_flush becomes the sync. I extended the writing of dirty pages through > WriteFile to writemap mode as well (for writing meta too), so that WriteFile > with write-through can be used to flush the data without ever needing to call > FlushViewOfFile or FlushFileBuffers. I also implemented support for write > gathering in writemap mode where contiguous file positions infers contiguous > memory (by tracking the starting position with wdp and writing contiguous pages > in single operations). Sorting of the dirty list is maintained even in writemap > mode for this purpose. What is the point of using writemap mode if you still need to use WriteFile on every individual page? > The performance benefits of this patch, in my testing, are considerable. Writing > out/syncing transactions is typically over 5x faster in writemap mode, and 2x > faster in standard mode. And perhaps more importantly (especially in environment > with many threads/processes), the efficiency benefits are even larger, > particularly in writemap mode, where there can be a 50-100x reduction in the > system CPU usage by using this patch. This brings windows performance with > sync'ed transactions in LMDB back into the range of "lightning" performance :). What is the performance difference between your patch using writemap, and just not using writemap in the first place? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs