openldap-bugs April 2019

openldap-bugs@openldap.org

16 participants
64 discussions

RE: (ITS#9017) Improving performance of commit sync in Windows
by kriszyp＠gmail.com 30 Apr '19

30 Apr '19

--_F6B70E9A-5F12-495E-A3D8-F48F4F20717D_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" > What is the point of using writemap mode if you still need to use WriteFi= le > on every individual page? As I understood from the documentation, and have observed, using writemap m= ode is faster (and uses less temporary memory) because it doesn=E2=80=99t r= equire mallocs to allocate pages (docs: =E2=80=9CThis is faster and uses fe= wer mallocs=E2=80=9D). To be clear though, LMDB is so incredibly fast and e= fficient, that in sync-mode, it takes enormous transactions before the time= spent allocating and creating the dirty pages with the updated b-tree is a= nywhere even remotely close to the time it takes to wait for disk flushing,= even with an SSD. But the more pertinent question is efficiency, and measu= ring CPU cycles rather than time spent (efficiency is more important than j= ust time spent). When I ran my tests this morning of 100 (sync) transaction= s with 100 puts per transaction, times varied quite a bit, but it seemed li= ke running with writemap enabled typically averages about 500ms of CPU and = with writemap disabled it typically averages around 600ms. Not a huge diffe= rence, but still definitely worthwhile, I think. Caveat emptor: Measuring LMDB performance with sync interactions on Windows= is one of the most frustratingly erratic things to measure. It is sunny ou= tside right now, times could be different when it starts raining later, but= , this is what I saw this morning... > What is the performance difference between your patch using writemap, and= just > not using writemap in the first place? Running 1000 sync transactions on 3GB db with a single put per transaction,= without writemap map, without the patch took about 60 seconds. And it took= about 1 second with the patch with writemap mode enabled! (there is no sig= nificant difference in sync times with writemap enabled or disabled with th= e patch.) So the difference was huge in my test. And not only that, without= the patch, the CPU usage was actually _higher_ during that 60 seconds (clo= se to 100% of a core) than during the execution with the patch for one seco= nd (close to 50%). Anyway, there are certainly tests I have run where the = differences are not as large (doing small commits on large dbs accentuates = the differences), but the patch always seems to win. It could also be that = my particular configuration causes bigger differences (on an SSD drive, and= maybe a more fragmented file?). Anyway, I added error handling for the malloc, and fixed/changed the other = things you suggested. Be happy to make any other changes you want. The upda= ted patch is here: https://github.com/kriszyp/node-lmdb/commit/25366dea9453749cf6637f43ec17b9b= 62094acde > OVERLAPPED* ov =3D malloc((pagecount - keep) * sizeof(OVERLAPPED)); > Probably this ought to just be pre-allocated based on the maximum number = of dirty pages a txn allows. I wasn=E2=80=99t sure I understood this comment. Are you suggesting we mall= oc(MDB_IDL_UM_MAX * sizeof(OVERLAPPED)) for each environment, and retain it= for the life of the environment? I think that is 4MB, if my math is right,= which seems like a lot of memory to keep allocated (we usually have a lot = of open environments). If the goal is to reduce the number of mallocs, how = about we retain the OVERLAPPED array, and only free and re-malloc if the pr= evious allocation wasn=E2=80=99t large enough? Then there isn=E2=80=99t unn= ecessary allocation, and we only malloc when there is a bigger transaction = than any previous. I put this together in a separate commit, as I wasn=E2= =80=99t sure if this what you wanted (can squash if you prefer): https://gi= thub.com/kriszyp/node-lmdb/commit/2fe68fb5269c843e2e789746a17a4b2adefaac40 Thank you for the review!=20 Thanks, Kris From: Howard Chu Sent: April 30, 2019 7:12 AM To: kriszyp(a)gmail.com; openldap-its(a)OpenLDAP.org Subject: Re: (ITS#9017) Improving performance of commit sync in Windows kriszyp(a)gmail.com wrote: > Full_Name: Kristopher William Zyp > Version: LMDB 0.9.23 > OS: Windows > URL: https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74= a0ab9332b7fc4ce9 > Submission from: (NULL) (71.199.6.148) >=20 >=20 > We have seen very poor performance on the sync of commits on large databa= ses in > Windows. On databases with 2GB of data, in writemap mode, the sync of eve= n small > commits is consistently well over 100ms (without writemap it is faster, b= ut > still slow). It is expected that a sync should take some time while waiti= ng for > disk confirmation of the writes, but more concerning is that these sync > operations (in writemap mode) are instead dominated by nearly 100% system= CPU > utilization, so operations that requires sub-millisecond b-tree update > operations are then dominated by very large amounts of system CPU cycles = during > the sync phase. >=20 > I think that the fundamental problem is that FlushViewOfFile seems to be = an O(n) > operation where n is the size of the file (or map). I presume that Window= s is > scanning the entire map/file for dirty pages to flush, I'm guessing becau= se it > doesn't have an internal index of all the dirty pages for every file/map-= view in > the OS disk cache. Therefore, the turns into an extremely expensive, CPU-= bound > operation to find the dirty pages for (large file) and initiate their wri= tes, > which, of course, is contrary to the whole goal of a scalable database sy= stem. > And FlushFileBuffers is also relatively slow as well. We have attempted t= o batch > as many operations into single transaction as possible, but this is still= a very > large overhead. >=20 > The Windows docs for FlushFileBuffers itself warns about the inefficienci= es of > this function (https://docs.microsoft.com/en-us/windows/desktop/api/filea= pi/nf-fileapi-flushfilebuffers). > Which also points to the solution: it is much faster to write out the dir= ty > pages with WriteFile through a sync file handle (FILE_FLAG_WRITE_THROUGH)= . >=20 > The associated patch > (https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab= 9332b7fc4ce9) > is my attempt at implementing this solution, for Windows. Fortunately, wi= th the > design of LMDB, this is relatively straightforward. LMDB already supports > writing out dirty pages with WriteFile calls. I added a write-through han= dle for > sending these writes directly to disk. I then made that file-handle > overlapped/asynchronously, so all the writes for a commit could be starte= d in > overlap mode, and (at least theoretically) transfer in parallel to the dr= ive and > then used GetOverlappedResult to wait for the completion. So basically > mdb_page_flush becomes the sync. I extended the writing of dirty pages th= rough > WriteFile to writemap mode as well (for writing meta too), so that WriteF= ile > with write-through can be used to flush the data without ever needing to = call > FlushViewOfFile or FlushFileBuffers. I also implemented support for write > gathering in writemap mode where contiguous file positions infers contigu= ous > memory (by tracking the starting position with wdp and writing contiguous= pages > in single operations). Sorting of the dirty list is maintained even in wr= itemap > mode for this purpose. What is the point of using writemap mode if you still need to use WriteFile on every individual page? > The performance benefits of this patch, in my testing, are considerable. = Writing > out/syncing transactions is typically over 5x faster in writemap mode, an= d 2x > faster in standard mode. And perhaps more importantly (especially in envi= ronment > with many threads/processes), the efficiency benefits are even larger, > particularly in writemap mode, where there can be a 50-100x reduction in = the > system CPU usage by using this patch. This brings windows performance wit= h > sync'ed transactions in LMDB back into the range of "lightning" performan= ce :). What is the performance difference between your patch using writemap, and j= ust not using writemap in the first place? --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ --_F6B70E9A-5F12-495E-A3D8-F48F4F20717D_ Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8" <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta ht= tp-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta name= =3DGenerator content=3D"Microsoft Word 15 (filtered medium)"><style></style></head><body lang=3DEN-CA link=3Dblue vlink=3D"#954F72"><div cla= ss=3DWordSection1><p class=3DMsoNormal>> What is the point of using writ= emap mode if you still need to use WriteFile<o:p></o:p></p><p class=3DMsoNo= rmal>> on every individual page?</p><p class=3DMsoNormal><o:p> </o:= p></p><p class=3DMsoNormal>As I understood from the documentation, and have= observed, using writemap mode is faster (and uses less temporary memory) b= ecause it doesn=E2=80=99t require mallocs to allocate pages (docs: =E2=80= =9CThis is faster and uses fewer mallocs=E2=80=9D). To be clear though, LMD= B is so incredibly fast and efficient, that in sync-mode, it takes enormous= transactions before the time spent allocating and creating the dirty pages= with the updated b-tree is anywhere even remotely close to the time it tak= es to wait for disk flushing, even with an SSD. But the more pertinent ques= tion is efficiency, and measuring CPU cycles rather than time spent (effici= ency is more important than just time spent). When I ran my tests this morn= ing of 100 (sync) transactions with 100 puts per transaction, times varied = quite a bit, but it seemed like running with writemap enabled typically ave= rages about 500ms of CPU and with writemap disabled it typically averages a= round 600ms. Not a huge difference, but still definitely worthwhile, I thin= k.</p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>Caveat= emptor: Measuring LMDB performance with sync interactions on Windows is on= e of the most frustratingly erratic things to measure. It is sunny outside = right now, times could be different when it starts raining later, but, this= is what I saw this morning...<o:p></o:p></p><p class=3DMsoNormal><o:p>&nbs= p;</o:p></p><p class=3DMsoNormal>> What is the performance difference be= tween your patch using writemap, and just<o:p></o:p></p><p class=3DMsoNorma= l>> not using writemap in the first place?<o:p></o:p></p><p class=3DMsoN= ormal><o:p> </o:p></p><p class=3DMsoNormal>Running 1000 sync transacti= ons on 3GB db with a single put per transaction, without writemap map, with= out the patch took about 60 seconds. And it took about 1 second with the pa= tch with writemap mode enabled! (there is no significant difference in sync= times with writemap enabled or disabled with the patch.) So the difference= was huge in my test. And not only that, without the patch, the CPU usage w= as actually _<i>higher</i>_ during that 60 seconds (close to 100% of a core= ) than during the execution with the patch for one second (close to 50%). = =C2=A0Anyway, there are certainly tests I have run where the differences ar= e not as large (doing small commits on large dbs accentuates the difference= s), but the patch always seems to win. It could also be that my particular = configuration causes bigger differences (on an SSD drive, and maybe a more = fragmented file?).</p><p class=3DMsoNormal><o:p> </o:p></p><p class=3D= MsoNormal>Anyway, I added error handling for the malloc, and fixed/changed = the other things you suggested. Be happy to make any other changes you want= . The updated patch is here:<o:p></o:p></p><p class=3DMsoNormal>https://git= hub.com/kriszyp/node-lmdb/commit/25366dea9453749cf6637f43ec17b9b62094acde<o= :p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal= >><span class=3Dblob-code-inner><span style=3D'font-size:9.0pt;font-fami= ly:Consolas;color:#24292E'> OVERLAPPED* ov =3D </span></span><span class=3D= pl-c1><span style=3D'font-size:9.0pt;font-family:Consolas;color:#005CC5'>ma= lloc</span></span><span class=3Dblob-code-inner><span style=3D'font-size:9.= 0pt;font-family:Consolas;color:#24292E'>((pagecount - keep) * </span></span= ><span class=3Dpl-k><span style=3D'font-size:9.0pt;font-family:Consolas;col= or:#D73A49'>sizeof</span></span><span class=3Dblob-code-inner><span style= =3D'font-size:9.0pt;font-family:Consolas;color:#24292E'>(OVERLAPPED));<o:p>= </o:p></span></span></p><p class=3DMsoNormal><span class=3Dblob-code-inner>= <span style=3D'font-size:9.0pt;font-family:Consolas;color:#24292E'>> </s= pan></span><span style=3D'font-size:10.5pt;font-family:"Segoe UI",sans-seri= f;color:#24292E;background:white'>Probably this ought to just be pre-alloca= ted based on the maximum number of dirty pages a txn allows.<o:p></o:p></sp= an></p><p class=3DMsoNormal><span style=3D'font-size:10.5pt;font-family:"Se= goe UI",sans-serif;color:#24292E;background:white'><o:p> </o:p></span>= </p><p class=3DMsoNormal><span style=3D'font-size:10.5pt;font-family:"Segoe= UI",sans-serif;color:#24292E;background:white'>I wasn=E2=80=99t sure I und= erstood this comment. Are you suggesting we </span>malloc(MDB_IDL_UM_MAX * = sizeof(OVERLAPPED)) for each environment, and retain it for the life of the= environment? I think that is 4MB, if my math is right, which seems like a = lot of memory to keep allocated (we usually have a lot of open environments= ). If the goal is to reduce the number of mallocs, how about we retain the = OVERLAPPED array, and only free and re-malloc if the previous allocation wa= sn=E2=80=99t large enough? Then there isn=E2=80=99t unnecessary allocation,= and we only malloc when there is a bigger transaction than any previous. I= put this together in a separate commit, as I wasn=E2=80=99t sure if this w= hat you wanted (can squash if you prefer): https://github.com/kriszyp/node-= lmdb/commit/2fe68fb5269c843e2e789746a17a4b2adefaac40</p><p class=3DMsoNorma= l><o:p> </o:p></p><p class=3DMsoNormal>Thank you for the review! <span= style=3D'font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#24292E;= background:white'><o:p></o:p></span></p><p class=3DMsoNormal><o:p> </o= :p></p><p class=3DMsoNormal>Thanks,<br>Kris</p><p class=3DMsoNormal><o:p>&n= bsp;</o:p></p><div style=3D'mso-element:para-border-div;border:none;border-= top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=3DMsoNormal sty= le=3D'border:none;padding:0cm'><b>From: </b><a href=3D"mailto:hyc@symas.com= ">Howard Chu</a><br><b>Sent: </b>April 30, 2019 7:12 AM<br><b>To: </b><a hr= ef=3D"mailto:kriszyp@gmail.com">kriszyp(a)gmail.com</a>; <a href=3D"mailto:op= enldap-its(a)OpenLDAP.org">openldap-its(a)OpenLDAP.org</a><br><b>Subject: </b>R= e: (ITS#9017) Improving performance of commit sync in Windows</p></div><p c= lass=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>kriszyp(a)gmail.co= m wrote:</p><p class=3DMsoNormal>> Full_Name: Kristopher William Zyp</p>= <p class=3DMsoNormal>> Version: LMDB 0.9.23</p><p class=3DMsoNormal>>= OS: Windows</p><p class=3DMsoNormal>> URL: https://github.com/kriszyp/n= ode-lmdb/commit/7ff525ae57684a163d32af74a0ab9332b7fc4ce9</p><p class=3DMsoN= ormal>> Submission from: (NULL) (71.199.6.148)</p><p class=3DMsoNormal>&= gt; </p><p class=3DMsoNormal>> </p><p class=3DMsoNormal>> We have see= n very poor performance on the sync of commits on large databases in</p><p = class=3DMsoNormal>> Windows. On databases with 2GB of data, in writemap = mode, the sync of even small</p><p class=3DMsoNormal>> commits is consis= tently well over 100ms (without writemap it is faster, but</p><p class=3DMs= oNormal>> still slow). It is expected that a sync should take some time = while waiting for</p><p class=3DMsoNormal>> disk confirmation of the wri= tes, but more concerning is that these sync</p><p class=3DMsoNormal>> op= erations (in writemap mode) are instead dominated by nearly 100% system CPU= </p><p class=3DMsoNormal>> utilization, so operations that requires sub-= millisecond b-tree update</p><p class=3DMsoNormal>> operations are then = dominated by very large amounts of system CPU cycles during</p><p class=3DM= soNormal>> the sync phase.</p><p class=3DMsoNormal>> </p><p class=3DM= soNormal>> I think that the fundamental problem is that FlushViewOfFile = seems to be an O(n)</p><p class=3DMsoNormal>> operation where n is the s= ize of the file (or map). I presume that Windows is</p><p class=3DMsoNormal= >> scanning the entire map/file for dirty pages to flush, I'm guessing b= ecause it</p><p class=3DMsoNormal>> doesn't have an internal index of al= l the dirty pages for every file/map-view in</p><p class=3DMsoNormal>> t= he OS disk cache. Therefore, the turns into an extremely expensive, CPU-bou= nd</p><p class=3DMsoNormal>> operation to find the dirty pages for (larg= e file) and initiate their writes,</p><p class=3DMsoNormal>> which, of c= ourse, is contrary to the whole goal of a scalable database system.</p><p c= lass=3DMsoNormal>> And FlushFileBuffers is also relatively slow as well.= We have attempted to batch</p><p class=3DMsoNormal>> as many operations= into single transaction as possible, but this is still a very</p><p class= =3DMsoNormal>> large overhead.</p><p class=3DMsoNormal>> </p><p class= =3DMsoNormal>> The Windows docs for FlushFileBuffers itself warns about = the inefficiencies of</p><p class=3DMsoNormal>> this function (https://d= ocs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-flushfilebuf= fers).</p><p class=3DMsoNormal>> Which also points to the solution: it i= s much faster to write out the dirty</p><p class=3DMsoNormal>> pages wit= h WriteFile through a sync file handle (FILE_FLAG_WRITE_THROUGH).</p><p cla= ss=3DMsoNormal>> </p><p class=3DMsoNormal>> The associated patch</p><= p class=3DMsoNormal>> (https://github.com/kriszyp/node-lmdb/commit/7ff52= 5ae57684a163d32af74a0ab9332b7fc4ce9)</p><p class=3DMsoNormal>> is my att= empt at implementing this solution, for Windows. Fortunately, with the</p><= p class=3DMsoNormal>> design of LMDB, this is relatively straightforward= . LMDB already supports</p><p class=3DMsoNormal>> writing out dirty page= s with WriteFile calls. I added a write-through handle for</p><p class=3DMs= oNormal>> sending these writes directly to disk. I then made that file-h= andle</p><p class=3DMsoNormal>> overlapped/asynchronously, so all the wr= ites for a commit could be started in</p><p class=3DMsoNormal>> overlap = mode, and (at least theoretically) transfer in parallel to the drive and</p= ><p class=3DMsoNormal>> then used GetOverlappedResult to wait for the co= mpletion. So basically</p><p class=3DMsoNormal>> mdb_page_flush becomes = the sync. I extended the writing of dirty pages through</p><p class=3DMsoNo= rmal>> WriteFile to writemap mode as well (for writing meta too), so tha= t WriteFile</p><p class=3DMsoNormal>> with write-through can be used to = flush the data without ever needing to call</p><p class=3DMsoNormal>> Fl= ushViewOfFile or FlushFileBuffers. I also implemented support for write</p>= <p class=3DMsoNormal>> gathering in writemap mode where contiguous file = positions infers contiguous</p><p class=3DMsoNormal>> memory (by trackin= g the starting position with wdp and writing contiguous pages</p><p class= =3DMsoNormal>> in single operations). Sorting of the dirty list is maint= ained even in writemap</p><p class=3DMsoNormal>> mode for this purpose.<= /p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>What is t= he point of using writemap mode if you still need to use WriteFile</p><p cl= ass=3DMsoNormal>on every individual page?</p><p class=3DMsoNormal><o:p>&nbs= p;</o:p></p><p class=3DMsoNormal>> The performance benefits of this patc= h, in my testing, are considerable. Writing</p><p class=3DMsoNormal>> ou= t/syncing transactions is typically over 5x faster in writemap mode, and 2x= </p><p class=3DMsoNormal>> faster in standard mode. And perhaps more imp= ortantly (especially in environment</p><p class=3DMsoNormal>> with many = threads/processes), the efficiency benefits are even larger,</p><p class=3D= MsoNormal>> particularly in writemap mode, where there can be a 50-100x = reduction in the</p><p class=3DMsoNormal>> system CPU usage by using thi= s patch. This brings windows performance with</p><p class=3DMsoNormal>> = sync'ed transactions in LMDB back into the range of "lightning" p= erformance :).</p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoN= ormal>What is the performance difference between your patch using writemap,= and just</p><p class=3DMsoNormal>not using writemap in the first place?</p= ><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>-- </p><p c= lass=3DMsoNormal>=C2=A0=C2=A0-- Howard Chu</p><p class=3DMsoNormal>=C2=A0 C= TO, Symas Corp.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= http://www.symas.com</p><p class=3DMsoNormal>=C2=A0 Director, Highland Sun= =C2=A0=C2=A0=C2=A0=C2=A0 http://highlandsun.com/hyc/</p><p class=3DMsoNorma= l>=C2=A0 Chief Architect, OpenLDAP=C2=A0 http://www.openldap.org/project/</= p><p class=3DMsoNormal><o:p> </o:p></p></div></body></html>= --_F6B70E9A-5F12-495E-A3D8-F48F4F20717D_--

1 0

Re: (ITS#9017) Improving performance of commit sync in Windows
by hyc＠symas.com 30 Apr '19

30 Apr '19

kriszyp(a)gmail.com wrote: > Full_Name: Kristopher William Zyp > Version: LMDB 0.9.23 > OS: Windows > URL: https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab933… > Submission from: (NULL) (71.199.6.148) > > > We have seen very poor performance on the sync of commits on large databases in > Windows. On databases with 2GB of data, in writemap mode, the sync of even small > commits is consistently well over 100ms (without writemap it is faster, but > still slow). It is expected that a sync should take some time while waiting for > disk confirmation of the writes, but more concerning is that these sync > operations (in writemap mode) are instead dominated by nearly 100% system CPU > utilization, so operations that requires sub-millisecond b-tree update > operations are then dominated by very large amounts of system CPU cycles during > the sync phase. > > I think that the fundamental problem is that FlushViewOfFile seems to be an O(n) > operation where n is the size of the file (or map). I presume that Windows is > scanning the entire map/file for dirty pages to flush, I'm guessing because it > doesn't have an internal index of all the dirty pages for every file/map-view in > the OS disk cache. Therefore, the turns into an extremely expensive, CPU-bound > operation to find the dirty pages for (large file) and initiate their writes, > which, of course, is contrary to the whole goal of a scalable database system. > And FlushFileBuffers is also relatively slow as well. We have attempted to batch > as many operations into single transaction as possible, but this is still a very > large overhead. > > The Windows docs for FlushFileBuffers itself warns about the inefficiencies of > this function (https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-flu…). > Which also points to the solution: it is much faster to write out the dirty > pages with WriteFile through a sync file handle (FILE_FLAG_WRITE_THROUGH). > > The associated patch > (https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab933…) > is my attempt at implementing this solution, for Windows. Fortunately, with the > design of LMDB, this is relatively straightforward. LMDB already supports > writing out dirty pages with WriteFile calls. I added a write-through handle for > sending these writes directly to disk. I then made that file-handle > overlapped/asynchronously, so all the writes for a commit could be started in > overlap mode, and (at least theoretically) transfer in parallel to the drive and > then used GetOverlappedResult to wait for the completion. So basically > mdb_page_flush becomes the sync. I extended the writing of dirty pages through > WriteFile to writemap mode as well (for writing meta too), so that WriteFile > with write-through can be used to flush the data without ever needing to call > FlushViewOfFile or FlushFileBuffers. I also implemented support for write > gathering in writemap mode where contiguous file positions infers contiguous > memory (by tracking the starting position with wdp and writing contiguous pages > in single operations). Sorting of the dirty list is maintained even in writemap > mode for this purpose. What is the point of using writemap mode if you still need to use WriteFile on every individual page? > The performance benefits of this patch, in my testing, are considerable. Writing > out/syncing transactions is typically over 5x faster in writemap mode, and 2x > faster in standard mode. And perhaps more importantly (especially in environment > with many threads/processes), the efficiency benefits are even larger, > particularly in writemap mode, where there can be a 50-100x reduction in the > system CPU usage by using this patch. This brings windows performance with > sync'ed transactions in LMDB back into the range of "lightning" performance :). What is the performance difference between your patch using writemap, and just not using writemap in the first place? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9017) Improving performance of commit sync in Windows
by kriszyp＠gmail.com 30 Apr '19

30 Apr '19

Full_Name: Kristopher William Zyp Version: LMDB 0.9.23 OS: Windows URL: https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab933… Submission from: (NULL) (71.199.6.148) We have seen very poor performance on the sync of commits on large databases in Windows. On databases with 2GB of data, in writemap mode, the sync of even small commits is consistently well over 100ms (without writemap it is faster, but still slow). It is expected that a sync should take some time while waiting for disk confirmation of the writes, but more concerning is that these sync operations (in writemap mode) are instead dominated by nearly 100% system CPU utilization, so operations that requires sub-millisecond b-tree update operations are then dominated by very large amounts of system CPU cycles during the sync phase. I think that the fundamental problem is that FlushViewOfFile seems to be an O(n) operation where n is the size of the file (or map). I presume that Windows is scanning the entire map/file for dirty pages to flush, I'm guessing because it doesn't have an internal index of all the dirty pages for every file/map-view in the OS disk cache. Therefore, the turns into an extremely expensive, CPU-bound operation to find the dirty pages for (large file) and initiate their writes, which, of course, is contrary to the whole goal of a scalable database system. And FlushFileBuffers is also relatively slow as well. We have attempted to batch as many operations into single transaction as possible, but this is still a very large overhead. The Windows docs for FlushFileBuffers itself warns about the inefficiencies of this function (https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-flu…). Which also points to the solution: it is much faster to write out the dirty pages with WriteFile through a sync file handle (FILE_FLAG_WRITE_THROUGH). The associated patch (https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab933…) is my attempt at implementing this solution, for Windows. Fortunately, with the design of LMDB, this is relatively straightforward. LMDB already supports writing out dirty pages with WriteFile calls. I added a write-through handle for sending these writes directly to disk. I then made that file-handle overlapped/asynchronously, so all the writes for a commit could be started in overlap mode, and (at least theoretically) transfer in parallel to the drive and then used GetOverlappedResult to wait for the completion. So basically mdb_page_flush becomes the sync. I extended the writing of dirty pages through WriteFile to writemap mode as well (for writing meta too), so that WriteFile with write-through can be used to flush the data without ever needing to call FlushViewOfFile or FlushFileBuffers. I also implemented support for write gathering in writemap mode where contiguous file positions infers contiguous memory (by tracking the starting position with wdp and writing contiguous pages in single operations). Sorting of the dirty list is maintained even in writemap mode for this purpose. The performance benefits of this patch, in my testing, are considerable. Writing out/syncing transactions is typically over 5x faster in writemap mode, and 2x faster in standard mode. And perhaps more importantly (especially in environment with many threads/processes), the efficiency benefits are even larger, particularly in writemap mode, where there can be a 50-100x reduction in the system CPU usage by using this patch. This brings windows performance with sync'ed transactions in LMDB back into the range of "lightning" performance :). All of these changes in the associated patch should only affect Windows. I actually had started with the approach of using a flag to indicate write-through behavior (here https://github.com/kriszyp/node-lmdb/commit/435ca423d0e13936f2a5f0193e994f5… if anyone wants to test it or play with it further). However, I didn't really see any substantive improvements in unix. It is possible that maybe with the use of dsync writes that are done asynchronously in parallel and wait for the completion (aio_write/aio_suspend), there might be a performance benefit opportunity, but I was less interested in this, and I assume you probably have already thoroughly explored these options already. In the end, I think it makes more sense to just make this the default behavior for Windows where the major improvement is, and don't change the unix behavior. Also, I don't think you can write to files that have an open writeable mapped region, so the implementation would be limited on unix anyway. Anyway, this is certainly a more involved patch, in a sophisticated code-base, so I humbly present this for consideration. I have tested both the performance and the sync safety of this code. Our application has a high enough intensity of db interaction that it is actually pretty easy to reproduce LMDB data corruption by powering off a VM with the app running, if sync-mode is not enabled. And with my testing so far, sync-mode with this patch seems to preserve the rock-solid crash-proof design of LMDB. But I'd be glad to make any changes or cleanup needed, or if you want the patch to be submitted differently (it seems like using patches from my node repo fork has worked in the past though).

1 0

Re: (ITS#7632) ldap_start_tls_s core dumps for 64 bit
by quanah＠symas.com 30 Apr '19

30 Apr '19

--On Wednesday, June 26, 2013 6:20 AM +0000 a16474(a)gmail.com wrote: > Full_Name: Amit Sinha > Version: openldap-2.4.35 > OS: Linux 2.6.18-308.11.1.el5 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (72.163.217.105) > > > When i compile and run the below code on linux 64 bit, it core dumps in > ldap_start_tls_s(). Hello, Thank you for the report! I apologize for the (very very long) delay in response. This issue no longer seems to occur with current OpenLDAP/OpenSSL releases (nor does it happen with OpenLDAP/GnuTLS). I believe one of the many fixes to OpenLDAP's TLS code has since resolved this issue. I tested with both OpenSSL 1.0.2k and 1.1.1b. This ITS will be closed. Warm regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#9015) Replication goes haywire querying promoted master
by ondra＠mistotebe.net 26 Apr '19

26 Apr '19

On Wed, Apr 24, 2019 at 01:30:18PM +0000, ondra(a)mistotebe.net wrote: > On Tue, Apr 23, 2019 at 11:28:40PM +0000, quanah(a)symas.com wrote: >> The consumer did not replicate the database as a part of the initial >> search, so I don't think it qualifies as "up to date". I.e., it's primary >> DB remains empty. > > Ah, that wasn't mentioned, my assumption was that the source DB was > actually empty. > > This is coming from ITS#8281 fix in cd8ff37629012c1676ef79de164a159da9b2ae89. I've pushed a change that makes sure we end up with a contexCSN, generating one if necessary whenever we're configured to be a master (olcMirrorMode is set or no olcSyncrepl exists) here: https://github.com/mistotebe/openldap/tree/its9015 It's kind of a followup on this (reverting part of ITS#8281 above): https://www.openldap.org/lists/openldap-devel/201904/msg00015.html -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

1 0

Re: (ITS#9014) OpenLDAP modifies user provided TLS certificate before sending it to client
by siddjain＠live.com 24 Apr '19

24 Apr '19

--_000_MWHPR08MB2400D7AE5E8EEC3D17192FACB53C0MWHPR08MB2400namp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thank you. we tried using another openldap image and that worked. so it see= ms the problem is with the osixia docker image we were using to run openlda= p. it is based on debian (which uses GnuTLS per your email) so tbh we are s= urprised it would have such a bug in it. the image that worked for us is ba= sed on alpine. https://github.com/osixia/docker-light-baseimage/blob/stable/image/Dockerfi= le https://github.com/tiredofit/docker-openldap/blob/master/Dockerfile but back to your comment, how can one isolate what TLS/SSL library OpenLDAP= is linked to in the environment you're using? [https://avatars0.githubusercontent.com/u/23528985?s=3D400&v=3D4]<https://g= ithub.com/tiredofit/docker-openldap/blob/master/Dockerfile> docker-openldap/Dockerfile at master =B7 tiredofit/docker-openldap =B7 GitH= ub<https://github.com/tiredofit/docker-openldap/blob/master/Dockerfile> Docker OpenLDAP Container w/TLS & Replication Support S6 Overlay, and Zabbi= x Monitoring based on Alpine - tiredofit/docker-openldap github.com [https://avatars0.githubusercontent.com/u/23528985?s=3D400&v=3D4]<https://g= ithub.com/tiredofit/docker-openldap/blob/master/Dockerfile> docker-openldap/Dockerfile at master =B7 tiredofit/docker-openldap =B7 GitH= ub<https://github.com/tiredofit/docker-openldap/blob/master/Dockerfile> Docker OpenLDAP Container w/TLS & Replication Support S6 Overlay, and Zabbi= x Monitoring based on Alpine - tiredofit/docker-openldap github.com ________________________________ From: Quanah Gibson-Mount <quanah(a)symas.com> Sent: Wednesday, April 24, 2019 1:06 PM To: siddjain(a)live.com; openldap-its(a)OpenLDAP.org Subject: Re: (ITS#9014) OpenLDAP modifies user provided TLS certificate bef= ore sending it to client --On Wednesday, April 24, 2019 6:43 PM +0000 hyc(a)symas.com wrote: > siddjain(a)live.com wrote: >> --_000_MWHPR08MB24000D77048AFCF7465C4397B53C0MWHPR08MB2400namp_ >> Content-Type: text/plain; charset=3D"iso-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> could you send me output of running >> >> openssl version -a >> >> on your system? thanks > >> openssl version -a > OpenSSL 1.1.1 11 Sep 2018 > built on: Tue Dec 4 13:15:09 2018 UTC > platform: debian-amd64 I would also note that not all OpenLDAP builds use OpenSSL. For example, OpenLDAP built on Debian/Ubuntu uses GnuTLS. OpenLDAP built on some versions of RedHat 7 use MozNSS. Current RedHat 7 builds use OpenSSL but have an odd MozNSS bridge for backwards compatibilty, and there may be all sorts of odd bugs in that. Apple links OpenLDAP to its own custom SSL libary. So really your first step should be isolating what TLS/SSL library OpenLDAP is linked to in the environment you're using. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <https://eur01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fwww.sym= as.com&data=3D02%7C01%7C%7C349b90be6afe4991a54b08d6c8f068b4%7C84df9e7fe= 9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917332202918260&sdata=3DNifWEVt269= tCTuar98XYUfNkaHWSFMffI3M4%2FJ7j8zI%3D&reserved=3D0> --_000_MWHPR08MB2400D7AE5E8EEC3D17192FACB53C0MWHPR08MB2400namp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style type=3D"text/css" style=3D"display:none;"> P {margin-top:0;margin-bo= ttom:0;} </style> </head> <body dir=3D"ltr"> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;= color: rgb(0, 0, 0);"> Thank you. we tried using another openldap image and that worked. so it see= ms the problem is with the osixia docker image we were using to run openlda= p. it is based on debian (which <span style=3D"color: rgb(51, 51, 51);= font-family: "Segoe UI", "Segoe UI Web (West European)&quot= ;, "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica = Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 25= 5, 255); display: inline !important">uses GnuTLS per your email</span>) so tbh we are surprised it would have such a= bug in it. the image that worked for us is based on alpine. </div> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;= color: rgb(0, 0, 0);"> <a href=3D"https://github.com/osixia/docker-light-baseimage/blob/stable/ima= ge/Dockerfile">https://github.com/osixia/docker-light-baseimage/blob/stable= /image/Dockerfile</a><br> </div> <div> <div id=3D"appendonsend"></div> <div style=3D"font-family:Calibri,Helvetica,sans-serif; font-size:12pt; col= or:rgb(0,0,0)"> <a href=3D"https://github.com/tiredofit/docker-openldap/blob/master/Dockerf= ile" id=3D"LPlnk147820">https://github.com/tiredofit/docker-openldap/blob/m= aster/Dockerfile</a></div> <div style=3D"font-family:Calibri,Helvetica,sans-serif; font-size:12pt; col= or:rgb(0,0,0)"> but back to your comment, how can one <span style=3D"color: rgb(51, 51= , 51); font-family: "Segoe UI", "Segoe UI Web (West European= )", "Segoe UI", -apple-system, system-ui, Roboto, "Helv= etica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(2= 55, 255, 255); display: inline !important">isolate what TLS/SSL library OpenLDAP<span> </span></span><span style=3D"colo= r: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (= West European)", "Segoe UI", -apple-system, system-ui, Robot= o, "Helvetica Neue", sans-serif; font-size: 14.6667px; background= -color: rgb(255, 255, 255); display: inline !important">is linked to in the environment you're using? </span><br style=3D"color:= rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (We= st European)", "Segoe UI", -apple-system, system-ui, Roboto,= "Helvetica Neue", sans-serif; font-size: 14.6667px; background-c= olor: rgb(255, 255, 255)"> <br> <div id=3D"LPBorder_GTaHR0cHM6Ly9naXRodWIuY29tL3RpcmVkb2ZpdC9kb2NrZXItb3Blb= mxkYXAvYmxvYi9tYXN0ZXIvRG9ja2VyZmlsZQ.." class=3D"LPBorder213343" contented= itable=3D"false" style=3D"width: 100%; margin-top: 16px; margin-bottom: 16p= x; position: relative; max-width: 800px; min-width: 424px;"> <table id=3D"LPContainer213343" role=3D"presentation" style=3D"padding: 12p= x 36px 12px 12px; width: 100%; border-width: 1px; border-style: solid; bord= er-color: rgb(200, 200, 200); border-radius: 2px;"> <tbody> <tr valign=3D"top" style=3D"border-spacing: 0px;"> <td> <div id=3D"LPImageContainer213343" style=3D"position: relative; margin-righ= t: 12px; height: 160px; overflow: hidden;"> <a target=3D"_blank" id=3D"LPImageAnchor213343" href=3D"https://github.com/= tiredofit/docker-openldap/blob/master/Dockerfile"><img id=3D"LPThumbnailIma= geId213343" alt=3D"" height=3D"160" style=3D"display: block;" width=3D"160"= src=3D"https://avatars0.githubusercontent.com/u/23528985?s=3D400&v=3D4= "></a></div> </td> <td style=3D"width: 100%;"> <div id=3D"LPTitle213343" style=3D"font-size: 21px; font-weight: 300; margi= n-right: 8px; font-family: wf_segoe-ui_light, "Segoe UI Light", &= quot;Segoe WP Light", "Segoe UI", "Segoe WP", Taho= ma, Arial, sans-serif; margin-bottom: 12px;"> <a target=3D"_blank" id=3D"LPUrlAnchor213343" href=3D"https://github.com/ti= redofit/docker-openldap/blob/master/Dockerfile" style=3D"text-decoration: n= one; color: var(--themePrimary);">docker-openldap/Dockerfile at master =B7 = tiredofit/docker-openldap =B7 GitHub</a></div> <div id=3D"LPDescription213343" style=3D"font-size: 14px; max-height: 100px= ; color: rgb(102, 102, 102); font-family: wf_segoe-ui_normal, "Segoe U= I", "Segoe WP", Tahoma, Arial, sans-serif; margin-bottom: 12= px; margin-right: 8px; overflow: hidden;"> Docker OpenLDAP Container w/TLS & Replication Support S6 Overlay, and Z= abbix Monitoring based on Alpine - tiredofit/docker-openldap</div> <div id=3D"LPMetadata213343" style=3D"font-size: 14px; font-weight: 400; co= lor: rgb(166, 166, 166); font-family: wf_segoe-ui_normal, "Segoe UI&qu= ot;, "Segoe WP", Tahoma, Arial, sans-serif;"> github.com</div> </td> </tr> </tbody> </table> </div> <br> <div id=3D"LPBorder_GTaHR0cHM6Ly9naXRodWIuY29tL3RpcmVkb2ZpdC9kb2NrZXItb3Blb= mxkYXAvYmxvYi9tYXN0ZXIvRG9ja2VyZmlsZQ.." class=3D"LPBorder356508" contented= itable=3D"false" style=3D"width: 100%; margin-top: 16px; margin-bottom: 16p= x; position: relative; max-width: 800px; min-width: 424px;"> <table id=3D"LPContainer356508" role=3D"presentation" style=3D"padding: 12p= x 36px 12px 12px; width: 100%; border-width: 1px; border-style: solid; bord= er-color: rgb(200, 200, 200); border-radius: 2px;"> <tbody> <tr valign=3D"top" style=3D"border-spacing: 0px;"> <td> <div id=3D"LPImageContainer356508" style=3D"position: relative; margin-righ= t: 12px; height: 160px; overflow: hidden;"> <a target=3D"_blank" id=3D"LPImageAnchor356508" href=3D"https://github.com/= tiredofit/docker-openldap/blob/master/Dockerfile"><img id=3D"LPThumbnailIma= geId356508" alt=3D"" height=3D"160" style=3D"display: block;" width=3D"160"= src=3D"https://avatars0.githubusercontent.com/u/23528985?s=3D400&v=3D4= "></a></div> </td> <td style=3D"width: 100%;"> <div id=3D"LPTitle356508" style=3D"font-size: 21px; font-weight: 300; margi= n-right: 8px; font-family: wf_segoe-ui_light, "Segoe UI Light", &= quot;Segoe WP Light", "Segoe UI", "Segoe WP", Taho= ma, Arial, sans-serif; margin-bottom: 12px;"> <a target=3D"_blank" id=3D"LPUrlAnchor356508" href=3D"https://github.com/ti= redofit/docker-openldap/blob/master/Dockerfile" style=3D"text-decoration: n= one; color: var(--themePrimary);">docker-openldap/Dockerfile at master =B7 = tiredofit/docker-openldap =B7 GitHub</a></div> <div id=3D"LPDescription356508" style=3D"font-size: 14px; max-height: 100px= ; color: rgb(102, 102, 102); font-family: wf_segoe-ui_normal, "Segoe U= I", "Segoe WP", Tahoma, Arial, sans-serif; margin-bottom: 12= px; margin-right: 8px; overflow: hidden;"> Docker OpenLDAP Container w/TLS & Replication Support S6 Overlay, and Z= abbix Monitoring based on Alpine - tiredofit/docker-openldap</div> <div id=3D"LPMetadata356508" style=3D"font-size: 14px; font-weight: 400; co= lor: rgb(166, 166, 166); font-family: wf_segoe-ui_normal, "Segoe UI&qu= ot;, "Segoe WP", Tahoma, Arial, sans-serif;"> github.com</div> </td> </tr> </tbody> </table> </div> <br> </div> <hr tabindex=3D"-1" style=3D"display:inline-block; width:98%"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" co= lor=3D"#000000" style=3D"font-size:11pt"><b>From:</b> Quanah Gibson-Mount &= lt;quanah(a)symas.com><br> <b>Sent:</b> Wednesday, April 24, 2019 1:06 PM<br> <b>To:</b> siddjain(a)live.com; openldap-its(a)OpenLDAP.org<br> <b>Subject:</b> Re: (ITS#9014) OpenLDAP modifies user provided TLS certific= ate before sending it to client</font> <div> </div> </div> <div class=3D"BodyFragment"><font size=3D"2"><span style=3D"font-size:11pt"= > <div class=3D"PlainText">--On Wednesday, April 24, 2019 6:43 PM +0000 h= yc(a)symas.com wrote:<br> <br> > siddjain(a)live.com wrote:<br> >> --_000_MWHPR08MB24000D77048AFCF7465C4397B53C0MWHPR08MB2400namp_<br= > >> Content-Type: text/plain; charset=3D"iso-8859-1"<br> >> Content-Transfer-Encoding: quoted-printable<br> >><br> >> could you send me output of running<br> >><br> >> openssl version -a<br> >><br> >> on your system? thanks<br> ><br> >> openssl version -a<br> > OpenSSL 1.1.1  11 Sep 2018<br> > built on: Tue Dec  4 13:15:09 2018 UTC<br> > platform: debian-amd64<br> <br> I would also note that not all OpenLDAP builds use OpenSSL.  For examp= le, <br> OpenLDAP built on Debian/Ubuntu uses GnuTLS.  OpenLDAP built on some <= br> versions of RedHat 7 use MozNSS.  Current RedHat 7 builds use OpenSSL = but <br> have an odd MozNSS bridge for backwards compatibilty, and there may be all = <br> sorts of odd bugs in that.  Apple links OpenLDAP to its own custom SSL= <br> libary.<br> <br> So really your first step should be isolating what TLS/SSL library OpenLDAP= <br> is linked to in the environment you're using.<br> <br> --Quanah<br> <br> <br> <br> --<br> <br> Quanah Gibson-Mount<br> Product Architect<br> Symas Corporation<br> Packaged, certified, and supported LDAP solutions powered by OpenLDAP:<br> <<a href=3D"https://eur01.safelinks.protection.outlook.com/?url=3Dhttp%3= A%2F%2Fwww.symas.com&amp;data=3D02%7C01%7C%7C349b90be6afe4991a54b08d6c8= f068b4%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917332202918260&= amp;sdata=3DNifWEVt269tCTuar98XYUfNkaHWSFMffI3M4%2FJ7j8zI%3D&amp;reserv= ed=3D0">https://eur01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2F= www.symas.com&amp;data=3D02%7C01%7C%7C349b90be6afe4991a54b08d6c8f068b4%= 7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917332202918260&amp;sda= ta=3DNifWEVt269tCTuar98XYUfNkaHWSFMffI3M4%2FJ7j8zI%3D&amp;reserved=3D0<= /a>><br> <br> </div> </span></font></div> </div> </body> </html> --_000_MWHPR08MB2400D7AE5E8EEC3D17192FACB53C0MWHPR08MB2400namp_--

1 0

Re: (ITS#9014) OpenLDAP modifies user provided TLS certificate before sending it to client
by quanah＠symas.com 24 Apr '19

24 Apr '19

--On Wednesday, April 24, 2019 6:43 PM +0000 hyc(a)symas.com wrote: > siddjain(a)live.com wrote: >> --_000_MWHPR08MB24000D77048AFCF7465C4397B53C0MWHPR08MB2400namp_ >> Content-Type: text/plain; charset="iso-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> could you send me output of running >> >> openssl version -a >> >> on your system? thanks > >> openssl version -a > OpenSSL 1.1.1 11 Sep 2018 > built on: Tue Dec 4 13:15:09 2018 UTC > platform: debian-amd64 I would also note that not all OpenLDAP builds use OpenSSL. For example, OpenLDAP built on Debian/Ubuntu uses GnuTLS. OpenLDAP built on some versions of RedHat 7 use MozNSS. Current RedHat 7 builds use OpenSSL but have an odd MozNSS bridge for backwards compatibilty, and there may be all sorts of odd bugs in that. Apple links OpenLDAP to its own custom SSL libary. So really your first step should be isolating what TLS/SSL library OpenLDAP is linked to in the environment you're using. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#9014) OpenLDAP modifies user provided TLS certificate before sending it to client
by hyc＠symas.com 24 Apr '19

24 Apr '19

siddjain(a)live.com wrote: > --_000_MWHPR08MB24000D77048AFCF7465C4397B53C0MWHPR08MB2400namp_ > Content-Type: text/plain; charset="iso-8859-1" > Content-Transfer-Encoding: quoted-printable > > could you send me output of running > > openssl version -a > > on your system? thanks > openssl version -a OpenSSL 1.1.1 11 Sep 2018 built on: Tue Dec 4 13:15:09 2018 UTC platform: debian-amd64 options: bn(64,64) rc4(8x,int) des(int) blowfish(ptr) compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-DovhWu/openssl-1.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2 OPENSSLDIR: "/usr/lib/ssl" ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-1.1" Seeding source: os-specific > > ________________________________ > From: Howard Chu <hyc(a)symas.com> > Sent: Wednesday, April 24, 2019 10:04 AM > To: Siddharth Jain; openldap-its(a)OpenLDAP.org > Subject: Re: (ITS#9014) OpenLDAP modifies user provided TLS certificate bef= > ore sending it to client > > Siddharth Jain wrote: >> Wow! Thanks for responding so fast. This could be a bug in docker-openlda= > p then. we have repro'ed this in two different environments - mac and ubunt= > u. Do you >> have a recommendation for docker image for openldap? > > As I said before, OpenLDAP doesn't touch the certificate files, it merely t= > ells the TLS > library where they are. You must likely have a broken TLS library. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9014) OpenLDAP modifies user provided TLS certificate before sending it to client
by siddjain＠live.com 24 Apr '19

24 Apr '19

--_000_MWHPR08MB24000D77048AFCF7465C4397B53C0MWHPR08MB2400namp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable could you send me output of running openssl version -a on your system? thanks ________________________________ From: Howard Chu <hyc(a)symas.com> Sent: Wednesday, April 24, 2019 10:04 AM To: Siddharth Jain; openldap-its(a)OpenLDAP.org Subject: Re: (ITS#9014) OpenLDAP modifies user provided TLS certificate bef= ore sending it to client Siddharth Jain wrote: > Wow! Thanks for responding so fast. This could be a bug in docker-openlda= p then. we have repro'ed this in two different environments - mac and ubunt= u. Do you > have a recommendation for docker image for openldap? As I said before, OpenLDAP doesn't touch the certificate files, it merely t= ells the TLS library where they are. You must likely have a broken TLS library. -------------------------------------------------------------------------= ---------------------------------------------------------------------------= ------------ > *From:* Howard Chu <hyc(a)symas.com> > *Sent:* Wednesday, April 24, 2019 9:42 AM > *To:* Siddharth Jain; openldap-its(a)OpenLDAP.org > *Subject:* Re: (ITS#9014) OpenLDAP modifies user provided TLS certificate= before sending it to client > > Siddharth Jain wrote: >> we have documented complete steps to repro the bug here <https://nam01.s= afelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithub.com%2Fsiddjain%= 2Fopenldap-bug&data=3D02%7C01%7C%7Cdeffc420629649af454408d6c8d6f129%7C8= 4df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917222820865922&sdata=3Dsx= jXXBtCMOjbK5AZCpLTObP%2BIlJRAxXUK7LpLzUDD%2FM%3D&reserved=3D0> with > container logs. > > I see no error here. > > Using your cert/key files: > There is no OpenLDAP bug here. Your server environment is broken. -- -- Howard Chu CTO, Symas Corp. https://nam01.safelinks.protection.outlook.com= /?url=3Dhttp%3A%2F%2Fwww.symas.com&data=3D02%7C01%7C%7Cdeffc420629649af= 454408d6c8d6f129%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C6369172228208= 65922&sdata=3DX5JT6j5%2BQ2BAsKGfNslnC%2FkQj%2BcSU4GAdTqmqqc3lWo%3D&= reserved=3D0 Director, Highland Sun https://nam01.safelinks.protection.outlook.com= /?url=3Dhttp%3A%2F%2Fhighlandsun.com%2Fhyc%2F&data=3D02%7C01%7C%7Cdeffc= 420629649af454408d6c8d6f129%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C63= 6917222820865922&sdata=3DSHju26Gxu5dToV%2BuCYDxBMZQS5qJZvREcg9q0CEg2bo%= 3D&reserved=3D0 Chief Architect, OpenLDAP https://nam01.safelinks.protection.outlook.com= /?url=3Dhttp%3A%2F%2Fwww.openldap.org%2Fproject%2F&data=3D02%7C01%7C%7C= deffc420629649af454408d6c8d6f129%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0= %7C636917222820865922&sdata=3DfJ7LIrWHv%2FG4CJGrx%2BClsFoldJfri%2Bdk7WN= 59Bt45jU%3D&reserved=3D0 --_000_MWHPR08MB24000D77048AFCF7465C4397B53C0MWHPR08MB2400namp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style type=3D"text/css" style=3D"display:none;"> P {margin-top:0;margin-bo= ttom:0;} </style> </head> <body dir=3D"ltr"> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;= color: rgb(0, 0, 0);"> could you send me output of running  <p style=3D"margin: 0px; font: 11px Menlo; background-color: rgb(255, 255, = 255); margin: 0px; background-color: rgb(255, 255, 255)"> <span style=3D"font-variant-ligatures: no-common-ligatures; font-variant-li= gatures: no-common-ligatures">openssl version -a</span></p> <p style=3D"margin: 0px; font: 11px Menlo; background-color: rgb(255, 255, = 255); margin: 0px; background-color: rgb(255, 255, 255)"> <span style=3D"font-variant-ligatures: no-common-ligatures; font-variant-li= gatures: no-common-ligatures">on your system? thanks</span></p> </div> <div> <div id=3D"appendonsend"></div> <div style=3D"font-family:Calibri,Helvetica,sans-serif; font-size:12pt; col= or:rgb(0,0,0)"> <br> </div> <hr tabindex=3D"-1" style=3D"display:inline-block; width:98%"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" co= lor=3D"#000000" style=3D"font-size:11pt"><b>From:</b> Howard Chu <hyc@sy= mas.com><br> <b>Sent:</b> Wednesday, April 24, 2019 10:04 AM<br> <b>To:</b> Siddharth Jain; openldap-its(a)OpenLDAP.org<br> <b>Subject:</b> Re: (ITS#9014) OpenLDAP modifies user provided TLS certific= ate before sending it to client</font> <div> </div> </div> <div class=3D"BodyFragment"><font size=3D"2"><span style=3D"font-size:11pt"= > <div class=3D"PlainText">Siddharth Jain wrote:<br> > Wow! Thanks for responding so fast. This could be a bug in docker-open= ldap then. we have repro'ed this in two different environments - mac and ub= untu. Do you<br> > have a recommendation for docker image for openldap?<br> <br> As I said before, OpenLDAP doesn't touch the certificate files, it merely t= ells the TLS<br> library where they are. You must likely have a broken TLS library.<br>   --------------------------------------------------------------------= ---------------------------------------------------------------------------= -----------------<br> > *From:* Howard Chu <hyc(a)symas.com><br> > *Sent:* Wednesday, April 24, 2019 9:42 AM<br> > *To:* Siddharth Jain; openldap-its(a)OpenLDAP.org<br> > *Subject:* Re: (ITS#9014) OpenLDAP modifies user provided TLS certific= ate before sending it to client<br> >  <br> > Siddharth Jain wrote:<br> >> we have documented complete steps to repro the bug here <<= a href=3D"https://nam01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F= %2Fgithub.com%2Fsiddjain%2Fopenldap-bug&amp;data=3D02%7C01%7C%7Cdeffc42= 0629649af454408d6c8d6f129%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C6369= 17222820865922&amp;sdata=3DsxjXXBtCMOjbK5AZCpLTObP%2BIlJRAxXUK7LpLzUDD%= 2FM%3D&amp;reserved=3D0">https://nam01.safelinks.protection.outlook.com= /?url=3Dhttps%3A%2F%2Fgithub.com%2Fsiddjain%2Fopenldap-bug&amp;data=3D0= 2%7C01%7C%7Cdeffc420629649af454408d6c8d6f129%7C84df9e7fe9f640afb435aaaaaaaa= aaaa%7C1%7C0%7C636917222820865922&amp;sdata=3DsxjXXBtCMOjbK5AZCpLTObP%2= BIlJRAxXUK7LpLzUDD%2FM%3D&amp;reserved=3D0</a>> with<br> > container logs.<br> > <br> > I see no error here.<br> > <br> > Using your cert/key files:<br> <br> > There is no OpenLDAP bug here. Your server environment is broken.<br> <br> <br> -- <br>   -- Howard Chu<br>   CTO, Symas Corp.        &nbs= p;  <a href=3D"https://nam01.safelinks.protection.outlook.com/?url=3Dh= ttp%3A%2F%2Fwww.symas.com&amp;data=3D02%7C01%7C%7Cdeffc420629649af45440= 8d6c8d6f129%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917222820865922= &amp;sdata=3DX5JT6j5%2BQ2BAsKGfNslnC%2FkQj%2BcSU4GAdTqmqqc3lWo%3D&a= mp;reserved=3D0"> https://nam01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fwww.syma= s.com&amp;data=3D02%7C01%7C%7Cdeffc420629649af454408d6c8d6f129%7C84df9e= 7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917222820865922&amp;sdata=3DX5J= T6j5%2BQ2BAsKGfNslnC%2FkQj%2BcSU4GAdTqmqqc3lWo%3D&amp;reserved=3D0</a><= br>   Director, Highland Sun     <a href=3D"https://na= m01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fhighlandsun.com%2F= hyc%2F&amp;data=3D02%7C01%7C%7Cdeffc420629649af454408d6c8d6f129%7C84df9= e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917222820865922&amp;sdata=3DSH= ju26Gxu5dToV%2BuCYDxBMZQS5qJZvREcg9q0CEg2bo%3D&amp;reserved=3D0"> https://nam01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fhighland= sun.com%2Fhyc%2F&amp;data=3D02%7C01%7C%7Cdeffc420629649af454408d6c8d6f1= 29%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917222820865922&amp;= sdata=3DSHju26Gxu5dToV%2BuCYDxBMZQS5qJZvREcg9q0CEg2bo%3D&amp;reserved= =3D0</a><br>   Chief Architect, OpenLDAP  <a href=3D"https://nam01.safelinks.p= rotection.outlook.com/?url=3Dhttp%3A%2F%2Fwww.openldap.org%2Fproject%2F&amp= ;amp;data=3D02%7C01%7C%7Cdeffc420629649af454408d6c8d6f129%7C84df9e7fe9f640a= fb435aaaaaaaaaaaa%7C1%7C0%7C636917222820865922&amp;sdata=3DfJ7LIrWHv%2F= G4CJGrx%2BClsFoldJfri%2Bdk7WN59Bt45jU%3D&amp;reserved=3D0"> https://nam01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fwww.open= ldap.org%2Fproject%2F&amp;data=3D02%7C01%7C%7Cdeffc420629649af454408d6c= 8d6f129%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917222820865922&amp= ;amp;sdata=3DfJ7LIrWHv%2FG4CJGrx%2BClsFoldJfri%2Bdk7WN59Bt45jU%3D&amp;r= eserved=3D0</a><br> </div> </span></font></div> </div> </body> </html> --_000_MWHPR08MB24000D77048AFCF7465C4397B53C0MWHPR08MB2400namp_--

1 0

Re: (ITS#9014) OpenLDAP modifies user provided TLS certificate before sending it to client
by hyc＠symas.com 24 Apr '19

24 Apr '19

Siddharth Jain wrote: > Wow! Thanks for responding so fast. This could be a bug in docker-openl= dap then. we have repro'ed this in two different environments - mac and u= buntu. Do you > have a recommendation for docker image for openldap? As I said before, OpenLDAP doesn't touch the certificate files, it merely= tells the TLS library where they are. You must likely have a broken TLS library. -----------------------------------------------------------------------= -------------------------------------------------------------------------= ---------------- > *From:* Howard Chu <hyc(a)symas.com> > *Sent:* Wednesday, April 24, 2019 9:42 AM > *To:* Siddharth Jain; openldap-its(a)OpenLDAP.org > *Subject:* Re: (ITS#9014) OpenLDAP modifies user provided TLS certifica= te before sending it to client > =A0 > Siddharth Jain wrote: >> we have documented complete steps to repro the bug=A0here <https://eur= 04.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithub.com%2Fsid= djain%2Fopenldap-bug&data=3D02%7C01%7C%7Caca4f78e53324b52690008d6c8d3= cc09%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917209315407238&= sdata=3D8VfRtnCNPd%2BFo2Sps%2BLftBG3XcC57ReIFFphK6noyLc%3D&reserved=3D= 0>=A0with > container logs. >=20 > I see no error here. >=20 > Using your cert/key files: > There is no OpenLDAP bug here. Your server environment is broken. --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs April 2019