- openldap-bugs - openldap.org

Re: (ITS#9017) Improving performance of commit sync in Windows
by kriszyp＠gmail.com 06 Feb '20

06 Feb '20

--000000000000bdb86d059deaf350 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sorry to keep pestering, but just pinging about this patch again, as I still think this fix could benefit windows users. And at this point, I think I can say we have tested it pretty well, running on our servers for almost a year :). Thanks, Kris On Wed, Sep 18, 2019 at 12:56 PM Kris Zyp <kriszyp(a)gmail.com> wrote: > Checking on this again, is this still a possibility for merging into LMDB= ? > This fix is still working great (improved performance) on our systems. > Thanks, > Kris > > On Mon, Jun 17, 2019 at 1:04 PM Kris Zyp <kriszyp(a)gmail.com> wrote: > >> Is this still being considered/reviewed? Let me know if there are any >> other changes you would like me to make. This patch has continued to yie= ld >> significant and reliable performance improvements for us, and seems like= it >> would be nice for this to be available for other Windows users. >> >> On Fri, May 3, 2019 at 3:52 PM Kris Zyp <kriszyp(a)gmail.com> wrote: >> >>> For the sake of putting this in the email thread (other code discussion >>> in GitHub), here is the latest squashed commit of the proposed patch (w= ith >>> the on-demand, retained overlapped array to reduce re-malloc and openin= g >>> event handles): >>> https://github.com/kriszyp/node-lmdb/commit/726a9156662c703bf3d453aab75= ee222072b990f >>> >>> >>> >>> Thanks, >>> Kris >>> >>> >>> >>> *From: *Kris Zyp <kriszyp(a)gmail.com> >>> *Sent: *April 30, 2019 12:43 PM >>> *To: *Howard Chu <hyc(a)symas.com>; openldap-its(a)OpenLDAP.org >>> *Subject: *RE: (ITS#9017) Improving performance of commit sync in >>> Windows >>> >>> >>> >>> > What is the point of using writemap mode if you still need to use >>> WriteFile >>> >>> > on every individual page? >>> >>> >>> >>> As I understood from the documentation, and have observed, using >>> writemap mode is faster (and uses less temporary memory) because it doe= sn=E2=80=99t >>> require mallocs to allocate pages (docs: =E2=80=9CThis is faster and us= es fewer >>> mallocs=E2=80=9D). To be clear though, LMDB is so incredibly fast and e= fficient, >>> that in sync-mode, it takes enormous transactions before the time spent >>> allocating and creating the dirty pages with the updated b-tree is anyw= here >>> even remotely close to the time it takes to wait for disk flushing, eve= n >>> with an SSD. But the more pertinent question is efficiency, and measuri= ng >>> CPU cycles rather than time spent (efficiency is more important than ju= st >>> time spent). When I ran my tests this morning of 100 (sync) transaction= s >>> with 100 puts per transaction, times varied quite a bit, but it seemed = like >>> running with writemap enabled typically averages about 500ms of CPU and >>> with writemap disabled it typically averages around 600ms. Not a huge >>> difference, but still definitely worthwhile, I think. >>> >>> >>> >>> Caveat emptor: Measuring LMDB performance with sync interactions on >>> Windows is one of the most frustratingly erratic things to measure. It = is >>> sunny outside right now, times could be different when it starts rainin= g >>> later, but, this is what I saw this morning... >>> >>> >>> >>> > What is the performance difference between your patch using writemap, >>> and just >>> >>> > not using writemap in the first place? >>> >>> >>> >>> Running 1000 sync transactions on 3GB db with a single put per >>> transaction, without writemap map, without the patch took about 60 seco= nds. >>> And it took about 1 second with the patch with writemap mode enabled! >>> (there is no significant difference in sync times with writemap enabled= or >>> disabled with the patch.) So the difference was huge in my test. And no= t >>> only that, without the patch, the CPU usage was actually _*higher*_ >>> during that 60 seconds (close to 100% of a core) than during the execut= ion >>> with the patch for one second (close to 50%). Anyway, there are certai= nly >>> tests I have run where the differences are not as large (doing small >>> commits on large dbs accentuates the differences), but the patch always >>> seems to win. It could also be that my particular configuration causes >>> bigger differences (on an SSD drive, and maybe a more fragmented file?)= . >>> >>> >>> >>> Anyway, I added error handling for the malloc, and fixed/changed the >>> other things you suggested. Be happy to make any other changes you want= . >>> The updated patch is here: >>> >>> >>> https://github.com/kriszyp/node-lmdb/commit/25366dea9453749cf6637f43ec1= 7b9b62094acde >>> >>> >>> >>> > OVERLAPPED* ov =3D malloc((pagecount - keep) * sizeof(OVERLAPPED)); >>> >>> > Probably this ought to just be pre-allocated based on the maximum >>> number of dirty pages a txn allows. >>> >>> >>> >>> I wasn=E2=80=99t sure I understood this comment. Are you suggesting we = malloc(MDB_IDL_UM_MAX >>> * sizeof(OVERLAPPED)) for each environment, and retain it for the life = of >>> the environment? I think that is 4MB, if my math is right, which seems = like >>> a lot of memory to keep allocated (we usually have a lot of open >>> environments). If the goal is to reduce the number of mallocs, how abou= t we >>> retain the OVERLAPPED array, and only free and re-malloc if the previou= s >>> allocation wasn=E2=80=99t large enough? Then there isn=E2=80=99t unnece= ssary allocation, >>> and we only malloc when there is a bigger transaction than any previous= . I >>> put this together in a separate commit, as I wasn=E2=80=99t sure if thi= s what you >>> wanted (can squash if you prefer): >>> https://github.com/kriszyp/node-lmdb/commit/2fe68fb5269c843e2e789746a17= a4b2adefaac40 >>> >>> >>> >>> Thank you for the review! >>> >>> >>> >>> Thanks, >>> Kris >>> >>> >>> >>> *From: *Howard Chu <hyc(a)symas.com> >>> *Sent: *April 30, 2019 7:12 AM >>> *To: *kriszyp(a)gmail.com; openldap-its(a)OpenLDAP.org >>> *Subject: *Re: (ITS#9017) Improving performance of commit sync in >>> Windows >>> >>> >>> >>> kriszyp(a)gmail.com wrote: >>> >>> > Full_Name: Kristopher William Zyp >>> >>> > Version: LMDB 0.9.23 >>> >>> > OS: Windows >>> >>> > URL: >>> https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0a= b9332b7fc4ce9 >>> >>> > Submission from: (NULL) (71.199.6.148) >>> >>> > >>> >>> > >>> >>> > We have seen very poor performance on the sync of commits on large >>> databases in >>> >>> > Windows. On databases with 2GB of data, in writemap mode, the sync of >>> even small >>> >>> > commits is consistently well over 100ms (without writemap it is >>> faster, but >>> >>> > still slow). It is expected that a sync should take some time while >>> waiting for >>> >>> > disk confirmation of the writes, but more concerning is that these sy= nc >>> >>> > operations (in writemap mode) are instead dominated by nearly 100% >>> system CPU >>> >>> > utilization, so operations that requires sub-millisecond b-tree updat= e >>> >>> > operations are then dominated by very large amounts of system CPU >>> cycles during >>> >>> > the sync phase. >>> >>> > >>> >>> > I think that the fundamental problem is that FlushViewOfFile seems to >>> be an O(n) >>> >>> > operation where n is the size of the file (or map). I presume that >>> Windows is >>> >>> > scanning the entire map/file for dirty pages to flush, I'm guessing >>> because it >>> >>> > doesn't have an internal index of all the dirty pages for every >>> file/map-view in >>> >>> > the OS disk cache. Therefore, the turns into an extremely expensive, >>> CPU-bound >>> >>> > operation to find the dirty pages for (large file) and initiate their >>> writes, >>> >>> > which, of course, is contrary to the whole goal of a scalable databas= e >>> system. >>> >>> > And FlushFileBuffers is also relatively slow as well. We have >>> attempted to batch >>> >>> > as many operations into single transaction as possible, but this is >>> still a very >>> >>> > large overhead. >>> >>> > >>> >>> > The Windows docs for FlushFileBuffers itself warns about the >>> inefficiencies of >>> >>> > this function ( >>> https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi= -flushfilebuffers >>> ). >>> >>> > Which also points to the solution: it is much faster to write out the >>> dirty >>> >>> > pages with WriteFile through a sync file handle >>> (FILE_FLAG_WRITE_THROUGH). >>> >>> > >>> >>> > The associated patch >>> >>> > ( >>> https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0a= b9332b7fc4ce9 >>> ) >>> >>> > is my attempt at implementing this solution, for Windows. Fortunately= , >>> with the >>> >>> > design of LMDB, this is relatively straightforward. LMDB already >>> supports >>> >>> > writing out dirty pages with WriteFile calls. I added a write-through >>> handle for >>> >>> > sending these writes directly to disk. I then made that file-handle >>> >>> > overlapped/asynchronously, so all the writes for a commit could be >>> started in >>> >>> > overlap mode, and (at least theoretically) transfer in parallel to th= e >>> drive and >>> >>> > then used GetOverlappedResult to wait for the completion. So basicall= y >>> >>> > mdb_page_flush becomes the sync. I extended the writing of dirty page= s >>> through >>> >>> > WriteFile to writemap mode as well (for writing meta too), so that >>> WriteFile >>> >>> > with write-through can be used to flush the data without ever needing >>> to call >>> >>> > FlushViewOfFile or FlushFileBuffers. I also implemented support for >>> write >>> >>> > gathering in writemap mode where contiguous file positions infers >>> contiguous >>> >>> > memory (by tracking the starting position with wdp and writing >>> contiguous pages >>> >>> > in single operations). Sorting of the dirty list is maintained even i= n >>> writemap >>> >>> > mode for this purpose. >>> >>> >>> >>> What is the point of using writemap mode if you still need to use >>> WriteFile >>> >>> on every individual page? >>> >>> >>> >>> > The performance benefits of this patch, in my testing, are >>> considerable. Writing >>> >>> > out/syncing transactions is typically over 5x faster in writemap mode= , >>> and 2x >>> >>> > faster in standard mode. And perhaps more importantly (especially in >>> environment >>> >>> > with many threads/processes), the efficiency benefits are even larger= , >>> >>> > particularly in writemap mode, where there can be a 50-100x reduction >>> in the >>> >>> > system CPU usage by using this patch. This brings windows performance >>> with >>> >>> > sync'ed transactions in LMDB back into the range of "lightning" >>> performance :). >>> >>> >>> >>> What is the performance difference between your patch using writemap, >>> and just >>> >>> not using writemap in the first place? >>> >>> >>> >>> -- >>> >>> -- Howard Chu >>> >>> CTO, Symas Corp. http://www.symas.com >>> >>> Director, Highland Sun http://highlandsun.com/hyc/ >>> >>> Chief Architect, OpenLDAP http://www.openldap.org/project/ >>> >>> >>> >>> >>> >> --000000000000bdb86d059deaf350 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Sorry to keep pestering, but just pinging about this patch= again, as I still think this fix could benefit windows users. And at this = point, I think I can say we have tested it pretty well, running on our serv= ers for almost a year :).<div>Thanks,</div><div>Kris</div></div> <div cl= ass=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Sep 18, 2= 019 at 12:56 PM Kris Zyp <<a href=3D"mailto:kriszyp@gmail.com">kriszyp@g= mail.com</a>> wrote: </div><blockquote class=3D"gmail_quote" style=3D= "margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-le= ft:1ex"><div dir=3D"ltr">Checking on this again, is this still a possibilit= y for merging into LMDB? This fix is still working great (improved performa= nce) on our systems.<div>Thanks,</div><div>Kris</div></div> <div class= =3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Jun 17, 2019= at 1:04 PM Kris Zyp <<a href=3D"mailto:kriszyp@gmail.com" target=3D"_bl= ank">kriszyp(a)gmail.com</a>> wrote: </div><blockquote class=3D"gmail_q= uote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,2= 04);padding-left:1ex"><div dir=3D"ltr">Is this still being considered/revie= wed? Let me know if there are any other changes you would like me to make. = This patch has continued to yield significant and reliable performance impr= ovements for us, and seems like it would be nice for this to be available f= or other Windows users.</div> <div class=3D"gmail_quote"><div dir=3D"ltr= " class=3D"gmail_attr">On Fri, May 3, 2019 at 3:52 PM Kris Zyp <<a href= =3D"mailto:kriszyp@gmail.com" target=3D"_blank">kriszyp(a)gmail.com</a>> w= rote: </div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0p= x 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang= =3D"EN-CA"><div>For the sake of putting this in the = email thread (other code discussion in GitHub), here is the latest squashed= commit of the proposed patch (with the on-demand, retained overlapped arra= y to reduce re-malloc and opening event handles): <a href=3D"https://github= .com/kriszyp/node-lmdb/commit/726a9156662c703bf3d453aab75ee222072b990f" tar= get=3D"_blank">https://github.com/kriszyp/node-lmdb/commit/726a9156662c703b= f3d453aab75ee222072b990f</a>=C2=A0= Thanks, Kris</u= >=C2=A0<div style=3D"border-right:none;border-bottom:none;border= -left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">From: <a href=3D= "mailto:kriszyp@gmail.com" target=3D"_blank">Kris Zyp</a> Sent: A= pril 30, 2019 12:43 PM To: <a href=3D"mailto:hyc@symas.com" targe= t=3D"_blank">Howard Chu</a>; <a href=3D"mailto:openldap-its@OpenLDAP.org" t= arget=3D"_blank">openldap-its(a)OpenLDAP.org</a> Subject: RE: (ITS#= 9017) Improving performance of commit sync in Windows</div>=C2=A0> What is the = point of using writemap mode if you still need to use WriteFile</= u>> on every individual page?</= p>=C2=A0As = I understood from the documentation, and have observed, using writemap mode= is faster (and uses less temporary memory) because it doesn=E2=80=99t requ= ire mallocs to allocate pages (docs: =E2=80=9CThis is faster and uses fewer= mallocs=E2=80=9D). To be clear though, LMDB is so incredibly fast and effi= cient, that in sync-mode, it takes enormous transactions before the time sp= ent allocating and creating the dirty pages with the updated b-tree is anyw= here even remotely close to the time it takes to wait for disk flushing, ev= en with an SSD. But the more pertinent question is efficiency, and measurin= g CPU cycles rather than time spent (efficiency is more important than just= time spent). When I ran my tests this morning of 100 (sync) transactions w= ith 100 puts per transaction, times varied quite a bit, but it seemed like = running with writemap enabled typically averages about 500ms of CPU and wit= h writemap disabled it typically averages around 600ms. Not a huge differen= ce, but still definitely worthwhile, I think.=C2=A0Caveat emptor: Me= asuring LMDB performance with sync interactions on Windows is one of the mo= st frustratingly erratic things to measure. It is sunny outside right now, = times could be different when it starts raining later, but, this is what I = saw this morning...=C2=A0<u= >> What is the performance difference bet= ween your patch using writemap, and just> not using writemap in the first place?=C2=A0Running 1000 = sync transactions on 3GB db with a single put per transaction, without writ= emap map, without the patch took about 60 seconds. And it took about 1 seco= nd with the patch with writemap mode enabled! (there is no significant diff= erence in sync times with writemap enabled or disabled with the patch.) So = the difference was huge in my test. And not only that, without the patch, t= he CPU usage was actually _higher_ during that 60 seconds (close to = 100% of a core) than during the execution with the patch for one second (cl= ose to 50%).=C2=A0 Anyway, there are certainly tests I have run where the d= ifferences are not as large (doing small commits on large dbs accentuates t= he differences), but the patch always seems to win. It could also be that m= y particular configuration causes bigger differences (on an SSD drive, and = maybe a more fragmented file?).= =C2=A0Anyway, I added error handling = for the malloc, and fixed/changed the other things you suggested. Be happy = to make any other changes you want. The updated patch is here:</u= ><a href=3D"https://github.com/kriszyp/node-lmdb= /commit/25366dea9453749cf6637f43ec17b9b62094acde" target=3D"_blank">https:/= /github.com/kriszyp/node-lmdb/commit/25366dea9453749cf6637f43ec17b9b62094ac= de</a>=C2=A0> OVERLAPPED* ov =3D malloc<= /span>((pagecount - keep) * sizeof<s= pan style=3D"font-size:9pt;font-family:Consolas;color:rgb(36,41,46)">(OVERL= APPED));> Probably th= is ought to just be pre-allocated based on the maximum number of dirty page= s a txn allows.=C2=A0<= /span>I w= asn=E2=80=99t sure I understood this comment. Are you suggesting we = malloc(MDB_IDL_UM_MAX * sizeof(OVERLAPPED)) for each environment, and retai= n it for the life of the environment? I think that is 4MB, if my math is ri= ght, which seems like a lot of memory to keep allocated (we usually have a = lot of open environments). If the goal is to reduce the number of mallocs, = how about we retain the OVERLAPPED array, and only free and re-malloc if th= e previous allocation wasn=E2=80=99t large enough? Then there isn=E2=80=99t= unnecessary allocation, and we only malloc when there is a bigger transact= ion than any previous. I put this together in a separate commit, as I wasn= =E2=80=99t sure if this what you wanted (can squash if you prefer): <a href= =3D"https://github.com/kriszyp/node-lmdb/commit/2fe68fb5269c843e2e789746a17= a4b2adefaac40" target=3D"_blank">https://github.com/kriszyp/node-lmdb/commi= t/2fe68fb5269c843e2e789746a17a4b2adefaac40</a>=C2=A0Thank you for th= e review! <= p class=3D"MsoNormal">=C2=A0Thanks= , Kris=C2=A0<= div style=3D"border-right:none;border-bottom:none;border-left:none;border-t= op:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm"><= b>From: <a href=3D"mailto:hyc@symas.com" target=3D"_blank">Howard Chu</= a> Sent: April 30, 2019 7:12 AM To: <a href=3D"mailto:k= riszyp(a)gmail.com" target=3D"_blank">kriszyp(a)gmail.com</a>; <a href=3D"mailt= o:openldap-its@OpenLDAP.org" target=3D"_blank">openldap-its(a)OpenLDAP.org</a= > Subject: Re: (ITS#9017) Improving performance of commit sync in= Windows</div>=C2=A0= <a href=3D"mailto:kriszyp@gmail.com" target=3D"_= blank">kriszyp(a)gmail.com</a> wrote:> Full_Name: Kristopher William Zyp> Version: LMDB 0.9.23>= OS: Windows> URL: <a href=3D"h= ttps://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab9332= b7fc4ce9" target=3D"_blank">https://github.com/kriszyp/node-lmdb/commit/7ff= 525ae57684a163d32af74a0ab9332b7fc4ce9</a>> Submission from: (NULL) (71.199.6.148)> > <= u>> We have seen very poor performance on= the sync of commits on large databases in> Windows. On databases with 2GB of data, in writemap mode, the = sync of even small> commits is = consistently well over 100ms (without writemap it is faster, but<= /u>> still slow). It is expected that a sync = should take some time while waiting for> disk confirmation of the writes, but more concerning is that thes= e sync> operations (in writemap= mode) are instead dominated by nearly 100% system CPU> utilization, so operations that requires sub-milli= second b-tree update> operation= s are then dominated by very large amounts of system CPU cycles during</= u>> the sync phase.<= p class=3D"MsoNormal">> > I = think that the fundamental problem is that FlushViewOfFile seems to be an O= (n)> operation where n is the s= ize of the file (or map). I presume that Windows is> scanning the entire map/file for dirty pages to flush= , I'm guessing because it> = doesn't have an internal index of all the dirty pages for every file/ma= p-view in> the OS disk cache. T= herefore, the turns into an extremely expensive, CPU-bound</p= >> operation to find the dirty pages for (large f= ile) and initiate their writes,&gt= ; which, of course, is contrary to the whole goal of a scalable database sy= stem.> And FlushFileBuffers is = also relatively slow as well. We have attempted to batch<= p class=3D"MsoNormal">> as many operations into single transaction as po= ssible, but this is still a very&g= t; large overhead.> <= /u>> The Windows docs for FlushFileBuffers it= self warns about the inefficiencies of> this function (<a href=3D"https://docs.microsoft.com/en-us/windows= /desktop/api/fileapi/nf-fileapi-flushfilebuffers" target=3D"_blank">https:/= /docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-flushfileb= uffers</a>).> Which also points= to the solution: it is much faster to write out the dirty</p= >> pages with WriteFile through a sync file handl= e (FILE_FLAG_WRITE_THROUGH).> <= u>> The associated patch= > (<a href=3D"https://github.com/kriszyp/= node-lmdb/commit/7ff525ae57684a163d32af74a0ab9332b7fc4ce9" target=3D"_blank= ">https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab9= 332b7fc4ce9</a>)> is my attempt= at implementing this solution, for Windows. Fortunately, with the<u= >> design of LMDB, this is relatively str= aightforward. LMDB already supports> writing out dirty pages with WriteFile calls. I added a write-through= handle for> sending these writ= es directly to disk. I then made that file-handle> overlapped/asynchronously, so all the writes for a comm= it could be started in> overlap= mode, and (at least theoretically) transfer in parallel to the drive and<u= >> then used GetOverlappedResult t= o wait for the completion. So basically> mdb_page_flush becomes the sync. I extended the writing of dirty = pages through> WriteFile to wri= temap mode as well (for writing meta too), so that WriteFile<= /p>> with write-through can be used to flush the = data without ever needing to call&= gt; FlushViewOfFile or FlushFileBuffers. I also implemented support for wri= te> gathering in writemap mode = where contiguous file positions infers contiguous> memory (by tracking the starting position with wdp and = writing contiguous pages> in si= ngle operations). Sorting of the dirty list is maintained even in writemap<= u>> mode for this purpose.<= u>=C2=A0What is the point of using writemap mode if you still need to use Writ= eFileon every individual page?<= /u>=C2=A0> The performance benefits of this patch, in my testing, are co= nsiderable. Writing> out/syncin= g transactions is typically over 5x faster in writemap mode, and 2x<= u>> faster in standard mode. And perhaps = more importantly (especially in environment> with many threads/processes), the efficiency benefits are eve= n larger,> particularly in writ= emap mode, where there can be a 50-100x reduction in the<= p class=3D"MsoNormal">> system CPU usage by using this patch. This bring= s windows performance with> syn= c'ed transactions in LMDB back into the range of "lightning" = performance :).=C2=A0</u= >What is the performance difference between your= patch using writemap, and justnot= using writemap in the first place?=C2=A0-- =C2=A0=C2=A0-- Howard Chu=C2=A0 CTO, Symas Corp.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 <a href=3D"http://www.symas.com" target=3D"_blank">http:= //www.symas.com</a>=C2=A0 Director= , Highland Sun=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"http://highlandsun.com/hy= c/" target=3D"_blank">http://highlandsun.com/hyc/</a>=C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http:/= /www.openldap.org/project/" target=3D"_blank">http://www.openldap.org/proje= ct/</a>=C2=A0=C2=A0</div></div></blockquote></div> </blockquote></div> </blockquote></div> --000000000000bdb86d059deaf350--

1 0

(ITS#9163) warning: logical not is only applied to the left hand side of bitwise operator
by noloader＠gmail.com 06 Feb '20

06 Feb '20

Full_Name: Jeffrey Walton Version: 2.4.47 OS: OS X 10.10 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (100.18.0.145) Hi Everyone. I'm building OpenSLAP 2.4.47 from the release tarball on OS X. Clang is triggering a warning that may be of interest: gcc -g2 -O2 -march=native -fPIC -pthread -I../../include -I/usr/local/include -DNDEBUG -c -o common.o common.c ... common.c:2329:10: warning: logical not is only applied to the left hand side of this bitwise operator [-Wlogical-not-parentheses] if ( !tool_ctrl_response[j].mask & tool_type ) { ^ ~ common.c:2329:10: note: add parentheses after the '!' to evaluate the bitwise operator first if ( !tool_ctrl_response[j].mask & tool_type ) { ^ ( ) common.c:2329:10: note: add parentheses around left hand side expression to silence this warning if ( !tool_ctrl_response[j].mask & tool_type ) { ^ ( ) As written the statement is being evaluated as: (!tool_ctrl_response[j].mask) & tool_type It looks like this is what was intended: !(tool_ctrl_response[j].mask & tool_type) Perhaps the project can add parenthesis to clear the warning.

1 0

Re: (ITS#8988) Undefined Behavior in slapadd
by openldap-technical＠kolttonen.fi 03 Feb '20

03 Feb '20

This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463811718-1330329680-1580755187=:215607 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 8BIT Hello, On Mon, 3 Feb 2020, hyc(a)symas.com wrote: > No. The POSIX spec is *not* a part of the C spec - yet it is still valid. POSIX spec *is* a C standard that exists in addition to the ISO C standard. ISO C defines the so-called "basic language and features" of C if you can call it that. > Which simply proves the point that just because something is not defined > in the C spec does not make it invalid. The compiler guys are idiots for > taking this position. Like Jeff already said, wrongly aligned object access *is* defined by the ISO C specification. It is wrong and explicitly defined as Undefined Behaviour. Invoking UB is playing with fire, and calling compiler writers with offending names does not change that fact at all. Spending long time on comp.lang.c should be mandatory for all C programmers out there. It is shocking to invoke UB and not bother to fix it, instead blaming compiler writers and C standard writers. Best regards, Jokke Hämäläinen ---1463811718-1330329680-1580755187=:215607--

1 0

Re: (ITS#9162) dbMaxSize property does not as expected
by quanah＠symas.com 03 Feb '20

03 Feb '20

--On Monday, February 3, 2020 9:49 AM +0000 pasumarthivijaykumar(a)gmail.com wrote: > Full_Name: Vijay Kumar > Version: 2.4.48 > OS: windows > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (103.204.96.6) Please stop filing ITSes, as you clearly are working with an already broken database. This ITS will be closed. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#8988) Undefined Behavior in slapadd
by hyc＠symas.com 03 Feb '20

03 Feb '20

openldap-technical(a)kolttonen.fi wrote: > This message is in MIME format. The first part should be readable text, > while the remaining parts are likely unreadable without MIME-aware tools. > > ---1463811718-806940296-1580670508=:196090 > Content-Type: text/plain; charset=ISO-8859-15 > Content-Transfer-Encoding: 8BIT > > > Hello, > > On Sat, 8 Jun 2019, hyc(a)symas.com wrote: > >> The gcc/clang folks have their heads up their asses. They've >> deliberately misinterpreted the spec claiming undefined behaviors are >> forbidden, supposedly to enable essential optimizations. Most of which >> only apply to obscure corner cases in compiler benchmark suites, that >> nobody in the real world ever benefits from. > > I realize this thread is very old, but Jeff and the C compiler folks are > right. All C programs that invoke undefined behavior are illegal C > programs and should be fixed. > > Posix threads are well-defined by Posix standards, so calling them > "undefined behaviour" is not a valid argument. No. The POSIX spec is *not* a part of the C spec - yet it is still valid. Which simply proves the point that just because something is not defined in the C spec does not make it invalid. The compiler guys are idiots for taking this position. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9162) dbMaxSize property does not as expected
by pasumarthivijaykumar＠gmail.com 03 Feb '20

03 Feb '20

Full_Name: Vijay Kumar Version: 2.4.48 OS: windows URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (103.204.96.6) if i try to set the value from property, i dont understand from where the value 1406945XXXXXX and getting set. Please check the details mentioned. 1) olcDbMaxSize: 10485760 >mdb_stat.exe -rne "C:\testMaxSize2\mdb\data.mdb" Environment Info Map address: 0000000000000000 Map size: 140694549168128 Page size: 4096 Max pages: 34349255168 Number of pages used: 2 Last transaction ID: 0 Max readers: 126 Number of readers used: 0 Reader Table Status (no active readers) >slapd.exe -T add -F ..\slapd.d -l backupContent.ldif 5e37a371 mdb_db_open: database "o=abc.net" cannot be opened: ⁿ (87). Restore from backup! 5e37a371 backend_startup_one (type=mdb, suffix="o=abc.net"): bi_db_open failed! (87) slap_startup failed 2) olcDbMaxSize: 1048576 >slapd.exe -T add -F ..\slapd.d -l backupContent.ldif 5e37a5ef mdb_db_open: database "o=abc.net" cannot be opened: ⁿ (87). Restore from backup! 5e37a5ef backend_startup_one (type=mdb, suffix="o=abc.net"): bi_db_open failed! (87) slap_startup failed >mdb_stat.exe -rne "C:\testMaxSize2\mdb\data.mdb" Environment Info Map address: 0000000000000000 Map size: 140694539730944 Page size: 4096 Max pages: 34349252864 Number of pages used: 2 Last transaction ID: 0 Max readers: 126 Number of readers used: 0 Reader Table Status (no active readers) 3) olcDbMaxSize: 1024 >slapd.exe -T add -F ..\slapd.d -l backupContent.ldif 5e37a7fd mdb_db_open: database "o=abc.net" cannot be opened: ⁿ (87). Restore from backup! 5e37a7fd backend_startup_one (type=mdb, suffix="o=abc.net"): bi_db_open failed! (87) slap_startup failed >mdb_stat.exe -rne "C:\testMaxSize2\mdb\data.mdb" Environment Info Map address: 0000000000000000 Map size: 140694538683392 Page size: 4096 Max pages: 34349252608 Number of pages used: 2 Last transaction ID: 0 Max readers: 126 Number of readers used: 0 Reader Table Status (no active readers) 5) olcDbMaxSize 10 >slapd.exe -T add -F ..\slapd.d -l backupContent.ldif 5e37a92b mdb_db_open: database "o=abc.net" cannot be opened: ⁿ (87). Restore from backup! 5e37a92b backend_startup_one (type=mdb, suffix="o=abc.net"): bi_db_open failed! (87) slap_startup failed >mdb_stat.exe -rne "C:\testMaxSize2\mdb\data.mdb" Environment Info Map address: 0000000000000000 Map size: 140694538682378 Page size: 4096 Max pages: 34349252608 Number of pages used: 2 Last transaction ID: 0 Max readers: 126 Number of readers used: 0 Reader Table Status (no active readers) 6) olcDbMaxSize 1 >slapd.exe -T add -F ..\slapd.d -l backupContent.ldif 5e37a92b mdb_db_open: database "o=abc.net" cannot be opened: ⁿ (87). Restore from backup! 5e37a92b backend_startup_one (type=mdb, suffix="o=abc.net"): bi_db_open failed! (87) slap_startup failed >mdb_stat.exe -rne "C:\testMaxSize2\mdb\data.mdb" Environment Info Map address: 0000000000000000 Map size: 140694538682369 Page size: 4096 Max pages: 34349252608 Number of pages used: 2 Last transaction ID: 0 Max readers: 126 Number of readers used: 0 Reader Table Status (no active readers) 7) olcDbMaxSize not set >mdb_stat.exe -rne "C:\testMaxSize2\mdb\data.mdb" Environment Info Map address: 0000000000000000 Map size: 1048576 Page size: 4096 Max pages: 256 Number of pages used: 254 Last transaction ID: 104 Max readers: 126 Number of readers used: 0 Reader Table Status (no active readers)

1 0

Re: (ITS#8988) Undefined Behavior in slapadd
by openldap-technical＠kolttonen.fi 02 Feb '20

02 Feb '20

This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463811718-326687762-1580674674=:196994 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 8BIT On Sun, 2 Feb 2020, openldap-technical(a)kolttonen.fi wrote: > I realize this thread is very old, but Jeff and the C compiler folks are > right. All C programs that invoke undefined behavior are illegal C > programs and should be fixed. > > Posix threads are well-defined by Posix standards, so calling them > "undefined behaviour" is not a valid argument. To make it clear that UB in C programs is horrible, let me provide a real world example of UB: Cyrus IMAPD had an unnoticed dormant strcpy() related UB bug for several years. You see, the manual page of strcpy() says that the src and dst strings must not overlap, or else the C program invokes UB. For many years, Cyrus operated correctly even though the src and dst strings *did* overlap in one part of their database code. UB of course allows this behaviour too. The reason was because GNU libc folks had written their strcpy() implementation in such a way that breaking the contract of having non-overlapping strings did not cause any problems. Then, years later, GNU libc folks deciced to optimize (or otherwise change) their strcpy() implementation. Now the new implementation punished all UB invokers, and we saw our Cyrus mailbox database slowly but surely getting corrupted. It is indeed shocking to hear that LMDB implementation invokes UB. GCC and Clang could change their behaviour tomorrow, and LMDB could get corrupted, crash or whatever. And certainly the blame would be on LMDB code, not the compiler writers. Best Regards, Jokke Hämäläinen ---1463811718-326687762-1580674674=:196994--

1 0

Re: (ITS#8988) Undefined Behavior in slapadd
by openldap-technical＠kolttonen.fi 02 Feb '20

02 Feb '20

This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463811718-806940296-1580670508=:196090 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 8BIT Hello, On Sat, 8 Jun 2019, hyc(a)symas.com wrote: > The gcc/clang folks have their heads up their asses. They've > deliberately misinterpreted the spec claiming undefined behaviors are > forbidden, supposedly to enable essential optimizations. Most of which > only apply to obscure corner cases in compiler benchmark suites, that > nobody in the real world ever benefits from. I realize this thread is very old, but Jeff and the C compiler folks are right. All C programs that invoke undefined behavior are illegal C programs and should be fixed. Posix threads are well-defined by Posix standards, so calling them "undefined behaviour" is not a valid argument. Best regards, Jokke Hämäläinen ---1463811718-806940296-1580670508=:196090--

1 0

Re: (ITS#9161) Codespell report for "OpenLDAP" (on fossies.org)
by quanah＠symas.com 30 Jan '20

30 Jan '20

--On Thursday, January 30, 2020 8:48 PM +0000 Jens.Schleusener(a)fossies.org wrote: > Full_Name: Jens Schleusener > Version: 2.4.49 > OS: (irrelevant) > URL: https://fossies.org/linux/misc/openldap/codespell.html > Submission from: (NULL) (2003:cf:71a:5655:2905:9b7a:fef1:ab74) > > > The FOSS server fossies.org - also supporting "OpenLDAP" - offers a new > feature "Source code misspelling reports": This is very nice, thank you. Definitely some items in here that need fixing. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9161) Codespell report for "OpenLDAP" (on fossies.org)
by Jens.Schleusener＠fossies.org 30 Jan '20

30 Jan '20

Full_Name: Jens Schleusener Version: 2.4.49 OS: (irrelevant) URL: https://fossies.org/linux/misc/openldap/codespell.html Submission from: (NULL) (2003:cf:71a:5655:2905:9b7a:fef1:ab74) The FOSS server fossies.org - also supporting "OpenLDAP" - offers a new feature "Source code misspelling reports": https://fossies.org/features.html#codespell Such reports are normally only generated on request, but as Fossies administrator I have just created (for testing purposes) an analysis for the new "OpenLDAP" release 2.4.49: https://fossies.org/linux/misc/openldap/codespell.html That version-independent (not linked) URL should redirect always to the last report (if available), so currently to https://fossies.org/linux/misc/openldap-2.4.49.tgz/codespell.html Although after a first review some obviously wrong matches ("false positives") are already filtered out (ignored) please inform me if you find more of them so that I can force a new improved check if applicable. Just for information there are also two supplemental pages https://fossies.org/linux/misc/openldap/codespell_conf.html showing some used "codespell" configurations and https://fossies.org/linux/misc/openldap/codespell_fps.html showing all resulting obvious "false positives". If appropriate, additional reports for the current or also future development versions (these in a special "test" folder) could be willingly created.

1 0

Re: (ITS#9159) mdb_put failed: MDB_MAP_FULL: Environment mapsize limit reached(-30792)
by quanah＠symas.com 30 Jan '20

30 Jan '20

--On Thursday, January 30, 2020 2:38 PM +0530 Vijay Kumar <pasumarthivijaykumar(a)gmail.com> wrote: > > > Thank you Quanah, > > I am not familiar about your process of subscribing to lists. > I have gone though the documentation to understand the mdb size for > windows. Can you please share me the documentation you might referring? The information on the mailing lists and how to subscribe is directly accessible from the main page for the OpenLDAP project: <https://www.openldap.org/> Click on the link that says "Mailing lists" Click on the link that says "OpenLDAP-technical" to get to: <https://www.openldap.org/lists/mm/listinfo/openldap-technical> The man page for slapd-mdb(5) has all of the documentation: <https://www.openldap.org/software/man.cgi?query=slapd-mdb&apropos=0&sektion…> Again, the default MAXIMUM size for an MDB database is 10MB if the maxsize parameter is not set. You *must* set this value to something larger. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#9159) mdb_put failed: MDB_MAP_FULL: Environment mapsize limit reached(-30792)
by pasumarthivijaykumar＠gmail.com 30 Jan '20

30 Jan '20

--0000000000008f08ba059d57cdcc Content-Type: text/plain; charset="UTF-8" Thank you Quanah, I am not familiar about your process of subscribing to lists. I have gone though the documentation to understand the mdb size for windows. Can you please share me the documentation you might referring? Thank you. Regards, Vijay Kumar On Thu, Jan 30, 2020 at 12:46 PM Quanah Gibson-Mount <quanah(a)symas.com> wrote: > > > --On Thursday, January 30, 2020 6:09 AM +0000 > pasumarthivijaykumar(a)gmail.com wrote: > > > --00000000000025b743059d554b79 > > Content-Type: text/plain; charset="UTF-8" > > Content-Transfer-Encoding: quoted-printable > > > > Thanks for your reply I do know that info for windows also is it the > > same.? I am unable to use stats tool also. > > As I already stated, the maxsize parameter specifies the maximum size the > database can reach. > > > As I don=E2=80=99t see any reply from our many mails to > > openldap-technical(a)openldap.org I have created this case to clarify. > > > The openldap-technical list is only open to subscribers, if you want to > post to it you must subscribe first. > > > > Why don=E2=80=99t you forward the query to the concerned team instead > > closi= ng. > > Your query would not be forwarded because you failed to read and/or > understand the documentation. This system is for bug reports only. > > > No reply to us.! > > I did reply. This is clearly shown in the ITS. > > > Please let us know answer to solve a issue.! > > > I already provided you the answer in my earlier response. Set an > approprate maxsize setting, the default of 10MB is clearly too low. > > Regards, > Quanah > > > -- > > Quanah Gibson-Mount > Product Architect > Symas Corporation > Packaged, certified, and supported LDAP solutions powered by OpenLDAP: > <http://www.symas.com> > -- Thanks & Regards, Vijay Kumar *+91-94944 44009* --0000000000008f08ba059d57cdcc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Thank you Quanah,</div><div> </div>I am not famili= ar about your process of subscribing to lists.<div>I have gone though the d= ocumentation to understand the mdb size for windows. Can you please share m= e the documentation you might referring?</div><div> </div><div>Thank you= .</div><div> </div><div>Regards,</div><div>Vijay Kumar <div> </div= ></div></div> <div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail= _attr">On Thu, Jan 30, 2020 at 12:46 PM Quanah Gibson-Mount <<a href=3D"= mailto:quanah@symas.com">quanah(a)symas.com</a>> wrote: </div><blockquo= te class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid rgb(204,204,204);padding-left:1ex"> --On Thursday, January 30, 2020 6:09 AM +0000 <a href=3D"mailto:pasumarthivijaykumar@gmail.com" target=3D"_blank">pasumar= thivijaykumar(a)gmail.com</a> wrote: > --00000000000025b743059d554b79 > Content-Type: text/plain; charset=3D"UTF-8" > Content-Transfer-Encoding: quoted-printable > > Thanks for your reply I do know that info for windows also is it the<b= r> > same.? I am unable to use stats tool also. As I already stated, the maxsize parameter specifies the maximum size the <= br> database can reach. > As I don=3DE2=3D80=3D99t see any reply from our many mails to > <a href=3D"mailto:openldap-technical@openldap.org" target=3D"_blank">o= penldap-technical(a)openldap.org</a> I have created this case to clarify. The openldap-technical list is only open to subscribers, if you want to <br= > post to it you must subscribe first. > Why don=3DE2=3D80=3D99t you forward the query to the concerned team in= stead > closi=3D ng. Your query would not be forwarded because you failed to read and/or understand the documentation.=C2=A0 This system is for bug reports only.<br= > > No reply to us.! I did reply.=C2=A0 This is clearly shown in the ITS. > Please let us know answer to solve a issue.! I already provided you the answer in my earlier response.=C2=A0 Set an approprate maxsize setting, the default of 10MB is clearly too low. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <<a href=3D"http://www.symas.com" rel=3D"noreferrer" target=3D"_blank">h= ttp://www.symas.com</a>> </blockquote></div> <div> </div>-- <div dir=3D"ltr"= class=3D"gmail_signature"><div dir=3D"ltr"><div><div>Thanks & Regards,</div> <div>=C2=A0</div> <div>Vijay Kumar</div> <div>+91-94944 44009</div></div></div></div> --0000000000008f08ba059d57cdcc--

1 0

Re: (ITS#9159) mdb_put failed: MDB_MAP_FULL: Environment mapsize limit reached(-30792)
by quanah＠symas.com 30 Jan '20

30 Jan '20

--On Thursday, January 30, 2020 6:09 AM +0000 pasumarthivijaykumar(a)gmail.com wrote: > --00000000000025b743059d554b79 > Content-Type: text/plain; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > Thanks for your reply I do know that info for windows also is it the > same.? I am unable to use stats tool also. As I already stated, the maxsize parameter specifies the maximum size the database can reach. > As I don=E2=80=99t see any reply from our many mails to > openldap-technical(a)openldap.org I have created this case to clarify. The openldap-technical list is only open to subscribers, if you want to post to it you must subscribe first. > Why don=E2=80=99t you forward the query to the concerned team instead > closi= ng. Your query would not be forwarded because you failed to read and/or understand the documentation. This system is for bug reports only. > No reply to us.! I did reply. This is clearly shown in the ITS. > Please let us know answer to solve a issue.! I already provided you the answer in my earlier response. Set an approprate maxsize setting, the default of 10MB is clearly too low. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#9159) mdb_put failed: MDB_MAP_FULL: Environment mapsize limit reached(-30792)
by pasumarthivijaykumar＠gmail.com 30 Jan '20

30 Jan '20

--00000000000025b743059d554b79 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks for your reply I do know that info for windows also is it the same.? I am unable to use stats tool also. As I don=E2=80=99t see any reply from our many mails to openldap-technical(a)openldap.org I have created this case to clarify. Why don=E2=80=99t you forward the query to the concerned team instead closi= ng. No reply to us.! Please let us know answer to solve a issue.! On Wed, Jan 29, 2020 at 4:24 PM <openldap-its(a)openldap.org> wrote: > > *** THIS IS AN AUTOMATICALLY GENERATED REPLY *** > > Thanks for your report to the OpenLDAP Issue Tracking System. Your > report has been assigned the tracking number ITS#9159. > > One of our support engineers will look at your report in due course. > Note that this may take some time because our support engineers > are volunteers. They only work on OpenLDAP when they have spare > time. > > If you need to provide additional information in regards to your > issue report, you may do so by replying to this message. Note that > any mail sent to openldap-its(a)openldap.org with (ITS#9159) > in the subject will automatically be attached to the issue report. > > mailto:openldap-its@openldap.org?subject=3D(ITS#9159) > > You may follow the progress of this report by loading the following > URL in a web browser: > http://www.OpenLDAP.org/its/index.cgi?findid=3D9159 > > Please remember to retain your issue tracking number (ITS#9159) > on any further messages you send to us regarding this report. If > you don't then you'll just waste our time and yours because we > won't be able to properly track the report. > > Please note that the Issue Tracking System is not intended to > be used to seek help in the proper use of OpenLDAP Software. > Such requests will be closed. > > OpenLDAP Software is user supported. > http://www.OpenLDAP.org/support/ > > -------------- > Copyright 1998-2007 The OpenLDAP Foundation, All Rights Reserved. > > --=20 Thanks & Regards, Vijay Kumar *+91-94944 44009* --00000000000025b743059d554b79 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><pre style=3D"background-image:initial;background-position= :initial;background-size:initial;background-repeat:initial;background-origi= n:initial;background-clip:initial;margin:0in 0in 0.0001pt;font-size:10pt;fo= nt-family:"Courier New"">Thanks for y= our reply I do know that info for windows also is it the same.? I am unable= to use stats tool also.</pre><pre style=3D"background-image:initial= ;background-position:initial;background-size:initial;background-repeat:init= ial;background-origin:initial;background-clip:initial;margin:0in 0in 0.0001= pt;font-size:10pt;font-family:"Courier New"">=C2=A0</pre><pre sty= le=3D"background-image:initial;background-position:initial;background-size:= initial;background-repeat:initial;background-origin:initial;background-clip= :initial;margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier N= ew"">As I don=E2=80=99t see any reply from= our many mails to <a href=3D"mailto:openldap-technical@openldap.org" style= =3D"color:rgb(5,99,193)">openldap-technical(a)openldap.org</a> I have created= this case to clarify.</pre><pre style=3D"background-image:initial;b= ackground-position:initial;background-size:initial;background-repeat:initia= l;background-origin:initial;background-clip:initial;margin:0in 0in 0.0001pt= ;font-size:10pt;font-family:"Courier New"">=C2=A0</pre><pre style=3D"background-image:initial;background-= position:initial;background-size:initial;background-repeat:initial;backgrou= nd-origin:initial;background-clip:initial;margin:0in 0in 0.0001pt;font-size= :10pt;font-family:"Courier New"">Why = don=E2=80=99t you forward the query to the concerned team instead closing.<= /span></pre><pre style=3D"background-image:initial;background-position:init= ial;background-size:initial;background-repeat:initial;background-origin:ini= tial;background-clip:initial;margin:0in 0in 0.0001pt;font-size:10pt;font-fa= mily:"Courier New"">=C2=A0</pr= e><pre style=3D"background-image:initial;background-position:initial;backgr= ound-size:initial;background-repeat:initial;background-origin:initial;backg= round-clip:initial;margin:0in 0in 0.0001pt;font-size:10pt;font-family:&quot= ;Courier New"">No reply to us.!</pr= e><pre style=3D"background-image:initial;background-position:initial;backgr= ound-size:initial;background-repeat:initial;background-origin:initial;backg= round-clip:initial;margin:0in 0in 0.0001pt;font-size:10pt;font-family:&quot= ;Courier New"">=C2=A0</pre><pre sty= le=3D"background-image:initial;background-position:initial;background-size:= initial;background-repeat:initial;background-origin:initial;background-clip= :initial;margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier N= ew"">Please let us know answer to solve a = issue.!</pre></div> <div class=3D"gmail_quote"><div dir=3D"ltr" c= lass=3D"gmail_attr">On Wed, Jan 29, 2020 at 4:24 PM <<a href=3D"mailto:o= penldap-its(a)openldap.org">openldap-its(a)openldap.org</a>> wrote: </div= ><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border= -left:1px solid rgb(204,204,204);padding-left:1ex"> *** THIS IS AN AUTOMATICALLY GENERATED REPLY *** Thanks for your report to the OpenLDAP Issue Tracking System.=C2=A0 Your<br= > report has been assigned the tracking number ITS#9159. One of our support engineers will look at your report in due course. Note that this may take some time because our support engineers are volunteers.=C2=A0 They only work on OpenLDAP when they have spare time. If you need to provide additional information in regards to your issue report, you may do so by replying to this message.=C2=A0 Note that<br= > any mail sent to <a href=3D"mailto:openldap-its@openldap.org" target=3D"_bl= ank">openldap-its(a)openldap.org</a> with (ITS#9159) in the subject will automatically be attached to the issue report. =C2=A0 =C2=A0 =C2=A0 =C2=A0 mailto:<a href=3D"mailto:openldap-its@openldap.= org" target=3D"_blank">openldap-its(a)openldap.org</a>?subject=3D(ITS#9159)<b= r> You may follow the progress of this report by loading the following URL in a web browser: =C2=A0 =C2=A0 <a href=3D"http://www.OpenLDAP.org/its/index.cgi?findid=3D915= 9" rel=3D"noreferrer" target=3D"_blank">http://www.OpenLDAP.org/its/index.c= gi?findid=3D9159</a> Please remember to retain your issue tracking number (ITS#9159) on any further messages you send to us regarding this report.=C2=A0 If you don't then you'll just waste our time and yours because we won't be able to properly track the report. Please note that the Issue Tracking System is not intended to be used to seek help in the proper use of OpenLDAP Software. Such requests will be closed. OpenLDAP Software is user supported. =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"http://www.OpenLDAP.org/support/" re= l=3D"noreferrer" target=3D"_blank">http://www.OpenLDAP.org/support/</a> -------------- Copyright 1998-2007 The OpenLDAP Foundation, All Rights Reserved. </blockquote></div> <div> </div>-- <div dir=3D"ltr"= class=3D"gmail_signature"><div dir=3D"ltr"><div><div>Thanks & Regards,</div> <div>=C2=A0</div> <div>Vijay Kumar</div> <div>+91-94944 44009</div></div></div></div> --00000000000025b743059d554b79--

1 0

Re: (ITS#9159) mdb_put failed: MDB_MAP_FULL: Environment mapsize limit reached(-30792)
by quanah＠symas.com 29 Jan '20

29 Jan '20

--On Wednesday, January 29, 2020 10:54 AM +0000 pasumarthivijaykumar(a)gmail.com wrote: > Full_Name: Vijay Kumar > Version: 2.4.48 > OS: Windows > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (103.204.96.6) > > > I am not setting any map size for openldap with mdb configuration in > windows. Hello, Usage questions need to be sent to the openldap-technical(a)openldap.org email discussion list. The ITS system is for bug reports only. However, I would suggest carefully reading the slapd-mdb(5) man page, and consider the fact that the maxsize parameter determines what the allowed maximum size of the database is. If you do not set the parameter, it limits the database to 10MB in size (The default value). Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#9160) Out-of-memory handling issues
by ondra＠openldap.org 29 Jan '20

29 Jan '20

On Wed, Jan 29, 2020 at 11:46:23AM +0000, Howard Chu wrote: > ondra(a)openldap.org wrote: >> --- malloc_check.cocci --- >> @@ >> expression E, E1; >> identifier ch_alloc =~ "^(ch|slap)_.*alloc.*"; >> identifier f =~ "(?i).*alloc.*"; >> identifier foo; >> @@ >> >> E = >> ( >> ch_alloc > > ch_malloc always aborts if it fails, therefore failure checks are not used. > >> | >> entry_alloc >> | >> attr_alloc >> | > > These rely on ch_malloc, therefore failure checks are not used. > >> metaconn_alloc >> | >> asyncmeta_conn_alloc >> | >> CompAlloc /* component matching has massively bitrotted */ >> | >> * f Which is why only the last one is actually used - '*'. >> )(...); >> ... when != E = E1 >> when != E == NULL >> when != E != NULL >> ( >> E->foo >> | >> E[...] >> | >> *(E) >> ) >> --- malloc_check.cocci --- -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

1 0

Re: (ITS#9160) Out-of-memory handling issues
by hyc＠symas.com 29 Jan '20

29 Jan '20

ondra(a)openldap.org wrote: > Full_Name: Ondrej Kuznik > Version: re24/master > OS: > URL: > Submission from: (NULL) (212.159.69.225) > > > A look at the code highlights a few places where *alloc responses are used > unchecked. > > The following semantic patch is a good starting point. Currently it lists 87 > sites most of which probably need adjusting. > > --- malloc_check.cocci --- > @@ > expression E, E1; > identifier ch_alloc =~ "^(ch|slap)_.*alloc.*"; > identifier f =~ "(?i).*alloc.*"; > identifier foo; > @@ > > E = > ( > ch_alloc ch_malloc always aborts if it fails, therefore failure checks are not used. > | > entry_alloc> | > attr_alloc > | These rely on ch_malloc, therefore failure checks are not used. > metaconn_alloc > | > asyncmeta_conn_alloc > | > CompAlloc /* component matching has massively bitrotted */ > | > * f > )(...); > ... when != E = E1 > when != E == NULL > when != E != NULL > ( > E->foo > | > E[...] > | > *(E) > ) > --- malloc_check.cocci --- > > Patches coming soon. > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9160) Out-of-memory handling issues
by ondra＠openldap.org 29 Jan '20

29 Jan '20

Full_Name: Ondrej Kuznik Version: re24/master OS: URL: Submission from: (NULL) (212.159.69.225) A look at the code highlights a few places where *alloc responses are used unchecked. The following semantic patch is a good starting point. Currently it lists 87 sites most of which probably need adjusting. --- malloc_check.cocci --- @@ expression E, E1; identifier ch_alloc =~ "^(ch|slap)_.*alloc.*"; identifier f =~ "(?i).*alloc.*"; identifier foo; @@ E = ( ch_alloc | entry_alloc | attr_alloc | metaconn_alloc | asyncmeta_conn_alloc | CompAlloc /* component matching has massively bitrotted */ | * f )(...); ... when != E = E1 when != E == NULL when != E != NULL ( E->foo | E[...] | *(E) ) --- malloc_check.cocci --- Patches coming soon.

1 0

(ITS#9159) mdb_put failed: MDB_MAP_FULL: Environment mapsize limit reached(-30792)
by pasumarthivijaykumar＠gmail.com 29 Jan '20

29 Jan '20

Full_Name: Vijay Kumar Version: 2.4.48 OS: Windows URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (103.204.96.6) I am not setting any map size for openldap with mdb configuration in windows. It throws me an error as mentioned below 5e315f30 mdb_id2entry_put: mdb_put failed: MDB_MAP_FULL: Environment mapsize limit reached(-30792) "cn=azrbasisschema.xsd,cn=nl.ciz.awbz.webservice.awbztransformaties.aanvraag_2_io31_transformatie,cn=method sets,o=system,cn=abc,cn=defaultinst,o=abc.net" 5e315f30 conn=1000 op=8 RESULT tag=105 err=80 text=entry store failed 5e315f30 slap_graduate_commit_csn: removing 0000006519FF99A0 20200129103216.169110Z#000000#000#000000 can you please let me know what could be the issue.? I checked the memory used by customer; file system 30GB available out of 200GB, RAM is 24GB. all the way i see suggestions on linux olxDbMaxsize but for windows we dont set that.

1 0

Re: (ITS#9157) return additional error code information when OPT_X_TLS_NEWCTX fails
by shawn.michael.mckinney＠gmail.com 27 Jan '20

27 Jan '20

In one of our test envs, had the path wrong in replication config for = encryption artifacts: *** Conf excerpt: *** syncrepl =E2=80=A6 tls_key=3D/opt/symas/etc/openldap/file-name.pem *** Which gives generic error: *** Log Trace: *** an 27 17:25:17 sapz1a slapd[6203]: slapd starting =E2=80=A6 TLS context initialization failed (-1) Jan 27 17:25:17 hostname slapd[6203]: do_syncrepl: rid=3D031 rc -1 = retrying (4 retries left) *** Would have been helpful for the message to specify which artifact = wasn=E2=80=99t found, the line number of the config, or some other way = of narrowing the problem. =E2=80=94 Shawn

1 0

(ITS#9157) return additional error code information when OPT_X_TLS_NEWCTX fails
by tagrain＠gmail.com 24 Jan '20

24 Jan '20

Full_Name: Thomas Grainger Version: libldap-2.4-2 OS: Ubuntu URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (62.60.6.23) when setting OPT_X_TLS_NEWCTX after an invalid OPT_X_TLS_CACERTFILE libldap returns a generic error code would it be possible to pass the underlying TLS library's error code instead or in addition? https://github.com/python-ldap/python-ldap/issues/55#issuecomment-577831055

1 0

(ITS#9156) latest ppolicy draft support
by ondra＠openldap.org 24 Jan '20

24 Jan '20

Full_Name: Ondrej Kuznik Version: master OS: URL: https://github.com/mistotebe/openldap/tree/ppolicy10 Submission from: (NULL) (212.159.69.225) There have been a few iterations of the ppolicy behera draft and it would be useful to support (some) of the documented features. The following are coming: - pwdGraceExpiry - pwdMaxLength - pwdStartTime and pwdEndTime - pwdMinDelay and pwdMaxDelay (as a soft-lockout, contingent with other implementations based on an informal survey during latest LDAPCon) - pwdMaxIdle (and moving the relevant parts of lastbind overlay into slapd code as it is a shared dependency of a few things now) - the ability to store per-policy arguments to check_password() - updates/fixes to the draft that should have been identified before some of the above were proposed (mostly typos and ASN.1 schema updates)

1 0

Re: (ITS#9155) LMDB: List of spill pages leaks after transaction
by hyc＠symas.com 23 Jan '20

23 Jan '20

babayants.alexander(a)gmail.com wrote: > Full_Name: Alexander Babayants > Version: LMDB 0.9.24 > OS: Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (188.170.78.53) > > > Hi! > > ASAN shows that the list of spill pages is not freed after transaction: Thanks for the report. Fixed now in mdb.master and mdb.RE/0.9 > ================================================================= > ==test==13046==ERROR: LeakSanitizer: detected memory leaks > > Direct leak of 20971376 byte(s) in 1 object(s) allocated from: > #0 0x4bcbb2 in realloc (test+0x4bcbb2) > #1 0x7f067ef5ed59 in mdb_midl_grow liblmdb/src/midl.c:135:8 > #2 0x7f067ef5ec83 in mdb_midl_append liblmdb/src/midl.c:162:7 > #3 0x7f067ef4456b in mdb_page_spill liblmdb/src/mdb.c:2077:13 > #4 0x7f067ef3960b in mdb_cursor_put liblmdb/src/mdb.c:6640:14 > #5 0x7f067ef5106c in mdb_put liblmdb/src/mdb.c:9027:7 > #6 0x4ec674 in main test.c:28:14 > #7 0x7f067b46d544 in __libc_start_main (/lib64/libc.so.6+0x22544) > > SUMMARY: AddressSanitizer: 20971376 byte(s) leaked in 1 allocation(s). > > Reproducible on LMDB 0.9.24 with the following code (anything doing > long write transaction will trigger this): > #include "lmdb.h" > > int main(int argc,char * argv[]) > { > int rc; > MDB_env *env; > MDB_dbi dbi; > MDB_val key, data; > MDB_txn *txn; > int kval; > char sval[1024] = {0}; // fine to fill with zeroes > > rc = mdb_env_create(&env); > rc = mdb_env_set_mapsize(env, 10ULL * 1024 * 1024 * 1024); // 10 GiB > rc = mdb_env_open(env, "./testdb", 0, 0664); > rc = mdb_txn_begin(env, NULL, 0, &txn); > rc = mdb_dbi_open(txn, NULL, 0, &dbi); > > key.mv_size = sizeof(int); > key.mv_data = &kval; > data.mv_size = sizeof(sval); > data.mv_data = sval; > > for (kval = 0; kval < 100'000'000; ++kval) { > rc = mdb_put(txn, dbi, &key, &data, 0); > } > rc = mdb_txn_commit(txn); > > mdb_dbi_close(env, dbi); > mdb_env_close(env); > return 0; > } > >>From my point of view the leak looks benign (=does not lead to > database corruption or such), but I didn't dive deep enough in LMDB > internals to be completely certain about that. > > Regards, > Alexander Babayants. > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9126) add: pwdChangedTime leads to seg fault
by ondra＠mistotebe.net 23 Jan '20

23 Jan '20

On Mon, Jan 13, 2020 at 09:35:18PM +0000, michael(a)stroeder.com wrote: > This still happens with current RE24 snapshot. > > Is more information needed to address this? Hi, had a look at it and a proposed patch is in a branch in my GitHub repo: https://github.com/mistotebe/openldap/tree/its9126 Let me know if this fixes the issue for you. As far as I have been able to figure out, the database content generated by ppolicy until now was inconsistent with the schema (normalised value was missing) and existing values will still cause the crash to happen. A fix is to replace the affected values or export/reload the database. Thanks, -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

1 0

(ITS#9155) LMDB: List of spill pages leaks after transaction
by babayants.alexander＠gmail.com 23 Jan '20

23 Jan '20

Full_Name: Alexander Babayants Version: LMDB 0.9.24 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (188.170.78.53) Hi! ASAN shows that the list of spill pages is not freed after transaction: ================================================================= ==test==13046==ERROR: LeakSanitizer: detected memory leaks Direct leak of 20971376 byte(s) in 1 object(s) allocated from: #0 0x4bcbb2 in realloc (test+0x4bcbb2) #1 0x7f067ef5ed59 in mdb_midl_grow liblmdb/src/midl.c:135:8 #2 0x7f067ef5ec83 in mdb_midl_append liblmdb/src/midl.c:162:7 #3 0x7f067ef4456b in mdb_page_spill liblmdb/src/mdb.c:2077:13 #4 0x7f067ef3960b in mdb_cursor_put liblmdb/src/mdb.c:6640:14 #5 0x7f067ef5106c in mdb_put liblmdb/src/mdb.c:9027:7 #6 0x4ec674 in main test.c:28:14 #7 0x7f067b46d544 in __libc_start_main (/lib64/libc.so.6+0x22544) SUMMARY: AddressSanitizer: 20971376 byte(s) leaked in 1 allocation(s). Reproducible on LMDB 0.9.24 with the following code (anything doing long write transaction will trigger this): #include "lmdb.h" int main(int argc,char * argv[]) { int rc; MDB_env *env; MDB_dbi dbi; MDB_val key, data; MDB_txn *txn; int kval; char sval[1024] = {0}; // fine to fill with zeroes rc = mdb_env_create(&env); rc = mdb_env_set_mapsize(env, 10ULL * 1024 * 1024 * 1024); // 10 GiB rc = mdb_env_open(env, "./testdb", 0, 0664); rc = mdb_txn_begin(env, NULL, 0, &txn); rc = mdb_dbi_open(txn, NULL, 0, &dbi); key.mv_size = sizeof(int); key.mv_data = &kval; data.mv_size = sizeof(sval); data.mv_data = sval; for (kval = 0; kval < 100'000'000; ++kval) { rc = mdb_put(txn, dbi, &key, &data, 0); } rc = mdb_txn_commit(txn); mdb_dbi_close(env, dbi); mdb_env_close(env); return 0; } >From my point of view the leak looks benign (=does not lead to database corruption or such), but I didn't dive deep enough in LMDB internals to be completely certain about that. Regards, Alexander Babayants.

1 0

Re: (ITS#9154) RFE: Number of entries in MDB database in cn=monitor
by michael＠stroeder.com 18 Jan '20

18 Jan '20

On 1/18/20 1:54 PM, Howard Chu wrote: > michael(a)stroeder.com wrote: >> The number of entries could be added there as attribute 'olmMDBEntries' or >> similar. > > Added in git master Great! Thank you. Any chance to see it in RE24? At least the back-port patch seems to work smoothly in my local test environment. Ciao, Michael.

1 0

Re: (ITS#9154) RFE: Number of entries in MDB database in cn=monitor
by hyc＠symas.com 18 Jan '20

18 Jan '20

michael(a)stroeder.com wrote: > Full_Name: Michael Str.der > Version: master > OS: > URL: > Submission from: (NULL) (213.240.182.99) > > > As a system engineer I want to see the number of entries within a mdb database > in the monitoring (e.g. to alarm unusual fast changes due to false deletions). > > While one can use mdb_stat or other custom scripts to monitor the number of > entries of the id2e sub-DB this requires read access to the MDB environment. But > for security reasons this should be avoided. > > olmMDBDatabase entries in cn=monitor already have attributes for MDB page > counts. > The number of entries could be added there as attribute 'olmMDBEntries' or > similar. Added in git master -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9154) RFE: Number of entries in MDB database in cn=monitor
by michael＠stroeder.com 17 Jan '20

17 Jan '20

Full_Name: Michael Str.der Version: master OS: URL: Submission from: (NULL) (213.240.182.99) As a system engineer I want to see the number of entries within a mdb database in the monitoring (e.g. to alarm unusual fast changes due to false deletions). While one can use mdb_stat or other custom scripts to monitor the number of entries of the id2e sub-DB this requires read access to the MDB environment. But for security reasons this should be avoided. olmMDBDatabase entries in cn=monitor already have attributes for MDB page counts. The number of entries could be added there as attribute 'olmMDBEntries' or similar.

1 0

(ITS#9153) replication.sdf inconsistent whitespace
by ryan＠openldap.org 14 Jan '20

14 Jan '20

Full_Name: Ryan Tandy Version: RE24 OS: URL: https://github.com/openldap/openldap/compare/OPENLDAP_REL_ENG_2_4...rtandy:… Submission from: (NULL) (70.66.128.207) Submitted by: ryan This patch fixes indentation of two 'maxsize' lines in the 'Syncrepl Proxy' example. For RE24 only as master lacks these lines (will fix separately).

1 0

(ITS#9152) Configuring autoca before database exists crashes slapd
by quanah＠openldap.org 13 Jan '20

13 Jan '20

Full_Name: Quanah Gibson-Mount Version: HEAD OS: N/A URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.143.26) If you configure autoca via cn=config when the underlying DB does not yet exist, slapd will crash. i.e., I added this entry to my cn=config db: dn: olcOverlay=autoca,olcDatabase={1}mdb,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcACAConfig olcOverlay: autoca olcACAlocalDN: cn=anvil4.rb.symas.net,ou=Servers,dc=rb,dc=symas,dc=net while the mdb database was not yet populated.

1 0

Re: (ITS#9126) add: pwdChangedTime leads to seg fault
by michael＠stroeder.com 13 Jan '20

13 Jan '20

This still happens with current RE24 snapshot. Is more information needed to address this?

1 0

Re: (ITS#9151) slapd-sock - stumped
by matthew＠schlutech.com 12 Jan '20

12 Jan '20

I will move this there thank you. It is difficult to know where to go - this seems like a bug as the behavior is pervasive in all my testing. I have now tried multiple OS and multiple versions of OpenLDAP. As for the path differences - that was just me copying/pasting from two separate tests. I assure you the problem persists when the paths are correct. As you pointed out - inconsistent in the information I provided therein - not in actual testing. Thank you. On 1/12/20 3:30 AM, Michael Ströder wrote: > On 1/11/20 9:03 PM, matthew(a)schlutech.com wrote: >> Always results in socket connect failed. >> >> database sock >> socketpath /tmp/example.sock >> [..] >> Always: socket connect(/root/example.sock) failed > /tmp/example.sock != /root/example.sock > > Either something's inconsistent in your config or the information you > provided herein. > > Also this does not seem to be a bug report but the ITS is only for > reporting bugs. Please send usage questions to openldap-technical > mailing list. > > Ciao, Michael. > >

1 0

Re: (ITS#9151) slapd-sock - stumped
by michael＠stroeder.com 12 Jan '20

12 Jan '20

On 1/11/20 9:03 PM, matthew(a)schlutech.com wrote: > Always results in socket connect failed. > > database sock > socketpath /tmp/example.sock > [..] > Always: socket connect(/root/example.sock) failed /tmp/example.sock != /root/example.sock Either something's inconsistent in your config or the information you provided herein. Also this does not seem to be a bug report but the ITS is only for reporting bugs. Please send usage questions to openldap-technical mailing list. Ciao, Michael.

1 0

(ITS#9151) slapd-sock - stumped
by matthew＠schlutech.com 11 Jan '20

11 Jan '20

Full_Name: W. Matthew Schlueter Version: 2.4.45+dfsg-1ubuntu1.4 OS: Ubuntu 18.04.3 LTS URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (172.101.204.19) Always results in socket connect failed. Using the examples from the source tree: modulepath /usr/lib/ldap moduleload back_sock.so include /etc/ldap/schema/core.schema database sock socketpath /tmp/example.sock suffix "dc=test,dc=com" Starting slapd: slapd -4 -f /etc/ldap/slapd.conf -u root -g root -d1 Using the perl script example from the source tree plus tried endless debugging with my own code. File permissions all look correct. Scoured for answers. Stumped. Always: socket connect(/root/example.sock) failed

1 0

Re: (ITS#9150) olcDbNoSync: FALSE has the same effect as olcDbNoSync: TRUE
by hyc＠symas.com 11 Jan '20

11 Jan '20

maxime.besson(a)worteks.com wrote: > Full_Name: Maxime Besson > Version: 2.4.48 > OS: Debian Buster / CentOS7 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (2a01:cb00:802:8400:2cbe:3c60:fca6:e50b) > > > While migrating configuration from slapd.conf to cn=config, I noticed something > pretty strange related to the dbnosync option Thanks for the report, fixed now in git master. > > The result of converting the following config: > > include ./core.schema > pidfile ./slapd.1.pid > argsfile ./slapd.1.args > > > database mdb > suffix "dc=example,dc=com" > rootdn "cn=Manager,dc=example,dc=com" > rootpw secret > directory db.1.a > index objectClass eq > index cn,sn,uid pres,eq,sub > index entryUUID,entryCSN eq > > > Is (only interested in MDB config): > # AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify. > # CRC32 3df9c3d6 > dn: olcDatabase={1}mdb > objectClass: olcDatabaseConfig > objectClass: olcMdbConfig > olcDatabase: {1}mdb > olcSuffix: dc=example,dc=com > olcAddContentAcl: FALSE > olcLastMod: TRUE > olcMaxDerefDepth: 15 > olcReadOnly: FALSE > olcRootDN: cn=Manager,dc=example,dc=com > olcRootPW:: c2VjcmV0 > olcSyncUseSubentry: FALSE > olcMonitoring: TRUE > olcDbDirectory: db.1.a > olcDbNoSync: FALSE > olcDbIndex: objectClass eq > olcDbIndex: entryUUID eq > olcDbIndex: entryCSN eq > olcDbIndex: cn pres,eq,sub > olcDbIndex: uid pres,eq,sub > olcDbIndex: sn pres,eq,sub > olcDbMaxReaders: 0 > olcDbMaxSize: 10485760 > olcDbMode: 0600 > olcDbSearchStack: 16 > olcDbMaxEntrySize: 0 > olcDbRtxnSize: 10000 > > > As you can see, olcDbNoSync has the value FALSE, because there was no "dbnosync" > option in slapd.conf, however, performance testing shows a dramatic increase in > write speed after slapd.d migration. Which seems odd to me. It seems like the > MDB database was not syncing to disk anymore despite olcDbNoSync being FALSE > > Removing olcDbNoSync: FALSE from cn=config did restore previous performance and > syncing behavior > > Looking at servers/slapd/back-mdb/config.c:723 I found this: > > case MDB_DBNOSYNC: > if ( c->value_int ) > mdb->mi_dbenv_flags |= MDB_NOSYNC; > else > mdb->mi_dbenv_flags ^= MDB_NOSYNC; > > > The else case seems wrong, I replaced it with > > case MDB_DBNOSYNC: > if ( c->value_int ) > mdb->mi_dbenv_flags |= MDB_NOSYNC; > else > mdb->mi_dbenv_flags &= ~MDB_NOSYNC; > > And it solves the issue on my system. > > Reproduced on master, and 2.4.48, on Debian > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9147) syncrepl connexion leak when provider uses an expired certificate
by hyc＠symas.com 11 Jan '20

11 Jan '20

maxime.besson(a)worteks.com wrote: > Full_Name: Maxime Besson > Version: 2.4.48 > OS: Debian Buster / CentOS7 > URL: https://cloud.worteks.com/index.php/s/9CE6ALLaAfrxZW4/download > Submission from: (NULL) (92.184.104.113) Thanks for the report and testcase, fixed now in git master. > > > I have two OpenLDAP (2.4.48, Ubuntu) servers running with syncrepl in > mirrormode. > > One of my server's X509 certificates has recently expired, and I noticed that > while it was expired, the other node's connection count kept climbing until it > a "Max open files" condition. It seems that when a Syncrepl consumer encounters > a certificate error, the outgoing LDAP connection to the provider is never > closed. > > Attached to this bug you will find a test case to reproduce this behavior > > # runs a provider with a bogus certificate > # and a consumer with retry=3+ > sh test.sh > ... > TLS certificate verification: Error, self signed certificate > TLS trace: SSL3 alert write:fatal:unknown CA > TLS trace: SSL_connect:error in error > TLS: can't connect: error:1416F086:SSL > routines:tls_process_server_certificate:certificate verify failed (self signed > certificate). > 5e160966 slap_client_connect: URI=ldaps://127.0.1.1:6636/ > DN="cn=manager,dc=example,dc=com" ldap_sasl_bind_s failed (-1) > 5e160966 do_syncrepl: rid=001 rc -1 retrying > ... > and so on, every 3 seconds > > While the consumer retries, running lsof on its PID will reveal the connection > leak: > > 42u TCP localhost:45964->localhost.lan:6636 (CLOSE_WAIT) > 43u TCP localhost:45966->localhost.lan:6636 (CLOSE_WAIT) > 44u TCP localhost:45968->localhost.lan:6636 (CLOSE_WAIT) > 45u TCP localhost:45970->localhost.lan:6636 (CLOSE_WAIT) > 46u TCP localhost:45972->localhost.lan:6636 (CLOSE_WAIT) > 47u TCP localhost:45974->localhost.lan:6636 (CLOSE_WAIT) > 48u TCP localhost:45976->localhost.lan:6636 (CLOSE_WAIT) > 49u TCP localhost:45978->localhost.lan:6636 (CLOSE_WAIT) > 50u TCP localhost:45980->localhost.lan:6636 (CLOSE_WAIT) > 51u TCP localhost:45982->localhost.lan:6636 (CLOSE_WAIT) > 52u TCP localhost:45984->localhost.lan:6636 (CLOSE_WAIT) > 53u TCP localhost:45986->localhost.lan:6636 (CLOSE_WAIT) > 54u TCP localhost:45988->localhost.lan:6636 (CLOSE_WAIT) > 55u TCP localhost:45990->localhost.lan:6636 (CLOSE_WAIT) > 56u TCP localhost:45992->localhost.lan:6636 (CLOSE_WAIT) > 57u TCP localhost:45994->localhost.lan:6636 (CLOSE_WAIT) > 58u TCP localhost:45996->localhost.lan:6636 (CLOSE_WAIT) > 59u TCP localhost:45998->localhost.lan:6636 (CLOSE_WAIT) > 60u TCP localhost:46000->localhost.lan:6636 (CLOSE_WAIT) > 61u TCP localhost:46002->localhost.lan:6636 (CLOSE_WAIT) > 62u TCP localhost:46004->localhost.lan:6636 (CLOSE_WAIT) > 63u TCP localhost:46006->localhost.lan:6636 (CLOSE_WAIT) > 64u TCP localhost:46008->localhost.lan:6636 (CLOSE_WAIT) > > > Modifying the provider URL in slapd.2.conf with a wrong port causes the > syncrepl > consumer to fail and retry just as much, but without connection piling up in > CLOSE_WAIT state. > > This is not a very critical issue because it only affects servers who are in > already degraded condition (broken replication, invalid certificate on the > provider) but I thought it still was worth reporting. > > I was able to reproduce this issue on the git master branch, on released 2.4.48, > on Centos7 and on Debian Buster. > OpenSSL version on the debian system: 1.1.1d-0+deb10u2 > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9124) CVE-ID?
by michael＠stroeder.com 10 Jan '20

10 Jan '20

On 1/10/20 2:28 PM, Stephan Zeisberg wrote: > So far I have not requested a CVE-Id for the issue. That's what Howard > wrote in this regard: > >> Usual practice for CVEs is not to make them public until fixes are >> released. In the future, you should tick the Major Security Issue >> button for potential CVEs so they can be handled privately before >> release.> > I am not aware of a release including the bugfix for the issue. If the > release already exists I am happy to request a CVE-Id for it First of all, many thanks for finding and submitting issues like this. Disclaimer: I'm not an official OpenLDAP project member and I'm not an expert for this CVE-ID process. >From my understanding you can request a CVE-ID which is kept confidential until the vendor developed a fix. This is useful to already have a unique reference for all the work done upstream to fix a particular security issue and for applying back-port patches to downstream packages (e.g. in Linux distributions). Furthermore OpenLDAP's ITS allows to mark an issue as security issue which hides it from public access. I read Howard's comment that he meant exactly this. Ciao, Michael.

1 0

Re: (ITS#9124) CVE-ID?
by stephan＠srlabs.de 10 Jan '20

10 Jan '20

--_000_e1c562ac33aa2ed6c8e93818cf4da3fasrlabsde_ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 SGkgTWljaGFlbCDigJQNCg0KU28gZmFyIEkgaGF2ZSBub3QgcmVxdWVzdGVkIGEgQ1ZFLUlkIGZv ciB0aGUgaXNzdWUuIFRoYXQncyB3aGF0IEhvd2FyZCB3cm90ZSBpbiB0aGlzIHJlZ2FyZDoNCg0K VXN1YWwgcHJhY3RpY2UgZm9yIENWRXMgaXMgbm90IHRvIG1ha2UgdGhlbSBwdWJsaWMgdW50aWwg Zml4ZXMgYXJlIHJlbGVhc2VkLiBJbg0KdGhlDQpmdXR1cmUsIHlvdSBzaG91bGQgdGljayB0aGUg TWFqb3IgU2VjdXJpdHkgSXNzdWUgYnV0dG9uIGZvciBwb3RlbnRpYWwgQ1ZFcyBzbw0KdGhleQ0K Y2FuIGJlIGhhbmRsZWQgcHJpdmF0ZWx5IGJlZm9yZSByZWxlYXNlLg0KDQoNCkkgYW0gbm90IGF3 YXJlIG9mIGEgcmVsZWFzZSBpbmNsdWRpbmcgdGhlIGJ1Z2ZpeCBmb3IgdGhlIGlzc3VlLiBJZiB0 aGUgcmVsZWFzZSBhbHJlYWR5IGV4aXN0cyBJIGFtIGhhcHB5IHRvIHJlcXVlc3QgYSBDVkUtSWQg Zm9yIGl0DQoNCkNoZWVycw0KDQogICAgLVN0ZXBoYW4NCg0KT24gMS8xMC8yMCAyOjA0IFBNLCBN aWNoYWVsIFN0csO2ZGVyIHdyb3RlOg0KDQpTdGVwaGFuLA0KDQpyZWdhcmRpbmc6DQoNCmh0dHBz Oi8vd3d3Lm9wZW5sZGFwLm9yZy9pdHMvaW5kZXguY2dpP2ZpbmRpZD05MTI0DQoNCldhcyB0aGVy ZSBldmVyIGEgQ1ZFLUlkIGFzc2lnbmVkIHRvIHRoaXMgaXNzdWU/IEknZCBsaWtlIHRvIHJlZmVy ZW5jZSBpdA0KaW4gYmFjay1wb3J0IHBhdGNoZXMgZm9yIGRvd25zdHJlYW0gcGFja2FnZXMuDQoN CkNpYW8sIE1pY2hhZWwuDQoNCg0K --_000_e1c562ac33aa2ed6c8e93818cf4da3fasrlabsde_ Content-Type: text/html; charset="utf-8" Content-ID: <53AC3735BCF1C94BAC7633E57FADE8D2(a)EURPRD10.PROD.OUTLOOK.COM> Content-Transfer-Encoding: base64 PGh0bWw+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0i dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjwvaGVhZD4NCjxib2R5Pg0KPHA+SGkgTWljaGFl bCDigJQ8L3A+DQo8cD5TbyBmYXIgSSBoYXZlIG5vdCByZXF1ZXN0ZWQgYSBDVkUtSWQgZm9yIHRo ZSBpc3N1ZS4gVGhhdCdzIHdoYXQgSG93YXJkIHdyb3RlIGluIHRoaXMgcmVnYXJkOjxicj4NCjwv cD4NCjxwcmUgc3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtc3R5bGU6IG5vcm1hbDsg Zm9udC12YXJpYW50LWxpZ2F0dXJlczogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFs OyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyBvcnBoYW5zOiAyOyB0 ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7 IHdpZG93czogMjsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6 IDBweDsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDI1NSwgMjU1LCAyNTUpOyB0ZXh0LWRlY29yYXRp b24tc3R5bGU6IGluaXRpYWw7IHRleHQtZGVjb3JhdGlvbi1jb2xvcjogaW5pdGlhbDsiPjxibG9j a3F1b3RlIHR5cGU9ImNpdGUiPjxwcmUgc3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQt c3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWxpZ2F0dXJlczogbm9ybWFsOyBmb250LXZhcmlh bnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFs OyBvcnBoYW5zOiAyOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10 cmFuc2Zvcm06IG5vbmU7IHdpZG93czogMjsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4 dC1zdHJva2Utd2lkdGg6IDBweDsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDI1NSwgMjU1LCAyNTUp OyB0ZXh0LWRlY29yYXRpb24tc3R5bGU6IGluaXRpYWw7IHRleHQtZGVjb3JhdGlvbi1jb2xvcjog aW5pdGlhbDsiPlVzdWFsIHByYWN0aWNlIGZvciBDVkVzIGlzIG5vdCB0byBtYWtlIHRoZW0gcHVi bGljIHVudGlsIGZpeGVzIGFyZSByZWxlYXNlZC4gSW4NCnRoZQ0KZnV0dXJlLCB5b3Ugc2hvdWxk IHRpY2sgdGhlIE1ham9yIFNlY3VyaXR5IElzc3VlIGJ1dHRvbiBmb3IgcG90ZW50aWFsIENWRXMg c28NCnRoZXkNCmNhbiBiZSBoYW5kbGVkIHByaXZhdGVseSBiZWZvcmUgcmVsZWFzZS4NCjwvcHJl PjwvYmxvY2txdW90ZT48L3ByZT4NCjxkaXYgY2xhc3M9Im1vei1jaXRlLXByZWZpeCI+SSBhbSBu b3QgYXdhcmUgb2YgYSByZWxlYXNlIGluY2x1ZGluZyB0aGUgYnVnZml4IGZvciB0aGUgaXNzdWUu IElmIHRoZSByZWxlYXNlIGFscmVhZHkgZXhpc3RzIEkgYW0gaGFwcHkgdG8gcmVxdWVzdCBhIENW RS1JZCBmb3IgaXQ8L2Rpdj4NCjxkaXYgY2xhc3M9Im1vei1jaXRlLXByZWZpeCI+PGJyPg0KPC9k aXY+DQo8ZGl2IGNsYXNzPSJtb3otY2l0ZS1wcmVmaXgiPkNoZWVyczwvZGl2Pg0KPGRpdiBjbGFz cz0ibW96LWNpdGUtcHJlZml4Ij48YnI+DQo8L2Rpdj4NCjxkaXYgY2xhc3M9Im1vei1jaXRlLXBy ZWZpeCI+Jm5ic3A7Jm5ic3A7Jm5ic3A7IC1TdGVwaGFuPGJyPg0KPC9kaXY+DQo8ZGl2IGNsYXNz PSJtb3otY2l0ZS1wcmVmaXgiPjxicj4NCjwvZGl2Pg0KPGRpdiBjbGFzcz0ibW96LWNpdGUtcHJl Zml4Ij5PbiAxLzEwLzIwIDI6MDQgUE0sIE1pY2hhZWwgU3Ryw7ZkZXIgd3JvdGU6PGJyPg0KPC9k aXY+DQo8YmxvY2txdW90ZSB0eXBlPSJjaXRlIiBjaXRlPSJtaWQ6MDA5YTgyNTgtZTgxNC1iYTI0 LTliYTktN2UyNDg0OTk3ZTRkQHN0cm9lZGVyLmNvbSI+DQo8cHJlIGNsYXNzPSJtb3otcXVvdGUt cHJlIiB3cmFwPSIiPlN0ZXBoYW4sDQoNCnJlZ2FyZGluZzoNCg0KPGEgY2xhc3M9Im1vei10eHQt bGluay1mcmVldGV4dCIgaHJlZj0iaHR0cHM6Ly93d3cub3BlbmxkYXAub3JnL2l0cy9pbmRleC5j Z2k/ZmluZGlkPTkxMjQiPmh0dHBzOi8vd3d3Lm9wZW5sZGFwLm9yZy9pdHMvaW5kZXguY2dpP2Zp bmRpZD05MTI0PC9hPg0KDQpXYXMgdGhlcmUgZXZlciBhIENWRS1JZCBhc3NpZ25lZCB0byB0aGlz IGlzc3VlPyBJJ2QgbGlrZSB0byByZWZlcmVuY2UgaXQNCmluIGJhY2stcG9ydCBwYXRjaGVzIGZv ciBkb3duc3RyZWFtIHBhY2thZ2VzLg0KDQpDaWFvLCBNaWNoYWVsLg0KDQo8L3ByZT4NCjwvYmxv Y2txdW90ZT4NCjwvYm9keT4NCjwvaHRtbD4NCg== --_000_e1c562ac33aa2ed6c8e93818cf4da3fasrlabsde_--

1 0

Re: (ITS#9124) CVE-ID?
by michael＠stroeder.com 10 Jan '20

10 Jan '20

Stephan, regarding: https://www.openldap.org/its/index.cgi?findid=9124 Was there ever a CVE-Id assigned to this issue? I'd like to reference it in back-port patches for downstream packages. Ciao, Michael.

1 0

Re: (ITS#9098) assert fails in meta_back_search in some cases after reconnect
by maxime.besson＠worteks.com 10 Jan '20

10 Jan '20

Hi, After a couple of months without any issues using 2.4.48, we suddenly encountered a crash again, but this time on 2.4.48. It was the exact same symptom, and the same assert failing as in my original message. It appears that the issue happens a lot more rarely in 2.4.48 compared to 2.4.47, so ITS#8841 possibly had an effect on my issue, but did not solve it completely. I have a core dump I can run tests against, if you need. -- Maxime Besson Expert Infrastructure - Worteks maxime.besson(a)worteks.com

1 0

(ITS#9150) olcDbNoSync: FALSE has the same effect as olcDbNoSync: TRUE
by maxime.besson＠worteks.com 09 Jan '20

09 Jan '20

Full_Name: Maxime Besson Version: 2.4.48 OS: Debian Buster / CentOS7 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2a01:cb00:802:8400:2cbe:3c60:fca6:e50b) While migrating configuration from slapd.conf to cn=config, I noticed something pretty strange related to the dbnosync option The result of converting the following config: include ./core.schema pidfile ./slapd.1.pid argsfile ./slapd.1.args database mdb suffix "dc=example,dc=com" rootdn "cn=Manager,dc=example,dc=com" rootpw secret directory db.1.a index objectClass eq index cn,sn,uid pres,eq,sub index entryUUID,entryCSN eq Is (only interested in MDB config): # AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify. # CRC32 3df9c3d6 dn: olcDatabase={1}mdb objectClass: olcDatabaseConfig objectClass: olcMdbConfig olcDatabase: {1}mdb olcSuffix: dc=example,dc=com olcAddContentAcl: FALSE olcLastMod: TRUE olcMaxDerefDepth: 15 olcReadOnly: FALSE olcRootDN: cn=Manager,dc=example,dc=com olcRootPW:: c2VjcmV0 olcSyncUseSubentry: FALSE olcMonitoring: TRUE olcDbDirectory: db.1.a olcDbNoSync: FALSE olcDbIndex: objectClass eq olcDbIndex: entryUUID eq olcDbIndex: entryCSN eq olcDbIndex: cn pres,eq,sub olcDbIndex: uid pres,eq,sub olcDbIndex: sn pres,eq,sub olcDbMaxReaders: 0 olcDbMaxSize: 10485760 olcDbMode: 0600 olcDbSearchStack: 16 olcDbMaxEntrySize: 0 olcDbRtxnSize: 10000 As you can see, olcDbNoSync has the value FALSE, because there was no "dbnosync" option in slapd.conf, however, performance testing shows a dramatic increase in write speed after slapd.d migration. Which seems odd to me. It seems like the MDB database was not syncing to disk anymore despite olcDbNoSync being FALSE Removing olcDbNoSync: FALSE from cn=config did restore previous performance and syncing behavior Looking at servers/slapd/back-mdb/config.c:723 I found this: case MDB_DBNOSYNC: if ( c->value_int ) mdb->mi_dbenv_flags |= MDB_NOSYNC; else mdb->mi_dbenv_flags ^= MDB_NOSYNC; The else case seems wrong, I replaced it with case MDB_DBNOSYNC: if ( c->value_int ) mdb->mi_dbenv_flags |= MDB_NOSYNC; else mdb->mi_dbenv_flags &= ~MDB_NOSYNC; And it solves the issue on my system. Reproduced on master, and 2.4.48, on Debian

1 0

Re: (ITS#9146) incomplete initialization of sessionlog structure may cause session log to grow indefinitely
by hyc＠symas.com 08 Jan '20

08 Jan '20

maxime.besson(a)worteks.com wrote: > Full_Name: Maxime Besson > Version: 2.4.48 > OS: Amazon Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (92.184.104.114) Thanks for the report, fixed now in git master. > > > During a cloud provider migration, I have migrated an OpenLDAP instance using > MDB + Syncrepl (in mirror mode, with session log) from Ubuntu to Amazon Linux. > > When starting a server with no data, syncrepl kicks in and starts replicating > the other server's database (12G, 7M entries). > > In Ubuntu this phase takes about 25 minutes (with dbnosync). In Amazon Linux, > with the exact same cn=config and daemon options, it took 10 days on first try, > at 100% CPU consumption the whole time. > > I observed that the start of the import is pretty fast (data.mdb grows by > 10M/s) but import speed slows down rapidly to a crawl. > > After some investigation, I discovered that disabling the sessionlog brings the > import times to normal. > > Digging further, I found that most of the CPU time is spent in > syncprov_add_slog, and particularly this line: > > > /* Keep the list in csn order. */ > ... > for ( sep = &sl->sl_head; *sep; sep = &(*sep)->se_next ) { >>>>>>>> if ( ber_bvcmp( &se->se_csn, &(*sep)->se_csn ) < 0 ) { > se->se_next = *sep; > *sep = se; > break; > } > } > > The sessionlog appears to be growing endlessly: > > (gdb) print sl->sl_num > $1 = 681896 > (gdb) print sl->sl_size > $3 = 100 > > Digging around the source, it seems that the session log is supposed to be > cleaned up at the end of syncprov_add_slog, if it's not playing: > > if (!sl->sl_playing) { > while ( sl->sl_num > sl->sl_size ) { > ... > > However: > > (gdb) print sl->sl_playing > $4 = 543388517 > > This looked like an uninitialized value to me, and indeed, sp_cf_gen > doesn't seem to initialize this field: > > if ( !sl ) { > sl = ch_malloc( sizeof( sessionlog )); > sl->sl_mincsn = NULL; > sl->sl_sids = NULL; > sl->sl_num = 0; > sl->sl_numcsns = 0; > sl->sl_head = sl->sl_tail = NULL; > ldap_pvt_thread_mutex_init( &sl->sl_mutex ); > si->si_logs = sl; > } > > > I tried the following patch: > > --- openldap-2.4.48.orig/servers/slapd/overlays/syncprov.c 2019-07-23 > 16:46:22.000000000 +0200 > +++ openldap-2.4.48/servers/slapd/overlays/syncprov.c 2020-01-08 > 11:33:16.770110282 +0100 > @@ -3082,6 +3082,7 @@ > sl = ch_malloc( sizeof( sessionlog )); > sl->sl_mincsn = NULL; > sl->sl_sids = NULL; > + sl->sl_playing = 0; > sl->sl_num = 0; > sl->sl_numcsns = 0; > sl->sl_head = sl->sl_tail = NULL; > > > And it got rid of the issue. My guess is that most of the time, sl_playing > happens to be 0, but on Amazon Linux, for some reason (patched glibc?), the > sessionlog lands in the wrong chunk of uninitilized memory > > Additional info: > > # rpm -q glibc > glibc-2.26-32.amzn2.0.2.x86_64 > > # uname -a > Linux 4.14.152-127.182.amzn2.x86_64 #1 SMP Thu Nov 14 17:32:43 UTC 2019 x86_64 > x86_64 x86_64 GNU/Linux > > # /usr/local/openldap/libexec/slapd -V > @(#) $OpenLDAP: slapd 2.4.48 (Jan 8 2020 11:39:38) $ > > (LTB build) > > > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9147) syncrepl connexion leak when provider uses an expired certificate
by maxime.besson＠worteks.com 08 Jan '20

08 Jan '20

Full_Name: Maxime Besson Version: 2.4.48 OS: Debian Buster / CentOS7 URL: https://cloud.worteks.com/index.php/s/9CE6ALLaAfrxZW4/download Submission from: (NULL) (92.184.104.113) I have two OpenLDAP (2.4.48, Ubuntu) servers running with syncrepl in mirrormode. One of my server's X509 certificates has recently expired, and I noticed that while it was expired, the other node's connection count kept climbing until it a "Max open files" condition. It seems that when a Syncrepl consumer encounters a certificate error, the outgoing LDAP connection to the provider is never closed. Attached to this bug you will find a test case to reproduce this behavior # runs a provider with a bogus certificate # and a consumer with retry=3+ sh test.sh ... TLS certificate verification: Error, self signed certificate TLS trace: SSL3 alert write:fatal:unknown CA TLS trace: SSL_connect:error in error TLS: can't connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed (self signed certificate). 5e160966 slap_client_connect: URI=ldaps://127.0.1.1:6636/ DN="cn=manager,dc=example,dc=com" ldap_sasl_bind_s failed (-1) 5e160966 do_syncrepl: rid=001 rc -1 retrying ... and so on, every 3 seconds While the consumer retries, running lsof on its PID will reveal the connection leak: 42u TCP localhost:45964->localhost.lan:6636 (CLOSE_WAIT) 43u TCP localhost:45966->localhost.lan:6636 (CLOSE_WAIT) 44u TCP localhost:45968->localhost.lan:6636 (CLOSE_WAIT) 45u TCP localhost:45970->localhost.lan:6636 (CLOSE_WAIT) 46u TCP localhost:45972->localhost.lan:6636 (CLOSE_WAIT) 47u TCP localhost:45974->localhost.lan:6636 (CLOSE_WAIT) 48u TCP localhost:45976->localhost.lan:6636 (CLOSE_WAIT) 49u TCP localhost:45978->localhost.lan:6636 (CLOSE_WAIT) 50u TCP localhost:45980->localhost.lan:6636 (CLOSE_WAIT) 51u TCP localhost:45982->localhost.lan:6636 (CLOSE_WAIT) 52u TCP localhost:45984->localhost.lan:6636 (CLOSE_WAIT) 53u TCP localhost:45986->localhost.lan:6636 (CLOSE_WAIT) 54u TCP localhost:45988->localhost.lan:6636 (CLOSE_WAIT) 55u TCP localhost:45990->localhost.lan:6636 (CLOSE_WAIT) 56u TCP localhost:45992->localhost.lan:6636 (CLOSE_WAIT) 57u TCP localhost:45994->localhost.lan:6636 (CLOSE_WAIT) 58u TCP localhost:45996->localhost.lan:6636 (CLOSE_WAIT) 59u TCP localhost:45998->localhost.lan:6636 (CLOSE_WAIT) 60u TCP localhost:46000->localhost.lan:6636 (CLOSE_WAIT) 61u TCP localhost:46002->localhost.lan:6636 (CLOSE_WAIT) 62u TCP localhost:46004->localhost.lan:6636 (CLOSE_WAIT) 63u TCP localhost:46006->localhost.lan:6636 (CLOSE_WAIT) 64u TCP localhost:46008->localhost.lan:6636 (CLOSE_WAIT) Modifying the provider URL in slapd.2.conf with a wrong port causes the syncrepl consumer to fail and retry just as much, but without connection piling up in CLOSE_WAIT state. This is not a very critical issue because it only affects servers who are in already degraded condition (broken replication, invalid certificate on the provider) but I thought it still was worth reporting. I was able to reproduce this issue on the git master branch, on released 2.4.48, on Centos7 and on Debian Buster. OpenSSL version on the debian system: 1.1.1d-0+deb10u2

1 0

(ITS#9146) incomplete initialization of sessionlog structure may cause session log to grow indefinitely
by maxime.besson＠worteks.com 08 Jan '20

08 Jan '20

Full_Name: Maxime Besson Version: 2.4.48 OS: Amazon Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (92.184.104.114) During a cloud provider migration, I have migrated an OpenLDAP instance using MDB + Syncrepl (in mirror mode, with session log) from Ubuntu to Amazon Linux. When starting a server with no data, syncrepl kicks in and starts replicating the other server's database (12G, 7M entries). In Ubuntu this phase takes about 25 minutes (with dbnosync). In Amazon Linux, with the exact same cn=config and daemon options, it took 10 days on first try, at 100% CPU consumption the whole time. I observed that the start of the import is pretty fast (data.mdb grows by 10M/s) but import speed slows down rapidly to a crawl. After some investigation, I discovered that disabling the sessionlog brings the import times to normal. Digging further, I found that most of the CPU time is spent in syncprov_add_slog, and particularly this line: /* Keep the list in csn order. */ ... for ( sep = &sl->sl_head; *sep; sep = &(*sep)->se_next ) { >>>>>>> if ( ber_bvcmp( &se->se_csn, &(*sep)->se_csn ) < 0 ) { se->se_next = *sep; *sep = se; break; } } The sessionlog appears to be growing endlessly: (gdb) print sl->sl_num $1 = 681896 (gdb) print sl->sl_size $3 = 100 Digging around the source, it seems that the session log is supposed to be cleaned up at the end of syncprov_add_slog, if it's not playing: if (!sl->sl_playing) { while ( sl->sl_num > sl->sl_size ) { ... However: (gdb) print sl->sl_playing $4 = 543388517 This looked like an uninitialized value to me, and indeed, sp_cf_gen doesn't seem to initialize this field: if ( !sl ) { sl = ch_malloc( sizeof( sessionlog )); sl->sl_mincsn = NULL; sl->sl_sids = NULL; sl->sl_num = 0; sl->sl_numcsns = 0; sl->sl_head = sl->sl_tail = NULL; ldap_pvt_thread_mutex_init( &sl->sl_mutex ); si->si_logs = sl; } I tried the following patch: --- openldap-2.4.48.orig/servers/slapd/overlays/syncprov.c 2019-07-23 16:46:22.000000000 +0200 +++ openldap-2.4.48/servers/slapd/overlays/syncprov.c 2020-01-08 11:33:16.770110282 +0100 @@ -3082,6 +3082,7 @@ sl = ch_malloc( sizeof( sessionlog )); sl->sl_mincsn = NULL; sl->sl_sids = NULL; + sl->sl_playing = 0; sl->sl_num = 0; sl->sl_numcsns = 0; sl->sl_head = sl->sl_tail = NULL; And it got rid of the issue. My guess is that most of the time, sl_playing happens to be 0, but on Amazon Linux, for some reason (patched glibc?), the sessionlog lands in the wrong chunk of uninitilized memory Additional info: # rpm -q glibc glibc-2.26-32.amzn2.0.2.x86_64 # uname -a Linux 4.14.152-127.182.amzn2.x86_64 #1 SMP Thu Nov 14 17:32:43 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux # /usr/local/openldap/libexec/slapd -V @(#) $OpenLDAP: slapd 2.4.48 (Jan 8 2020 11:39:38) $ (LTB build)

1 0

Re: (ITS#6104) race condition with cancel operation
by ondra＠mistotebe.net 03 Jan '20

03 Jan '20

This might be related or the same issue. With ITS#9124 and ITS#9145 fixed, if you run: ldapsearch -E '!sync=rp' -E '!cancel' syncprov has freed the operation and cancel_extop() still tries to touch the same Operation pointer without checking it's still in the connection list. At least valgrind isn't happy. -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

1 0

(ITS#9145) ldap tools try to attach configured controls to cancel/abandon operations
by ondra＠openldap.org 03 Jan '20

03 Jan '20

Full_Name: Ondrej Kuznik Version: re24/master OS: URL: Submission from: (NULL) (146.90.154.72) Take this command for example: ldapsearch -E '!sync=rp' -E '!cancel' Here, tool_check_abandon() calls ldap_cancel_s() with sctrls == NULL, but ld->ld_sctrls contains the syncrepl control as a default so it gets used instead, same for abandon. This means the option is broken. tool_check_abandon() should pass an zero-length LDAPControl list instead. Patch coming shortly.

1 0

Re: (ITS#9144) Remove ICU detection
by ryan＠nardis.ca 19 Dec '19

19 Dec '19

Patch available: https://github.com/openldap/openldap/compare/master...rtandy:its9144.patch

1 0

Re: (ITS#9143) Segfault in libcrypto.so.1.1
by quanah＠symas.com 19 Dec '19

19 Dec '19

--On Thursday, December 19, 2019 9:46 PM +0000 quanah(a)symas.com wrote: > > > --On Saturday, December 14, 2019 10:54 PM +0000 s.peillon(a)caenlamer.fr > wrote: > >> Can you tell me what to modify to correct this error? > > Hello, > > Libcrypto is not provided by the OpenLDAP project. I would also note > that Debian & Ubuntu use GnuTLS by default rather than OpenSSL. I would > advise contacting the Debian/Ubuntu project and their GnuTLS package > maintainers specifically. If you are building your own OpenLDAP linked to your own OpenSSL build, you'll need to dig deeper into the issues, as you're a bit outside of what Debian/Ubuntu do with OpenLDAP. I would note that at least for Ubuntu 18, there is already a backport provided of the 2.4.48 release that does not appear to exhibit the issues you describe. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#9143) Segfault in libcrypto.so.1.1
by quanah＠symas.com 19 Dec '19

19 Dec '19

--On Saturday, December 14, 2019 10:54 PM +0000 s.peillon(a)caenlamer.fr wrote: > Can you tell me what to modify to correct this error? Hello, Libcrypto is not provided by the OpenLDAP project. I would also note that Debian & Ubuntu use GnuTLS by default rather than OpenSSL. I would advise contacting the Debian/Ubuntu project and their GnuTLS package maintainers specifically. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#9144) Remove ICU detection
by quanah＠symas.com 19 Dec '19

19 Dec '19

--On Thursday, December 19, 2019 9:36 PM +0000 quanah(a)openldap.org wrote: > Full_Name: Quanah Gibson-Mount > Version: 2.4.48 > OS: N/A > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (47.208.143.26) > > > In 2005, ICU library detection was added to the build process, however > nothing was ever added to use it. This can result in an unwanted > dependency when building on system where the library is present, causing > issues later down the line. See also commits a956f7592 and 1cf5838e0 I found this to be a problem when slapd is compiled on a system where the ICU libs are present and then the build is later deployed on a system where it doesn't exist, causing slapd to fail. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs