February 2020 - openldap-bugs

Re: (ITS#9017) Improving performance of commit sync in Windows

by hyc＠symas.com

Howard Chu wrote: > There should be an off64_t type instead. > But it looks to me like only the env->me_size field could ever overflow, > and using size_t for that should be fine. The other uses are for the meta page, which is always either offset 0 or > offset 4096 (with 4KB pagesize), and the lockfile, which is always smaller than 2GB. Ignore that, I was looking at the 0.9 branch when I wrote that. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

3 years, 9 months

1
0
0 / 0

Re: (ITS#9017) Improving performance of commit sync in Windows

by hyc＠symas.com

kriszyp(a)gmail.com wrote: > However, in trying to test the mdb.master branch code with our application/server (we were previously using the 0.9 branch), there was a regression causing it > to crash pretty much any time we attempted to write to a db that was larger than 2GB. After some considerable testing and investigation (which is why it took a > while to get this finished), it seems the cause is the use of the off_t type for file pointer/positions, which appears to be a 32-bit signed integer compiling > on Windows. This overflows for dbs over 2GB and causes references that crash the process. Replacing all the off_t types with size_t (unsigned 64-bit), fixed > this issue. I am not sure if this is the right replacement type. I think this could also be addressed with compiler option for changing the file offset type > size, but that seems like a hazardous hoop to jump through. But the third commit, replacing the off_t with size_t definitely fixed the problem in our application. There should be an off64_t type instead. But it looks to me like only the env->me_size field could ever overflow, and using size_t for that should be fine. The other uses are for the meta page, which is always either offset 0 or offset 4096 (with 4KB pagesize), and the lockfile, which is always smaller than 2GB. > > https://github.com/LMDB/lmdb/commit/45cf4b6ede38565cfab1c40d0d77961a0cb22... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

3 years, 9 months

1
0
0 / 0

RE: RE: Re: (ITS#9017) Improving performance of commit sync in Windows

by kriszyp＠gmail.com

<html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta ht= tp-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta name= =3DGenerator content=3D"Microsoft Word 15 (filtered medium)"><style></style></head><body lang=3DEN-CA link=3Dblue vlink=3D"#954F72"><div cla= ss=3DWordSection1>Here is a pull request for these win= dows upgrades (I know it won=E2=80=99t be applied directly in GitHub, but s= eems a convenient host for the patches):<a href=3D= "https://github.com/LMDB/lmdb/pull/18">https:= //github.com/LMDB/lmdb/pull/18</a><o:p></o:p><o:p> </o:p>The first commit is basically t= he changes mentioned in this thread for improving sync performance by using= write-through writes instead FlushFileBuffers/FlushViewOfFile, reapplied t= o the mdb.master branch, with conflicts resolved.<= a href=3D"https://github.com/LMDB/lmdb/commit/204d1710d7a34eff142f2654d7678= 182e1b9e054.patch">https://github.com/LMDB/lm= db/commit/204d1710d7a34eff142f2654d7678182e1b9e054.patch</a><o:p> </o:p>The second commi= t adds support for an option for opening map files by providing the mapsize= as the file size (MaximumSize) to NtCreateSection (which also improves per= formance and prevents freezing caused by file growth). I implemented this a= s a compiler preprocessor define, as requested. However, I would note that = I would still prefer this to be a runtime option. When bundling LMDB in a n= ode package, there is no way to parameterize the compiler options (that I k= now of) for distribution, and it would be nice to have this an option for d= ifferent users of the node package.<a href=3D"http= s://github.com/LMDB/lmdb/commit/6f94bb1da1812c1ddb68e33448c9cd381674b02d.pa= tch">https://github.com/LMDB/lmdb/commit/6f94= bb1da1812c1ddb68e33448c9cd381674b02d.patch</a><o:p> </o:p>However, in trying to test the= mdb.master branch code with our application/server (we were previously usi= ng the 0.9 branch), there was a regression causing it to crash pretty much = any time we attempted to write to a db that was larger than 2GB. After some= considerable testing and investigation (which is why it took a while to ge= t this finished), it seems the cause is the use of the off_t type for file = pointer/positions, which appears to be a 32-bit signed integer compiling on= Windows. This overflows for dbs over 2GB and causes references that crash = the process. Replacing all the off_t types with size_t (unsigned 64-bit), f= ixed this issue. I am not sure if this is the right replacement type. I thi= nk this could also be addressed with compiler option for changing the file = offset type size, but that seems like a hazardous hoop to jump through. But= the third commit, replacing the off_t with size_t definitely fixed the pro= blem in our application.<a href=3D"https://github.= com/LMDB/lmdb/commit/45cf4b6ede38565cfab1c40d0d77961a0cb22b9e.patch">https://github.com/LMDB/lmdb/commit/45cf4b6ede38565= cfab1c40d0d77961a0cb22b9e.patch</a><o:p>&nb= sp;</o:p>With these three commits we are seeing so= lid stability and performance on Windows with the master branch. Thanks for= considering this!<o:p> </o:p>Thanks, Kris<o:p> </o:p><div= style=3D'mso-element:para-border-div;border:none;border-top:solid #E1E1E1 = 1.0pt;padding:3.0pt 0cm 0cm 0cm'>From: <a href=3D"mailto:kriszyp@gmail.com">kriszyp@gmai= l.com</a> Sent: February 18, 2020 1:06 PM To: <a href= =3D"mailto:hyc@symas.com">Howard Chu</a> Cc: <a href=3D"mailto:op= enldap-its(a)openldap.org">openldap-its(a)OpenLDAP.org</a> Subject: R= E: Re: (ITS#9017) Improving performance of commit sync in Windows</div>= <o:p> </o:p>Looking into= the APIs, I would guess that CreateFileMapping=E2=80=99s dwMaximumSizeHigh= and dwMaximumSizeLow arguments are mapped to NtCreateSection=E2=80=99s Max= imumSize argument (4th argument). In LMDB=E2=80=99s 0.9 CreateFi= leMapping call, the size is provided (since it is required), whereas NtCrea= teSection has MaximumSize as optional, and in LMDB (mdb.master), it is not = provided. Passing the size in (with MaximumSize argument) to NtCreateSectio= n seems to result in the same behavior as CreateFileMapping of preallocatin= g a file size equivalent to the max/mapping size, rather than dynamically g= rowing.<o:p></o:p><o:p> </o:p>It is challenging to readily test for the performance issues; it = took a fair bit of effort and load testing to trigger the long-duration per= formance/freezing issues we had seen before on Azure. However, in some more= isolated tests, providing a MaximumSize to NtCreateSection does seem to pe= rform better than leaving it unspecified (specifically the system CPU usage= seems to about be about twice as much with dynamic growth than fixed file = size), and the performance seems about the same as CreateFileMapping. And I= think we both had suspected that it was likely due to the overhead or inef= ficiencies in how Windows was dynamically growing the (size of the) mapped = file, and having Windows used a predefined fixed file size, whether through= NtCreateSection or CreateFileMapping, probably would prevent these slowdow= n issues.<o:p></o:p><o:p> </o:p>But, whether we use NtCreateSection or CreateFileMapping, we a= re still faced with the same trade-off of whether to use dynamic Windows-co= ntrolled file growth (convenient since the map size can be set arbitrarily = large), or using pre-specified file size (to avoid the slowing/freezing tha= t seems to sometimes result from dynamic Windows growth). So I assume it wo= uld still be reasonable to offer a compile time option, so users could opt-= in to using a fixed file size (provide the MaximumSize to NtCreateSection)?= Fortunately this would be a trivial addition, just a few lines of code to = use the preprocessor switch to supply this argument.<o:p></o:p><o:p> </o:p><a href=3D"http://un= documented.ntinternals.net/index.html?page=3DUserMode%2FUndocumented%20Func= tions%2FNT%20Objects%2FSection%2FNtMapViewOfSection.html">http://undocument= ed.ntinternals.net/index.html?page=3DUserMode%2FUndocumented%20Functions%2F= NT%20Objects%2FSection%2FNtMapViewOfSection.html</a><o:p></o:p><o:p> </o:p>Thanks, Kris<o:p>= </o:p><o:p> </o:p><div style=3D'border:no= ne;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'>From: <a href=3D"mailto:hyc@symas.com">Howard Chu</a> S= ent: February 18, 2020 8:15 AM To: <a href=3D"mailto:kriszyp@= gmail.com">Kris Zyp</a> Cc: <a href=3D"mailto:openldap-its@openld= ap.org">openldap-its(a)OpenLDAP.org</a> Subject: Re: (ITS#9017) Imp= roving performance of commit sync in Windows<o:p></o:p></div><o:p> </o:p>Kris Zyp wrote:<o:p>= </o:p>> Yes, I will work on upgrading this patc= h to 1.0.<o:p></o:p>> <o:p></o:p>> However, I believe in order to realize optimal Windows pe= rformance without regression/slow-downs, I will also need to disable the us= e of NTDLL (and use direct<o:p></o:p>> CreateFi= leMapping memory maps instead), as mentioned  http://www.openldap.org/= lists/openldap-bugs/201902/msg00009.html. You had mentioned in that thread = that<o:p></o:p>> the NTDLL code wasn't destined= for release, is that still the case (it is still in the mdb.master branch)= ? If you are intending to release (1.0) with the<o:p></o:p>> NTDLL-based memory maps (which is understandable, can certain= ly be an easier path for Windows users getting started), I will also need t= o add the compiler<o:p></o:p>> option to disabl= e it. Anyway, I will work on creating a PR with both these patches.<o:p></o= :p><o:p> </o:p>Might= be interesting to investigate why the behavior with NTDLL is so much slowe= r on Azure. Since<o:p></o:p>the Win32 APIs are imp= lemented on top of the NT APIs, you would expect them to perform identicall= y.<o:p></o:p>Perhaps there's some additional flag = that the Win32 API is passing to NT that is needed on Azure.<o:p></o:p>= <o:p> </o:p>> <o:p></= o:p>> Thanks,<o:p></o:p>> Kris<o:p></o:p>> <o:p></o:p>> On Mon, Feb 17, 2020 at 8:14 AM Howard Chu <hyc(a)symas.= com <mailto:hyc@symas.com>> wrote:<o:p></o:p>> <o:p></o:p>>     K= ris Zyp wrote:<o:p></o:p>>   &nb= sp; > Sorry to keep pestering, but just pinging about this patch again, = as I still think this fix could benefit windows users. And at this point, I= think I can<o:p></o:p>>   &nbsp= ; say we<o:p></o:p>>     &g= t; have tested it pretty well, running on our servers for almost a year :).= <o:p></o:p>> <o:p></o:p>>     Looks like this patch is against the 0.9 rel= ease branch. I hit a bunch of conflicts<o:p></o:p>= >     trying to apply it to mdb.master. We'll be sto= pping work on 0.9 soon, and getting<o:p></o:p>>=      LMDB 1.0 out the door finally, so can you please v= erify that your changes will work<o:p></o:p>>&n= bsp;    on mdb.master as well?<o:p></o:p>> <o:p></o:p>>    = > Thanks,<o:p></o:p>>   &nbs= p; > Kris<o:p></o:p>>   &nbsp= ; ><o:p></o:p>>     >= On Wed, Sep 18, 2019 at 12:56 PM Kris Zyp <kriszyp(a)gmail.com <mailto= :kriszyp@gmail.com> <mailto:kriszyp@gmail.com <mailto:kriszyp@gmai= l.com>>> wrote:<o:p></o:p>> &nbsp= ;   ><o:p></o:p>>  &nbsp= ;  >     Checking on this again, is this still a pos= sibility for merging into LMDB? This fix is still working great (improved p= erformance) on our systems.<o:p></o:p>> &n= bsp;   >     Thanks,<o:p></o:p>>     >     Kris<o:p></o:= p>>     ><o:p></o:p>= >     >     O= n Mon, Jun 17, 2019 at 1:04 PM Kris Zyp <kriszyp(a)gmail.com <mailto:kr= iszyp(a)gmail.com> <mailto:kriszyp@gmail.com <mailto:kriszyp@gmail.c= om>>> wrote:<o:p></o:p>>  &n= bsp;  ><o:p></o:p>>   &n= bsp; >         Is this still being considered/r= eviewed? Let me know if there are any other changes you would like me to ma= ke. This patch has continued to yield<o:p></o:p>&g= t;     >         significan= t and reliable performance improvements for us, and seems like it would be = nice for this to be available for other Windows users.<o:p></o:p>>     ><o:p></o:p>>     >        &nbsp= ;On Fri, May 3, 2019 at 3:52 PM Kris Zyp <kriszyp(a)gmail.com <mailto:k= riszyp(a)gmail.com> <mailto:kriszyp@gmail.com <mailto:kriszyp@gmail.= com>>> wrote:<o:p></o:p>>  &= nbsp;  ><o:p></o:p>>   &= nbsp; >             For the sake of p= utting this in the email thread (other code discussion in GitHub), here is = the latest squashed commit of the proposed patch (with<o:p></o:p>>     >       =      the on-demand, retained overlapped array to reduce re-= malloc and opening event handles):<o:p></o:p>>&= nbsp;    >            &nbsp= ;https://github.com/kriszyp/node-lmdb/commit/726a9156662c703bf3d453aab75ee2= 22072b990f<o:p></o:p>>     = ><o:p></o:p>> <o:p></o:p>> <o:p></o:p>>    = -- <o:p></o:p>>      = -- Howard Chu<o:p></o:p>>   &nb= sp;   CTO, Symas Corp.           http://= www.symas.com<o:p></o:p>>   &nbs= p;   Director, Highland Sun     http://highlandsun.com/= hyc/<o:p></o:p>>      = Chief Architect, OpenLDAP  http://www.openldap.org/project/<o:p></o:p= >> <o:p></o:p><o:p>&nb= sp;</o:p><o:p> </o:p>-- <o:p></o:p>  -- Howard Chu<o:p></o:p= >  CTO, Symas Corp.    &n= bsp;      http://www.symas.com<o:p></o:p>  Director, Highland Sun     http:= //highlandsun.com/hyc/<o:p></o:p>  Chief Arch= itect, OpenLDAP  http://www.openldap.org/project/<o:p></o:p><o:p> </o:p><o:p> </o:p><= /p></div></body></html>=

3 years, 9 months

1
0
0 / 0

(ITS#9176) libldap support for TLSv1.3 Encrypted SNI

by ryan＠openldap.org

Full_Name: Ryan Tandy Version: master OS: Debian URL: Submission from: (NULL) (70.66.128.207) Submitted by: ryan ldap.google.com requires clients to support SNI and sends back an intentionally invalid certificate if it is not used. Affects libldap with both OpenSSL and GnuTLS. (Might want a separate ITS for each fix.) With OpenSSL: $ ./clients/tools/ldapsearch -H ldaps://ldap.google.com -x -d1 ldap_url_parse_ext(ldaps://ldap.google.com) ldap_create ldap_url_parse_ext(ldaps://ldap.google.com:636/??base) ldap_sasl_bind ldap_send_initial_request ldap_new_connection 1 1 0 ldap_int_open_connection ldap_connect_to_host: TCP ldap.google.com:636 ldap_new_socket: 3 ldap_prepare_socket: 3 ldap_connect_to_host: Trying 216.239.32.58:636 ldap_pvt_connect: fd: 3 tm: -1 async: 0 attempting to connect: connect success TLS trace: SSL_connect:before SSL initialization TLS trace: SSL_connect:SSLv3/TLS write client hello TLS trace: SSL_connect:SSLv3/TLS write client hello TLS trace: SSL_connect:SSLv3/TLS read server hello TLS trace: SSL_connect:TLSv1.3 read encrypted extensions TLS trace: SSL_connect:SSLv3/TLS read server certificate request TLS certificate verification: depth: 0, err: 18, subject: /OU=No SNI provided; please fix your client./CN=invalid2.invalid, issuer: /OU=No SNI provided; please fix your client./CN=invalid2.invalid TLS certificate verification: Error, self signed certificate TLS trace: SSL3 alert write:fatal:unknown CA TLS trace: SSL_connect:error in error TLS: can't connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed (self signed certificate). ldap_err2string ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1) With GnuTLS: $ ./clients/tools/ldapsearch -H ldaps://ldap.google.com -x -d1 ldap_url_parse_ext(ldaps://ldap.google.com) ldap_create ldap_url_parse_ext(ldaps://ldap.google.com:636/??base) ldap_sasl_bind ldap_send_initial_request ldap_new_connection 1 1 0 ldap_int_open_connection ldap_connect_to_host: TCP ldap.google.com:636 ldap_new_socket: 3 ldap_prepare_socket: 3 ldap_connect_to_host: Trying 216.239.32.58:636 ldap_pvt_connect: fd: 3 tm: -1 async: 0 attempting to connect: connect success TLS: peer cert untrusted or revoked (0x42) TLS: can't connect: (unknown error code). ldap_err2string ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1) When the client does not support SNI, this server sends back a dummy certificate: OU = "No SNI provided; please fix your client.", CN = invalid2.invalid which obviously does not validate. The same behaviour can be seen with "openssl s_client -noservername" or "gnutls-cli --disable-sni". For GnuTLS, we have to call gnutls_server_name_set(3) to enable SNI. The documentation notes that it should only be called with a DNS name, not with an IP address. I guess there's a corresponding call for OpenSSL. With GnuTLS, users can work around it for the short term by disabling TLSv1.3 via priority string, i.e.: LDAPTLS_CIPHER_SUITE='NORMAL:!VERS-TLS1.3' ./clients/tools/ldapsearch -H ldaps://ldap.google.com -x Not sure if the same can be done with OpenSSL.

3 years, 9 months

1
0
0 / 0

(ITS#9175) ldapsearch segfault

by lhaley＠meditech.com

Full_Name: Lexi Haley Version: 02/20/2020 trunk commit 299fb490a27e7b0e5a60464e33f5ea04d00f0f7c OS: CYGWIN on Windows URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (208.206.3.188) Using clients/tools/ldapsearch to interact with my company's active directory, and I receive a segfault, in libraries/liblbver/decode.c . the 'gist' of the command I executed was: ldapsearch.exe -h companyADserver -b cn=users,dc=company,dc=com -s sub '(cn=z*)' cn the gdb backtrace is (omitting all the (arg=val) items for brevity: #0 ber_get_stringbvl (...) at decode.c:445 #1 0x0000000100435781 in ber_scanf (...) at decode.c:820 #2 0x000000010041a605 in ldap_get_attribute_ber (...) at getattr.c:149 #3 0x0000000100401d93 in print_entry (...) at ldapsearch.c:1758 #5 0x000000010043bfd7 in main (...) at ldapsearch.c:1510 and digging around - here is what I see going on. in decode.c, during the ber_get_stringbvl function, the passed in b (aka cookie from caller), has the following values: {choice = BvOff, option = 0, siz = 1, off = 4294967296, result = 0x800055410} notice that off is 0x100000000 - a likely ridiculous (and thus a possible MAGIC) value. examining in ber_scanf, looking at the memory for the va_args - i.e. examining ap - I see (sure enough) 0x100000000... As a hacky temporary workaround - I added to my ber_get_stringbvl , before even pulling the tag: } res; + if (b->off == 0x100000000) + return LBER_DEFAULT; tag = ber_skip_tag( ber, &bv.bv_len ); I did try rewinding my source back to OPENLDAP_REL_ENG_2_4_40 (2014) and then to OPENLDAP_REL_ENG_2_4_30 (2012) - and both compilations still yielded the segfault error I described above. I'm afraid I do not know much about ldap or active directory - so I easily might be omitting something relevant, and/or completely missing the mark in different ways. Feel free to ask me for more information, and/or suggest what I might try out. Thanks, Lexi

3 years, 9 months

1
0
0 / 0

Re: (ITS#8639) Remove support for LANMAN

by fumiyas＠osstech.co.jp

Hi, I've created a path for smbk5pwd to disable the insecure sambaLMPassword support by default. https://github.com/osstech-jp/openldap/commit/bba50bf6533d8f67dcbfc990b6b... https://github.com/osstech-jp/openldap/commit/bba50bf6533d8f67dcbfc990b6b... -- -- Name: SATOH Fumiyasu @ OSS Technology Corp. (fumiyas @ osstech co jp) -- Business Home: https://www.OSSTech.co.jp/ -- GitHub Home: https://GitHub.com/fumiyas/ -- PGP Fingerprint: BBE1 A1C9 525A 292E 6729 CDEC ADC2 9DCA 5E1C CBCA

3 years, 9 months

1
0
0 / 0

(ITS#9993) 1

by sample＠email.tst

Full_Name: tsSLAueP OS: 1 URL: ftp://ftp.openldap.org/incoming/ Version: 1 Submission from: (NULL) (178.238.229.54) 555

3 years, 9 months

1
0
0 / 0

(ITS#9991) 1

by sample＠email.tst

Full_Name: tsSLAueP OS: 1 URL: ftp://ftp.openldap.org/incoming/ Version: 1 Submission from: (NULL) (178.238.229.54) 555

3 years, 9 months

1
0
0 / 0

(ITS#9989) 1

by sample＠email.tst

Full_Name: tsSLAueP OS: 1 URL: ftp://ftp.openldap.org/incoming/ Version: 1 Submission from: (NULL) (178.238.229.54) 555

3 years, 9 months

1
0
0 / 0

(ITS#9988) 1

by sample＠email.tst

Full_Name: tsSLAueP OS: 1 URL: ftp://ftp.openldap.org/incoming/ Version: 1 Submission from: (NULL) (178.238.229.54) 555

3 years, 9 months

1
0
0 / 0

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs February 2020