Full_Name: Kristopher William Zyp
Version: LMDB 0.9.23
OS: Windows
URL: https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab933…
Submission from: (NULL) (71.199.6.148)
We have seen very poor performance on the sync of commits on large databases in
Windows. On databases with 2GB of data, in writemap mode, the sync of even small
commits is consistently well over 100ms (without writemap it is faster, but
still slow). It is expected that a sync should take some time while waiting for
disk confirmation of the writes, but more concerning is that these sync
operations (in writemap mode) are instead dominated by nearly 100% system CPU
utilization, so operations that requires sub-millisecond b-tree update
operations are then dominated by very large amounts of system CPU cycles during
the sync phase.
I think that the fundamental problem is that FlushViewOfFile seems to be an O(n)
operation where n is the size of the file (or map). I presume that Windows is
scanning the entire map/file for dirty pages to flush, I'm guessing because it
doesn't have an internal index of all the dirty pages for every file/map-view in
the OS disk cache. Therefore, the turns into an extremely expensive, CPU-bound
operation to find the dirty pages for (large file) and initiate their writes,
which, of course, is contrary to the whole goal of a scalable database system.
And FlushFileBuffers is also relatively slow as well. We have attempted to batch
as many operations into single transaction as possible, but this is still a very
large overhead.
The Windows docs for FlushFileBuffers itself warns about the inefficiencies of
this function (https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-flu…).
Which also points to the solution: it is much faster to write out the dirty
pages with WriteFile through a sync file handle (FILE_FLAG_WRITE_THROUGH).
The associated patch
(https://github.com/kriszyp/node-lmdb/commit/7ff525ae57684a163d32af74a0ab933…)
is my attempt at implementing this solution, for Windows. Fortunately, with the
design of LMDB, this is relatively straightforward. LMDB already supports
writing out dirty pages with WriteFile calls. I added a write-through handle for
sending these writes directly to disk. I then made that file-handle
overlapped/asynchronously, so all the writes for a commit could be started in
overlap mode, and (at least theoretically) transfer in parallel to the drive and
then used GetOverlappedResult to wait for the completion. So basically
mdb_page_flush becomes the sync. I extended the writing of dirty pages through
WriteFile to writemap mode as well (for writing meta too), so that WriteFile
with write-through can be used to flush the data without ever needing to call
FlushViewOfFile or FlushFileBuffers. I also implemented support for write
gathering in writemap mode where contiguous file positions infers contiguous
memory (by tracking the starting position with wdp and writing contiguous pages
in single operations). Sorting of the dirty list is maintained even in writemap
mode for this purpose.
The performance benefits of this patch, in my testing, are considerable. Writing
out/syncing transactions is typically over 5x faster in writemap mode, and 2x
faster in standard mode. And perhaps more importantly (especially in environment
with many threads/processes), the efficiency benefits are even larger,
particularly in writemap mode, where there can be a 50-100x reduction in the
system CPU usage by using this patch. This brings windows performance with
sync'ed transactions in LMDB back into the range of "lightning" performance :).
All of these changes in the associated patch should only affect Windows. I
actually had started with the approach of using a flag to indicate write-through
behavior (here https://github.com/kriszyp/node-lmdb/commit/435ca423d0e13936f2a5f0193e994f5…
if anyone wants to test it or play with it further). However, I didn't really
see any substantive improvements in unix. It is possible that maybe with the use
of dsync writes that are done asynchronously in parallel and wait for the
completion (aio_write/aio_suspend), there might be a performance benefit
opportunity, but I was less interested in this, and I assume you probably have
already thoroughly explored these options already. In the end, I think it makes
more sense to just make this the default behavior for Windows where the major
improvement is, and don't change the unix behavior. Also, I don't think you can
write to files that have an open writeable mapped region, so the implementation
would be limited on unix anyway.
Anyway, this is certainly a more involved patch, in a sophisticated code-base,
so I humbly present this for consideration. I have tested both the performance
and the sync safety of this code. Our application has a high enough intensity of
db interaction that it is actually pretty easy to reproduce LMDB data corruption
by powering off a VM with the app running, if sync-mode is not enabled. And with
my testing so far, sync-mode with this patch seems to preserve the rock-solid
crash-proof design of LMDB. But I'd be glad to make any changes or cleanup
needed, or if you want the patch to be submitted differently (it seems like
using patches from my node repo fork has worked in the past though).
--On Wednesday, June 26, 2013 6:20 AM +0000 a16474(a)gmail.com wrote:
> Full_Name: Amit Sinha
> Version: openldap-2.4.35
> OS: Linux 2.6.18-308.11.1.el5
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (72.163.217.105)
>
>
> When i compile and run the below code on linux 64 bit, it core dumps in
> ldap_start_tls_s().
Hello,
Thank you for the report! I apologize for the (very very long) delay in
response. This issue no longer seems to occur with current
OpenLDAP/OpenSSL releases (nor does it happen with OpenLDAP/GnuTLS).
I believe one of the many fixes to OpenLDAP's TLS code has since resolved
this issue. I tested with both OpenSSL 1.0.2k and 1.1.1b.
This ITS will be closed.
Warm regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
On Wed, Apr 24, 2019 at 01:30:18PM +0000, ondra(a)mistotebe.net wrote:
> On Tue, Apr 23, 2019 at 11:28:40PM +0000, quanah(a)symas.com wrote:
>> The consumer did not replicate the database as a part of the initial
>> search, so I don't think it qualifies as "up to date". I.e., it's primary
>> DB remains empty.
>
> Ah, that wasn't mentioned, my assumption was that the source DB was
> actually empty.
>
> This is coming from ITS#8281 fix in cd8ff37629012c1676ef79de164a159da9b2ae89.
I've pushed a change that makes sure we end up with a contexCSN,
generating one if necessary whenever we're configured to be a master
(olcMirrorMode is set or no olcSyncrepl exists) here:
https://github.com/mistotebe/openldap/tree/its9015
It's kind of a followup on this (reverting part of ITS#8281 above):
https://www.openldap.org/lists/openldap-devel/201904/msg00015.html
--
OndÅ™ej KuznÃk
Senior Software Engineer
Symas Corporation http://www.symas.com
Packaged, certified, and supported LDAP solutions powered by OpenLDAP
--_000_MWHPR08MB2400D7AE5E8EEC3D17192FACB53C0MWHPR08MB2400namp_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Thank you. we tried using another openldap image and that worked. so it see=
ms the problem is with the osixia docker image we were using to run openlda=
p. it is based on debian (which uses GnuTLS per your email) so tbh we are s=
urprised it would have such a bug in it. the image that worked for us is ba=
sed on alpine.
https://github.com/osixia/docker-light-baseimage/blob/stable/image/Dockerfi=
le
https://github.com/tiredofit/docker-openldap/blob/master/Dockerfile
but back to your comment, how can one isolate what TLS/SSL library OpenLDAP=
is linked to in the environment you're using?
[https://avatars0.githubusercontent.com/u/23528985?s=3D400&v=3D4]<https://g=ithub.com/tiredofit/docker-openldap/blob/master/Dockerfile>
docker-openldap/Dockerfile at master =B7 tiredofit/docker-openldap =B7 GitH=
ub<https://github.com/tiredofit/docker-openldap/blob/master/Dockerfile>
Docker OpenLDAP Container w/TLS & Replication Support S6 Overlay, and Zabbi=
x Monitoring based on Alpine - tiredofit/docker-openldap
github.com
[https://avatars0.githubusercontent.com/u/23528985?s=3D400&v=3D4]<https://g=ithub.com/tiredofit/docker-openldap/blob/master/Dockerfile>
docker-openldap/Dockerfile at master =B7 tiredofit/docker-openldap =B7 GitH=
ub<https://github.com/tiredofit/docker-openldap/blob/master/Dockerfile>
Docker OpenLDAP Container w/TLS & Replication Support S6 Overlay, and Zabbi=
x Monitoring based on Alpine - tiredofit/docker-openldap
github.com
________________________________
From: Quanah Gibson-Mount <quanah(a)symas.com>
Sent: Wednesday, April 24, 2019 1:06 PM
To: siddjain(a)live.com; openldap-its(a)OpenLDAP.org
Subject: Re: (ITS#9014) OpenLDAP modifies user provided TLS certificate bef=
ore sending it to client
--On Wednesday, April 24, 2019 6:43 PM +0000 hyc(a)symas.com wrote:
> siddjain(a)live.com wrote:
>> --_000_MWHPR08MB24000D77048AFCF7465C4397B53C0MWHPR08MB2400namp_
>> Content-Type: text/plain; charset=3D"iso-8859-1"
>> Content-Transfer-Encoding: quoted-printable
>>
>> could you send me output of running
>>
>> openssl version -a
>>
>> on your system? thanks
>
>> openssl version -a
> OpenSSL 1.1.1 11 Sep 2018
> built on: Tue Dec 4 13:15:09 2018 UTC
> platform: debian-amd64
I would also note that not all OpenLDAP builds use OpenSSL. For example,
OpenLDAP built on Debian/Ubuntu uses GnuTLS. OpenLDAP built on some
versions of RedHat 7 use MozNSS. Current RedHat 7 builds use OpenSSL but
have an odd MozNSS bridge for backwards compatibilty, and there may be all
sorts of odd bugs in that. Apple links OpenLDAP to its own custom SSL
libary.
So really your first step should be isolating what TLS/SSL library OpenLDAP
is linked to in the environment you're using.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<https://eur01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fwww.sym=
as.com&data=3D02%7C01%7C%7C349b90be6afe4991a54b08d6c8f068b4%7C84df9e7fe=
9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917332202918260&sdata=3DNifWEVt269=
tCTuar98XYUfNkaHWSFMffI3M4%2FJ7j8zI%3D&reserved=3D0>
--_000_MWHPR08MB2400D7AE5E8EEC3D17192FACB53C0MWHPR08MB2400namp_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style type=3D"text/css" style=3D"display:none;"> P {margin-top:0;margin-bo=
ttom:0;} </style>
</head>
<body dir=3D"ltr">
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0);">
Thank you. we tried using another openldap image and that worked. so it see=
ms the problem is with the osixia docker image we were using to run openlda=
p. it is based on debian (which <span style=3D"color: rgb(51, 51, 51);=
font-family: "Segoe UI", "Segoe UI Web (West European)"=
;, "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica =
Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 25=
5, 255); display: inline !important">uses
GnuTLS per your email</span>) so tbh we are surprised it would have such a=
bug in it. the image that worked for us is based on alpine. </div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0);">
<a href=3D"https://github.com/osixia/docker-light-baseimage/blob/stable/ima=
ge/Dockerfile">https://github.com/osixia/docker-light-baseimage/blob/stable=
/image/Dockerfile</a><br>
</div>
<div>
<div id=3D"appendonsend"></div>
<div style=3D"font-family:Calibri,Helvetica,sans-serif; font-size:12pt; col=
or:rgb(0,0,0)">
<a href=3D"https://github.com/tiredofit/docker-openldap/blob/master/Dockerf=
ile" id=3D"LPlnk147820">https://github.com/tiredofit/docker-openldap/blob/m=
aster/Dockerfile</a></div>
<div style=3D"font-family:Calibri,Helvetica,sans-serif; font-size:12pt; col=
or:rgb(0,0,0)">
but back to your comment, how can one <span style=3D"color: rgb(51, 51=
, 51); font-family: "Segoe UI", "Segoe UI Web (West European=
)", "Segoe UI", -apple-system, system-ui, Roboto, "Helv=
etica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(2=
55, 255, 255); display: inline !important">isolate
what TLS/SSL library OpenLDAP<span> </span></span><span style=3D"colo=
r: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (=
West European)", "Segoe UI", -apple-system, system-ui, Robot=
o, "Helvetica Neue", sans-serif; font-size: 14.6667px; background=
-color: rgb(255, 255, 255); display: inline !important">is
linked to in the environment you're using? </span><br style=3D"color:=
rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (We=
st European)", "Segoe UI", -apple-system, system-ui, Roboto,=
"Helvetica Neue", sans-serif; font-size: 14.6667px; background-c=
olor: rgb(255, 255, 255)">
<br>
<div id=3D"LPBorder_GTaHR0cHM6Ly9naXRodWIuY29tL3RpcmVkb2ZpdC9kb2NrZXItb3Blb=
mxkYXAvYmxvYi9tYXN0ZXIvRG9ja2VyZmlsZQ.." class=3D"LPBorder213343" contented=
itable=3D"false" style=3D"width: 100%; margin-top: 16px; margin-bottom: 16p=
x; position: relative; max-width: 800px; min-width: 424px;">
<table id=3D"LPContainer213343" role=3D"presentation" style=3D"padding: 12p=
x 36px 12px 12px; width: 100%; border-width: 1px; border-style: solid; bord=
er-color: rgb(200, 200, 200); border-radius: 2px;">
<tbody>
<tr valign=3D"top" style=3D"border-spacing: 0px;">
<td>
<div id=3D"LPImageContainer213343" style=3D"position: relative; margin-righ=
t: 12px; height: 160px; overflow: hidden;">
<a target=3D"_blank" id=3D"LPImageAnchor213343" href=3D"https://github.com/=
tiredofit/docker-openldap/blob/master/Dockerfile"><img id=3D"LPThumbnailIma=
geId213343" alt=3D"" height=3D"160" style=3D"display: block;" width=3D"160"=
src=3D"https://avatars0.githubusercontent.com/u/23528985?s=3D400&v=3D4=
"></a></div>
</td>
<td style=3D"width: 100%;">
<div id=3D"LPTitle213343" style=3D"font-size: 21px; font-weight: 300; margi=
n-right: 8px; font-family: wf_segoe-ui_light, "Segoe UI Light", &=
quot;Segoe WP Light", "Segoe UI", "Segoe WP", Taho=
ma, Arial, sans-serif; margin-bottom: 12px;">
<a target=3D"_blank" id=3D"LPUrlAnchor213343" href=3D"https://github.com/ti=
redofit/docker-openldap/blob/master/Dockerfile" style=3D"text-decoration: n=
one; color: var(--themePrimary);">docker-openldap/Dockerfile at master =B7 =
tiredofit/docker-openldap =B7 GitHub</a></div>
<div id=3D"LPDescription213343" style=3D"font-size: 14px; max-height: 100px=
; color: rgb(102, 102, 102); font-family: wf_segoe-ui_normal, "Segoe U=
I", "Segoe WP", Tahoma, Arial, sans-serif; margin-bottom: 12=
px; margin-right: 8px; overflow: hidden;">
Docker OpenLDAP Container w/TLS & Replication Support S6 Overlay, and Z=
abbix Monitoring based on Alpine - tiredofit/docker-openldap</div>
<div id=3D"LPMetadata213343" style=3D"font-size: 14px; font-weight: 400; co=
lor: rgb(166, 166, 166); font-family: wf_segoe-ui_normal, "Segoe UI&qu=
ot;, "Segoe WP", Tahoma, Arial, sans-serif;">
github.com</div>
</td>
</tr>
</tbody>
</table>
</div>
<br>
<div id=3D"LPBorder_GTaHR0cHM6Ly9naXRodWIuY29tL3RpcmVkb2ZpdC9kb2NrZXItb3Blb=
mxkYXAvYmxvYi9tYXN0ZXIvRG9ja2VyZmlsZQ.." class=3D"LPBorder356508" contented=
itable=3D"false" style=3D"width: 100%; margin-top: 16px; margin-bottom: 16p=
x; position: relative; max-width: 800px; min-width: 424px;">
<table id=3D"LPContainer356508" role=3D"presentation" style=3D"padding: 12p=
x 36px 12px 12px; width: 100%; border-width: 1px; border-style: solid; bord=
er-color: rgb(200, 200, 200); border-radius: 2px;">
<tbody>
<tr valign=3D"top" style=3D"border-spacing: 0px;">
<td>
<div id=3D"LPImageContainer356508" style=3D"position: relative; margin-righ=
t: 12px; height: 160px; overflow: hidden;">
<a target=3D"_blank" id=3D"LPImageAnchor356508" href=3D"https://github.com/=
tiredofit/docker-openldap/blob/master/Dockerfile"><img id=3D"LPThumbnailIma=
geId356508" alt=3D"" height=3D"160" style=3D"display: block;" width=3D"160"=
src=3D"https://avatars0.githubusercontent.com/u/23528985?s=3D400&v=3D4=
"></a></div>
</td>
<td style=3D"width: 100%;">
<div id=3D"LPTitle356508" style=3D"font-size: 21px; font-weight: 300; margi=
n-right: 8px; font-family: wf_segoe-ui_light, "Segoe UI Light", &=
quot;Segoe WP Light", "Segoe UI", "Segoe WP", Taho=
ma, Arial, sans-serif; margin-bottom: 12px;">
<a target=3D"_blank" id=3D"LPUrlAnchor356508" href=3D"https://github.com/ti=
redofit/docker-openldap/blob/master/Dockerfile" style=3D"text-decoration: n=
one; color: var(--themePrimary);">docker-openldap/Dockerfile at master =B7 =
tiredofit/docker-openldap =B7 GitHub</a></div>
<div id=3D"LPDescription356508" style=3D"font-size: 14px; max-height: 100px=
; color: rgb(102, 102, 102); font-family: wf_segoe-ui_normal, "Segoe U=
I", "Segoe WP", Tahoma, Arial, sans-serif; margin-bottom: 12=
px; margin-right: 8px; overflow: hidden;">
Docker OpenLDAP Container w/TLS & Replication Support S6 Overlay, and Z=
abbix Monitoring based on Alpine - tiredofit/docker-openldap</div>
<div id=3D"LPMetadata356508" style=3D"font-size: 14px; font-weight: 400; co=
lor: rgb(166, 166, 166); font-family: wf_segoe-ui_normal, "Segoe UI&qu=
ot;, "Segoe WP", Tahoma, Arial, sans-serif;">
github.com</div>
</td>
</tr>
</tbody>
</table>
</div>
<br>
</div>
<hr tabindex=3D"-1" style=3D"display:inline-block; width:98%">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" co=
lor=3D"#000000" style=3D"font-size:11pt"><b>From:</b> Quanah Gibson-Mount &=
lt;quanah(a)symas.com><br>
<b>Sent:</b> Wednesday, April 24, 2019 1:06 PM<br>
<b>To:</b> siddjain(a)live.com; openldap-its(a)OpenLDAP.org<br>
<b>Subject:</b> Re: (ITS#9014) OpenLDAP modifies user provided TLS certific=
ate before sending it to client</font>
<div> </div>
</div>
<div class=3D"BodyFragment"><font size=3D"2"><span style=3D"font-size:11pt"=
>
<div class=3D"PlainText">--On Wednesday, April 24, 2019 6:43 PM +0000 h=
yc(a)symas.com wrote:<br>
<br>
> siddjain(a)live.com wrote:<br>
>> --_000_MWHPR08MB24000D77048AFCF7465C4397B53C0MWHPR08MB2400namp_<br=
>
>> Content-Type: text/plain; charset=3D"iso-8859-1"<br>
>> Content-Transfer-Encoding: quoted-printable<br>
>><br>
>> could you send me output of running<br>
>><br>
>> openssl version -a<br>
>><br>
>> on your system? thanks<br>
><br>
>> openssl version -a<br>
> OpenSSL 1.1.1 11 Sep 2018<br>
> built on: Tue Dec 4 13:15:09 2018 UTC<br>
> platform: debian-amd64<br>
<br>
I would also note that not all OpenLDAP builds use OpenSSL. For examp=
le, <br>
OpenLDAP built on Debian/Ubuntu uses GnuTLS. OpenLDAP built on some <=
br>
versions of RedHat 7 use MozNSS. Current RedHat 7 builds use OpenSSL =
but <br>
have an odd MozNSS bridge for backwards compatibilty, and there may be all =
<br>
sorts of odd bugs in that. Apple links OpenLDAP to its own custom SSL=
<br>
libary.<br>
<br>
So really your first step should be isolating what TLS/SSL library OpenLDAP=
<br>
is linked to in the environment you're using.<br>
<br>
--Quanah<br>
<br>
<br>
<br>
--<br>
<br>
Quanah Gibson-Mount<br>
Product Architect<br>
Symas Corporation<br>
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:<br>
<<a href=3D"https://eur01.safelinks.protection.outlook.com/?url=3Dhttp%3=
A%2F%2Fwww.symas.com&amp;data=3D02%7C01%7C%7C349b90be6afe4991a54b08d6c8=
f068b4%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917332202918260&=
amp;sdata=3DNifWEVt269tCTuar98XYUfNkaHWSFMffI3M4%2FJ7j8zI%3D&amp;reserv=
ed=3D0">https://eur01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2F=www.symas.com&amp;data=3D02%7C01%7C%7C349b90be6afe4991a54b08d6c8f068b4%=
7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917332202918260&amp;sda=
ta=3DNifWEVt269tCTuar98XYUfNkaHWSFMffI3M4%2FJ7j8zI%3D&amp;reserved=3D0<=
/a>><br>
<br>
</div>
</span></font></div>
</div>
</body>
</html>
--_000_MWHPR08MB2400D7AE5E8EEC3D17192FACB53C0MWHPR08MB2400namp_--
--On Wednesday, April 24, 2019 6:43 PM +0000 hyc(a)symas.com wrote:
> siddjain(a)live.com wrote:
>> --_000_MWHPR08MB24000D77048AFCF7465C4397B53C0MWHPR08MB2400namp_
>> Content-Type: text/plain; charset="iso-8859-1"
>> Content-Transfer-Encoding: quoted-printable
>>
>> could you send me output of running
>>
>> openssl version -a
>>
>> on your system? thanks
>
>> openssl version -a
> OpenSSL 1.1.1 11 Sep 2018
> built on: Tue Dec 4 13:15:09 2018 UTC
> platform: debian-amd64
I would also note that not all OpenLDAP builds use OpenSSL. For example,
OpenLDAP built on Debian/Ubuntu uses GnuTLS. OpenLDAP built on some
versions of RedHat 7 use MozNSS. Current RedHat 7 builds use OpenSSL but
have an odd MozNSS bridge for backwards compatibilty, and there may be all
sorts of odd bugs in that. Apple links OpenLDAP to its own custom SSL
libary.
So really your first step should be isolating what TLS/SSL library OpenLDAP
is linked to in the environment you're using.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
siddjain(a)live.com wrote:
> --_000_MWHPR08MB24000D77048AFCF7465C4397B53C0MWHPR08MB2400namp_
> Content-Type: text/plain; charset="iso-8859-1"
> Content-Transfer-Encoding: quoted-printable
>
> could you send me output of running
>
> openssl version -a
>
> on your system? thanks
> openssl version -a
OpenSSL 1.1.1 11 Sep 2018
built on: Tue Dec 4 13:15:09 2018 UTC
platform: debian-amd64
options: bn(64,64) rc4(8x,int) des(int) blowfish(ptr)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-DovhWu/openssl-1.1.1=.
-fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2
-DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM
-DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-1.1"
Seeding source: os-specific
>
> ________________________________
> From: Howard Chu <hyc(a)symas.com>
> Sent: Wednesday, April 24, 2019 10:04 AM
> To: Siddharth Jain; openldap-its(a)OpenLDAP.org
> Subject: Re: (ITS#9014) OpenLDAP modifies user provided TLS certificate bef=
> ore sending it to client
>
> Siddharth Jain wrote:
>> Wow! Thanks for responding so fast. This could be a bug in docker-openlda=
> p then. we have repro'ed this in two different environments - mac and ubunt=
> u. Do you
>> have a recommendation for docker image for openldap?
>
> As I said before, OpenLDAP doesn't touch the certificate files, it merely t=
> ells the TLS
> library where they are. You must likely have a broken TLS library.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
--_000_MWHPR08MB24000D77048AFCF7465C4397B53C0MWHPR08MB2400namp_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
could you send me output of running
openssl version -a
on your system? thanks
________________________________
From: Howard Chu <hyc(a)symas.com>
Sent: Wednesday, April 24, 2019 10:04 AM
To: Siddharth Jain; openldap-its(a)OpenLDAP.org
Subject: Re: (ITS#9014) OpenLDAP modifies user provided TLS certificate bef=
ore sending it to client
Siddharth Jain wrote:
> Wow! Thanks for responding so fast. This could be a bug in docker-openlda=
p then. we have repro'ed this in two different environments - mac and ubunt=
u. Do you
> have a recommendation for docker image for openldap?
As I said before, OpenLDAP doesn't touch the certificate files, it merely t=
ells the TLS
library where they are. You must likely have a broken TLS library.
-------------------------------------------------------------------------=
---------------------------------------------------------------------------=
------------
> *From:* Howard Chu <hyc(a)symas.com>
> *Sent:* Wednesday, April 24, 2019 9:42 AM
> *To:* Siddharth Jain; openldap-its(a)OpenLDAP.org
> *Subject:* Re: (ITS#9014) OpenLDAP modifies user provided TLS certificate=
before sending it to client
>
> Siddharth Jain wrote:
>> we have documented complete steps to repro the bug here <https://nam01.s=afelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithub.com%2Fsiddjain%=
2Fopenldap-bug&data=3D02%7C01%7C%7Cdeffc420629649af454408d6c8d6f129%7C8=
4df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917222820865922&sdata=3Dsx=
jXXBtCMOjbK5AZCpLTObP%2BIlJRAxXUK7LpLzUDD%2FM%3D&reserved=3D0> with
> container logs.
>
> I see no error here.
>
> Using your cert/key files:
> There is no OpenLDAP bug here. Your server environment is broken.
--
-- Howard Chu
CTO, Symas Corp. https://nam01.safelinks.protection.outlook.com=
/?url=3Dhttp%3A%2F%2Fwww.symas.com&data=3D02%7C01%7C%7Cdeffc420629649af=
454408d6c8d6f129%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C6369172228208=
65922&sdata=3DX5JT6j5%2BQ2BAsKGfNslnC%2FkQj%2BcSU4GAdTqmqqc3lWo%3D&=
reserved=3D0
Director, Highland Sun https://nam01.safelinks.protection.outlook.com=
/?url=3Dhttp%3A%2F%2Fhighlandsun.com%2Fhyc%2F&data=3D02%7C01%7C%7Cdeffc=
420629649af454408d6c8d6f129%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C63=
6917222820865922&sdata=3DSHju26Gxu5dToV%2BuCYDxBMZQS5qJZvREcg9q0CEg2bo%=
3D&reserved=3D0
Chief Architect, OpenLDAP https://nam01.safelinks.protection.outlook.com=
/?url=3Dhttp%3A%2F%2Fwww.openldap.org%2Fproject%2F&data=3D02%7C01%7C%7C=
deffc420629649af454408d6c8d6f129%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0=
%7C636917222820865922&sdata=3DfJ7LIrWHv%2FG4CJGrx%2BClsFoldJfri%2Bdk7WN=
59Bt45jU%3D&reserved=3D0
--_000_MWHPR08MB24000D77048AFCF7465C4397B53C0MWHPR08MB2400namp_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style type=3D"text/css" style=3D"display:none;"> P {margin-top:0;margin-bo=
ttom:0;} </style>
</head>
<body dir=3D"ltr">
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0);">
could you send me output of running
<p style=3D"margin: 0px; font: 11px Menlo; background-color: rgb(255, 255, =
255); margin: 0px; background-color: rgb(255, 255, 255)">
<span style=3D"font-variant-ligatures: no-common-ligatures; font-variant-li=
gatures: no-common-ligatures">openssl version -a</span></p>
<p style=3D"margin: 0px; font: 11px Menlo; background-color: rgb(255, 255, =
255); margin: 0px; background-color: rgb(255, 255, 255)">
<span style=3D"font-variant-ligatures: no-common-ligatures; font-variant-li=
gatures: no-common-ligatures">on your system? thanks</span></p>
</div>
<div>
<div id=3D"appendonsend"></div>
<div style=3D"font-family:Calibri,Helvetica,sans-serif; font-size:12pt; col=
or:rgb(0,0,0)">
<br>
</div>
<hr tabindex=3D"-1" style=3D"display:inline-block; width:98%">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" co=
lor=3D"#000000" style=3D"font-size:11pt"><b>From:</b> Howard Chu <hyc@sy=
mas.com><br>
<b>Sent:</b> Wednesday, April 24, 2019 10:04 AM<br>
<b>To:</b> Siddharth Jain; openldap-its(a)OpenLDAP.org<br>
<b>Subject:</b> Re: (ITS#9014) OpenLDAP modifies user provided TLS certific=
ate before sending it to client</font>
<div> </div>
</div>
<div class=3D"BodyFragment"><font size=3D"2"><span style=3D"font-size:11pt"=
>
<div class=3D"PlainText">Siddharth Jain wrote:<br>
> Wow! Thanks for responding so fast. This could be a bug in docker-open=
ldap then. we have repro'ed this in two different environments - mac and ub=
untu. Do you<br>
> have a recommendation for docker image for openldap?<br>
<br>
As I said before, OpenLDAP doesn't touch the certificate files, it merely t=
ells the TLS<br>
library where they are. You must likely have a broken TLS library.<br>
--------------------------------------------------------------------=
---------------------------------------------------------------------------=
-----------------<br>
> *From:* Howard Chu <hyc(a)symas.com><br>
> *Sent:* Wednesday, April 24, 2019 9:42 AM<br>
> *To:* Siddharth Jain; openldap-its(a)OpenLDAP.org<br>
> *Subject:* Re: (ITS#9014) OpenLDAP modifies user provided TLS certific=
ate before sending it to client<br>
> <br>
> Siddharth Jain wrote:<br>
>> we have documented complete steps to repro the bug here <<=
a href=3D"https://nam01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F=
%2Fgithub.com%2Fsiddjain%2Fopenldap-bug&amp;data=3D02%7C01%7C%7Cdeffc42=
0629649af454408d6c8d6f129%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C6369=
17222820865922&amp;sdata=3DsxjXXBtCMOjbK5AZCpLTObP%2BIlJRAxXUK7LpLzUDD%=
2FM%3D&amp;reserved=3D0">https://nam01.safelinks.protection.outlook.com=
/?url=3Dhttps%3A%2F%2Fgithub.com%2Fsiddjain%2Fopenldap-bug&amp;data=3D0=
2%7C01%7C%7Cdeffc420629649af454408d6c8d6f129%7C84df9e7fe9f640afb435aaaaaaaa=
aaaa%7C1%7C0%7C636917222820865922&amp;sdata=3DsxjXXBtCMOjbK5AZCpLTObP%2=
BIlJRAxXUK7LpLzUDD%2FM%3D&amp;reserved=3D0</a>> with<br>
> container logs.<br>
> <br>
> I see no error here.<br>
> <br>
> Using your cert/key files:<br>
<br>
> There is no OpenLDAP bug here. Your server environment is broken.<br>
<br>
<br>
-- <br>
-- Howard Chu<br>
CTO, Symas Corp. &nbs=
p; <a href=3D"https://nam01.safelinks.protection.outlook.com/?url=3Dh=
ttp%3A%2F%2Fwww.symas.com&amp;data=3D02%7C01%7C%7Cdeffc420629649af45440=
8d6c8d6f129%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917222820865922=
&amp;sdata=3DX5JT6j5%2BQ2BAsKGfNslnC%2FkQj%2BcSU4GAdTqmqqc3lWo%3D&a=
mp;reserved=3D0">
https://nam01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fwww.syma=
s.com&amp;data=3D02%7C01%7C%7Cdeffc420629649af454408d6c8d6f129%7C84df9e=
7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917222820865922&amp;sdata=3DX5J=
T6j5%2BQ2BAsKGfNslnC%2FkQj%2BcSU4GAdTqmqqc3lWo%3D&amp;reserved=3D0</a><=
br>
Director, Highland Sun <a href=3D"https://na=m01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fhighlandsun.com%2F=
hyc%2F&amp;data=3D02%7C01%7C%7Cdeffc420629649af454408d6c8d6f129%7C84df9=
e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917222820865922&amp;sdata=3DSH=
ju26Gxu5dToV%2BuCYDxBMZQS5qJZvREcg9q0CEg2bo%3D&amp;reserved=3D0">
https://nam01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fhighland=
sun.com%2Fhyc%2F&amp;data=3D02%7C01%7C%7Cdeffc420629649af454408d6c8d6f1=
29%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917222820865922&amp;=
sdata=3DSHju26Gxu5dToV%2BuCYDxBMZQS5qJZvREcg9q0CEg2bo%3D&amp;reserved=
=3D0</a><br>
Chief Architect, OpenLDAP <a href=3D"https://nam01.safelinks.p=rotection.outlook.com/?url=3Dhttp%3A%2F%2Fwww.openldap.org%2Fproject%2F&=
;amp;data=3D02%7C01%7C%7Cdeffc420629649af454408d6c8d6f129%7C84df9e7fe9f640a=
fb435aaaaaaaaaaaa%7C1%7C0%7C636917222820865922&amp;sdata=3DfJ7LIrWHv%2F=
G4CJGrx%2BClsFoldJfri%2Bdk7WN59Bt45jU%3D&amp;reserved=3D0">
https://nam01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fwww.open=
ldap.org%2Fproject%2F&amp;data=3D02%7C01%7C%7Cdeffc420629649af454408d6c=
8d6f129%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917222820865922&=
;amp;sdata=3DfJ7LIrWHv%2FG4CJGrx%2BClsFoldJfri%2Bdk7WN59Bt45jU%3D&amp;r=
eserved=3D0</a><br>
</div>
</span></font></div>
</div>
</body>
</html>
--_000_MWHPR08MB24000D77048AFCF7465C4397B53C0MWHPR08MB2400namp_--
Siddharth Jain wrote:
> Wow! Thanks for responding so fast. This could be a bug in docker-openl=
dap then. we have repro'ed this in two different environments - mac and u=
buntu. Do you
> have a recommendation for docker image for openldap?
As I said before, OpenLDAP doesn't touch the certificate files, it merely=
tells the TLS
library where they are. You must likely have a broken TLS library.
-----------------------------------------------------------------------=
-------------------------------------------------------------------------=
----------------
> *From:* Howard Chu <hyc(a)symas.com>
> *Sent:* Wednesday, April 24, 2019 9:42 AM
> *To:* Siddharth Jain; openldap-its(a)OpenLDAP.org
> *Subject:* Re: (ITS#9014) OpenLDAP modifies user provided TLS certifica=
te before sending it to client
> =A0
> Siddharth Jain wrote:
>> we have documented complete steps to repro the bug=A0here <https://eur=04.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithub.com%2Fsid=
djain%2Fopenldap-bug&data=3D02%7C01%7C%7Caca4f78e53324b52690008d6c8d3=
cc09%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636917209315407238&=
sdata=3D8VfRtnCNPd%2BFo2Sps%2BLftBG3XcC57ReIFFphK6noyLc%3D&reserved=3D=
0>=A0with
> container logs.
>=20
> I see no error here.
>=20
> Using your cert/key files:
> There is no OpenLDAP bug here. Your server environment is broken.
--=20
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/