--0000000000000dacee05826a34a9
Content-Type: text/plain; charset="UTF-8"
Thank you, I appreciate it! Do you know how often or when this will get
into the github mirror https://github.com/LMDB/lmdb/commits/mdb.RE/0.9 (or
if I am misunderstanding how the branches are synced)?
On Sun, Feb 10, 2019 at 3:14 AM Howard Chu <hyc(a)symas.com> wrote:
> kriszyp(a)gmail.com wrote:
> > Full_Name: Kristopher William Zyp
> > Version: LMDB 0.9.23
> > OS: Windows Server 2012 R2, Windows 10
> > URL:
> https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45bd32…
> > Submission from: (NULL) (71.199.6.148)
> >
>
> Thanks for the report and patch, added to mdb.RE/0.9
>
> > Calling mdb_env_set_mapsize to increase the map size, when a DB is also
> in use
> > by other processes, when MDB_WRITEMAP is enabled (and the db file has
> been
> > opened with PAGE_READWRITE access), on Windows, will occasionally (seems
> like
> > about 1/100 attempts fails) produce an error "The requested operation
> cannot be
> > performed on a file with a user-mapped section open", or segfaults. The
> error
> > occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map that
> is
> > performed to increase the allocated file size to the map size, prior to
> > CreateFileMapping.
> >
> > As it turns out this is pretty easy to solve, because manually expanding
> the
> > file size is not necessary when calling CreateFileMapping with
> PAGE_READWRITE
> > access, as Windows will automatically expand the file size for us, when
> opened
> > with the page write access enabled. Of course, this means all processes
> must be
> > consistent in use of MDB_WRITEMAP, but the documentation already makes
> this
> > explicit and clear.
> >
> > I believe this can be fixed by simply adding a check for MDB_WRITEMAP in
> the if
> > statement that calls SetFilePointer:
> >
> > if (!(flags & MDB_WRITEMAP) && (SetFilePointer(env->me_fd,
> sizelo, &sizehi, 0)
> > != (DWORD)sizelo
> > || !SetEndOfFile(env->me_fd)
> > || SetFilePointer(env->me_fd, 0, NULL, 0) != 0))
> > return ErrCode();
> >
> > The attached URL has the change as a patch/diff as applied to our node
> package.
> >
> > I am certainly happy to just keep this change on our own branches. There
> may be
> > nuances of this that I might not be aware of, but it seems to be working
> great
> > for us and I have tested this with MDB_WRITEMAP enabled and disabled. So
> I
> > thought I would offer/suggest this change, as it seems like it is
> > straightforward change to improve stability. Thank you!
> >
> >
>
>
> --
> -- Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
>
--0000000000000dacee05826a34a9
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr">Thank you, I appreciate it! Do you know h=
ow often or when this will get into the github mirror <a href=3D"https://gi=thub.com/LMDB/lmdb/commits/mdb.RE/0.9">https://github.com/LMDB/lmdb/commits=
/mdb.RE/0.9</a> (or if I am misunderstanding how the branches are synced)?<=
/div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_a=
ttr">On Sun, Feb 10, 2019 at 3:14 AM Howard Chu <<a href=3D"mailto:hyc@s=
ymas.com">hyc(a)symas.com</a>> wrote:<br></div><blockquote class=3D"gmail_=
quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,=
204);padding-left:1ex"><a href=3D"mailto:kriszyp@gmail.com" target=3D"_blan=
k">kriszyp(a)gmail.com</a> wrote:<br>
> Full_Name: Kristopher William Zyp<br>
> Version: LMDB 0.9.23<br>
> OS: Windows Server 2012 R2, Windows 10<br>
> URL: <a href=3D"https://github.com/kriszyp/node-lmdb/commit/6df903907f=
5516320e9a8afce45bd32ab4e8e1f2.patch" rel=3D"noreferrer" target=3D"_blank">=
https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45bd32=
ab4e8e1f2.patch</a><br>
> Submission from: (NULL) (71.199.6.148)<br>
> <br>
<br>
Thanks for the report and patch, added to mdb.RE/0.9<br>
<br>
> Calling mdb_env_set_mapsize to increase the map size, when a DB is als=
o in use<br>
> by other processes, when MDB_WRITEMAP is enabled (and the db file has =
been<br>
> opened with PAGE_READWRITE access), on Windows, will occasionally (see=
ms like<br>
> about 1/100 attempts fails) produce an error "The requested opera=
tion cannot be<br>
> performed on a file with a user-mapped section open", or segfault=
s. The error<br>
> occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map tha=
t is<br>
> performed to increase the allocated file size to the map size, prior t=
o<br>
> CreateFileMapping.<br>
> <br>
> As it turns out this is pretty easy to solve, because manually expandi=
ng the<br>
> file size is not necessary when calling CreateFileMapping with PAGE_RE=
ADWRITE<br>
> access, as Windows will automatically expand the file size for us, whe=
n opened<br>
> with the page write access enabled. Of course, this means all processe=
s must be<br>
> consistent in use of MDB_WRITEMAP, but the documentation already makes=
this<br>
> explicit and clear.<br>
> <br>
> I believe this can be fixed by simply adding a check for MDB_WRITEMAP =
in the if<br>
> statement that calls SetFilePointer:<br>
> <br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (!(flags &=
; MDB_WRITEMAP) && (SetFilePointer(env->me_fd, sizelo, &size=
hi, 0)<br>
> !=3D (DWORD)sizelo<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0|| !SetEndOfFile(env->me_fd)<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0|| SetFilePointer(env->me_fd, 0, NULL, 0) !=3D 0))<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0return ErrCode();<br>
> <br>
> The attached URL has the change as a patch/diff as applied to our node=
package.<br>
> <br>
> I am certainly happy to just keep this change on our own branches. The=
re may be<br>
> nuances of this that I might not be aware of, but it seems to be worki=
ng great<br>
> for us and I have tested this with MDB_WRITEMAP enabled and disabled. =
So I<br>
> thought I would offer/suggest this change, as it seems like it is<br>
> straightforward change to improve stability. Thank you!<br>
> <br>
> <br>
<br>
<br>
-- <br>
=C2=A0 -- Howard Chu<br>
=C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"=
http://www.symas.com" rel=3D"noreferrer" target=3D"_blank">http://www.symas=
.com</a><br>
=C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a href=3D"http://highland=sun.com/hyc/" rel=3D"noreferrer" target=3D"_blank">http://highlandsun.com/h=
yc/</a><br>
=C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http://www.openldap.org/p=
roject/" rel=3D"noreferrer" target=3D"_blank">http://www.openldap.org/proje=
ct/</a><br>
</blockquote></div>
--0000000000000dacee05826a34a9--
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><font face="Arial,Verdana,Helvetica">The attached files are
derived from OpenLDAP Software.
All of the modifications to OpenLDAP Software represented in the
following patch(es) were developed by Interlink Networks LLC. </font><font
face="Arial,Verdana,Helvetica"><font
face="Arial,Verdana,Helvetica">Interlink Networks LLC</font>
has not assigned rights and/or interest in
this work to any party. I, Vernon Smith am authorized by </font><font
face="Arial,Verdana,Helvetica"><font
face="Arial,Verdana,Helvetica">Interlink Networks LLC</font>,
my employer, to release this work under
the following terms.</font></p>
<p><font face="Arial,Verdana,Helvetica"><font
face="Arial,Verdana,Helvetica">Interlink Networks LLC </font></font><font
face="Arial,Verdana,Helvetica">hereby place the following
modifications to
OpenLDAP Software (and only these modifications) into the public
domain. Hence, these modifications may be freely used and/or
redistributed for any purpose with or without attribution and/or
other notice.</font></p>
<p><font face="Arial,Verdana,Helvetica">Thanks, Vern<br>
</font></p>
<div class="moz-cite-prefix">On 2/20/2019 9:42 AM, Quanah
Gibson-Mount wrote:<br>
</div>
<blockquote type="cite"
cite="mid:8A8B819A395EEAE290F20248@[192.168.1.39]">--On Tuesday,
February 19, 2019 2:41 PM +0000 <a class="moz-txt-link-abbreviated" href="mailto:quanah@symas.com">quanah(a)symas.com</a> wrote:
<br>
<br>
<blockquote type="cite">--On Tuesday, February 19, 2019 2:12 AM
-0500 Vern Smith
<br>
<a class="moz-txt-link-rfc2396E" href="mailto:vsmith@interlinknetworks.com"><vsmith(a)interlinknetworks.com></a> wrote:
<br>
<br>
<blockquote type="cite">ITS#8957 and ITS#8968 are my submits and
I was using those changes.
<br>
ITS#8963 is for OpenLDAP servers and I only an using the
libraries so it
<br>
does not apply to my updates.
<br>
</blockquote>
<br>
Please keep replies to the ITS system.
<br>
</blockquote>
<br>
Hi,
<br>
<br>
Although each change is individually tiny, the sum of changes is
not. Can you please add an IPR to this ITS, as documented at
<a class="moz-txt-link-rfc2396E" href="https://www.openldap.org/devel/contributing.html#notice"><https://www.openldap.org/devel/contributing.html#notice></a>
<br>
<br>
Thanks!
<br>
<br>
Regards,
<br>
Quanah
<br>
<br>
<br>
--
<br>
<br>
Quanah Gibson-Mount
<br>
Product Architect
<br>
Symas Corporation
<br>
Packaged, certified, and supported LDAP solutions powered by
OpenLDAP:
<br>
<a class="moz-txt-link-rfc2396E" href="http://www.symas.com"><http://www.symas.com></a>
<br>
<br>
<br>
</blockquote>
</body>
</html>
--On Tuesday, February 19, 2019 2:41 PM +0000 quanah(a)symas.com wrote:
> --On Tuesday, February 19, 2019 2:12 AM -0500 Vern Smith
> <vsmith(a)interlinknetworks.com> wrote:
>
>> ITS#8957 and ITS#8968 are my submits and I was using those changes.
>> ITS#8963 is for OpenLDAP servers and I only an using the libraries so it
>> does not apply to my updates.
>
> Please keep replies to the ITS system.
Hi,
Although each change is individually tiny, the sum of changes is not. Can
you please add an IPR to this ITS, as documented at
<https://www.openldap.org/devel/contributing.html#notice>
Thanks!
Regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
quanah(a)openldap.org wrote:
> Full_Name: Quanah Gibson-Mount
> Version: RE25
> OS: N/A
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (47.208.148.239)
>
>
> It's currently difficult for downstream packagers to create a working
> development package as some items required to build external modules are only
> included what are considered "private" headers, such as slap.h (See ITS#7283).
> SLAP_CB_{BYPASS,CONTINUE} are examples of items that are contained in slap.h
> that are potentially required for building external modules.
>
> IIRC, it's also been discussed previously to rename "ldap_pvt.h" to "openldap.h"
> to indicate it's ok to package this header file in a -devel package (I could
> have the initial header name incorrect, it's been a while).
This discussion is a bit awkward. OpenLDAP is an open source project. What we provide
is a distribution of source code. It was never intended to be sliced and diced into
individual binary packages. If you're developing modules for OpenLDAP, you're expected
to have a complete OpenLDAP build tree in which your modules reside. Modules meant to
run inside slapd are by definition *internal* - they are running inside slapd after all.
The fact that they're written and maintained by 3rd parties doesn't change this.
Header files marked "private" mean they're only for internal use within an OpenLDAP
build tree. Applications that are generic LDAP clients should only use the public APIs.
We can still discuss formally making some OpenLDAP-specific APIs public, but that
should probably be done on a case by case basis. LDAP is supposed to be interoperable;
we should not encourage broader use of OpenLDAP-specific APIs that rely on details
of OpenLDAP's internal implementation. If they have no particular dependency on
internals, then we may safely expose them.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
Full_Name: Quanah Gibson-Mount
Version: RE25
OS: N/A
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (47.208.148.239)
It's currently difficult for downstream packagers to create a working
development package as some items required to build external modules are only
included what are considered "private" headers, such as slap.h (See ITS#7283).
SLAP_CB_{BYPASS,CONTINUE} are examples of items that are contained in slap.h
that are potentially required for building external modules.
IIRC, it's also been discussed previously to rename "ldap_pvt.h" to "openldap.h"
to indicate it's ok to package this header file in a -devel package (I could
have the initial header name incorrect, it's been a while).
--On Tuesday, February 19, 2019 2:12 AM -0500 Vern Smith
<vsmith(a)interlinknetworks.com> wrote:
> ITS#8957 and ITS#8968 are my submits and I was using those changes.
> ITS#8963 is for OpenLDAP servers and I only an using the libraries so it
> does not apply to my updates.
Please keep replies to the ITS system.
When filing an ITS, please note specifically which commits past a release
you've applied, so the ITS is a clear representation of the issue you're
reporting (and thus avoiding questions such as those I posited). Thanks!
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
--On Tuesday, February 19, 2019 5:57 AM +0000 vsmith(a)interlinknetworks.com
wrote:
> Full_Name: Vernon Smith
> Version: 2.4.47
> OS: Linux, Solaris
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (2601:40d:4300:679a:d18c:3060:e826:d35c)
>
>
> I am using libldap library built with -DLDAP_USE_NON_BLOCKING_TLS and
> configured for Async connection mode. I test making connections using the
> library to servers that are hung to verify that my application will not
> hang in those cases. I have found 3 issues. The first is that the
> ldap_pvt_connect() clears non-blocking socket setup after the connection
> is made even when Async mode was configured. So here is my patch for that.
Did you pick up the fix for ITS#8957 from RE24?
Did you pick up the fix for ITS#8968 from RE24?
Did you pick up the fix for ITS#8963 from RE24?
Those are all post 2.4.47 fixes.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
Full_Name: Vernon Smith
Version: 2.4.47
OS: Linux, Solaris
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (2601:40d:4300:679a:d18c:3060:e826:d35c)
I am using libldap library built with -DLDAP_USE_NON_BLOCKING_TLS and configured
for Async connection mode. I test making connections using the library to
servers that are hung to verify that my application will not hang in those
cases. I have found 3 issues. The first is that the ldap_pvt_connect() clears
non-blocking socket setup after the connection is made even when Async mode was
configured. So here is my patch for that.
diff --git a/libraries/libldap/os-ip.c b/libraries/libldap/os-ip.c
index a823cc6..d7927e5 100644
--- a/libraries/libldap/os-ip.c
+++ b/libraries/libldap/os-ip.c
@@ -443,7 +443,7 @@ ldap_pvt_connect(LDAP *ld, ber_socket_t s,
if ( connect(s, sin, addrlen) != AC_SOCKET_ERROR ) {
osip_debug(ld, "connect success\n", 0, 0, 0);
- if ( opt_tv && ldap_pvt_ndelay_off(ld, s) == -1 )
+ if ( !async && opt_tv && ldap_pvt_ndelay_off(ld, s) == -1 )
return ( -1 );
return ( 0 );
}
The second issue is that the tlso_session_connect() routine does not correctly
handle the return code from SSL_connect(), it just returns it to the caller. For
LDAP_USE_NON_BLOCKING_TLS, the return code must be checked and an appropriate
return used. Here is my patch.
diff --git a/libraries/libldap/tls_o.c b/libraries/libldap/tls_o.c
index e95a448..7a31b5e 100644
--- a/libraries/libldap/tls_o.c
+++ b/libraries/libldap/tls_o.c
@@ -531,7 +531,23 @@ tlso_session_connect( LDAP *ld, tls_session *sess )
tlso_session *s = (tlso_session *)sess;
/* Caller expects 0 = success, OpenSSL returns 1 = success */
- return SSL_connect( s ) - 1;
+ int rc = SSL_connect( s ) - 1;
+
+#ifdef LDAP_USE_NON_BLOCKING_TLS
+ int sslerr = SSL_get_error(s, rc+1);
+ int sockerr = sock_errno();
+
+ if ( rc < 0 ) {
+ if ( sslerr == SSL_ERROR_WANT_READ || sslerr == SSL_ERROR_WANT_WRITE ) {
+ rc = 0;
+ } else if (( sslerr == SSL_ERROR_SYSCALL ) &&
+ ( sockerr == EAGAIN || sockerr == ENOTCONN )) {
+ rc = 0;
+ }
+ }
+#endif /* LDAP_USE_NON_BLOCKING_TLS */
+
+ return rc;
}
static int
The third issue is that ldap_int_tls_start() compiled with
-DLDAP_USE_NON_BLOCKING_TLS Plays with the socket non-blocking setting even if
Async mode is configured. Here is mt patch to only play with the socket setting
if not in Async mode.
diff --git a/libraries/libldap/tls2.c b/libraries/libldap/tls2.c
index d9b2d27..69b749c 100644
--- a/libraries/libldap/tls2.c
+++ b/libraries/libldap/tls2.c
@@ -1075,8 +1075,11 @@ ldap_int_tls_start ( LDAP *ld, LDAPConn *conn,
LDAPURLDesc *srv )
/*
* Use non-blocking io during SSL Handshake when a timeout is configured
*/
+ int async = LDAP_BOOL_GET( &ld->ld_options, LDAP_BOOL_CONNECT_ASYNC );
if ( ld->ld_options.ldo_tm_net.tv_sec >= 0 ) {
- ber_sockbuf_ctrl( sb, LBER_SB_OPT_SET_NONBLOCK, (void*)1 );
+ if ( ! async ) {
+ ber_sockbuf_ctrl( sb, LBER_SB_OPT_SET_NONBLOCK, (void*)1 );
+ }
ber_sockbuf_ctrl( sb, LBER_SB_OPT_GET_FD, &sd );
tv = ld->ld_options.ldo_tm_net;
tv0 = tv;
@@ -1110,8 +1113,10 @@ ldap_int_tls_start ( LDAP *ld, LDAPConn *conn,
LDAPURLDesc *srv )
ld->ld_errno = LDAP_TIMEOUT;
break;
} else {
- /* ldap_int_poll called ldap_pvt_ndelay_off */
- ber_sockbuf_ctrl( sb, LBER_SB_OPT_SET_NONBLOCK, (void*)1 );
+ /* ldap_int_poll called ldap_pvt_ndelay_off if not in async mode */
+ if ( ! async ) {
+ ber_sockbuf_ctrl( sb, LBER_SB_OPT_SET_NONBLOCK, (void*)1 );
+ }
ret = ldap_int_tls_connect( ld, conn, host );
if ( ret > 0 ) { /* need to call tls_connect once more */
struct timeval curr_time_tv, delta_tv;
@@ -1159,7 +1164,9 @@ ldap_int_tls_start ( LDAP *ld, LDAPConn *conn, LDAPURLDesc
*srv )
}
}
if ( ld->ld_options.ldo_tm_net.tv_sec >= 0 ) {
- ber_sockbuf_ctrl( sb, LBER_SB_OPT_SET_NONBLOCK, NULL );
+ if ( ! async ) {
+ ber_sockbuf_ctrl( sb, LBER_SB_OPT_SET_NONBLOCK, NULL );
+ }
}
#endif /* LDAP_USE_NON_BLOCKING_TLS */
Thanks, Vern