Re: (ITS#8980) Async mode and TLS Non-Blocking Issue
by quanah＠symas.com
--On Wednesday, February 20, 2019 2:30 PM -0500 Vern Smith
<vsmith(a)interlinknetworks.com> wrote:
>
> The attached files are derived from OpenLDAP Software. All of the
> modifications to OpenLDAP Software represented in the following patch(es)
> were developed by Interlink Networks LLC. Interlink Networks LLC has not
> assigned rights and/or interest in this work to any party. I, Vernon
> Smith am authorized by Interlink Networks LLC, my employer, to release
> this work under the following terms.
Thanks! This has been applied to git master.
Regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
4 years, 3 months
(ITS#8985) Fix of an assertion error in back-asyncmeta during restart of remote target
by nivanova＠symas.com
Full_Name: Nadezhda Ivanova
Version: 2.5
OS: Ubuntu 18.04
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (93.152.142.241)
The attached patches provide a fix for a problem, reported by a customer:
Occasionally, when the remote target of an asyncmeta database is restarted,
slapd crashes with assertion error in connection_init:
assert( c->c_struct_state == SLAP_C_UNUSED );
The issue occurs very rarely, but I managed to reproduce it.
It was caused by the fact that on socket error, try_read1msg would close the
LDAP connection (ld) to that target, making the fd available to be used again
for another connection. Asyncmeta is not aware of that, so it doesn't stop the
listener on the corresponding Connection. If a new LDAP connection is opened to
a different, active target using the same fd, asyncmeta would call
connection_init on an already used Connection, which causes the assertion
failure.
The fix consists of three separate patches:
ftp://ftp.openldap.org/incoming/nadezhda-ivanova-190228_2.patch
This moves initizalization of c_mutex to connections_init, and therefore any
further access to a Connection can be synchronized, eliminating some race
conditions.
ftp://ftp.openldap.org/incoming/nadezhda-ivanova-190228_3.patch
This adds the LDAP_OPT_KEEPCONN option, which instructs try_read1msg that the
connection will be freed by the client.
ftp://ftp.openldap.org/incoming/nadezhda-ivanova-190228_4.patch
This updates asyncmeta to make use of LDAP_OPT_KEEPCONN.
IPR:
The attached files are derived from OpenLDAP Software. All of the modifications
to OpenLDAP Software represented in the following patch were developed by Symas
Corporation. Symas Corporation has not assigned rights and/or interest in this
work to any party. I, Nadezhda Ivanova, am authorized by Symas Corporation, my
employer, to release this work under the following terms.
The attached modifications to OpenLDAP Software are subject to the following
notice:
Copyright 2019 Symas Corporation
Redistribution and use in source and binary forms, with or without
modification,
are permitted only as authorized by the OpenLDAP Public License.
4 years, 3 months
Re: (ITS#8734) Fixes for multiple back-asyncmeta issues
by nivanova＠symas.com
A new version of the patch has been uploaded at ftp://ftp.openldap.org/incoming/nadezhda-ivanova-190228.patch
The patch has been integrated with master and tested.
It includes all the changes necessary to fix back-asyncmeta issues
discovered during on-site testing since the start of 2016.
These include:
Issues with stability - crashes and assertion failures
Incorrect behavior during unstable network conditions, such as inability to reset connections
or process responses, or "hanging" to wait for a response that would never be received.
Memory leaks and memory management fixes - major redesign of the way back-asyncmeta
works with memory contexts.
Rewrite was replaced with suffix-massage in configuration, and the network-timeout value was changed to milliseconds.
Incorrect behavior when SASL is used to bind to a target.
Many problems caused by race conditions
Fixes for compiler warnings, and tests.
Cleanup of unused code.
Nadezhda Ivanova
Software Engineer
Symas Corporation http://www.symas.com
4 years, 3 months
(ITS#8984) Issue with socket buffer assertion failed
by gandhali.atre＠plantronics.com
Full_Name: Gandhali Atre
Version: 2.4.44
OS: Android
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (140.242.26.200)
Hi,
I have a product that allows to configure LDAP servers. Looks like when
changing the configurations or doing some search 1/10 times there is a crash and
below are the crash details.
messages 2019-02-22 22:11:38.543 logcat:DEBUG(6046):Abort message
'../../../libraries/liblber/sockbuf.c:89: ber_sockbuf_ctrl: assertion
"SOCKBUF_VALID( sb )" failed'
messages 2019-02-22 22:11:38.543 logcat:DEBUG(6046) r0 00000000
r1 000011bb r2 00000006 r3 00000008
messages 2019-02-22 22:11:38.543 logcat:DEBUG(6046) r4 00001176
r5 000011bb r6 ed27df1c r7 0000010c
messages 2019-02-22 22:11:38.543 logcat:DEBUG(6046) r8 00000000
r9 ed27f290 sl 025b6410 fp ed27ef64
messages 2019-02-22 22:11:38.543 logcat:DEBUG(6046) ip 00000000
sp ed27df08 lr ef37c7f3 pc ef376774 cpsr 20000030
messages 2019-02-22 22:11:38.545 logcat:DEBUG(6046)
messages 2019-02-22 22:11:38.545 backtrace
messages 2019-02-22 22:11:38.545 logcat:DEBUG(6046) #00 pc
0001a774 /system/lib/libc.so (abort+63)
messages 2019-02-22 22:11:38.548 logcat:DEBUG(6046) #01 pc
0001a963 /system/lib/libc.so (__assert2+22)
messages 2019-02-22 22:11:38.548 logcat:DEBUG(6046) #02 pc
0000a860 /system/lib/liblber-2.4.so.2 (ber_sockbuf_ctrl+188)
I read about some fixes already made for this assertion failures. Looks like it
isn't working or this is something new. Any help with this would be
appreciated.,
Thanks.
4 years, 3 months
RE: (ITS#8975) Calling mdb_env_set_mapsize on a shared DB failssometimes
by kriszyp＠gmail.com
--_36998F79-0E7E-483F-8FFD-6401D532832A_
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="utf-8"
I see it now! I really did check even earlier this morning and wasn=E2=80=
=99t there until recently! Anyway, thank you, sorry for the noise.
Thanks,
Kris
From: Quanah Gibson-Mount
Sent: February 21, 2019 10:12 AM
To: kriszyp(a)gmail.com; openldap-its(a)OpenLDAP.org
Subject: Re: (ITS#8975) Calling mdb_env_set_mapsize on a shared DB failssom=
etimes
Hi,
It is literally the first set of commits listed at that URL, so it's=20
clearly already there.
Hope that helps!
--Quanah
--On Thursday, February 21, 2019 4:45 PM +0000 kriszyp(a)gmail.com wrote:
> --0000000000000dacee05826a34a9
> Content-Type: text/plain; charset=3D"UTF-8"
>
> Thank you, I appreciate it! Do you know how often or when this will get
> into the github mirror https://github.com/LMDB/lmdb/commits/mdb.RE/0.9 (o=
r
> if I am misunderstanding how the branches are synced)?
>
> On Sun, Feb 10, 2019 at 3:14 AM Howard Chu <hyc(a)symas.com> wrote:
>
>> kriszyp(a)gmail.com wrote:
>> > Full_Name: Kristopher William Zyp
>> > Version: LMDB 0.9.23
>> > OS: Windows Server 2012 R2, Windows 10
>> > URL:
>> https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45b
>> d32ab4e8e1f2.patch
>> > Submission from: (NULL) (71.199.6.148)
>> >
>>
>> Thanks for the report and patch, added to mdb.RE/0.9
>>
>> > Calling mdb_env_set_mapsize to increase the map size, when a DB is als=
o
>> in use
>> > by other processes, when MDB_WRITEMAP is enabled (and the db file has
>> been
>> > opened with PAGE_READWRITE access), on Windows, will occasionally
>> > (seems
>> like
>> > about 1/100 attempts fails) produce an error "The requested operation
>> cannot be
>> > performed on a file with a user-mapped section open", or segfaults. Th=
e
>> error
>> > occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map tha=
t
>> is
>> > performed to increase the allocated file size to the map size, prior t=
o
>> > CreateFileMapping.
>> >
>> > As it turns out this is pretty easy to solve, because manually
>> > expanding
>> the
>> > file size is not necessary when calling CreateFileMapping with
>> PAGE_READWRITE
>> > access, as Windows will automatically expand the file size for us, whe=
n
>> opened
>> > with the page write access enabled. Of course, this means all processe=
s
>> must be
>> > consistent in use of MDB_WRITEMAP, but the documentation already makes
>> this
>> > explicit and clear.
>> >
>> > I believe this can be fixed by simply adding a check for MDB_WRITEMAP
>> > in
>> the if
>> > statement that calls SetFilePointer:
>> >
>> > if (!(flags & MDB_WRITEMAP) &&
>> > (SetFilePointer(env->me_fd,
>> sizelo, &sizehi, 0)
>> > !=3D (DWORD)sizelo
>> > || !SetEndOfFile(env->me_fd)
>> > || SetFilePointer(env->me_fd, 0, NULL, 0) !=3D 0=
))
>> > return ErrCode();
>> >
>> > The attached URL has the change as a patch/diff as applied to our node
>> package.
>> >
>> > I am certainly happy to just keep this change on our own branches.
>> > There
>> may be
>> > nuances of this that I might not be aware of, but it seems to be
>> > working
>> great
>> > for us and I have tested this with MDB_WRITEMAP enabled and disabled.
>> > So
>> I
>> > thought I would offer/suggest this change, as it seems like it is
>> > straightforward change to improve stability. Thank you!
>> >
>> >
>>
>>
>> --
>> -- Howard Chu
>> CTO, Symas Corp. http://www.symas.com
>> Director, Highland Sun http://highlandsun.com/hyc/
>> Chief Architect, OpenLDAP http://www.openldap.org/project/
>>
>
> --0000000000000dacee05826a34a9
> Content-Type: text/html; charset=3D"UTF-8"
> Content-Transfer-Encoding: quoted-printable
>
> <div dir=3D3D"ltr"><div dir=3D3D"ltr">Thank you, I appreciate it! Do you =
know
> h=3D ow often or when this will get into the github mirror <a
> href=3D3D"https://gi=3D
> thub.com/LMDB/lmdb/commits/mdb.RE/0.9">https://github.com/LMDB/lmdb/commi
> ts=3D /mdb.RE/0.9</a> (or if I am misunderstanding how the branches are
> synced)?<=3D /div></div><br><div class=3D3D"gmail_quote"><div dir=3D3D"lt=
r"
> class=3D3D"gmail_a=3D ttr">On Sun, Feb 10, 2019 at 3:14 AM Howard Chu <=
;<a
> href=3D3D"mailto:hyc@s=3D ymas.com">hyc(a)symas.com</a>>
> wrote:<br></div><blockquote class=3D3D"gmail_=3D quote" style=3D3D"margin=
:0px
> 0px 0px 0.8ex;border-left:1px solid rgb(204,204,=3D
> 204);padding-left:1ex"><a href=3D3D"mailto:kriszyp@gmail.com"
> target=3D3D"_blan=3D k">kriszyp(a)gmail.com</a> wrote:<br>
> > Full_Name: Kristopher William Zyp<br>
> > Version: LMDB 0.9.23<br>
> > OS: Windows Server 2012 R2, Windows 10<br>
> > URL: <a
> href=3D3D"https://github.com/kriszyp/node-lmdb/commit/6df903907f=3D
> 5516320e9a8afce45bd32ab4e8e1f2.patch" rel=3D3D"noreferrer"
> target=3D3D"_blank">=3D
> https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45bd
> 32=3D ab4e8e1f2.patch</a><br>
> > Submission from: (NULL) (71.199.6.148)<br>
> > <br>
> <br>
> Thanks for the report and patch, added to mdb.RE/0.9<br>
> <br>
> > Calling mdb_env_set_mapsize to increase the map size, when a DB is
> als=3D o in use<br>
> > by other processes, when MDB_WRITEMAP is enabled (and the db file
> has =3D been<br>
> > opened with PAGE_READWRITE access), on Windows, will occasionally
> (see=3D ms like<br>
> > about 1/100 attempts fails) produce an error "The requested
> opera=3D tion cannot be<br>
> > performed on a file with a user-mapped section open", or
> segfault=3D s. The error<br>
> > occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map
> tha=3D t is<br>
> > performed to increase the allocated file size to the map size, prior
> t=3D o<br>
> > CreateFileMapping.<br>
> > <br>
> > As it turns out this is pretty easy to solve, because manually
> expandi=3D ng the<br>
> > file size is not necessary when calling CreateFileMapping with
> PAGE_RE=3D ADWRITE<br>
> > access, as Windows will automatically expand the file size for us,
> whe=3D n opened<br>
> > with the page write access enabled. Of course, this means all
> processe=3D s must be<br>
> > consistent in use of MDB_WRITEMAP, but the documentation already
> makes=3D this<br>
> > explicit and clear.<br>
> > <br>
> > I believe this can be fixed by simply adding a check for
> MDB_WRITEMAP =3D in the if<br>
> > statement that calls SetFilePointer:<br>
> > <br>
> >=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3D=
C2=3DA0 =3DC2=3DA0if (!(flags
> &=3D ; MDB_WRITEMAP) && (SetFilePointer(env->me_fd, sizelo,
> &size=3D hi, 0)<br>
> > !=3D3D (DWORD)sizelo<br>
> >=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3D=
C2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0
> =3D =3DC2=3DA0 =3DC2=3DA0|| !SetEndOfFile(env->me_fd)<br>
> >=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3D=
C2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0
> =3D =3DC2=3DA0 =3DC2=3DA0|| SetFilePointer(env->me_fd, 0, NULL, 0) !=
=3D3D 0))<br>
> >=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3D=
C2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0
> =3D =3DC2=3DA0 =3DC2=3DA0return ErrCode();<br>
> > <br>
> > The attached URL has the change as a patch/diff as applied to our
> node=3D package.<br>
> > <br>
> > I am certainly happy to just keep this change on our own branches.
> The=3D re may be<br>
> > nuances of this that I might not be aware of, but it seems to be
> worki=3D ng great<br>
> > for us and I have tested this with MDB_WRITEMAP enabled and
> disabled. =3D So I<br>
> > thought I would offer/suggest this change, as it seems like it is<br=
>
> > straightforward change to improve stability. Thank you!<br>
> > <br>
> > <br>
> <br>
> <br>
> -- <br>
> =3DC2=3DA0 -- Howard Chu<br>
> =3DC2=3DA0 CTO, Symas Corp.=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =
=3DC2=3DA0 =3DC2=3DA0<a
> href=3D3D"=3D http://www.symas.com" rel=3D3D"noreferrer"
> target=3D3D"_blank">http://www.symas=3D .com</a><br>
> =3DC2=3DA0 Director, Highland Sun=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0<a
> href=3D3D"http://highland=3D sun.com/hyc/" rel=3D3D"noreferrer"
> target=3D3D"_blank">http://highlandsun.com/h=3D yc/</a><br>
> =3DC2=3DA0 Chief Architect, OpenLDAP=3DC2=3DA0 <a
> href=3D3D"http://www.openldap.org/p=3D roject/" rel=3D3D"noreferrer"
> target=3D3D"_blank">http://www.openldap.org/proje=3D ct/</a><br>
> </blockquote></div>
>
> --0000000000000dacee05826a34a9--
>
>
>
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
--_36998F79-0E7E-483F-8FFD-6401D532832A_
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset="utf-8"
<html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc=
hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of=
fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta ht=
tp-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta name=
=3DGenerator content=3D"Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=3DEN-CA link=3Dblue vlink=3D"#954F72"><div cla=
ss=3DWordSection1><p class=3DMsoNormal>I see it now! I really did check eve=
n earlier this morning and wasn=E2=80=99t there until recently! Anyway, tha=
nk you, sorry for the noise.</p><p class=3DMsoNormal><o:p> </o:p></p><=
p class=3DMsoNormal>Thanks,<br>Kris</p><p class=3DMsoNormal><o:p> </o:=
p></p><div style=3D'mso-element:para-border-div;border:none;border-top:soli=
d #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=3DMsoNormal style=3D'bo=
rder:none;padding:0cm'><b>From: </b><a href=3D"mailto:quanah@symas.com">Qua=
nah Gibson-Mount</a><br><b>Sent: </b>February 21, 2019 10:12 AM<br><b>To: <=
/b><a href=3D"mailto:kriszyp@gmail.com">kriszyp(a)gmail.com</a>; <a href=3D"m=
ailto:openldap-its@OpenLDAP.org">openldap-its(a)OpenLDAP.org</a><br><b>Subjec=
t: </b>Re: (ITS#8975) Calling mdb_env_set_mapsize on a shared DB failssomet=
imes</p></div><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNorma=
l>Hi,</p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>It =
is literally the first set of commits listed at that URL, so it's </p><p cl=
ass=3DMsoNormal>clearly already there.</p><p class=3DMsoNormal><o:p> <=
/o:p></p><p class=3DMsoNormal>Hope that helps!</p><p class=3DMsoNormal><o:p=
> </o:p></p><p class=3DMsoNormal>--Quanah</p><p class=3DMsoNormal><o:p=
> </o:p></p><p class=3DMsoNormal>--On Thursday, February 21, 2019 4:45=
PM +0000 kriszyp(a)gmail.com wrote:</p><p class=3DMsoNormal><o:p> </o:p=
></p><p class=3DMsoNormal>> --0000000000000dacee05826a34a9</p><p class=
=3DMsoNormal>> Content-Type: text/plain; charset=3D"UTF-8"</p>=
<p class=3DMsoNormal>><o:p> </o:p></p><p class=3DMsoNormal>> Tha=
nk you, I appreciate it! Do you know how often or when this will get</p><p =
class=3DMsoNormal>> into the github mirror https://github.com/LMDB/lmdb/=
commits/mdb.RE/0.9 (or</p><p class=3DMsoNormal>> if I am misunderstandin=
g how the branches are synced)?</p><p class=3DMsoNormal>><o:p> </o:=
p></p><p class=3DMsoNormal>> On Sun, Feb 10, 2019 at 3:14 AM Howard Chu =
<hyc(a)symas.com> wrote:</p><p class=3DMsoNormal>><o:p> </o:p><=
/p><p class=3DMsoNormal>>> kriszyp(a)gmail.com wrote:</p><p class=3DMso=
Normal>>> > Full_Name: Kristopher William Zyp</p><p class=3DMsoNor=
mal>>> > Version: LMDB 0.9.23</p><p class=3DMsoNormal>>> >=
; OS: Windows Server 2012 R2, Windows 10</p><p class=3DMsoNormal>>> &=
gt; URL:</p><p class=3DMsoNormal>>> https://github.com/kriszyp/node-l=
mdb/commit/6df903907f5516320e9a8afce45b</p><p class=3DMsoNormal>>> d3=
2ab4e8e1f2.patch</p><p class=3DMsoNormal>>> > Submission from: (NU=
LL) (71.199.6.148)</p><p class=3DMsoNormal>>> ></p><p class=3DMsoN=
ormal>>><o:p> </o:p></p><p class=3DMsoNormal>>> Thanks for=
the report and patch, added to mdb.RE/0.9</p><p class=3DMsoNormal>>>=
<o:p> </o:p></p><p class=3DMsoNormal>>> > Calling mdb_env_set=
_mapsize to increase the map size, when a DB is also</p><p class=3DMsoNorma=
l>>> in use</p><p class=3DMsoNormal>>> > by other processes,=
when MDB_WRITEMAP is enabled (and the db file has</p><p class=3DMsoNormal>=
>> been</p><p class=3DMsoNormal>>> > opened with PAGE_READWR=
ITE access), on Windows, will occasionally</p><p class=3DMsoNormal>>>=
> (seems</p><p class=3DMsoNormal>>> like</p><p class=3DMsoNormal>=
>> > about 1/100 attempts fails) produce an error "The reques=
ted operation</p><p class=3DMsoNormal>>> cannot be</p><p class=3DMsoN=
ormal>>> > performed on a file with a user-mapped section open&quo=
t;, or segfaults. The</p><p class=3DMsoNormal>>> error</p><p class=3D=
MsoNormal>>> > occurs in the SetFilePointer (or SetEndOfFile) call=
in mdb_env_map that</p><p class=3DMsoNormal>>> is</p><p class=3DMsoN=
ormal>>> > performed to increase the allocated file size to the ma=
p size, prior to</p><p class=3DMsoNormal>>> > CreateFileMapping.</=
p><p class=3DMsoNormal>>> ></p><p class=3DMsoNormal>>> > =
As it turns out this is pretty easy to solve, because manually</p><p class=
=3DMsoNormal>>> > expanding</p><p class=3DMsoNormal>>> the</=
p><p class=3DMsoNormal>>> > file size is not necessary when callin=
g CreateFileMapping with</p><p class=3DMsoNormal>>> PAGE_READWRITE</p=
><p class=3DMsoNormal>>> > access, as Windows will automatically e=
xpand the file size for us, when</p><p class=3DMsoNormal>>> opened</p=
><p class=3DMsoNormal>>> > with the page write access enabled. Of =
course, this means all processes</p><p class=3DMsoNormal>>> must be</=
p><p class=3DMsoNormal>>> > consistent in use of MDB_WRITEMAP, but=
the documentation already makes</p><p class=3DMsoNormal>>> this</p><=
p class=3DMsoNormal>>> > explicit and clear.</p><p class=3DMsoNorm=
al>>> ></p><p class=3DMsoNormal>>> > I believe this can b=
e fixed by simply adding a check for MDB_WRITEMAP</p><p class=3DMsoNormal>&=
gt;> > in</p><p class=3DMsoNormal>>> the if</p><p class=3DMsoNo=
rmal>>> > statement that calls SetFilePointer:</p><p class=3DMsoNo=
rmal>>> ></p><p class=3DMsoNormal>>> >=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (!(fl=
ags & MDB_WRITEMAP) &&</p><p class=3DMsoNormal>>> >=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 (SetFilePointer(env->me_fd,</p><p class=3DMsoNormal>>> size=
lo, &sizehi, 0)</p><p class=3DMsoNormal>>> > !=3D (DWORD)sizel=
o</p><p class=3DMsoNormal>>> >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0 || !SetEndOfFile(env->me_fd)</p><p class=3DMsoNorm=
al>>> >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
|| SetFilePointer(env->me_fd, 0, NULL, 0) !=3D 0))</p><p class=3DMsoNor=
mal>>> >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
return ErrCode();</p><p class=3DMsoNormal>>> ></p><p class=3DMsoN=
ormal>>> > The attached URL has the change as a patch/diff as appl=
ied to our node</p><p class=3DMsoNormal>>> package.</p><p class=3DMso=
Normal>>> ></p><p class=3DMsoNormal>>> > I am certainly h=
appy to just keep this change on our own branches.</p><p class=3DMsoNormal>=
>> > There</p><p class=3DMsoNormal>>> may be</p><p class=3DM=
soNormal>>> > nuances of this that I might not be aware of, but it=
seems to be</p><p class=3DMsoNormal>>> > working</p><p class=3DMs=
oNormal>>> great</p><p class=3DMsoNormal>>> > for us and I h=
ave tested this with MDB_WRITEMAP enabled and disabled.</p><p class=3DMsoNo=
rmal>>> > So</p><p class=3DMsoNormal>>> I</p><p class=3DMsoN=
ormal>>> > thought I would offer/suggest this change, as it seems =
like it is</p><p class=3DMsoNormal>>> > straightforward change to =
improve stability. Thank you!</p><p class=3DMsoNormal>>> ></p><p c=
lass=3DMsoNormal>>> ></p><p class=3DMsoNormal>>><o:p> <=
/o:p></p><p class=3DMsoNormal>>><o:p> </o:p></p><p class=3DMsoNo=
rmal>>> --</p><p class=3DMsoNormal>>>=C2=A0=C2=A0 -- Howard Chu=
</p><p class=3DMsoNormal>>>=C2=A0=C2=A0 CTO, Symas Corp.=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 http://www.symas.com</p><p=
class=3DMsoNormal>>>=C2=A0=C2=A0 Director, Highland Sun=C2=A0=C2=A0=
=C2=A0=C2=A0 http://highlandsun.com/hyc/</p><p class=3DMsoNormal>>>=
=C2=A0=C2=A0 Chief Architect, OpenLDAP=C2=A0 http://www.openldap.org/projec=
t/</p><p class=3DMsoNormal>>><o:p> </o:p></p><p class=3DMsoNorma=
l>><o:p> </o:p></p><p class=3DMsoNormal>> --0000000000000dacee05=
826a34a9</p><p class=3DMsoNormal>> Content-Type: text/html; charset=3D&q=
uot;UTF-8"</p><p class=3DMsoNormal>> Content-Transfer-Encoding: quo=
ted-printable</p><p class=3DMsoNormal>><o:p> </o:p></p><p class=3DM=
soNormal>> <div dir=3D3D"ltr"><div dir=3D3D"ltr&=
quot;>Thank you, I appreciate it! Do you know</p><p class=3DMsoNormal>&g=
t; h=3D ow often or when this will get into the github mirror <a</p><p c=
lass=3DMsoNormal>> href=3D3D"https://gi=3D</p><p class=3DMsoNormal>=
> thub.com/LMDB/lmdb/commits/mdb.RE/0.9">https://github.com/LMDB=
/lmdb/commi</p><p class=3DMsoNormal>> ts=3D /mdb.RE/0.9</a> (or if=
I am misunderstanding how the branches are</p><p class=3DMsoNormal>> sy=
nced)?<=3D /div></div><br><div class=3D3D"gmail_q=
uote"><div dir=3D3D"ltr"</p><p class=3DMsoNormal>> =
class=3D3D"gmail_a=3D ttr">On Sun, Feb 10, 2019 at 3:14 AM How=
ard Chu &lt;<a</p><p class=3DMsoNormal>> href=3D3D"mailto:hy=
c@s=3D ymas.com">hyc(a)symas.com</a>&gt;</p><p class=3DMsoN=
ormal>> wrote:<br></div><blockquote class=3D3D"gmail=
_=3D quote" style=3D3D"margin:0px</p><p class=3DMsoNormal>> 0p=
x 0px 0.8ex;border-left:1px solid rgb(204,204,=3D</p><p class=3DMsoNormal>&=
gt; 204);padding-left:1ex"><a href=3D3D"mailto:kriszyp@gmai=
l.com"</p><p class=3DMsoNormal>> target=3D3D"_blan=3D k"&=
gt;kriszyp(a)gmail.com</a> wrote:<br></p><p class=3DMsoNormal>>=
; &gt; Full_Name: Kristopher William Zyp<br></p><p class=3DMsoNor=
mal>> &gt; Version: LMDB 0.9.23<br></p><p class=3DMsoNormal>&g=
t; &gt; OS: Windows Server 2012 R2, Windows 10<br></p><p class=3D=
MsoNormal>> &gt; URL: <a</p><p class=3DMsoNormal>> href=3D3D&q=
uot;https://github.com/kriszyp/node-lmdb/commit/6df903907f=3D</p><p class=
=3DMsoNormal>> 5516320e9a8afce45bd32ab4e8e1f2.patch" rel=3D3D"=
noreferrer"</p><p class=3DMsoNormal>> target=3D3D"_blank"=
>=3D</p><p class=3DMsoNormal>> https://github.com/kriszyp/node-lmdb/c=
ommit/6df903907f5516320e9a8afce45bd</p><p class=3DMsoNormal>> 32=3D ab4e=
8e1f2.patch</a><br></p><p class=3DMsoNormal>> &gt; Submi=
ssion from: (NULL) (71.199.6.148)<br></p><p class=3DMsoNormal>> &a=
mp;gt; <br></p><p class=3DMsoNormal>> <br></p><p class=3DMso=
Normal>> Thanks for the report and patch, added to mdb.RE/0.9<br><=
/p><p class=3DMsoNormal>> <br></p><p class=3DMsoNormal>> &g=
t; Calling mdb_env_set_mapsize to increase the map size, when a DB is</p><p=
class=3DMsoNormal>> als=3D o in use<br></p><p class=3DMsoNormal>&=
gt; &gt; by other processes, when MDB_WRITEMAP is enabled (and the db f=
ile</p><p class=3DMsoNormal>> has =3D been<br></p><p class=3DMsoNo=
rmal>> &gt; opened with PAGE_READWRITE access), on Windows, will occ=
asionally</p><p class=3DMsoNormal>> (see=3D ms like<br></p><p clas=
s=3DMsoNormal>> &gt; about 1/100 attempts fails) produce an error &a=
mp;quot;The requested</p><p class=3DMsoNormal>> opera=3D tion cannot be&=
lt;br></p><p class=3DMsoNormal>> &gt; performed on a file with a =
user-mapped section open&quot;, or</p><p class=3DMsoNormal>> segfaul=
t=3D s. The error<br></p><p class=3DMsoNormal>> &gt; occurs in=
the SetFilePointer (or SetEndOfFile) call in mdb_env_map</p><p class=3DMso=
Normal>> tha=3D t is<br></p><p class=3DMsoNormal>> &gt; per=
formed to increase the allocated file size to the map size, prior</p><p cla=
ss=3DMsoNormal>> t=3D o<br></p><p class=3DMsoNormal>> &gt; =
CreateFileMapping.<br></p><p class=3DMsoNormal>> &gt; <br&g=
t;</p><p class=3DMsoNormal>> &gt; As it turns out this is pretty eas=
y to solve, because manually</p><p class=3DMsoNormal>> expandi=3D ng the=
<br></p><p class=3DMsoNormal>> &gt; file size is not necessary=
when calling CreateFileMapping with</p><p class=3DMsoNormal>> PAGE_RE=
=3D ADWRITE<br></p><p class=3DMsoNormal>> &gt; access, as Wind=
ows will automatically expand the file size for us,</p><p class=3DMsoNormal=
>> whe=3D n opened<br></p><p class=3DMsoNormal>> &gt; with =
the page write access enabled. Of course, this means all</p><p class=3DMsoN=
ormal>> processe=3D s must be<br></p><p class=3DMsoNormal>> &am=
p;gt; consistent in use of MDB_WRITEMAP, but the documentation already</p><=
p class=3DMsoNormal>> makes=3D=C2=A0 this<br></p><p class=3DMsoNor=
mal>> &gt; explicit and clear.<br></p><p class=3DMsoNormal>>=
; &gt; <br></p><p class=3DMsoNormal>> &gt; I believe this =
can be fixed by simply adding a check for</p><p class=3DMsoNormal>> MDB_=
WRITEMAP =3D in the if<br></p><p class=3DMsoNormal>> &gt; stat=
ement that calls SetFilePointer:<br></p><p class=3DMsoNormal>> &am=
p;gt; <br></p><p class=3DMsoNormal>> &gt;=3DC2=3DA0 =3DC2=3DA0=
=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0if (!(fla=
gs</p><p class=3DMsoNormal>> &amp=3D ; MDB_WRITEMAP) &amp;&a=
mp; (SetFilePointer(env-&gt;me_fd, sizelo,</p><p class=3DMsoNormal>>=
&amp;size=3D hi, 0)<br></p><p class=3DMsoNormal>> &gt; !=
=3D3D (DWORD)sizelo<br></p><p class=3DMsoNormal>> &gt;=3DC2=3D=
A0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=
=3DA0 =3DC2=3DA0 =3DC2=3DA0</p><p class=3DMsoNormal>> =3D =3DC2=3DA0 =3D=
C2=3DA0|| !SetEndOfFile(env-&gt;me_fd)<br></p><p class=3DMsoNorma=
l>> &gt;=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=
=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0</p><p class=3DMsoNormal>&=
gt; =3D =3DC2=3DA0 =3DC2=3DA0|| SetFilePointer(env-&gt;me_fd, 0, NULL, =
0) !=3D3D 0))<br></p><p class=3DMsoNormal>> &gt;=3DC2=3DA0 =3D=
C2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =
=3DC2=3DA0 =3DC2=3DA0</p><p class=3DMsoNormal>> =3D =3DC2=3DA0 =3DC2=3DA=
0return ErrCode();<br></p><p class=3DMsoNormal>> &gt; <br&g=
t;</p><p class=3DMsoNormal>> &gt; The attached URL has the change as=
a patch/diff as applied to our</p><p class=3DMsoNormal>> node=3D=C2=A0 =
package.<br></p><p class=3DMsoNormal>> &gt; <br></p><p c=
lass=3DMsoNormal>> &gt; I am certainly happy to just keep this chang=
e on our own branches.</p><p class=3DMsoNormal>> The=3D re may be<br&=
gt;</p><p class=3DMsoNormal>> &gt; nuances of this that I might not =
be aware of, but it seems to be</p><p class=3DMsoNormal>> worki=3D ng gr=
eat<br></p><p class=3DMsoNormal>> &gt; for us and I have teste=
d this with MDB_WRITEMAP enabled and</p><p class=3DMsoNormal>> disabled.=
=3D So I<br></p><p class=3DMsoNormal>> &gt; thought I would o=
ffer/suggest this change, as it seems like it is<br></p><p class=3DMs=
oNormal>> &gt; straightforward change to improve stability. Thank yo=
u!<br></p><p class=3DMsoNormal>> &gt; <br></p><p class=
=3DMsoNormal>> &gt; <br></p><p class=3DMsoNormal>> <br&g=
t;</p><p class=3DMsoNormal>> <br></p><p class=3DMsoNormal>> -- =
<br></p><p class=3DMsoNormal>> =3DC2=3DA0 -- Howard Chu<br><=
/p><p class=3DMsoNormal>> =3DC2=3DA0 CTO, Symas Corp.=3DC2=3DA0 =3DC2=3D=
A0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0<a</p><p class=3DMsoNormal=
>> href=3D3D"=3D http://www.symas.com" rel=3D3D"noreferre=
r"</p><p class=3DMsoNormal>> target=3D3D"_blank">http:=
//www.symas=3D .com</a><br></p><p class=3DMsoNormal>> =3DC2=
=3DA0 Director, Highland Sun=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0<a</p><p cla=
ss=3DMsoNormal>> href=3D3D"http://highland=3D sun.com/hyc/" re=
l=3D3D"noreferrer"</p><p class=3DMsoNormal>> target=3D3D"=
_blank">http://highlandsun.com/h=3D yc/</a><br></p><p c=
lass=3DMsoNormal>> =3DC2=3DA0 Chief Architect, OpenLDAP=3DC2=3DA0 <a<=
/p><p class=3DMsoNormal>> href=3D3D"http://www.openldap.org/p=3D ro=
ject/" rel=3D3D"noreferrer"</p><p class=3DMsoNormal>> tar=
get=3D3D"_blank">http://www.openldap.org/proje=3D ct/</a>=
;<br></p><p class=3DMsoNormal>> </blockquote></div></p=
><p class=3DMsoNormal>><o:p> </o:p></p><p class=3DMsoNormal>> --=
0000000000000dacee05826a34a9--</p><p class=3DMsoNormal>><o:p> </o:p=
></p><p class=3DMsoNormal>><o:p> </o:p></p><p class=3DMsoNormal>>=
;<o:p> </o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3D=
MsoNormal><o:p> </o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p=
class=3DMsoNormal>--</p><p class=3DMsoNormal><o:p> </o:p></p><p class=
=3DMsoNormal>Quanah Gibson-Mount</p><p class=3DMsoNormal>Product Architect<=
/p><p class=3DMsoNormal>Symas Corporation</p><p class=3DMsoNormal>Packaged,=
certified, and supported LDAP solutions powered by OpenLDAP:</p><p class=
=3DMsoNormal><http://www.symas.com></p><p class=3DMsoNormal><o:p>&nbs=
p;</o:p></p><p class=3DMsoNormal><o:p> </o:p></p></div></body></html>=
--_36998F79-0E7E-483F-8FFD-6401D532832A_--
4 years, 3 months
Re: (ITS#8975) Calling mdb_env_set_mapsize on a shared DB fails sometimes
by quanah＠symas.com
Hi,
It is literally the first set of commits listed at that URL, so it's
clearly already there.
Hope that helps!
--Quanah
--On Thursday, February 21, 2019 4:45 PM +0000 kriszyp(a)gmail.com wrote:
> --0000000000000dacee05826a34a9
> Content-Type: text/plain; charset="UTF-8"
>
> Thank you, I appreciate it! Do you know how often or when this will get
> into the github mirror https://github.com/LMDB/lmdb/commits/mdb.RE/0.9 (or
> if I am misunderstanding how the branches are synced)?
>
> On Sun, Feb 10, 2019 at 3:14 AM Howard Chu <hyc(a)symas.com> wrote:
>
>> kriszyp(a)gmail.com wrote:
>> > Full_Name: Kristopher William Zyp
>> > Version: LMDB 0.9.23
>> > OS: Windows Server 2012 R2, Windows 10
>> > URL:
>> https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45b
>> d32ab4e8e1f2.patch
>> > Submission from: (NULL) (71.199.6.148)
>> >
>>
>> Thanks for the report and patch, added to mdb.RE/0.9
>>
>> > Calling mdb_env_set_mapsize to increase the map size, when a DB is also
>> in use
>> > by other processes, when MDB_WRITEMAP is enabled (and the db file has
>> been
>> > opened with PAGE_READWRITE access), on Windows, will occasionally
>> > (seems
>> like
>> > about 1/100 attempts fails) produce an error "The requested operation
>> cannot be
>> > performed on a file with a user-mapped section open", or segfaults. The
>> error
>> > occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map that
>> is
>> > performed to increase the allocated file size to the map size, prior to
>> > CreateFileMapping.
>> >
>> > As it turns out this is pretty easy to solve, because manually
>> > expanding
>> the
>> > file size is not necessary when calling CreateFileMapping with
>> PAGE_READWRITE
>> > access, as Windows will automatically expand the file size for us, when
>> opened
>> > with the page write access enabled. Of course, this means all processes
>> must be
>> > consistent in use of MDB_WRITEMAP, but the documentation already makes
>> this
>> > explicit and clear.
>> >
>> > I believe this can be fixed by simply adding a check for MDB_WRITEMAP
>> > in
>> the if
>> > statement that calls SetFilePointer:
>> >
>> > if (!(flags & MDB_WRITEMAP) &&
>> > (SetFilePointer(env->me_fd,
>> sizelo, &sizehi, 0)
>> > != (DWORD)sizelo
>> > || !SetEndOfFile(env->me_fd)
>> > || SetFilePointer(env->me_fd, 0, NULL, 0) != 0))
>> > return ErrCode();
>> >
>> > The attached URL has the change as a patch/diff as applied to our node
>> package.
>> >
>> > I am certainly happy to just keep this change on our own branches.
>> > There
>> may be
>> > nuances of this that I might not be aware of, but it seems to be
>> > working
>> great
>> > for us and I have tested this with MDB_WRITEMAP enabled and disabled.
>> > So
>> I
>> > thought I would offer/suggest this change, as it seems like it is
>> > straightforward change to improve stability. Thank you!
>> >
>> >
>>
>>
>> --
>> -- Howard Chu
>> CTO, Symas Corp. http://www.symas.com
>> Director, Highland Sun http://highlandsun.com/hyc/
>> Chief Architect, OpenLDAP http://www.openldap.org/project/
>>
>
> --0000000000000dacee05826a34a9
> Content-Type: text/html; charset="UTF-8"
> Content-Transfer-Encoding: quoted-printable
>
> <div dir=3D"ltr"><div dir=3D"ltr">Thank you, I appreciate it! Do you know
> h= ow often or when this will get into the github mirror <a
> href=3D"https://gi=
> thub.com/LMDB/lmdb/commits/mdb.RE/0.9">https://github.com/LMDB/lmdb/commi
> ts= /mdb.RE/0.9</a> (or if I am misunderstanding how the branches are
> synced)?<= /div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr"
> class=3D"gmail_a= ttr">On Sun, Feb 10, 2019 at 3:14 AM Howard Chu <<a
> href=3D"mailto:hyc@s= ymas.com">hyc(a)symas.com</a>>
> wrote:<br></div><blockquote class=3D"gmail_= quote" style=3D"margin:0px
> 0px 0px 0.8ex;border-left:1px solid rgb(204,204,=
> 204);padding-left:1ex"><a href=3D"mailto:kriszyp@gmail.com"
> target=3D"_blan= k">kriszyp(a)gmail.com</a> wrote:<br>
> > Full_Name: Kristopher William Zyp<br>
> > Version: LMDB 0.9.23<br>
> > OS: Windows Server 2012 R2, Windows 10<br>
> > URL: <a
> href=3D"https://github.com/kriszyp/node-lmdb/commit/6df903907f=
> 5516320e9a8afce45bd32ab4e8e1f2.patch" rel=3D"noreferrer"
> target=3D"_blank">=
> https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45bd
> 32= ab4e8e1f2.patch</a><br>
> > Submission from: (NULL) (71.199.6.148)<br>
> > <br>
> <br>
> Thanks for the report and patch, added to mdb.RE/0.9<br>
> <br>
> > Calling mdb_env_set_mapsize to increase the map size, when a DB is
> als= o in use<br>
> > by other processes, when MDB_WRITEMAP is enabled (and the db file
> has = been<br>
> > opened with PAGE_READWRITE access), on Windows, will occasionally
> (see= ms like<br>
> > about 1/100 attempts fails) produce an error "The requested
> opera= tion cannot be<br>
> > performed on a file with a user-mapped section open", or
> segfault= s. The error<br>
> > occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map
> tha= t is<br>
> > performed to increase the allocated file size to the map size, prior
> t= o<br>
> > CreateFileMapping.<br>
> > <br>
> > As it turns out this is pretty easy to solve, because manually
> expandi= ng the<br>
> > file size is not necessary when calling CreateFileMapping with
> PAGE_RE= ADWRITE<br>
> > access, as Windows will automatically expand the file size for us,
> whe= n opened<br>
> > with the page write access enabled. Of course, this means all
> processe= s must be<br>
> > consistent in use of MDB_WRITEMAP, but the documentation already
> makes= this<br>
> > explicit and clear.<br>
> > <br>
> > I believe this can be fixed by simply adding a check for
> MDB_WRITEMAP = in the if<br>
> > statement that calls SetFilePointer:<br>
> > <br>
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (!(flags
> &= ; MDB_WRITEMAP) && (SetFilePointer(env->me_fd, sizelo,
> &size= hi, 0)<br>
> > !=3D (DWORD)sizelo<br>
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> = =C2=A0 =C2=A0|| !SetEndOfFile(env->me_fd)<br>
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> = =C2=A0 =C2=A0|| SetFilePointer(env->me_fd, 0, NULL, 0) !=3D 0))<br>
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> = =C2=A0 =C2=A0return ErrCode();<br>
> > <br>
> > The attached URL has the change as a patch/diff as applied to our
> node= package.<br>
> > <br>
> > I am certainly happy to just keep this change on our own branches.
> The= re may be<br>
> > nuances of this that I might not be aware of, but it seems to be
> worki= ng great<br>
> > for us and I have tested this with MDB_WRITEMAP enabled and
> disabled. = So I<br>
> > thought I would offer/suggest this change, as it seems like it is<br>
> > straightforward change to improve stability. Thank you!<br>
> > <br>
> > <br>
> <br>
> <br>
> -- <br>
> =C2=A0 -- Howard Chu<br>
> =C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a
> href=3D"= http://www.symas.com" rel=3D"noreferrer"
> target=3D"_blank">http://www.symas= .com</a><br>
> =C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a
> href=3D"http://highland= sun.com/hyc/" rel=3D"noreferrer"
> target=3D"_blank">http://highlandsun.com/h= yc/</a><br>
> =C2=A0 Chief Architect, OpenLDAP=C2=A0 <a
> href=3D"http://www.openldap.org/p= roject/" rel=3D"noreferrer"
> target=3D"_blank">http://www.openldap.org/proje= ct/</a><br>
> </blockquote></div>
>
> --0000000000000dacee05826a34a9--
>
>
>
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
4 years, 3 months
Re: (ITS#8975) Calling mdb_env_set_mapsize on a shared DB fails sometimes
by kriszyp＠gmail.com
--0000000000000dacee05826a34a9
Content-Type: text/plain; charset="UTF-8"
Thank you, I appreciate it! Do you know how often or when this will get
into the github mirror https://github.com/LMDB/lmdb/commits/mdb.RE/0.9 (or
if I am misunderstanding how the branches are synced)?
On Sun, Feb 10, 2019 at 3:14 AM Howard Chu <hyc(a)symas.com> wrote:
> kriszyp(a)gmail.com wrote:
> > Full_Name: Kristopher William Zyp
> > Version: LMDB 0.9.23
> > OS: Windows Server 2012 R2, Windows 10
> > URL:
> https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45bd...
> > Submission from: (NULL) (71.199.6.148)
> >
>
> Thanks for the report and patch, added to mdb.RE/0.9
>
> > Calling mdb_env_set_mapsize to increase the map size, when a DB is also
> in use
> > by other processes, when MDB_WRITEMAP is enabled (and the db file has
> been
> > opened with PAGE_READWRITE access), on Windows, will occasionally (seems
> like
> > about 1/100 attempts fails) produce an error "The requested operation
> cannot be
> > performed on a file with a user-mapped section open", or segfaults. The
> error
> > occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map that
> is
> > performed to increase the allocated file size to the map size, prior to
> > CreateFileMapping.
> >
> > As it turns out this is pretty easy to solve, because manually expanding
> the
> > file size is not necessary when calling CreateFileMapping with
> PAGE_READWRITE
> > access, as Windows will automatically expand the file size for us, when
> opened
> > with the page write access enabled. Of course, this means all processes
> must be
> > consistent in use of MDB_WRITEMAP, but the documentation already makes
> this
> > explicit and clear.
> >
> > I believe this can be fixed by simply adding a check for MDB_WRITEMAP in
> the if
> > statement that calls SetFilePointer:
> >
> > if (!(flags & MDB_WRITEMAP) && (SetFilePointer(env->me_fd,
> sizelo, &sizehi, 0)
> > != (DWORD)sizelo
> > || !SetEndOfFile(env->me_fd)
> > || SetFilePointer(env->me_fd, 0, NULL, 0) != 0))
> > return ErrCode();
> >
> > The attached URL has the change as a patch/diff as applied to our node
> package.
> >
> > I am certainly happy to just keep this change on our own branches. There
> may be
> > nuances of this that I might not be aware of, but it seems to be working
> great
> > for us and I have tested this with MDB_WRITEMAP enabled and disabled. So
> I
> > thought I would offer/suggest this change, as it seems like it is
> > straightforward change to improve stability. Thank you!
> >
> >
>
>
> --
> -- Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
>
--0000000000000dacee05826a34a9
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr">Thank you, I appreciate it! Do you know h=
ow often or when this will get into the github mirror <a href=3D"https://gi=
thub.com/LMDB/lmdb/commits/mdb.RE/0.9">https://github.com/LMDB/lmdb/commits=
/mdb.RE/0.9</a> (or if I am misunderstanding how the branches are synced)?<=
/div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_a=
ttr">On Sun, Feb 10, 2019 at 3:14 AM Howard Chu <<a href=3D"mailto:hyc@s=
ymas.com">hyc(a)symas.com</a>> wrote:<br></div><blockquote class=3D"gmail_=
quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,=
204);padding-left:1ex"><a href=3D"mailto:kriszyp@gmail.com" target=3D"_blan=
k">kriszyp(a)gmail.com</a> wrote:<br>
> Full_Name: Kristopher William Zyp<br>
> Version: LMDB 0.9.23<br>
> OS: Windows Server 2012 R2, Windows 10<br>
> URL: <a href=3D"https://github.com/kriszyp/node-lmdb/commit/6df903907f=
5516320e9a8afce45bd32ab4e8e1f2.patch" rel=3D"noreferrer" target=3D"_blank">=
https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45bd32=
ab4e8e1f2.patch</a><br>
> Submission from: (NULL) (71.199.6.148)<br>
> <br>
<br>
Thanks for the report and patch, added to mdb.RE/0.9<br>
<br>
> Calling mdb_env_set_mapsize to increase the map size, when a DB is als=
o in use<br>
> by other processes, when MDB_WRITEMAP is enabled (and the db file has =
been<br>
> opened with PAGE_READWRITE access), on Windows, will occasionally (see=
ms like<br>
> about 1/100 attempts fails) produce an error "The requested opera=
tion cannot be<br>
> performed on a file with a user-mapped section open", or segfault=
s. The error<br>
> occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map tha=
t is<br>
> performed to increase the allocated file size to the map size, prior t=
o<br>
> CreateFileMapping.<br>
> <br>
> As it turns out this is pretty easy to solve, because manually expandi=
ng the<br>
> file size is not necessary when calling CreateFileMapping with PAGE_RE=
ADWRITE<br>
> access, as Windows will automatically expand the file size for us, whe=
n opened<br>
> with the page write access enabled. Of course, this means all processe=
s must be<br>
> consistent in use of MDB_WRITEMAP, but the documentation already makes=
this<br>
> explicit and clear.<br>
> <br>
> I believe this can be fixed by simply adding a check for MDB_WRITEMAP =
in the if<br>
> statement that calls SetFilePointer:<br>
> <br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (!(flags &=
; MDB_WRITEMAP) && (SetFilePointer(env->me_fd, sizelo, &size=
hi, 0)<br>
> !=3D (DWORD)sizelo<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0|| !SetEndOfFile(env->me_fd)<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0|| SetFilePointer(env->me_fd, 0, NULL, 0) !=3D 0))<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0return ErrCode();<br>
> <br>
> The attached URL has the change as a patch/diff as applied to our node=
package.<br>
> <br>
> I am certainly happy to just keep this change on our own branches. The=
re may be<br>
> nuances of this that I might not be aware of, but it seems to be worki=
ng great<br>
> for us and I have tested this with MDB_WRITEMAP enabled and disabled. =
So I<br>
> thought I would offer/suggest this change, as it seems like it is<br>
> straightforward change to improve stability. Thank you!<br>
> <br>
> <br>
<br>
<br>
-- <br>
=C2=A0 -- Howard Chu<br>
=C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"=
http://www.symas.com" rel=3D"noreferrer" target=3D"_blank">http://www.symas=
.com</a><br>
=C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a href=3D"http://highland=
sun.com/hyc/" rel=3D"noreferrer" target=3D"_blank">http://highlandsun.com/h=
yc/</a><br>
=C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http://www.openldap.org/p=
roject/" rel=3D"noreferrer" target=3D"_blank">http://www.openldap.org/proje=
ct/</a><br>
</blockquote></div>
--0000000000000dacee05826a34a9--
4 years, 3 months
