From kriszyp@gmail.com Thu Feb 21 16:45:45 2019 From: kriszyp@gmail.com To: openldap-bugs@openldap.org Subject: Re: (ITS#8975) Calling mdb_env_set_mapsize on a shared DB fails sometimes Date: Thu, 21 Feb 2019 16:45:43 +0000 Message-ID: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6445727098354042906==" --===============6445727098354042906== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable --0000000000000dacee05826a34a9 Content-Type: text/plain; charset=3D"UTF-8" Thank you, I appreciate it! Do you know how often or when this will get into the github mirror https://github.com/LMDB/lmdb/commits/mdb.RE/0.9 (or if I am misunderstanding how the branches are synced)? On Sun, Feb 10, 2019 at 3:14 AM Howard Chu wrote: > kriszyp(a)gmail.com wrote: > > Full_Name: Kristopher William Zyp > > Version: LMDB 0.9.23 > > OS: Windows Server 2012 R2, Windows 10 > > URL: > https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45bd32= ab4e8e1f2.patch > > Submission from: (NULL) (71.199.6.148) > > > > Thanks for the report and patch, added to mdb.RE/0.9 > > > Calling mdb_env_set_mapsize to increase the map size, when a DB is also > in use > > by other processes, when MDB_WRITEMAP is enabled (and the db file has > been > > opened with PAGE_READWRITE access), on Windows, will occasionally (seems > like > > about 1/100 attempts fails) produce an error "The requested operation > cannot be > > performed on a file with a user-mapped section open", or segfaults. The > error > > occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map that > is > > performed to increase the allocated file size to the map size, prior to > > CreateFileMapping. > > > > As it turns out this is pretty easy to solve, because manually expanding > the > > file size is not necessary when calling CreateFileMapping with > PAGE_READWRITE > > access, as Windows will automatically expand the file size for us, when > opened > > with the page write access enabled. Of course, this means all processes > must be > > consistent in use of MDB_WRITEMAP, but the documentation already makes > this > > explicit and clear. > > > > I believe this can be fixed by simply adding a check for MDB_WRITEMAP in > the if > > statement that calls SetFilePointer: > > > > if (!(flags & MDB_WRITEMAP) && (SetFilePointer(env->me_fd, > sizelo, &sizehi, 0) > > !=3D (DWORD)sizelo > > || !SetEndOfFile(env->me_fd) > > || SetFilePointer(env->me_fd, 0, NULL, 0) !=3D 0)) > > return ErrCode(); > > > > The attached URL has the change as a patch/diff as applied to our node > package. > > > > I am certainly happy to just keep this change on our own branches. There > may be > > nuances of this that I might not be aware of, but it seems to be working > great > > for us and I have tested this with MDB_WRITEMAP enabled and disabled. So > I > > thought I would offer/suggest this change, as it seems like it is > > straightforward change to improve stability. Thank you! > > > > > > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ > --0000000000000dacee05826a34a9 Content-Type: text/html; charset=3D"UTF-8" Content-Transfer-Encoding: quoted-printable
Thank you, I appreciate it! Do you know= h=3D ow often or when this will get into the github mirror https://github.com/LMDB/lmdb/commits=3D /mdb.RE/0.9 (or if I am misunderstanding how the branches are synced)?<=3D /div>

On Sun, Feb 10, 2019 at 3:14 AM Howard Chu <hyc(a)symas.com> wrote:
kriszyp(a)gmail.com wrote:
> Full_Name: Kristopher William Zyp
> Version: LMDB 0.9.23
> OS: Windows Server 2012 R2, Windows 10
> URL: =3D https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45bd32=3D ab4e8e1f2.patch
> Submission from: (NULL) (71.199.6.148)
>

Thanks for the report and patch, added to mdb.RE/0.9

> Calling mdb_env_set_mapsize to increase the map size, when a DB is als=3D o in use
> by other processes, when MDB_WRITEMAP is enabled (and the db file has =3D been
> opened with PAGE_READWRITE access), on Windows, will occasionally (see=3D ms like
> about 1/100 attempts fails) produce an error "The requested opera=3D tion cannot be
> performed on a file with a user-mapped section open", or segfault=3D s. The error
> occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map tha=3D t is
> performed to increase the allocated file size to the map size, prior t=3D o
> CreateFileMapping.
>
> As it turns out this is pretty easy to solve, because manually expandi=3D ng the
> file size is not necessary when calling CreateFileMapping with PAGE_RE=3D ADWRITE
> access, as Windows will automatically expand the file size for us, whe=3D n opened
> with the page write access enabled. Of course, this means all processe=3D s must be
> consistent in use of MDB_WRITEMAP, but the documentation already makes=3D this
> explicit and clear.
>
> I believe this can be fixed by simply adding a check for MDB_WRITEMAP =3D in the if
> statement that calls SetFilePointer:
>
>=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2= =3DA0 =3DC2=3DA0if (!(flags &=3D ; MDB_WRITEMAP) && (SetFilePointer(env->me_fd, sizelo, &size=3D hi, 0)
> !=3D3D (DWORD)sizelo
>=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2= =3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3D =3DC2=3DA0 =3DC2=3DA0|| !SetEndOfFile(env->me_fd)
>=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2= =3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3D =3DC2=3DA0 =3DC2=3DA0|| SetFilePointer(env->me_fd, 0, NULL, 0) !=3D3D 0))<= br> >=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2= =3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3D =3DC2=3DA0 =3DC2=3DA0return ErrCode();
>
> The attached URL has the change as a patch/diff as applied to our node=3D package.
>
> I am certainly happy to just keep this change on our own branches. The=3D re may be
> nuances of this that I might not be aware of, but it seems to be worki=3D ng great
> for us and I have tested this with MDB_WRITEMAP enabled and disabled. =3D So I
> thought I would offer/suggest this change, as it seems like it is
> straightforward change to improve stability. Thank you!
>
>


--
=3DC2=3DA0 -- Howard Chu
=3DC2=3DA0 CTO, Symas Corp.=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2= =3DA0 =3DC2=3DA0http://www.sym= as=3D .com
=3DC2=3DA0 Director, Highland Sun=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0http://highlandsun.com= /h=3D yc/
=3DC2=3DA0 Chief Architect, OpenLDAP=3DC2=3DA0 http://www.openldap.org/pro= je=3D ct/
--0000000000000dacee05826a34a9-- --===============6445727098354042906==--