--000000000000856dab058992e756 Content-Type: text/plain; charset="UTF-8"
The short answer is yes. Our mappings are now bigger than 1TiB (out of 2TiB total available). Also, it *seems* to be that it is just way faster for Linux to map things in order all at once. Additionally, we have some boxes that still have spinning disks, and sequential access is a couple orders of magnitude faster on those.
I haven't looked at the logic in the Linux pager, but it seems to be fairly aggressive about taking memory pages back from large memory maps, even though it doesn't strictly need to. Locking the memory is a bit of a sledgehammer, but in our experience it is an effective one.
On Thu, May 23, 2019 at 2:23 PM Howard Chu hyc@symas.com wrote:
github@nicwatson.org wrote:
Full_Name: Nic Watson Version: OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (73.132.68.128)
Goal: I'd like a clean way to get at the address of the data memory map in
LMDB.
MDB_envinfo.me_mapaddr only returns the map address if MAP_FIXED is used.
Current Workarounds:
- Use OS-specific mechanism to retrieve all memory maps (e.g.
/proc/<pid>/smaps).
- Defeat opaque handle and reach into the MDB_env struct directly and
grab the
me_map field.
Justification: In my current application, I notice a significant performance increase
if I
mlock the mapfile. In order to do that cleanly, I need the address of
the map.
That sounds odd. Is there a lot of memory pressure from other processes on the machine? Where is the performance loss or gain coming from?
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
--000000000000856dab058992e756 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">The short answer is yes. Our mappings are now bigger than = 1TiB (out of 2TiB total available). Also, it *seems* to be that it is just = way faster for Linux to map things in order all at once. Additionally, we h= ave some boxes that still have spinning disks, and sequential access is a c= ouple orders of magnitude faster on those.<div><br></div><div>I haven't= looked at the logic in the Linux pager, but it seems to be fairly aggressi= ve about taking memory pages back from large memory maps, even though it do= esn't strictly need to. Locking the memory is a bit of a sledgehammer, = but in our experience it is an effective one.</div></div><br><div class=3D"= gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Thu, May 23, 2019 at = 2:23 PM Howard Chu <<a href=3D"mailto:hyc@symas.com">hyc@symas.com</a>&g= t; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0p= x 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><a hre= f=3D"mailto:github@nicwatson.org" target=3D"_blank">github@nicwatson.org</a=
wrote:<br>
> Full_Name: Nic Watson<br> > Version: <br> > OS: Linux<br> > URL: <a href=3D"ftp://ftp.openldap.org/incoming/" rel=3D"noreferrer" t= arget=3D"_blank">ftp://ftp.openldap.org/incoming/</a><br> > Submission from: (NULL) (73.132.68.128)<br> > <br> > <br> > Goal:<br> > I'd like a clean way to get at the address of the data memory map = in LMDB.<br> > <br> > MDB_envinfo.me_mapaddr only returns the map address if MAP_FIXED is us= ed.<br> > <br> > Current Workarounds:<br> > * Use OS-specific mechanism to retrieve all memory maps (e.g.<br> > /proc/<pid>/smaps). <br> > * Defeat opaque handle and reach into the MDB_env struct directly and = grab the<br> > me_map field.<br> > <br> > Justification:<br> > In my current application, I notice a significant performance increase= if I<br> > mlock the mapfile.=C2=A0 In order to do that cleanly, I need the addre= ss of the map.<br> <br> That sounds odd. Is there a lot of memory pressure from other processes on = the machine?<br> Where is the performance loss or gain coming from?<br> <br> -- <br> =C2=A0 -- Howard Chu<br> =C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"= http://www.symas.com" rel=3D"noreferrer" target=3D"_blank">http://www.symas= .com</a><br> =C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a href=3D"http://highland= sun.com/hyc/" rel=3D"noreferrer" target=3D"_blank">http://highlandsun.com/h= yc/</a><br> =C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http://www.openldap.org/p= roject/" rel=3D"noreferrer" target=3D"_blank">http://www.openldap.org/proje= ct/</a><br> </blockquote></div>
--000000000000856dab058992e756--
-
nic＠nicwatson.org