Jeremiah Morrill wrote:
This sort of coincides with ITS issue 8347 I just reported.
With ldmb VL32, I’ve noticed that it can eat up a lot of memory space. Sometimes ~500mb for an 800mb db after reading every record, one by one using the same readonly transaction/cursor. I found that modifying MDB_ERPAGE_SIZE from 16384 to 256 brought virtual memory usage down substantially (~30megs). I have not noticed any adverse effects from this change or MDB_MAP_FULL. The mapped address spaces to seem to get deallocated on the close of the environment, not the transaction, which sounds like its being cached in the env->me_rpages for reasons explained in mdb_rpage_get comments.
Is the mem map cache supposed to be this aggressive by design,
Yes. It is designed to use up to 1GB.
or just because
this area of the code is so new...or simply a usage error on my part? If it’s by design, could there be a consideration to make MDB_ERPAGE_SIZE a #define’able value in the public lmdb.h so one could tweak the behavior?
You're welcome to tweak it in whatever way you see fit. MDB_VL32 is only ever used as a captive library, not linked in from a system package. How you build it is entirely up to you.