Hi
I'm using LMDB (original C interface) to store and read some data. However, I notice that my resident memory usage (RES in `htop`) keeps rising with each call to mdb_get with unique keys. It does not change if I do a repeated call to mdb_get with a key I've used before.
It looks like mdb_get is caching the reads, but I want to make sure what effect it is really having on my system. Is there some way to limit the amount of memory lmdb can use for this? Or is it something strange and should not be happening?
Thanks Rohit
Rohit Girdhar wrote:
Hi
I'm using LMDB (original C interface) to store and read some data. However, I notice that my resident memory usage (RES in `htop`) keeps rising with each call to mdb_get with unique keys. It does not change if I do a repeated call to mdb_get with a key I've used before.
It looks like mdb_get is caching the reads,
LMDB does no caching of its own. Read the docs. Learn how mmap works.
but I want to make sure what effect it is really having on my system. Is there some way to limit the amount of memory lmdb can use for this? Or is it something strange and should not be happening?
There's nothing strange going on. LMDB's memory use is entirely constrained by how much demand for memory other processes on the system are imposing, i.e. system memory pressure. If there is no pressure, LMDB will use all available RAM (or up to the configured mapsize, whichever is less). If there is memory pressure, LMDB will use less. Memory used by LMDB grows and shrinks dynamically and automatically with the system load.
Thanks, that helps!
On Fri, Jul 31, 2015 at 10:02 AM, Howard Chu hyc@symas.com wrote:
Rohit Girdhar wrote:
Hi
I'm using LMDB (original C interface) to store and read some data. However, I notice that my resident memory usage (RES in `htop`) keeps rising with each call to mdb_get with unique keys. It does not change if I do a repeated call to mdb_get with a key I've used before.
It looks like mdb_get is caching the reads,
LMDB does no caching of its own. Read the docs. Learn how mmap works.
but I want to make sure what
effect it is really having on my system. Is there some way to limit the amount of memory lmdb can use for this? Or is it something strange and should not be happening?
There's nothing strange going on. LMDB's memory use is entirely constrained by how much demand for memory other processes on the system are imposing, i.e. system memory pressure. If there is no pressure, LMDB will use all available RAM (or up to the configured mapsize, whichever is less). If there is memory pressure, LMDB will use less. Memory used by LMDB grows and shrinks dynamically and automatically with the system load.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
openldap-technical@openldap.org
-
Howard Chu -
Rohit Girdhar