Hey, I am using lmdb as a backend to a big database (and its awesome, thanks!). While trying to understand memory consumption of my process I came across something interesting I'd like to get an expert's opinion about.
I am seeing that the "Virtual" memory is quite large, larger than I expected, and it *might* be a problem in my case, I am being hit with lots of page ins/outs (I have a 500GB RAM machines).
When I lsof my process, I am seeing that each mdb file is appearing twice (I use many), eg: _ lsof -p 59709 | grep 86510580 test 59709 vaknin mem REG 8,17 625500160 86510580 /fs/test1.mdb test 59709 vaknin 239r REG 8,17 625500160 86510580 /fs/test1.mdb
I know that after mmaping a file, it is not needed to be kept open, and it seems it is in lmdb. I then tried to see if this actually uses memory, and I used memstat for this: _ memstat -p 59709 -w | grep 86510580 1221680k(1221680k): [08:11]:86510580 59709
From this it seems that the file is indeed using twice as much (virtual)
memory as it should.. 2*625500160 ≈ 1221680000
(this test was repeated for many different files, all have a factor of ≈2)
I know this might simply be an artifact and might not actually be a contributing reason for my swap ins/outs, but I wanted to hear what do you think about it? Why does lmdb keep the file handles open?
By the way the swap ins/outs are happening later in the process, after querying lmdb and processing its results.
Thanks!