Howard Chu hyc@symas.com schrieb am 15.01.2014 um 23:10 in Nachricht
Luc Vlaming wrote:
Hi,
Currently I am creating support for using LMDB as a new storage backend for one of our products. At the moment I am testing import bulk data into lmdb using transactions
that
span a single record of 10MB. The total db size afterwards is 5GB. I also tested with records of 1MB.
I noticed a very odd thing: when using the MDB_WRITEMAP option, memory usage grows very quickly and linear with the amount of data stored into the database. (memory usage ends up a bit higher than 5GB). when not using
Maybe for the future make a difference between virtual memory usage and real (resident) memory usage. Especially for Linux this makes a big difference, because a malloc(1GB) actually does not consume any memory until it is actually used.
There's also the "pmap" Utility that can show the detailed difference. For example my small (bdb) slapd has: # pmap 3668 3668: slapd START SIZE RSS PSS DIRTY SWAP PERM MAPPING [...] 00007f601a7f4000 8192K 120K 120K 120K 0K rw-p [anon] [...] 00007f603db4c000 18320K 184K 184K 84K 0K rw-s /var/lib/ldap/__db.003 [...] Total: 808004K 29768K 28657K 27016K 32040K
So of 800MB virtual memory there is only 30MB actually in use...
MDB_WRITEMAP, however, memory usage stays very low. Does anyone have a suggestion what might be wrong and what causes such different behaviour with and without using the memorymap option?
There is nothing wrong. It is simply writing to the shared memory map.
Off-topic: I can remember a statement of the late 80ies where a programmer claimed the 32-bit address space is so large that one does not have to care about garbage collection in virtual address space; just use new addresses. I think even with 64 bit one should always try not to waste address space.
Regards, Ulrich
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/