Re: Re: lmdb memory usage when writing lots of data with memorymap option

>>> Howard Chu <hyc(a)symas.com&gt; schrieb am 15.01.2014 um 23:10 in Nachricht <52D7076B.3020003(a)symas.com&gt;: > Luc Vlaming wrote: >> Hi, >> >> Currently I am creating support for using LMDB as a new storage backend for >> one of our products. >> At the moment I am testing import bulk data into lmdb using transactions > that >> span a single record of 10MB. The total db size afterwards is 5GB. I also >> tested with records of 1MB. >> >> I noticed a very odd thing: when using the MDB_WRITEMAP option, memory usage >> grows very quickly and linear with the amount of data stored into the >> database. (memory usage ends up a bit higher than 5GB). when not using Maybe for the future make a difference between virtual memory usage and real (resident) memory usage. Especially for Linux this makes a big difference, because a malloc(1GB) actually does not consume any memory until it is actually used. There's also the "pmap" Utility that can show the detailed difference. For example my small (bdb) slapd has: # pmap 3668 3668: slapd START SIZE RSS PSS DIRTY SWAP PERM MAPPING [...] 00007f601a7f4000 8192K 120K 120K 120K 0K rw-p [anon] [...] 00007f603db4c000 18320K 184K 184K 84K 0K rw-s /var/lib/ldap/__db.003 [...] Total: 808004K 29768K 28657K 27016K 32040K So of 800MB virtual memory there is only 30MB actually in use... >> MDB_WRITEMAP, however, memory usage stays very low. Does anyone have a >> suggestion what might be wrong and what causes such different behaviour with >> and without using the memorymap option? > > There is nothing wrong. It is simply writing to the shared memory map. Off-topic: I can remember a statement of the late 80ies where a programmer claimed the 32-bit address space is so large that one does not have to care about garbage collection in virtual address space; just use new addresses. I think even with 64 bit one should always try not to waste address space. Regards, Ulrich > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/