>> Howard Chu <hyc(a)symas.com> schrieb am 15.01.2014 um
23:10 in Nachricht
<52D7076B.3020003(a)symas.com>:
Luc Vlaming wrote:
> Hi,
>
> Currently I am creating support for using LMDB as a new storage backend for
> one of our products.
> At the moment I am testing import bulk data into lmdb using transactions
that
> span a single record of 10MB. The total db size afterwards is 5GB. I also
> tested with records of 1MB.
>
> I noticed a very odd thing: when using the MDB_WRITEMAP option, memory usage
> grows very quickly and linear with the amount of data stored into the
> database. (memory usage ends up a bit higher than 5GB). when not using
Maybe for the future make a difference between virtual memory usage and real (resident)
memory usage. Especially for Linux this makes a big difference, because a malloc(1GB)
actually does not consume any memory until it is actually used.
There's also the "pmap" Utility that can show the detailed difference. For
example my small (bdb) slapd has:
# pmap 3668
3668: slapd
START SIZE RSS PSS DIRTY SWAP PERM MAPPING
[...]
00007f601a7f4000 8192K 120K 120K 120K 0K rw-p [anon]
[...]
00007f603db4c000 18320K 184K 184K 84K 0K rw-s /var/lib/ldap/__db.003
[...]
Total: 808004K 29768K 28657K 27016K 32040K
So of 800MB virtual memory there is only 30MB actually in use...
> MDB_WRITEMAP, however, memory usage stays very low. Does anyone
have a
> suggestion what might be wrong and what causes such different behaviour with
> and without using the memorymap option?
There is nothing wrong. It is simply writing to the shared memory map.
Off-topic: I can remember a statement of the late 80ies where a programmer claimed the
32-bit address space is so large that one does not have to care about garbage collection
in virtual address space; just use new addresses. I think even with 64 bit one should
always try not to waste address space.
Regards,
Ulrich
--
-- Howard Chu
CTO, Symas Corp.
http://www.symas.com
Director, Highland Sun
http://highlandsun.com/hyc/
Chief Architect, OpenLDAP
http://www.openldap.org/project/