my bad. Nothing wrong with LMDB
Christian Sell <christian(a)gsvitec.com> hat am 26. November 2015
I forgot to mention: I am using MDB_RESERVE to avoid extra memcpy. Could it
be that this is the cause for the extreme database bloat?
> something wrong with the question below?
> I am trying to use LMDB to store large (huge) amounts of binary data which,
> the reason of limiting memory footprint, are split into chunks. Each chunk
> stored under a separate key, made up of [collectionId, chunkId], so that I
> later iterate the chunks using a LMDB cursor. Chunk size is configurable.
> During my tests, I encountered a strange scenario where, after inserting
> 2000 chunks consisting of 512KB each, the database size had grown to a
> that was roughly 135 times the calculated size of the data. I ran the stat
> utility over the db and saw that there were > 12000 overflow pages vs.
> 2000 data pages. When I reduced the chunk size to 4060 bytes, the number of
> overflow pages went down to 1000, and the database size went down to the
> expected number (I experimented with different sizes, this was the best
> I did not find any documentation to explain this behaviour, or how to deal
> it. Of course it makes me worry about database bloat and the consequences.
> anyone shed light on this?
GS Vitec GmbH
Im Ziegelhaus 6-8
mobil: +49 (0) 173 5384289
Tel: +49 (0) 6051 601.26-90
Fax: +49 (0) 6051 601.26-91