Hi,
as far as my scenario goes, I shortly afterward detected serious errors in my setup that were responsible to the buildup. There is an article on the symas website (https://symas.com/understanding-lmdb-database-file-sizes-and-memory-utilizat...) that adresses the question/issue of database growth, I assume you have read that. Crashes sould of course not happen, but what exactly do you mean by "crash" anyway?
regards, Christian
salesman1 IIS Advertising sales-iss@outlook.com hat am 26. Juni 2016 um 00:59 geschrieben:
Re: I HAVE THE SAME PROBLEMS, TOO !
from your comments
http://www.openldap.org/lists/openldap-technical/201511/msg00201.html
I noticed the following using lmdb. Tested version is up-to-date 0.9.18, from 2016.
Database grows much larger than expected in many use cases, and eventually crashes, also was
not a well-behaved database when simple, missing or misbehaved client code executes it.
.
Do you think the same ?
Maybe lbmd is was fine-tuned for other especific needs, or is a toy database, or a complicated
case of application in our scenario. Unfortunatelly is not like Redis yet.
We can not throw anything special on it without some effort.
We really want do use lbmd, it's fast, but can't get it to work as expected. Lmbd may be incomplete,
buggy for (the common coder) or || (the experienced coder) who tries do use it for other needs
than designed. So it's now a dream but not easily realistic. Needs an expert.
- Database size grows 3x the data we put in, than stops growing - when
records (key+data) from 410 to 4010 bytes are added.
-> and consumed much more space than expected for 5000*400 bytes = 2.000.000 bytes
- Second, when we put key+data > 4096 bytes, database grows much larger and
crashes a lot of times, with a number
of page overflows shown on 'mdb_dump ./testdb'
We made keys ("k:%d", i%5000) for (i=0; i < 1000000; i++) // so there are 5000 keys, else is overwriting;
Data is random ascii or fixed memset( 'x',400 )
Program name: sample2
We are overwriting old values AND NOT appending, but it grows anyway. Larger data than 4096 bytes per key-value
seems impossible or non-sense at this time. Are we using wrong flags or program structure ?
trials - first run goes fine: #./sample2
[root@ip-172-31-13-74 liblmdb]# ./sample2; ls ./testdb/* -l Start Init done written 100000 pairs of len=400 bytes -rw-r--r-- 1 root root 2584576 Jun 25 21:43 ./testdb/data.mdb -rw-r--r-- 1 root root 8192 Jun 25 21:43 ./testdb/lock.mdb [root@ip-172-31-13-74 liblmdb]# ./sample2; ls ./testdb/* -l Start Init done written 100000 pairs of len=400 bytes -rw-r--r-- 1 root root 5173248 Jun 25 21:43 ./testdb/data.mdb -rw-r--r-- 1 root root 8192 Jun 25 21:43 ./testdb/lock.mdb [root@ip-172-31-13-74 liblmdb]# ./sample2; ls ./testdb/* -l Start Init done written 100000 pairs of len=400 bytes -rw-r--r-- 1 root root 7761920 Jun 25 21:43 ./testdb/data.mdb -rw-r--r-- 1 root root 8192 Jun 25 21:43 ./testdb/lock.mdb [root@ip-172-31-13-74 liblmdb]#
Looks like it's not replacing old keys and have a page/data size problem,
Any suggestions ?
REGARDS !!
Fabio Martinez - Buenos Aires and São Paulo
gcc developer
Brazil
--
Hello,
I am trying to use LMDB to store large (huge) amounts of binary data which, for the reason of limiting memory footprint, are split into chunks. Each chunk ist stored under a separate key, made up of [collectionId, chunkId], so that I can later iterate the chunks using a LMDB cursor. Chunk size is configurable.
During my tests, I encountered a strange scenario where, after inserting some 2000 chunks consisting of 512KB each, the database size had grown to a value that was roughly 135 times the calculated size of the data. I ran stat over the db and saw that there were > 12000 overflow pages vs. approx. 2000 data pages. When I reduced the chunk size to 4060 bytes, the number of overflow pages went down to 1000, and the database size went down to the expected number (I experimented with different sizes, this was the best result). I did not find any documentation that would explain this behavior, or how to deal with it. Of course it makes me worry about database bloat and the consequences. Can anyone shed light on this?
thanks, Christian