Hello,
I'm playing around with LMDB and I'd like to know more about how it deletes items and reclaims space. I've been having some problems with this lately.
For simplicity I'm testing smaller 20MiB DBs with data values of sizes from 40B to 4000B. The test checks all kinds of stuff, but mainly tries to do a few fillups and flushes in short sequence. The first cycle slowly increases the size of the data until the DB is full, then it flushes. Then 10 cycles of fillup-flush of static-sized data follow. For flushing I've been doing both mdb_drop and removing batches of stored data. I've tried batches of sizes between 2 and 100 items. The results differ very much, depending on all these parameters: (only the 10 equal cycles counted):
batch:2, data size: 40B first three runs pass, 7 fail
batch:50, data size: 40B first three runs pass, 7 fail
batch:100, data size: 40B first three runs pass, 7 fail
mdb_drop, data size: 40B first run only passes, all other 9 fail
batch:20, data size: 400B all runs pass
batch:100, data size: 400B all runs pass
mdb_drop, data size: 400B all runs pass
batch:2, data size: 4000B all runs pass
batch:50, data size: 4000B first run only passes, all other 9 fail
batch:100, data size: 4000B first run only passes, all other 9 fail
mdb_drop, data size: 4000B all odd runs pass, all even runs fail
Note: I've implemented a simple protection from hitting MDB_MAP_FULL by some arithmetic over fields in MDB_stat and MDB_envinfo structures. The protection reserves at least 32 pages - every insert checks whether it would violate this reservation and if so returns with an error.
I've read through a previous thread[1] which was similar in its nature. From that I gathered I need 3 cleanup commits to reclaim all that space. This would explain why the 4000-byte runs succeeded only when flushing using batches of 2 (about 40 4KB-sized items fit into the DB). But it doesn't explain the 40-byte runs which should all succeed apart from the mdb_drop version. And I have no idea why mdb_drop works well in the 400-byte-sized case.
Can anybody shed some light into this for me, please? What can I do about it? I need to provide some guarantee the DB will stay functional after fillup.
Thanks,
Dominik
[1] http://www.openldap.org/lists/openldap-technical/201306/msg00116.html