On 11/25/2015 02:27 PM, Howard Chu wrote:
Dominik Taborsky wrote:
> Hello,
>
> I've tried stress-testing LMDB in our use case and I've discovered something
I consider a bug.
>
> In our use case we need to keep inserting to the DB until it's full, flush it and
repeat. We've designed the insert function to fail if there are not enough headroom
pages for safe delete / DB drop. During the tests the DB works correctly for some amount
of repeats, but then it suddenly refuses to insert anything. It seems LMDB doesn't
clear or reuse all of its pages. I would like to get anyone's opinion on this.
>
> I have attached the source file containing the test. The DB size is set to 20MB, the
inserted values have by default 25kB. In the first run the DB accepts 365 inserts, which
decreases over time and stabilizes at 280. After 18 repeats the DB does not accept any
inserts at all:
> ok 1 - pass #1 fillup run (365 inserts)
> ok 2 - pass #2 fillup run (300 inserts)
> ok 3 - pass #3 fillup run (286 inserts)
> ok 4 - pass #4 fillup run (283 inserts)
> ok 5 - pass #5 fillup run (281 inserts)
> ok 6 - pass #6 fillup run (280 inserts)
> ok 7 - pass #7 fillup run (280 inserts)
> ok 8 - pass #8 fillup run (280 inserts)
> ok 9 - pass #9 fillup run (280 inserts)
> ok 10 - pass #10 fillup run (280 inserts)
> ok 11 - pass #11 fillup run (280 inserts)
> ok 12 - pass #12 fillup run (280 inserts)
> ok 13 - pass #13 fillup run (280 inserts)
> ok 14 - pass #14 fillup run (280 inserts)
> ok 15 - pass #15 fillup run (280 inserts)
> ok 16 - pass #16 fillup run (280 inserts)
> ok 17 - pass #17 fillup run (280 inserts)
> ok 18 - pass #18 fillup run (108 inserts)
> not ok 19 - pass #19 fillup run (0 inserts)
> not ok 20 - pass #20 fillup run (0 inserts)
>
> The test is run 4 times with different approaches that we thought could have had some
impact. These approaches combine opening the DB per insert or per single fillup, and
reopening the DB just for the DB drop. In this case these options have no effect and the
results are the same, but originally a unit test that this originates from had some
differences so I kept them in.
>
> Note that I also tried calling mdb_drop with "1" as the last parameter, but
it had no effect on the result.
>
> I am linking against the latest LMDB from sources.
>
> I also tried adding a constant number of pages to the reserve before the insert
function fails. This only changes the amount of inserted items but doesn't change the
functionality.
>
> If you compile the program you can supply it with an argument to change the default
size of the item value size. Lower values break the DB sooner (20kB requires only 12
repeated fillups and drops).
>
> If anyone has any idea what might be the issue, please let me know. I know deleting
the DB from filesystem and creating a new one would work, but that's a hack, not a
fix.
Your calculation of max_pages vs used_pages is totally bogus. There's nothing wrong
with LMDB.
Thanks for checking it out, Howard, but can you please advise on how to compute a safe
buffer then?