Patrick Hawley wrote:
I've been puzzled by lmdb's free list use. In some experiments, I've been seeing that it sometimes grabs pages from the end instead of using free list pages. This means that the number of pages used tends to grow until it uses them all. I've tried increasing the map size (to even quite a large value), and ensuring that write transactions are not held open long but that doesn't seem to help.
Length of write transactions is irrelevant. Nothing in the LMDB documentation talks about length of write transactions; perhaps you need to reread the docs.
The issue is keeping a lot of read transactions open when writes are in progress.
I've tried with
subdatabases and without subdatabases. It does appear to increase faster when overflow pages are used, but even without overflow pages the effect occurs. I can see in the code some places where it might choose not to use the free list, but I haven't been able to identify a problem in the code. Wondering whether anyone else has observed this also. The library otherwise looks quite useful and performant. Using lmdb via python (0.92). Thanks.
Patrick