On 8/19/20 7:11 PM, Howard Chu wrote:
We have a process that needs to analyze the contents of a whole LMDB. So far the approach was to open a read-only transaction and use a cursor to iterate over the whole contents in order. This long transaction apparently causes other concurrent process(es) to get MDB_MAP_FULL from mdb_put() even though that LMDB has plenty free pages at that moment.
[...]
It's a documented feature.
I haven't seen that in docs http://www.lmdb.tech/doc/ (I even skimmed the "internals" section) Am I missing some part? It's just stated that _put functions can fail with MDB_MAP_FULL if "the database is full" (and I wouldn't consider it "full" in my case).
If that long transaction was getting MDB_TXN_FULL (or another special error), that would be more understandable to me, but those transactions appear OK and other short ones are getting MDB_MAP_FULL. Right now I have no indication *when* to break those read-only transactions, except for guessing based on experiments.
On 8/20/20 9:05 AM, Ulrich Windl wrote:
if you can measure the "fullness" of LMDB while you read, you could suspend/resume the reading in the way I described at the beginning before LMDB gets too full.
I have no idea how to measure *this* kind of fullness. Our estimate shows there should be many free pages (even half of all)
MDB_stat st; // and use mdb_stat() size_t pgs_used = st.ms_branch_pages + st.ms_leaf_pages + st.ms_overflow_pages;
and this estimate won't change at all when done during the read-only transaction. So the only thing to do that I can see is at the RW transaction living in a different process: if a write gives me MDB_MAP_FULL and the fullness estimate shows that a large fraction of pages should be free, I might assume that the error will just disappear soon (when the other process finishes its RO transaction). That's not nice at all.
--Vladimir