On 8/19/20 7:11 PM, Howard Chu wrote:
> We have a process that needs to analyze the contents of a whole
LMDB. So far the approach was to open a read-only transaction and use a cursor to iterate
over the whole contents in order. This long transaction apparently causes other
concurrent process(es) to get MDB_MAP_FULL from mdb_put() even though that LMDB has plenty
free pages at that moment.
It's a documented feature.
I haven't seen that in docs http://www.lmdb.tech/doc/
(I even skimmed
the "internals" section) Am I missing some part? It's just stated that
_put functions can fail with MDB_MAP_FULL if "the database is full" (and
I wouldn't consider it "full" in my case).
If that long transaction was getting MDB_TXN_FULL (or another special
error), that would be more understandable to me, but those transactions
appear OK and other short ones are getting MDB_MAP_FULL. Right now I
have no indication *when* to break those read-only transactions, except
for guessing based on experiments.
On 8/20/20 9:05 AM, Ulrich Windl wrote:
if you can measure the "fullness" of LMDB
while you read, you could suspend/resume the reading in the way I described at
the beginning before LMDB gets too full.
I have no idea how to measure *this* kind of fullness. Our estimate
shows there should be many free pages (even half of all)
MDB_stat st; // and use mdb_stat()
size_t pgs_used = st.ms_branch_pages + st.ms_leaf_pages + st.ms_overflow_pages;
and this estimate won't change at all when done during the read-only
transaction. So the only thing to do that I can see is at the RW
transaction living in a different process: if a write gives me
MDB_MAP_FULL and the fullness estimate shows that a large fraction of
pages should be free, I might assume that the error will just disappear
soon (when the other process finishes its RO transaction). That's not
nice at all.