Hi!
I'm trying to use LMDB in a way that automatically resizes the map whenever it becomes too small, i.e. a function fails with MDB_MAP_FULL. I've created and attached a program that resembles my real usecase in a very simplified fashion: It just writes data to the database, and tries to double the map size whenever MDB_MAP_FULL is received.
However, for me this program (as well as my real code) produces a segfault in the mdb_put call that follows after the first resize. What am I doing wrong here, and how can I correctly implement a resize when full?
According to the docs [1], it is ok to call mdb_env_set_mapsize for an open environment as long as no transactions are active. My code ensures this by aborting the current transaction before resizing the map. Also, there seems to be other code out there [2] that does the resize in a way that looks like what I'm trying to do, and presumably works fine.
[1] http://www.lmdb.tech/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5 [2] https://github.com/BVLC/caffe/pull/3731
Yours, Daniel
Daniel Kraft wrote:
Hi!
I'm trying to use LMDB in a way that automatically resizes the map whenever it becomes too small, i.e. a function fails with MDB_MAP_FULL. I've created and attached a program that resembles my real usecase in a very simplified fashion: It just writes data to the database, and tries to double the map size whenever MDB_MAP_FULL is received.
However, for me this program (as well as my real code) produces a segfault in the mdb_put call that follows after the first resize. What am I doing wrong here, and how can I correctly implement a resize when full?
You're attempting to use the txn after committing it. This is basically a use-after-free attempt, which is why you get a SEGV.
Hi Howard!
On 2018-11-12 21:21, Howard Chu wrote:
Daniel Kraft wrote:
I'm trying to use LMDB in a way that automatically resizes the map whenever it becomes too small, i.e. a function fails with MDB_MAP_FULL. I've created and attached a program that resembles my real usecase in a very simplified fashion: It just writes data to the database, and tries to double the map size whenever MDB_MAP_FULL is received.
However, for me this program (as well as my real code) produces a segfault in the mdb_put call that follows after the first resize. What am I doing wrong here, and how can I correctly implement a resize when full?
You're attempting to use the txn after committing it. This is basically a use-after-free attempt, which is why you get a SEGV.
Ah ok, so mdb_txn_commit also frees the transaction if it then actually fails with MDB_MAP_FULL -- I was missing that. By avoiding the mdb_txn_abort in that case, my example code indeed works now!
So is it correct to assume that the transaction is "effectively aborted" after mdb_txn_commit fails with MDB_MAP_FULL (or, for that matter, any error)?
Thanks a lot!
Yours, Daniel
openldap-technical@openldap.org
-
Daniel Kraft -
Howard Chu