Hi,
I have a question regarding growing an LMDB database when a write transaction hits MDB_MAP_FULL. I would like to avoid defining a high mapsize value because my application will contain many MDB_envs, and because I have Windows users (Windows allocates the whole file on the disk).
Based on the intuition that MDB_MAP_FULL should not leave the database in a weird state, I have made the following little experiment. When MDB_MAP_FULL is encountered I tried to:
* copy the current env (mdb_env_copy) into another directory (fine: it does not seem to contain uncommited data)
* reset the transaction < error bit > (modified LMDB code to introduce a < txn->mt_flags &= ~MDB_TXN_ERROR > somewhere)
* commit the transaction
* close the database
* close the env
* reopen it with a higher mapsize value
* reopen the database
* create another transaction
* continue writing ... and it seems to be working pretty well.
Assuming I am ready to < relax > some of the ACID requirements, does it sound reasonable to think that MDB_MAP_FULL does not leave LMDB is a weird state? And that the < trick > described above should always be working? By < working > I mean: the copied environment will never contain uncommitted data (so I can rely on it to implement a kind of rollback), the reopened environment will always be valid and contain the expected data (data written before hitting MDB_MAP_FULL)?
Thanks in advance for any insight, Best regards,
Bruno.
Bruno Freudensprung wrote:
Hi,
I have a question regarding growing an LMDB database when a write transaction hits MDB_MAP_FULL.
I would like to avoid defining a high mapsize value because my application will contain many MDB_envs, and because I have Windows users
My condolences.
(Windows allocates the whole file on the disk).
See ITS#8324.
Based on the intuition that MDB_MAP_FULL should not leave the database in a weird state, I have made the following little experiment. When MDB_MAP_FULL is encountered I tried to:
·copy the current env (mdb_env_copy) into another directory (fine: it does not seem to contain uncommited data)
·reset the transaction « error bit » (modified LMDB code to introduce a « txn->mt_flags &= ~MDB_TXN_ERROR » somewhere)
You're being stupid. You think we add error flags just for the fun of it? When you get an error like this you must abort the current transaction. Does the documentation not state this clearly enough?
Hi Howard,
Hmm well... how embarrassing. To me it seemed probable that LMDB has been designed with these rationales in mind: one must have a clear idea of the maximum order of magnitude of the volume to be held by the database, the database must have ACID properties, and (less probably) a single machine will not host a lot of LMDB databases. Given that it seemed probable that when the database is full, you probably had no choice but to fail a transaction (in order to be consistent). But ok, that was a stupid guess and a misunderstanding of LMDB rationales. I'm sorry; I didn't mean to offend anyone.
Anyway, thanks a lot for ITS#8324! I'll give it a try!
Best regards,
Bruno.
-----Message d'origine----- De : openldap-technical [mailto:openldap-technical-bounces@openldap.org] De la part de Howard Chu Envoyé : lundi 30 novembre 2015 20:15 À : Bruno Freudensprung bruno.freudensprung@temis.com; openldap-technical@openldap.org Objet : Re: Growing an LMDB database after MDB_MAP_FULL
Bruno Freudensprung wrote:
Hi,
I have a question regarding growing an LMDB database when a write transaction hits MDB_MAP_FULL.
I would like to avoid defining a high mapsize value because my application will contain many MDB_envs, and because I have Windows users
My condolences.
(Windows allocates the whole file on the disk).
See ITS#8324.
Based on the intuition that MDB_MAP_FULL should not leave the database in a weird state, I have made the following little experiment. When MDB_MAP_FULL is encountered I tried to:
·copy the current env (mdb_env_copy) into another directory (fine: it does not seem to contain uncommited data)
·reset the transaction « error bit » (modified LMDB code to introduce a « txn->mt_flags &= ~MDB_TXN_ERROR » somewhere)
You're being stupid. You think we add error flags just for the fun of it? When you get an error like this you must abort the current transaction. Does the documentation not state this clearly enough?
Hi Howard,
I've tested the ITS#8324 fix this week-end and it's working nicely, thanks again! Regards,
Bruno.
-----Message d'origine----- De : openldap-technical [mailto:openldap-technical-bounces@openldap.org] De la part de Howard Chu Envoyé : lundi 30 novembre 2015 20:15 À : Bruno Freudensprung bruno.freudensprung@temis.com; openldap-technical@openldap.org Objet : Re: Growing an LMDB database after MDB_MAP_FULL
(Windows allocates the whole file on the disk).
See ITS#8324.
Bruno Freudensprung wrote:
Hi Howard,
I've tested the ITS#8324 fix this week-end and it's working nicely, thanks again!
Great, thanks for the feedback.
Regards,
Bruno.
-----Message d'origine----- De : openldap-technical [mailto:openldap-technical-bounces@openldap.org] De la part de Howard Chu Envoyé : lundi 30 novembre 2015 20:15 À : Bruno Freudensprung bruno.freudensprung@temis.com; openldap-technical@openldap.org Objet : Re: Growing an LMDB database after MDB_MAP_FULL
(Windows allocates the whole file on the disk).
See ITS#8324.
openldap-technical@openldap.org
-
Bruno Freudensprung -
Howard Chu