I need to better understand the best way to configure the various environment flags related to sync and map.
When we converted from BDB to LMDB, we instinctively kept #MDB_NOSYNC true. This proved to make the database subject to corruption if it was killed or stopped too harshly. It also had the effect on Windows at least to make Windows gradually use up all available RAM for the memory mapped file and to bring the server to its knees.
After carefully reading the documentation about leaving "the system with no hint for when to write transactions to disk" when MDB_NOSYNC was turned on, we completely went without setting any options. The results were that we can't corrupt the database, but the speed degradation makes this approach unusable, as we are very heavily write focused (at least in this part of our process, where we perform analysis and load the database with 100's millions of records).
In looking at the documentation I'm tempted to try MDB_WRITEMAP with MDB_MAPASYNC but do I then need to also issue some manual mdb_env_sync and if so at what frequency and what should trigger this?
What are best practices combinations of those flags here. We absolutely can't afford database corruption, but we can deal with one (or maybe more with some re-design) transactions that are lost.
Please guide us.
Thanks Alain