Austin L wrote:
Howard,
Again thanks for the support. I have tested the change you made and noticed a curious drop off in performance but not as drastic as when using FILE_FLAG_WRITE_THROUGH
Did you also define MDB_FIXEDSIZE?
I'd guess this is overhead from dynamically growing the map.
The case where you only changed the MDB_O_OVERLAPPED code lacks an fsync, so transaction commits aren't actually durable.
It looks to me like the WRITEMAP case is still missing fsync too, will look into that a bit more. Since there's no actual WriteFile calls, the WRITE_THROUGH flag shouldn't have changed things one way or the other.
I assume related to the following change "#if defined(_WIN32) && MDB_USE_WRITE_THROUGH"
Storing 6k byte struct with int as key. Writes are performed in batches of 1k Writes for this test are sorted. These tests test performance of 5k, 10k, 100k, and 1M entries.
LMDB 0.9.35
LMDBBenchmark/DirectLMDB/5000/iterations:1/real_time 58.6 ms 46.9 ms 1 bytes_per_second=470.466Mi/s items_per_second=85.3493k/s time_per_item=11.7166us LMDBBenchmark/DirectLMDB/10000/iterations:1/real_time 125 ms 78.1 ms 1 bytes_per_second=442.574Mi/s items_per_second=80.2894k/s time_per_item=12.4549us LMDBBenchmark/DirectLMDB/100000/iterations:1/real_time 1138 ms 734 ms 1 bytes_per_second=484.43Mi/s items_per_second=87.8827k/s time_per_item=11.3788us LMDBBenchmark/DirectLMDB/1000000/iterations:1/real_time 10792 ms 6938 ms 1 bytes_per_second=510.755Mi/s items_per_second=92.6584k/s time_per_item=10.7923us
LMDB 1.0.0 with FILE_FLAG_WRITE_THROUGH removed from case MDB_O_OVERLAPPED Includes build fixes from ITS#10539 lmdb: windows cleanup (ac041af3dd044d433e73432839cc323bffa357f3)
LMDBBenchmark/DirectLMDB/5000/iterations:1/real_time 47.2 ms 46.9 ms 1 bytes_per_second=584.429Mi/s items_per_second=106.024k/s time_per_item=9.43184us LMDBBenchmark/DirectLMDB/10000/iterations:1/real_time 93.5 ms 93.8 ms 1 bytes_per_second=589.774Mi/s items_per_second=106.994k/s time_per_item=9.34636us LMDBBenchmark/DirectLMDB/100000/iterations:1/real_time 858 ms 844 ms 1 bytes_per_second=642.65Mi/s items_per_second=116.586k/s time_per_item=8.57736us LMDBBenchmark/DirectLMDB/1000000/iterations:1/real_time 13081 ms 9219 ms 1 bytes_per_second=421.389Mi/s items_per_second=76.446k/s time_per_item=13.0811us
LMDB 1.0.0 Includes suggested commit: ITS#10542 lmdb: Add compile time flag MDB_USE_WRITE_THROUGH for Windows (62e2a60e71cd58e6fdd83a31af3d3c7fe103483d) Includes build fixes from ITS#10539 lmdb: windows cleanup (ac041af3dd044d433e73432839cc323bffa357f3)
LMDBBenchmark/DirectLMDB/5000/iterations:1/real_time 61.9 ms 46.9 ms 1 bytes_per_second=445.215Mi/s items_per_second=80.7685k/s time_per_item=12.3811us LMDBBenchmark/DirectLMDB/10000/iterations:1/real_time 126 ms 109 ms 1 bytes_per_second=437.474Mi/s items_per_second=79.3641k/s time_per_item=12.6002us LMDBBenchmark/DirectLMDB/100000/iterations:1/real_time 1686 ms 1375 ms 1 bytes_per_second=327.037Mi/s items_per_second=59.3292k/s time_per_item=16.8551us LMDBBenchmark/DirectLMDB/1000000/iterations:1/real_time 60717 ms 56750 ms 1 bytes_per_second=90.786Mi/s items_per_second=16.4699k/s time_per_item=60.7168us
Both these 1.0.0 tests include an additional change below for MSVC compatibility, because of my intent to update conan-center-index in order to be in compliance the recipe must support MSVC to be in compliance. Their recipes support embedded patch files. @@ -7213,7 +7217,7 @@ ok: #endif *ret = p; if (enc && env->me_encfunc) {
*enc = (MDB_page *)(id3.mptr + rem * env->me_psize);
*enc = (MDB_page *)((char*)id3.mptr + rem * env->me_psize);