Hi all!
Could someone authoritative let me know what the current state of native windows support (I'm trying to compile with MSVC, not an emulation layer) is for the LMDB project?
I notice that I can get 0.9.31 libraries/liblmdb to compile without error with MSVC tools (up to the C4333 warning which I understand from the issues tracker is routinely ignored/suppressed).
However, the 1.0 release does not compile at all, with multiple errors and warnings that look very much like platform issues - stuff about the sizes of basic types and attempts to use unsupported features (like void* arithmetic).
Was MSVC support present in 0.9.31 and then dropped? Am I missing some compiler flags?
I can't really find any good information on the website or elsewhere about the possibility of building a stable, reliable LMDB with MSVC 1.0.
Thanks in advance, Dave Trombley
Dave Trombley wrote:
Hi all!
Could someone authoritative let me know what the current state of native windows support (I'm trying to compile with MSVC, not an emulation layer) is for the LMDB project?
LMDB on Windows is built with gcc. The OpenLDAP Project doesn't use proprietary software.
Howard,
Thanks I had the exact same question as I am investigating the use of LMDB for my company. I was specifically interested in 1.0.0 because of the new MDB_VL32 support for windowing, as we have to support Linux and Windows 64 bit and annoyingly support 32 bit Windows applications because of legacy support.
I found I had to patch mdb.c to compile with MSVC 2026.
Given your statement of building with GCC on Windows, I assume you and your team are using MinGW? I would like to know the tool set you are using as I was planning to submit a pull request to Conan Center Index to update their recipe for LDMB which currently only supports up to 0.9.32, to add support for 0.9.35 and 1.0.0.
And perhaps I should open a new thread for my next question. But my patched mdb.c to compile with MSVC I also saw an 8 fold drop in write performance on Windows Only in my test application moving from 0.9.32 to 1.0.0. I did not see this when moving to 0.9.35.
I assume you and your team have never seen this. It very well could be a result of my terrible patch to mdb.c to compile with MSVC and may be solved as soon as I recompile LMDB on windows with GCC.
Thanks For your Time -Austin L.
ajldigger@gmail.com wrote:
Howard,
Thanks I had the exact same question as I am investigating the use of LMDB for my company. I was specifically interested in 1.0.0 because of the new MDB_VL32 support for windowing, as we have to support Linux and Windows 64 bit and annoyingly support 32 bit Windows applications because of legacy support.
I found I had to patch mdb.c to compile with MSVC 2026.
Given your statement of building with GCC on Windows, I assume you and your team are using MinGW?
We currently use msys2, with the mingw64-ucrt environment.
I would like to know the tool set you are using as I was planning to submit a pull request to Conan Center Index to update their recipe for LDMB which currently only supports up to 0.9.32, to add support for 0.9.35 and 1.0.0.
Looks like 1.0.0 won't compile cleanly, this has been fixed in git rev 8827c7db612cd6c2690336e914f25d2e9b0abc40
And perhaps I should open a new thread for my next question. But my patched mdb.c to compile with MSVC I also saw an 8 fold drop in write performance on Windows Only in my test application moving from 0.9.32 to 1.0.0. I did not see this when moving to 0.9.35.
Probably an effect of ITS#8324, which was never enabled in 0.9. But should have been mitigated by ITS#9017. You can also try compiling with -DMDB_FIXEDSIZE instead, to use the pre-8324 behavior.
I assume you and your team have never seen this. It very well could be a result of my terrible patch to mdb.c to compile with MSVC and may be solved as soon as I recompile LMDB on windows with GCC.
Thanks For your Time -Austin L.
Thank you Howard. Turns out my patch was basically the same as compile fix you provided.
I have tried the -DMDB_FIXEDSIZE as you suggested but that has minimal or no improvement to my write speeds. I am still seeing an 8~10 fold decrease in writes speeds on Windows moving from 0.9.35 to 1.0.0.
Reported by my google benchmark test: 0.9.35: bytes_per_second=613.809Mi/s 1.0.0: bytes_per_second=70.3283Mi/s
Austin L wrote:
Thank you Howard. Turns out my patch was basically the same as compile fix you provided.
I have tried the -DMDB_FIXEDSIZE as you suggested but that has minimal or no improvement to my write speeds. I am still seeing an 8~10 fold decrease in writes speeds on Windows moving from 0.9.35 to 1.0.0.
Reported by my google benchmark test: 0.9.35: bytes_per_second=613.809Mi/s 1.0.0: bytes_per_second=70.3283Mi/s
You can try turning off FILE_FLAG_WRITE_THROUGH on the writes. Change that in the `case MDB_O_OVERLAPPED:` clause.
But other people have reported that using writethrough was faster than using buffered writes with an explicit FlushFileBuffers().
Your the man Howard, that did the trick. Thanks so much for the help!!!
Austin L wrote:
Your the man Howard, that did the trick. Thanks so much for the help!!!
Try this out too https://bugs.openldap.org/show_bug.cgi?id=10542
Howard,
Again thanks for the support. I have tested the change you made and noticed a curious drop off in performance but not as drastic as when using FILE_FLAG_WRITE_THROUGH
I assume related to the following change "#if defined(_WIN32) && MDB_USE_WRITE_THROUGH"
Storing 6k byte struct with int as key. Writes are performed in batches of 1k Writes for this test are sorted. These tests test performance of 5k, 10k, 100k, and 1M entries.
LMDB 0.9.35
LMDBBenchmark/DirectLMDB/5000/iterations:1/real_time 58.6 ms 46.9 ms 1 bytes_per_second=470.466Mi/s items_per_second=85.3493k/s time_per_item=11.7166us LMDBBenchmark/DirectLMDB/10000/iterations:1/real_time 125 ms 78.1 ms 1 bytes_per_second=442.574Mi/s items_per_second=80.2894k/s time_per_item=12.4549us LMDBBenchmark/DirectLMDB/100000/iterations:1/real_time 1138 ms 734 ms 1 bytes_per_second=484.43Mi/s items_per_second=87.8827k/s time_per_item=11.3788us LMDBBenchmark/DirectLMDB/1000000/iterations:1/real_time 10792 ms 6938 ms 1 bytes_per_second=510.755Mi/s items_per_second=92.6584k/s time_per_item=10.7923us
LMDB 1.0.0 with FILE_FLAG_WRITE_THROUGH removed from case MDB_O_OVERLAPPED Includes build fixes from ITS#10539 lmdb: windows cleanup (ac041af3dd044d433e73432839cc323bffa357f3)
LMDBBenchmark/DirectLMDB/5000/iterations:1/real_time 47.2 ms 46.9 ms 1 bytes_per_second=584.429Mi/s items_per_second=106.024k/s time_per_item=9.43184us LMDBBenchmark/DirectLMDB/10000/iterations:1/real_time 93.5 ms 93.8 ms 1 bytes_per_second=589.774Mi/s items_per_second=106.994k/s time_per_item=9.34636us LMDBBenchmark/DirectLMDB/100000/iterations:1/real_time 858 ms 844 ms 1 bytes_per_second=642.65Mi/s items_per_second=116.586k/s time_per_item=8.57736us LMDBBenchmark/DirectLMDB/1000000/iterations:1/real_time 13081 ms 9219 ms 1 bytes_per_second=421.389Mi/s items_per_second=76.446k/s time_per_item=13.0811us
LMDB 1.0.0 Includes suggested commit: ITS#10542 lmdb: Add compile time flag MDB_USE_WRITE_THROUGH for Windows (62e2a60e71cd58e6fdd83a31af3d3c7fe103483d) Includes build fixes from ITS#10539 lmdb: windows cleanup (ac041af3dd044d433e73432839cc323bffa357f3)
LMDBBenchmark/DirectLMDB/5000/iterations:1/real_time 61.9 ms 46.9 ms 1 bytes_per_second=445.215Mi/s items_per_second=80.7685k/s time_per_item=12.3811us LMDBBenchmark/DirectLMDB/10000/iterations:1/real_time 126 ms 109 ms 1 bytes_per_second=437.474Mi/s items_per_second=79.3641k/s time_per_item=12.6002us LMDBBenchmark/DirectLMDB/100000/iterations:1/real_time 1686 ms 1375 ms 1 bytes_per_second=327.037Mi/s items_per_second=59.3292k/s time_per_item=16.8551us LMDBBenchmark/DirectLMDB/1000000/iterations:1/real_time 60717 ms 56750 ms 1 bytes_per_second=90.786Mi/s items_per_second=16.4699k/s time_per_item=60.7168us
Both these 1.0.0 tests include an additional change below for MSVC compatibility, because of my intent to update conan-center-index in order to be in compliance the recipe must support MSVC to be in compliance. Their recipes support embedded patch files. @@ -7213,7 +7217,7 @@ ok: #endif *ret = p; if (enc && env->me_encfunc) { - *enc = (MDB_page *)(id3.mptr + rem * env->me_psize); + *enc = (MDB_page *)((char*)id3.mptr + rem * env->me_psize);
Austin L wrote:
Howard,
Again thanks for the support. I have tested the change you made and noticed a curious drop off in performance but not as drastic as when using FILE_FLAG_WRITE_THROUGH
Did you also define MDB_FIXEDSIZE?
I'd guess this is overhead from dynamically growing the map.
The case where you only changed the MDB_O_OVERLAPPED code lacks an fsync, so transaction commits aren't actually durable.
It looks to me like the WRITEMAP case is still missing fsync too, will look into that a bit more. Since there's no actual WriteFile calls, the WRITE_THROUGH flag shouldn't have changed things one way or the other.
I assume related to the following change "#if defined(_WIN32) && MDB_USE_WRITE_THROUGH"
Storing 6k byte struct with int as key. Writes are performed in batches of 1k Writes for this test are sorted. These tests test performance of 5k, 10k, 100k, and 1M entries.
LMDB 0.9.35
LMDBBenchmark/DirectLMDB/5000/iterations:1/real_time 58.6 ms 46.9 ms 1 bytes_per_second=470.466Mi/s items_per_second=85.3493k/s time_per_item=11.7166us LMDBBenchmark/DirectLMDB/10000/iterations:1/real_time 125 ms 78.1 ms 1 bytes_per_second=442.574Mi/s items_per_second=80.2894k/s time_per_item=12.4549us LMDBBenchmark/DirectLMDB/100000/iterations:1/real_time 1138 ms 734 ms 1 bytes_per_second=484.43Mi/s items_per_second=87.8827k/s time_per_item=11.3788us LMDBBenchmark/DirectLMDB/1000000/iterations:1/real_time 10792 ms 6938 ms 1 bytes_per_second=510.755Mi/s items_per_second=92.6584k/s time_per_item=10.7923us
LMDB 1.0.0 with FILE_FLAG_WRITE_THROUGH removed from case MDB_O_OVERLAPPED Includes build fixes from ITS#10539 lmdb: windows cleanup (ac041af3dd044d433e73432839cc323bffa357f3)
LMDBBenchmark/DirectLMDB/5000/iterations:1/real_time 47.2 ms 46.9 ms 1 bytes_per_second=584.429Mi/s items_per_second=106.024k/s time_per_item=9.43184us LMDBBenchmark/DirectLMDB/10000/iterations:1/real_time 93.5 ms 93.8 ms 1 bytes_per_second=589.774Mi/s items_per_second=106.994k/s time_per_item=9.34636us LMDBBenchmark/DirectLMDB/100000/iterations:1/real_time 858 ms 844 ms 1 bytes_per_second=642.65Mi/s items_per_second=116.586k/s time_per_item=8.57736us LMDBBenchmark/DirectLMDB/1000000/iterations:1/real_time 13081 ms 9219 ms 1 bytes_per_second=421.389Mi/s items_per_second=76.446k/s time_per_item=13.0811us
LMDB 1.0.0 Includes suggested commit: ITS#10542 lmdb: Add compile time flag MDB_USE_WRITE_THROUGH for Windows (62e2a60e71cd58e6fdd83a31af3d3c7fe103483d) Includes build fixes from ITS#10539 lmdb: windows cleanup (ac041af3dd044d433e73432839cc323bffa357f3)
LMDBBenchmark/DirectLMDB/5000/iterations:1/real_time 61.9 ms 46.9 ms 1 bytes_per_second=445.215Mi/s items_per_second=80.7685k/s time_per_item=12.3811us LMDBBenchmark/DirectLMDB/10000/iterations:1/real_time 126 ms 109 ms 1 bytes_per_second=437.474Mi/s items_per_second=79.3641k/s time_per_item=12.6002us LMDBBenchmark/DirectLMDB/100000/iterations:1/real_time 1686 ms 1375 ms 1 bytes_per_second=327.037Mi/s items_per_second=59.3292k/s time_per_item=16.8551us LMDBBenchmark/DirectLMDB/1000000/iterations:1/real_time 60717 ms 56750 ms 1 bytes_per_second=90.786Mi/s items_per_second=16.4699k/s time_per_item=60.7168us
Both these 1.0.0 tests include an additional change below for MSVC compatibility, because of my intent to update conan-center-index in order to be in compliance the recipe must support MSVC to be in compliance. Their recipes support embedded patch files. @@ -7213,7 +7217,7 @@ ok: #endif *ret = p; if (enc && env->me_encfunc) {
*enc = (MDB_page *)(id3.mptr + rem * env->me_psize);
*enc = (MDB_page *)((char*)id3.mptr + rem * env->me_psize);
Yes sir, this was with MDB_FIXEDSIZE:
I re-built and ran to confirm LMDBBenchmark/DirectLMDB/5000/iterations:1/real_time 60.1 ms 62.5 ms 1 bytes_per_second=458.212Mi/s items_per_second=83.1263k/s time_per_item=12.0299us LMDBBenchmark/DirectLMDB/10000/iterations:1/real_time 115 ms 78.1 ms 1 bytes_per_second=478.553Mi/s items_per_second=86.8166k/s time_per_item=11.5185us LMDBBenchmark/DirectLMDB/100000/iterations:1/real_time 1589 ms 1391 ms 1 bytes_per_second=346.968Mi/s items_per_second=62.945k/s time_per_item=15.8869us LMDBBenchmark/DirectLMDB/1000000/iterations:1/real_time 64060 ms 59797 ms 1 bytes_per_second=86.0486Mi/s items_per_second=15.6105k/s time_per_item=64.0596us
WIthout MDB_FIXEDSZIE: LMDBBenchmark/DirectLMDB/5000/iterations:1/real_time 263 ms 234 ms 1 bytes_per_second=104.879Mi/s items_per_second=19.0266k/s time_per_item=52.5579us LMDBBenchmark/DirectLMDB/10000/iterations:1/real_time 541 ms 438 ms 1 bytes_per_second=101.861Mi/s items_per_second=18.479k/s time_per_item=54.1155us LMDBBenchmark/DirectLMDB/100000/iterations:1/real_time 4983 ms 4156 ms 1 bytes_per_second=110.612Mi/s items_per_second=20.0666k/s time_per_item=49.834us LMDBBenchmark/DirectLMDB/1000000/iterations:1/real_time 101823 ms 92297 ms 1 bytes_per_second=54.1355Mi/s items_per_second=9.82097k/s time_per_item=101.823us
Austin L wrote:
Yes sir, this was with MDB_FIXEDSIZE:
I re-built and ran to confirm LMDBBenchmark/DirectLMDB/5000/iterations:1/real_time 60.1 ms 62.5 ms 1 bytes_per_second=458.212Mi/s items_per_second=83.1263k/s time_per_item=12.0299us LMDBBenchmark/DirectLMDB/10000/iterations:1/real_time 115 ms 78.1 ms 1 bytes_per_second=478.553Mi/s items_per_second=86.8166k/s time_per_item=11.5185us LMDBBenchmark/DirectLMDB/100000/iterations:1/real_time 1589 ms 1391 ms 1 bytes_per_second=346.968Mi/s items_per_second=62.945k/s time_per_item=15.8869us LMDBBenchmark/DirectLMDB/1000000/iterations:1/real_time 64060 ms 59797 ms 1 bytes_per_second=86.0486Mi/s items_per_second=15.6105k/s time_per_item=64.0596us
WIthout MDB_FIXEDSZIE: LMDBBenchmark/DirectLMDB/5000/iterations:1/real_time 263 ms 234 ms 1 bytes_per_second=104.879Mi/s items_per_second=19.0266k/s time_per_item=52.5579us LMDBBenchmark/DirectLMDB/10000/iterations:1/real_time 541 ms 438 ms 1 bytes_per_second=101.861Mi/s items_per_second=18.479k/s time_per_item=54.1155us LMDBBenchmark/DirectLMDB/100000/iterations:1/real_time 4983 ms 4156 ms 1 bytes_per_second=110.612Mi/s items_per_second=20.0666k/s time_per_item=49.834us LMDBBenchmark/DirectLMDB/1000000/iterations:1/real_time 101823 ms 92297 ms 1 bytes_per_second=54.1355Mi/s items_per_second=9.82097k/s time_per_item=101.823us
The only other apparent difference is the use of the OVERLAPPED flag itself. You could try removing that too.
openldap-technical@openldap.org
-
ajldigger＠gmail.com -
Austin L
-
Dave Trombley -
Howard Chu