kriszyp@gmail.com wrote:
Full_Name: Kristopher William Zyp Version: LMDB 0.9.70 OS: Windows Server 2012 R2 URL: Submission from: (NULL) (71.199.6.148)
I guess Windows users are never happy.
Note that you're using the LMDB work-in-progress code. You ought to be using the LMDB 0.9 release branch, which just uses Win32.
The fact is that you get optimal performance by using LMDB as originally designed and documented - choose as large a mapsize as possible and then forget about it. Preallocation is always more efficient than incremental growth.
Clearly Windows will struggle to allocate disk space to grow files and address space to grow maps while there's competing system activity occurring, and this problem is worse when running under a hypervisor. This is the reason we never merged the NTDLL code into the release branch; we knew from the outset that it would destroy performance.
https://openldap.org/lists/openldap-technical/201511/msg00107.html
As a general rule, we're opposed to adding platform-specific API options to LMDB. The API is supposed to be uniform across all supported platforms. If you want to submit a patch for compile-time selection of this feature, I guess that could work. As it is, LMDB is clearly not going to be a preinstalled Windows DLL so it should be being built privately for every app that uses it anyway, so this shouldn't cause compatibility issues.
We have experienced significant performance issues in the form of strange pauses or freezing of any processes that are using our LMDB databases. These pauses are periodic, often every few minutes, and typically completely freeze a process for about 5 seconds, sometimes less, sometimes as much as a minute. They often occur when the process is completely idle (we have a sleep timer to monitor for them), but we also see a lot of unusually large pauses on commits too. We have never been able to reproduce any of these issues with our local machines running Windows 10, only on the virtual Azure Windows servers.
After weeks of investigation, we finally realized that all of these strange performance issues seemed to be due to the use of NTDLL for memory mapping. I swapped out the NtCreateSection/NtMapViewOfSection calls for standard Win32 CreateFileMapping/MapViewOfFile calls to create the memory map, and suddenly these issues went away, and performance dramatically improved. Making this switch does mean you lose the progressive file allocation functionality of NtCreateSection, but this is a small price to pay for the vast performance benefit of using standard file mapping.
Anyway, my request is to simply to provide an option to use CreateFileMapping API instead of NtCreateSection. I recognize that the NTDLL based APIs may be more convenient for many users, but an option to use the standard APIs, even if it means the entire map is pre-allocated in the file size, is well worth it for the gains. I also recognize that Windows isn't the first choice of LMDB, but performance with these standard APIs is very good.
I'd be happy to work on putting together a branch/pull request to address this. However, your repository is a little different than typical github repositories, so I wanted to know if that was doable, and if there was a proper way to submit a PR, or if it needs to be submitted through this ticket system as a patch.
http://www.openldap.org/devel/contributing.html