Victor Baybekov wrote:

Thanks for the explanation! The ultimate speed is needed for server usage
where, e.g. now, I have several TB free space and could allocate the entire
space at start. For clients, where I use LMDB for replicated cache of data
subsets, even the 3x perf drop is kind of tolerable given the absolute
numbers. Could we have a flag to use the patch optionally? So that on a server
where I could allocate 1+Tb and use hacks with WriteMap as we discussed
previously, I could enjoy the speedup of WriteMap and the fact that the
pointer addresses from MDB_RESERVE remain fixed over the lifetime of a process?

I am loathe to add more option flags. All of this violates the principle of simplicity which is core to LMDB.

Another question: with this patch, do addresses of pointers remain the same
after the map file size increases, or we must follow the LMDB rules as if
there is no WriteMap flag is used? Does Windows reserve the entire virtual
memory space, and the patch just affects the file size on a disk, or file
growth with the patch is equivalent to remapping and pointer addresses become
invalid every time? I am referring to this discussion
<http://www.openldap.org/lists/openldap-technical/201510/msg00022.html> we had
before.

The entire virtual memory space is reserved, so there is no change in behavior here. The patch would have been pointless if it required such a change.

P.S. I know your hate for Windows and am starting to share it given rich tools
available on *nix, but for small non-hardcore-programmers teams doing
exploratory number crunching the benefits of RDP & GUIs etc. outweigh the
benefits *nix gives for high load production systems, so we use Windows as an
internal server. LMDB is just a perfect off-heap data structure for many use
cases.


On Wed, Dec 30, 2015 at 11:32 PM, Howard Chu <hyc@symas.com

<mailto:hyc@symas.com>> wrote:

    Victor Baybekov wrote:

        Hi,

        Thanks a lot for ITS#8324! For embedded, not server, use case that
        change adds
        much convenience.

        I have tested the master from .NET via P/Invoke and do not see any major
        slowdown with default options. To insert 10M <int32,int32> pairs inside a
        single transaction v.0.9.14 takes minimum 3400 msec, latest master takes
        minimum 3750 msec. This is not scientific, just best result from 10 runs.
        Sometimes both timings increase to 5000+ msecs. On average slowdown is
        visible
        but tolerable - from 2.9 Mops to 2.6 Mops (absolute numbers are still
        awesome!). With Append and NoSync I could get 3.45 Mops on the same
        test with
        master build.

        However, with WriteMap performance of |master| drops 3x to 10000 msec
        or just
        1 Mops, while for the |v.0.9.14| performance with WriteMap improvesto 2350
        msec or 4.25 Mops.

        Is this the cost of convenience or it could be fixed so that WriteMap
        still "is faster and uses fewer mallocs" as the docs say?


    That's pretty much the cost of this patch, it has the biggest impact on
    WriteMap usage. In default mode, regular Writes are done to grow the file
    so the code path is basically unchanged from before. In WriteMap mode the
    file has to be grown explicitly, right before accessing a new page and
    apparently the VirtualAlloc call that does this is expensive. Since it's
    the equivalent of both a malloc and a write together, it's actually more
    expensive than the default mode.

    Please followup to the ITS so this conversation stays with the ticket.

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/