Mohammed Muneer wrote:

 "Windows is like that."

 I know.

Please keep your replies on the mailing list.

 "Have no idea what you're actually doing, so it's impossible to answer.

Learn to ask better questions and provide better information."

 Sorry that I didn't sound intelligent. I was in a hurry when I wrote it.

I will now try to give you a detailed account of what I am doing.

 "What revision of LMDB?"

 The latest hosted at https://github.com/LMDB/lmdb.

"The latest" in which branch? Note that mdb.master is the main development branch and contains work-in-progress code that is probably not stable. As a non-LMDB-developer you should only be using actual tagged releases. You may occasionally want to use the mdb.RE/xx release engineering branches if they contain a fix for a specific issue that you encountered.

 "Which version of Windows?"

 All are Windows 7 Ultimate x64


 "How much free disk space is there?"

 Machine 1 has around 26 GB free. Machine 2 has around 40 GB free.

Virtualbox guest has only has around 10 GB free.

 "What else is your application code doing?"

 This is silly. My application was trying to open an env on windows. Here

is the code for it.

   #define MB * 1024 * 1024
   #define GB * 1024 * 1024 * 1024

   MDB_env *env = 0;

   bool openEnv() {
       mdb_size_t mapSize = 390 MB;
       MDB_dbi maxNamedDatabases = 100;


       if (mdb_env_create(&env) != MDB_SUCCESS)
       return false;

       if (mdb_env_set_mapsize(env, mapSize) != MDB_SUCCESS)
       return false;

       if (mdb_env_set_maxdbs(env, maxNamedDatabases) != MDB_SUCCESS)
       return false;

       int err = mdb_env_open(env, "test", 0, 0664);

       if (err != MDB_SUCCESS) {
       fprintf(stderr, "Error opening env %i", err);
       return false;
       }
       fprintf(stderr, "Success opening env %i", err);

       return true;
   }

 int main() {

     if (!openEnv())
       return 1;

 }




 "How do you know that only XX MB were "opened"

 Trial and error on windows showed that mdb_env_open returns success when

I give a mapSize of 390 MB on Machine 1, etc... But on further testing, we can open 2 GB sometimes. But after a system restart, situation is still the same.

 "what stats are you checking? "

 Well as you can see from the above code, I am checking the return value

of mdb_env_open().

 "What other differences exist between machine 1 and machine 2?"

 Not much. One being a core-i3 and the other core-i5. Both have 4 GB RAM.

Both are similiar in that none of them have so many applications installed and all have used memory around 500 MB

 "What other processes are running,"

  As I stated its mostly a default installation of Windows x64 on all

machines.

 "how much memory are they consuming on each machine?"

  Are you referring to other processes or my application. Assuming you are

talking about my application, it is just a main function. So that might be irrelevant

 "Note that just because you create a 1GB memory map of a file doesn't

mean that it will actually consume 1GB of RAM. That's not how memory mapping works."

  Never said that. Didn't even imply it. Don't know where you are getting

this. I know how memory map works. Having implemented one in a custom hobby OS 5 years ago. Your implementation leveraging COW along with performance and MVCC was the whole reason I was interested in your project.

  "Read your Windows developer documentation, MSDN is always there for you

to read. We're not here to teach you how your OS works."

  You don't have to teach me how my OS works. Sorry if I gave you that

impression.

Apparently we do. See below.

  "Hangs where, exactly? Have you gotten a stack trace from such a hang?

That's the obvious first step that any application developer should know to do."

  On windows, if I dont set env_set_mapsize(env, 0) just before beginning

a write transaction, then first insert will complete successfully. On the second write transaction while the application is running, and we try to for eg: mdb_cursor_get(), it doesn't return. This can be reproduced with the below stated toolchain.

  -----------

  For further notes, I am using official QtSDK-mingw on windows which is a

32 bit compiler. But sometimes I can get LMDB to open the env succesffully for 2 gb, etc. But on a system restart, the situation is still the same.

You can't address more than 2GB of memory in 32 bit code. You generally won't be able to create a 2GB mmap in a 32 bit process because other DLLs will already be occupying large chunks of the 32 bit address space.

  But I downloaded an unofficial QtSDK-mingw-64 with the same result. But
  when I change
 mdb_size_t mapSize = 2147483648;    // 2 GB
  to
 mdb_size_t mapSize = 2147483648LL;

 it works.


 But setting 2147483648LL does not work (meaning does not return

MDB_SUCCESS) most of the times on mingw-32-bit

That is expected. Again, you should already know how your OS works.

 So the quesstion really is how to get it working when compiled on a

32-bit compiler and executed on both 32 and 64 bit Windows?

 Note: I am not referring to vl32 branch because I don't want to have to

use more than 2GB as the size of my env file.

It can't be done.

On Mon, Dec 28, 2015 at 2:17 PM, Howard Chu <hyc@symas.com mailto:hyc@symas.com> wrote:

Mohammed Muneer wrote:


    I can env_set_mapsize(env, 1 GB) without any problems in linux.

    But on windows, it is a different story


Windows is like that.

       Machine 1) Windows 64 only opens with upto 10 MB (apporx)
       Machine 2 Windows 64 only opens with upto 500 MB (apporx)
       Machine 3) Virtual box Windows 64 running on linux works with 1 GB

    Q1) So why is this happening. Am I doing anything wrong.


Have no idea what you're actually doing, so it's impossible to answer.
Learn to ask better questions and provide better information.

What revision of LMDB? Which version of Windows? How much free disk space
is there? What else is your application code doing? How do you know that
only XX MB were "opened", what stats are you checking? What other
differences exist between machine 1 and machine 2?

What other processes are running, how much memory are they consuming on
each machine? Note that just because you create a 1GB memory map of a file
doesn't mean that it will actually consume 1GB of RAM. That's not how
memory mapping works. Read your Windows developer documentation, MSDN is
always there for you to read. We're not here to teach you how your OS works.

    Q2) And setting env_set_mapsize before calling open is not enough. I
    need to set
            env_set_mapsize(env, 0) just before beginning a write transaction
    inorder to
           insert something or otherwise the application indefinitely
    hangs on the
    second insert transaction.


Hangs where, exactly? Have you gotten a stack trace from such a hang?
That's the obvious first step that any application developer should know
to do.


    Both the above questions concerns just Windows. Everything is fine
    with linux.