Hello,
we would like to change our backend from hdb to mdb.
I did some initial tests and found out the following: When setting maxsize to 1.4 GB, the size of the database on the file system is about 1.4 GB. I thought I read that the maxsize has nothing to do with the actual physical size needed. (If I have only 10 MB of data my database should be about 10 MB). Even worse is the fact, that slapd.exe then needs about 1.5 GB virtual size memory. And if I use 32 Bit slapd.exe the process will crash when reaching 2 GB virtual size. (as every 32 bit process will do)
Now my conclusion is, that the things described in the mdb paper are only valid for unix/linux, because windows uses another memory system. Is this correct? Or is there a way I haven't found to configure OpenLDAP on windows so that it does not need so much RAM. The size on the disk does not matter for us, but since we use only 32 bit slapd.exe the RAM does matter.
Best regards, Frank
Regarding size.
"If an application specifies a size for the file mapping object that is larger than the size of the actual named file on disk and if the page protection allows write access (that is, the flProtect parameter specifies PAGE_READWRITE or PAGE_EXECUTE_READWRITE), then the file on disk is increased to match the specified size of the file mapping object. If the file is extended, the contents of the file between the old end of the file and the new end of the file are not guaranteed to be zero; the behavior is defined by the file system. If the file on disk cannot be increased, CreateFileMapping fails and GetLastError returns ERROR_DISK_FULL."
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366537%28v=vs.85%2...
On Wed, Jan 28, 2015 at 3:27 PM, Frank Offermanns < Frank.Offermanns@caseris.de> wrote:
Hello,
we would like to change our backend from hdb to mdb.
I did some initial tests and found out the following: When setting maxsize to 1.4 GB, the size of the database on the file system is about 1.4 GB. I thought I read that the maxsize has nothing to do with the actual physical size needed. (If I have only 10 MB of data my database should be about 10 MB). Even worse is the fact, that slapd.exe then needs about 1.5 GB virtual size memory. And if I use 32 Bit slapd.exe the process will crash when reaching 2 GB virtual size. (as every 32 bit process will do)
Now my conclusion is, that the things described in the mdb paper are only valid for unix/linux, because windows uses another memory system. Is this correct? Or is there a way I haven't found to configure OpenLDAP on windows so that it does not need so much RAM. The size on the disk does not matter for us, but since we use only 32 bit slapd.exe the RAM does matter.
Best regards, Frank
Hi,
thanks for the information, but unfortunately this information is not helpfull for me.
Is there a way to configure LMDB on Windows 32 bit, so that a database can be larger? Or is it mandantory to use 64 bit with lmdb (at least on windows file systems) when handling larger data?
Regards, Frank
Von: Kristoffer Sjögren stoffe@gmail.com An: Frank Offermanns Frank.Offermanns@caseris.de, Kopie: openldap-technical@openldap.org Datum: 28.01.2015 16:32 Betreff: Re: LMDB usage on windows - to much memory needed
Regarding size.
"If an application specifies a size for the file mapping object that is larger than the size of the actual named file on disk and if the page protection allows write access (that is, the flProtect parameter specifies PAGE_READWRITE or PAGE_EXECUTE_READWRITE), then the file on disk is increased to match the specified size of the file mapping object. If the file is extended, the contents of the file between the old end of the file and the new end of the file are not guaranteed to be zero; the behavior is defined by the file system. If the file on disk cannot be increased, CreateFileMapping fails and GetLastError returns ERROR_DISK_FULL."
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366537%28v=vs.85%2...
On Wed, Jan 28, 2015 at 3:27 PM, Frank Offermanns < Frank.Offermanns@caseris.de> wrote: Hello,
we would like to change our backend from hdb to mdb.
I did some initial tests and found out the following: When setting maxsize to 1.4 GB, the size of the database on the file system is about 1.4 GB. I thought I read that the maxsize has nothing to do with the actual physical size needed. (If I have only 10 MB of data my database should be about 10 MB). Even worse is the fact, that slapd.exe then needs about 1.5 GB virtual size memory. And if I use 32 Bit slapd.exe the process will crash when reaching 2 GB virtual size. (as every 32 bit process will do)
Now my conclusion is, that the things described in the mdb paper are only valid for unix/linux, because windows uses another memory system. Is this correct? Or is there a way I haven't found to configure OpenLDAP on windows so that it does not need so much RAM. The size on the disk does not matter for us, but since we use only 32 bit slapd.exe the RAM does matter.
Best regards, Frank
Hi,
On Thu, 29 Jan 2015, Frank Offermanns wrote:
Hi,
thanks for the information, but unfortunately this information is not helpfull for me.
Is there a way to configure LMDB on Windows 32 bit, so that a database can be larger? Or is it mandantory to use 64 bit with lmdb (at least on windows file systems) when handling larger data?
You need 64 bits on any kind of OS to support multi gigabyte databases.
No 32 bit OS will allow a single process to memory map much more than 1 or perhaps 2 GB of address space.
Greetings Christian
Regards, Frank
Von: Kristoffer Sjögren stoffe@gmail.com An: Frank Offermanns Frank.Offermanns@caseris.de, Kopie: openldap-technical@openldap.org Datum: 28.01.2015 16:32 Betreff: Re: LMDB usage on windows - to much memory needed
Regarding size.
"If an application specifies a size for the file mapping object that is larger than the size of the actual named file on disk and if the page protection allows write access (that is, the flProtect parameter specifies PAGE_READWRITE or PAGE_EXECUTE_READWRITE), then the file on disk is increased to match the specified size of the file mapping object. If the file is extended, the contents of the file between the old end of the file and the new end of the file are not guaranteed to be zero; the behavior is defined by the file system. If the file on disk cannot be increased, CreateFileMapping fails and GetLastError returns ERROR_DISK_FULL."
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366537%28v=vs.85%2...
On Wed, Jan 28, 2015 at 3:27 PM, Frank Offermanns < Frank.Offermanns@caseris.de> wrote: Hello,
we would like to change our backend from hdb to mdb.
I did some initial tests and found out the following: When setting maxsize to 1.4 GB, the size of the database on the file system is about 1.4 GB. I thought I read that the maxsize has nothing to do with the actual physical size needed. (If I have only 10 MB of data my database should be about 10 MB). Even worse is the fact, that slapd.exe then needs about 1.5 GB virtual size memory. And if I use 32 Bit slapd.exe the process will crash when reaching 2 GB virtual size. (as every 32 bit process will do)
Now my conclusion is, that the things described in the mdb paper are only valid for unix/linux, because windows uses another memory system. Is this correct? Or is there a way I haven't found to configure OpenLDAP on windows so that it does not need so much RAM. The size on the disk does not matter for us, but since we use only 32 bit slapd.exe the RAM does matter.
Best regards, Frank
Hello Christian,
thanks for the answer.
What you say is true, but with hdb backend I could use larger databases, because it has no memory map file, hasn't it?
So I hoped, that I might have missed a configuration parameter which would have made mdb be more usefull for 32 bit. But it seems, that I haven't missed such a parameter, so I have to use 64 bit openldap (or use smaller databases).
Best regards, Frank
Von: Christian Kratzer ck-lists@cksoft.de An: Frank Offermanns Frank.Offermanns@caseris.de, Kopie: Kristoffer Sjögren stoffe@gmail.com, openldap-technical@openldap.org Datum: 29.01.2015 14:55 Betreff: Re: Re: LMDB usage on windows - to much memory needed
Hi,
On Thu, 29 Jan 2015, Frank Offermanns wrote:
Hi,
thanks for the information, but unfortunately this information is not helpfull for me.
Is there a way to configure LMDB on Windows 32 bit, so that a database
can
be larger? Or is it mandantory to use 64 bit with lmdb (at least on windows file systems) when handling larger data?
You need 64 bits on any kind of OS to support multi gigabyte databases.
No 32 bit OS will allow a single process to memory map much more than 1 or perhaps 2 GB of address space.
Greetings Christian
Regards, Frank
Von: Kristoffer Sjögren stoffe@gmail.com An: Frank Offermanns Frank.Offermanns@caseris.de, Kopie: openldap-technical@openldap.org Datum: 28.01.2015 16:32 Betreff: Re: LMDB usage on windows - to much memory needed
Regarding size.
"If an application specifies a size for the file mapping object that is larger than the size of the actual named file on disk and if the page protection allows write access (that is, the flProtect parameter
specifies
PAGE_READWRITE or PAGE_EXECUTE_READWRITE), then the file on disk is increased to match the specified size of the file mapping object. If the file is extended, the contents of the file between the old end of the
file
and the new end of the file are not guaranteed to be zero; the behavior
is
defined by the file system. If the file on disk cannot be increased, CreateFileMapping fails and GetLastError returns ERROR_DISK_FULL."
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366537%28v=vs.85%2...
On Wed, Jan 28, 2015 at 3:27 PM, Frank Offermanns < Frank.Offermanns@caseris.de> wrote: Hello,
we would like to change our backend from hdb to mdb.
I did some initial tests and found out the following: When setting maxsize to 1.4 GB, the size of the database on the file system is about 1.4 GB. I thought I read that the maxsize has nothing to do with the actual physical size needed. (If I have only 10 MB of data
my
database should be about 10 MB). Even worse is the fact, that slapd.exe then needs about 1.5 GB virtual size memory. And if I use 32 Bit slapd.exe the process will crash when reaching 2 GB virtual size. (as every 32 bit process will do)
Now my conclusion is, that the things described in the mdb paper are
only
valid for unix/linux, because windows uses another memory system. Is
this
correct? Or is there a way I haven't found to configure OpenLDAP on windows so that it does not need so much RAM. The size on the disk does not matter for us, but since we use only 32
bit
slapd.exe the RAM does matter.
Best regards, Frank
Frank Offermanns wrote:
Hello Christian,
thanks for the answer.
What you say is true, but with hdb backend I could use larger databases, because it has no memory map file, hasn't it?
So I hoped, that I might have missed a configuration parameter which would have made mdb be more usefull for 32 bit.
32 bit servers were obsolete at least 5 years ago, maybe 10 years.
But it seems, that I haven't missed such a parameter, so I have to use 64 bit openldap (or use smaller databases).
I am philosophically opposed to supporting 32 bit architectures in LMDB. http://www.openldap.org/lists/openldap-devel/201405/msg00014.html
However, for the few people who insist, there is an experimental branch available that can support larger DBs on 32 bit CPUs, by unmapping and remapping segments of the DB.
https://gitorious.org/mdb/mdb/source/69d7cb8d44e04f02d8d0c923ae71fbaaa9f42f3...:
Due to the added system calls involved, it is significantly slower. We have plans to merge it into LMDB 1.0 as an optional feature, but for the moment it's only in that branch.
Fundamentally, supporting this feature in LMDB is the wrong thing to do. It adds code/bloat and slows down the overall codebase just in order to support a dying technology. Even in embedded processing 64 bit CPUs are available cheaply now; anyone still pouring money into 32 bit CPUs needs to have their head examined.
Best regards, Frank
Von: Christian Kratzer ck-lists@cksoft.de An: Frank Offermanns Frank.Offermanns@caseris.de, Kopie: Kristoffer Sjögren stoffe@gmail.com, openldap-technical@openldap.org Datum: 29.01.2015 14:55 Betreff: Re: Re: LMDB usage on windows - to much memory needed
Hi,
On Thu, 29 Jan 2015, Frank Offermanns wrote:
Hi,
thanks for the information, but unfortunately this information is not helpfull for me.
Is there a way to configure LMDB on Windows 32 bit, so that a
database can
be larger? Or is it mandantory to use 64 bit with lmdb (at least on windows file systems) when handling larger data?
You need 64 bits on any kind of OS to support multi gigabyte databases.
No 32 bit OS will allow a single process to memory map much more than 1 or perhaps 2 GB of address space.
Greetings Christian
Regards, Frank
Von: Kristoffer Sjögren stoffe@gmail.com An: Frank Offermanns Frank.Offermanns@caseris.de, Kopie: openldap-technical@openldap.org Datum: 28.01.2015 16:32 Betreff: Re: LMDB usage on windows - to much memory needed
Regarding size.
"If an application specifies a size for the file mapping object that is larger than the size of the actual named file on disk and if the page protection allows write access (that is, the flProtect parameter
specifies
PAGE_READWRITE or PAGE_EXECUTE_READWRITE), then the file on disk is increased to match the specified size of the file mapping object. If the file is extended, the contents of the file between the old end of the
file
and the new end of the file are not guaranteed to be zero; the
behavior is
defined by the file system. If the file on disk cannot be increased, CreateFileMapping fails and GetLastError returns ERROR_DISK_FULL."
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366537%28v=vs.85%2...
On Wed, Jan 28, 2015 at 3:27 PM, Frank Offermanns < Frank.Offermanns@caseris.de> wrote: Hello,
we would like to change our backend from hdb to mdb.
I did some initial tests and found out the following: When setting maxsize to 1.4 GB, the size of the database on the file system is about 1.4 GB. I thought I read that the maxsize has nothing to do with the actual physical size needed. (If I have only 10 MB of data my database should be about 10 MB). Even worse is the fact, that slapd.exe then needs about 1.5 GB virtual size memory. And if I use 32 Bit slapd.exe the process will crash when reaching 2 GB virtual size. (as every 32 bit process will do)
Now my conclusion is, that the things described in the mdb paper are only valid for unix/linux, because windows uses another memory system. Is this correct? Or is there a way I haven't found to configure OpenLDAP on windows so that it does not need so much RAM. The size on the disk does not matter for us, but since we use only 32 bit slapd.exe the RAM does matter.
Best regards, Frank
-- Christian Kratzer CK Software GmbH Email: ck@cksoft.de Wildberger Weg 24/2 Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian Kratzer Web: http://www.cksoft.de/
openldap-technical@openldap.org
-
Christian Kratzer -
Frank Offermanns -
Howard Chu -
Kristoffer Sjögren