I've started testing an LDAP server here using MDB and ran across a few caveats that might be of use to others looking into using it. But first off, let me say a hearty THANKS to anyone who's contributed to it. In this first OpenLDAP server I've converted over to MDB it's *dramatically* faster and it's definitely nice to not worry about having to setup script/s to occasionally (carefully) commit/flush DB logs, etc.
One caveat that might be worth mentioning in release notes somewhere... Not all implementations of memory mapped I/O are created equal. I ran into this a long time back when I wrote a multi-threaded quicksort program for a friend to had to sort text files bigger than 10 gigs and didn't want to wait for the unix sort command. :-) The program I banged together for him used memory mapped I/O and one of the things I found was that while Solaris would let me memory map a file bigger than I had physical or virtual memory for, linux wouldn't. It appeared that some versions of the 2.x kernels wouldn't let me memory-map a file bigger than the total *virtual* memory size, and I think MDB is running into the same limitation. On a SLES11 system, for instance with the 2.6.32.12 kernel, I can't specify a maxsize bigger than the total of my physical memory and swap space. So just something to keep in mind if you're using MDB on the 2.x kernels - you may need a big swap area even though the memory mapped I/O routines in the kernel seem to be smart enough to avoid swapping like mad.
On a newish ubuntu system with a 3.5 kernel this doesn't seem to be an issue - tell OpenLDAP to use whatever maxsize you want and it just works. :-)
I'd also only use MDB on a 64 bit linux system. One of the other headaches I remember running into with memory mapped I/O was adding in support for 64 bit I/O on 32 bit systems. Best to avoid that whole mess and just use a 64 bit OS in the first place.
Lastly... At the risk of making Howard and Quanah cringe... :-) The OpenLDAP DB I've been testing this with is the back-end to an email tracking tool I setup several years ago. More as an excuse to edjimicate myself on the java API for LDAP than anything else, I wrote a quick bit of java that watches postfix and sendmail logs and writes pertinent bits of info into an LDAP database, and a few PHP scripts to then query that database for things like to/from addresses, queue IDs, and message IDs. 'Makes it easy for junior admins to quickly search through gigabytes of logs to see what path an email took to get from point A to point B, who all received it (after it went through one or more list servers and a few aliases got de-ref'd, etc).
Yeah, it's an utter abuse of LDAP which is supposed to be write-rarely and read-mostly, especially as our postfix relays handle anywhere from 1 to 10 messages per second on average. :-) But what the heck, it works fine and was a fun weekend project. It's also served as a way to stress-test new versions of OpenLDAP before I deploy them elsewhere. :-)
Anyway, thanks again to everyone who contributed to MDB. It's lots faster than BerkeleyDB in all of my testing so far. 'Looking forward to gradually shifting more of my LDAP servers over to it.
Brent
--On Wednesday, September 18, 2013 2:01 PM -0600 Brent Bice bbice@sgi.com wrote:
On a SLES11 system, for instance with the 2.6.32.12 kernel
This is because the SLES folks seem to love to seek out ways to break things for servers. I'm not really sure why they call themselves an "enterprise server" product anymore. It is trivial to fix this:
ulimit -v unlimited
On a newish ubuntu system with a 3.5 kernel this doesn't seem to bean issue - tell OpenLDAP to use whatever maxsize you want and it just works. :-)
There's a known issue in 3.0 through 3.9 kernels that is fixed in 3.10 that will affect your write speed with mdb. See:
http://www.openldap.org/lists/openldap-devel/201309/msg00008.html
I have 3.10 running on my Ubuntu12 box now, it is a significant improvement. It also appears this breakage was backported on some Linux OSes into their 2.6 kernels.
I'd also only use MDB on a 64 bit linux system. One of the otherheadaches I remember running into with memory mapped I/O was adding in support for 64 bit I/O on 32 bit systems. Best to avoid that whole mess and just use a 64 bit OS in the first place.
Correct.
Anyway, thanks again to everyone who contributed to MDB. It's lotsfaster than BerkeleyDB in all of my testing so far. 'Looking forward to gradually shifting more of my LDAP servers over to it.
For me, the back-mdb write speed is about 50x to 70x faster than the BDB based backends.
http://wiki.zimbra.com/wiki/OpenLDAP_MDB_vs_HDB_performance
Also quite a bit faster for reads:
http://mishikal.wordpress.com/2013/05/16/openldap-a-comparison-of-back-mdb-and-back-hdb-performance/
Glad to hear MDB is working out well for you!
--Quanah
--
Quanah Gibson-Mount Lead Engineer Zimbra Software, LLC -------------------- Zimbra :: the leader in open source messaging and collaboration
Brent Bice wrote:
I've started testing an LDAP server here using MDB and ran across afew caveats that might be of use to others looking into using it. But first off, let me say a hearty THANKS to anyone who's contributed to it. In this first OpenLDAP server I've converted over to MDB it's *dramatically* faster and it's definitely nice to not worry about having to setup script/s to occasionally (carefully) commit/flush DB logs, etc.
One caveat that might be worth mentioning in release notessomewhere... Not all implementations of memory mapped I/O are created equal. I ran into this a long time back when I wrote a multi-threaded quicksort program for a friend to had to sort text files bigger than 10 gigs and didn't want to wait for the unix sort command. :-) The program I banged together for him used memory mapped I/O and one of the things I found was that while Solaris would let me memory map a file bigger than I had physical or virtual memory for, linux wouldn't. It appeared that some versions of the 2.x kernels wouldn't let me memory-map a file bigger than the total *virtual* memory size, and I think MDB is running into the same limitation. On a SLES11 system, for instance with the 2.6.32.12 kernel, I can't specify a maxsize bigger than the total of my physical memory and swap space. So just something to keep in mind if you're using MDB on the 2.x kernels - you may need a big swap area even though the memory mapped I/O routines in the kernel seem to be smart enough to avoid swapping like mad.
Some Linux distros also ship with a default VM ulimit that will get in your way, but Linux kernel "overcommit" settings have been around for a long time. I don't think what you saw was particular to the 2.6.32 kernel, most likely related to your distro's default config.
On a newish ubuntu system with a 3.5 kernel this doesn't seem to bean issue - tell OpenLDAP to use whatever maxsize you want and it just works. :-)
I'd also only use MDB on a 64 bit linux system. One of the otherheadaches I remember running into with memory mapped I/O was adding in support for 64 bit I/O on 32 bit systems. Best to avoid that whole mess and just use a 64 bit OS in the first place.
We definitely don't recommend using 32 bit servers with LMDB.
Lastly... At the risk of making Howard and Quanah cringe... :-) TheOpenLDAP DB I've been testing this with is the back-end to an email tracking tool I setup several years ago. More as an excuse to edjimicate myself on the java API for LDAP than anything else, I wrote a quick bit of java that watches postfix and sendmail logs and writes pertinent bits of info into an LDAP database, and a few PHP scripts to then query that database for things like to/from addresses, queue IDs, and message IDs. 'Makes it easy for junior admins to quickly search through gigabytes of logs to see what path an email took to get from point A to point B, who all received it (after it went through one or more list servers and a few aliases got de-ref'd, etc).
Yeah, it's an utter abuse of LDAP which is supposed to bewrite-rarely and read-mostly, especially as our postfix relays handle anywhere from 1 to 10 messages per second on average. :-) But what the heck, it works fine and was a fun weekend project. It's also served as a way to stress-test new versions of OpenLDAP before I deploy them elsewhere. :-)
Anyway, thanks again to everyone who contributed to MDB. It's lotsfaster than BerkeleyDB in all of my testing so far. 'Looking forward to gradually shifting more of my LDAP servers over to it.
You're welcome.
On 09/18/2013 02:26 PM, Quanah Gibson-Mount wrote:
This is because the SLES folks seem to love to seek out ways to break things for servers. I'm not really sure why they call themselves an "enterprise server" product anymore. It is trivial to fix this:
ulimit -v unlimited
I didn't think it was a ulimit issue (pretty sure I checked this) but I'll double-check later today on a different OpenLDAP server running SLES11. :-)
There's a known issue in 3.0 through 3.9 kernels that is fixed in 3.10 that will affect your write speed with mdb. See:
http://www.openldap.org/lists/openldap-devel/201309/msg00008.html
Hmm, thanks for the tip. 'Didn't know about that. I know there were some significant performance fixes in XFS somewhere in the 3.x line of kernels too. Just 'nother reason to upgrade though I don't think I have any distros (yet) using a kernel newer than 3.5. :-/
Brent
Anyway, thanks again to everyone who contributed to MDB. It's lotsfaster than BerkeleyDB in all of my testing so far. 'Looking forward to gradually shifting more of my LDAP servers over to it.
For me, the back-mdb write speed is about 50x to 70x faster than the BDB based backends.
http://wiki.zimbra.com/wiki/OpenLDAP_MDB_vs_HDB_performance
Also quite a bit faster for reads:
http://mishikal.wordpress.com/2013/05/16/openldap-a-comparison-of-back-mdb-and-back-hdb-performance/
Glad to hear MDB is working out well for you!
--Quanah
--
Quanah Gibson-Mount Lead Engineer Zimbra Software, LLC
Zimbra :: the leader in open source messaging and collaboration
--On Wednesday, September 18, 2013 2:41 PM -0600 Brent Bice bbice@sgi.com wrote:
On 09/18/2013 02:26 PM, Quanah Gibson-Mount wrote:
This is because the SLES folks seem to love to seek out ways to break things for servers. I'm not really sure why they call themselves an "enterprise server" product anymore. It is trivial to fix this:
ulimit -v unlimited
I didn't think it was a ulimit issue (pretty sure I checked this) butI'll double-check later today on a different OpenLDAP server running SLES11. :-)
Oh, we hit this issue immediately in Zimbra 8 for SLES11 when we moved everything to mdb. ;) Which is why I know the fix so well. :P
--Quanah
--
Quanah Gibson-Mount Lead Engineer Zimbra Software, LLC -------------------- Zimbra :: the leader in open source messaging and collaboration
>>> Brent Bice bbice@sgi.com schrieb am 18.09.2013 um 22:01 in Nachricht 523A068F.1000100@sgi.com:
I've started testing an LDAP server here using MDB and ran across a few caveats that might be of use to others looking into using it. But first off, let me say a hearty THANKS to anyone who's contributed to it. In this first OpenLDAP server I've converted over to MDB it's *dramatically* faster and it's definitely nice to not worry about having to setup script/s to occasionally (carefully) commit/flush DB logs, etc.
One caveat that might be worth mentioning in release notessomewhere... Not all implementations of memory mapped I/O are created equal. I ran into this a long time back when I wrote a multi-threaded quicksort program for a friend to had to sort text files bigger than 10 gigs and didn't want to wait for the unix sort command. :-) The program I banged together for him used memory mapped I/O and one of the things I found was that while Solaris would let me memory map a file bigger than I had physical or virtual memory for, linux wouldn't. It appeared that
I doubt that Solaris allows you to mmap() a file to an area larger than the virtual address space, however you can mmap() a file area larger than RAM+swap when a demand paging strategy is used. However once you start modifying the mapped pages you may run out of memory, so thing twice.
some versions of the 2.x kernels wouldn't let me memory-map a file bigger than the total *virtual* memory size, and I think MDB is running into the same limitation. On a SLES11 system, for instance with the 2.6.32.12 kernel, I can't specify a maxsize bigger than the total of my physical memory and swap space. So just something to keep in mind if
Also be aware that in SLES11 SP2 the kernel update release some weeks ago strengthened the checks for mmap()ed areas: I had a program that started to fail when I tried to change one byte after the end of the file, while this worked with the kernel before.
you're using MDB on the 2.x kernels - you may need a big swap area even though the memory mapped I/O routines in the kernel seem to be smart enough to avoid swapping like mad.
I'd like to object: AFAIR, MDB used mmap()ed areas in strictly read-only fashion, so the backing store is the original file, being demand paged. When data is write()n, the system will dirty buffers in real RAM that are eventually written back to the file blocks. I see no path where dirty buffers should be swapped unless the mapping is PRIVATE.
On a newish ubuntu system with a 3.5 kernel this doesn't seem to bean issue - tell OpenLDAP to use whatever maxsize you want and it just works. :-)
I'd also only use MDB on a 64 bit linux system. One of the otherheadaches I remember running into with memory mapped I/O was adding in support for 64 bit I/O on 32 bit systems. Best to avoid that whole mess and just use a 64 bit OS in the first place.
For 32bit remember that the thread's stacks also consume virtual address space. So the maximum database size may be significantly below 4GB.
Lastly... At the risk of making Howard and Quanah cringe... :-) TheOpenLDAP DB I've been testing this with is the back-end to an email tracking tool I setup several years ago. More as an excuse to edjimicate myself on the java API for LDAP than anything else, I wrote a quick bit of java that watches postfix and sendmail logs and writes pertinent bits of info into an LDAP database, and a few PHP scripts to then query that database for things like to/from addresses, queue IDs, and message IDs. 'Makes it easy for junior admins to quickly search through gigabytes of logs to see what path an email took to get from point A to point B, who all received it (after it went through one or more list servers and a few aliases got de-ref'd, etc).
Yeah, it's an utter abuse of LDAP which is supposed to bewrite-rarely and read-mostly, especially as our postfix relays handle anywhere from 1 to 10 messages per second on average. :-) But what the heck, it works fine and was a fun weekend project. It's also served as a way to stress-test new versions of OpenLDAP before I deploy them elsewhere. :-)
Anyway, thanks again to everyone who contributed to MDB. It's lotsfaster than BerkeleyDB in all of my testing so far. 'Looking forward to gradually shifting more of my LDAP servers over to it.
Brent
Ulrich Windl wrote:
>>> Brent Bice <bbice@sgi.com> schrieb am 18.09.2013 um 22:01 in NachrichtI've started testing an LDAP server here using MDB and ran across a few caveats that might be of use to others looking into using it. But first off, let me say a hearty THANKS to anyone who's contributed to it. In this first OpenLDAP server I've converted over to MDB it's *dramatically* faster and it's definitely nice to not worry about having to setup script/s to occasionally (carefully) commit/flush DB logs, etc.
One caveat that might be worth mentioning in release notessomewhere... Not all implementations of memory mapped I/O are created equal. I ran into this a long time back when I wrote a multi-threaded quicksort program for a friend to had to sort text files bigger than 10 gigs and didn't want to wait for the unix sort command. :-) The program I banged together for him used memory mapped I/O and one of the things I found was that while Solaris would let me memory map a file bigger than I had physical or virtual memory for, linux wouldn't. It appeared that
I doubt that Solaris allows you to mmap() a file to an area larger than the
virtual address space, however you can mmap() a file area larger than RAM+swap when a demand paging strategy is used. However once you start modifying the mapped pages you may run out of memory, so thing twice.
No OS can let you mmap a single region larger than the address space. But mapping a file larger than RAM is no problem, the OS will swap pages in and out as needed.
some versions of the 2.x kernels wouldn't let me memory-map a file bigger than the total *virtual* memory size, and I think MDB is running into the same limitation. On a SLES11 system, for instance with the 2.6.32.12 kernel, I can't specify a maxsize bigger than the total of my physical memory and swap space. So just something to keep in mind if
Also be aware that in SLES11 SP2 the kernel update release some weeks ago
strengthened the checks for mmap()ed areas: I had a program that started to fail when I tried to change one byte after the end of the file, while this worked with the kernel before.
Irrelevant for LMDB since we never do such a thing.
you're using MDB on the 2.x kernels - you may need a big swap area even though the memory mapped I/O routines in the kernel seem to be smart enough to avoid swapping like mad.
I'd like to object: AFAIR, MDB used mmap()ed areas in strictly read-only
fashion, so the backing store is the original file, being demand paged. When data is write()n, the system will dirty buffers in real RAM that are eventually written back to the file blocks. I see no path where dirty buffers should be swapped unless the mapping is PRIVATE.
Correct; since LMDB uses an mmap'd file it will *never* use swap space.
openldap-technical@openldap.org
-
Brent Bice -
Howard Chu -
Quanah Gibson-Mount -
Ulrich Windl