On 8/28/07, Quanah Gibson-Mount quanah@zimbra.com wrote:
--On Tuesday, August 28, 2007 4:10 PM +0100 Gavin Henry gavin.henry@gmail.com wrote:
On 28/08/07, Sam Tran stlist@gmail.com wrote:
Hi All,
Does OpenLDAP 2.3.38 support the new Berkeley DB 4.6.19?
Nope, only CVS HEAD does just now.
I suggest reading the discussion about 4.6.19 on the -devel list. Especially, note the serious degradation to performance that occurs when using it.
That's an unfortunate development from oracle considering the presentation that was released showing bdb4.6 to be very fast.
Here's a link to that thread, fyi: http://www.openldap.org/lists/openldap-devel/200708/msg00040.html
--On Tuesday, August 28, 2007 2:44 PM -0400 matthew sporleder msporleder@gmail.com wrote:
That's an unfortunate development from oracle considering the presentation that was released showing bdb4.6 to be very fast.
Yup. I believe Howard is working with Sleepycat to find out what was done to 4.6 to make it suddenly be 3-4 times slower instead of significantly faster as it was in the early releases.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
--On Tuesday, August 28, 2007 2:44 PM -0400 matthew sporleder msporleder@gmail.com wrote:
That's an unfortunate development from oracle considering the presentation that was released showing bdb4.6 to be very fast.
Yup. I believe Howard is working with Sleepycat to find out what was done to 4.6 to make it suddenly be 3-4 times slower instead of significantly faster as it was in the early releases.
Yes, it's been a long conversation so far. While their new memory manager (inspired by Jong-Hyuk Choi's research on BerkeleyDB performance in OpenLDAP) is a great improvement, their new lock manager is not.
This may not be an issue on all platforms. On x86 though, they have a hybrid mutex mechanism which is enabled by default. It uses both assembly language spinlocks and pthread mutexes, first spinning in the assembly language lock some number of times before falling back onto the mutex. They claimed that this improved performance in their tests, because pthread mutexes can be very expensive on some platforms. In my tests on x86_64 Linux however, it just forced CPU usage to 100% (200% actually, dualcore) and slowed down overall throughput.
You can give an explicit "--with-mutex=POSIX/pthread" argument when configuring BerkeleyDB to avoid the hybrid mutex scheme, in which case performance of BDB 4.6.19 seems to match what we obtained with BDB 4.6.3.
If you were using BerkeleyDB 4.5.20 before, the early BDB 4.6 releases will work well for you. The only difference between 4.5.20 and 4.6.1 was in the memory manager. 4.6.2 just tweaked some portability issues to support BREW (Qualcomm's cellphone programming environment). 4.6.3 added the ability to specify different cache priorities per database operation. (I didn't track what changed between 4.6.3 and 4.6.18.)
And for anyone curious - you can read ITS#3851 http://www.openldap.org/its/index.cgi/Development?id=3851 for the background on the problems in BerkeleyDB's memory allocator. While there was a fair amount of debate as to whether Jong's proposed solution was of any benefit, it was pretty clear that the existing code in BerkeleyDB could be improved. But the changes in the memory allocator may not visibly affect you if you're running a small-to-medium sized database. It's only apparent when the total volume of data is much larger than the BerkeleyDB cache, because that's the condition that leads to memory fragmentation, and it's the fragmentation that causes the allocator's performance to degrade.
The same kind of problem affects the slapd entry cache, when the number of active entries is much larger than the entry cache, and you're using the glibc malloc. (At least, for glibc 2.3 and older. I haven't retested with glibc 2.5 yet.)
To sum up - if you're already using BDB 4.5.20 with OpenLDAP 2.3 and the performance is acceptable, I wouldn't be in any hurry to upgrade to BDB 4.6. Releases 4.6.18 and 4.6.19 are incompatible with OpenLDAP 2.3. BDB 4.6.1 is a drop-in replacement for 4.5.20 though.
On 8/28/07, Howard Chu hyc@symas.com wrote:
Quanah Gibson-Mount wrote:
--On Tuesday, August 28, 2007 2:44 PM -0400 matthew sporleder msporleder@gmail.com wrote:
That's an unfortunate development from oracle considering the presentation that was released showing bdb4.6 to be very fast.
Yup. I believe Howard is working with Sleepycat to find out what was done to 4.6 to make it suddenly be 3-4 times slower instead of significantly faster as it was in the early releases.
Yes, it's been a long conversation so far. While their new memory manager (inspired by Jong-Hyuk Choi's research on BerkeleyDB performance in OpenLDAP) is a great improvement, their new lock manager is not.
This may not be an issue on all platforms. On x86 though, they have a hybrid mutex mechanism which is enabled by default. It uses both assembly language spinlocks and pthread mutexes, first spinning in the assembly language lock some number of times before falling back onto the mutex. They claimed that this improved performance in their tests, because pthread mutexes can be very expensive on some platforms. In my tests on x86_64 Linux however, it just forced CPU usage to 100% (200% actually, dualcore) and slowed down overall throughput.
You can give an explicit "--with-mutex=POSIX/pthread" argument when configuring BerkeleyDB to avoid the hybrid mutex scheme, in which case performance of BDB 4.6.19 seems to match what we obtained with BDB 4.6.3.
If you were using BerkeleyDB 4.5.20 before, the early BDB 4.6 releases will work well for you. The only difference between 4.5.20 and 4.6.1 was in the memory manager. 4.6.2 just tweaked some portability issues to support BREW (Qualcomm's cellphone programming environment). 4.6.3 added the ability to specify different cache priorities per database operation. (I didn't track what changed between 4.6.3 and 4.6.18.)
And for anyone curious - you can read ITS#3851 http://www.openldap.org/its/index.cgi/Development?id=3851 for the background on the problems in BerkeleyDB's memory allocator. While there was a fair amount of debate as to whether Jong's proposed solution was of any benefit, it was pretty clear that the existing code in BerkeleyDB could be improved. But the changes in the memory allocator may not visibly affect you if you're running a small-to-medium sized database. It's only apparent when the total volume of data is much larger than the BerkeleyDB cache, because that's the condition that leads to memory fragmentation, and it's the fragmentation that causes the allocator's performance to degrade.
The same kind of problem affects the slapd entry cache, when the number of active entries is much larger than the entry cache, and you're using the glibc malloc. (At least, for glibc 2.3 and older. I haven't retested with glibc 2.5 yet.)
To sum up - if you're already using BDB 4.5.20 with OpenLDAP 2.3 and the performance is acceptable, I wouldn't be in any hurry to upgrade to BDB 4.6. Releases 4.6.18 and 4.6.19 are incompatible with OpenLDAP 2.3. BDB 4.6.1 is a drop-in replacement for 4.5.20 though.
What do you think your chances are at getting --with-mutex=POSIX/pthread made the default? If not very good, it might be worth mentioning in openldap docs if it's that much of a performance gain.
matthew sporleder wrote:
What do you think your chances are at getting --with-mutex=POSIX/pthread made the default? If not very good, it might be worth mentioning in openldap docs if it's that much of a performance gain.
It's too early to tell yet. We're still working on identifying why the hybrid mutex scheme performs so poorly. In fact the hybrid scheme has been around for a long time, in several other BDB releases. But somewhere along the line in 4.6 they changed the actual sequence of assembly language instructions used on x86 (to correct a bug). It's possible that the old sequence was always broken, so it never performed any actual locking in prior releases, or that the new sequence is broken, or that the new sequence is working as designed, but is just causing more bus contention than expected.
Testing on other platforms may help pinpoint the problem. In general though, I think current Linux kernels already have an extremely efficient mutex implementation, so there's not much point in using the assembly language spinlock here.
And for anyone curious - you can read ITS#3851 http://www.openldap.org/its/index.cgi/Development?id=3851 for the background on the problems in BerkeleyDB's memory allocator. While there was a fair amount of debate as to whether Jong's proposed solution was of any benefit, it was pretty clear that the existing code in BerkeleyDB could be improved. But the changes in the memory allocator may not visibly affect you if you're running a small-to-medium sized database. It's only apparent when the total volume of data is much larger than the BerkeleyDB cache, because that's the condition that leads to memory fragmentation, and it's the fragmentation that causes the allocator's performance to degrade.
To sum up - if you're already using BDB 4.5.20 with OpenLDAP 2.3 and the performance is acceptable, I wouldn't be in any hurry to upgrade to BDB 4.6. Releases 4.6.18 and 4.6.19 are incompatible with OpenLDAP 2.3. BDB 4.6.1 is a drop-in replacement for 4.5.20 though.
So currently the recommended BDB version to use with 2.3 is either 4.5.20 or 4.6.1 (or good old 4.2.52 + patches).
Ok, thanks.
openldap-software@openldap.org
-
Gavin Henry -
Howard Chu -
matthew sporleder -
Quanah Gibson-Mount