Quanah Gibson-Mount wrote:
--On Tuesday, August 28, 2007 2:44 PM -0400 matthew sporleder msporleder@gmail.com wrote:
That's an unfortunate development from oracle considering the presentation that was released showing bdb4.6 to be very fast.
Yup. I believe Howard is working with Sleepycat to find out what was done to 4.6 to make it suddenly be 3-4 times slower instead of significantly faster as it was in the early releases.
Yes, it's been a long conversation so far. While their new memory manager (inspired by Jong-Hyuk Choi's research on BerkeleyDB performance in OpenLDAP) is a great improvement, their new lock manager is not.
This may not be an issue on all platforms. On x86 though, they have a hybrid mutex mechanism which is enabled by default. It uses both assembly language spinlocks and pthread mutexes, first spinning in the assembly language lock some number of times before falling back onto the mutex. They claimed that this improved performance in their tests, because pthread mutexes can be very expensive on some platforms. In my tests on x86_64 Linux however, it just forced CPU usage to 100% (200% actually, dualcore) and slowed down overall throughput.
You can give an explicit "--with-mutex=POSIX/pthread" argument when configuring BerkeleyDB to avoid the hybrid mutex scheme, in which case performance of BDB 4.6.19 seems to match what we obtained with BDB 4.6.3.
If you were using BerkeleyDB 4.5.20 before, the early BDB 4.6 releases will work well for you. The only difference between 4.5.20 and 4.6.1 was in the memory manager. 4.6.2 just tweaked some portability issues to support BREW (Qualcomm's cellphone programming environment). 4.6.3 added the ability to specify different cache priorities per database operation. (I didn't track what changed between 4.6.3 and 4.6.18.)
And for anyone curious - you can read ITS#3851 http://www.openldap.org/its/index.cgi/Development?id=3851 for the background on the problems in BerkeleyDB's memory allocator. While there was a fair amount of debate as to whether Jong's proposed solution was of any benefit, it was pretty clear that the existing code in BerkeleyDB could be improved. But the changes in the memory allocator may not visibly affect you if you're running a small-to-medium sized database. It's only apparent when the total volume of data is much larger than the BerkeleyDB cache, because that's the condition that leads to memory fragmentation, and it's the fragmentation that causes the allocator's performance to degrade.
The same kind of problem affects the slapd entry cache, when the number of active entries is much larger than the entry cache, and you're using the glibc malloc. (At least, for glibc 2.3 and older. I haven't retested with glibc 2.5 yet.)
To sum up - if you're already using BDB 4.5.20 with OpenLDAP 2.3 and the performance is acceptable, I wouldn't be in any hurry to upgrade to BDB 4.6. Releases 4.6.18 and 4.6.19 are incompatible with OpenLDAP 2.3. BDB 4.6.1 is a drop-in replacement for 4.5.20 though.