OK, so 4.6.18 changes the lock management system in ways that are incompatible with previous releases. The advice from Oracle is to just use a read-only transaction (which is actually what we used to do, several revisions back). Of course with BDB 4.2 that causes the problem of log files staying open unless you apply the patch I wrote for that issue. It seems we may need to either resurrect that patch (it should still be in CVS) or drop support for DB 4.2 so we can rewrite the lock management (yet again) to support DB 4.6.
Our early tests with DB 4.6.1 thru 4.6.3 showed impressive performance gains, making it seem worthwhile. But we have no way of knowing how the current 4.6.18 will perform at the moment.
Howard Chu wrote:
Our early tests with DB 4.6.1 thru 4.6.3 showed impressive performance gains, making it seem worthwhile. But we have no way of knowing how the current 4.6.18 will perform at the moment.
Were those speedups confined to, say, write, or general?
p.
Ing. Pierangelo Masarati OpenLDAP Core Team
SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati@sys-net.it ---------------------------------------
Pierangelo Masarati wrote:
Howard Chu wrote:
Our early tests with DB 4.6.1 thru 4.6.3 showed impressive performance gains, making it seem worthwhile. But we have no way of knowing how the current 4.6.18 will perform at the moment.
Were those speedups confined to, say, write, or general?
In general. You might recall that Jong-Hyuk Choi had done some work identifying bottlenecks in BDB's memory management; Sleepycat never used the code that Jong sent them but they still rewrote their memory manager between DB 4.5.20 and 4.6.1 as a result of Jong's work. Since the memory manager affects all operations in the DB library, the speedups can be seen everywhere.
<quote who="Howard Chu">
Pierangelo Masarati wrote:
Howard Chu wrote:
Our early tests with DB 4.6.1 thru 4.6.3 showed impressive performance gains, making it seem worthwhile. But we have no way of knowing how the current 4.6.18 will perform at the moment.
Were those speedups confined to, say, write, or general?
In general. You might recall that Jong-Hyuk Choi had done some work identifying bottlenecks in BDB's memory management; Sleepycat never used the code that Jong sent them but they still rewrote their memory manager between DB 4.5.20 and 4.6.1 as a result of Jong's work. Since the memory manager affects all operations in the DB library, the speedups can be seen everywhere. --
If support for 4.2 is dropped in the next 2.4 beta, what is the earliest version we think we will support?
Gavin.
Gavin Henry wrote:
<quote who="Howard Chu"> > Pierangelo Masarati wrote: >> Howard Chu wrote: >> >>> Our early tests with DB 4.6.1 thru 4.6.3 showed impressive performance >>> gains, making it seem worthwhile. But we have no way of knowing how the >>> current 4.6.18 will perform at the moment. >> Were those speedups confined to, say, write, or general? > In general. You might recall that Jong-Hyuk Choi had done some work > identifying > bottlenecks in BDB's memory management; Sleepycat never used the code that > Jong > sent them but they still rewrote their memory manager between DB 4.5.20 > and > 4.6.1 as a result of Jong's work. Since the memory manager affects all > operations in the DB library, the speedups can be seen everywhere. > --
If support for 4.2 is dropped in the next 2.4 beta, what is the earliest version we think we will support?
We already disallow use of 4.3 due to its instability, so 4.4 would be the minimum.
Also the BDB support guys at Oracle are now saying they'd like 4.2 to be dropped anyway, as it's nearing its 4th birthday, and they don't want to have to deal with it any more. I guess that's reasonable...
--On August 14, 2007 8:39:20 AM -0700 Howard Chu hyc@symas.com wrote:
Gavin Henry wrote:
<quote who="Howard Chu"> > Pierangelo Masarati wrote: >> Howard Chu wrote: >> >>> Our early tests with DB 4.6.1 thru 4.6.3 showed impressive performance >>> gains, making it seem worthwhile. But we have no way of knowing how >>> the current 4.6.18 will perform at the moment. >> Were those speedups confined to, say, write, or general? > In general. You might recall that Jong-Hyuk Choi had done some work > identifying > bottlenecks in BDB's memory management; Sleepycat never used the code > that Jong > sent them but they still rewrote their memory manager between DB 4.5.20 > and > 4.6.1 as a result of Jong's work. Since the memory manager affects all > operations in the DB library, the speedups can be seen everywhere. > --
If support for 4.2 is dropped in the next 2.4 beta, what is the earliest version we think we will support?
We already disallow use of 4.3 due to its instability, so 4.4 would be the minimum.
Also the BDB support guys at Oracle are now saying they'd like 4.2 to be dropped anyway, as it's nearing its 4th birthday, and they don't want to have to deal with it any more. I guess that's reasonable...
There's a lot of people who'd like to drop support for 4.2 (distros in particular). I'm going to take a snapshot of HEAD today before the 4.6 changes, so I can do comparison tests once the 4.6 support is in, but I think that dropping support for 4.2 is the right way to go.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Michael Ströder wrote:
Quanah Gibson-Mount wrote:
but I think that dropping support for 4.2 is the right way to go.
+1 (since the developers are not willing to support it anyway)
Ciao, Michael.
What ever is supported by the vendor is fine by me.
On Tuesday 14 August 2007 17:39:20 Howard Chu wrote:
Gavin Henry wrote:
<quote who="Howard Chu">
Pierangelo Masarati wrote:
Howard Chu wrote:
Our early tests with DB 4.6.1 thru 4.6.3 showed impressive performance gains, making it seem worthwhile. But we have no way of knowing how the current 4.6.18 will perform at the moment.
[...]
If support for 4.2 is dropped in the next 2.4 beta, what is the earliest version we think we will support?
We already disallow use of 4.3 due to its instability, so 4.4 would be the minimum.
Also the BDB support guys at Oracle are now saying they'd like 4.2 to be dropped anyway, as it's nearing its 4th birthday, and they don't want to have to deal with it any more. I guess that's reasonable...
Most distros that have db4.2 with all the recommended patches already have a 4.4 or later. On distros that don't ... you'd probably be bundling a version of Berkely DB anyway, it might as well give performance benefits as well
(while trying to finish up db4.6 packages for Mandriva ...)
Regards, Buchan
Howard Chu writes:
If support for 4.2 is dropped in the next 2.4 beta, what is the earliest version we think we will support?
We already disallow use of 4.3 due to its instability, so 4.4 would be the minimum.
What's the best BerkeleyDB version to use with OpenLDAP 2.3 anyway?
--On August 15, 2007 5:53:04 PM +0200 Hallvard B Furuseth h.b.furuseth@usit.uio.no wrote:
Howard Chu writes:
If support for 4.2 is dropped in the next 2.4 beta, what is the earliest version we think we will support?
We already disallow use of 4.3 due to its instability, so 4.4 would be the minimum.
What's the best BerkeleyDB version to use with OpenLDAP 2.3 anyway?
From my testing, currently it is 4.2.52+patches. However, testing with the
pre-releases from Sleepycat of 4.6 indicated it was faster than 4.2.52 across the board. Both 4.4 and 4.5 come in slower than 4.2 and 4.6 in my testing. So from a throughput (read & write) standpoint, 4.2 is what I've found to be the "best" until 4.6 came along.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
--On August 15, 2007 5:53:04 PM +0200 Hallvard B Furuseth h.b.furuseth@usit.uio.no wrote:
Howard Chu writes:
If support for 4.2 is dropped in the next 2.4 beta, what is the earliest version we think we will support?
We already disallow use of 4.3 due to its instability, so 4.4 would be the minimum.
What's the best BerkeleyDB version to use with OpenLDAP 2.3 anyway?
From my testing, currently it is 4.2.52+patches. However, testing with the
pre-releases from Sleepycat of 4.6 indicated it was faster than 4.2.52 across the board. Both 4.4 and 4.5 come in slower than 4.2 and 4.6 in my testing. So from a throughput (read & write) standpoint, 4.2 is what I've found to be the "best" until 4.6 came along.
Chaps,
Am I the only one noticing this:
The recommended/optimal Berkeley DB 4.2.52 for OpenLDAP 2.3.x might not even be supported with the next major release 2.4.x.
I've been reading this thread and know about the background on this, but still..
--Quanah
//Kari Mattsson
--On August 15, 2007 8:44:41 PM +0300 Kari Mattsson kari@trivore.com wrote:
Chaps,
Am I the only one noticing this:
The recommended/optimal Berkeley DB 4.2.52 for OpenLDAP 2.3.x might not even be supported with the next major release 2.4.x.
I've been reading this thread and know about the background on this, but still..
Right. The point is, that 4.6 makes an incompatible change with 4.2.52. On the upside, we've only stayed with recommending 4.2 because neither 4.4 or 4.5 did better. 4.6 does do better. I've already discussed with Howard that once the changes for 4.6 are in HEAD, I'll do further benchmarking to compare HEAD w/4.2 support and HEAD w/ 4.6 support to verify that 4.6 continues to give the better performance.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
--On August 15, 2007 8:44:41 PM +0300 Kari Mattsson kari@trivore.com wrote:
Chaps,
Am I the only one noticing this:
The recommended/optimal Berkeley DB 4.2.52 for OpenLDAP 2.3.x might not even be supported with the next major release 2.4.x.
Right. The point is, that 4.6 makes an incompatible change with 4.2.52. On the upside, we've only stayed with recommending 4.2 because neither 4.4 or 4.5 did better.
That's not strictly true; 4.4 and 4.5 have better 64-bit support and we've been recommending them for 64-bit servers. So BDB 4.2 has not been the sole recommended version for OL 2.3.
4.6 does do better. I've already discussed with Howard that once the changes for 4.6 are in HEAD, I'll do further benchmarking to compare HEAD w/4.2 support and HEAD w/ 4.6 support to verify that 4.6 continues to give the better performance.
--On August 15, 2007 11:10:31 AM -0700 Howard Chu hyc@symas.com wrote:
Quanah Gibson-Mount wrote:
--On August 15, 2007 8:44:41 PM +0300 Kari Mattsson kari@trivore.com wrote:
Chaps,
Am I the only one noticing this:
The recommended/optimal Berkeley DB 4.2.52 for OpenLDAP 2.3.x might not even be supported with the next major release 2.4.x.
Right. The point is, that 4.6 makes an incompatible change with 4.2.52. On the upside, we've only stayed with recommending 4.2 because neither 4.4 or 4.5 did better.
That's not strictly true; 4.4 and 4.5 have better 64-bit support and we've been recommending them for 64-bit servers. So BDB 4.2 has not been the sole recommended version for OL 2.3.
Huh, I've been using BDB 4.2 on 64-bit servers no problem... And it still performed better than 4.4 or 4.5 on 64-bit.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount writes:
That's not strictly true; 4.4 and 4.5 have better 64-bit support and we've been recommending them for 64-bit servers. So BDB 4.2 has not been the sole recommended version for OL 2.3.
Huh, I've been using BDB 4.2 on 64-bit servers no problem... And it still performed better than 4.4 or 4.5 on 64-bit.
It sounds to me like it's a bit early to remove support for the possibly-best BerkeleyDB version we have so far. 4.6+OpenLDAP hasn't seen much exposure yet after all. At least, I suggest to delay changes which will be hard to revert if 4.3-like problems do crop up.
Hallvard B Furuseth wrote:
Quanah Gibson-Mount writes:
That's not strictly true; 4.4 and 4.5 have better 64-bit support and we've been recommending them for 64-bit servers. So BDB 4.2 has not been the sole recommended version for OL 2.3.
Huh, I've been using BDB 4.2 on 64-bit servers no problem... And it still performed better than 4.4 or 4.5 on 64-bit.
It sounds to me like it's a bit early to remove support for the possibly-best BerkeleyDB version we have so far. 4.6+OpenLDAP hasn't seen much exposure yet after all. At least, I suggest to delay changes which will be hard to revert if 4.3-like problems do crop up.
Agreed. The current code in CVS is working with 4.6.19, but performance is terrible, using up over 3x more CPU time and still running over 3x slower than 4.2.52 or 4.6.3. I don't know what they've done between 4.6.3 and current, but it's definitely not ready for primetime.
I've also got a version of the code written to Oracle's recommendation (using read-only transactions) which works with 4.3 and up, but again, the performance is abysmal. As such, I'm not committing that version.
Still talking to the Oracle/BDB folks to see where to go next.
Kari Mattsson wrote:
Quanah Gibson-Mount wrote:
pre-releases from Sleepycat of 4.6 indicated it was faster than 4.2.52 across the board. Both 4.4 and 4.5 come in slower than 4.2 and 4.6 in my testing. So from a throughput (read & write) standpoint, 4.2 is what I've found to be the "best" until 4.6 came along.
Chaps,
Am I the only one noticing this:
The recommended/optimal Berkeley DB 4.2.52 for OpenLDAP 2.3.x might not even be supported with the next major release 2.4.x.
I've been reading this thread and know about the background on this, but still..
And your point is?
BDB 4.1 was the best available when OpenLDAP 2.1 was released; it was deprecated in favor of BDB 4.2 when OpenLDAP 2.2 was released. How is this any different?
BDB 4.3 would have been the preferred version when OpenLDAP 2.3 was initially released, but we found that 4.3 was terribly broken. It's great that BDB 4.2 has worked for us so long, but it's time to move on.
Howard Chu wrote:
Kari Mattsson wrote:
Quanah Gibson-Mount wrote:
pre-releases from Sleepycat of 4.6 indicated it was faster than 4.2.52 across the board. Both 4.4 and 4.5 come in slower than 4.2 and 4.6 in my testing. So from a throughput (read & write) standpoint, 4.2 is what I've found to be the "best" until 4.6 came along.
Chaps,
Am I the only one noticing this:
The recommended/optimal Berkeley DB 4.2.52 for OpenLDAP 2.3.x might not even be supported with the next major release 2.4.x.
I've been reading this thread and know about the background on this, but still..
And your point is?
Simple, yet philosophical:
If 'D' is the best companion for 'O', it is evolutionary to allow 'O+1' to still work with 'D'.
Requiring 'D+1' with 'O+1' makes upgrades more difficult to arrange as a whole.
So, maybe my point is change management.
BDB 4.1 was the best available when OpenLDAP 2.1 was released; it was deprecated in favor of BDB 4.2 when OpenLDAP 2.2 was released. How is this any different?
Wasn't here during that time ;-) ..meaning this same thing has already happened before.
BDB 4.3 would have been the preferred version when OpenLDAP 2.3 was initially released, but we found that 4.3 was terribly broken. It's great that BDB 4.2 has worked for us so long, but it's time to move on.
Yep. Please do not take this note as a negative one. It was just a note.
--On August 15, 2007 9:22:54 PM +0300 Kari Mattsson kari@trivore.com wrote:
Simple, yet philosophical:
If 'D' is the best companion for 'O', it is evolutionary to allow 'O+1' to still work with 'D'.
Requiring 'D+1' with 'O+1' makes upgrades more difficult to arrange as a whole.
So, maybe my point is change management.
I think you also miss the point that every OpenLDAP minor release I've experienced has required the database to be reloaded (2.1->2.2, 2.2->2.3), so this again is no different than the past, although for different reasons this time around, I believe.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
--On August 15, 2007 9:22:54 PM +0300 Kari Mattsson kari@trivore.com wrote:
Simple, yet philosophical:
If 'D' is the best companion for 'O', it is evolutionary to allow 'O+1' to still work with 'D'.
Requiring 'D+1' with 'O+1' makes upgrades more difficult to arrange as a whole.
So, maybe my point is change management.
I think you also miss the point that every OpenLDAP minor release I've experienced has required the database to be reloaded (2.1->2.2, 2.2->2.3), so this again is no different than the past, although for different reasons this time around, I believe.
You think right. My first OpenLDAP was during 2.1 era, and I wasn't really thinking this through.
--Quanah
//Kari
Kari Mattsson wrote:
If 'D' is the best companion for 'O', it is evolutionary to allow 'O+1' to still work with 'D'.
Requiring 'D+1' with 'O+1' makes upgrades more difficult to arrange as a whole.
So, maybe my point is change management.
As Quanah already said, change management is no different here than it has ever been: slapcat with the old version, slapadd with the new. That has always been the recommended upgrade procedure.
As an aside, the current back-bdb code will still compile and work properly with everything going back to BDB 4.0 (if you can still find a copy of that anywhere). You just have to tell the configure script to skip the version check if you really want to run things that way. But there are plenty of good reasons not to keep using those old BDB releases, which is why we raise the requirements in the configure script.
-
Buchan Milne -
Gavin Henry -
Hallvard B Furuseth -
Howard Chu -
Kari Mattsson -
Michael Ströder -
Pierangelo Masarati -
Quanah Gibson-Mount