Running openldap 2.3.32 with bdb 4.2 (and using syncrepl if that's relevant).
I need to deal with the issue of how to safely delete old bdb log files on our many replicas.
In a previous thread, Aaron Richton wrote:
I'd recommend DB_LOG_AUTOREMOVE. Barring that, you can run db_archive manually. Check Sleepycat docs for details on either.
Well, I just ran db_archive and caused widespread chaos because most (all?) of the replicas stopped responding to queries. (I have yet to perform a post-mortem)
I know that there's a bug in bdb 4.2 that causes logs to be held open even though they're no longer required. Upgrading bdb is not on the cards right now so I need to work around that problem by stopping and starting openldap.
So the question I have just at the moment is, when I run db_archive, should openldap be running or not running?
I've seen nothing in any docs that suggest it should be stopped, and the bdb docs simply imply that applications are expected to be running (but I'm not a programmer so my I interpreted it wrongly). So I ran db_archive just after starting openldap. Was that the wrong thing to do, and is it an obvious cause for the meltdown?
Cheers, Lesley W
--On Wednesday, February 28, 2007 1:31 PM +1300 Lesley Walker lesley.walker@opus.co.nz wrote:
Running openldap 2.3.32 with bdb 4.2 (and using syncrepl if that's relevant).
So, are you running BDB 4.2 with or without the patches that sleepycat says you must use to use BDB 4.2? Because I use the patches that sleepycat says to use, and I have no problems with log purging.
http://www.stanford.edu/services/directory/openldap/configuration/bdb-build-notes.html
contains all the patches I apply to BDB 4.2.52 (which is the BDB 4.2 release I'll just assume you are using, since you don't say).
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
So, are you running BDB 4.2 with or without the patches that sleepycat says you must use to use BDB 4.2?
I would have to investigate that. We're running the Debian packaged version 4.2.52-18. I'm guessing possibly not.
Because I use the patches that sleepycat says to use, and I have no problems with log purging.
That's probably it, then. <sigh> So now we'll have to decide whether to do a custom build or find a workaround.
Thanks!
Lesley W
--On Wednesday, February 28, 2007 3:12 PM +1300 Lesley Walker lesley.walker@opus.co.nz wrote:
So, are you running BDB 4.2 with or without the patches that sleepycat says you must use to use BDB 4.2?
I would have to investigate that. We're running the Debian packaged version 4.2.52-18. I'm guessing possibly not.
Because I use the patches that sleepycat says to use, and I have no problems with log purging.
That's probably it, then. <sigh> So now we'll have to decide whether to do a custom build or find a workaround.
My personal experience is that using the Debian packages for anything is always questionable. I use Debian as my OS, but everything from gcc up is built into /usr/local. However, I would hope Debian would have included those patches.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
My personal experience is that using the Debian packages for anything is always questionable. I use Debian as my OS, but everything from gcc up is built into /usr/local.
We use the packaged versions where they work well enough, but custom builds for quite a few other things.
However, I would hope Debian would have included those patches.
As far as I can tell from the changelog, they have got at least three of them, they probably have the Java stuff (not using Java with it anyway) but they may have missed the one relating to the btree bug. Could that have caused a problem? Could it have fooled the consumers into deleting and re-creating all their records? Because I *think* that's what happened.
Lesley W
On Tue, 27 Feb 2007, Quanah Gibson-Mount wrote:
That's probably it, then. <sigh> So now we'll have to decide whether to do a custom build or find a workaround.
My personal experience is that using the Debian packages for anything is always questionable. I use Debian as my OS, but everything from gcc up is built into /usr/local. However, I would hope Debian would have included those patches.
Same here with Ubuntu, which I'm now forced to use as well as FreeBSD (better eye-candy, you see).
Relevance to OpenLDAP? Don't use Linux packages - always build from source.
Quanah Gibson-Mount wrote, on 28. feb 2007 03:11:
[...]
My personal experience is that using the Debian packages for anything is
always questionable. I use Debian as my OS, but everything from gcc up is built into /usr/local. However, I would hope Debian would have included those patches.
FWIW the Red Hat RHAS4 db-4.2.52 installed as-is causes OpenLDAP to puke; I built and install my own rpms, patched to the hilt, and install them with my own Cyrus SASL 2.2.22 rpms on any RHAS4 system I install. Buchan also builds his own discrete-sonamed db-4.2.52 stuff for his OpenLDAP 2.3 on systems where db-4.2.52 might give problems.
--Tonni
--On Tuesday, February 27, 2007 6:11 PM -0800 Quanah Gibson-Mount quanah@stanford.edu wrote:
--On Wednesday, February 28, 2007 3:12 PM +1300 Lesley Walker lesley.walker@opus.co.nz wrote:
So, are you running BDB 4.2 with or without the patches that sleepycat says you must use to use BDB 4.2?
I would have to investigate that. We're running the Debian packaged version 4.2.52-18. I'm guessing possibly not.
The version of BDB in Debian is currently 4.2.52-24. BDB 4.2.52-18 is part of Sarge. Sarge ships with OpenLDAP 2.2.23, which is a very old release of OpenLDAP, and there were some bugs fixed with checkpointing and open transactions *after* that release. So that may be the issue you are hitting. Are you *sure* you are running OpenLDAP 2.3.32 on those boxes? Because that wouldn't have been provided by the basic Sarge release. In fact, even in etch, the latest release is OpenLDAP 2.3.30 (with a few patches backported).
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
So, are you running BDB 4.2 with or without the patches that sleepycat says you must use to use BDB 4.2?
I would have to investigate that. We're running the Debian packaged version 4.2.52-18. I'm guessing possibly not.
I asked the Debian BDB maintainers, and the answer is copied below.
The version of BDB in Debian is currently 4.2.52-24. BDB 4.2.52-18 is part of Sarge. Sarge ships with OpenLDAP 2.2.23, which is a very old release of OpenLDAP, and there were some bugs fixed with checkpointing and open transactions *after* that release. So that may be the issue you are hitting. Are you *sure* you are running OpenLDAP 2.3.32 on those boxes?
Yes, I built it myself, and included the sync logging patch that Aaron Richton kindly pointed me to back in January.
We build most of our critical apps (OpenLDAP, Samba, Apache and a few others) but BDB isn't on that list.
Reply from Debian maintainers re the BDB patches:
Clint Adams wrote:
Sent: Wednesday, 28 February 2007 5:11 p.m. To: Lesley Walker Cc: pkg-db-devel@lists.alioth.debian.org Subject: Re: [Pkg-db-devel] Db4.2 patch query
Can you confirm/deny that all 5 of the patches listed at the following URL have been incorporated into the Debian build?
I can confirm that patches #1 and #2 have, and that #4 and #5 have NOT.
The URL I gave was http://www.oracle.com/technology/products/berkeley-db/db/update/4.2.52/patch .4.2.52.html (beware link breakage)
--On Tuesday, March 06, 2007 10:14 AM +1300 Lesley Walker lesley.walker@opus.co.nz wrote:
The URL I gave was http://www.oracle.com/technology/products/berkeley-db/db/update/4.2.52/pa tch .4.2.52.html (beware link breakage)
Interesting... Did they say if they were included in later releases?
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Quanah wrote regarding DBD patches on Debian:
Interesting... Did they say if they were included in later releases?
I wasn't 100% sure which release he was referring to since he didn't say, and I probably didn't phrase my original question as clearly as I could have. But I asked another question about future plans and got a reply that pretty much answers that one:
Clint Adams wrote:
Sent: Thursday, 1 March 2007 3:16 p.m. To: Lesley Walker Cc: pkg-db-devel@lists.alioth.debian.org Subject: Re: [Pkg-db-devel] Db4.2 patch query
Thanks for the quick answer!
Are you planning to incorporate the later ones at some stage?
As far as I am aware, there are no firm plans either way. Chances of them getting into sarge are infinitesimal. For etch, there is a possibility, at the discretion of the release managers, assuming release-critical bugs are filed. For lenny, we will either drop db4.2 entirely or apply the patches.
Please file bugs regarding the deficiencies that would be corrected by the application of the missing patches, if you wish to facilitate this process.
If you are unfamiliar with the bug reporting process, see http://www.debian.org/Bugs/Reporting
If you have a use case which exposes the data loss corrected by the patch, list it and set the severity according to http://www.debian.org/Bugs/Developer#severities
Hope this helps.
I replied that if I could pin down any specific problems I would file bug a report, but it probably makes more sense to move forward to 4.4.
--On Tuesday, March 06, 2007 11:19 AM +1300 Lesley Walker lesley.walker@opus.co.nz wrote:
Quanah wrote regarding DBD patches on Debian:
Interesting... Did they say if they were included in later releases?
I wasn't 100% sure which release he was referring to since he didn't say, and I probably didn't phrase my original question as clearly as I could have. But I asked another question about future plans and got a reply that pretty much answers that one:
Clint Adams wrote:
Sent: Thursday, 1 March 2007 3:16 p.m. To: Lesley Walker Cc: pkg-db-devel@lists.alioth.debian.org Subject: Re: [Pkg-db-devel] Db4.2 patch query
Thanks for the quick answer!
Are you planning to incorporate the later ones at some stage?
As far as I am aware, there are no firm plans either way. Chances of them getting into sarge are infinitesimal. For etch, there is a possibility, at the discretion of the release managers, assuming release-critical bugs are filed. For lenny, we will either drop db4.2 entirely or apply the patches.
Please file bugs regarding the deficiencies that would be corrected by the application of the missing patches, if you wish to facilitate this process.
If you are unfamiliar with the bug reporting process, see http://www.debian.org/Bugs/Reporting
If you have a use case which exposes the data loss corrected by the patch, list it and set the severity according to http://www.debian.org/Bugs/Developer#severities
Hope this helps.
I replied that if I could pin down any specific problems I would file bug a report, but it probably makes more sense to move forward to 4.4.
4.5 would work too. although I'd personally prefer 4.6 if it is released by that time. ;)
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Quanah Gibson-Mount wrote:
4.5 would work too. although I'd personally prefer 4.6 if it is released by that time. ;)
Gah! The numbers keep changing. I'm sure the FAQ said 4.4 was recommended, last time I looked. :-)
--On Tuesday, March 06, 2007 2:40 PM +1300 Lesley Walker lesley.walker@opus.co.nz wrote:
Quanah Gibson-Mount wrote:
4.5 would work too. although I'd personally prefer 4.6 if it is released by that time. ;)
Gah! The numbers keep changing. I'm sure the FAQ said 4.4 was recommended, last time I looked. :-)
In my testing, any of 4.2, 4.4, and 4.5 are acceptable. ;) 4.2 still has the best performance until you get to 4.6.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Quanah Gibson-Mount escribió:
--On Tuesday, March 06, 2007 11:19 AM +1300 Lesley Walker lesley.walker@opus.co.nz wrote:
Quanah wrote regarding DBD patches on Debian:
Interesting... Did they say if they were included in later releases?
I wasn't 100% sure which release he was referring to since he didn't say, and I probably didn't phrase my original question as clearly as I could have. But I asked another question about future plans and got a reply that pretty much answers that one:
Clint Adams wrote:
Sent: Thursday, 1 March 2007 3:16 p.m. To: Lesley Walker Cc: pkg-db-devel@lists.alioth.debian.org Subject: Re: [Pkg-db-devel] Db4.2 patch query
Thanks for the quick answer!
Are you planning to incorporate the later ones at some stage?
As far as I am aware, there are no firm plans either way. Chances of them getting into sarge are infinitesimal. For etch, there is a possibility, at the discretion of the release managers, assuming release-critical bugs are filed. For lenny, we will either drop db4.2 entirely or apply the patches.
Please file bugs regarding the deficiencies that would be corrected by the application of the missing patches, if you wish to facilitate this process.
If you are unfamiliar with the bug reporting process, see http://www.debian.org/Bugs/Reporting
If you have a use case which exposes the data loss corrected by the patch, list it and set the severity according to http://www.debian.org/Bugs/Developer#severities
Hope this helps.
I replied that if I could pin down any specific problems I would file bug a report, but it probably makes more sense to move forward to 4.4.
4.5 would work too. although I'd personally prefer 4.6 if it is released by that time. ;)
--Quanah
Hello!
I have followed the thread, because we're upgrading to the lastest stable release of OpenLDAP, compiling from the sources. In the other hand, we're doing this on Red Hat Enterprise Linux, and we have linking agaisnt db4-4.2.52-7.1.
Do you recommend compiling db4 4.5.20 from the sources and link OpenLDAP 2.3.32 agaisnt it ?
Regards,
Manuel Molina Cuberos wrote, on 09. mar 2007 13:39:
[...]
I have followed the thread, because we're upgrading to the lastest stable release of OpenLDAP, compiling from the sources. In the other hand, we're doing this on Red Hat Enterprise Linux, and we have linking agaisnt db4-4.2.52-7.1.
Do you recommend compiling db4 4.5.20 from the sources and link OpenLDAP 2.3.32 agaisnt it ?
Whatever you do, don't use Red Hat's standard db4-4.2.52-7.1 with OL 2.3: it isn't patched as required (it's from 2004 and has never been modified). You should either build Buchan Milne's OL 2.3 latest srpm configured to build and use a correctly patched, discrete version of db4-4.2.52 or I can give you correctly patched 4.2.52 rpms.
Do try *not* to use source-compiled db4 (in your case 4.5.20) on RHAS/RHEL, as it installs to non-standard (non-FHS) directories and will get you into all kinds of a mess later. As a pointer, try never to install non-rpm software on RHAS/RHEL, this after 4-5 years as Red Hat administrator and having done it all myself, earlier ;)
Best,
--Tonni
--On Friday, March 09, 2007 1:39 PM +0100 Manuel Molina Cuberos manuel.molina@corp.ya.com wrote:
4.5 would work too. although I'd personally prefer 4.6 if it is released by that time. ;)
--Quanah
Hello!
I have followed the thread, because we're upgrading to the lastest stable release of OpenLDAP, compiling from the sources. In the other hand, we're doing this on Red Hat Enterprise Linux, and we have linking agaisnt db4-4.2.52-7.1.
Do you recommend compiling db4 4.5.20 from the sources and link OpenLDAP 2.3.32 agaisnt it ?
4.2 remains the fastest BDB release outside of BDB 4.6 that I've tested. I would, of course, suggest building BDB yourself, and making sure you have all the applicable patches that go with it.
http://www.stanford.edu/services/directory/openldap/configuration/bdb-build-notes.html
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Hi: Why am I getting No such object?
#/usr/local/bin/ldapwhoami -x -D "cn=Directory Manager,ou=people,dc=sprint,dc=com" -W -h ldaphost Enter LDAP Password: ldap_bind: No such object (32) matched DN: ou=people,dc=sprint,dc=com
Thanks Ravi
On Sun, Mar 11, 2007 at 09:26:45AM -0400, Ravi wrote:
Hi: Why am I getting No such object?
#/usr/local/bin/ldapwhoami -x -D "cn=Directory Manager,ou=people,dc=sprint,dc=com" -W -h ldaphost Enter LDAP Password: ldap_bind: No such object (32) matched DN: ou=people,dc=sprint,dc=com
does ou=people,dc=sprint,dc=com exist ? what does the slapd log's say ?
Thanks Ravi
--On Tuesday, March 06, 2007 11:19 AM +1300 Lesley Walker lesley.walker@opus.co.nz wrote:
I replied that if I could pin down any specific problems I would file bug a report, but it probably makes more sense to move forward to 4.4.
I'm raising this as an issue with the pkg-openldap-devel folks as well.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
On 2/27/07, Lesley Walker lesley.walker@opus.co.nz wrote:
Running openldap 2.3.32 with bdb 4.2 (and using syncrepl if that's relevant).
I need to deal with the issue of how to safely delete old bdb log files on our many replicas.
In a previous thread, Aaron Richton wrote:
I'd recommend DB_LOG_AUTOREMOVE. Barring that, you can run db_archive manually. Check Sleepycat docs for details on either.
Well, I just ran db_archive and caused widespread chaos because most (all?) of the replicas stopped responding to queries. (I have yet to perform a post-mortem)
I know that there's a bug in bdb 4.2 that causes logs to be held open even though they're no longer required. Upgrading bdb is not on the cards right now so I need to work around that problem by stopping and starting openldap.
So the question I have just at the moment is, when I run db_archive, should openldap be running or not running?
I've seen nothing in any docs that suggest it should be stopped, and the bdb docs simply imply that applications are expected to be running (but I'm not a programmer so my I interpreted it wrongly). So I ran db_archive just after starting openldap. Was that the wrong thing to do, and is it an obvious cause for the meltdown?
What options to db_archive did you use? I'm safely using -d and -a | xargs without any issues. Are you sure you used the db_archive that corresponds to the version of bdb that's being used?
After you ran db_recover and the server stopped responding, did you restart slapd and it started responding again? If not, did you try to start with debugging info to figure out why? How many log files did it remove? Maybe openldap hadn't fully initialized, or had a lot of writes pending when you did the db_archive. (although I don't think that should matter)
I would also recommend not using distro packages for anything you rely on beyond what the OS needs to run.
matthew sporleder wrote:
What options to db_archive did you use? I'm safely using -d and -a | xargs without any issues. Are you sure you used the db_archive that corresponds to the version of bdb that's being used?
The command I ran was /usr/bin/db4.2_archive -d
After you ran db_recover and the server stopped responding, did you restart slapd and it started responding again?
I think when the servers stopped responding they were all busy deleting and re-creating all their entries.
I stopped slapd then ran slapd -c rid=123,csn=0 which brought everything back to its normal state over about ten minutes or so.
How many log files did it remove?
In most cases, about 50.
Maybe openldap hadn't fully initialized, or had a lot of writes pending when you did the db_archive.
There were probably some writes pending, but it's not a busy environment in terms of updates.
--On Thursday, March 01, 2007 11:03 AM +1300 Lesley Walker lesley.walker@opus.co.nz wrote:
matthew sporleder wrote:
What options to db_archive did you use? I'm safely using -d and -a | xargs without any issues. Are you sure you used the db_archive that corresponds to the version of bdb that's being used?
The command I ran was /usr/bin/db4.2_archive -d
The question is, what version of BDB is slapd linked to? For example, if slapd is linked against BDB 4.4, and you ran the BDB 4.2 db_archive, that's bad.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
openldap-software@openldap.org
-
Alex Samad -
Dave Horsfall -
Lesley Walker -
Manuel Molina Cuberos -
matthew sporleder -
Quanah Gibson-Mount -
Ravi -
Tony Earnshaw