Marten Lehmann wrote:

we are currently running openldap 2.2.13 with a bdb-4.2.52 backend as it comes with RHEL4.

As stated on this list many times this version of OpenLDAP is historic, outdated and was quite buggy although Red Hat ships it with its Enterprise product. There have been tons of fixes and many other important improvements applied to the code since then. So you should definitely upgrade!

I read through the backend documentation of openldap and berkeley db is praised as very fast and efficient.

Do have the six patches applied to 4.2.52 needed for OpenLDAP?

Ciao, Michael.

Marten Lehmann wrote:

Hello,

we are currently running openldap 2.2.13 with a bdb-4.2.52 backend as it comes with RHEL4. We have about 50.000 cn's each having about 5-10 attributes with a length of no longer than 100 characters. So our dataset is not very big.

We have a lot of concurrent reads since or ldap provides the data for incoming and pop3/imap servers. But we have only about 100 changes/adds a day, not more.

Yesterday we had the second bdb problem in our 1 year ldap setup. The problem is, that we cannot really detect the crash, since ldap queries just hang, but neither they timeout nor openldap is crashing. We just notice the mailserver issues and have to track it down to openldap and bdb. As with last time, we had to shutdown openldap and recover bdb.

Fortunately we have been able to recover the data-directory both times using dv_recover. But in fact, we want to have a backend, that doesn't crash twice a year. And as our setup is growing and we want to start with replication, we want a backend to whom we can trust.

I read through the backend documentation of openldap and berkeley db is praised as very fast and efficient. Amazing, that berkeleydb is not called stable anywhere in the documentation. And from my experience, bdb is not stable. Whenever you here about bdb, you here about it because a database went corrupt.

Hm, according to Red Hat, their build of OpenLDAP 2.2 is extremely stable. Perhaps, since you paid them for RHEL4, you should file a bug report with them to let them know how wrong they are. They don't listen to us, nor do they give away Red Hat Network accounts to mere Open Source Developers whose code they resell, so we can't file bug reports for you.

bdb is just a key/value database that seems to work fine as long it is read-only. Different projects, e.g. subversion, have turned away from bdb and use a different backend.

Often they use SQLite. Since SQLite handles complete table layouts, it would also be possible to create one table with two columns (key and value as in bdb). SQLite shall also be transactional.

But why is openldap sticking on bdb? Does bdb have any other important features the SQLite doesn't have? Benchmark issues? Replication?

I have to admit, that I'm not using the latest releases of bdb. But I'm using and watching bdb for years and I hardly believe, that it has become stable in the latest release.

4.2.52 (with documented patches) has never failed for me. 4.3 was known to blow up in many situations. 4.6.21 looks pretty good, there's no reported problems with it. BDB 4.7 is in the works already. On my last test, 4.7.13 SEGV'd in its deadlock detector, so yeah, I guess their release record is kind of spotty. But that's also the nature of newer software; as 4.7 is still early in its release cycle I expect it will improve just as the others did.

On Wednesday 27 February 2008 18:42:05 Marten Lehmann wrote:

Hello,

we are currently running openldap 2.2.13 with a bdb-4.2.52 backend as it comes with RHEL4. We have about 50.000 cn's

How many cn's you have is irrelevant, so I assume you mean entries.

each having about 5-10 attributes with a length of no longer than 100 characters. So our dataset is not very big.

However, it is most definitely bigger that what you can run on the default BDB cache (256kB). You didn't provide any details of BDB tuning (the contents of your DB_CONFIG file, for example), so I assume you have none, in which case you can expect problems.

We have a lot of concurrent reads since or ldap provides the data for incoming and pop3/imap servers. But we have only about 100 changes/adds a day, not more.

We briefly ran 2.2.13 (before I joined, and for a short time thereafter), and there weren't huge problems, on a database with (at that time) ~ 800 000 entries, each server doing > 100 operations/sec, with about 2 writes/sec.

We currently run 2.3.40 (plus one patch ...) with 1.5 million entries, each slave doing > 200 operations/sec, with about 4 writes/sec.

In neither case have we had significant problems. But, we always had an appropriately tuned DB_CONFIG

Yesterday we had the second bdb problem in our 1 year ldap setup. The problem is, that we cannot really detect the crash, since ldap queries just hang, but neither they timeout nor openldap is crashing. We just notice the mailserver issues and have to track it down to openldap and bdb.

So, on a 50,000 user mail system, you don't monitor all the components of the mail system?

As with last time, we had to shutdown openldap and recover bdb.

The 2nd may be as a result of the first, which could be as a result of your lack of tuning, or a bug.

Fortunately we have been able to recover the data-directory both times using dv_recover.

Red Hat should have included automatic database recovery in their init script, as other a number of other distributions did for 2.1 and 2.2. Then you would probably not have noticed much ...

But in fact, we want to have a backend, that doesn't crash twice a year.

Are you sure it was the backend? Or, was the fact that you needed to recover the result of shutting down slapd uncleanly?

And as our setup is growing and we want to start with replication, we want a backend to whom we can trust.

I read through the backend documentation of openldap and berkeley db is praised as very fast and efficient. Amazing, that berkeleydb is not called stable anywhere in the documentation. And from my experience, bdb is not stable. Whenever you here about bdb, you here about it because a database went corrupt. bdb is just a key/value database that seems to work fine as long it is read-only. Different projects, e.g. subversion, have turned away from bdb and use a different backend.

You are assuming that the reason they stopped using BDB was problems in BDB itself, whereas it could have been in the way the software used the library ...

Often they use SQLite. Since SQLite handles complete table layouts, it would also be possible to create one table with two columns (key and value as in bdb). SQLite shall also be transactional.

BDB is transactional. OpenLDAP has no need of SQL support etc. AFAIK, the biggest driver for people to use SQLite is related to licensing issues.

But why is openldap sticking on bdb? Does bdb have any other important features the SQLite doesn't have? Benchmark issues? Replication?

OpenLDAP doesn't use BDB replication features.

I have to admit, that I'm not using the latest releases of bdb. But I'm using and watching bdb for years and I hardly believe, that it has become stable in the latest release.

I really don't think it is beneficial for you to criticise the software ... if you want solutions to your problems, provide more detail on your configuration, and maybe you will have a quicker resolution ...

You may want to upgrade. This may not fix all your issues (as upgrading doesn't magically tune your configuration for you), however: -There have been lots of fixes since 2.2.13 -Features have been added that would mitigate your problems, even if you didn't do tuning (such as slapd doing automatic database recovery when necessary) -Warning messages when no DB_CONFIG file is found etc.

Upgrading may not be very much work ... you may want to look at:

http://staff.telkomsa.net/packages/rhel4/openldap/ http://staff.telkomsa.net/packages/ (for a .repo file for yum, if you use yum)

However, I am curious as to why this question ("Why does OpenLDAP on RHEL4 suck") gets asked so often. How did you come to the conclusion that BDB should be dropped in favour of some other database library? What searches did you do to find solutions to your problem? Maybe we can improve the content on the FAQ-o-matic to ensure people don't end up in your situation, or at least find real solutions sooner.

Regards, Buchan

andylockran wrote:

Howard Chu wrote:

...

4.2.52 (with documented patches) has never failed for me. 4.3 was known to blow up in many situations. 4.6.21 looks pretty good, there's no reported problems with it. BDB 4.7 is in the works already. On my last test, 4.7.13 SEGV'd in its deadlock detector, so yeah, I guess their release record is kind of spotty. But that's also the nature of newer software; as 4.7 is still early in its release cycle I expect it will improve just as the others did.

Howard Chu wrote:

...

4.2.52 (with documented patches) has never failed for me.

Apologies for reviving a dead thread - but I've had a couple of occasions this year when I've had to restore the bdb backend using db_recover.

I notice I'm using the version of bdb mentioned on this thread.. but am ignorant to the patches. I installed Buchan Milne's repo in order to get a stable openldap on RHEL4 - but my bdb version is still stuck in 2003.

What is the best advised route to upgrading - is there another repo with bdb in it, or should I compile bdb manually - then link to it later?

I'm currently manually building more modern BDB variants, as I wasn't able to find a repo with a more updated BDB for CentOS 4.5. Admittedly, I haven't really looked, so I can't say they're not out there.

Building OpenLDAP against a custom BDB isn't overly difficult, the majority being remembering to set the library envars, as ./configure doesn't take a parameter for BDB paths.

HTH,