masarati@aero.polimi.it wrote:

...

We are experiencing a decay in query performance over time using OpenLDAP 2.3.43 on CentOS 5.

After populating the database with about 70 000 entries, the query response time for one entry is about 0.4s. After about two hours the same query takes over 1s. The slapd.conf file and DB_CONFIG files that we use can be found at the following URLs.

http://svnweb.cern.ch/trac/gridinfo/browser/bdii/trunk/etc/bdii-slapd.conf http://svnweb.cern.ch/trac/gridinfo/browser/bdii/trunk/etc/DB_CONFIG

Does anyone have an idea about what is happening.

Not sure whether it matters or not, but your DB_CONFIG's set_cachesize looks very small, about 50MB. Are the same DB_CONFIG settings used for all databases? What database, of the 3 defined in your slapd.conf, is suffering from this problem? Are critical searches related to indexed attrs? Can you reproduce the problem with recent releases (e.g. 2.4.19)?

Also, I note that only in the first database you define an equality index on objectClass. This is instrumental for decent performances of slapd, as documented in slapd-bdb(5) (the presence index on objectClass is pointless).

The DB_CONFIG is the same for all databases. The first database has about 70K entries and the others only have a couple of entries. What is interesting is that queries to the all databases are really slow even though there is almost no data in two of them. I am have tried the 2.4.18 packages from Buchan Milne on RHEL5 and they seem to work fine.

The performance seems to be a great deal worse for none index searches. Searching for entries using the object class takes about 1s, however, using an attribute and a wild card takes 26s. The performance of both become worse with each successive query and this affects all databases no matter which one queried.

Thanks for your help

Laurence

Laurence Field wrote:

masarati@aero.polimi.it wrote:

...

...

We are experiencing a decay in query performance over time using OpenLDAP 2.3.43 on CentOS 5.

After populating the database with about 70 000 entries, the query response time for one entry is about 0.4s. After about two hours the same query takes over 1s. The slapd.conf file and DB_CONFIG files that we use can be found at the following URLs.

http://svnweb.cern.ch/trac/gridinfo/browser/bdii/trunk/etc/bdii-slapd.conf http://svnweb.cern.ch/trac/gridinfo/browser/bdii/trunk/etc/DB_CONFIG

Does anyone have an idea about what is happening.

Not sure whether it matters or not, but your DB_CONFIG's set_cachesize looks very small, about 50MB. Are the same DB_CONFIG settings used for all databases? What database, of the 3 defined in your slapd.conf, is suffering from this problem? Are critical searches related to indexed attrs? Can you reproduce the problem with recent releases (e.g. 2.4.19)?

So here are the results of the performance test. The the machines used were Xeon 2.33GHz 2 quad core CPUs with 16 GB RAM. The database was populated with 80K entries, approximately 70MB. A query for all entries was run using a single loop with a 5s sleep. The resulting graphs of the network utilization for the different LDAP versions can be found at the URLs below.

http://lfield.web.cern.ch/lfield/2.3.43.png http://lfield.web.cern.ch/lfield/2.4.18.png

The decrease in throughput corresponds to an increase in the response time. Why is the performance of version 2.3.43 decreasing over time?

The answer to this question can probably be gleaned from reading the openldap-devel archives from the time 2.4 was being first prepared for release. A lot of profiling and refactoring went into the 2.4 code to improve performance relative to 2.3. At this point 2.3 is ancient and I've forgotten everything that we've changed internally, and it's not worth my while to dig back to remember what we fixed.

Laurence Field wrote:

...

The answer to this question can probably be gleaned from reading the openldap-devel archives from the time 2.4 was being first prepared for release. A lot of profiling and refactoring went into the 2.4 code to improve performance relative to 2.3. At this point 2.3 is ancient and I've forgotten everything that we've changed internally, and it's not worth my while to dig back to remember what we fixed.

Thanks, will start digging.

Laurence

Just to complete this thread. In the devel list there was quite a bit of discussion about the bdb backend. Debugging suggested that most of the time was spent in the following library.

/usr/lib64/tls/libslapd_db-4.4.so

As a solution, I have just taken the openldap source rpm from Fedora 12 and rebuilt it on CENTOS 5, relocating it to /opt/openldap2.4 and turning off auto-provides.

Thanks for your help.

Laurence

Howard Chu wrote:

Laurence Field wrote:

...

Laurence Field wrote:

...

...

The answer to this question can probably be gleaned from reading the openldap-devel archives from the time 2.4 was being first prepared for release. A lot of profiling and refactoring went into the 2.4 code to improve performance relative to 2.3. At this point 2.3 is ancient and I've forgotten everything that we've changed internally, and it's not worth my while to dig back to remember what we fixed.

Thanks, will start digging.

Laurence

Just to complete this thread. In the devel list there was quite a bit of discussion about the bdb backend. Debugging suggested that most of the time was spent in the following library.

/usr/lib64/tls/libslapd_db-4.4.so

As a solution, I have just taken the openldap source rpm from Fedora 12 and rebuilt it on CENTOS 5, relocating it to /opt/openldap2.4 and turning off auto-provides.

And now you know why (a) we recommend never using slapd as built by your distro vendor (particularly Red Hat and its derivatives) and (b) we always recommend using the most current code.

As this seems to be the one and only, always true answer, which covers all common openldap server scenarios, I can't help but wonder:

Why are there no pre-built, openldap-crew-certified, latest stable version, auto-updateable packages made available for most common server distros, including RHEL, CentOS, SLES, Ubuntu LTS and Debian Stable?

When software are provided in source form only, the burden of pre-compilation configuration, compilation and quality assurance are placed on the user, which in most cases are less qualified, less informed, and less able to keep up to date with the work. Even more important, the risk of human error are multiplied by the number of users that accept this burden.

The combination of only releasing the software in source form, frequent development in the stable branch (too early too frequent), and continuously stating that vendor provided binary packages should NOT be used, places this software project way down on the list of software we want to include in a production environment, I am sorry to say. Sorry both because the alternatives are still quite expensive, and because openldap is what we currently use on our rather large and critical LDAP cluster.

This LDAP implementation has the potential to be on the very top of the list. But right now, it's found wanting.

I'm quite sure that we are not the only organization willing to pay good money for support contracts that includes up to date packages for our server distributions at any time. Feel free to cash in! :)

Kind regards

c

On Wednesday, 25 November 2009 14:57:12 Christian Haugan Toldnes wrote:

Howard Chu wrote:

...

Laurence Field wrote:

...

Laurence Field wrote:

...

...

The answer to this question can probably be gleaned from reading the openldap-devel archives from the time 2.4 was being first prepared for release. A lot of profiling and refactoring went into the 2.4 code to improve performance relative to 2.3. At this point 2.3 is ancient and I've forgotten everything that we've changed internally, and it's not worth my while to dig back to remember what we fixed.

Thanks, will start digging.

Laurence

Just to complete this thread. In the devel list there was quite a bit of discussion about the bdb backend. Debugging suggested that most of the time was spent in the following library.

/usr/lib64/tls/libslapd_db-4.4.so

As a solution, I have just taken the openldap source rpm from Fedora 12 and rebuilt it on CENTOS 5, relocating it to /opt/openldap2.4 and turning off auto-provides.

And now you know why (a) we recommend never using slapd as built by your distro vendor (particularly Red Hat and its derivatives) and (b) we always recommend using the most current code.

As this seems to be the one and only, always true answer, which covers all common openldap server scenarios, I can't help but wonder:

Why are there no pre-built, openldap-crew-certified, latest stable version, auto-updateable packages made available for most common server distros, including RHEL, CentOS, SLES, Ubuntu LTS and Debian Stable?

The same question could be asked about kernel, glibc, Berkeley DB, cyrus- sasl, MIT Kerberos and/or Heimdal etc. etc. etc.

Symas does provide builds, but they have to pay developers, so you may have to pay for builds from them, depending on your requirements.

When software are provided in source form only, the burden of pre-compilation configuration, compilation and quality assurance are placed on the user,

Or the organisation that provided the user with the rest of their OS, or third parties who provide supported packages.

which in most cases are less qualified, less informed, and less able to keep up to date with the work. Even more important, the risk of human error are multiplied by the number of users that accept this burden.

The combination of only releasing the software in source form, frequent development in the stable branch (too early too frequent), and continuously stating that vendor provided binary packages should NOT be used, places this software project way down on the list of software we want to include in a production environment, I am sorry to say. Sorry both because the alternatives are still quite expensive, and because openldap is what we currently use on our rather large and critical LDAP cluster.

This LDAP implementation has the potential to be on the very top of the list. But right now, it's found wanting.

I'm quite sure that we are not the only organization willing to pay good money for support contracts that includes up to date packages for our server distributions at any time. Feel free to cash in! :)

I know of at least two companies provided supported binaries: -Symas -OpenLogic: https://olex.openlogic.com/packages/openldap

I have used neither, as I build my own (and those for Mandriva), e.g.:

http://staff.telkomsa.net/packages/rhel5/openldap/

If OpenLDAP is your biggest requirement, I would probably go with Symas, as they employ OpenLDAP developers, if you need support on other packages OpenLogic supplies, it might make more sense to go with them.

2.4.19 is building at the moment, I hope packages will be available, at least for RHEL5/CentOS 5, by early next week.

Regards, Buchan

--On Wednesday, December 09, 2009 12:27 PM +0100 Christian Haugan Toldnes christian.toldnes@ntnu.no wrote:

Buchan Milne wrote:

...

I know of at least two companies provided supported binaries: -Symas -OpenLogic: https://olex.openlogic.com/packages/openldap

I have used neither, as I build my own (and those for Mandriva), e.g.:

http://staff.telkomsa.net/packages/rhel5/openldap/

If OpenLDAP is your biggest requirement, I would probably go with Symas, as they employ OpenLDAP developers, if you need support on other packages OpenLogic supplies, it might make more sense to go with them.

Of all the open source software we use, openldap seems to be the only server software not usable in the state shipped from our OS distributors, so in that case Symas sounds better. Howver, according to their website, they're heavily outdated: "Symas OpenLDAP Version 3 is based on OpenLDAP Version 2.3.".

Url: http://www.symas.com/cds.shtml

Doesn't really matter that they employ Openldap developers when said developer(s) always encurage latest version and they provide only older versions.

I might get the wrong idea from their website of course, if it's outdated or poorly presents current state of their services.

Elsewhere it seems like they do in fact ship openldap 2.4.x: http://www.symas.com/updates/

Anyway. Will look into it, as well as trying to get decent pricing from RedHat on their Directory Server.

If you have performance requirements, I'd be very careful to benchmark RH's directory server before you consider using it. It is orders of magnitudes slower than OpenLDAP in anything I've tried with it. slamd is an excellent tool for doing such benchmarking.

--Quanah

--

Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration