I am working on setting up OpenLDAP for a web project. I had originally planned on using the accesslog overlay to track all access to the LDAP. I have discovered that the use of this overlay has a tremendous impact on performance.
I started out by auditing all operations. With this configuration I was getting 40-50 reads/sec max. I then decided that I only needed to audit writes. Making this change sent the reads through the roof, but the writes were still only averaging 16-20 per second. I am in the process of retesting the writes with no accesslog. My assumption is that the writes per second will jump dramtically.
So I have to question what is the purpose of the accesslog overlay? Is it really needed and if so is there are way to increase its performance. I have to admit that I have not paid attention to the settings for the accesslog backend. Do I need to tweak these settings just like I did for my primary backend? If so what are the optimal settings for a write intensive database?
Thanks, Pete
--On Friday, March 20, 2009 3:08 PM -0400 Pete Giesin pgiesin@hubcitymedia.com wrote:
I am working on setting up OpenLDAP for a web project. I had originally planned on using the accesslog overlay to track all access to the LDAP. I have discovered that the use of this overlay has a tremendous impact on performance.
I started out by auditing all operations. With this configuration I was getting 40-50 reads/sec max. I then decided that I only needed to audit writes. Making this change sent the reads through the roof, but the writes were still only averaging 16-20 per second. I am in the process of retesting the writes with no accesslog. My assumption is that the writes per second will jump dramtically.
So I have to question what is the purpose of the accesslog overlay? Is it really needed and if so is there are way to increase its performance. I have to admit that I have not paid attention to the settings for the accesslog backend. Do I need to tweak these settings just like I did for my primary backend? If so what are the optimal settings for a write intensive database?
It's a database like any other, so yes, it needs to be tuned appropriately to your work load. The accesslog overlay is most often used for things like delta-syncrepl replication. It can also be used to trigger actions based off what it writes out, as well.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Pete Giesin wrote:
I am working on setting up OpenLDAP for a web project. I had originally planned on using the accesslog overlay to track all access to the LDAP. I have discovered that the use of this overlay has a tremendous impact on performance.
I started out by auditing all operations. With this configuration I was getting 40-50 reads/sec max. I then decided that I only needed to audit writes. Making this change sent the reads through the roof, but the writes were still only averaging 16-20 per second. I am in the process of retesting the writes with no accesslog. My assumption is that the writes per second will jump dramtically.
So I have to question what is the purpose of the accesslog overlay? Is it really needed and if so is there are way to increase its performance. I have to admit that I have not paid attention to the settings for the accesslog backend. Do I need to tweak these settings just like I did for my primary backend? If so what are the optimal settings for a write intensive database?
I'm sure there will be disagreement over what I'm about to say, but if your application is write intensive, OpenLDAP is probably the wrong tool for the job. That said, if you tune the bdb backend for the access log correctly, you should be able to get many order of magnitudes better performance out of it.
Turning off the accesslog completely did increase writes but only to ~200 per sec. That is still not an acceptable number. So I need to figure out how to tweak OpenLDAP to increase write performance for both my primary backend and the accesslog backend.
Here is my configuration for the accesslog database. These are basically straight out of the box. I am not sure how best to tweak these to increase writes. My current configuration has both the db and log files for both the log and primary databases on the same disk. I realize that split these across multiple disks would help with the write performance.
Any other suggestions on how to imrpove write performance would be greatly appreciated.
Thanks, Pete
slapd.conf ------------------------- #Access Log database database hdb suffix "cn=log" rootdn "cn=Manager,cn=log" rootpw secret directory /opt/symas/var/openldap-data/access # Indices to maintain index reqStart eq index objectClass eq
DB_CONFIG ------------------------ # one 0.25 GB cache set_cachesize 0 268435456 1
# Data Directory #set_data_dir db
# Transaction Log settings set_lg_regionmax 262144 set_lg_bsize 2097152
set_lg_dir /opt/symas/var/openldap-logs/access
On Fri, Mar 20, 2009 at 3:56 PM, Russell Jackson raj@csub.edu wrote:
Pete Giesin wrote:
I am working on setting up OpenLDAP for a web project. I had originally
planned on
using the accesslog overlay to track all access to the LDAP. I have
discovered that the
use of this overlay has a tremendous impact on performance.
I started out by auditing all operations. With this configuration I was
getting 40-50
reads/sec max. I then decided that I only needed to audit writes. Making
this change
sent the reads through the roof, but the writes were still only averaging
16-20 per
second. I am in the process of retesting the writes with no accesslog. My
assumption is
that the writes per second will jump dramtically.
So I have to question what is the purpose of the accesslog overlay? Is it
really needed
and if so is there are way to increase its performance. I have to admit
that I have not
paid attention to the settings for the accesslog backend. Do I need to
tweak these
settings just like I did for my primary backend? If so what are the
optimal settings
for a write intensive database?
I'm sure there will be disagreement over what I'm about to say, but if your application is write intensive, OpenLDAP is probably the wrong tool for the job. That said, if you tune the bdb backend for the access log correctly, you should be able to get many order of magnitudes better performance out of it.
-- Russell A. Jackson raj@csub.edu Network Analyst California State University, Bakersfield
What the scientists have in their briefcases is terrifying. -- Nikita Khruschev
Pete Giesin wrote:
Turning off the accesslog completely did increase writes but only to ~200 per sec. That is still not an acceptable number.
For determining what's an acceptable number one would need some more information: - which OpenLDAP version (2.4 is faster on writing) - how large is your whole database? - indexing configuration? (the more the slower) - how big are the entries written? (the bigger the slower) - hardware used?
Ciao, Michael.
--On Friday, March 20, 2009 10:53 PM +0100 Michael Ströder michael@stroeder.com wrote:
Pete Giesin wrote:
Turning off the accesslog completely did increase writes but only to ~200 per sec. That is still not an acceptable number.
For determining what's an acceptable number one would need some more information:
- which OpenLDAP version (2.4 is faster on writing)
- how large is your whole database?
- indexing configuration? (the more the slower)
- how big are the entries written? (the bigger the slower)
- hardware used?
BDB release? Database backend? Disk-based BDB cache or shared memory BDB cache? checkpoint interval?
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
openldap-software@openldap.org
-
Michael Ströder -
Pete Giesin -
Quanah Gibson-Mount -
Russell Jackson