Hi guys,
First of all, I would like to thank the community for such a great piece of software and encourage the developers to keep up the good work.
Until now, the setup of the ldap and the file configuration has been pretty straightforward. The ldap was performing great with few entries, but things got worse as the number of entries in the ldap increased. Currently we are experiencing serious performance problems when the searches performed exceed certain amount of entries. The "ldap_search" calls usually return after a few seconds but as the number of results matching the filter entered increases, the waiting time varies from one minute to many minutes.
The scenario is the following: * We are accessing openldap via PHP function calls * The number of entries in the ldap is close to 100.000 * The hardware is the following: Intel(R) Xeon(TM) CPU 3.33GHz 1Gb RAM * The slapd.conf file is the following:
/************************************************************************************************************/ modulepath /usr/lib/ldap moduleload back_bdb sizelimit -1 tool-threads 1 backend bdb checkpoint 512 30 database bdb directory "/var/lib/ldap" dbconfig set_cachesize 0 2097152 0 cachesize 100000 /************************************************************************************************************/
We have tried to increase both set_cachesize and cachesize (multiplying both by 100) and have not noticed a performance increase. I hope any of you can point us in the right direction since we are beginning to get desperated...
Thanks in advance,
Mikel
You need to add indexes for whatever you filter for. Without them, every search must examine all entries in scope for the search. Include 'index objectClass eq' no matter what else, since slapd often adds an objectClass component to your filter internally.
You can use loglevel 256 to see what your clients search for.
Also messges like <= bdb_<inequality, substring, etc>_candidates: (foo) not indexed if I remember correctly.
If the clients search for (&(foo=...)(bar=...)) you might only need an index for one of foo and bar, since each 'and' component narrows down the search. If they search for (|(foo=...)(bar=...)) you need both. Even if you don't have both attributes in the database, because slapd doesn't know that. Except you don't need (and can't) index attributes your included schema doesn't define.
If you add indexes by editing slapd.conf rather than via cn=config, stop slapd and run sbin/slapindex to actually build the indexes. Without slapindex, slapd will think the empty index means nothing in the database matches the indexed attribute.
On Friday 25 April 2008 11:28:44 Mikel.Santos@idom.com wrote:
Hi guys,
First of all, I would like to thank the community for such a great piece of software and encourage the developers to keep up the good work.
Until now, the setup of the ldap and the file configuration has been pretty straightforward. The ldap was performing great with few entries, but things got worse as the number of entries in the ldap increased. Currently we are experiencing serious performance problems when the searches performed exceed certain amount of entries. The "ldap_search" calls usually return after a few seconds but as the number of results matching the filter entered increases, the waiting time varies from one minute to many minutes.
This would appear to be a client-side problem. In my experience, the performance (rate at which entries are returned, minus overhead of connection setup) on an LDAP server will degrade as the number of entries in the database increases, not as the number of entries returned increases (assuming your cache is not interfering).
The scenario is the following:
- We are accessing openldap via PHP function calls
I think you should test via some other method to ensure that PHP is not the bottleneck. For example, maybe the amount of memory available to PHP is not sufficient for the number of entries you are returning.
You can easily determine this by running the ldapsearch utility with equivalent parameters to your php case. To eliminate any IO issues, redirect the results to /dev/null, e.g.:
$ time ldapsearch -x -D $binddn -b $basedn -w $pass -z $entries >/dev/null
(set the environment variables appropriately or replace them)
- The number of entries in the ldap is close to 100.000
- The hardware is the following: Intel(R) Xeon(TM) CPU 3.33GHz 1Gb RAM
- The slapd.conf file is the following:
/************************************************************************** **********************************/ modulepath /usr/lib/ldap moduleload back_bdb sizelimit -1 tool-threads 1 backend bdb checkpoint 512 30 database bdb directory "/var/lib/ldap" dbconfig set_cachesize 0 2097152 0 cachesize 100000 /************************************************************************** **********************************/
We have tried to increase both set_cachesize and cachesize (multiplying both by 100) and have not noticed a performance increase. I hope any of you can point us in the right direction since we are beginning to get desperated...
Unless you can reproduce the problem without php, I point you in the direction of php ...
Regards, Buchan
Hi Hallvard,
The fields I want to perform searches on are all indexed and I have run slapindex after adding "index" directives in slapd.conf. Still no way...
Thank you
Hallvard B Furuseth <h.b.furuseth@usi Para t.uio.no> Mikel.Santos@idom.com Enviado por: cc Hallvard Breien openldap-technical@openldap.org Furuseth Asunto <h.b.furuseth@usi Re: Serious performance problems t.uio.no>
25/04/2008 15:15
You need to add indexes for whatever you filter for. Without them, every search must examine all entries in scope for the search. Include 'index objectClass eq' no matter what else, since slapd often adds an objectClass component to your filter internally.
You can use loglevel 256 to see what your clients search for.
Also messges like <= bdb_<inequality, substring, etc>_candidates: (foo) not indexed if I remember correctly.
If the clients search for (&(foo=...)(bar=...)) you might only need an index for one of foo and bar, since each 'and' component narrows down the search. If they search for (|(foo=...)(bar=...)) you need both. Even if you don't have both attributes in the database, because slapd doesn't know that. Except you don't need (and can't) index attributes your included schema doesn't define.
If you add indexes by editing slapd.conf rather than via cn=config, stop slapd and run sbin/slapindex to actually build the indexes. Without slapindex, slapd will think the empty index means nothing in the database matches the indexed attribute.
-- Hallvard
Hi Buchan,
I have tested ldap performance with both slapcat and ldapsearch redirecting the output to /dev/null. The searches returning lots of matches, take from less than a second to three seconds max, which I guess it is the "expected" performance. Performance decreases when output is shown on the screen, though (which I find normal). So the bottleneck seems to be apache+PHP. Actually, I am pretty sure about this, since I have been watching slapd and apache processes and their CPU consumption lately, and apache raises up to 99-100% when these ldap searches are performed. I know this might not be the place to ask, but if anyone has experienced similar problems, please let me know.
Thank you for your time
Mikel
Buchan Milne <bgmilne@staff.te lkomsa.net> Para openldap-technical@openldap.org 25/04/2008 16:13 cc Mikel.Santos@idom.com Asunto Re: Serious performance problems
On Friday 25 April 2008 11:28:44 Mikel.Santos@idom.com wrote:
Hi guys,
First of all, I would like to thank the community for such a great piece
of
software and encourage the developers to keep up the good work.
Until now, the setup of the ldap and the file configuration has been
pretty
straightforward. The ldap was performing great with few entries, but
things
got worse as the number of entries in the ldap increased. Currently we are experiencing serious performance problems when the searches performed exceed certain amount of entries. The "ldap_search" calls usually return after a few seconds but as the number of results matching the filter entered increases, the waiting time varies from one minute to many minutes.
This would appear to be a client-side problem. In my experience, the performance (rate at which entries are returned, minus overhead of connection setup) on an LDAP server will degrade as the number of entries in the database increases, not as the number of entries returned increases (assuming your cache is not interfering).
The scenario is the following:
- We are accessing openldap via PHP function calls
I think you should test via some other method to ensure that PHP is not the
bottleneck. For example, maybe the amount of memory available to PHP is not
sufficient for the number of entries you are returning.
You can easily determine this by running the ldapsearch utility with equivalent parameters to your php case. To eliminate any IO issues, redirect the results to /dev/null, e.g.:
$ time ldapsearch -x -D $binddn -b $basedn -w $pass -z $entries >/dev/null
(set the environment variables appropriately or replace them)
- The number of entries in the ldap is close to 100.000
- The hardware is the following: Intel(R) Xeon(TM) CPU 3.33GHz 1Gb RAM
- The slapd.conf file is the following:
/**************************************************************************
**********************************/ modulepath /usr/lib/ldap moduleload back_bdb sizelimit -1 tool-threads 1 backend bdb checkpoint 512 30 database bdb directory "/var/lib/ldap" dbconfig set_cachesize 0 2097152 0 cachesize 100000
/**************************************************************************
**********************************/
We have tried to increase both set_cachesize and cachesize (multiplying both by 100) and have not noticed a performance increase. I hope any of you can point us in the right direction since we are beginning to get desperated...
Unless you can reproduce the problem without php, I point you in the direction of php ...
Regards, Buchan
openldap-technical@openldap.org
-
Buchan Milne -
Hallvard B Furuseth -
Mikel.Santos＠idom.com