New subject: Help: Slow LDAP search with high %iowait

10 Sep 2008

      Would you folks mind sharing some thoughts/ideas on this? Thanks a lot!
--- On Mon, 9/8/08, Victor victorfuman@yahoo.com wrote:
...
From: Victor victorfuman@yahoo.com
Subject: Help: Slow LDAP search with high %iowait
To: openldap-technical@openldap.org
Date: Monday, September 8, 2008, 5:35 PM
Hi ,
I did quite a bit reading and research before I send email
to this list for help. If I have missed some basic concepts
here, please execuse my ignorance. Thanks for your help and
time in advance.

Summary

The initial search in my prototyping with OpenLDAP (slapd +
BDB) seemed to be  slow.  What is the reason and How could I
fix it?

Configuration

2.1 Environment
Linux CentOS, 1 hard disk (therefore unfortunately the BDB
transaction logs and database files  are written to the same
disk), 120GB disk space (80% unused), 1GB RAM, reserved for
this prototyping, OpenLDAP 2.3.39 with default BDB
installation
2.2   slapd.conf (modified trivially for discussion
purpose)
# global configuration
loglevel        0
# BDB
database        bdb
suffix          "dc=test,dc=dummy,dc=com"
rootdn         
"cn=Manager,dc=test,dc=dummy,dc=com"
# Cleartext passwords, especially for the rootdn, should
# be avoid.  See slappasswd(8) and slapd.conf(5) for
details.
# Use of strong authentication encouraged.
rootpw          secret
# The database directory MUST exist prior to running slapd
AND
# should only be accessible by the slapd and slap tools.
# Mode 700 recommended.
directory       /usr/local/var/openldap-data
#Other DB configuration
idlcachesize    60000
cachesize       20000
# Indices to maintain
# the indexes are to support search in first name, last
name and email for both exact match and wild cards in the
end
index   objectClass     eq
index   gn              pres,eq,sub
index   sn              pres,eq,sub
index   mail            pres,eq,sub
2.3 DB_CONFIG (for BDB)
set_cachesize 0 52428800 1
set_lg_bsize 2097512
set_flags DB_LOG_AUTOREMOVE
set_lg_regionmax 262144
2.4 Data setup
2 million records (users with gn, sn, email, mobile, street
address, etc. in the BDB; all records are indexed using the
index in the above slapd.conf;  grouped by the first
character of lastName. For example,
dn: ou=Z,dc=test,dc=dummy,dc=com
objectclass: organizationalUnit
ou: Z
Sample LDIF entry:
#Directory Entry
dn:
uid=ABCDEFGHIJKLMNOPQRSTUVWXYZ123459,ou=F,dc=test,dc=dummy,dc=com
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
uid: ABCDEFGHIJKLMNOPQRSTUVWXYZ123459
...... (details omitted)

Symptom/Problem

It was very slow in the first (fresh) search if I searched
by wildcard firstname only like "Larry*" (which
returned 478 entries/users). The response time was generally
higher than 5 seconds Depending the count of records found,
the response time might exceed 20 or even 50 seconds. During
the search, the "iostat" result showed +95%
%iowait, await was much higher that svctm, the device %util
was over 96%. Here is the "iostat" output:
Time: 10:51:34 AM
avg-cpu:  %user   %nice    %sys %iowait   %idle
           3.10    0.00    1.40   95.50    0.00
Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s   
rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
hda          0.00   2.90 64.94 65.33 1322.68  580.22  
661.34   290.11    14.61    51.99  343.51   7.44  96.92
dm-0         0.00   0.00 64.94 72.53 1322.68  580.22  
661.34   290.11    13.84    55.44  330.62   7.06  96.99
dm-1         0.00   0.00  0.00  0.00    0.00    0.00    
0.00     0.00     0.00     0.00    0.00   0.00   0.00
However, The subsequent search (using the exact search
criteria) is  much faster (within 200ms). I believe it is
because of the cache.
I did a "db_stat -m" check and saw +90% cache hit
rate (I guess it is normal?). The detailed output is in the
attachment.

Questions

The "iostat" output showed obvious I/O
bottleneck. Assuming I can't upgrade my hardware (for
example, adding another disk specifically for writing
transaction logs to), assuming I won't set a limit to
the max number of entried returned, is there anything else I
can do (typically BDB/slapd tuning or configuration) to make
the fresh/first search much faster (say within 2 seconds for
the worst case)? Did I do anything wrong? Please  advise.
Thanks a lot!
Vic

Re: Help: Slow LDAP search with high %iowait