On Monday 15 June 2009 04:44:21 Rodrigo Costa wrote:
OpenLdap group,
I'm having a possible issue that could be a problem. I have a DB with
around 4 million entrances. In my slapd.conf I use the following cache
constraints :
#Cache values
cachesize 10000
dncachesize 3000000
idlcachesize 10000
cachefree 10
Your idlcachesize might be too small for your cachesize (but, you don't say
which backend you are using). You haven't listed any Berkeley DB tuning.
I'm also running the system in 2 machines in MirrorMode without problems
about this configuration.
My DB has exactly 3882992 entrances. In this way the dncachesize is
smaller than the number of records. After I move the dncache constraint
to a size smaller than the number of records(memory concern) I start to
have some issues related with ldapsearch, for example.
After the number of entrances in cache match the constraints(always pass
a little) the system hangs for new searches. It appears to be for
records "not yet" cached. If a record not cached is searched the
ldapsearch bind but hangs during search. One example can be seen below :
[root@brtldp11 ~]# time ldapsearch -LLL -x -D
"cn=admin,ou=CONTENT,o=domain,c=fr" -w secret -b
"ou=CONTENT,o=domain,c=fr" -H ldap://10.142.15.170:389
'pnnumber=+554184011071'
real 0m40.140s
user 0m0.003s
sys 0m0.001s
Just after I press CTRL-C the command stopped. It would stay forever in
this state. This happens after I ldapsearch the full DB and cache is
filled.
But, not every entry in the DB is cached. slapd needs to access the remaining
entries by some means to determine if they match the filter or not.
If I then search this same record in the mirror I have the return
very
fast. See example :
[root@brtldp11 ~]# time ldapsearch -LLL -x -D
"cn=admin,ou=CONTENT,o=domain,c=fr" -w secret -b
"ou=CONTENT,o=domain,c=fr" -H ldap://10.142.15.172:389
'pnnumber=+554184011071'
dn:
pnnumber=\2B554184011071,uid=1219843774965\2B554184011071,ou=REPOSITORY,ou
=CONTENT,o=domain,c=fr
subpnid: 0
pntype: 2
pncaps: 7
objectClass: phoneinfo
pnnumber: +554184011071
real 0m0.257s
user 0m0.002s
sys 0m0.003s
Most likely because the OS filesystem cache has the index cached in memory. You
may want to consider filling the OS filesystem cache (e.g. by copying files
larger than the total system memory) to check ...
So this is doesn't appear to be a record problem since also both
systems
are in mirror mode. Also the return is in a reasonable time, in around
257 miliseconds. After cached it can even be faster.
What is cached? Cached in which cache? Are you sure? How did you check?
See the number of entrances I could search before system hangs :
[root@brtldp12 ~]# wc -l /backup/temp2.txt
3078804 /backup/temp2.txt
How did you populate this file? With what search (specifically the filter).
The number was 3,078,804 records since even dncache boundary is
3,000,000 it always pass a little.
Appears that if a cache boundary is smaller than number of DB records
slapd can hang when searching for new records not yet cached. Since I
have multiple DBs with this order of records I needed to impose this
boundary smaller than the number of records. My expectation was some
performance degradation after cache is filled but not that system could
hang(at least for non cached records).
You may be thrashing the Berkeley DB cache, if your indexes are too big for
the cachesize you have set in DB_CONFIG (if you haven't then the default of
256k is definitely way too much for 3 million records).
Also, the fact that you seem to only be searching on the naming attribute may
be giving you a false idea of the real performance of your directory within
the dncachesize limits (I am not sure if slapd can answer searches on the
naming attribute faster from the dn cache than searchs on a non-naming
attribute ...).
After this situation happens the slapd process does not ends
correctly
only after kill -9 is given in the process.
I would like to know if someone already passed by this situation and I
believe it can be easily reproduced with cache configuration smaller
than the number of records. I believe any DB in this situation could
reproduce the same behavior.
Anyone running an OpenLDAP directory with the bdb or hdb backend with more
than 10 000 entries who doesn't know what their Berkeley DB configuration is
like in terms of DB cache is looking for problems.
Regards,
Buchan