Howard,
I tried bigger caches but I do not have enough memory to apply them. This was the reason I only tried the dncachesize to speed up search queries.
I also have this same database running in a old openldap version(2.1.30) even with a little more records on it. In this version, as I understand, there isn't any cache at openLDAP but only at BDB. See how it is behaving in terms of memory and CPU from the openLDAP 2.1.30:
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND 3651 root 24 0 136M 135M 126M S 0.0 1.1 0:00 1 slapd
See a really small memory consumption and I really reasonable performance. The only issue I have with this version is about the replication mechanism which I would like to increase its availability using syncrepl unless slurpd.
The problem is that for version after 2.1 looks like we need to have enough memory to cache all database since there are many situations where slapd appear increase CPU or memory usage and considerably reduce performance.
I tried to remove from slapd.conf any cache constraint seeing if the previous version performance directly from disk reads would be reproduced. I saw some good startup, like 500 returns per second, but after sometime slapd hanged and did not return any more records to ldapseach. Also it consumed all 4 CPUs :
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4409 ldap 15 0 400m 183m 132m S 400 1.5 27:22.70 slapd
And even after I stop the ldapsearch the CPU consumption continues consuming all CPUs processing. I believe it entered in a dead loop.
I do not have a heavily load system but based in records and DBs I have some memory resource constraints.
I also tried some smaller caches, like :
cachesize 500000 dncachesize 500000 idlcachesize 500000 cachefree 10000
But it also hangs the search after sometime.
I was wondering if there is a way to run slapd without cache, reading from disk(like first time read to insert record in cache), what is enough for small/medium systems in terms of consulting. In this way I could use the behavior as the 2.1.30 system and the new syncrepl replication.
Thanks a lot !
Rodrigo.
Howard Chu wrote:
Quanah Gibson-Mount wrote:
--On Wednesday, June 17, 2009 5:25 AM -0700 Rodrigo Costa rlvcosta@yahoo.com wrote:
Could this be a configuration issue? I do not think but I'm putting below my cache configuration :
Clearly it is a configuration issue.
# Cache values cachesize 10000 dncachesize 3000000 idlcachesize 10000 cachefree 10
These values are extremely low (except for dncachesize) for a system with 4 million records. I'd expect something more like:
cachesize 4000000 dncachesize 3000000
dncachesize must always be >= cachesize
idlcachesize 8000000 cachefree 100000
or something along those lines. Particularly in the case of your ldapsearch, cachesize is likely the most relevant. Try playing with the settings a bit more.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Rodrigo Costa wrote:
Howard,
I tried bigger caches but I do not have enough memory to apply them. This was the reason I only tried the dncachesize to speed up search queries.
I also have this same database running in a old openldap version(2.1.30) even with a little more records on it. In this version, as I understand, there isn't any cache at openLDAP but only at BDB.
False. back-bdb was originally written without entry caching, but it was never released that way, and entry and IDL caching are both present in 2.1.30.
See how it is behaving in terms of memory and CPU from the openLDAP 2.1.30:
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND 3651 root 24 0 136M 135M 126M S 0.0 1.1 0:00 1 slapd
See a really small memory consumption and I really reasonable performance. The only issue I have with this version is about the replication mechanism which I would like to increase its availability using syncrepl unless slurpd.
The problem is that for version after 2.1 looks like we need to have enough memory to cache all database since there are many situations where slapd appear increase CPU or memory usage and considerably reduce performance.
Not all, but as the documentation says, the dncache needs to be that large. None of the other caches are as critical.
I tried to remove from slapd.conf any cache constraint seeing if the previous version performance directly from disk reads would be reproduced. I saw some good startup, like 500 returns per second, but after sometime slapd hanged and did not return any more records to ldapseach. Also it consumed all 4 CPUs :
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4409 ldap 15 0 400m 183m 132m S 400 1.5 27:22.70 slapd
And even after I stop the ldapsearch the CPU consumption continues consuming all CPUs processing. I believe it entered in a dead loop.
You should get a gdb snapshot of this situation so we can see where the loop is occurring.
I do not have a heavily load system but based in records and DBs I have some memory resource constraints.
Performance is directly related to memory. If the DB is too large to fit in memory then you're stuck with disk I/Os for most operations and nothing can improve that. You're being mislead by your observation of "initially good results" - you're just seeing the filesystem cache at first, but when it gets exhausted then you see the true performance of your disks.
I also tried some smaller caches, like :
cachesize 500000 dncachesize 500000 idlcachesize 500000 cachefree 10000
But it also hangs the search after sometime.
I was wondering if there is a way to run slapd without cache, reading from disk(like first time read to insert record in cache), what is enough for small/medium systems in terms of consulting. In this way I could use the behavior as the 2.1.30 system and the new syncrepl replication.
The 2.4 code is quite different from 2.1, there is no way to get the same behavior.
Howard Chu writes:
Not all, but as the documentation says, the dncache needs to be that large.
Where does it say that? I see nothing in the Guide and merely recommended sizes in the slapd-bdb(5) manpage.
--On Wednesday, June 17, 2009 11:50 PM +0200 Hallvard B Furuseth h.b.furuseth@usit.uio.no wrote:
Howard Chu writes:
Not all, but as the documentation says, the dncache needs to be that large.
Where does it say that? I see nothing in the Guide and merely recommended sizes in the slapd-bdb(5) manpage.
It doesn't, but it should note the lower bound.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
openldap-software@openldap.org
-
Hallvard B Furuseth -
Howard Chu -
Quanah Gibson-Mount -
Rodrigo Costa