On Mon, Mar 09, 2009 at 05:06:37PM -0700, Rodrigo Costa wrote:
I'm using BDB4.7 with all patches available and using OpenLDAP 2.4.15 HEAD since I'm also testing ITS#5860 resolution I posted sometime ago.
[snip]
The problem is exactly after sometime passes and then slapd into consumer has CPU utilization almost fixed in 200% and the slapd process in provider becomes iddle(CPU 0% utilized by slapd).
This appears to indicate the search ended but something in consumer is not working ok since too much CPU is being utilized.
Other strange thing is if I start a ldapsearch with consumer(slave) the responsiveness is very low indicating something is really wrong. I put a ldapsearch and after some yours and killing the slapd process I could see that only around 50 thousands entraces were passed. This is a really slow performance.
[snip]
Other strange behavior I was not expecting, is that even the search is from the consumer to provider I see the Cache information, using monitor interface, growing in consumer. So I see in consumer something like :
[snip]
olmBDBEntryCache: 9999 olmBDBDNCache: 1309605 olmBDBIDLCache: 4715
We're seeing the same thing, with 2.4.15 + the second-round fix from ITS#5860 and fully patched BDB 4.7. FWIW, this is with existing databases, not a fresh syncrepl.
After a couple days of slapd uptime, its BDB entry and DN caches have grown to about 400k entries each and slapd's memory footprint is increasing at ~100MB/hour. CPU consumption skyrockets and performance takes a nose dive. The CPU consumption seems to be spread across all threads; I can get another backtrace next time it happens, if that will help.
openldap 1992 1992 0.0 82.1 5274396 3337904 ? Ssl Mar09 0:00 /usr/sbin/slapd -h ldap:/// ldaps:/// -g openldap -u openldap -f /etc/ldap/slapd.conf openldap 1992 1994 0.6 82.1 5274396 3337904 ? Rsl Mar09 9:57 /usr/sbin/slapd -h ldap:/// ldaps:/// -g openldap -u openldap -f /etc/ldap/slapd.conf openldap 1992 1995 6.6 82.1 5274396 3337904 ? Rsl Mar09 97:36 /usr/sbin/slapd -h ldap:/// ldaps:/// -g openldap -u openldap -f /etc/ldap/slapd.conf openldap 1992 1996 6.5 82.1 5274396 3337904 ? Rsl Mar09 96:24 /usr/sbin/slapd -h ldap:/// ldaps:/// -g openldap -u openldap -f /etc/ldap/slapd.conf openldap 1992 1997 6.6 82.1 5274396 3337904 ? Rsl Mar09 97:13 /usr/sbin/slapd -h ldap:/// ldaps:/// -g openldap -u openldap -f /etc/ldap/slapd.conf openldap 1992 1998 6.6 82.1 5274396 3337904 ? Rsl Mar09 97:13 /usr/sbin/slapd -h ldap:/// ldaps:/// -g openldap -u openldap -f /etc/ldap/slapd.conf openldap 1992 1999 6.5 82.1 5274396 3337904 ? Rsl Mar09 96:13 /usr/sbin/slapd -h ldap:/// ldaps:/// -g openldap -u openldap -f /etc/ldap/slapd.conf openldap 1992 2000 6.6 82.1 5274396 3337904 ? Rsl Mar09 97:17 /usr/sbin/slapd -h ldap:/// ldaps:/// -g openldap -u openldap -f /etc/ldap/slapd.conf openldap 1992 2001 6.6 82.1 5274396 3337904 ? Rsl Mar09 96:56 /usr/sbin/slapd -h ldap:/// ldaps:/// -g openldap -u openldap -f /etc/ldap/slapd.conf openldap 1992 2002 6.7 82.1 5274396 3337904 ? Rsl Mar09 99:02 /usr/sbin/slapd -h ldap:/// ldaps:/// -g openldap -u openldap -f /etc/ldap/slapd.conf openldap 1992 2003 6.6 82.1 5274396 3337904 ? Ssl Mar09 97:18 /usr/sbin/slapd -h ldap:/// ldaps:/// -g openldap -u openldap -f /etc/ldap/slapd.conf openldap 1992 2004 6.7 82.1 5274396 3337904 ? Rsl Mar09 99:31 /usr/sbin/slapd -h ldap:/// ldaps:/// -g openldap -u openldap -f /etc/ldap/slapd.conf openldap 1992 2005 6.6 82.1 5274396 3337904 ? Rsl Mar09 96:44 /usr/sbin/slapd -h ldap:/// ldaps:/// -g openldap -u openldap -f /etc/ldap/slapd.conf openldap 1992 2006 6.6 82.1 5274396 3337904 ? Rsl Mar09 97:10 /usr/sbin/slapd -h ldap:/// ldaps:/// -g openldap -u openldap -f /etc/ldap/slapd.conf
john