I had an unexpected and completely undocumented crash of slapd this morning. I'm looking for some hints on tracking it down.
Here's the background.
We are running 2.3.41 (locally built RPM) on RedHat EL 4 with four slave servers (running the same 2.3.41 and RHEL4). We use a nightly update process where we slapcat the master database, apply the changes from the systems of record (students, employees, retirees, etc) to the LDIF, generate an ldapmodify data stream and run ldapmodify to apply the changes.
The student system made some massive changes this morning which caused us to generate an ldapmodify input file with 31,973 changes (adds, modifies, modrdn's) in it. The ldapmodify on the master took 8 minutes. The delta-syncrepl to the slave/replica servers took 33 to 44 minutes. The replica delta-syncrepl processes seem to have been averaging about 800 changes per minute, which is quite slow for what I was expecting.
Since it took so long for the replica's to get all the changes, they fell more than the 10 minutes behind the master server and the person on call got paged (nagios monitoring of the replica and master CSN's). The person on call had not been properly trained (my fault) to look for the syncrepl messages in the syslog on the replica servers and thus they issued a restart on one of the replicas (thinking that something was hung). The replica restarted properly, but the master seems to have crashed without a sound at the same time. There was no core file generated and I haven't found anything logged in the syslog on the master. slapd was started on the master, and the output of the startup says that the accesslog database had an unclean shutdown and needed to be recovered (which it was successfully).
I'm wondering the following things:
1) Is it possible that one of the ITS's for syncrepl that will be included in 2.3.42 would address this crash? Any suggestions on tracking down why it crashed?
2) Does it appear that I have a configuration problem (the delta-syncrepl taking about five times as long to get the changes out to the replicas as it took to apply them on the master)? Where would you suggest I look if it is likely?
Thanks,
--On Thursday, May 22, 2008 10:07 AM -0400 Francis Swasey Frank.Swasey@uvm.edu wrote:
I'm wondering the following things:
- Is it possible that one of the ITS's for syncrepl that will be
included in 2.3.42 would address this crash? Any suggestions on tracking down why it crashed?
You may have hit the accesslog callbacks bug (ITS#5490).
- Does it appear that I have a configuration problem (the delta-syncrepl
taking about five times as long to get the changes out to the replicas as it took to apply them on the master)? Where would you suggest I look if it is likely?
Have you set up a DB_CONFIG file for your accesslog DB that's reasonable?
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On 5/22/08 12:12 PM, Quanah Gibson-Mount wrote:
--On Thursday, May 22, 2008 10:07 AM -0400 Francis Swasey Frank.Swasey@uvm.edu wrote:
I'm wondering the following things:
- Is it possible that one of the ITS's for syncrepl that will be
included in 2.3.42 would address this crash? Any suggestions on tracking down why it crashed?
You may have hit the accesslog callbacks bug (ITS#5490).
Looks likely. Then, I need to research why I didn't get a core file (likely a ulimit problem)
- Does it appear that I have a configuration problem (the delta-syncrepl
taking about five times as long to get the changes out to the replicas as it took to apply them on the master)? Where would you suggest I look if it is likely?
Have you set up a DB_CONFIG file for your accesslog DB that's reasonable?
I think it's reasonable ;-) I have two DB's and 4GB of memory on this system. The DB_CONFIG for the accesslog DB is as follows:
set_cachesize 0 524288000 2 set_lg_bsize 2097152 set_lg_regionmax 262144
Have I missed something that would be more reasonable?
The DB_CONFIG for the other DB has the following: set_cachesize 0 524288000 2 set_lg_bsize 2097152 set_lg_regionmax 262144 set_lk_max_locks 4000 set_lk_max_objects 2000
(The lock/objects were increased from the defaults in response to lock/object issues with 2.2 a couple of years ago).
Frank
--On Thursday, May 22, 2008 12:54 PM -0400 Francis Swasey Frank.Swasey@uvm.edu wrote:
On 5/22/08 12:12 PM, Quanah Gibson-Mount wrote:
--On Thursday, May 22, 2008 10:07 AM -0400 Francis Swasey Frank.Swasey@uvm.edu wrote:
I'm wondering the following things:
- Is it possible that one of the ITS's for syncrepl that will be
included in 2.3.42 would address this crash? Any suggestions on tracking down why it crashed?
You may have hit the accesslog callbacks bug (ITS#5490).
Looks likely. Then, I need to research why I didn't get a core file (likely a ulimit problem)
- Does it appear that I have a configuration problem (the
delta-syncrepl taking about five times as long to get the changes out to the replicas as it took to apply them on the master)? Where would you suggest I look if it is likely?
Have you set up a DB_CONFIG file for your accesslog DB that's reasonable?
I think it's reasonable ;-) I have two DB's and 4GB of memory on this system. The DB_CONFIG for the accesslog DB is as follows:
set_cachesize 0 524288000 2 set_lg_bsize 2097152 set_lg_regionmax 262144
Have I missed something that would be more reasonable?
I don't know without knowing the size of your accesslog db. ;)
Personally I avoid splitting memory regions if possible, but that may only apply to BDB 4.2.52, I've never tested that aspect of configuration with anything more recent.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On 5/22/08 12:59 PM, Quanah Gibson-Mount wrote:
set_cachesize 0 524288000 2 set_lg_bsize 2097152 set_lg_regionmax 262144
Have I missed something that would be more reasonable?
I don't know without knowing the size of your accesslog db. ;)
Sorry...
My accesslog DB currently has just over 48000 entries. The automatic purging runs every 8 hours and gets rid of everything that is older than 5 days.
If you wanted the actual file sizes: -rw-r--r-- 1 ldap ldap 383 Jan 30 2006 DB_CONFIG -rw------- 1 ldap ldap 24576 Jul 10 2007 __db.006 -rw------- 1 ldap ldap 450560 Jul 10 2007 __db.005 -rw------- 1 ldap ldap 2359296 Jul 10 2007 __db.004 -rw------- 1 ldap ldap 262144000 Jul 10 2007 __db.003 -rw------- 1 ldap ldap 262144000 Jul 10 2007 __db.002 -rw------- 1 ldap ldap 16384 Jul 10 2007 __db.001 -rw------- 1 ldap ldap 1323008 May 22 13:05 reqStart.bdb -rw------- 1 ldap ldap 679936 May 22 13:05 reqResult.bdb -rw------- 1 ldap ldap 1323008 May 22 13:05 reqEnd.bdb -rw------- 1 ldap ldap 2732032 May 22 13:05 objectClass.bdb -rw------- 1 ldap ldap 79740928 May 22 13:05 id2entry.bdb -rw------- 1 ldap ldap 1327104 May 22 13:05 entryCSN.bdb -rw------- 1 ldap ldap 12718080 May 22 13:05 dn2id.bdb -rw-r--r-- 1 ldap ldap 4096 May 22 13:05 alock
Personally I avoid splitting memory regions if possible, but that may only apply to BDB 4.2.52, I've never tested that aspect of configuration with anything more recent.
I am running 4.2.52... and the split memory regions are a carry over from when this was running on a 2GB machine and it seemed to run better with multiple segments (or at least that was the theory at the time).
--On Thursday, May 22, 2008 1:15 PM -0400 Francis Swasey Frank.Swasey@uvm.edu wrote:
Sorry...
My accesslog DB currently has just over 48000 entries. The automatic purging runs every 8 hours and gets rid of everything that is older than 5 days.
Hm, well, I hit an interesting bug with purging and delta-sync recently. It can't be fixed in 2.3 because the fix causes another bug to get worse. There will be a workaround for it in 2.3.42.
Everything tuning wise seems reasonable as far as the database goes. I agree the performance on replication can be a bit slow in a high-write scenario particularly if there are multiple replicas competing for resources.
How many threads does the master slapd have available to it? Also, how much of a cache do you have defined for syncprov and the accesslog db?
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On 5/22/08 1:59 PM, Quanah Gibson-Mount wrote:
--On Thursday, May 22, 2008 1:15 PM -0400 Francis Swasey Frank.Swasey@uvm.edu wrote:
Sorry...
My accesslog DB currently has just over 48000 entries. The automatic purging runs every 8 hours and gets rid of everything that is older than 5 days.
Hm, well, I hit an interesting bug with purging and delta-sync recently. It can't be fixed in 2.3 because the fix causes another bug to get worse. There will be a workaround for it in 2.3.42.
Hm, I vaguely remember that discussion.
Everything tuning wise seems reasonable as far as the database goes. I agree the performance on replication can be a bit slow in a high-write scenario particularly if there are multiple replicas competing for resources.
How many threads does the master slapd have available to it? Also, how much of a cache do you have defined for syncprov and the accesslog db?
The master has the default of 16 threads (ie, I don't specify the threads keyword in the slapd.conf file).
As for the cache for syncprov and accesslog db, I am not certain of what you are asking for. I don't have any "cache" type statements for either of them in the slapd.conf file and I'm not setting syncprov-sessionlog either.
Frank
--On Thursday, May 22, 2008 2:55 PM -0400 Francis Swasey Frank.Swasey@uvm.edu wrote:
How many threads does the master slapd have available to it? Also, how much of a cache do you have defined for syncprov and the accesslog db?
The master has the default of 16 threads (ie, I don't specify the threads keyword in the slapd.conf file).
As for the cache for syncprov and accesslog db, I am not certain of what you are asking for. I don't have any "cache" type statements for either of them in the slapd.conf file and I'm not setting syncprov-sessionlog either.
You definitely should have a syncprov-sessionlog setting (syncprov cache). It'll help with anything except ADD ops.
Also, for the accesslog DB, you may want to set a cachesize value. I haven't really experimented with that, but it could be worthwhile.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
--On Thursday, May 22, 2008 2:55 PM -0400 Francis Swasey Frank.Swasey@uvm.edu wrote:
How many threads does the master slapd have available to it? Also, how much of a cache do you have defined for syncprov and the accesslog db?
The master has the default of 16 threads (ie, I don't specify the threads keyword in the slapd.conf file).
As for the cache for syncprov and accesslog db, I am not certain of what you are asking for. I don't have any "cache" type statements for either of them in the slapd.conf file and I'm not setting syncprov-sessionlog either.
You definitely should have a syncprov-sessionlog setting (syncprov cache). It'll help with anything except ADD ops.
No, the syncprov-sessionlog is totally superfluous when you're using delta-syncrepl.
Also, for the accesslog DB, you may want to set a cachesize value. I haven't really experimented with that, but it could be worthwhile.
It doesn't need to be anything huge. The volume of entries in the accesslog DB is pretty much always going to be much smaller than the main DB.
On 5/22/08 4:13 PM, Howard Chu wrote:
No, the syncprov-sessionlog is totally superfluous when you're using delta-syncrepl.
Ok, I'll stop looking at that one.
Also, for the accesslog DB, you may want to set a cachesize value. I haven't really experimented with that, but it could be worthwhile.
It doesn't need to be anything huge. The volume of entries in the accesslog DB is pretty much always going to be much smaller than the main DB.
Re-reading the slapd-hdb man page again, looks like the cachesize defaults to 1000 and the idlcachesize defaults to 0. The man page suggests that the idlcachesize should be at least 3x the cachesize. Does that guideline hold for accesslog DB as well?
As an aside, researching the past 30 days of nightly updates, I find that the 32000 updates that happened yesterday are abnormal (happens only at semester change -- so, four times a year). The normal number of updates is in the 200-2300 range (the average being 842, the median being 806).
So, I'm thinking something along the lines of a cachesize of 5000 and idlcachesize of 15000. Does that make sense?
Thanks,
openldap-software@openldap.org
-
Francis Swasey -
Howard Chu -
Quanah Gibson-Mount