Hello,
Here at UCI we have just deployed syncrepl to replicate our LDAP directory. Unfortunately, we use an old directory structure (PH/QI) for our master database, and ldap gets updates from that system in batch. The batch process updates our master LDAP server, and syncrepl is used to push the changes out to all the other LDAP servers. This causes somewhere around 100 to 1000 entries to be updated all at once (out of 80k + entries in the LDAP directory).
The problem we are experiencing is that when a syncrepl slave receives a bunch of updates, queries to these slaves slow down tremendously. We are talking going from a sub second query time for a single LDAP entry, to, in some cases, over 20 seconds response time for simple queries. This is causing all sorts of problems for us. One thing to note: the master does basically the same updates, but through an normal ldap client, rather than through syncrepl -- and it does not experience this slowness.
We are using BDB, Openldap ver. 2.3.28 and our syncrepl entry looks something like this:
syncrepl rid=11 provider=ldap://ldap3.nac.uci.edu:389 type=refreshAndPersist interval=0:00:00:05 searchbase="OU=University of California Irvine, O=University of California, C=US" filter="(objectClass=*)" scope=sub schemachecking=off sizelimit=0 timelimit=0 updatedn="cn=root,OU=University of California Irvine, O=Universi ty of California, C=US" bindmethod=simple binddn="uid=nsp,OU=University of California Irvine, O=University of California, C=US" credentials="REMOVED" starttls=yes retry=1,2,3,4,5,+
Any help would be appreciated,
-Paul Main Network and Academic Computing Services University of California, Irvine
--On Tuesday, December 12, 2006 12:44 PM -0800 Paul Main pmain@uci.edu wrote:
Hello,
Here at UCI we have just deployed syncrepl to replicate our LDAPdirectory. Unfortunately, we use an old directory structure (PH/QI)for our master database, and ldap gets updates from that system inbatch. The batch process updates our master LDAP server, andsyncrepl is used to push the changes out to all the other LDAPservers. This causes somewhere around 100 to 1000 entries to beupdated all at once (out of 80k + entries in the LDAP directory).
The problem we are experiencing is that when a syncrepl slavereceives a bunch of updates, queries to these slaves slow downtremendously. We are talking going from a sub second query time fora single LDAP entry, to, in some cases, over 20 seconds response timefor simple queries. This is causing all sorts of problems for us. One thing to note: the master does basically the same updates, butthrough an normal ldap client, rather than through syncrepl -- and itdoes not experience this slowness.
We are using BDB, Openldap ver. 2.3.28 and our syncrepl entry lookssomething like this:
syncrepl rid=11 provider=ldap://ldap3.nac.uci.edu:389 type=refreshAndPersist interval=0:00:00:05 searchbase="OU=University of California Irvine,O=University of California, C=US" filter="(objectClass=*)" scope=sub schemachecking=off sizelimit=0 timelimit=0 updatedn="cn=root,OU=University of CaliforniaIrvine, O=Universi ty of California, C=US" bindmethod=simple binddn="uid=nsp,OU=University of California Irvine,O=University of California, C=US" credentials="REMOVED" starttls=yes retry=1,2,3,4,5,+
To start...
(a) interval is not valid when using refreshAndPersist. This is a very common error, and I don't get why people make it, the man page clearly states:
In the refreshOnly operation, the next synchronization search operation is periodically rescheduled at an interval time (specified by interval parameter; 1 day by default)
(b) Your values for sizelimit and timelimit are suspect. Per the man page:
The sizelimit and timelimit only accept "unlimited" and positive integers, and both default to "unlimited".
(c) There's no need to specify the filter since you are using the default filter
In other bits, you don't provide your BDB configuration at all (i.e., what the DB_CONFIG file has), nor do you note db size vs. memory size, cache settings, etc.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
causing all sorts of problems for us. One thing to note: the master does basically the same updates, but through an normal ldap client, rather than through syncrepl -- and it does not experience this slowness.
syncrepl (re)sends the entire entry, so it's a lot more intense than your typical MOD. I seem to be the only person willing to live with this, but it's quite common to use "delta-syncrepl" instead, which only sends the changed attributes. There should be ample documentation out there; read up and come back if you have questions. (I'll probably change off of traditional syncrepl when I move production to 2.4.)
Apart from that, make sure that your slaves are sanely configured in terms of cache, DB_CONFIG, etc. It sounds like you've got your master tuned to your liking, so you could follow lessons learned there.
--On Tuesday, December 12, 2006 7:57 PM -0500 Aaron Richton richton@nbcs.rutgers.edu wrote:
causing all sorts of problems for us. One thing to note: the master does basically the same updates, but through an normal ldap client, rather than through syncrepl -- and it does not experience this slowness.
syncrepl (re)sends the entire entry, so it's a lot more intense than your typical MOD. I seem to be the only person willing to live with this, but it's quite common to use "delta-syncrepl" instead, which only sends the changed attributes. There should be ample documentation out there; read up and come back if you have questions. (I'll probably change off of traditional syncrepl when I move production to 2.4.)
Oh, very good point, I'd forgotten that bit (I'm one of those delta-syncrepl users. :P ).
Symas has a tech tip on this at:
http://www.connexitor.com/forums/viewtopic.php?t=3
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
openldap-software@openldap.org
-
Aaron Richton -
Paul Main -
Quanah Gibson-Mount