I'm having trouble getting the consumer synced in reasonable time. My
tests were with fewer than 20 entries in the datastore and I saw no
problems.
But we have 260,000 inetOrgPersons (with only a few attributes for
each user: uid cn sn givenName mail userPassword).
I've set up syncrepl:
PROVIDER
> # Indices to maintain for this database
> index objectclass,entryCSN,entryUUID eq
> index ou,cn,mail,surname,givenname eq,sub
> index uidNumber,gidNumber,loginShell eq
> index uid,memberUid eq,sub
> index nisMapName,nisMapEntry eq,sub
>
> overlay syncprov
> syncprov-checkpoint 100 1
> syncprov-sessionlog 100
>
> limits dn.children="ou=replicators,dc=service,dc=utoronto,dc=ca"
> size=unlimited time=unlimited
(I index attributes I'm not currently using. I presume that's not the
problem.)
CONSUMER
> syncrepl rid=123
> provider=ldap://PROVIDER:389
> type=refreshAndPersist
> interval=00:00:10:00
> retry="60 10 300 +"
> searchbase="dc=service,dc=utoronto,dc=ca"
> filter="(objectClass=*)"
> scope=sub
> schemachecking=off
> starttls=critical
> bindmethod=simple
>
> binddn="uid=replicator,ou=replicators,dc=service,dc=utoronto,dc=ca"
I've tried with and without slapcat/slapadd to initialize the
consumer. On our slower system, slapadd took 98 minutes to rebuild the
database; the faster was 35 minutes (and I have only one consumer
right now).
A full transfer via syncrepl is slow: 10 entries per second:
> # < /var/log/daemon egrep 'bdb_add: added id=' | cut -b1-50 | uniq -
> c | tail -10
> 10 Apr 1 10:57:31 ldap2 slapd[3126]: bdb_add: added
> 11 Apr 1 10:57:32 ldap2 slapd[3126]: bdb_add: added
> 9 Apr 1 10:57:33 ldap2 slapd[3126]: bdb_add: added
> 9 Apr 1 10:57:34 ldap2 slapd[3126]: bdb_add: added
> 10 Apr 1 10:57:35 ldap2 slapd[3126]: bdb_add: added
> 9 Apr 1 10:57:36 ldap2 slapd[3126]: bdb_add: added
> 10 Apr 1 10:57:37 ldap2 slapd[3126]: bdb_add: added
> 10 Apr 1 10:57:38 ldap2 slapd[3126]: bdb_add: added
> 10 Apr 1 10:57:39 ldap2 slapd[3126]: bdb_add: added
> 5 Apr 1 10:57:40 ldap2 slapd[3126]: bdb_add: added
With the consumer using the slapadded initial database, syncrepl seems
to be reviewing every entry, 15 entries per second:
> # tail -10000 /var/log/daemon | egrep 'entry unchanged' | cut -
> b1-83 | uniq -c
> 8 Apr 1 11:30:57 ldap2 slapd[3782]: syncrepl_entry: rid=123
> entry unchanged, ignored
> 15 Apr 1 11:30:58 ldap2 slapd[3782]: syncrepl_entry: rid=123
> entry unchanged, ignored
> 14 Apr 1 11:30:59 ldap2 slapd[3782]: syncrepl_entry: rid=123
> entry unchanged, ignored
> 15 Apr 1 11:31:00 ldap2 slapd[3782]: syncrepl_entry: rid=123
> entry unchanged, ignored
> 15 Apr 1 11:31:01 ldap2 slapd[3782]: syncrepl_entry: rid=123
> entry unchanged, ignored
> 15 Apr 1 11:31:02 ldap2 slapd[3782]: syncrepl_entry: rid=123
> entry unchanged, ignored
> 14 Apr 1 11:31:03 ldap2 slapd[3782]: syncrepl_entry: rid=123
> entry unchanged, ignored
> 14 Apr 1 11:31:04 ldap2 slapd[3782]: syncrepl_entry: rid=123
> entry unchanged, ignored
> 15 Apr 1 11:31:05 ldap2 slapd[3782]: syncrepl_entry: rid=123
> entry unchanged, ignored
> 2 Apr 1 11:31:06 ldap2 slapd[3782]: syncrepl_entry: rid=123
> entry unchanged, ignored
I can only hope it'll be done in 5 hours. The datastore isn't active,
so the consumer is up to date, for now, but this needless work is time
consuming.
Is this normal?
What happens when I restart the consumer? Why should I expect it to be
faster restarting?
I changed one entry (adding displayName to my own entry) after the
slapcat, so the consumer did not have the change when the consumer
started 90 minutes ago. That update has not yet propagated.
Can I prime the consumer's syncrepl cookie (if that's an appropriate
term)? Is that a solution? And how would I do that?
Thanks for your time,
Paul