I have to mention that openldap 2.4.23 is an ubuntu package and compiled with db4.8.30

I did some other tries: using just a standard replication (one provider) it works. I stopped the consumer during the mass delete and start again, and it's synced

I noticed some messages: Mar 23 15:36:14 ldaprelay slapd[24616]: syncrepl_del_nonpresent: rid=020 be_delete uid=t606435,ou=people,dc=univ-yy,dc=fr (0)

I then restarted again with two providers, but modify two things. To prevent a warning I added a memberof overlay, modify the url to add a trailing /, the stopped that server during mass deletes and it stays like this:

Mar 23 15:41:39 ldaprelay slapd[24632]: @(#) $OpenLDAP: slapd 2.4.23 (Nov 19 2010 17:41:28) $#012#011buildd@allspice :/build/buildd/openldap-2.4.23/debian/build/servers/slapd Mar 23 15:41:39 ldaprelay slapd[24634]: slapd starting Mar 23 15:41:39 ldaprelay slapd[24634]: do_syncrep2: rid=040 LDAP_RES_INTERMEDIATE - REFRESH_DELETE Mar 23 15:41:39 ldaprelay slapd[24634]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - SYNC_ID_SET Mar 23 15:41:42 ldaprelay slapd[24634]: last message repeated 175 times Mar 23 15:41:42 ldaprelay slapd[24634]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - REFRESH_PRESENT Mar 23 15:41:42 ldaprelay slapd[24634]: do_syncrep2: rid=020 cookie=rid=020,sid=020,csn=20110323144119.644587Z#000000#020#000000 Mar 23 15:41:42 ldaprelay slapd[24634]: slap_queue_csn: queing 0x7f674545b4b0 20110323144119.644587Z#000000#020#000000 Mar 23 15:41:42 ldaprelay slapd[24634]: slap_graduate_commit_csn: removing 0x7f674545b570 20110323144119.644587Z#000000#020#000000

In this case, still 14000 entries not deleted. I don't understand what to do. One provider, it's working, two providers no. The second one has just 10 entries and no traffic. So is me setup OK, should we add something on the provider side?

Thanks

Dom

2011/3/23 LALOT Dominique dom.lalot@gmail.com

Hello,

I am testing the replication feature in a multimaster environment replicating into a single database. As stated before, I added serverid to my providers. I just have two providers for test purpose. I tested mass updates on a provider, stopped my replica during updates, then start again and it's OK, it updates the entries If I do the same for mass deletes. I deleted 40000 entries while stopping the consumer. My consumer is still with 30000 undeleted entries. I left the consumer for hours, restarting it twice. It seems there is no regular compare between consumer or provider in such situation. I'll simplify to test in a single provider setup, to see if it works.

All servers are 2.4.23

consumer side. A script is populating the DIT via slapadd first to create the dc entries.

database bdb suffix "dc=fr"

serverid 0x40 ldap://ldapmaitre.univ-yy.fr syncrepl rid=040 provider=ldap://ldapmaitre.univ-yy.fr/ type=refreshAndPersist searchbase="dc=univ-yy,dc=fr" retry="60 10 300 +" scope=sub filter="(objectClass=*)" attrs="*,+" schemachecking=off bindmethod=simple

serverid 0x20 ldap://anutest.univ-xx.fr syncrepl rid=020 provider=ldap://anutest.univ-xx.fr/ type=refreshAndPersist searchbase="dc=univ-xx,dc=fr" retry="60 10 300 +" scope=sub filter="(objectClass=*)" attrs="*,+" schemachecking=off bindmethod=simple

Mar 23 13:29:11 ldaprelay slapd[24319]: @(#) $OpenLDAP: slapd 2.4.23 (Nov 19 2010 17:41:28) $#012#011buildd@allspice :/build/buildd/openldap-2.4.23/debian/build/servers/slapd Mar 23 13:29:11 ldaprelay slapd[24320]: slapd starting Mar 23 13:29:11 ldaprelay slapd[24320]: do_syncrep2: rid=040 LDAP_RES_INTERMEDIATE - REFRESH_DELETE Mar 23 13:29:11 ldaprelay slapd[24320]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - REFRESH_DELETE

root@ldaprelay:/etc/ldap# ldapsearch -x -LLL -s base contextcsn dn: dc=fr contextCSN: 20110323101636.221613Z#000000#040#000000 contextCSN: 20110323100346.844382Z#000000#020#000000

Thanks for any hints

Dom

-- Dominique LALOT Ingénieur Systèmes et Réseaux http://annuaire.univmed.fr/showuser.php?uid=lalot

Hi Howard,

We were told to migrate to 2.4.23 sometimes ago, and we did some work to update our production servers. Can I try 2.4.24 only on the consumer side? It would be a pain to migrate all servers to 2.4.24 without package.

is this related to the last fixes?

Fixed slapd syncrepl reuse of presence list (ITS#6707) Fixed slapd syncrepl uninitialized return code (ITS#6719) Fixed slapd syncrepl variable initialization (ITS#6739) Fixed slapd syncrepl refresh to use complete cookie (ITS#6807)

Thanks

Dom

2011/3/23 Howard Chu hyc@symas.com

LALOT Dominique wrote:

...

Hello,

I am testing the replication feature in a multimaster environment replicating into a single database. As stated before, I added serverid to my providers. I just have two providers for test purpose. I tested mass updates on a provider, stopped my replica during updates, then start again and it's OK, it updates the entries If I do the same for mass deletes. I deleted 40000 entries while stopping the consumer. My consumer is still with 30000 undeleted entries. I left the consumer for hours, restarting it twice. It seems there is no regular compare between consumer or provider in such situation. I'll simplify to test in a single provider setup, to see if it works.

All servers are 2.4.23

Please try your test with 2.4.24 instead.

-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

LALOT Dominique wrote:

Howard,

I was obliged to remove slapd package on consumer. Then compile in 2.4.24 and restart. doing the same tests, there was nothing diffferent. provider is still 2.4.23 My test: delete 30000 entries, stop consumer when deleting; start again consumer when it's finished. Consumer is then out of sync.

You haven't posted your provider config. It appears you're using syncprov-sessionlog. Obviously both ends of this system are vital to solving the puzzle.

Mar 23 17:27:50 ldaprelay slapd[4358]: slapd stopped. Mar 23 17:34:45 ldaprelay slapd[4577]: @(#) $OpenLDAP: slapd 2.4.24 (Mar 23 2011 16:52:04) $#012#011root@ldaprelay.univ-amu.fr:/usr/local/src/openldap-2.4.24/servers/slapd Mar 23 17:34:45 ldaprelay slapd[4579]: slapd starting Mar 23 17:34:45 ldaprelay slapd[4579]: do_syncrep2: rid=040 LDAP_RES_INTERMEDIATE - REFRESH_DELETE Mar 23 17:34:45 ldaprelay slapd[4579]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - SYNC_ID_SET Mar 23 17:34:48 ldaprelay slapd[4579]: last message repeated 175 times Mar 23 17:34:48 ldaprelay slapd[4579]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - REFRESH_PRESENT Mar 23 17:34:48 ldaprelay slapd[4579]: do_syncrep2: rid=020 cookie=rid=020,sid=020,csn=20110323163416.105518Z#000000#020#000000 Mar 23 17:34:48 ldaprelay slapd[4579]: slap_queue_csn: queing 0x1a35560 20110323163416.105518Z#000000#020#000000 Mar 23 17:34:48 ldaprelay slapd[4579]: slap_graduate_commit_csn: removing 0x1a35620 20110323163416.105518Z#000000#020#000000

I changed the provider to 2.4.24 that makes deletes. Hopefully this one was built on tar.gz That makes no difference, after restarting the consumer, it does not delete extra entries

Dom

2011/3/23 LALOT Dominique <dom.lalot@gmail.com mailto:dom.lalot@gmail.com>

Hi Howard,

We were told to migrate to 2.4.23 sometimes ago, and we did some work to
update our production servers. Can I try 2.4.24 only on the consumer side?
It would be a pain to migrate all servers to 2.4.24 without package.

is this related to the last fixes?

Fixed slapd syncrepl reuse of presence list (ITS#6707)
Fixed slapd syncrepl uninitialized return code (ITS#6719)
Fixed slapd syncrepl variable initialization (ITS#6739)


Fixed slapd syncrepl refresh to use complete cookie (ITS#6807)

Thanks

Dom


2011/3/23 Howard Chu <hyc@symas.com <mailto:hyc@symas.com>>

    LALOT Dominique wrote:

        Hello,

        I am testing the replication feature in a multimaster environment
        replicating
        into a single database. As stated before, I added serverid to my
        providers. I
        just have two providers for test purpose.
        I tested mass updates on a provider, stopped my replica during
        updates, then
        start again and it's OK, it updates the entries
        If I do the same for mass deletes. I deleted 40000 entries while
        stopping the
        consumer. My consumer is still with 30000 undeleted entries. I
        left the
        consumer for hours, restarting it twice.
        It seems there is no regular compare between consumer or provider
        in such
        situation. I'll simplify to test in a single provider setup, to
        see if it works.

        All servers are 2.4.23


    Please try your test with 2.4.24 instead.

Well,

To resume my tests:

provider and consumer in 2.4.24 -> same behaviour Get rid of session log -> same behaviour

If I removed one of the providers of my consumer, everything went fine. Even, if I stop the consumer during mass deletes, it sync very fast as soon as it starts again That's related to several providers for one consumer and may be something else I don't see. The second provider (first rid in my conf) has less that 15 entries and no updates at all.

I changed the order of the providers in my slapd.conf. There is no difference.

Dom

Mar 24 11:36:32 ldaprelay slapd[5444]: slapd starting Mar 24 11:36:32 ldaprelay slapd[5444]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - SYNC_ID_SET Mar 24 11:36:32 ldaprelay slapd[5444]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - SYNC_ID_SET Mar 24 11:36:32 ldaprelay slapd[5444]: do_syncrep2: rid=040 LDAP_RES_INTERMEDIATE - REFRESH_DELETE Mar 24 11:36:32 ldaprelay slapd[5444]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - SYNC_ID_SET Mar 24 11:36:33 ldaprelay slapd[5444]: last message repeated 200 times Mar 24 11:36:33 ldaprelay slapd[5444]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - REFRESH_PRESENT Mar 24 11:36:33 ldaprelay slapd[5444]: do_syncrep2: rid=020 cookie=rid=020,sid=020,csn=20110324103606.185647Z#000000#020#000000 Mar 24 11:36:33 ldaprelay slapd[5444]: slap_queue_csn: queing 0x227dfe0 20110324103606.185647Z#000000#020#000000 Mar 24 11:36:33 ldaprelay slapd[5444]: slap_graduate_commit_csn: removing 0x227e0a0 20110324103606.185647Z#000000#020#000000

# consumer # big provider and mass deletes serverid 0x20 ldap://anutest.univ.fr/ syncrepl rid=020 provider=ldap://anutest.univ.fr/ type=refreshAndPersist searchbase="dc=univ,dc=fr" retry="60 10 300 +" scope=sub filter="(objectClass=*)" attrs="*,+" schemachecking=off bindmethod=simple

# small provider serverid 0x40 ldap://ldapmaitre.univ-a.fr/ syncrepl rid=040 provider=ldap://ldapmaitre.univ-a.fr/ type=refreshAndPersist searchbase="dc=univ-a,dc=fr" retry="60 10 300 +" scope=sub filter="(objectClass=*)" attrs="*,+" schemachecking=off bindmethod=simple

context on the servers

# univ.fr dn: dc=univ,dc=fr contextCSN: 20110324103606.185647Z#000000#020#000000

# univ-a.fr dn: dc=univ-a,dc=fr contextCSN: 20110323101636.221613Z#000000#040#000000

dn: dc=fr contextCSN: 20110324103606.185647Z#000000#020#000000 contextCSN: 20110323101636.221613Z#000000#040#000000

2011/3/24 LALOT Dominique dom.lalot@gmail.com

Here is my provider, and we use syncprov-sessionlog

database bdb suffix "dc=univ-yy,dc=fr" directory /var/ldap/base/ sizelimit -1 cachesize 50000 dbconfig set_cachesize 0 140000000 1 dbconfig set_flags DB_LOG_AUTOREMOVE dbconfig set_tas_spins 1 dbconfig set_lg_regionmax 262144 idlcachesize 150000 checkpoint 1024 5 # flush tous les 1Mo ou 5 minutes #dbnosync syncprov-checkpoint 100 10 syncprov-sessionlog 100 index objectClass,entryCSN,entryUUID eq

...

Should we use a session log? I'll try without. That's tricky to understand each parameter. Would be great to have something anaylizing more than syntax, relations between parameters and database cache optimization.

Dom

2011/3/23 Howard Chu hyc@symas.com

...

LALOT Dominique wrote:

...

Howard,

I was obliged to remove slapd package on consumer. Then compile in 2.4.24 and restart. doing the same tests, there was nothing diffferent. provider is still 2.4.23 My test: delete 30000 entries, stop consumer when deleting; start again consumer when it's finished. Consumer is then out of sync.

You haven't posted your provider config. It appears you're using syncprov-sessionlog. Obviously both ends of this system are vital to solving the puzzle.

Mar 23 17:27:50 ldaprelay slapd[4358]: slapd stopped.

...

Mar 23 17:34:45 ldaprelay slapd[4577]: @(#) $OpenLDAP: slapd 2.4.24 (Mar 23 2011 16:52:04) $#012#011root@ldaprelay.univ-amu.fr: /usr/local/src/openldap-2.4.24/servers/slapd Mar 23 17:34:45 ldaprelay slapd[4579]: slapd starting Mar 23 17:34:45 ldaprelay slapd[4579]: do_syncrep2: rid=040 LDAP_RES_INTERMEDIATE - REFRESH_DELETE Mar 23 17:34:45 ldaprelay slapd[4579]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - SYNC_ID_SET Mar 23 17:34:48 ldaprelay slapd[4579]: last message repeated 175 times Mar 23 17:34:48 ldaprelay slapd[4579]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - REFRESH_PRESENT Mar 23 17:34:48 ldaprelay slapd[4579]: do_syncrep2: rid=020 cookie=rid=020,sid=020,csn=20110323163416.105518Z#000000#020#000000 Mar 23 17:34:48 ldaprelay slapd[4579]: slap_queue_csn: queing 0x1a35560 20110323163416.105518Z#000000#020#000000 Mar 23 17:34:48 ldaprelay slapd[4579]: slap_graduate_commit_csn: removing 0x1a35620 20110323163416.105518Z#000000#020#000000

I changed the provider to 2.4.24 that makes deletes. Hopefully this one was built on tar.gz That makes no difference, after restarting the consumer, it does not delete extra entries

Dom

2011/3/23 LALOT Dominique <dom.lalot@gmail.com mailto: dom.lalot@gmail.com>

Hi Howard,

We were told to migrate to 2.4.23 sometimes ago, and we did some work to update our production servers. Can I try 2.4.24 only on the consumer side? It would be a pain to migrate all servers to 2.4.24 without package.

is this related to the last fixes?

Fixed slapd syncrepl reuse of presence list (ITS#6707) Fixed slapd syncrepl uninitialized return code (ITS#6719) Fixed slapd syncrepl variable initialization (ITS#6739)

Fixed slapd syncrepl refresh to use complete cookie (ITS#6807)

Thanks

Dom

2011/3/23 Howard Chu <hyc@symas.com mailto:hyc@symas.com>

   LALOT Dominique wrote:

       Hello,

       I am testing the replication feature in a multimaster

environment replicating into a single database. As stated before, I added serverid to my providers. I just have two providers for test purpose. I tested mass updates on a provider, stopped my replica during updates, then start again and it's OK, it updates the entries If I do the same for mass deletes. I deleted 40000 entries while stopping the consumer. My consumer is still with 30000 undeleted entries. I left the consumer for hours, restarting it twice. It seems there is no regular compare between consumer or provider in such situation. I'll simplify to test in a single provider setup, to see if it works.

       All servers are 2.4.23


   Please try your test with 2.4.24 instead.

-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

-- Dominique LALOT Ingénieur Systèmes et Réseaux http://annuaire.univmed.fr/showuser.php?uid=lalot