Im running OpenLDAP 2.4.17 with 3 servers configured in multi-master mode (server 1 replicates to server 2 and server 3, server 2 replicates to server 1 and server 3, and server 3 replicates to server 1 and server 2).  While executing shell scripts to test replication, all 3 slapd instances began returning error code 32 to all requests.  No information could be retrieved from any of the 3 LDAP servers using ldapsearch, Apache Directory Studio, LDAP Administrator tool, or the LDAP Browser tool.  The slapd log records with debug level = sync contained numerous findbase failed! 32 errors. Starting and restarting the slapd instances still resulted in error code 32 being returned for any LDAP request.  I had to restore the LDAP databases to be able to view/update records in any of the 3 LDAP databases.

Server 1 configuration snippet:

overlay syncprov

syncprov-checkpoint 100 15

syncprov-sessionlog 5000

syncrepl rid=001

        provider=ldap://localhost:3892

        type=refreshAndPersist

        retry="60 60 300 +"

        searchbase="dc=authentx"

        scope=sub

        schemachecking=off

        bindmethod=simple

        binddn="cn=xroot,dc=authentx"

        credentials="980170contact"

syncrepl rid=002

        provider=ldap://localhost:3893

        type=refreshAndPersist

        retry="60 60 300 +"

        searchbase="dc=authentx"

        scope=sub

        schemachecking=off

        bindmethod=simple

        binddn="cn=xroot,dc=authentx"

        credentials="980170contact"

mirrormode      on

Server 2 configuration snippet:

overlay syncprov

syncprov-checkpoint 100 15

syncprov-sessionlog 5000

syncrepl rid=001

        provider=ldap://localhost:3891

        type=refreshAndPersist

        retry="60 60 300 +"

        searchbase="dc=authentx"

        scope=sub

        schemachecking=off

        bindmethod=simple

        binddn="cn=xroot,dc=authentx"

        credentials="980170contact"

syncrepl rid=002

        provider=ldap://localhost:3893

        type=refreshAndPersist

        retry="60 60 300 +"

        searchbase="dc=authentx"

        scope=sub

        schemachecking=off

        bindmethod=simple

        binddn="cn=xroot,dc=authentx"

        credentials="980170contact"

mirrormode      on

Server 3 configuration snippet:

overlay syncprov

syncprov-checkpoint 100 15

syncprov-sessionlog 5000

syncrepl rid=001

        provider=ldap://localhost:3891

        type=refreshAndPersist

        retry="60 60 300 +"

        searchbase="dc=authentx"

        scope=sub

        schemachecking=off

        bindmethod=simple

        binddn="cn=xroot,dc=authentx"

        credentials="980170contact"

syncrepl rid=002

        provider=ldap://localhost:3892

        type=refreshAndPersist

        retry="60 60 300 +"

        searchbase="dc=authentx"

        scope=sub

        schemachecking=off

        bindmethod=simple

        binddn="cn=xroot,dc=authentx"

        credentials="980170contact"

mirrormode      on

Snippets from log file from server 1 around the slapd error 32 problem time:

syncprov_sendresp: to=002, cookie=rid=001,sid=001,csn=20090811143737.466654Z#000000#001#000000

syncprov_sendresp: to=003, cookie=rid=001,sid=001,csn=20090811143737.466654Z#000000#001#000000

slap_queue_csn: queing 0xa2452d98 20090811143737.567781Z#000000#001#000000

syncprov_sendresp: to=002, cookie=rid=001,sid=001,csn=20090811143737.567781Z#000000#001#000000

slap_graduate_commit_csn: removing 0xa17098a8 20090811143737.567781Z#000000#001#000000

syncprov_sendresp: to=003, cookie=rid=001,sid=001,csn=20090811143737.567781Z#000000#001#000000

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

connection_input: conn=456 deferring operation: binding

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

findbase failed! 32

connection_input: conn=461 deferring operation: binding

Snippets from log file from server 2 around the slapd error 32 problem time:

slap_queue_csn: queing 0xa1709520 20090811143737.567781Z#000000#001#000000

slap_graduate_commit_csn: removing 0xa1709650 20090811143737.567781Z#000000#001#000000

syncrepl_updateCookie: rid=001 be_modify failed (32)

do_syncrepl: rid=001 rc 32 retrying (59 retries left)

syncprov_sendresp: to=003, cookie=rid=002,sid=002,csn=20090811143737.567781Z#000000#001#000000

do_syncrep2: cookie=rid=002,sid=003,csn=20090811143737.567781Z#000000#001#000000

syncrepl_entry: rid=002 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_DELETE)

syncrepl_entry: rid=002 be_search (32)

syncrepl_entry: rid=002 (null)

slap_queue_csn: queing 0xa1714320 20090811143737.567781Z#000000#001#000000

slap_graduate_commit_csn: removing 0xa1716558 20090811143737.567781Z#000000#001#000000

do_syncrepl: rid=002 rc 32 retrying (59 retries left)

findbase failed! 32

do_syncrep2: rid=001 LDAP_RES_SEARCH_RESULT

do_syncrep2: rid=001 LDAP_RES_SEARCH_RESULT (32) No such object

do_syncrep2: rid=001 (32) No such object

do_syncrepl: rid=001 rc -2 retrying (59 retries left)

do_syncrep2: rid=002 LDAP_RES_SEARCH_RESULT

do_syncrep2: rid=002 LDAP_RES_SEARCH_RESULT (32) No such object

do_syncrep2: rid=002 (32) No such object

do_syncrepl: rid=002 rc -2 retrying (59 retries left)

findbase failed! 32

Snippets from log file from server 3 around the slapd error 32 problem time:

slap_graduate_commit_csn: removing 0x835e498 20090811143737.466654Z#000000#001#000000

do_syncrep2: cookie=rid=001,sid=001,csn=20090811143737.567781Z#000000#001#000000

syncrepl_entry: rid=001 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_DELETE)

syncrepl_entry: rid=001 be_search (0)

syncrepl_entry: rid=001 dc=authentx

slap_queue_csn: queing 0xa160ee80 20090811143737.567781Z#000000#001#000000

syncprov_matchops: skipping original sid 001

slap_graduate_commit_csn: removing 0xa16134f8 20090811143737.567781Z#000000#001#000000

syncrepl_entry: rid=001 be_delete dc=authentx (0)

slap_queue_csn: queing 0xa160ee80 20090811143737.567781Z#000000#001#000000

slap_graduate_commit_csn: removing 0xa16134f8 20090811143737.567781Z#000000#001#000000

syncrepl_updateCookie: rid=001 be_modify failed (32)

do_syncrepl: rid=001 rc 32 retrying (59 retries left)

syncprov_sendresp: to=002, cookie=rid=002,sid=003,csn=20090811143737.567781Z#000000#001#000000

do_syncrep2: cookie=rid=002,sid=002,csn=20090811143737.567781Z#000000#001#000000

syncrepl_entry: rid=002 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_DELETE)

syncrepl_entry: rid=002 be_search (32)

syncrepl_entry: rid=002 (null)

slap_queue_csn: queing 0x8338688 20090811143737.567781Z#000000#001#000000

slap_graduate_commit_csn: removing 0x8360318 20090811143737.567781Z#000000#001#000000

do_syncrepl: rid=002 rc 32 retrying (59 retries left)

do_syncrep2: rid=002 LDAP_RES_SEARCH_RESULT

do_syncrep2: rid=002 LDAP_RES_SEARCH_RESULT (32) No such object

do_syncrep2: rid=002 (32) No such object

do_syncrepl: rid=002 rc -2 retrying (59 retries left)

findbase failed! 32

I have been unable to reproduce this problem.  Any suggestions on how to prevent this problem from occurring and how to recover when this problem occurs?

Thanks,

Barry Colston