syncrepl findbase failed! 32 errors

I’m running OpenLDAP 2.4.17 with 3 servers configured in multi-master mode (server 1 replicates to server 2 and server 3, server 2 replicates to server 1 and server 3, and server 3 replicates to server 1 and server 2). While executing shell scripts to test replication, all 3 slapd instances began returning error code 32 to all requests. No information could be retrieved from any of the 3 LDAP servers using ldapsearch, Apache Directory Studio, LDAP Administrator tool, or the LDAP Browser tool. The slapd log records with debug level = sync contained numerous “findbase failed! 32” errors. Starting and restarting the slapd instances still resulted in error code 32 being returned for any LDAP request. I had to restore the LDAP databases to be able to view/update records in any of the 3 LDAP databases.

Server 1 configuration snippet:

overlay syncprov

syncprov-checkpoint 100 15

syncprov-sessionlog 5000

syncrepl rid=001

provider=ldap://localhost:3892

type=refreshAndPersist

retry="60 60 300 +"

searchbase="dc=authentx"

scope=sub

schemachecking=off

bindmethod=simple

binddn="cn=xroot,dc=authentx"

credentials="980170contact"

syncrepl rid=002

provider=ldap://localhost:3893

type=refreshAndPersist

retry="60 60 300 +"

searchbase="dc=authentx"

scope=sub

schemachecking=off

bindmethod=simple

binddn="cn=xroot,dc=authentx"

credentials="980170contact"

mirrormode on

Server 2 configuration snippet:

overlay syncprov

syncprov-checkpoint 100 15

syncprov-sessionlog 5000

syncrepl rid=001

provider=ldap://localhost:3891

type=refreshAndPersist

retry="60 60 300 +"

searchbase="dc=authentx"

scope=sub

schemachecking=off

bindmethod=simple

binddn="cn=xroot,dc=authentx"

credentials="980170contact"

syncrepl rid=002

provider=ldap://localhost:3893

type=refreshAndPersist

retry="60 60 300 +"

searchbase="dc=authentx"

scope=sub

schemachecking=off

bindmethod=simple

binddn="cn=xroot,dc=authentx"

credentials="980170contact"

mirrormode on

Server 3 configuration snippet:

overlay syncprov

syncprov-checkpoint 100 15

syncprov-sessionlog 5000

syncrepl rid=001

provider=ldap://localhost:3891

type=refreshAndPersist

retry="60 60 300 +"

searchbase="dc=authentx"

scope=sub

schemachecking=off

bindmethod=simple

binddn="cn=xroot,dc=authentx"

credentials="980170contact"

syncrepl rid=002

provider=ldap://localhost:3892

type=refreshAndPersist

retry="60 60 300 +"

searchbase="dc=authentx"

scope=sub

schemachecking=off

bindmethod=simple

binddn="cn=xroot,dc=authentx"

credentials="980170contact"

mirrormode on

Snippets from log file from server 1 around the slapd error 32 problem time:

syncprov_sendresp: to=002, cookie=rid=001,sid=001,csn=20090811143737.466654Z#000000#001#000000

syncprov_sendresp: to=003, cookie=rid=001,sid=001,csn=20090811143737.466654Z#000000#001#000000

slap_queue_csn: queing 0xa2452d98 20090811143737.567781Z#000000#001#000000

syncprov_sendresp: to=002, cookie=rid=001,sid=001,csn=20090811143737.567781Z#000000#001#000000

slap_graduate_commit_csn: removing 0xa17098a8 20090811143737.567781Z#000000#001#000000

syncprov_sendresp: to=003, cookie=rid=001,sid=001,csn=20090811143737.567781Z#000000#001#000000

findbase failed! 32

connection_input: conn=456 deferring operation: binding

findbase failed! 32

connection_input: conn=461 deferring operation: binding

Snippets from log file from server 2 around the slapd error 32 problem time:

slap_queue_csn: queing 0xa1709520 20090811143737.567781Z#000000#001#000000

slap_graduate_commit_csn: removing 0xa1709650 20090811143737.567781Z#000000#001#000000

syncrepl_updateCookie: rid=001 be_modify failed (32)

do_syncrepl: rid=001 rc 32 retrying (59 retries left)

syncprov_sendresp: to=003, cookie=rid=002,sid=002,csn=20090811143737.567781Z#000000#001#000000

do_syncrep2: cookie=rid=002,sid=003,csn=20090811143737.567781Z#000000#001#000000

syncrepl_entry: rid=002 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_DELETE)

syncrepl_entry: rid=002 be_search (32)

syncrepl_entry: rid=002 (null)

slap_queue_csn: queing 0xa1714320 20090811143737.567781Z#000000#001#000000

slap_graduate_commit_csn: removing 0xa1716558 20090811143737.567781Z#000000#001#000000

do_syncrepl: rid=002 rc 32 retrying (59 retries left)

findbase failed! 32

do_syncrep2: rid=001 LDAP_RES_SEARCH_RESULT

do_syncrep2: rid=001 LDAP_RES_SEARCH_RESULT (32) No such object

do_syncrep2: rid=001 (32) No such object

do_syncrepl: rid=001 rc -2 retrying (59 retries left)

do_syncrep2: rid=002 LDAP_RES_SEARCH_RESULT

do_syncrep2: rid=002 LDAP_RES_SEARCH_RESULT (32) No such object

do_syncrep2: rid=002 (32) No such object

do_syncrepl: rid=002 rc -2 retrying (59 retries left)

findbase failed! 32

Snippets from log file from server 3 around the slapd error 32 problem time:

slap_graduate_commit_csn: removing 0x835e498 20090811143737.466654Z#000000#001#000000

do_syncrep2: cookie=rid=001,sid=001,csn=20090811143737.567781Z#000000#001#000000

syncrepl_entry: rid=001 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_DELETE)

syncrepl_entry: rid=001 be_search (0)

syncrepl_entry: rid=001 dc=authentx

slap_queue_csn: queing 0xa160ee80 20090811143737.567781Z#000000#001#000000

syncprov_matchops: skipping original sid 001

slap_graduate_commit_csn: removing 0xa16134f8 20090811143737.567781Z#000000#001#000000

syncrepl_entry: rid=001 be_delete dc=authentx (0)

slap_queue_csn: queing 0xa160ee80 20090811143737.567781Z#000000#001#000000

slap_graduate_commit_csn: removing 0xa16134f8 20090811143737.567781Z#000000#001#000000

syncrepl_updateCookie: rid=001 be_modify failed (32)

do_syncrepl: rid=001 rc 32 retrying (59 retries left)

syncprov_sendresp: to=002, cookie=rid=002,sid=003,csn=20090811143737.567781Z#000000#001#000000

do_syncrep2: cookie=rid=002,sid=002,csn=20090811143737.567781Z#000000#001#000000

syncrepl_entry: rid=002 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_DELETE)

syncrepl_entry: rid=002 be_search (32)

syncrepl_entry: rid=002 (null)

slap_queue_csn: queing 0x8338688 20090811143737.567781Z#000000#001#000000

slap_graduate_commit_csn: removing 0x8360318 20090811143737.567781Z#000000#001#000000

do_syncrepl: rid=002 rc 32 retrying (59 retries left)

do_syncrep2: rid=002 LDAP_RES_SEARCH_RESULT

do_syncrep2: rid=002 LDAP_RES_SEARCH_RESULT (32) No such object

do_syncrep2: rid=002 (32) No such object

do_syncrepl: rid=002 rc -2 retrying (59 retries left)

findbase failed! 32

I have been unable to reproduce this problem. Any suggestions on how to prevent this problem from occurring and how to recover when this problem occurs?

Thanks,

Barry Colston