Full_Name: Rein Tollevik Version: CVS head OS: linux and solaris URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (81.93.160.250)
We have seen deadlock/hang situations in our master server today, which looks like a deadlock caused by the si_csn_rwlock lock being held while syncprov_checkpoint() is running. The patch at the end should (I hope) fix this.
The scenario is a server with a the syncprov overlay on a glue database with multiple bdb backends as subordinate. One of them is a syncrepl consumer, although I don't know if that matters.
A modify operation on the syncrepl consumer causes a checkpoint of the contextCSN. The resulting modify on the glue backend hangs in bdb_cache_modify() awaiting the bdb_cache_entry_db_relock() to return a write lock on the glue suffix entry. At the same time is a modify on one of the other backends hanging in syncprov_op_response() waiting for a lock on the si_csn_rwlock which the first thread holds. The remaining threads quickly blocks waiting for a read lock on the glue suffix entry (the most common search base). I recon the read locks are queued to give priority to the thread waiting for the write lock.
What I haven't figured out is who that holds the lock on the glue suffix entry in the first place... Anyway, holding the si_csn_rwlock while the contextCSN is written to the database should not be necessary. But I don't know what consequences it whould have if more than one thread should end up checkpointing competing values, so I have added a new flag to the syncprov_info structure to prevent simultaneons checkpoints from occuring rather than using locks to do it.
Rein Tollevik Basefarm AS
Index: OpenLDAP/servers/slapd/overlays/syncprov.c diff -u OpenLDAP/servers/slapd/overlays/syncprov.c:1.1.1.13 OpenLDAP/servers/slapd/overlays/syncprov.c:1.12 --- OpenLDAP/servers/slapd/overlays/syncprov.c:1.1.1.13 Mon Mar 31 17:29:48 2008 +++ OpenLDAP/servers/slapd/overlays/syncprov.c Tue Apr 1 21:54:05 2008 @@ -128,6 +128,7 @@ int si_numops; /* number of ops since last checkpoint */ int si_nopres; /* Skip present phase */ int si_usehint; /* use reload hint */ + int si_check; /* are we checkpointing? */ time_t si_chklast; /* time of last checkpoint */ Avlnode *si_mods; /* entries being modified */ sessionlog *si_logs; @@ -1308,8 +1309,10 @@ slap_callback cb = {0}; BackendDB be;
+ ldap_pvt_thread_rdwr_rlock( &si->si_csn_rwlock ); mod.sml_numvals = si->si_numcsns; mod.sml_values = si->si_ctxcsn; + ldap_pvt_thread_rdwr_runlock( &si->si_csn_rwlock ); mod.sml_nvalues = NULL; mod.sml_desc = slap_schema.si_ad_contextCSN; mod.sml_op = LDAP_MOD_REPLACE; @@ -1335,6 +1338,8 @@ if ( mod.sml_next != NULL ) { slap_mods_free( mod.sml_next, 1 ); } + + si->si_check = 0; }
static void @@ -1612,7 +1617,7 @@ }
si->si_numops++; - if ( si->si_chkops || si->si_chktime ) { + if ( !si->si_check && (si->si_chkops || si->si_chktime) ) { if ( si->si_chkops && si->si_numops >= si->si_chkops ) { do_check = 1; si->si_numops = 0; @@ -1627,12 +1632,14 @@ } } } + + if ( do_check ) + si->si_check = 1; + ldap_pvt_thread_rdwr_wunlock( &si->si_csn_rwlock );
if ( do_check ) { - ldap_pvt_thread_rdwr_rlock( &si->si_csn_rwlock ); syncprov_checkpoint( op, rs, on ); - ldap_pvt_thread_rdwr_runlock( &si->si_csn_rwlock ); }
opc->sctxcsn.bv_len = maxcsn.bv_len;