On Thu, 3 Apr 2008, rein@basefarm.no wrote:
Our persistent syncrepl consumers stops receiving data after a while, with no indication of why :-( They don't recognize restarts of the producer, so the only way to get the replication running again is to restart the consumers.
So far it looks to me as if the syncrepl thread has managed to return without adding the connection socket back to slap_daemon.sd_readers. But whether that is correct or not, and how it has managed to do it if so, I cannot tell.
Hm, I hope I have found the race condition that causes this :-) I'm now running with the patch at the end to see if that solves it, only time will tell..
The race is that between the time selecting on the syncrepl socket is enabled by the call to connection_client_enable() and the release of the si_mutex a new message may arrive. If so, the next call to do_syncrepl may fail in its attempt to trylock the mutex and no-one will re-enable selecting on it again. My patch delays enabling of the socket until the mutex has been released, which looks safe to me. Or can the access to si->si_conn without a lock be a problem?
Rein
Index: OpenLDAP/servers/slapd/syncrepl.c diff -u OpenLDAP/servers/slapd/syncrepl.c:1.9 OpenLDAP/servers/slapd/syncrepl.c:1.10 --- OpenLDAP/servers/slapd/syncrepl.c:1.9 Fri Mar 28 14:25:55 2008 +++ OpenLDAP/servers/slapd/syncrepl.c Fri Apr 4 16:54:05 2008 @@ -1166,6 +1166,7 @@ ber_socket_t s; int i, defer = 1, fail = 0; Backend *be; + int enable_conn = 0;
Debug( LDAP_DEBUG_TRACE, "=>do_syncrepl %s\n", si->si_ridtxt, 0, 0 );
@@ -1271,11 +1272,7 @@ * If we failed, tear down the connection and reschedule. */ if ( rc == LDAP_SUCCESS ) { - if ( si->si_conn ) { - connection_client_enable( si->si_conn ); - } else { - si->si_conn = connection_client_setup( s, do_syncrepl, arg ); - } + enable_conn = 1; } else if ( si->si_conn ) { dostop = 1; } @@ -1342,6 +1339,14 @@ ldap_pvt_thread_mutex_unlock( &slapd_rq.rq_mutex ); ldap_pvt_thread_mutex_unlock( &si->si_mutex );
+ if ( enable_conn ) { + if ( si->si_conn ) { + connection_client_enable( si->si_conn ); + } else { + si->si_conn = connection_client_setup( s, do_syncrepl, arg ); + } + } + if ( rc ) { if ( fail == RETRYNUM_TAIL ) { Debug( LDAP_DEBUG_ANY,