> Will have to do some more thinking. There is a race condition here somewhere, just not sure where it is yet.
The race seems to be here, in bdb_tool_entry_close(), introduced by commit 31175da3 while fixing ITS#6853:
slapd_shutdown = 1; #ifdef USE_TRICKLE ldap_pvt_thread_mutex_lock( &bdb_tool_trickle_mutex );
/* trickle thread may not have started yet */ while ( !bdb_tool_trickle_active ) ldap_pvt_thread_cond_wait( &bdb_tool_trickle_cond_end, &bdb_tool_trickle_mutex );
The comment says "trickle thread may not have started yet", but doesn't seem to account for the fact that the trickle thread may already have finished. This is primarily because this function is not the only one that can cond_signal on bdb_tool_trickle_cond, here's another one in bdb_tool_entry_put():
#ifdef USE_TRICKLE if (( slapMode & SLAP_TOOL_QUICK ) && (( e->e_id & 0xfff ) == 0xfff )) { ldap_pvt_thread_cond_signal( &bdb_tool_trickle_cond ); } #endif
... and we were running slapadd with the -q option. I think the chain of events that can lead to the situation I saw is:
bdb_tool_entry_close(): slapd_shutdown = 1; bdb_tool_entry_put(): ldap_pvt_thread_cond_signal( &bdb_tool_trickle_cond ); bdb_tool_trickle_task(): bdb_tool_trickle_active = 0; bdb_tool_entry_close(): ldap_pvt_thread_mutex_lock( &bdb_tool_trickle_mutex ); bdb_tool_trickle_task(): ldap_pvt_thread_cond_signal( &bdb_tool_trickle_cond_end ); bdb_tool_trickle_task(): ldap_pvt_thread_mutex_unlock( &bdb_tool_trickle_mutex ); bdb_tool_entry_close(): while ( !bdb_tool_trickle_active ) bdb_tool_entry_close(): ldap_pvt_thread_cond_wait( &bdb_tool_trickle_cond_end, &bdb_tool_trickle_mutex );
So perhaps we need bdb_tool_trickle_active to additionally indicate whether in fact bdb_tool_trickle_task() has finished already. How about this?
--- servers/slapd/back-bdb/tools.c.orig 2012-02-29 17:37:09.000000000 +0000 +++ servers/slapd/back-bdb/tools.c 2016-05-27 13:10:31.453451000 +0100 @@ -174,7 +174,7 @@ &bdb_tool_trickle_mutex );
ldap_pvt_thread_cond_signal( &bdb_tool_trickle_cond ); - while ( bdb_tool_trickle_active ) + while ( bdb_tool_trickle_active > 0 ) ldap_pvt_thread_cond_wait( &bdb_tool_trickle_cond_end, &bdb_tool_trickle_mutex ); ldap_pvt_thread_mutex_unlock( &bdb_tool_trickle_mutex ); @@ -1283,7 +1283,7 @@ break; env->memp_trickle( env, 30, &wrote ); } - bdb_tool_trickle_active = 0; + bdb_tool_trickle_active = -1; ldap_pvt_thread_cond_signal( &bdb_tool_trickle_cond_end ); ldap_pvt_thread_mutex_unlock( &bdb_tool_trickle_mutex );
--------------------------------------------------------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.