Am Freitag 21 November 2008 14:21:06 schrieb Ralf Haferkamp: > Hi, > > While doing a few slapadd testruns comparing the RE23 with the RE24 version > I ran into a strange issue. I ran test with different LDIFs (100k, 500k and > 1000k Entries) and especially with the 500k and 1000k LDIF, slapadd from > 2.3.43 was significantly faster than the 2.4.12 version. > > 2.3.43 loaded the 500k database in 13m54s while it took 33m49s with 2.4.12. > The 1000k testcase to 29m41s on 2.3 (still faster than the 500k on 2.4). I > didn't finish the 2.4 run with the 1000k database I stopped it after about > an hour. > > I used exactly the same configuration on exactly the same hardware/os for > the tests (a HP Proliant DL 580 G3 with 4 3.33GHz Xeons, 8GB of RAM, > SLES10-SP2, ext3 filesystem). BerkelyDB Version was 4.5.20 with the > following DB_CONFIG: > > set_cachesize 0 4294967295 1 > set_lg_regionmax 262144 > set_lg_bsize 2097152 > > cachesize in slapd.conf was large enough to hold the entire database and > tool- threads was set to 4. > > I did some profiling (with valgrinds callgrind tool) to find out where all > the time is spend and it revealed that 2.4 spend a significantly larger > amount of systime in the pwrite() function than 2.3. Most of that seemed to > come from the bdb_tool_trickle_task() that calls libdb's memp_trickle() > function. Just to verify this I ran a testbuild with the trickle_task > disabled(). And slapadd's performance was back to a normal level, > comparable to the 2.3.43 release. > > AFAIK the trickle_task() was introduced into 2.4 to increase slapadd > throughput but has exactly the opposite effect on my test system. Did > anybody else make similar experiences? Or do you see anything thats > obviously wrong with my testcases? Btw, I just re-ran the test with an unmodified 2.4.12 compiled against db-4.7.25. With that combination slapadd is still a bit slower than the OpenLDAP 2.3/db-4.5.20 combination, but the difference not quite as huge: 16m8s compared to 13m54s for the 500k Entries.

Something else came to mind. The trickle_task obviously cannot increase the overall volume of I/O, so there's no good reason for it to make things more than twice as slow.

Since you said that your BDB cache size is large enough to contain the entire DB in each case, try this: preload the LDIF into the FS cache before running slapadd. (dd if=<ldif> of=/dev/null bs=1024k)

Ralf Haferkamp wrote: > Am Freitag 21 November 2008 14:21:06 schrieb Ralf Haferkamp: >> I did some profiling (with valgrinds callgrind tool) to find out where >> all the time is spend and it revealed that 2.4 spend a significantly >> larger amount of systime in the pwrite() function than 2.3. Most of that >> seemed to come from the bdb_tool_trickle_task() that calls libdb's >> memp_trickle() function. Just to verify this I ran a testbuild with the >> trickle_task disabled(). And slapadd's performance was back to a normal >> level, comparable to the 2.3.43 release. Something else came to mind. The trickle_task obviously cannot increase the overall volume of I/O, so there's no good reason for it to make things more than twice as slow. Except, that if you're getting reads mixed with writes you will lose a lot in disk seek time. Since you said that your BDB cache size is large enough to contain the entire DB in each case, try this: preload the LDIF into the FS cache before running slapadd. (dd if=<ldif> of=/dev/null bs=1024k) That will eliminate any seek overhead during the run.

Ralf Haferkamp writes: > When linking against db-4.7.25 slapadd is still a little faster when the > trickle-task is disabled: 15m19s (without the trickle-task) vs. 16m12s > for the 500k Entries LDIF. Due to this mutex lock in bdb_tool_entry_put(), perhaps? if (( slapMode& SLAP_TOOL_QUICK )&& (( e->e_id& 0xfff ) == 0xfff )) { ldap_pvt_thread_mutex_lock(&bdb_tool_trickle_mutex ); ldap_pvt_thread_cond_signal(&bdb_tool_trickle_cond ); ldap_pvt_thread_mutex_unlock(&bdb_tool_trickle_mutex ); } Weird code. That's not the normal way to use a cond. Why wait for a mutex before signalling, when the code doesn't change any variable which is tested around the cond_wait?

Howard Chu writes: > That was to prevent the signal from being lost. Though I suppose it > doesn't matter, since the signal would get sent again soon enough. If it's lost, that'll usually be because memp_trickle() is busily flusing pages. It won't return until it has nothing to do, in which case immediately calling it again won't be very useful.

A few other thoughts: Could memp_trickle() have a bug so it loses "page is frequently used" info or whatever Berkeley DB is using? Then it could cause such pages to need to be re-read when they'd otherwise not need to be.

The memp_trickle() call the dirty page percentage to 70%. Might be interesting to see how a run without the trickle task compares to one with trickle + 43% extra DB cache. (1/0.7 = 1.43).

In the first message, Ralf Haferkamp wrote: > I did some profiling (with valgrinds callgrind tool) to find out where > all the time is spend and it revealed that 2.4 spend a significantly > larger amount of systime in the pwrite() function than 2.3. Most of > that seemed to come from the bdb_tool_trickle_task() that calls > libdb's memp_trickle() function. The trickle task takes over write work that is otherwise done from other functions. If it spends as much or more_ time in pwrite() from trickle_task() than it does in pwrite() when there is no trickle task, that'd be interesting...

Am Dienstag 25 November 2008 15:07:21 schrieb Hallvard B Furuseth: > Did I send this? Trying again - I wrote: >> ...unless the doc is inaccurate. > which in any case would be no reason to make bdb_tool_entry_put() > wait for env->memp_trickle(). If you keep the mutex lock, > set a flag while locked and move memp_trickle() out of the lock. I gave your attached patch a short try and it is a bit slower (between 10 and 20 seconds in my tests), than the one with the lock/unlock calls removed. The question is, if it is really needed to catch every signal. AFAIK the signal is sent every 4096th entry anyway, so if one is lost there will soon be another one that triggers memp_trickle().

Howard, what's your opinion on this?

Ralf Haferkamp writes: > When linking against db-4.7.25 slapadd is still a little faster when the > trickle-task is disabled: 15m19s (without the trickle-task) vs. 16m12s > for the 500k Entries LDIF. Due to this mutex lock in bdb_tool_entry_put(), perhaps? if (( slapMode & SLAP_TOOL_QUICK ) && (( e->e_id & 0xfff ) == 0xfff )) { ldap_pvt_thread_mutex_lock( &bdb_tool_trickle_mutex ); ldap_pvt_thread_cond_signal( &bdb_tool_trickle_cond ); ldap_pvt_thread_mutex_unlock( &bdb_tool_trickle_mutex ); } Weird code. That's not the normal way to use a cond. Why wait for a mutex before signalling, when the code doesn't change any variable which is tested around the cond_wait?