truss is pretty much useless in this context. Most of BDB's activity is thru memory-mapping, which involves no system calls for truss to trace. You need an actual profile (e.g. using oprofile) to identify where the time is going.
This is very true. But reach for the tools you have, even if it is a hammer. I guess Purify would be the Solaris equivalent, unless we find the problem also occurs on a Linux box.
truss has a simple profiler, but only for System Calls which do not help in this case:
Library: Function calls libaio: close 16 libc: membar_exit 1633814 libc: thr_self 653455 libc: _lock_clear 326545 libc: _lock_try 326545 libc: memcpy 163311 libc: memcmp 242 libc: strcasecmp 195 libc: free 77
sys totals: .921 303 22 usr time: 21.669 elapsed: 829.890
Now repeat the db_stat call and see how long it takes the 2nd time.
It does indeed speed up, if I do not wait too long between tests.
real 1m27.712s real 0m29.696s real 0m4.332s real 0m4.452s
4 seconds is much nicer. So what you are saying is that BDB uses mmap, and operations inside this memory will trigger reads inside the kernel which do not show as libc syscalls. Rats. So it may be IO? I need to throw even more memory at it, and live with the increasing startup times?
How does the "set_cachesize" relate to the mmap usage?