Hi,
I have two node MMR ldap cluster with hdb, but for some reason one of the servers CPU get spiked every minute, I am not sure what is causing this spike every minute, we have same server configuration in both the server. Also when i do db_stat on *.bdb files this server is taking too long to respond for db_stat, i am pasting top and strace out put of that thread here. Also attach a graph shows cpu spikes and active threads spikes in both the servers (green and yellow lines),
*Openldap Version: openldap-2.4.43*
[root@localhost ldap]# db_stat -d id2entry.bdb Thu Sep 21 22:16:15 2017 Local time
date ^CKilled [root@localhost ldap]# date Thu Sep 21 22:27:54 UTC 2017
* du -sh id2entry.bdb* *7.6G id2entry.bdb*
DB_CONFIG
sudo cat /var/lib/ldap/DB_CONFIG |grep -v "^#"
set_cachesize 8 0 1
set_lg_regionmax 262144 set_lg_bsize 2097152
set_lk_max_locks 3000 set_lk_max_objects 1500 set_lk_max_lockers 1500
*Top output of busy thread at the time of cpu spike:*
ldap_top..20170922.165808.886367620: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165811.390769103: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165813.895158022: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165816.399627768: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165818.903936890: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165821.408275241: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165823.912707039: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.05 slapd ldap_top..20170922.165826.417287284: 61437 ldap 20 0 14.8g 13g 8.2g R 101.8 47.5 18:02.51 slapd ldap_top..20170922.165828.922186726: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:05.01 slapd ldap_top..20170922.165831.427101922: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:07.51 slapd ldap_top..20170922.165833.931996934: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:09.92 slapd ldap_top..20170922.165836.436814788: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:12.42 slapd ldap_top..20170922.165838.941597309: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:14.92 slapd ldap_top..20170922.165841.446448385: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:17.18 slapd ldap_top..20170922.165843.951329579: 61437 ldap 20 0 14.8g 13g 8.2g R 97.8 47.5 18:19.67 slapd ldap_top..20170922.165846.456172321: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:21.85 slapd ldap_top..20170922.165848.961017603: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:24.35 slapd ldap_top..20170922.165851.465897714: 61437 ldap 20 0 14.8g 13g 8.2g R 101.8 47.5 18:26.49 slapd ldap_top..20170922.165853.970797751: 61437 ldap 20 0 14.8g 13g 8.2g S 57.9 47.5 18:28.77 slapd ldap_top..20170922.165856.475641225: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.77 slapd ldap_top..20170922.165858.980268851: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.77 slapd ldap_top..20170922.165901.485817704: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.77 slapd ldap_top..20170922.165903.990499397: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165906.495260168: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165909.078879876: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165911.583657585: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165914.088384025: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165916.593114635: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165919.097562041: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165921.602221629: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165924.106739600: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165926.611115008: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165929.115600777: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165931.620082217: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165934.124916755: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd
*strace output on that thread around that time:*
grep "16:58:26" ldap_strace.61437 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x560271548120, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x560271548120, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0
*Here is the disk activity metrics and corresponding pstack ouput captured at different time when the system is not serving any traffic other than replication:*
04:20:32 PM 19169 0.00 10042.00 0.00 slapd 04:20:34 PM 19169 0.00 45850.00 0.00 slapd 04:20:36 PM 19169 0.00 35706.00 0.00 slapd 04:20:38 PM 19169 0.00 37096.00 0.00 slapd 04:20:40 PM 19169 0.00 43654.00 0.00 slapd
04:20:40 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 04:20:42 PM 19169 0.00 42948.00 0.00 slapd 04:20:44 PM 19169 0.00 43272.00 0.00 slapd 04:20:46 PM 19169 0.00 37084.00 0.00 slapd 04:20:48 PM 19169 0.00 22740.00 0.00 slapd 04:20:50 PM 19169 0.00 23540.00 0.00 slapd 04:20:52 PM 19169 0.00 23540.00 0.00 slapd 04:20:54 PM 19169 0.00 7458.00 0.00 slapd 04:20:56 PM 19169 0.00 0.00 0.00 slapd 04:20:58 PM 19169 0.00 0.00 0.00 slapd 04:21:00 PM 19169 0.00 0.00 0.00 slapd 04:21:02 PM 19169 0.00 0.00 0.00 slapd 04:21:04 PM 19169 0.00 0.00 0.00 slapd 04:21:06 PM 19169 0.00 0.00 0.00 slapd 04:21:08 PM 19169 0.00 0.00 0.00 slapd 04:21:10 PM 19169 0.00 30.00 0.00 slapd 04:21:12 PM 19169 0.00 0.00 0.00 slapd 04:21:14 PM 19169 0.00 0.00 0.00 slapd 04:21:16 PM 19169 0.00 0.00 0.00 slapd 04:21:18 PM 19169 4.00 346.00 0.00 slapd 04:21:20 PM 19169 24.00 1012.00 0.00 slapd
04:21:20 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 04:21:22 PM 19169 0.00 14.00 0.00 slapd 04:21:24 PM 19169 6.00 1098.00 0.00 slapd 04:21:26 PM 19169 24.00 302.00 0.00 slapd 04:21:28 PM 19169 82.00 598.00 0.00 slapd 04:21:30 PM 19169 82.00 582.00 0.00 slapd 04:21:32 PM 19169 10.00 114.00 0.00 slapd 04:21:34 PM 19169 14.00 1854.00 0.00 slapd 04:21:36 PM 19169 6.00 608.00 0.00 slapd 04:21:38 PM 19169 6.00 502.00 0.00 slapd 04:21:40 PM 19169 10.00 992.00 0.00 slapd 04:21:42 PM 19169 28.00 5620.00 0.00 slapd 04:21:44 PM 19169 0.00 0.00 0.00 slapd 04:21:46 PM 19169 0.00 0.00 0.00 slapd 04:21:48 PM 19169 0.00 0.00 0.00 slapd 04:21:50 PM 19169 0.00 0.00 0.00 slapd 04:21:52 PM 19169 0.00 0.00 0.00 slapd 04:21:54 PM 19169 0.00 1018.00 0.00 slapd 04:21:56 PM 19169 0.00 44146.00 0.00 slapd 04:21:58 PM 19169 0.00 39188.00 0.00 slapd 04:22:00 PM 19169 0.00 43206.00 0.00 slapd
04:22:00 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 04:22:02 PM 19169 0.00 39266.00 0.00 slapd 04:22:04 PM 19169 0.00 43240.00 0.00 slapd 04:22:06 PM 19169 0.00 44358.00 0.00 slapd 04:22:08 PM 19169 0.00 43790.00 0.00 slapd 04:22:10 PM 19169 0.00 38186.00 0.00 slapd 04:22:12 PM 19169 0.00 36304.00 0.00 slapd 04:22:14 PM 19169 0.00 42903.48 0.00 slapd 04:22:16 PM 19169 0.00 42246.00 0.00 slapd 04:22:18 PM 19169 0.00 9720.00 0.00 slapd
*Thu Sep 21 16:20:40 UTC 2017* Thread 14 (Thread 0x7fc4eb57c700 (LWP 19171)): #0 0x00007fc702d3d8e3 in epoll_wait () from /lib64/libc.so.6 #1 0x00005638f66beda9 in ?? () #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc4ead7b700 (LWP 19172)): #0 0x00007fc704acce6c in __db_tas_mutex_unlock () from /lib64/libdb-4.7.so #1 0x00007fc704b74738 in __dbc_close () from /lib64/libdb-4.7.so #2 0x00007fc704b7492d in ?? () from /lib64/libdb-4.7.so #3 0x00007fc704b750da in __dbc_get () from /lib64/libdb-4.7.so #4 0x00007fc704b7f9dd in __dbc_get_pp () from /lib64/libdb-4.7.so #5 0x00005638f67a458c in hdb_id2entry () #6 0x00005638f679b6f8 in hdb_cache_find_id () #7 0x00005638f675041d in hdb_search () #8 0x00005638f67304c2 in overlay_op_walk () #9 0x00005638f6730624 in ?? () #10 0x00005638f66c69e9 in fe_op_search () #11 0x00005638f66c62b6 in do_search () #12 0x00005638f66c3ad4 in ?? () #13 0x00005638f66c3e3b in ?? () #14 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #15 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc4ea57a700 (LWP 19173)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fc4e9d79700 (LWP 19174)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fc4e9578700 (LWP 19175)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fc4e8d77700 (LWP 19176)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc4d7fff700 (LWP 19179)): #0 0x00007fc7032229ad in connect () from /lib64/libpthread.so.0 #1 0x00007fc705458d0c in ldap_connect_to_host () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc705442ece in ldap_int_open_connection () from /lib64/libldap_r-2.4.so.2 #3 0x00007fc705455f1d in ldap_new_connection () from /lib64/libldap_r-2.4.so.2 #4 0x00007fc7054424bf in ldap_open_defconn () from /lib64/libldap_r-2.4.so.2 #5 0x00007fc705457458 in ldap_send_initial_request () from /lib64/libldap_r-2.4.so.2 #6 0x00007fc70544c63f in ldap_sasl_bind () from /lib64/libldap_r-2.4.so.2 #7 0x00007fc70544ca19 in ldap_sasl_bind_s () from /lib64/libldap_r-2.4.so.2 #8 0x00005638f66baf52 in slap_client_connect () #9 0x00005638f672997d in ?? () #10 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #11 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc4d77fe700 (LWP 19293)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc4d6ffd700 (LWP 23630)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc4d67fc700 (LWP 23631)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fc4d5ffb700 (LWP 23632)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc4d57fa700 (LWP 23633)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc4d4ff9700 (LWP 23634)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc705895780 (LWP 19169)): #0 0x00007fc70321cf17 in pthread_join () from /lib64/libpthread.so.0 #1 0x00005638f66c0dc9 in slapd_daemon () #2 0x00005638f66a7d68 in main () *Thu Sep 21 16:20:42 UTC 2017* Thread 14 (Thread 0x7fc4eb57c700 (LWP 19171)): #0 0x00007fc702d3d8e3 in epoll_wait () from /lib64/libc.so.6 #1 0x00005638f66beda9 in ?? () #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc4ead7b700 (LWP 19172)): #0 0x00005638f66cfa88 in entry_decode () #1 0x00005638f67a44a2 in hdb_id2entry () #2 0x00005638f679b6f8 in hdb_cache_find_id () #3 0x00005638f675041d in hdb_search () #4 0x00005638f67304c2 in overlay_op_walk () #5 0x00005638f6730624 in ?? () #6 0x00005638f66c69e9 in fe_op_search () #7 0x00005638f66c62b6 in do_search () #8 0x00005638f66c3ad4 in ?? () #9 0x00005638f66c3e3b in ?? () #10 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #11 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc4ea57a700 (LWP 19173)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fc4e9d79700 (LWP 19174)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fc4e9578700 (LWP 19175)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fc4e8d77700 (LWP 19176)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc4d7fff700 (LWP 19179)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc4d77fe700 (LWP 19293)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc4d6ffd700 (LWP 23630)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc4d67fc700 (LWP 23631)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fc4d5ffb700 (LWP 23632)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc4d57fa700 (LWP 23633)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc4d4ff9700 (LWP 23634)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc705895780 (LWP 19169)): #0 0x00007fc70321cf17 in pthread_join () from /lib64/libpthread.so.0 #1 0x00005638f66c0dc9 in slapd_daemon () #2 0x00005638f66a7d68 in main () Thu Sep 21 16:20:44 UTC 2017 Thread 14 (Thread 0x7fc4eb57c700 (LWP 19171)): #0 0x00007fc702d3d8e3 in epoll_wait () from /lib64/libc.so.6 #1 0x00005638f66beda9 in ?? () #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc4ead7b700 (LWP 19172)): #0 0x00005638f6706b30 in ad_keystring () #1 0x00005638f6706c6e in slap_bv2ad () #2 0x00005638f66cf9d6 in entry_decode () #3 0x00005638f67a44a2 in hdb_id2entry () #4 0x00005638f679b6f8 in hdb_cache_find_id () #5 0x00005638f675041d in hdb_search () #6 0x00005638f67304c2 in overlay_op_walk () #7 0x00005638f6730624 in ?? () #8 0x00005638f66c69e9 in fe_op_search () #9 0x00005638f66c62b6 in do_search () #10 0x00005638f66c3ad4 in ?? () #11 0x00005638f66c3e3b in ?? () #12 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #13 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc4ea57a700 (LWP 19173)):
After i did db_recover db_stat is responding fast but cpu spikes are still there, is there any way to find if my db is bad?
Thanks, Ram
I suggest you to use the latest slapd version and mdb instead of bdb/hdb which is deprecated.
Em 22-09-2017 16:00, rammohan ganapavarapu escreveu: Hi,
I have two node MMR ldap cluster with hdb, but for some reason one of the servers CPU get spiked every minute, I am not sure what is causing this spike every minute, we have same server configuration in both the server. Also when i do db_stat on *.bdb files this server is taking too long to respond for db_stat, i am pasting top and strace out put of that thread here. Also attach a graph shows cpu spikes and active threads spikes in both the servers (green and yellow lines),
Openldap Version: openldap-2.4.43
[root@localhost ldap]# db_stat -d id2entry.bdb Thu Sep 21 22:16:15 2017 Local time
date ^CKilled [root@localhost ldap]# date Thu Sep 21 22:27:54 UTC 2017
du -sh id2entry.bdb 7.6G id2entry.bdb
DB_CONFIG
sudo cat /var/lib/ldap/DB_CONFIG |grep -v "^#"
set_cachesize 8 0 1
set_lg_regionmax 262144 set_lg_bsize 2097152
set_lk_max_locks 3000 set_lk_max_objects 1500 set_lk_max_lockers 1500
Top output of busy thread at the time of cpu spike:
ldap_top..20170922.165808.886367620: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165811.390769103: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165813.895158022: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165816.399627768: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165818.903936890: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165821.408275241: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165823.912707039: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.05 slapd ldap_top..20170922.165826.417287284: 61437 ldap 20 0 14.8g 13g 8.2g R 101.8 47.5 18:02.51 slapd ldap_top..20170922.165828.922186726: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:05.01 slapd ldap_top..20170922.165831.427101922: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:07.51 slapd ldap_top..20170922.165833.931996934: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:09.92 slapd ldap_top..20170922.165836.436814788: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:12.42 slapd ldap_top..20170922.165838.941597309: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:14.92 slapd ldap_top..20170922.165841.446448385: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:17.18 slapd ldap_top..20170922.165843.951329579: 61437 ldap 20 0 14.8g 13g 8.2g R 97.8 47.5 18:19.67 slapd ldap_top..20170922.165846.456172321: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:21.85 slapd ldap_top..20170922.165848.961017603: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:24.35 slapd ldap_top..20170922.165851.465897714: 61437 ldap 20 0 14.8g 13g 8.2g R 101.8 47.5 18:26.49 slapd ldap_top..20170922.165853.970797751: 61437 ldap 20 0 14.8g 13g 8.2g S 57.9 47.5 18:28.77 slapd ldap_top..20170922.165856.475641225: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.77 slapd ldap_top..20170922.165858.980268851: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.77 slapd ldap_top..20170922.165901.485817704: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.77 slapd ldap_top..20170922.165903.990499397: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165906.495260168: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165909.078879876: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165911.583657585: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165914.088384025: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165916.593114635: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165919.097562041: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165921.602221629: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165924.106739600: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165926.611115008: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165929.115600777: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165931.620082217: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165934.124916755: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd
strace output on that thread around that time:
grep "16:58:26" ldap_strace.61437 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x560271548120, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x560271548120, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0
Here is the disk activity metrics and corresponding pstack ouput captured at different time when the system is not serving any traffic other than replication:
04:20:32 PM 19169 0.00 10042.00 0.00 slapd 04:20:34 PM 19169 0.00 45850.00 0.00 slapd 04:20:36 PM 19169 0.00 35706.00 0.00 slapd 04:20:38 PM 19169 0.00 37096.00 0.00 slapd 04:20:40 PM 19169 0.00 43654.00 0.00 slapd
04:20:40 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 04:20:42 PM 19169 0.00 42948.00 0.00 slapd 04:20:44 PM 19169 0.00 43272.00 0.00 slapd 04:20:46 PM 19169 0.00 37084.00 0.00 slapd 04:20:48 PM 19169 0.00 22740.00 0.00 slapd 04:20:50 PM 19169 0.00 23540.00 0.00 slapd 04:20:52 PM 19169 0.00 23540.00 0.00 slapd 04:20:54 PM 19169 0.00 7458.00 0.00 slapd 04:20:56 PM 19169 0.00 0.00 0.00 slapd 04:20:58 PM 19169 0.00 0.00 0.00 slapd 04:21:00 PM 19169 0.00 0.00 0.00 slapd 04:21:02 PM 19169 0.00 0.00 0.00 slapd 04:21:04 PM 19169 0.00 0.00 0.00 slapd 04:21:06 PM 19169 0.00 0.00 0.00 slapd 04:21:08 PM 19169 0.00 0.00 0.00 slapd 04:21:10 PM 19169 0.00 30.00 0.00 slapd 04:21:12 PM 19169 0.00 0.00 0.00 slapd 04:21:14 PM 19169 0.00 0.00 0.00 slapd 04:21:16 PM 19169 0.00 0.00 0.00 slapd 04:21:18 PM 19169 4.00 346.00 0.00 slapd 04:21:20 PM 19169 24.00 1012.00 0.00 slapd
04:21:20 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 04:21:22 PM 19169 0.00 14.00 0.00 slapd 04:21:24 PM 19169 6.00 1098.00 0.00 slapd 04:21:26 PM 19169 24.00 302.00 0.00 slapd 04:21:28 PM 19169 82.00 598.00 0.00 slapd 04:21:30 PM 19169 82.00 582.00 0.00 slapd 04:21:32 PM 19169 10.00 114.00 0.00 slapd 04:21:34 PM 19169 14.00 1854.00 0.00 slapd 04:21:36 PM 19169 6.00 608.00 0.00 slapd 04:21:38 PM 19169 6.00 502.00 0.00 slapd 04:21:40 PM 19169 10.00 992.00 0.00 slapd 04:21:42 PM 19169 28.00 5620.00 0.00 slapd 04:21:44 PM 19169 0.00 0.00 0.00 slapd 04:21:46 PM 19169 0.00 0.00 0.00 slapd 04:21:48 PM 19169 0.00 0.00 0.00 slapd 04:21:50 PM 19169 0.00 0.00 0.00 slapd 04:21:52 PM 19169 0.00 0.00 0.00 slapd 04:21:54 PM 19169 0.00 1018.00 0.00 slapd 04:21:56 PM 19169 0.00 44146.00 0.00 slapd 04:21:58 PM 19169 0.00 39188.00 0.00 slapd 04:22:00 PM 19169 0.00 43206.00 0.00 slapd
04:22:00 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 04:22:02 PM 19169 0.00 39266.00 0.00 slapd 04:22:04 PM 19169 0.00 43240.00 0.00 slapd 04:22:06 PM 19169 0.00 44358.00 0.00 slapd 04:22:08 PM 19169 0.00 43790.00 0.00 slapd 04:22:10 PM 19169 0.00 38186.00 0.00 slapd 04:22:12 PM 19169 0.00 36304.00 0.00 slapd 04:22:14 PM 19169 0.00 42903.48 0.00 slapd 04:22:16 PM 19169 0.00 42246.00 0.00 slapd 04:22:18 PM 19169 0.00 9720.00 0.00 slapd
Thu Sep 21 16:20:40 UTC 2017 Thread 14 (Thread 0x7fc4eb57c700 (LWP 19171)): #0 0x00007fc702d3d8e3 in epoll_wait () from /lib64/libc.so.6 #1 0x00005638f66beda9 in ?? () #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc4ead7b700 (LWP 19172)): #0 0x00007fc704acce6c in __db_tas_mutex_unlock () from /lib64/libdb-4.7.sohttp://libdb-4.7.so #1 0x00007fc704b74738 in __dbc_close () from /lib64/libdb-4.7.sohttp://libdb-4.7.so #2 0x00007fc704b7492d in ?? () from /lib64/libdb-4.7.sohttp://libdb-4.7.so #3 0x00007fc704b750da in __dbc_get () from /lib64/libdb-4.7.sohttp://libdb-4.7.so #4 0x00007fc704b7f9dd in __dbc_get_pp () from /lib64/libdb-4.7.sohttp://libdb-4.7.so #5 0x00005638f67a458c in hdb_id2entry () #6 0x00005638f679b6f8 in hdb_cache_find_id () #7 0x00005638f675041d in hdb_search () #8 0x00005638f67304c2 in overlay_op_walk () #9 0x00005638f6730624 in ?? () #10 0x00005638f66c69e9 in fe_op_search () #11 0x00005638f66c62b6 in do_search () #12 0x00005638f66c3ad4 in ?? () #13 0x00005638f66c3e3b in ?? () #14 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #15 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc4ea57a700 (LWP 19173)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fc4e9d79700 (LWP 19174)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fc4e9578700 (LWP 19175)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fc4e8d77700 (LWP 19176)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc4d7fff700 (LWP 19179)): #0 0x00007fc7032229ad in connect () from /lib64/libpthread.so.0 #1 0x00007fc705458d0c in ldap_connect_to_host () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc705442ece in ldap_int_open_connection () from /lib64/libldap_r-2.4.so.2 #3 0x00007fc705455f1d in ldap_new_connection () from /lib64/libldap_r-2.4.so.2 #4 0x00007fc7054424bf in ldap_open_defconn () from /lib64/libldap_r-2.4.so.2 #5 0x00007fc705457458 in ldap_send_initial_request () from /lib64/libldap_r-2.4.so.2 #6 0x00007fc70544c63f in ldap_sasl_bind () from /lib64/libldap_r-2.4.so.2 #7 0x00007fc70544ca19 in ldap_sasl_bind_s () from /lib64/libldap_r-2.4.so.2 #8 0x00005638f66baf52 in slap_client_connect () #9 0x00005638f672997d in ?? () #10 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #11 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc4d77fe700 (LWP 19293)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc4d6ffd700 (LWP 23630)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc4d67fc700 (LWP 23631)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fc4d5ffb700 (LWP 23632)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc4d57fa700 (LWP 23633)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc4d4ff9700 (LWP 23634)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc705895780 (LWP 19169)): #0 0x00007fc70321cf17 in pthread_join () from /lib64/libpthread.so.0 #1 0x00005638f66c0dc9 in slapd_daemon () #2 0x00005638f66a7d68 in main () Thu Sep 21 16:20:42 UTC 2017 Thread 14 (Thread 0x7fc4eb57c700 (LWP 19171)): #0 0x00007fc702d3d8e3 in epoll_wait () from /lib64/libc.so.6 #1 0x00005638f66beda9 in ?? () #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc4ead7b700 (LWP 19172)): #0 0x00005638f66cfa88 in entry_decode () #1 0x00005638f67a44a2 in hdb_id2entry () #2 0x00005638f679b6f8 in hdb_cache_find_id () #3 0x00005638f675041d in hdb_search () #4 0x00005638f67304c2 in overlay_op_walk () #5 0x00005638f6730624 in ?? () #6 0x00005638f66c69e9 in fe_op_search () #7 0x00005638f66c62b6 in do_search () #8 0x00005638f66c3ad4 in ?? () #9 0x00005638f66c3e3b in ?? () #10 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #11 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc4ea57a700 (LWP 19173)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fc4e9d79700 (LWP 19174)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fc4e9578700 (LWP 19175)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fc4e8d77700 (LWP 19176)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc4d7fff700 (LWP 19179)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc4d77fe700 (LWP 19293)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc4d6ffd700 (LWP 23630)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc4d67fc700 (LWP 23631)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fc4d5ffb700 (LWP 23632)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc4d57fa700 (LWP 23633)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc4d4ff9700 (LWP 23634)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2mailto:pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc705895780 (LWP 19169)): #0 0x00007fc70321cf17 in pthread_join () from /lib64/libpthread.so.0 #1 0x00005638f66c0dc9 in slapd_daemon () #2 0x00005638f66a7d68 in main () Thu Sep 21 16:20:44 UTC 2017 Thread 14 (Thread 0x7fc4eb57c700 (LWP 19171)): #0 0x00007fc702d3d8e3 in epoll_wait () from /lib64/libc.so.6 #1 0x00005638f66beda9 in ?? () #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc4ead7b700 (LWP 19172)): #0 0x00005638f6706b30 in ad_keystring () #1 0x00005638f6706c6e in slap_bv2ad () #2 0x00005638f66cf9d6 in entry_decode () #3 0x00005638f67a44a2 in hdb_id2entry () #4 0x00005638f679b6f8 in hdb_cache_find_id () #5 0x00005638f675041d in hdb_search () #6 0x00005638f67304c2 in overlay_op_walk () #7 0x00005638f6730624 in ?? () #8 0x00005638f66c69e9 in fe_op_search () #9 0x00005638f66c62b6 in do_search () #10 0x00005638f66c3ad4 in ?? () #11 0x00005638f66c3e3b in ?? () #12 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #13 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc4ea57a700 (LWP 19173)):
After i did db_recover db_stat is responding fast but cpu spikes are still there, is there any way to find if my db is bad?
Thanks, Ram
-- Matheus Eduardo Bonifácio Morais Analista de Infraestrutura de TI, Plataforma e Aplicações Confederação Sicredi
Centro Administrativo Sicredi – Porto Alegre +55 51 3358 7143
sicredi.com.br
Esta mensagem é somente para uso do destinatário informado e pode conter informações privilegiadas, proprietárias ou privadas. Se você recebeu esta mensagem por engano, por favor, notifique o remetente imediatamente e apague a original. Qualquer outro uso deste e-mail é proibido.
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.
Any pointers to identify the issue?
Thanks, Ram
On Fri, Sep 22, 2017 at 12:00 PM, rammohan ganapavarapu < rammohanganap@gmail.com> wrote:
Hi,
I have two node MMR ldap cluster with hdb, but for some reason one of the servers CPU get spiked every minute, I am not sure what is causing this spike every minute, we have same server configuration in both the server. Also when i do db_stat on *.bdb files this server is taking too long to respond for db_stat, i am pasting top and strace out put of that thread here. Also attach a graph shows cpu spikes and active threads spikes in both the servers (green and yellow lines),
*Openldap Version: openldap-2.4.43*
[root@localhost ldap]# db_stat -d id2entry.bdb Thu Sep 21 22:16:15 2017 Local time
date ^CKilled [root@localhost ldap]# date Thu Sep 21 22:27:54 UTC 2017
- du -sh id2entry.bdb*
*7.6G id2entry.bdb*
DB_CONFIG
sudo cat /var/lib/ldap/DB_CONFIG |grep -v "^#"
set_cachesize 8 0 1
set_lg_regionmax 262144 set_lg_bsize 2097152
set_lk_max_locks 3000 set_lk_max_objects 1500 set_lk_max_lockers 1500
*Top output of busy thread at the time of cpu spike:*
ldap_top..20170922.165808.886367620: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165811.390769103: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165813.895158022: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165816.399627768: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165818.903936890: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165821.408275241: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165823.912707039: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.05 slapd ldap_top..20170922.165826.417287284: 61437 ldap 20 0 14.8g 13g 8.2g R 101.8 47.5 18:02.51 slapd ldap_top..20170922.165828.922186726: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:05.01 slapd ldap_top..20170922.165831.427101922: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:07.51 slapd ldap_top..20170922.165833.931996934: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:09.92 slapd ldap_top..20170922.165836.436814788: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:12.42 slapd ldap_top..20170922.165838.941597309: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:14.92 slapd ldap_top..20170922.165841.446448385: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:17.18 slapd ldap_top..20170922.165843.951329579: 61437 ldap 20 0 14.8g 13g 8.2g R 97.8 47.5 18:19.67 slapd ldap_top..20170922.165846.456172321: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:21.85 slapd ldap_top..20170922.165848.961017603: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:24.35 slapd ldap_top..20170922.165851.465897714: 61437 ldap 20 0 14.8g 13g 8.2g R 101.8 47.5 18:26.49 slapd ldap_top..20170922.165853.970797751: 61437 ldap 20 0 14.8g 13g 8.2g S 57.9 47.5 18:28.77 slapd ldap_top..20170922.165856.475641225: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.77 slapd ldap_top..20170922.165858.980268851: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.77 slapd ldap_top..20170922.165901.485817704: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.77 slapd ldap_top..20170922.165903.990499397: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165906.495260168: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165909.078879876: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165911.583657585: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165914.088384025: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165916.593114635: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165919.097562041: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165921.602221629: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165924.106739600: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165926.611115008: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165929.115600777: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165931.620082217: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165934.124916755: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd
*strace output on that thread around that time:*
grep "16:58:26" ldap_strace.61437 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x560271548120, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x560271548120, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0
*Here is the disk activity metrics and corresponding pstack ouput captured at different time when the system is not serving any traffic other than replication:*
04:20:32 PM 19169 0.00 10042.00 0.00 slapd 04:20:34 PM 19169 0.00 45850.00 0.00 slapd 04:20:36 PM 19169 0.00 35706.00 0.00 slapd 04:20:38 PM 19169 0.00 37096.00 0.00 slapd 04:20:40 PM 19169 0.00 43654.00 0.00 slapd
04:20:40 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 04:20:42 PM 19169 0.00 42948.00 0.00 slapd 04:20:44 PM 19169 0.00 43272.00 0.00 slapd 04:20:46 PM 19169 0.00 37084.00 0.00 slapd 04:20:48 PM 19169 0.00 22740.00 0.00 slapd 04:20:50 PM 19169 0.00 23540.00 0.00 slapd 04:20:52 PM 19169 0.00 23540.00 0.00 slapd 04:20:54 PM 19169 0.00 7458.00 0.00 slapd 04:20:56 PM 19169 0.00 0.00 0.00 slapd 04:20:58 PM 19169 0.00 0.00 0.00 slapd 04:21:00 PM 19169 0.00 0.00 0.00 slapd 04:21:02 PM 19169 0.00 0.00 0.00 slapd 04:21:04 PM 19169 0.00 0.00 0.00 slapd 04:21:06 PM 19169 0.00 0.00 0.00 slapd 04:21:08 PM 19169 0.00 0.00 0.00 slapd 04:21:10 PM 19169 0.00 30.00 0.00 slapd 04:21:12 PM 19169 0.00 0.00 0.00 slapd 04:21:14 PM 19169 0.00 0.00 0.00 slapd 04:21:16 PM 19169 0.00 0.00 0.00 slapd 04:21:18 PM 19169 4.00 346.00 0.00 slapd 04:21:20 PM 19169 24.00 1012.00 0.00 slapd
04:21:20 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 04:21:22 PM 19169 0.00 14.00 0.00 slapd 04:21:24 PM 19169 6.00 1098.00 0.00 slapd 04:21:26 PM 19169 24.00 302.00 0.00 slapd 04:21:28 PM 19169 82.00 598.00 0.00 slapd 04:21:30 PM 19169 82.00 582.00 0.00 slapd 04:21:32 PM 19169 10.00 114.00 0.00 slapd 04:21:34 PM 19169 14.00 1854.00 0.00 slapd 04:21:36 PM 19169 6.00 608.00 0.00 slapd 04:21:38 PM 19169 6.00 502.00 0.00 slapd 04:21:40 PM 19169 10.00 992.00 0.00 slapd 04:21:42 PM 19169 28.00 5620.00 0.00 slapd 04:21:44 PM 19169 0.00 0.00 0.00 slapd 04:21:46 PM 19169 0.00 0.00 0.00 slapd 04:21:48 PM 19169 0.00 0.00 0.00 slapd 04:21:50 PM 19169 0.00 0.00 0.00 slapd 04:21:52 PM 19169 0.00 0.00 0.00 slapd 04:21:54 PM 19169 0.00 1018.00 0.00 slapd 04:21:56 PM 19169 0.00 44146.00 0.00 slapd 04:21:58 PM 19169 0.00 39188.00 0.00 slapd 04:22:00 PM 19169 0.00 43206.00 0.00 slapd
04:22:00 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 04:22:02 PM 19169 0.00 39266.00 0.00 slapd 04:22:04 PM 19169 0.00 43240.00 0.00 slapd 04:22:06 PM 19169 0.00 44358.00 0.00 slapd 04:22:08 PM 19169 0.00 43790.00 0.00 slapd 04:22:10 PM 19169 0.00 38186.00 0.00 slapd 04:22:12 PM 19169 0.00 36304.00 0.00 slapd 04:22:14 PM 19169 0.00 42903.48 0.00 slapd 04:22:16 PM 19169 0.00 42246.00 0.00 slapd 04:22:18 PM 19169 0.00 9720.00 0.00 slapd
*Thu Sep 21 16:20:40 UTC 2017* Thread 14 (Thread 0x7fc4eb57c700 (LWP 19171)): #0 0x00007fc702d3d8e3 in epoll_wait () from /lib64/libc.so.6 #1 0x00005638f66beda9 in ?? () #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc4ead7b700 (LWP 19172)): #0 0x00007fc704acce6c in __db_tas_mutex_unlock () from /lib64/ libdb-4.7.so #1 0x00007fc704b74738 in __dbc_close () from /lib64/libdb-4.7.so #2 0x00007fc704b7492d in ?? () from /lib64/libdb-4.7.so #3 0x00007fc704b750da in __dbc_get () from /lib64/libdb-4.7.so #4 0x00007fc704b7f9dd in __dbc_get_pp () from /lib64/libdb-4.7.so #5 0x00005638f67a458c in hdb_id2entry () #6 0x00005638f679b6f8 in hdb_cache_find_id () #7 0x00005638f675041d in hdb_search () #8 0x00005638f67304c2 in overlay_op_walk () #9 0x00005638f6730624 in ?? () #10 0x00005638f66c69e9 in fe_op_search () #11 0x00005638f66c62b6 in do_search () #12 0x00005638f66c3ad4 in ?? () #13 0x00005638f66c3e3b in ?? () #14 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #15 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc4ea57a700 (LWP 19173)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fc4e9d79700 (LWP 19174)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fc4e9578700 (LWP 19175)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fc4e8d77700 (LWP 19176)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc4d7fff700 (LWP 19179)): #0 0x00007fc7032229ad in connect () from /lib64/libpthread.so.0 #1 0x00007fc705458d0c in ldap_connect_to_host () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc705442ece in ldap_int_open_connection () from /lib64/libldap_r-2.4.so.2 #3 0x00007fc705455f1d in ldap_new_connection () from /lib64/libldap_r-2.4.so.2 #4 0x00007fc7054424bf in ldap_open_defconn () from /lib64/libldap_r-2.4.so.2 #5 0x00007fc705457458 in ldap_send_initial_request () from /lib64/libldap_r-2.4.so.2 #6 0x00007fc70544c63f in ldap_sasl_bind () from /lib64/libldap_r-2.4.so.2 #7 0x00007fc70544ca19 in ldap_sasl_bind_s () from /lib64/libldap_r-2.4.so.2 #8 0x00005638f66baf52 in slap_client_connect () #9 0x00005638f672997d in ?? () #10 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #11 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc4d77fe700 (LWP 19293)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc4d6ffd700 (LWP 23630)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc4d67fc700 (LWP 23631)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fc4d5ffb700 (LWP 23632)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc4d57fa700 (LWP 23633)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc4d4ff9700 (LWP 23634)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc705895780 (LWP 19169)): #0 0x00007fc70321cf17 in pthread_join () from /lib64/libpthread.so.0 #1 0x00005638f66c0dc9 in slapd_daemon () #2 0x00005638f66a7d68 in main () *Thu Sep 21 16:20:42 UTC 2017* Thread 14 (Thread 0x7fc4eb57c700 (LWP 19171)): #0 0x00007fc702d3d8e3 in epoll_wait () from /lib64/libc.so.6 #1 0x00005638f66beda9 in ?? () #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc4ead7b700 (LWP 19172)): #0 0x00005638f66cfa88 in entry_decode () #1 0x00005638f67a44a2 in hdb_id2entry () #2 0x00005638f679b6f8 in hdb_cache_find_id () #3 0x00005638f675041d in hdb_search () #4 0x00005638f67304c2 in overlay_op_walk () #5 0x00005638f6730624 in ?? () #6 0x00005638f66c69e9 in fe_op_search () #7 0x00005638f66c62b6 in do_search () #8 0x00005638f66c3ad4 in ?? () #9 0x00005638f66c3e3b in ?? () #10 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #11 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc4ea57a700 (LWP 19173)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fc4e9d79700 (LWP 19174)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fc4e9578700 (LWP 19175)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fc4e8d77700 (LWP 19176)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc4d7fff700 (LWP 19179)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc4d77fe700 (LWP 19293)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc4d6ffd700 (LWP 23630)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc4d67fc700 (LWP 23631)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fc4d5ffb700 (LWP 23632)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc4d57fa700 (LWP 23633)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc4d4ff9700 (LWP 23634)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc705895780 (LWP 19169)): #0 0x00007fc70321cf17 in pthread_join () from /lib64/libpthread.so.0 #1 0x00005638f66c0dc9 in slapd_daemon () #2 0x00005638f66a7d68 in main () Thu Sep 21 16:20:44 UTC 2017 Thread 14 (Thread 0x7fc4eb57c700 (LWP 19171)): #0 0x00007fc702d3d8e3 in epoll_wait () from /lib64/libc.so.6 #1 0x00005638f66beda9 in ?? () #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc4ead7b700 (LWP 19172)): #0 0x00005638f6706b30 in ad_keystring () #1 0x00005638f6706c6e in slap_bv2ad () #2 0x00005638f66cf9d6 in entry_decode () #3 0x00005638f67a44a2 in hdb_id2entry () #4 0x00005638f679b6f8 in hdb_cache_find_id () #5 0x00005638f675041d in hdb_search () #6 0x00005638f67304c2 in overlay_op_walk () #7 0x00005638f6730624 in ?? () #8 0x00005638f66c69e9 in fe_op_search () #9 0x00005638f66c62b6 in do_search () #10 0x00005638f66c3ad4 in ?? () #11 0x00005638f66c3e3b in ?? () #12 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #13 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc4ea57a700 (LWP 19173)):
After i did db_recover db_stat is responding fast but cpu spikes are still there, is there any way to find if my db is bad?
Thanks, Ram
Still looking for help
On Sun, Sep 24, 2017 at 12:29 PM, rammohan ganapavarapu < rammohanganap@gmail.com> wrote:
Any pointers to identify the issue?
Thanks, Ram
On Fri, Sep 22, 2017 at 12:00 PM, rammohan ganapavarapu < rammohanganap@gmail.com> wrote:
Hi,
I have two node MMR ldap cluster with hdb, but for some reason one of the servers CPU get spiked every minute, I am not sure what is causing this spike every minute, we have same server configuration in both the server. Also when i do db_stat on *.bdb files this server is taking too long to respond for db_stat, i am pasting top and strace out put of that thread here. Also attach a graph shows cpu spikes and active threads spikes in both the servers (green and yellow lines),
*Openldap Version: openldap-2.4.43*
[root@localhost ldap]# db_stat -d id2entry.bdb Thu Sep 21 22:16:15 2017 Local time
date ^CKilled [root@localhost ldap]# date Thu Sep 21 22:27:54 UTC 2017
- du -sh id2entry.bdb*
*7.6G id2entry.bdb*
DB_CONFIG
sudo cat /var/lib/ldap/DB_CONFIG |grep -v "^#"
set_cachesize 8 0 1
set_lg_regionmax 262144 set_lg_bsize 2097152
set_lk_max_locks 3000 set_lk_max_objects 1500 set_lk_max_lockers 1500
*Top output of busy thread at the time of cpu spike:*
ldap_top..20170922.165808.886367620: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165811.390769103: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165813.895158022: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165816.399627768: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165818.903936890: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165821.408275241: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.04 slapd ldap_top..20170922.165823.912707039: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:01.05 slapd ldap_top..20170922.165826.417287284: 61437 ldap 20 0 14.8g 13g 8.2g R 101.8 47.5 18:02.51 slapd ldap_top..20170922.165828.922186726: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:05.01 slapd ldap_top..20170922.165831.427101922: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:07.51 slapd ldap_top..20170922.165833.931996934: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:09.92 slapd ldap_top..20170922.165836.436814788: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:12.42 slapd ldap_top..20170922.165838.941597309: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:14.92 slapd ldap_top..20170922.165841.446448385: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:17.18 slapd ldap_top..20170922.165843.951329579: 61437 ldap 20 0 14.8g 13g 8.2g R 97.8 47.5 18:19.67 slapd ldap_top..20170922.165846.456172321: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:21.85 slapd ldap_top..20170922.165848.961017603: 61437 ldap 20 0 14.8g 13g 8.2g R 99.8 47.5 18:24.35 slapd ldap_top..20170922.165851.465897714: 61437 ldap 20 0 14.8g 13g 8.2g R 101.8 47.5 18:26.49 slapd ldap_top..20170922.165853.970797751: 61437 ldap 20 0 14.8g 13g 8.2g S 57.9 47.5 18:28.77 slapd ldap_top..20170922.165856.475641225: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.77 slapd ldap_top..20170922.165858.980268851: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.77 slapd ldap_top..20170922.165901.485817704: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.77 slapd ldap_top..20170922.165903.990499397: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165906.495260168: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165909.078879876: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165911.583657585: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165914.088384025: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165916.593114635: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165919.097562041: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165921.602221629: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165924.106739600: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165926.611115008: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165929.115600777: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165931.620082217: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd ldap_top..20170922.165934.124916755: 61437 ldap 20 0 14.8g 13g 8.2g S 0.0 47.5 18:28.78 slapd
*strace output on that thread around that time:*
grep "16:58:26" ldap_strace.61437 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea40, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x560271548120, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x560271548120, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0 16:58:26 futex(0x5602707aea00, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) 16:58:26 futex(0x5602707aea00, FUTEX_WAKE_PRIVATE, 1) = 0
*Here is the disk activity metrics and corresponding pstack ouput captured at different time when the system is not serving any traffic other than replication:*
04:20:32 PM 19169 0.00 10042.00 0.00 slapd 04:20:34 PM 19169 0.00 45850.00 0.00 slapd 04:20:36 PM 19169 0.00 35706.00 0.00 slapd 04:20:38 PM 19169 0.00 37096.00 0.00 slapd 04:20:40 PM 19169 0.00 43654.00 0.00 slapd
04:20:40 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 04:20:42 PM 19169 0.00 42948.00 0.00 slapd 04:20:44 PM 19169 0.00 43272.00 0.00 slapd 04:20:46 PM 19169 0.00 37084.00 0.00 slapd 04:20:48 PM 19169 0.00 22740.00 0.00 slapd 04:20:50 PM 19169 0.00 23540.00 0.00 slapd 04:20:52 PM 19169 0.00 23540.00 0.00 slapd 04:20:54 PM 19169 0.00 7458.00 0.00 slapd 04:20:56 PM 19169 0.00 0.00 0.00 slapd 04:20:58 PM 19169 0.00 0.00 0.00 slapd 04:21:00 PM 19169 0.00 0.00 0.00 slapd 04:21:02 PM 19169 0.00 0.00 0.00 slapd 04:21:04 PM 19169 0.00 0.00 0.00 slapd 04:21:06 PM 19169 0.00 0.00 0.00 slapd 04:21:08 PM 19169 0.00 0.00 0.00 slapd 04:21:10 PM 19169 0.00 30.00 0.00 slapd 04:21:12 PM 19169 0.00 0.00 0.00 slapd 04:21:14 PM 19169 0.00 0.00 0.00 slapd 04:21:16 PM 19169 0.00 0.00 0.00 slapd 04:21:18 PM 19169 4.00 346.00 0.00 slapd 04:21:20 PM 19169 24.00 1012.00 0.00 slapd
04:21:20 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 04:21:22 PM 19169 0.00 14.00 0.00 slapd 04:21:24 PM 19169 6.00 1098.00 0.00 slapd 04:21:26 PM 19169 24.00 302.00 0.00 slapd 04:21:28 PM 19169 82.00 598.00 0.00 slapd 04:21:30 PM 19169 82.00 582.00 0.00 slapd 04:21:32 PM 19169 10.00 114.00 0.00 slapd 04:21:34 PM 19169 14.00 1854.00 0.00 slapd 04:21:36 PM 19169 6.00 608.00 0.00 slapd 04:21:38 PM 19169 6.00 502.00 0.00 slapd 04:21:40 PM 19169 10.00 992.00 0.00 slapd 04:21:42 PM 19169 28.00 5620.00 0.00 slapd 04:21:44 PM 19169 0.00 0.00 0.00 slapd 04:21:46 PM 19169 0.00 0.00 0.00 slapd 04:21:48 PM 19169 0.00 0.00 0.00 slapd 04:21:50 PM 19169 0.00 0.00 0.00 slapd 04:21:52 PM 19169 0.00 0.00 0.00 slapd 04:21:54 PM 19169 0.00 1018.00 0.00 slapd 04:21:56 PM 19169 0.00 44146.00 0.00 slapd 04:21:58 PM 19169 0.00 39188.00 0.00 slapd 04:22:00 PM 19169 0.00 43206.00 0.00 slapd
04:22:00 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 04:22:02 PM 19169 0.00 39266.00 0.00 slapd 04:22:04 PM 19169 0.00 43240.00 0.00 slapd 04:22:06 PM 19169 0.00 44358.00 0.00 slapd 04:22:08 PM 19169 0.00 43790.00 0.00 slapd 04:22:10 PM 19169 0.00 38186.00 0.00 slapd 04:22:12 PM 19169 0.00 36304.00 0.00 slapd 04:22:14 PM 19169 0.00 42903.48 0.00 slapd 04:22:16 PM 19169 0.00 42246.00 0.00 slapd 04:22:18 PM 19169 0.00 9720.00 0.00 slapd
*Thu Sep 21 16:20:40 UTC 2017* Thread 14 (Thread 0x7fc4eb57c700 (LWP 19171)): #0 0x00007fc702d3d8e3 in epoll_wait () from /lib64/libc.so.6 #1 0x00005638f66beda9 in ?? () #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc4ead7b700 (LWP 19172)): #0 0x00007fc704acce6c in __db_tas_mutex_unlock () from /lib64/ libdb-4.7.so #1 0x00007fc704b74738 in __dbc_close () from /lib64/libdb-4.7.so #2 0x00007fc704b7492d in ?? () from /lib64/libdb-4.7.so #3 0x00007fc704b750da in __dbc_get () from /lib64/libdb-4.7.so #4 0x00007fc704b7f9dd in __dbc_get_pp () from /lib64/libdb-4.7.so #5 0x00005638f67a458c in hdb_id2entry () #6 0x00005638f679b6f8 in hdb_cache_find_id () #7 0x00005638f675041d in hdb_search () #8 0x00005638f67304c2 in overlay_op_walk () #9 0x00005638f6730624 in ?? () #10 0x00005638f66c69e9 in fe_op_search () #11 0x00005638f66c62b6 in do_search () #12 0x00005638f66c3ad4 in ?? () #13 0x00005638f66c3e3b in ?? () #14 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #15 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc4ea57a700 (LWP 19173)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fc4e9d79700 (LWP 19174)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fc4e9578700 (LWP 19175)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fc4e8d77700 (LWP 19176)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc4d7fff700 (LWP 19179)): #0 0x00007fc7032229ad in connect () from /lib64/libpthread.so.0 #1 0x00007fc705458d0c in ldap_connect_to_host () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc705442ece in ldap_int_open_connection () from /lib64/libldap_r-2.4.so.2 #3 0x00007fc705455f1d in ldap_new_connection () from /lib64/libldap_r-2.4.so.2 #4 0x00007fc7054424bf in ldap_open_defconn () from /lib64/libldap_r-2.4.so.2 #5 0x00007fc705457458 in ldap_send_initial_request () from /lib64/libldap_r-2.4.so.2 #6 0x00007fc70544c63f in ldap_sasl_bind () from /lib64/libldap_r-2.4.so.2 #7 0x00007fc70544ca19 in ldap_sasl_bind_s () from /lib64/libldap_r-2.4.so.2 #8 0x00005638f66baf52 in slap_client_connect () #9 0x00005638f672997d in ?? () #10 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #11 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc4d77fe700 (LWP 19293)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc4d6ffd700 (LWP 23630)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc4d67fc700 (LWP 23631)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fc4d5ffb700 (LWP 23632)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc4d57fa700 (LWP 23633)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc4d4ff9700 (LWP 23634)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc705895780 (LWP 19169)): #0 0x00007fc70321cf17 in pthread_join () from /lib64/libpthread.so.0 #1 0x00005638f66c0dc9 in slapd_daemon () #2 0x00005638f66a7d68 in main () *Thu Sep 21 16:20:42 UTC 2017* Thread 14 (Thread 0x7fc4eb57c700 (LWP 19171)): #0 0x00007fc702d3d8e3 in epoll_wait () from /lib64/libc.so.6 #1 0x00005638f66beda9 in ?? () #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc4ead7b700 (LWP 19172)): #0 0x00005638f66cfa88 in entry_decode () #1 0x00005638f67a44a2 in hdb_id2entry () #2 0x00005638f679b6f8 in hdb_cache_find_id () #3 0x00005638f675041d in hdb_search () #4 0x00005638f67304c2 in overlay_op_walk () #5 0x00005638f6730624 in ?? () #6 0x00005638f66c69e9 in fe_op_search () #7 0x00005638f66c62b6 in do_search () #8 0x00005638f66c3ad4 in ?? () #9 0x00005638f66c3e3b in ?? () #10 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #11 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc4ea57a700 (LWP 19173)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fc4e9d79700 (LWP 19174)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fc4e9578700 (LWP 19175)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fc4e8d77700 (LWP 19176)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc4d7fff700 (LWP 19179)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc4d77fe700 (LWP 19293)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc4d6ffd700 (LWP 23630)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc4d67fc700 (LWP 23631)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fc4d5ffb700 (LWP 23632)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc4d57fa700 (LWP 23633)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc4d4ff9700 (LWP 23634)): #0 0x00007fc70321f905 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc705441b4b in ?? () from /lib64/libldap_r-2.4.so.2 #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc705895780 (LWP 19169)): #0 0x00007fc70321cf17 in pthread_join () from /lib64/libpthread.so.0 #1 0x00005638f66c0dc9 in slapd_daemon () #2 0x00005638f66a7d68 in main () Thu Sep 21 16:20:44 UTC 2017 Thread 14 (Thread 0x7fc4eb57c700 (LWP 19171)): #0 0x00007fc702d3d8e3 in epoll_wait () from /lib64/libc.so.6 #1 0x00005638f66beda9 in ?? () #2 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc4ead7b700 (LWP 19172)): #0 0x00005638f6706b30 in ad_keystring () #1 0x00005638f6706c6e in slap_bv2ad () #2 0x00005638f66cf9d6 in entry_decode () #3 0x00005638f67a44a2 in hdb_id2entry () #4 0x00005638f679b6f8 in hdb_cache_find_id () #5 0x00005638f675041d in hdb_search () #6 0x00005638f67304c2 in overlay_op_walk () #7 0x00005638f6730624 in ?? () #8 0x00005638f66c69e9 in fe_op_search () #9 0x00005638f66c62b6 in do_search () #10 0x00005638f66c3ad4 in ?? () #11 0x00005638f66c3e3b in ?? () #12 0x00007fc705441afa in ?? () from /lib64/libldap_r-2.4.so.2 #13 0x00007fc70321bde5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fc702d3d30d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc4ea57a700 (LWP 19173)):
After i did db_recover db_stat is responding fast but cpu spikes are still there, is there any way to find if my db is bad?
Thanks, Ram
openldap-technical@openldap.org
-
Matheus Eduardo Bonifacio Morais -
rammohan ganapavarapu