Slapd is coming down

Hello Guys, how are you doing ?

In the last weeks, my openldaps servers were suffering with many come down. I have a peak about the 400 connections per peer ( We have 2 ldap servers ). Although I has added more memory (4 GB), the problem decreased, but not resolved.

So, How Could I troubleshooting it ?

My enviroment:

- Two Red Hat Enterprise Linux Server release 6.6 (Santiago) - Two processors and 12 GB of RAM - Both environments are virtualized.

*************************************** My slapd.conf:

moduleload back_ldap moduleload translucent moduleload dynlist moduleload back_monitor backend mdb backend ldap

allow bind_v2 allow bind_anon_dn

database mdb directory /usr/local/openldap/var/openldap-translucent suffix "*****" rootdn "*************" rootpw **************************** maxsize 4294967296 sizelimit 100000 overlay translucent uri "ldap://localhost:1389/" translucent_bind_local on translucent_pwmod_local on translucent_local ****************************** idassert-bind bindmethod=none

overlay dynlist dynlist-attrset groupOfURLs memberURL member:uniqueMember

************************** My slapd-syncrepl.conf:

allow bind_v2 allow bind_anon_dn moduleload syncprov moduleload dynlist

# Definicoes da base primaria database mdb suffix "**" rootdn "******" directory /usr/local/openldap/var/openldap-syncrepl rootpw ****** sizelimit 100000 maxsize 8589934592

overlay dynlist dynlist-attrset groupOfURLs memberURL member:uniqueMember

loglevel sync stats

idletimeout 0

# ACLs include /usr/local/openldap/etc/openldap/schema/sicredi.acl

overlay syncprov

# Começo do Consumidor index entryUUID eq

# syncrepl directives syncrepl rid=0 provider=ldap://MASTER_Production:389 bindmethod=simple binddn="************" credentials=**** searchbase="****" logbase="cn=accesslog" logfilter="(&(objectClass=auditWriteObject)(reqResult=0))" type=refreshAndPersist retry="60 +" syncdata=accesslog

# Refer updates to the master updateref ldap://MASTER_Production:389

***** My openldap version: openldap-ltb-2.4.39-1.el6.x86_64

And What I saw in the logs:

Aug 19 09:51:43 ds1openldap2h kernel: slapd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Aug 19 09:51:43 ds1openldap2h kernel: slapd cpuset=/ mems_allowed=0 Aug 19 09:51:43 ds1openldap2h kernel: Pid: 4233, comm: slapd Not tainted 2.6.32-504.23.4.el6.x86_64 #1 Aug 19 09:51:43 ds1openldap2h kernel: Call Trace: Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff810d4241>] ? cpuset_print_task_mems_allowed+0x91/0xb0 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff81127500>] ? dump_header+0x90/0x1b0 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8122ee7c>] ? security_real_capable_noaudit+0x3c/0x70 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff81127982>] ? oom_kill_process+0x82/0x2a0 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff811278c1>] ? select_bad_process+0xe1/0x120 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff81127dc0>] ? out_of_memory+0x220/0x3c0 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff811346ff>] ? __alloc_pages_nodemask+0x89f/0x8d0 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8116c9aa>] ? alloc_pages_current+0xaa/0x110 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff811248f7>] ? __page_cache_alloc+0x87/0x90 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff811242de>] ? find_get_page+0x1e/0xa0 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff81125897>] ? filemap_fault+0x1a7/0x500 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8114ed04>] ? __do_fault+0x54/0x530 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8114f2d7>] ? handle_pte_fault+0xf7/0xb00 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8114ff79>] ? handle_mm_fault+0x299/0x3d0 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8104d096>] ? __do_page_fault+0x146/0x500 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff81529a1e>] ? thread_return+0x4e/0x7d0 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8153001e>] ? do_page_fault+0x3e/0xa0 Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8152d3d5>] ? page_fault+0x25/0x30 Aug 19 09:51:44 ds1openldap2h kernel: Mem-Info: Aug 19 09:51:44 ds1openldap2h kernel: Node 0 DMA per-cpu: Aug 19 09:51:44 ds1openldap2h kernel: CPU 0: hi: 0, btch: 1 usd: 0 Aug 19 09:51:44 ds1openldap2h kernel: CPU 1: hi: 0, btch: 1 usd: 0 Aug 19 09:51:44 ds1openldap2h kernel: Node 0 DMA32 per-cpu: Aug 19 09:51:44 ds1openldap2h kernel: CPU 0: hi: 186, btch: 31 usd: 0 Aug 19 09:51:44 ds1openldap2h kernel: CPU 1: hi: 186, btch: 31 usd: 0 Aug 19 09:51:44 ds1openldap2h kernel: Node 0 Normal per-cpu: Aug 19 09:51:44 ds1openldap2h kernel: CPU 0: hi: 186, btch: 31 usd: 10 Aug 19 09:51:44 ds1openldap2h kernel: CPU 1: hi: 186, btch: 31 usd: 0 Aug 19 09:51:44 ds1openldap2h kernel: active_anon:2571953 inactive_anon:399686 isolated_anon:0 Aug 19 09:51:44 ds1openldap2h kernel: active_file:305 inactive_file:604 isolated_file:0 Aug 19 09:51:44 ds1openldap2h kernel: unevictable:0 dirty:1 writeback:0 unstable:0 Aug 19 09:51:44 ds1openldap2h kernel: free:29811 slab_reclaimable:2307 slab_unreclaimable:7010 Aug 19 09:51:44 ds1openldap2h kernel: Node 0 DMA free:15276kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14884kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 19 09:51:44 ds1openldap2h kernel: lowmem_reserve[]: 0 3000 12090 12090 Aug 19 09:51:44 ds1openldap2h kernel: Node 0 DMA32 free:53124kB min:16748kB low:20932kB high:25120kB active_anon:2127764kB inactive_anon:559804kB active_file:0kB inactive_file:152kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:0kB mapped:128kB shmem:0kB slab_reclaimable:8kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:900kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 19 09:51:44 ds1openldap2h kernel: lowmem_reserve[]: 0 0 9090 9090 Aug 19 09:51:44 ds1openldap2h kernel: Node 0 Normal free:51016kB min:50752kB low:63440kB high:76128kB active_anon:8160048kB inactive_anon:1038940kB active_file:1224kB inactive_file:2052kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:9308160kB mlocked:0kB dirty:4kB writeback:0kB mapped:1288kB shmem:0kB slab_reclaimable:9220kB slab_unreclaimable:28028kB kernel_stack:1392kB pagetables:44504kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 19 09:51:44 ds1openldap2h kernel: lowmem_reserve[]: 0 0 0 0 Aug 19 09:51:44 ds1openldap2h kernel: Node 0 DMA: 1*4kB 1*8kB 2*16kB 2*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 3*4096kB = 15276kB Aug 19 09:51:44 ds1openldap2h kernel: Node 0 DMA32: 2599*4kB 1163*8kB 354*16kB 75*32kB 28*64kB 15*128kB 9*256kB 12*512kB 9*1024kB 0*2048kB 1*4096kB = 53236kB Aug 19 09:51:44 ds1openldap2h kernel: Node 0 Normal: 846*4kB 640*8kB 413*16kB 236*32kB 119*64kB 48*128kB 28*256kB 10*512kB 2*1024kB 0*2048kB 0*4096kB = 50760kB Aug 19 09:51:44 ds1openldap2h kernel: 5598 total pagecache pages Aug 19 09:51:44 ds1openldap2h kernel: 4605 pages in swap cache Aug 19 09:51:44 ds1openldap2h kernel: Swap cache stats: add 1173695, delete 1169090, find 121530/123860 Aug 19 09:51:44 ds1openldap2h kernel: Free swap = 0kB Aug 19 09:51:44 ds1openldap2h kernel: Total swap = 2097148kB Aug 19 09:51:44 ds1openldap2h kernel: 3145712 pages RAM Aug 19 09:51:44 ds1openldap2h kernel: 97157 pages reserved Aug 19 09:51:44 ds1openldap2h kernel: 797 pages shared Aug 19 09:51:44 ds1openldap2h kernel: 3014707 pages non-shared Aug 19 09:51:44 ds1openldap2h kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Aug 19 09:51:44 ds1openldap2h kernel: [ 441] 0 441 2795 0 0 -17 -1000 udevd Aug 19 09:51:44 ds1openldap2h kernel: [ 1251] 0 1251 47346 150 0 0 0 vmtoolsd Aug 19 09:51:44 ds1openldap2h kernel: [ 1290] 0 1290 23283 37 1 -17 -1000 auditd Aug 19 09:51:44 ds1openldap2h kernel: [ 1326] 65 1326 107809 573 1 0 0 nslcd Aug 19 09:51:44 ds1openldap2h kernel: [ 1340] 0 1340 62279 388 1 0 0 rsyslogd Aug 19 09:51:44 ds1openldap2h kernel: [ 1353] 0 1353 2707 46 0 0 0 irqbalance Aug 19 09:51:44 ds1openldap2h kernel: [ 1369] 32 1369 4744 15 1 0 0 rpcbind Aug 19 09:51:44 ds1openldap2h kernel: [ 1380] 81 1380 5881 36 1 0 0 dbus-daemon Aug 19 09:51:44 ds1openldap2h kernel: [ 1413] 0 1413 1020 0 1 0 0 acpid Aug 19 09:51:44 ds1openldap2h kernel: [ 1423] 68 1423 10041 151 0 0 0 hald Aug 19 09:51:44 ds1openldap2h kernel: [ 1424] 0 1424 5100 2 1 0 0 hald-runner Aug 19 09:51:44 ds1openldap2h kernel: [ 1456] 0 1456 5630 2 1 0 0 hald-addon-inpu Aug 19 09:51:44 ds1openldap2h kernel: [ 1470] 68 1470 4502 2 1 0 0 hald-addon-acpi Aug 19 09:51:44 ds1openldap2h kernel: [ 1510] 55 1510 2488944 50131 1 0 0 slapd Aug 19 09:51:44 ds1openldap2h kernel: [ 1572] 28 1572 241540 244 1 0 0 nscd Aug 19 09:51:44 ds1openldap2h kernel: [ 1597] 0 1597 16081 20 1 -17 -1000 sshd Aug 19 09:51:44 ds1openldap2h kernel: [ 1606] 38 1606 6566 72 1 0 0 ntpd Aug 19 09:51:44 ds1openldap2h kernel: [ 1629] 0 1629 28188 2 0 0 0 abrtd Aug 19 09:51:44 ds1openldap2h kernel: [ 1637] 0 1637 28131 23 1 0 0 abrt-dump-oops Aug 19 09:51:44 ds1openldap2h kernel: [ 1651] 0 1651 51804 2959 1 0 0 osad Aug 19 09:51:44 ds1openldap2h kernel: [ 1661] 0 1661 28742 18 1 0 0 crond Aug 19 09:51:44 ds1openldap2h kernel: [ 1771] 497 1771 257947 665 1 0 0 icinga2 Aug 19 09:51:44 ds1openldap2h kernel: [ 1795] 0 1795 25232 26 1 0 0 rhnsd Aug 19 09:51:44 ds1openldap2h kernel: [ 1804] 0 1804 27085 23 0 0 0 rhsmcertd Aug 19 09:51:44 ds1openldap2h kernel: [ 1818] 0 1818 1016 1 0 0 0 mingetty Aug 19 09:51:44 ds1openldap2h kernel: [ 1820] 0 1820 1016 1 0 0 0 mingetty Aug 19 09:51:44 ds1openldap2h kernel: [ 1822] 0 1822 1016 1 0 0 0 mingetty Aug 19 09:51:44 ds1openldap2h kernel: [ 1824] 0 1824 1016 1 0 0 0 mingetty Aug 19 09:51:44 ds1openldap2h kernel: [ 1826] 0 1826 1016 1 0 0 0 mingetty Aug 19 09:51:44 ds1openldap2h kernel: [ 1828] 0 1828 2794 0 1 -17 -1000 udevd Aug 19 09:51:44 ds1openldap2h kernel: [ 1829] 0 1829 2794 0 1 -17 -1000 udevd Aug 19 09:51:44 ds1openldap2h kernel: [ 1830] 0 1830 1016 1 0 0 0 mingetty Aug 19 09:51:44 ds1openldap2h kernel: [21760] 55 21760 4578590 2911874 1 0 0 slapd Aug 19 09:51:44 ds1openldap2h kernel: [16962] 0 16962 3105 113 1 0 0 nmon_x86_64_rhe Aug 19 09:51:44 ds1openldap2h kernel: [20314] 0 20314 3059 113 0 0 0 nmon_x86_64_rhe Aug 19 09:51:44 ds1openldap2h kernel: Out of memory: Kill process 21760 (slapd) score 957 or sacrifice child Aug 19 09:51:44 ds1openldap2h kernel: Killed process 21760, UID 55, (slapd) total-vm:18314360kB, anon-rss:11646816kB, file-rss:680kB

Thanks for your attention.