Full_Name: Josiah Webb Version: 2.4.23 OS: Debian Lenny (5.0.2) URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (146.115.147.2)

When running 5 concurrent ldapsearch clients (each with 10 concurrent threads) for unindexed attributes, slapd terminates with a SIGABRT about 3/4 of the way through all dn's.

here is the gdb interpreted core file dump and backtrace:

-----------------------------------------------------------------

awesome@ldapmaster-dev:~$ gdb /usr/local/libexec/slapd core.12907 GNU gdb 6.8-debian Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i486-linux-gnu"...

warning: Can't read pathname for load map: Input/output error. Reading symbols from /usr/local/lib/libtcmalloc.so.0...done. Loaded symbols for /usr/local/lib/libtcmalloc.so.0 Reading symbols from /usr/lib/libltdl.so.3...done. Loaded symbols for /usr/lib/libltdl.so.3 Reading symbols from /lib/i686/cmov/libdl.so.2...done. Loaded symbols for /lib/i686/cmov/libdl.so.2 Reading symbols from /usr/lib/libdb-4.6.so...done. Loaded symbols for /usr/lib/libdb-4.6.so Reading symbols from /lib/i686/cmov/libpthread.so.0...done. Loaded symbols for /lib/i686/cmov/libpthread.so.0 Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.8...done. Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.8 Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.8...done. Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.8 Reading symbols from /lib/i686/cmov/libcrypt.so.1...done. Loaded symbols for /lib/i686/cmov/libcrypt.so.1 Reading symbols from /lib/i686/cmov/libresolv.so.2...done. Loaded symbols for /lib/i686/cmov/libresolv.so.2 Reading symbols from /lib/i686/cmov/libc.so.6...done. Loaded symbols for /lib/i686/cmov/libc.so.6 Reading symbols from /usr/lib/libstdc++.so.6...done. Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/i686/cmov/libm.so.6...done. Loaded symbols for /lib/i686/cmov/libm.so.6 Reading symbols from /lib/libgcc_s.so.1...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /usr/lib/libz.so.1...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /lib/i686/cmov/libnss_files.so.2...done. Loaded symbols for /lib/i686/cmov/libnss_files.so.2 Core was generated by `/usr/local/libexec/slapd -d -1'. Program terminated with signal 6, Aborted. [New process 12911] [New process 12935] [New process 12934] [New process 12933] [New process 12932] [New process 12931] [New process 12930] [New process 12929] [New process 12928] [New process 12927] [New process 12926] [New process 12925] [New process 12924] [New process 12912] [New process 12907] [New process 12910] [New process 12909] [New process 12908] #0 0xb7f43424 in __kernel_vsyscall ()

(gdb) backtrace #0 0xb7f43424 in __kernel_vsyscall () #1 0xb7a7c640 in raise () from /lib/i686/cmov/libc.so.6 #2 0xb7a7e018 in abort () from /lib/i686/cmov/libc.so.6 #3 0xb7a755be in __assert_fail () from /lib/i686/cmov/libc.so.6 #4 0x0809853b in ch_malloc (size=1407) at ch_malloc.c:57 #5 0x08132a7a in hdb_id2entry (be=0x15ef5f64, tid=0xbb232d0, id=170395, e=0x15e35d08) at id2entry.c:143 #6 0x08128fea in hdb_cache_find_id (op=0xc429000, tid=0xbb232d0, id=170395, eip=0x15ef5eec, flag=2, lock=0x15ef5e44) at cache.c:998 #7 0x08103923 in hdb_search (op=0xc429000, rs=0x15ef7148) at search.c:707 #8 0x080e40dc in overlay_op_walk (op=0xc429000, rs=0x15ef7148, which=op_search, oi=0x9e39100, on=0x9e39400) at backover.c:669 #9 0x080e4ba1 in over_op_func (op=0xc429000, rs=0x15ef7148, which=op_search) at backover.c:721 #10 0x0807dc63 in fe_op_search (op=0xc429000, rs=0x15ef7148) at search.c:366 #11 0x0807e4d8 in do_search (op=0xc429000, rs=0x15ef7148) at search.c:217 #12 0x0807b546 in connection_operation (ctx=0x15ef7228, arg_v=0xc429000) at connection.c:1109 #13 0x0807c17f in connection_read_thread (ctx=0x15ef7228, argv=0x13) at connection.c:1245 #14 0x081590ec in ldap_int_thread_pool_wrapper (xpool=0x9e0a0b0) at tpool.c:685 #15 0xb7d924c0 in start_thread () from /lib/i686/cmov/libpthread.so.0 #16 0xb7b316de in clone () from /lib/i686/cmov/libc.so.6

---------------------------------------------------------------

the slapd standard log (/var/log/syslog), logged no errors whatsoever, slapd simply "stopped:

no "out of memory" issues either

free -m (taken at the exact time of the SIGABRT):

total used free shared buffers cached Mem: 8115 7492 623 0 226 1895 -/+ buffers/cache: 5370 2744 Swap: 2055 0 2055

Running a freshly compiled 2.4.23 openldap, which was compiled against tcmalloc (malloc replacement) from google-perftools-1.7 (standard build commands: ./configure, make, make install (no errors)

openldap compiled as follows:

./configure LDFLAGS="-fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -ltcmalloc" --enable-slapd --enable-bdb --enable-modules --enable-accesslog --enable-syncprov --enable-crypt --enable-cleartext --enable-syslog --enable-debug make depend make make test make install STRIP=""

no errors during build.

backend is Berkely DB 4.6.21-16, which contains the following patches:

http://www.oracle.com/technetwork/database/berkeleydb/patch-098875.html

slapd.conf as follows:

##################### # Global Directives # ##################### allow bind_v2 include /usr/local/etc/openldap/schema/core.schema include /usr/local/etc/openldap/schema/cosine.schema include /usr/local/etc/openldap/schema/nis.schema include /usr/local/etc/openldap/schema/inetorgperson.schema include /usr/local/etc/openldap/schema/qmail.schema include /usr/local/etc/openldap/schema/radius.schema pidfile /var/run/slapd/slapd.pid argsfile /var/run/slapd/slapd.args loglevel stats sync config none threads 16 gentlehup on sizelimit 500 tool-threads 4 backend hdb

############################## # Main Backend DB Directives # ############################## database hdb suffix "dc=localnet,dc=com" rootdn "cn=Manager,dc=localnet,dc=com" rootpw {SSHA}Kcv02GxN6YBuUVV+ct10VAxrc3c1iPei directory "/local/var/openldap-bdb" dbconfig set_cachesize 2 147483648 0 dbconfig set_lk_max_objects 1500 dbconfig set_lk_max_locks 1500 dbconfig set_lk_max_lockers 1500 cachesize 50000 idlcachesize 200000 cachefree 1000 index objectClass eq index cn eq index sn eq index uid eq index mail eq index mailAlternateAddress eq index mailForwardingAddress eq index radiusUID eq index uidNumber eq index gidNumber eq index overquota eq index mailhost eq index entryCSN eq index entryUUID eq overlay accesslog logdb "cn=accesslog" logops writes logsuccess TRUE logpurge 02+00:00 01+00:00 overlay syncprov syncprov-checkpoint 10 5 checkpoint 1024 5 limits dn.exact="cn=Manager,dc=localnet,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited

################################### # Accesslog Backend DB Directives # ################################### database hdb suffix "cn=accesslog" directory "/usr/local/var/openldap-data" dbconfig set_cachesize 0 268435456 1 dbconfig set_lg_regionmax 1048576 dbconfig set_lg_max 10485760 dbconfig set_lg_bsize 2097152 dbconfig set_lg_dir /usr/local/var/openldap-data dbconfig set_tmp_dir /tmp dbconfig set_flags DB_LOG_AUTOREMOVE cachesize 2000 idlcachesize 8000 cachefree 100 checkpoint 1024 5 lastmod on index default eq index objectClass index entryCSN index reqEnd index reqResult index reqStart overlay syncprov syncprov-nopresent TRUE syncprov-reloadhint TRUE limits dn.exact="cn=Manager,dc=localnet,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited

######################### # Access ACL Directives # ######################### access to * by dn="cn=Manager,dc=localnet,dc=com" write by * read by anonymous auth by self write

# end slapd.conf

Server is a 4 core Intel Xeon E5320 Debian Lenny box with 2.6.26-2-686-bigmem kernel and 8GB of physical memory.

Our entire data set (.bdb) including indexes totals 1.3GB. we have a large enough set_cachesize to store our entire data set in memory.

Our LDAP tree contains around 200,000 dn's

please advise, or let me know if you need further details or information

thanks!