On 05/18/2011 12:42 AM, Tim.Mooney@ndsu.edu wrote:

Full_Name: Tim Mooney Version: 2.4.25 OS: Linux, RHEL 5.6 x86_64 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2001:4930:106:0:18bb:1140:fa3d:f713)

Recently replaced an OpenLDAP 2.3.x server with a newer system running RHEL 5.6 x86_64. OpenLDAP 2.4.25 + Berkeley DB 4.8.30, built as x86_64 also. We've had slapd segfault several times now, and the most recent time it happened, I had slapd running under gdb and caught where it was happening.

As part of the server and OpenLDAP replacement, we're beginning to support ldap authentication via SASL pass-through to Kerberos. With the authentication, we also have the need to do DN rewriting at auth time, which is why I've enabled rwm and have the following in slapd-config:

dn: olcOverlay={0}rwm,olcDatabase={-1}frontend,cn=config objectClass: olcOverlayConfig objectClass: olcRwmConfig olcOverlay: {0}rwm olcRwmRewrite: {0}rwm-rewriteEngine "on" olcRwmRewrite: {1}rwm-rewriteMap "ldap" "attr2dn" "ldap://localhost/dc=nodak,d c=edu?dn?sub" olcRwmRewrite: {2}rwm-rewriteContext "bindDN" olcRwmRewrite: {3}rwm-rewriteRule "^(iid=[^, ]+).*" "${attr2dn($1)}" ":@I" olcRwmTFSupport: false olcRwmNormalizeMapped: FALSE structuralObjectClass: olcRwmConfig

This was generated via the automatic conversion from slapd.conf, the original entries in slapd.conf were:

# # TVM: new with our OpenLDAP 2.4.x install: load the rwm overlay # and add rules so that binds with the iid work. # overlay rwm rwm-rewriteEngine on

# define a rewriteMap function that returns the dn for a particular attr # This is straight out of the first bindDN example in slapo-rwm(5) rwm-rewriteMap ldap attr2dn "ldap://localhost/dc=nodak,dc=edu?dn?sub"

rwm-rewriteContext bindDN # and now the magic: parse out the IID and pass it to the attr2dn function. # This is also almost exactly taken from slapo-rwm(5), though I'm using iid # instead of mail and I'm not anchoring the regex and using $1, so it doesn't # matter if it's qualified or not. rwm-rewriteRule "^(iid=[^, ]+).*" "${attr2dn($1)}" ":@I"

With those rewrite rules in place, certain searches have been triggering the slapd segfault.

The segfault:

[root@server2 ldap]# /etc/init.d/NDUS_ldap stop; gdb /usr/local/sbin/slapd Stopping slapd: [ OK ] GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5_6.2) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or laterhttp://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/...

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x2aaaaaf47000 [Thread debugging using libthread_db enabled] [New process 5168] [Thread debugging using libthread_db enabled] [New Thread 0x40800940 (LWP 5169)] [New Thread 0x41001940 (LWP 5170)] [New Thread 0x41802940 (LWP 5182)] [New Thread 0x42003940 (LWP 5183)] [New Thread 0x42804940 (LWP 5302)] [New Thread 0x43005940 (LWP 5303)] [New Thread 0x43806940 (LWP 7677)]

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x42804940 (LWP 5302)] 0x00002aaab0a4e95e in map_attr_value (dc=0x428028a0, adp=0x428026b8, mapped_attr=0x428026a0, value=0x2aaae10015d0, mapped_value=0x42802690, remap=0, memctx=0x2aaadbb91740) at ../../../../servers/slapd/overlays/rwmmap.c:439 439 } else if ( ad->ad_type->sat_equality->smr_usage& SLAP_MR_MUTATION_NORMALIZER ) { Load new symbol table from "/usr/local/sbin/slapd"? (y or n) y Reading symbols from /usr/local/sbin/slapd...done.

(gdb) where #0 0x00002aaab0a4e95e in map_attr_value (dc=0x428028a0, adp=0x428026b8, mapped_attr=0x428026a0, value=0x2aaae10015d0, mapped_value=0x42802690, remap=0, memctx=0x2aaadbb91740) at ../../../../servers/slapd/overlays/rwmmap.c:439 #1 0x00002aaab0a4ee87 in rwm_int_filter_map_rewrite (op=0x2aaadbb91380, dc=0x428028a0, f=0x2aaae1001688, fstr=0x42802730) at ../../../../servers/slapd/overlays/rwmmap.c:528 #2 0x00002aaab0a4edba in rwm_int_filter_map_rewrite (op=0x2aaadbb91380, dc=0x428028a0, f=0x2aaae10016b0, fstr=0x428027d0) at ../../../../servers/slapd/overlays/rwmmap.c:691 #3 0x00002aaab0a4edba in rwm_int_filter_map_rewrite (op=0x2aaadbb91380, dc=0x428028a0, f=0x0, fstr=0x428028c0) at ../../../../servers/slapd/overlays/rwmmap.c:691 #4 0x00002aaab0a4f5eb in rwm_filter_map_rewrite (op=0x2aaaaaf29fc0, dc=0x2aaaaaeab420, f=0x0, fstr=0x2aaaaaf67740) at ../../../../servers/slapd/overlays/rwmmap.c:793 #5 0x00002aaab0a4b542 in rwm_op_search (op=0x2aaadbb91380, rs=0x42803c10) at ../../../../servers/slapd/overlays/rwm.c:969 #6 0x00002aaaaab5ee0a in overlay_op_walk (op=0x2aaadbb91380, rs=0x42803c10, which=op_search, oi=0x2aaaab01e060, on=0x2aaaab01e240) at ../../../servers/slapd/backover.c:659 #7 0x00002aaaaab5f418 in over_op_func (op=0x2aaadbb91380, rs=0x42803c10, which=op_search) at ../../../servers/slapd/backover.c:721 #8 0x00002aaaaaaf7325 in do_search (op=0x2aaadbb91380, rs=0x42803c10) at ../../../servers/slapd/search.c:217 #9 0x00002aaaaaaf4644 in connection_operation (ctx=0x42803d60, arg_v=<value optimized out>) at ../../../servers/slapd/connection.c:1113 #10 0x00002aaaaaaf4ca1 in connection_read_thread (ctx=0x42803d60, argv=<value optimized out>) at ../../../servers/slapd/connection.c:1249 #11 0x00002aaaaabf3d08 in ldap_int_thread_pool_wrapper (xpool=0x2aaaaaf7dfa0) at ../../../libraries/libldap_r/tpool.c:685 #12 0x00002aaaad5b973d in start_thread () from /lib64/libpthread.so.0 #13 0x00002aaaadaac0cd in clone () from /lib64/libc.so.6 (gdb) print dc $1 = (dncookie *) 0x428028a0 (gdb) print *dc $2 = {rwmap = 0x2aaaab01e420, conn = 0x2aaab0c70330, ctx = 0x2aaab0a50519 "searchFilterAttrDN", rs = 0x42803c10} (gdb) print dc->rwmap $3 = (struct ldaprwmap *) 0x2aaaab01e420 (gdb) print *(dc->rwmap) $4 = {rwm_rw = 0x2aaaab01e480, rwm_bva_rewrite = 0x2aaaab023a50, rwm_oc = { drop_missing = 0, map = 0x0, remap = 0x0}, rwm_at = {drop_missing = 0, map = 0x0, remap = 0x0}, rwm_bva_map = 0x0, rwm_flags = 2}

(gdb) print adp $5 = (AttributeDescription **) 0x428026b8 (gdb) print *adp $6 = (AttributeDescription *) 0x2aaae10015f0 (gdb) print **adp $7 = {ad_next = 0x7564652e6b61646f, ad_type = 0x2aaaaaeab420, ad_cname = { bv_len = 23, bv_val = 0x2aaae1001628 "apple-group-nestedgroup"}, ad_tags = {bv_len = 145, bv_val = 0x91<Address 0x91 out of bounds>}, ad_flags = 4096} (gdb) print ad->ad_type->sat_equality->smr_usage Cannot access memory at address 0x58

(gdb) print ad->ad_type->sat_equality $8 = (MatchingRule *) 0x0

(gdb) print *(ad->ad_type) $10 = {sat_atype = {at_oid = 0x2aaaaac38a54 "1.1.1", at_names = 0x0, at_desc = 0x2aaaaac35d10 "Catchall for undefined attribute types", at_obsolete = 1, at_sup_oid = 0x0, at_equality_oid = 0x0, at_ordering_oid = 0x0, at_substr_oid = 0x0, at_syntax_oid = 0x0, at_syntax_len = 0, at_single_value = 0, at_collective = 0, at_no_user_mod = 1, at_usage = 3, at_extensions = 0x0}, sat_cname = { bv_len = 9, bv_val = 0x2aaaaac38a5a "UNDEFINED"}, sat_sup = 0x0, sat_subtypes = 0x0, sat_equality = 0x0, sat_approx = 0x0, sat_ordering = 0x0, sat_substr = 0x0, sat_syntax = 0x2aaaaaf694e0, sat_check = 0, sat_oidmacro = 0x0, sat_soidmacro = 0x0, sat_flags = 768, sat_next = {stqe_next = 0x0}, sat_ad = 0x0, sat_ad_mutex = {__data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000'<repeats 39 times>, __align = 0}} (gdb) print value $11 = (struct berval *) 0x2aaae10015d0 (gdb) print *value $12 = {bv_len = 36, bv_val = 0x2aaae1001650 "ABCDEFAB-CDEF-ABCD-EFAB-CDEF0000001B"} (gdb) print *mapped_value $13 = {bv_len = 0, bv_val = 0x2aaae10015a0 "\243"} (gdb) print *mapped_attr $14 = {bv_len = 23, bv_val = 0x2aaae1001628 "apple-group-nestedgroup"}

At the time of the search, the very last thing that was logged was

May 17 17:03:03 server2 slapd[5168]: conn=28588 op=3 SRCH base="cn=groups,dc=ndsu,dc=nodak,dc=edu" scope=2 deref=0 filter="(&(?objectClass=posixGroup)(?objectClass=apple-group)(objectClass=extensibleObject)(|(?apple-group-nestedgroup=ABCDEFAB-CDEF-ABCD-EFAB-CDEF0000001B)))"

May 17 17:03:03 server2 slapd[5168]: conn=28588 op=3 SRCH attr=cn apple-generateduid gidNumber apple-group-realname ttl sambaSID rid primaryGroupID apple-keyword apple-group-nestedgroup

I'll happily provide any details that I've mistakenly left out or that would aid in debugging the issue.

The issue certainly could be caused by an error in my rwmRewriteRule, but I imagine that slapd shouldn't segfault even if my rwmRewriteRule is wrong.

Yes (I believe), and yes. I believe the mapping is being applied to an attribute that is not explicitly defined in the schema, but rather proxied or somehow treated as undefined. For this reason, the matching rule pointer is NULL. Can you check the definition of "apple-group-nestedgroup", if any? Of course, slapo-rwm should not crash on something like this.

p.