This appears to be due to an incorrect patch for ITS#5849. A fix is now in HEAD. (libldap/tls_o.c)
Mathieu MILLET wrote:
Hi everyone,
We have a configuration with 2 Openldap in Multimaster Replication mode, using TLS, client certificate and SASL EXTERNAL to secure the replication. (Two sets of certificate are used to differentiate the replication of cn=config and the data backend)
It is working in 2.4.13 (on Red Hat Entreprise Linux 4.5 and Debian 5), compiled from sources, with openssl libs (not gnutls).
Being affected by ITS#5906 (slapo-rwm with back-config) and ITS#5843 (slapd syncrepl MMR with deleted entries), I decided to try on a (test) environment this new version.
With 2.4.15 (and also reproduced in 2.4.14), our configuration segfaults on one of the two nodes at a short period of time after the 1st replication. When restarting the segfaulted node, the other segfaults and so on.
The segfault happens when just adding the syncrepl configuration for the cn=config backend, but some times they are alive long enough to enable syncrepl options for the databackend, but then again, segfaults always happen.
During some segfaults, I got some backtraces that follow : *** glibc detected *** /usr/local/libexec/slapd: realloc(): invalid pointer: 0xb6db9260 *** ======= Backtrace: ========= /lib/i686/cmov/libc.so.6[0xb6ccf624] /lib/i686/cmov/libc.so.6(realloc+0x242)[0xb6cd3c82] /usr/lib/i686/cmov/libcrypto.so.0.9.8[0xb6e224c5] /usr/lib/i686/cmov/libcrypto.so.0.9.8(CRYPTO_realloc+0xab)[0xb6e22c0b] /usr/lib/i686/cmov/libcrypto.so.0.9.8(BUF_MEM_grow+0x75)[0xb6e83415] /usr/lib/i686/cmov/libcrypto.so.0.9.8[0xb6ea95a4] /usr/lib/i686/cmov/libcrypto.so.0.9.8(ASN1_item_ex_i2d+0x14d)Abandon
or *** glibc detected *** /usr/local/libexec/slapd: realloc(): invalid pointer: 0xb6de4260 *** ======= Backtrace: ========= /lib/i686/cmov/libc.so.6[0xb6cfa624] /lib/i686/cmov/libc.so.6(realloc+0x242)[0xb6cfec82] /usr/lib/i686/cmov/libcrypto.so.0.9.8[0xb6e4d4c5] /usr/lib/i686/cmov/libcrypto.so.0.9.8(CRYPTO_realloc+0xab)[0xb6e4dc0b] /usr/lib/i686/cmov/libcrypto.so.0.9.8(BUF_MEM_grow+0x75)[0xb6eae415] /usr/lib/i686/cmov/libcrypto.so.0.9.8[0xb6ed45a4] /usr/lib/i686/cmov/libcrypto.so.0.9.8(ASN1_item_ex_i2d+0x14d)[0xb6edbfbd] /usr/lib/i686/cmov/libcrypto.so.0.9.8[0xb6edc5b5] /usr/lib/i686/cmov/libcrypto.so.0.9.8(ASN1_item_ex_i2d+0x2f3)[0xb6edc163] /usr/lib/i686/cmov/libcrypto.so.0.9.8[0xb6edc5b5] /usr/lib/i686/cmov/libcrypto.so.0.9.8(ASN1_item_ex_i2d+0x2f3)[0xb6edc163] /usr/lib/i686/cmov/libcrypto.so.0.9.8(ASN1_item_i2d+0x53)[0xb6edc923] /usr/lib/i686/cmov/libcrypto.so.0.9.8(i2d_X509+0x2e)[0xb6ed506e] /usr/lib/i686/cmov/libssl.so.0.9.8(ssl3_output_cert_chain+0x3d4)[0xb6f7b824] /usr/lib/i686/cmov/libssl.so.0.9.8(ssl3_send_client_certificate+0x142)[0xb6f721b2] /usr/lib/i686/cmov/libssl.so.0.9.8(ssl3_connect+0xb3)[0xb6f759d3] /usr/lib/i686/cmov/libssl.so.0.9.8(SSL_connect+0x2a)[0xb6f89c1a] /usAbandon
It definitely has something to do with TLS stuff.
After more testing, the ldap* clients also segfault when performing TLS and SASL External with Client Certificate.
Has anybody encounter this behaviour ?
Thanks in advance for any help, Sincerely yours, Mathieu MILLET.
******************* Startup config (of one node) **************
slapd.d/cn=config/olcDatabase={-1}frontend.ldif
dn: olcDatabase={-1}frontend objectClass: olcDatabaseConfig objectClass: olcFrontendConfig olcDatabase: {-1}frontend olcAccess: {0}to dn.base="" by * read olcAccess: {1}to dn.base="cn=subschema" by * read olcAccess: {2}to * by self write by users read by anonymous auth olcAddContentAcl: FALSE olcLastMod: TRUE olcMaxDerefDepth: 0 olcReadOnly: FALSE olcSchemaDN: cn=Subschema olcMonitoring: FALSE structuralObjectClass: olcDatabaseConfig entryUUID: 10002a99-3485-4805-a247-9e4ee777135d creatorsName: cn=config createTimestamp: 20090224192423Z entryCSN: 20090224192423.202231Z#000000#000#000000 modifiersName: cn=config modifyTimestamp: 20090224192423Z
slapd.d/cn=config/olcDatabase={0}config.ldif
dn: olcDatabase={0}config objectClass: olcDatabaseConfig olcDatabase: {0}config olcAccess: {0}to * by * none olcAddContentAcl: TRUE olcLastMod: TRUE olcMaxDerefDepth: 15 olcReadOnly: FALSE olcRootDN: cn=config olcRootPW:: c2VjcmV0 olcMonitoring: FALSE structuralObjectClass: olcDatabaseConfig entryUUID: fc35a505-ba8f-4bbf-828e-b061bb3aabba creatorsName: cn=config createTimestamp: 20090224192423Z entryCSN: 20090224192423.202231Z#000000#000#000000 modifiersName: cn=config modifyTimestamp: 20090224192423Z
slapd.d/cn=config/olcDatabase={2}hdb/olcOverlay={0}ppolicy.ldif
dn: olcOverlay={0}ppolicy objectClass: olcConfig objectClass: olcOverlayConfig objectClass: olcPPolicyConfig olcOverlay: {0}ppolicy olcPPolicyDefault: cn=default,ou=policies,dc=htam,dc=net olcPPolicyHashCleartext: FALSE olcPPolicyUseLockout: FALSE structuralObjectClass: olcPPolicyConfig entryUUID: 8078dd1d-369e-4c62-9fdc-1ce6820482d8 creatorsName: cn=config createTimestamp: 20090302125140Z entryCSN: 20090302125140.681319Z#000000#001#000000 modifiersName: cn=config modifyTimestamp: 20090302125140Z
slapd.d/cn=config/olcDatabase={2}hdb/olcOverlay={1}memberof.ldif
dn: olcOverlay={1}memberof objectClass: olcConfig objectClass: olcOverlayConfig objectClass: olcMemberOf olcOverlay: {1}memberof olcMemberOfDangling: ignore olcMemberOfRefInt: TRUE olcMemberOfGroupOC: groupOfUniqueNames olcMemberOfMemberAD: uniqueMember olcMemberOfMemberOfAD: memberOf structuralObjectClass: olcMemberOf entryUUID: b0a0abdd-77ef-47f6-a1e1-52637e30ebcc creatorsName: cn=config createTimestamp: 20090302125140Z entryCSN: 20090302125140.683800Z#000000#001#000000 modifiersName: cn=config modifyTimestamp: 20090302125140Z
slapd.d/cn=config/olcDatabase={2}hdb/olcOverlay={2}refint.ldif
dn: olcOverlay={2}refint objectClass: olcConfig objectClass: olcOverlayConfig objectClass: olcRefintConfig olcOverlay: {2}refint olcRefintAttribute: uniqueMember olcRefintNothing: cn=Manager,dc=htam,dc=net structuralObjectClass: olcRefintConfig entryUUID: 13d0a0a0-8284-447c-9d49-426e37692f57 creatorsName: cn=config createTimestamp: 20090302125140Z entryCSN: 20090302125140.685440Z#000000#001#000000 modifiersName: cn=config modifyTimestamp: 20090302125140Z
slapd.d/cn=config/cn=module{0}.ldif
dn: cn=module{0} objectClass: olcModuleList cn: module{0} olcModulePath: /usr/local/libexec/openldap olcModuleLoad: {0}memberof.la olcModuleLoad: {1}ppolicy.la olcModuleLoad: {2}refint.la olcModuleLoad: {3}retcode.la olcModuleLoad: {4}rwm.la olcModuleLoad: {5}syncprov.la olcModuleLoad: {6}unique.la olcModuleLoad: {7}back_monitor.la olcModuleLoad: {8}back_hdb.la olcModuleLoad: {9}back_relay.la structuralObjectClass: olcModuleList entryUUID: 353f4a38-3a12-446f-9176-570021c59341 creatorsName: cn=config createTimestamp: 20090224192423Z entryCSN: 20090224192423.202231Z#000000#001#000000 modifiersName: cn=config modifyTimestamp: 20090224192423Z
slapd.d/cn=config/olcDatabase={2}hdb.ldif
dn: olcDatabase={2}hdb objectClass: olcDatabaseConfig objectClass: olcHdbConfig olcDatabase: {2}hdb olcDbDirectory: /usr/local/var/openldap-data/ olcSuffix: dc=htam,dc=net olcAccess: {0}to attrs=userPassword by group/groupOfUniqueNames/uniqueMember=" cn=ldapadmins,ou=groups,dc=htam,dc=net" write by dn.subtree="ou=replicators,d c=htam,dc=net" read by dn.subtree="ou=computers,dc=htam,dc=net" auth by self =xwd by anonymous auth olcAccess: {1}to attrs=entry,objectClass,uid,uidNumber,gidNumber,loginShell,cn ,gecos,description,homeDirectory by group/groupOfUniqueNames/uniqueMember="cn =ldapadmins,ou=groups,dc=htam,dc=net" write by dn.subtree="ou=replicators,dc= htam,dc=net" read by dn.subtree="ou=computers,dc=htam,dc=net" read by self re ad olcAccess:: ezJ9dG8gYXR0cnM9dW5pcXVlTWVtYmVyIGJ5IGdyb3VwL2dyb3VwT2ZVbmlxdWVOYW 1lcy91bmlxdWVNZW1iZXI9ImNuPWxkYXBhZG1pbnMsb3U9Z3JvdXBzLGRjPWh0YW0sZGM9bmV0IiB 3cml0ZSBieSBkbi5zdWJ0cmVlPSJvdT1yZXBsaWNhdG9ycyxkYz1odGFtLGRjPW5ldCIgcmVhZCBi eSBkbi5zdWJ0cmVlPSJvdT1jb21wdXRlcnMsZGM9aHRhbSxkYz1uZXQiIHJlYWQg olcAccess: {3}to * by group/groupOfUniqueNames/uniqueMember="cn=ldapadmins,ou= groups,dc=htam,dc=net" write by dn.subtree="ou=replicators,dc=htam,dc=net" re ad by self read olcLastMod: TRUE olcMaxDerefDepth: 15 olcReadOnly: FALSE olcRootDN: cn=Manager,dc=htam,dc=net olcRootPW:: c2VjcmV0 olcMonitoring: TRUE olcDbCacheSize: 1000 olcDbConfig: {0}set_cachesize 0 268435456 1 olcDbConfig: {1}set_lg_regionmax 262144 olcDbConfig: {2}set_lg_bsize 2097152 olcDbConfig: {3}set_flags DB_LOG_AUTOREMOVE olcDbIndex: objectClass eq olcDbIndex: uid pres,eq olcDbIndex: cn pres,eq,sub olcDbIndex: sn pres,eq,sub olcDbIndex: givenname pres,eq,sub olcDbIndex: uniqueMember pres,eq olcDbIndex: memberUid pres,eq olcDbIndex: entryCSN eq olcDbIndex: entryUUID eq olcDbIndex: ipServicePort eq olcDbIndex: ipServiceProtocol eq olcDbIndex: oncRpcNumber eq olcDbIndex: ipProtocolNumber eq structuralObjectClass: olcHdbConfig entryUUID: 9f1eb1ca-a001-46db-aa58-4fc7897c64cc creatorsName: cn=config createTimestamp: 20090302125140Z entryCSN: 20090302125140.183122Z#000000#001#000000 modifiersName: cn=config modifyTimestamp: 20090302125140Z
slapd.d/cn=config/olcDatabase={1}monitor.ldif
dn: olcDatabase={1}monitor objectClass: olcDatabaseConfig olcDatabase: {1}monitor olcAccess: {0}to * by dn.base="cn=Manager,dc=htam,dc=net" read by * none olcLastMod: TRUE olcMaxDerefDepth: 15 olcReadOnly: FALSE olcMonitoring: FALSE structuralObjectClass: olcDatabaseConfig entryUUID: 6d366d19-e3ce-417b-a0b6-fd41bc690d83 creatorsName: cn=config createTimestamp: 20090302125140Z entryCSN: 20090302125140.118423Z#000000#001#000000 modifiersName: cn=config modifyTimestamp: 20090302125140Z
slapd.d/cn=config.ldif
dn: cn=config objectClass: olcGlobal cn: config olcConfigFile: slapd.conf.start olcConfigDir: slapd.d.start olcArgsFile: /usr/local/var/run/slapd.args olcAttributeOptions: lang- olcAuthzPolicy: none olcConcurrency: 0 olcConnMaxPending: 100 olcConnMaxPendingAuth: 1000 olcGentleHUP: FALSE olcIdleTimeout: 0 olcIndexSubstrIfMaxLen: 4 olcIndexSubstrIfMinLen: 2 olcIndexSubstrAnyLen: 4 olcIndexSubstrAnyStep: 2 olcIndexIntLen: 4 olcLocalSSF: 71 olcLogLevel: Packets olcLogLevel: Config olcLogLevel: Stats olcLogLevel: Sync olcPidFile: /usr/local/var/run/slapd.pid olcReadOnly: FALSE olcSaslSecProps: noplain,noanonymous olcServerID: 1 ldap://vmlinux01/ olcSockbufMaxIncoming: 262143 olcSockbufMaxIncomingAuth: 16777215 olcThreads: 16 olcTLSCACertificateFile: /usr/local/etc/openldap/cacerts/cacert.pem olcTLSCertificateFile: /usr/local/etc/openldap/slapd.cert.pem olcTLSCertificateKeyFile: /usr/local/etc/openldap/slapd.key.pem olcTLSCipherSuite: HIGH:MEDIUM olcTLSCRLCheck: none olcTLSVerifyClient: try olcToolThreads: 1 structuralObjectClass: olcGlobal entryUUID: 67b85bb6-58a2-4c6e-abd5-2bf7ce077d69 creatorsName: cn=config createTimestamp: 20090224192423Z entryCSN: 20090302142216.165509Z#000000#001#000000 modifiersName: cn=config modifyTimestamp: 20090302142216Z
******************* LDIF for activating syncrepl on cn=config
dn: cn=config changetype: modify replace: olcServerID olcServerID: 1 "ldap://vmlinux01" olcServerID: 2 "ldap://vmlinux02"
add: olcAuthzRegexp olcAuthzRegexp: "cn=.*_repl_config,o=Htam.net Inc.,c=fr" "cn=config"
dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov
dn: olcDatabase={0}config,cn=config changetype: modify replace: olcSyncRepl olcSyncRepl: rid=001 provider="ldap://vmlinux01" bindmethod=sasl saslmech="EXTERNAL" searchbase="cn=config" type=refreshAndPersist starttls=critical retry="5 5 60 +" timeout=1 tls_cacert=/usr/local/etc/openldap/cacerts/cacert.pem tls_cert=/usr/local/etc/openldap/slapd_repl_config.cert.pem tls_key=/usr/local/etc/openldap/slapd_repl_config.key.pem olcSyncRepl: rid=002 provider="ldap://vmlinux02" bindmethod=sasl saslmech="EXTERNAL" searchbase="cn=config" type=refreshAndPersist starttls=critical retry="5 5 60 +" timeout=1 tls_cacert=/usr/local/etc/openldap/cacerts/cacert.pem tls_cert=/usr/local/etc/openldap/slapd_repl_config.cert.pem tls_key=/usr/local/etc/openldap/slapd_repl_config.key.pem
add: olcMirrorMode olcMirrorMode: TRUE
add: olcLimits olcLimits: dn="cn=config" size=unlimited time=unlimited
******************* LDIF for activating syncrepl on data backend
dn: cn=config changetype: modify add: olcAuthzRegexp olcAuthzRegexp: "cn=.*_replicator,o=Htam.net Inc.,c=FR" cn=Replicator,ou=replicators,dc=htam,dc=net
dn: olcOverlay=syncprov,olcDatabase={2}hdb,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov
dn: olcDatabase={2}hdb,cn=config changetype: modify add: olcLimits olcLimits: dn.subtree="ou=replicators,dc=htam,dc=net" size=unlimited time=unlimited
add: olcSyncRepl olcSyncRepl: rid=201 provider="ldap://vmlinux01" bindmethod=sasl saslmech="EXTERNAL" searchbase="dc=htam,dc=net" type=refreshOnly interval=00:00:00:10 retry="5 5 300 +" timeout=1 starttls=critical tls_cacert=/usr/local/etc/openldap/cacerts/cacert.pem tls_cert=/usr/local/etc/openldap/slapd_replicator.cert tls_key=/usr/local/etc/openldap/slapd_replicator.key olcSyncRepl: rid=202 provider="ldap://vmlinux02" bindmethod=sasl saslmech="EXTERNAL" searchbase="dc=htam,dc=net" type=refreshOnly interval=00:00:00:10 retry="5 5 300 +" timeout=1 starttls=critical tls_cacert=/usr/local/etc/openldap/cacerts/cacert.pem tls_cert=/usr/local/etc/openldap/slapd_replicator.cert tls_key=/usr/local/etc/openldap/slapd_replicator.key
add: olcMirrorMode olcMirrorMode: TRUE
-- Mathieu MILLET mailto:ldap@htam.net