openldap-bugs

openldap-bugs@openldap.org

1 participants
20961 discussions

Re: (ITS#8384) LDAP Servers in Multi-master configuration locked up and unresponsive
by mhonek＠redhat.com 16 Mar '16

16 Mar '16

Here is the bug filed in Red Hat Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1317000 On 03/15/2016 03:33 PM, Michael Ströder wrote: > mhonek(a)redhat.com wrote: > > this seems to be a RHEL/CentOS-specific issue. Please contact Red Hat Support. > > I think this ITS ticket may be closed. > > How about adding a link to Red Hat's issue tracker in this ticket? > > Ciao, Michael. > -- Matúš Honěk

1 0

Re: (ITS#8385) Use After Free of struct ldap_common in slap_client_connect
by mkp37215＠gmail.com 15 Mar '16

15 Mar '16

On Mon, Mar 14, 2016 at 8:04 PM, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > > The Debian/Ubuntu maintainers are quite aware of the problems with using > GnuTLS, and maintain they cannot use OpenSSL due to licensing issues, > despite the fact every other major distribution uses OpenSSL (except RHEL, > which switched to MozNSS which resulted in much disaster, and is likely > going to switch back to OpenSSL). > > There is no reason you cannot use OpenSSL, regardless of what > Debian/Ubuntu's broken decisions are. > > --Quanah Quanah, thank you for your reply. However, I do not want to engage in a political discussion, particularly on the matter that is beyond my control. It is also out of topic, as the bug discussed here was in libldap code, and not in GnuTLS. I would like to again thank Howard for his work on this bug and coming up with a working fix very quickly. Unless any new issue related to this bug or the fix comes to light, I think we can consider the matter closed. Maciej Puzio

1 0

Re: (ITS#8384) LDAP Servers in Multi-master configuration locked up and unresponsive
by michael＠stroeder.com 15 Mar '16

15 Mar '16

mhonek(a)redhat.com wrote: > this seems to be a RHEL/CentOS-specific issue. Please contact Red Hat Support. > I think this ITS ticket may be closed. How about adding a link to Red Hat's issue tracker in this ticket? Ciao, Michael.

1 0

(ITS#8387) online olcDbConfig change fails with syncprov
by ryan＠openldap.org 15 Mar '16

15 Mar '16

Full_Name: Ryan Tandy Version: 2.4, master OS: Debian URL: Submission from: (NULL) (24.68.37.4) Submitted by: ryan Forwarding from a Debian bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816294 Configure a BDB or HDB database with syncprov: dn: olcDatabase={1}hdb,cn=config objectClass: olcHdbConfig olcDbDirectory: data olcSuffix: dc=example,dc=comD%D dn: olcOverlay={0}syncprov,olcDatabase={1}hdb,cn=config objectClass: olcSyncProvConfig perform some kind of modification to the database (so that a syncprov checkpoint is pending), and perform an online olcDbConfig change that reopens the database. dn: olcDatabase={1}hdb,cn=config changetype: modify replace: olcDbConfig olcDbConfig: set_cachesize 1 0 1 Reopening the database fails: 56e76e17 bdb(dc=example,dc=com): BDB4511 Error: closing the transaction region with active transactions 56e76e17 bdb_db_close: database "dc=example,dc=com": close failed: Invalid argument (22) and slapd crashes shortly after, when it tries to syncprov_checkpoint while the database is already gone. What appears to be happening is that "ctx" is different between bdb_reader_get and bdb_reader_flush in this case. During a normal slapd startup and shutdown: (gdb) thread apply all frame Thread 1 (Thread 0x7ffff7fed700 (LWP 21570)): #0 hdb_reader_get (op=0x7fffffffd8d0, env=0xa93fa0, txn=0x7fffffffd610) at cache.c:1666 1666 if ( !ctx ) { (gdb) p ctx $1 = (void *) 0x8b34a0 <ldap_int_main_thrctx> [ ... killall slapd ... ] (gdb) thread apply all frame Thread 1 (Thread 0x7ffff7fed700 (LWP 21570)): #0 hdb_reader_flush (env=0xa93fa0) at cache.c:1643 1643 if ( !ldap_pvt_thread_pool_getkey( ctx, env, &data, NULL ) ) { (gdb) p ctx $2 = (void *) 0x8b34a0 <ldap_int_main_thrctx> In this case, the readers are cleared correctly. Another startup, this time the hdb_db_close is triggered by performing an olcDbConfig change: (gdb) thread apply all frame Thread 1 (Thread 0x7ffff7fed700 (LWP 21624)): #0 hdb_reader_get (op=0x7fffffffd8d0, env=0xa93fa0, txn=0x7fffffffd610) at cache.c:1666 1666 if ( !ctx ) { (gd2929 p ctx $1 = (void *) 0x8b34a0 <ldap_int_main_thrctx> [ ... ldapmodify ... ] (gdb) thread apply all frame Thread 3 (Thread 0x7ffff362e700 (LWP 21633)): #0 hdb_reader_flush (env=0xa93fa0) at cache.c:1643 1643 if ( !ldap_pvt_thread_pool_getkey( ctx, env, &data, NULL ) ) { Thread 2 (Thread 0x7ffff3e2f700 (LWP 21631)): #0 0x00007ffff732d4d3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84 84 ../sysdeps/unix/syscall-template.S: No such file or directory. Thread 1 (Thread 0x7ffff7fed700 (LWP 21624)): #0 0x00007ffff75f06dd in pthread_join (threadid=140737285125888, thread_return=0x0) at pthread_join.c:90 90 pthread_join.c: No such file or directory. (gdb) p ctx $2 = (void *) 0x7ffff362dbf0 This time we have a different ctx, so the readers are not cleared. This when we get to db->close there is still an active txn. The comment "free up any keys used by the main thread" seems to assume bdb_reader_flush will be called on the main thread only.

1 0

Re: (ITS#8385) Use After Free of struct ldap_common in slap_client_connect
by quanah＠zimbra.com 15 Mar '16

15 Mar '16

--On Tuesday, March 15, 2016 1:34 AM +0000 mkp37215(a)gmail.com wrote: > --089e0139fc98189d9d052e0b902f > Content-Type: text/plain; charset=UTF-8 > > Howard, thank you very much for quickly fixing this issue. My tests show > that the replication is now working fine. > Regarding your recommendation of OpenSSL, that should rather go to > OpenLDAP package maintainers at Ubuntu and Debian. I use what they build > and the choice of the TLS library is not mine. The Debian/Ubuntu maintainers are quite aware of the problems with using GnuTLS, and maintain they cannot use OpenSSL due to licensing issues, despite the fact every other major distribution uses OpenSSL (except RHEL, which switched to MozNSS which resulted in much disaster, and is likely going to switch back to OpenSSL). There is no reason you cannot use OpenSSL, regardless of what Debian/Ubuntu's broken decisions are. --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc

1 0

Re: (ITS#8385) Use After Free of struct ldap_common in slap_client_connect
by mkp37215＠gmail.com 15 Mar '16

15 Mar '16

--089e0139fc98189d9d052e0b902f Content-Type: text/plain; charset=UTF-8 Howard, thank you very much for quickly fixing this issue. My tests show that the replication is now working fine. Regarding your recommendation of OpenSSL, that should rather go to OpenLDAP package maintainers at Ubuntu and Debian. I use what they build and the choice of the TLS library is not mine. --089e0139fc98189d9d052e0b902f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Howard, thank you very much for quickly fixing this i= ssue. My tests show that the replication is now working fine.<br></div>Rega= rding your recommendation of OpenSSL, that should rather go to OpenLDAP pac= kage maintainers at Ubuntu and Debian. I use what they build and the choice= of the TLS library is not mine.<br></div> --089e0139fc98189d9d052e0b902f--

1 0

(ITS#8386) Small LMDB Documentation patch
by philipp.storz＠bareos.com 14 Mar '16

14 Mar '16

Full_Name: Philipp Storz Version: LMDB 0.9.18 OS: linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (94.79.184.234) >From ede152281433d1e0bdc12741923c715360c4664a Mon Sep 17 00:00:00 2001 From: Philipp Storz <philipp.storz(a)bareos.com> Date: Mon, 14 Mar 2016 10:07:35 +0100 Subject: [PATCH] be a bit more precise that mdb_get retrieves data --- libraries/liblmdb/intro.doc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libraries/liblmdb/intro.doc b/libraries/liblmdb/intro.doc index 9fe9c22..2886c20 100644 --- a/libraries/liblmdb/intro.doc +++ b/libraries/liblmdb/intro.doc @@ -51,7 +51,7 @@ databases should only be opened once, by the first transaction in the process. After the first transaction completes, the database handles can freely be used by all subsequent transactions. -Within a transaction, #mdb_get() and #mdb_put(% c can store single +Within a transaction, #mdb_get() can retrieve and #mdb_put() can store single key/value pairs if that is all you need to do (but see \ref Cursors below if you want to do more). -- 2.6.2

1 0

Re: (ITS#8385) Use After Free of struct ldap_common in slap_client_connect
by hyc＠symas.com 12 Mar '16

12 Mar '16

> Thanks for the report. Inspection shows that the issue only exists in > libldap's GnuTLS support. As always, the Project recommends you use OpenSSL. > > It also looks like only the ldo_tls_require_cert field is being used so we can > probably just copy that flag instead of keeping a pointer to the ldap options > structure. > Fixed now in git master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8385) Use After Free of struct ldap_common in slap_client_connect
by hyc＠symas.com 12 Mar '16

12 Mar '16

mkp37215(a)gmail.com wrote: > Full_Name: Maciej Puzio > Version: 2.4.44 and current sources from git repo > OS: Ubuntu 15.10 amd64 > URL: > Submission from: (NULL) (129.112.109.41) > > > While configuring two servers in a dual-master setup I encountered an issue > causing replication failures, when replication is done on a TLS connection. In > this particular setup the issue manifested itself by replication retries > failing, even when the cause of the original failure no longer existed. For > example, let's consider that dual-master setup consists of servers A and B. If > server A was started before server B, its on-startup replication would fail (as > expected, since B is down). Server B would then be started and its replication > attempt would succeed (since A is already running). However A's subsequent > retries would fail, even though B was now accessible. If I then restarted server > A, its replication will succeed, but B will now lose connection to A and all B's > subsequent retries would fail. This issue has ~50% reproducibility, but if it > occurs once, then it would continue failing. On occasion both servers would > replicate fine, but obviously the whole situation was unacceptable on servers > being prepared for production use. > > I tested this issue thoroughly eliminating various hypotheses, and ended up > debugging OpenLDAP code and finding a bug causing the above behavior. In short, > on failure slap_client_connect deallocates LDAP object ld and objects pointed by > it, including ldap_common which contains ld_options. Pointer to ld_options is > held in sb->sb_tls_ctx and reused in subsequent slap_client_connect calls, at > this point referencing freed memory. > > This issue occurs in OpenLDAP newest stable version 2.4.44, as well as in > current source code from git repo (as of March 11, 2016), obtained from > git://git.openldap.org/openldap.git > Here, instead of freeing ld->ldc (struct ldap_common), I overwrite its contents > with a magic debug value 0xdeadbeef. I overwrite ldo_tls_require_cert, as an > incorrect value of this particular variable turned out to be the direct cause of > replication failures mentioned at the beginning (more about it later). However, > other fields of ldap_common could be used as well. > > File ldap-int.h is included to be able to examine contents of *ld, while the > declaration of struct tlsg_ctx has been copied verbatim from > libraries/libldap/tls_g.c, in order to examine the contents of > ld->ld_options.ldo_tls_ctx. I use GnuTLS library following Ubuntu build options. > But the issue in question is not GnuTLS-related, though it may cause different > symptoms with OpenSSL or Mozilla NSS (I did not test these options). Thanks for the report. Inspection shows that the issue only exists in libldap's GnuTLS support. As always, the Project recommends you use OpenSSL. It also looks like only the ldo_tls_require_cert field is being used so we can probably just copy that flag instead of keeping a pointer to the ldap options structure. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8385) Use After Free of struct ldap_common in slap_client_connect
by mkp37215＠gmail.com 12 Mar '16

12 Mar '16

Full_Name: Maciej Puzio Version: 2.4.44 and current sources from git repo OS: Ubuntu 15.10 amd64 URL: Submission from: (NULL) (129.112.109.41) While configuring two servers in a dual-master setup I encountered an issue causing replication failures, when replication is done on a TLS connection. In this particular setup the issue manifested itself by replication retries failing, even when the cause of the original failure no longer existed. For example, let's consider that dual-master setup consists of servers A and B. If server A was started before server B, its on-startup replication would fail (as expected, since B is down). Server B would then be started and its replication attempt would succeed (since A is already running). However A's subsequent retries would fail, even though B was now accessible. If I then restarted server A, its replication will succeed, but B will now lose connection to A and all B's subsequent retries would fail. This issue has ~50% reproducibility, but if it occurs once, then it would continue failing. On occasion both servers would replicate fine, but obviously the whole situation was unacceptable on servers being prepared for production use. I tested this issue thoroughly eliminating various hypotheses, and ended up debugging OpenLDAP code and finding a bug causing the above behavior. In short, on failure slap_client_connect deallocates LDAP object ld and objects pointed by it, including ldap_common which contains ld_options. Pointer to ld_options is held in sb->sb_tls_ctx and reused in subsequent slap_client_connect calls, at this point referencing freed memory. This issue occurs in OpenLDAP newest stable version 2.4.44, as well as in current source code from git repo (as of March 11, 2016), obtained from git://git.openldap.org/openldap.git In order to demonstrate that the issue is indeed use-after-free, I made the following debug modifications to the code. (The diffs are against the source code from git repo as of 2016/03/11) --- servers/slapd/config.c.orig 2016-03-11 14:52:14.000000000 -0600 +++ servers/slapd/config.c 2016-03-11 17:20:55.216170535 -0600 @@ -60,6 +60,26 @@ #define ARGS_STEP 512 + +/* DEBUG */ +#include <syslog.h> +#include <gnutls/gnutls.h> +#include "../../libraries/libldap/ldap-int.h" + +typedef struct tlsg_ctx { + struct ldapoptions *lo; + gnutls_certificate_credentials_t cred; + gnutls_dh_params_t dh_params; + unsigned long verify_depth; + int refcount; + gnutls_priority_t prios; +#ifdef LDAP_R_COMPILE + ldap_pvt_thread_mutex_t ref_mutex; +#endif +} tlsg_ctx; +/* END DEBUG */ + + /* * defaults for various global variables */ @@ -2001,9 +2021,11 @@ #ifdef HAVE_TLS if ( sb->sb_tls_do_init ) { + syslog(LOG_DEBUG, "DEBUG: slap_client_connect sb->sb_tls_do_init branch\n"); rc = bindconf_tls_set( sb, ld ); } else if ( sb->sb_tls_ctx ) { + syslog(LOG_DEBUG, "DEBUG: slap_client_connect sb->sb_tls_ctx branch\n"); rc = ldap_set_option( ld, LDAP_OPT_X_TLS_CTX, sb->sb_tls_ctx ); } @@ -2015,6 +2037,7 @@ sb->sb_uri.bv_val, rc, 0 ); goto done; } + syslog(LOG_DEBUG, "DEBUG: slap_client_connect ldo_tls_require_cert=%X\n", ((tlsg_x*x*)ld->ld_options.ldo_tls_ctx)->lo->ldo_tls_require_cert); #endif --- libraries/libldap/unbind.c.orig 2016-03-11 14:52:14.000000000 -0600 +++ libraries/libldap/unbind.c 2016-03-11 15:49:18.603447766 -0600 @@ -27,6 +27,8 @@ #include "ldap-int.h" +#include <syslog.h> /* DEBUG */ + /* An Unbind Request looks like this: * * UnbindRequest ::= [APPLICATION 2] NULL @@ -208,6 +210,8 @@ #ifdef HAVE_TLS ldap_int_tls_destroy( &ld->ld_options ); + syslog(LOG_DEBUG, "DEBUG: ldap_ld_destroy\n"); + ld->ld_options.ldo_tls_require_cert = 0xdeadbeef; /* DEBUG */ #endif if ( ld->ld_options.ldo_sctrls != NULL ) { @@ -233,7 +237,7 @@ #ifndef NDEBUG LDAP_TRASH(ld); #endif - LDAP_FREE( (char *) ld->ldc ); + //LDAP_FREE( (char *) ld->ldc ); /* DEBUG */ LDAP_FREE( (char *) ld ); return( err ); Here, instead of freeing ld->ldc (struct ldap_common), I overwrite its contents with a magic debug value 0xdeadbeef. I overwrite ldo_tls_require_cert, as an incorrect value of this particular variable turned out to be the direct cause of replication failures mentioned at the beginning (more about it later). However, other fields of ldap_common could be used as well. File ldap-int.h is included to be able to examine contents of *ld, while the declaration of struct tlsg_ctx has been copied verbatim from libraries/libldap/tls_g.c, in order to examine the contents of ld->ld_options.ldo_tls_ctx. I use GnuTLS library following Ubuntu build options. But the issue in question is not GnuTLS-related, though it may cause different symptoms with OpenSSL or Mozilla NSS (I did not test these options). One would expect that the only effect of these modifications would be a mere memory leak, but in addition to that we see that the magic value gets logged. As you can see, the second replication attempts uses memory that, if executing non-debug code, would be already freed: The results are as follows (only one server started): Mar 11 17:42:21 srvA slapd[14360]: slapd starting Mar 11 17:42:21 srvA slapd[14360]: =>do_syncrepl rid=001 Mar 11 17:42:21 srvA slapd[14360]: DEBUG: slap_client_connect sb->sb_tls_do_init branch Mar 11 17:42:21 srvA slapd[14355]: ...done. Mar 11 17:42:21 srvA slapd[14360]: DEBUG: slap_client_connect ldo_tls_require_cert=0 Mar 11 17:42:21 srvA slapd[14360]: DEBUG: ldap_ld_destroy Mar 11 17:42:21 srvA slapd[14360]: do_syncrepl: rid=001 rc -1 retrying (9 retries left) Mar 11 17:42:51 srvA slapd[14360]: =>do_syncrepl rid=001 Mar 11 17:42:51 srvA slapd[14360]: DEBUG: slap_client_connect sb->sb_tls_ctx branch Mar 11 17:42:51 srvA slapd[14360]: DEBUG: slap_client_connect ldo_tls_require_cert=DEADBEEF Mar 11 17:42:51 srvA slapd[14360]: DEBUG: ldap_ld_destroy Mar 11 17:42:51 srvA slapd[14360]: do_syncrepl: rid=001 rc -1 retrying (8 retries left) When running unmodified non-debug code, on replication retry ld_options contains either (A) previous values, (B) zeros or (C) random values. Occasionally a segfault occurs (although I observed them only during debugging). Replication fails much more readily if, in addition to the issue discussed here, there is a TLS certificate problem. In this case connection tends to fail in tlsg_cert_verify, even if tls_reqcert is set to 'allow' or 'never'. This is because random value of ldo_tls_require_cert will cause the certificate to be checked regardless of tls_reqcert setting. If certificates are correct, the issue may well remain hidden, as TLS connection will be established regardless of whether A, B or C occurs. In this case an occasional segfault may be the only symptom. Of course, if by chance B occurs, certificate verification will be silently skipped and this may lead to a security problem. A quick and dirty fix is to comment out few lines in servers/slapd/config.c so that "rc = bindconf_tls_set( sb, ld )" is always executed, which avoids the reuse of sb->sb_tls_ctx. However, I think a better fix would involve changing the handling of ld->ldoptions and related objects. Given the complexity of interactions between various objects involved, and my limited familiarity with OpenLDAP design, I feel that I am not in position to propose a patch. For reference, OpenLDAP was build with this configuration: ./configure --prefix=/opt/openldap --enable-debug --enable-dynamic --enable-syslog --enable-local --enable-slapd --enable-crypt --enable-modules --enable-backends --enable-ndb=no --enable-shell=no --enable-perl=no --enable-sql=no --enable-wt=no --enable-overlays --with-tls=gnutls CPPFLAGS="-Wno-format-extra-args" The relevant parts of slapd.conf are [redacted to anonymize]: loglevel 1 serverID 001 moduleload syncprov TLSCipherSuite SECURE256:-VERS-SSL3.0 TLSCACertificateFile /etc/ldap/ssl/ca.pem TLSCertificateFile /etc/ldap/ssl/srvA.pem TLSCertificateKeyFile /etc/ldap/ssl/srvA.key syncrepl rid=001 provider=ldaps://10.0.0.2 type=refreshAndPersist retry="30 10 300 +" searchbase="dc=test" attrs="*,+" bindmethod=simple binddn="cn=root,dc=test" credentials="plaintext-password" tls_reqcert=never keepalive="240:5:10" mirrormode TRUE overlay syncprov syncprov-checkpoint 10 1440 Test machine hardware: CPU: Intel Atom C2558 @ 2.40GHz Motherboard: Supermicro A1SRM-2558F NIC: Intel Corporation Ethernet Connection I354 (rev 03) Software used: OS: Ubuntu 15.10 Kernel: 4.2.0-30-generic #36-Ubuntu SMP x86_64 OpenLDAP: slapd 2.4.44 and slapd 2.X (from git repo) GnuTLS: 3.3.15-5ubuntu2 Please let me know if you need further information. Thank you. Maciej Puzio

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs