Re: (ITS#8385) Use After Free of struct ldap_common in slap_client_connect
by hyc＠symas.com
mkp37215(a)gmail.com wrote:
> Full_Name: Maciej Puzio
> Version: 2.4.44 and current sources from git repo
> OS: Ubuntu 15.10 amd64
> URL:
> Submission from: (NULL) (129.112.109.41)
>
>
> While configuring two servers in a dual-master setup I encountered an issue
> causing replication failures, when replication is done on a TLS connection. In
> this particular setup the issue manifested itself by replication retries
> failing, even when the cause of the original failure no longer existed. For
> example, let's consider that dual-master setup consists of servers A and B. If
> server A was started before server B, its on-startup replication would fail (as
> expected, since B is down). Server B would then be started and its replication
> attempt would succeed (since A is already running). However A's subsequent
> retries would fail, even though B was now accessible. If I then restarted server
> A, its replication will succeed, but B will now lose connection to A and all B's
> subsequent retries would fail. This issue has ~50% reproducibility, but if it
> occurs once, then it would continue failing. On occasion both servers would
> replicate fine, but obviously the whole situation was unacceptable on servers
> being prepared for production use.
>
> I tested this issue thoroughly eliminating various hypotheses, and ended up
> debugging OpenLDAP code and finding a bug causing the above behavior. In short,
> on failure slap_client_connect deallocates LDAP object ld and objects pointed by
> it, including ldap_common which contains ld_options. Pointer to ld_options is
> held in sb->sb_tls_ctx and reused in subsequent slap_client_connect calls, at
> this point referencing freed memory.
>
> This issue occurs in OpenLDAP newest stable version 2.4.44, as well as in
> current source code from git repo (as of March 11, 2016), obtained from
> git://git.openldap.org/openldap.git
> Here, instead of freeing ld->ldc (struct ldap_common), I overwrite its contents
> with a magic debug value 0xdeadbeef. I overwrite ldo_tls_require_cert, as an
> incorrect value of this particular variable turned out to be the direct cause of
> replication failures mentioned at the beginning (more about it later). However,
> other fields of ldap_common could be used as well.
>
> File ldap-int.h is included to be able to examine contents of *ld, while the
> declaration of struct tlsg_ctx has been copied verbatim from
> libraries/libldap/tls_g.c, in order to examine the contents of
> ld->ld_options.ldo_tls_ctx. I use GnuTLS library following Ubuntu build options.
> But the issue in question is not GnuTLS-related, though it may cause different
> symptoms with OpenSSL or Mozilla NSS (I did not test these options).
Thanks for the report. Inspection shows that the issue only exists in
libldap's GnuTLS support. As always, the Project recommends you use OpenSSL.
It also looks like only the ldo_tls_require_cert field is being used so we can
probably just copy that flag instead of keeping a pointer to the ldap options
structure.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
7 years, 6 months
(ITS#8385) Use After Free of struct ldap_common in slap_client_connect
by mkp37215＠gmail.com
Full_Name: Maciej Puzio
Version: 2.4.44 and current sources from git repo
OS: Ubuntu 15.10 amd64
URL:
Submission from: (NULL) (129.112.109.41)
While configuring two servers in a dual-master setup I encountered an issue
causing replication failures, when replication is done on a TLS connection. In
this particular setup the issue manifested itself by replication retries
failing, even when the cause of the original failure no longer existed. For
example, let's consider that dual-master setup consists of servers A and B. If
server A was started before server B, its on-startup replication would fail (as
expected, since B is down). Server B would then be started and its replication
attempt would succeed (since A is already running). However A's subsequent
retries would fail, even though B was now accessible. If I then restarted server
A, its replication will succeed, but B will now lose connection to A and all B's
subsequent retries would fail. This issue has ~50% reproducibility, but if it
occurs once, then it would continue failing. On occasion both servers would
replicate fine, but obviously the whole situation was unacceptable on servers
being prepared for production use.
I tested this issue thoroughly eliminating various hypotheses, and ended up
debugging OpenLDAP code and finding a bug causing the above behavior. In short,
on failure slap_client_connect deallocates LDAP object ld and objects pointed by
it, including ldap_common which contains ld_options. Pointer to ld_options is
held in sb->sb_tls_ctx and reused in subsequent slap_client_connect calls, at
this point referencing freed memory.
This issue occurs in OpenLDAP newest stable version 2.4.44, as well as in
current source code from git repo (as of March 11, 2016), obtained from
git://git.openldap.org/openldap.git
In order to demonstrate that the issue is indeed use-after-free, I made the
following debug modifications to the code. (The diffs are against the source
code from git repo as of 2016/03/11)
--- servers/slapd/config.c.orig 2016-03-11 14:52:14.000000000 -0600
+++ servers/slapd/config.c 2016-03-11 17:20:55.216170535 -0600
@@ -60,6 +60,26 @@
#define ARGS_STEP 512
+
+/* DEBUG */
+#include <syslog.h>
+#include <gnutls/gnutls.h>
+#include "../../libraries/libldap/ldap-int.h"
+
+typedef struct tlsg_ctx {
+ struct ldapoptions *lo;
+ gnutls_certificate_credentials_t cred;
+ gnutls_dh_params_t dh_params;
+ unsigned long verify_depth;
+ int refcount;
+ gnutls_priority_t prios;
+#ifdef LDAP_R_COMPILE
+ ldap_pvt_thread_mutex_t ref_mutex;
+#endif
+} tlsg_ctx;
+/* END DEBUG */
+
+
/*
* defaults for various global variables
*/
@@ -2001,9 +2021,11 @@
#ifdef HAVE_TLS
if ( sb->sb_tls_do_init ) {
+ syslog(LOG_DEBUG, "DEBUG: slap_client_connect sb->sb_tls_do_init
branch\n");
rc = bindconf_tls_set( sb, ld );
} else if ( sb->sb_tls_ctx ) {
+ syslog(LOG_DEBUG, "DEBUG: slap_client_connect sb->sb_tls_ctx
branch\n");
rc = ldap_set_option( ld, LDAP_OPT_X_TLS_CTX,
sb->sb_tls_ctx );
}
@@ -2015,6 +2037,7 @@
sb->sb_uri.bv_val, rc, 0 );
goto done;
}
+ syslog(LOG_DEBUG, "DEBUG: slap_client_connect
ldo_tls_require_cert=%X\n",
((tlsg_x*x*)ld->ld_options.ldo_tls_ctx)->lo->ldo_tls_require_cert);
#endif
--- libraries/libldap/unbind.c.orig 2016-03-11 14:52:14.000000000 -0600
+++ libraries/libldap/unbind.c 2016-03-11 15:49:18.603447766 -0600
@@ -27,6 +27,8 @@
#include "ldap-int.h"
+#include <syslog.h> /* DEBUG */
+
/* An Unbind Request looks like this:
*
* UnbindRequest ::= [APPLICATION 2] NULL
@@ -208,6 +210,8 @@
#ifdef HAVE_TLS
ldap_int_tls_destroy( &ld->ld_options );
+ syslog(LOG_DEBUG, "DEBUG: ldap_ld_destroy\n");
+ ld->ld_options.ldo_tls_require_cert = 0xdeadbeef; /* DEBUG */
#endif
if ( ld->ld_options.ldo_sctrls != NULL ) {
@@ -233,7 +237,7 @@
#ifndef NDEBUG
LDAP_TRASH(ld);
#endif
- LDAP_FREE( (char *) ld->ldc );
+ //LDAP_FREE( (char *) ld->ldc ); /* DEBUG */
LDAP_FREE( (char *) ld );
return( err );
Here, instead of freeing ld->ldc (struct ldap_common), I overwrite its contents
with a magic debug value 0xdeadbeef. I overwrite ldo_tls_require_cert, as an
incorrect value of this particular variable turned out to be the direct cause of
replication failures mentioned at the beginning (more about it later). However,
other fields of ldap_common could be used as well.
File ldap-int.h is included to be able to examine contents of *ld, while the
declaration of struct tlsg_ctx has been copied verbatim from
libraries/libldap/tls_g.c, in order to examine the contents of
ld->ld_options.ldo_tls_ctx. I use GnuTLS library following Ubuntu build options.
But the issue in question is not GnuTLS-related, though it may cause different
symptoms with OpenSSL or Mozilla NSS (I did not test these options).
One would expect that the only effect of these modifications would be a mere
memory leak, but in addition to that we see that the magic value gets logged. As
you can see, the second replication attempts uses memory that, if executing
non-debug code, would be already freed:
The results are as follows (only one server started):
Mar 11 17:42:21 srvA slapd[14360]: slapd starting
Mar 11 17:42:21 srvA slapd[14360]: =>do_syncrepl rid=001
Mar 11 17:42:21 srvA slapd[14360]: DEBUG: slap_client_connect sb->sb_tls_do_init
branch
Mar 11 17:42:21 srvA slapd[14355]: ...done.
Mar 11 17:42:21 srvA slapd[14360]: DEBUG: slap_client_connect
ldo_tls_require_cert=0
Mar 11 17:42:21 srvA slapd[14360]: DEBUG: ldap_ld_destroy
Mar 11 17:42:21 srvA slapd[14360]: do_syncrepl: rid=001 rc -1 retrying (9
retries left)
Mar 11 17:42:51 srvA slapd[14360]: =>do_syncrepl rid=001
Mar 11 17:42:51 srvA slapd[14360]: DEBUG: slap_client_connect sb->sb_tls_ctx
branch
Mar 11 17:42:51 srvA slapd[14360]: DEBUG: slap_client_connect
ldo_tls_require_cert=DEADBEEF
Mar 11 17:42:51 srvA slapd[14360]: DEBUG: ldap_ld_destroy
Mar 11 17:42:51 srvA slapd[14360]: do_syncrepl: rid=001 rc -1 retrying (8
retries left)
When running unmodified non-debug code, on replication retry ld_options contains
either (A) previous values, (B) zeros or (C) random values. Occasionally a
segfault occurs (although I observed them only during debugging). Replication
fails much more readily if, in addition to the issue discussed here, there is a
TLS certificate problem. In this case connection tends to fail in
tlsg_cert_verify, even if tls_reqcert is set to 'allow' or 'never'. This is
because random value of ldo_tls_require_cert will cause the certificate to be
checked regardless of tls_reqcert setting. If certificates are correct, the
issue may well remain hidden, as TLS connection will be established regardless
of whether A, B or C occurs. In this case an occasional segfault may be the only
symptom. Of course, if by chance B occurs, certificate verification will be
silently skipped and this may lead to a security problem.
A quick and dirty fix is to comment out few lines in servers/slapd/config.c so
that "rc = bindconf_tls_set( sb, ld )" is always executed, which avoids the
reuse of sb->sb_tls_ctx. However, I think a better fix would involve changing
the handling of ld->ldoptions and related objects. Given the complexity of
interactions between various objects involved, and my limited familiarity with
OpenLDAP design, I feel that I am not in position to propose a patch.
For reference, OpenLDAP was build with this configuration:
./configure --prefix=/opt/openldap --enable-debug --enable-dynamic
--enable-syslog --enable-local --enable-slapd --enable-crypt --enable-modules
--enable-backends --enable-ndb=no --enable-shell=no --enable-perl=no
--enable-sql=no --enable-wt=no --enable-overlays --with-tls=gnutls
CPPFLAGS="-Wno-format-extra-args"
The relevant parts of slapd.conf are [redacted to anonymize]:
loglevel 1
serverID 001
moduleload syncprov
TLSCipherSuite SECURE256:-VERS-SSL3.0
TLSCACertificateFile /etc/ldap/ssl/ca.pem
TLSCertificateFile /etc/ldap/ssl/srvA.pem
TLSCertificateKeyFile /etc/ldap/ssl/srvA.key
syncrepl rid=001
provider=ldaps://10.0.0.2
type=refreshAndPersist
retry="30 10 300 +"
searchbase="dc=test"
attrs="*,+"
bindmethod=simple
binddn="cn=root,dc=test"
credentials="plaintext-password"
tls_reqcert=never
keepalive="240:5:10"
mirrormode TRUE
overlay syncprov
syncprov-checkpoint 10 1440
Test machine hardware:
CPU: Intel Atom C2558 @ 2.40GHz
Motherboard: Supermicro A1SRM-2558F
NIC: Intel Corporation Ethernet Connection I354 (rev 03)
Software used:
OS: Ubuntu 15.10
Kernel: 4.2.0-30-generic #36-Ubuntu SMP x86_64
OpenLDAP: slapd 2.4.44 and slapd 2.X (from git repo)
GnuTLS: 3.3.15-5ubuntu2
Please let me know if you need further information. Thank you.
Maciej Puzio
7 years, 6 months
Re: (ITS#8384) LDAP Servers in Multi-master configuration locked up and unresponsive
by mhonek＠redhat.com
Hello,
this seems to be a RHEL/CentOS-specific issue. Please contact Red Hat Support.
I think this ITS ticket may be closed.
Regards.
On 03/11/2016 04:01 PM, kevin4sullivan(a)gmail.com wrote:
> Full_Name: Kevin Sullivan
> Version: 2.4.39-8
> OS: RHEL 6.4
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (38.127.164.138)
>
>
> Setup:
>
> I have two OpenLDAP servers running on separate hosts and replicating via delta
> syncrepl in mirrormode.
>
> Problem:
>
> On rare occasions when one of my servers comes online, both servers will lock up
> and be unresponsive. Any query to either server will time out. The servers will
> stay in this state indefinitely.
>
> Workaround:
>
> Killing either of the servers will resolve this problem.
>
> Observations:
>
> - I don't know how to reproduce this reliably.
> - Both servers have a thread that is stuck in a do_syncrep1() call.
> - Netstat shows that each server has unacknowledged data in one of their
> socket's receive queue.
>
>
> GDB on host1:
>
> (gdb) info threads
> 18 Thread 0x7f8b7b80d700 (LWP 31620) 0x00007f8ba7fa6f03 in epoll_wait () from
> /lib64/libc.so.6
> 17 Thread 0x7f8b7b00c700 (LWP 31621) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 16 Thread 0x7f8b7a80b700 (LWP 31622) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 15 Thread 0x7f8b7a00a700 (LWP 31623) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 14 Thread 0x7f8b79809700 (LWP 31625) 0x00007f8ba846a54d in read () from
> /lib64/libpthread.so.0
> 13 Thread 0x7f8b79008700 (LWP 31626) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 12 Thread 0x7f8b59ffd700 (LWP 25459) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 11 Thread 0x7f8b597fc700 (LWP 10195) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 10 Thread 0x7f8b70b37700 (LWP 28996) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 9 Thread 0x7f8b58ffb700 (LWP 8763) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 8 Thread 0x7f8b4bfff700 (LWP 28614) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 7 Thread 0x7f8b4b7fe700 (LWP 23367) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 6 Thread 0x7f8b4affd700 (LWP 13359) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 5 Thread 0x7f8b4a7fc700 (LWP 29064) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 4 Thread 0x7f8b49ffb700 (LWP 2184) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 3 Thread 0x7f8b497fa700 (LWP 31175) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 2 Thread 0x7f8b48ff9700 (LWP 2316) 0x00007f8ba846a054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> * 1 Thread 0x7f8baa8db700 (LWP 31615) 0x00007f8ba84640ad in pthread_join ()
> from /lib64/libpthread.so.0
> (gdb) bt
> #0 0x00007f8ba84640ad in pthread_join () from /lib64/libpthread.so.0
> #1 0x00007f8baa93c969 in slapd_daemon () at
> ../../%./servers/slapd/daemon.c:2929
> #2 0x00007f8baa927733 in main (argc=6, argv=<value optimized out>) at
> ../../../servers/slapd/main.c:1012
> (gdb) thread 2
> [Switching to thread 2 (Thread 0x7f8b48ff9700 (LWP 2316))]#0 0x00007f8ba846a054
> in __lll_lock_wait (929 from /lib64/libpthread.so.0
> (gdb) bt
> #0 0x00007f8ba846a054 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x00007f8ba8465388 in _L_lock_854 () from /lib64/libpthread.so.0
> #2 0x00007f8ba8465257 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3 0x00007f8baa4b96a0 in tlsm_session_accept_or_connect
> (session=0x7f8b2c104110, is_accept=<value optimized out>) at tls_m.c:2656
> #4 0x00007f8baa4b8787 in ldap_pvt_tls_accept (sb=0x7f8b2c104150,
> ctx_arg=0x7f8bac6f8230) at tls2.c:425
> #5 0x00007f8baa943f23 in connection_read (ctx=0x7f8b48ff8b70, argv=0x2f) at
> ../../../servers/slapd/connection.c:1372
> #6 connection_read_thread (ctx=0x7f8b48ff8b70, argv=0x2f) at
> ../../../servers/slapd/connection.c:1284
> #7 0x00007f8baa490a98 in ldap_int_thread_pool_wrapper (xpool=0x7f8bac5c8fb0) at
> ../../../libraries/libldap_r/tpool.c:688
> #8 0x00007f8ba8463851 in start_thread () from /lib64/libpthread.so.0
> #9 0x00007f8ba7fa690d in clone () from /lib64/libc.so.6
> (gdb) thread 14
> [Switching to thread 14 (Thread 0x7f8b79809700 (LWP 31625))]#0
> 0x00007f8ba846a54d in read () from /lib64/libpthread.so.0
> (gdb) bt
> #0 0x00007f8ba846a54d in read () from /lib64/libpthread.so.0
> #1 0x00007f8baa278b1c in sb_debug_read (sbiod=0x7f8b681def80,
> buf=0x7f8b6819c830, len=5) at ../../../libraries/liblber/sockbuf.c:829
> #2 0x00007f8baa4b9984 in tlsm_PR_Recv (fd=<value optimized out>,
> buf=0x7f8b6819c830, len=5, flags=<value optimized out>, timeout=<value optimized
> out>) at tls_m.c:3007
> #3 0x00007f8ba946c1ed in ?? () from /usr/lib64/libssl3.so
> #4 0000007f8ba9467480 in ?? () from /usr/lib64/libssl3.so
> #5 0x00007f8ba9469ed2 in ?? () from /usr/lib64/libssl3.so
> #6 0x00007f8ba9470135 in ?? () from /usr/lib64/libssl3.so
> #7 0x00007f8ba947196f in SSL_ForceHandshake () from /usr/lib64/libssl3o%o
> #8 0x00007f8baa4b96a8 in tlsm_session_accept_or_connect
> (session=0x7f8b681b0f20, is_accept=<value optimized out>) at tls_m.c:2658
> #9 0x00007f8baa4b82e2 in ldap_int_tls_connect (ld=0x7f8b681d9250, conn=<value
> optimized out>) at tls2.c:362
> #10 0x00007f8baa4b857inin ldap_int_tls_start (ld=0x7f8b681d9250,
> conn=0x7f8b681b0ba0, srv=<value optimized out>) at tls2.c:860
> #11 0x00007f8baa4b86ce in ldap_start_tls_s (ld=0x7f8b681d9250, serverctrls=0x0,
> clientctrls=0x0) at tls2.c:1040
> #12 0x00007f8baa937ff6 in slap_client_connect (ldp=0x7f8bac6d35a8,
> sb=0x7f8bac6d3380) at ../../../servers/slapd/config.c:2012
> #13 0x00007f8baa9ae3f5 in do_syncrep1 (ctx=<value optimized out>,
> arg=0x7f8bac6cc430) at ../../../servers/slapd/syncrepl.c:613
> #14 do_syncrepl (ctx=<value optimized out>, arg=0x7f8bac6cc430) at
> ../../../servers/slapd/syncrepl.c:1527
> #15 0x00007f8baa490a98 in ldap_int_thread_pool_wrapper (xpool=0x7f8bac5c8fb0) at
> ../../../libraries/libldap_r/tpool.c:688
> #16 0x00007f8ba8463851 in start_thread () from /lib64/libpthread.so.0
> #17 0x00007f8ba7fa690d in clone () from /lib64/libc.so.6
> (gdb) thread 18
> [Switching to thread 18 (Thread 0x7f8b7b80d700 (LWP 31620))]#0
> 0x00007f8ba7fa6f03 in epoll_wait () from /lib64/libc.so.6
> (gdb) bt
> #0 0x00007f8ba7fa6f03 in epoll_wait () from /lib64/libc.so.6
> #1 0x00007f8baa93d9e2 in slapd_daemon_task (ptr=<value optimized out>) at
> ../../../servers/slapd/daemon.c:2536
> #2 0x00007f8ba8463851 in start_thread () from /lib64/libpthread.so.0
> #3 0x00007f8ba7fa690d in clone () from /lib64/libc.so.6
>
>
> GDB on host2:
>
> (gdb) info threads
> 7 Thread 0x7f5c942d5700 (LWP 3869) 0x00007f5cc0797f03 in epoll_wait () from
> /lib64/libc.so.6
> 6 Thread 0x7f5c93ad4700 (LWP 3870) 0x00007f5cc0c5b54d in read () from
> /lib64/libpthread.so.0
> 5 Thread 0x7f5c932d3700 (LWP 3871) 0x00007f5cc0c5b054 in __lll_lock_wait ()
> from /lib64/libpthread.so.0
> 4 Thread 0x7f5c92ad2700 (LWP 3872) 0x00007f5cc0c5843c in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 3 Thread 0x7f5c922d1700 (LWP 3873) 0x00007f5cc0c5843c in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 2 Thread 0x7f5c91ad0700 (LWP 3874) 0x00007f5cc0c5843c in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> * 1 Thread 0x7f5cc30cc700 (LWP 3868) 0x00007f5cc0c550ad in pthread_join () from
> /lib64/libpthread.so.0
> (gdb) bt
> #0 0x00007f5cc0c550ad in pthread_join () from /lib64/libpthread.so.0
> #1 0x00007f5cc312d969 in slapd_daemon () at
> ../../../servers/slapd/daemon.c:2929
> #2 0x00007f5cc3118733 in main (argc=6, argv=<value optimized out>) at
> ../../../servers/slapd/main.c:1012
> (gdb) thread 2
> [Switching to thread 2 (Thread 0x7f5c91ad0700 (LWP 3874))]#0 0x00007f5cc0c5843c
> in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> (gdb) bt
> #0 0x00007f5cc0c5843c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007f5cc2c81af5 in ldap_int_thread_pool_wrapper (xpool=0x7f5cc3d18030) at
> ../../../libraries/libldap_r/tpool.c:675
> #2 0x00007f5cc0c54851 in start_thread () from /lib64/libpthread.so.0
> #3 0x00007f5cc079790d in clone () from /lib64/libc.so.6
> (gdb) thread 5
> [Switching to thread 5 (Thread 0x7f5c932d3700 (LWP 3871))]#0 0x00007f5cc0c5b054
> in __lll_lock_wait () from /lib64/libpthread.so.0
> (gdb) bt
> #0 0x00007f5cc0c5b054 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x00007f5cc0c56388 in _L_lock_854 () from /lib64/libpthread.so.0
> #2 0x00007f5cc0c56257 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3 0x00007f5cc2caa6a0 in tlsm_session_accept_or_connect
> (session=0x7f5c841165c0, is_accept=<value optimized out>) at tls_m.c:2656
> #4 0x00007f5cc2ca9787 in ldap_pvt_tls_accept (sb=0x7f5c800008c0,
> ctx_arg=0x7f5cc3e3b240) at tls2.c:425
> #5 0x00007f5cc3134f23 in connection_read (ctx=0x7f5c932d2b70, argv=0x10) at
> ../../../servers/slapd/connection.c:1372
> #6 connection_read_thread (ctx=0x7f5c932d2b70, argv=0x10) at
> ../../../servers/slapd/connection.c:1284
> #7 0x00007f5cc2c81a98 in ldap_int_thread_pool_wrapper (xpool=0x7f5cc3d18030) at
> ../../../libraries/libldap_r%tptpool.c:688
> #8 0x00007f5cc0c54851 in start_thread () from /lib64/libpthread.so.0
> #9 0x00007f5cc079790d in clone () from /lib64/libc.so.6
> (gdb) thread 6
> [Switching to thread 6 (Thread 0x7f5c93ad4700 (LWP 3870))]#0 0x00007f5cc0c5b54d
> in read () from /lib64/libpthread.so.0
> (gdb) bt
> #0 0x00007f5cc0c5b54d in read () from /lib64/libpthread.so.0
> #1 0x00007f5cc2a69b1c in sb_debug_read (sbiod=0x7f5c7c185e10,
> buf=0x7f5c7c18a7f0, len=5) at ../../../libraries/liblber/sockbuf.c:829
> #2 0x00007f5cc2caa984 in tlsm_PR_Recv (fd=<value optimized out>,
> buf=0x7f5c7c18a7f0, len=5, flags=<value optimized out>, timeout=<value optimized
> out>) at tls_m.c:3007
> #3 0x00007f5cc1c5d1ed in ?? () from /usr/lib64/libssl3.so
> #4 0x00007f5cc1c58480 in ?? () from /usr/lib64/libssl3.so
> #5 0x00007f5cc1c5aed2 in ?? () from /usr/lib64/libssl3.so
> #6 0x00007f5cc1c61135 in ?? () from /usr/lib64/libssl3.so
> #7 0x00007f5cc1c6296f in SSL_ForceHandshake () from /usr/lib64/libssl3.so
> #8 0x00007f5cc2caa6a8 in tlsm_session_accept_or_connect
> (session=0x7f5c7c182100, is_accept=<value optimized out>) at tls_m.c:2658
> #9 0x00007f5cc2ca92e2 in ldap_int_tls_connect (ld=0x7f5c7c100910, conn=<value
> optimized out>) at tls2.c:362
> #10 0x00007f5cc2ca9574 in ldap_int_tls_start (ld=0x7f5c7c100910,
> conn=0x7f5c7c10a020, srv=<value optimized out>) at tls2.c:860
> #11 0x00007f5cc2ca96ce in ldap_start_tls_s (ld=0x7f5c7c100910, serverctrls=0x0,
> clientctrls=0x0) at tls2.c:1040
> #12 0x00007f5cc3128ff6 in slap_client_connect (ldp=0x7f5cc3e22558,
> sb=0x7f5cc3e22330) at ../../../servers/slapd/config.c:2012
> #13 0x00007f5cc319f3f5 in do_syncrep1 (ctx=<value optimized out>,
> arg=0x7f5cc3e1f410) at ../../../servers/slapd/syncrepl.c:613
> #14 do_syncrepl (ctx=<value optimized out>, arg=0x7f5cc3e1f410) at
> ../../../servers/slapd/syncrepl.c:1527
> #15 0x00007f5cc2c81a98 in ldap_int_thread_pool_wrapper (xpool=0x7f5cc3d18030) at
> ../../../libraries/libldap_r/tpool.c:688
> #16 0x00007f5cc0c54851 in start_thread () from /lib64/libpthread.so.0
> #17 0x00007f5cc079790d in clone () from /lib64/libc.so.6
> (gdb) thread 7
> [Switching to thread 7 (Thread 0x7f5c942d5700 (LWP 3869))]#0 0x00007f5cc0797f03
> in epoll_wait () from /lib64/libc.so.6
> (gdb) bt
> #0 0x00007f5cc0797f03 in epoll_wait () from /lib64/libc.so.6
> #1 0x00007f5cc312e9e2 in slapd_daemon_task (ptr=<value optimized out>) at
> ../../../servers/slapd/daemon.c:2536
> #2 0x00007f5cc0c54851 in start_thread () from /lib64/libpthread.so.0
> #3 0x00007f5cc079790d in clone () from /lib64/libc.so.6
>
>
> Relevant configuration information (from host2):
>
> syncrepl rid=001
> provider=ldap://host1/
> type=refreshAndPersist
> retry="10 +"
> searchbase="dc=example,dc=com"
> starttls=critical
> tls_cacert=/etc/openldap/ssl-certs/cacert.pem
> tls_cert=/etc/openldap/ssl-certs/host1.crt
> tls_key=/etc/openldap/ssl-certs/host1.key
> tls_reqcert=demand
> bindmethod=sasl
> saslmech=EXTERNAL
> sizelimit=unlimited
> keepalive="10:2:10"
> logfilter="(&(objectClass=auditWriteObject)(reqResult=0))"
> logbase="cn=accesslog"
> syncdata=accesslog
>
>
> Netstat of host1:
>
> [root@host1 ~]# netstat -anp | grep slapd
> tcp 0 0 0.0.0.0:389 0.0.0.0:*
> LISTEN 31615/slapd
> tcp 56 0 192.168.1.1:389 192.168.1.2:57276
> ESTABLISHED 31615/slapd
> tcp 0 0 192.168.1.1:45569 192.168.1.2:389
> ESTABLISHED 31615/slapd
> unix 2 [ ] STREAM CONNECTED 45346152 31615/slapd
> unix 2 [ ] DGRAM 45346144 31615/slapd
>
> Netstat of host2:
>
> [root@host2 ~]# netstat -anp | grep slapd
> tcp 0 0 0.0.0.0:389 0.0.0.0:*
> LISTEN 3868/slapd
> tcp 56 0 192.168.1.2:389 192.168.1.1:45569
> ESTABLISHED 3868/slapd
> tcp 0 0 192.168.1.2:57276 192.168.1.1:389
> ESTABLISHED 3868/slapd
> unix 2 [ ] STREAM CONNECTED 19144 3868/slapd
> unix [ ] DGRAM 19136 3868/slapd
>
>
--
Matúš Honěk
7 years, 6 months
(ITS#8384) LDAP Servers in Multi-master configuration locked up and unresponsive
by kevin4sullivan＠gmail.com
Full_Name: Kevin Sullivan
Version: 2.4.39-8
OS: RHEL 6.4
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (38.127.164.138)
Setup:
I have two OpenLDAP servers running on separate hosts and replicating via delta
syncrepl in mirrormode.
Problem:
On rare occasions when one of my servers comes online, both servers will lock up
and be unresponsive. Any query to either server will time out. The servers will
stay in this state indefinitely.
Workaround:
Killing either of the servers will resolve this problem.
Observations:
- I don't know how to reproduce this reliably.
- Both servers have a thread that is stuck in a do_syncrep1() call.
- Netstat shows that each server has unacknowledged data in one of their
socket's receive queue.
GDB on host1:
(gdb) info threads
18 Thread 0x7f8b7b80d700 (LWP 31620) 0x00007f8ba7fa6f03 in epoll_wait () from
/lib64/libc.so.6
17 Thread 0x7f8b7b00c700 (LWP 31621) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
16 Thread 0x7f8b7a80b700 (LWP 31622) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
15 Thread 0x7f8b7a00a700 (LWP 31623) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
14 Thread 0x7f8b79809700 (LWP 31625) 0x00007f8ba846a54d in read () from
/lib64/libpthread.so.0
13 Thread 0x7f8b79008700 (LWP 31626) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
12 Thread 0x7f8b59ffd700 (LWP 25459) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
11 Thread 0x7f8b597fc700 (LWP 10195) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
10 Thread 0x7f8b70b37700 (LWP 28996) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
9 Thread 0x7f8b58ffb700 (LWP 8763) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
8 Thread 0x7f8b4bfff700 (LWP 28614) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
7 Thread 0x7f8b4b7fe700 (LWP 23367) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
6 Thread 0x7f8b4affd700 (LWP 13359) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
5 Thread 0x7f8b4a7fc700 (LWP 29064) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
4 Thread 0x7f8b49ffb700 (LWP 2184) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
3 Thread 0x7f8b497fa700 (LWP 31175) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
2 Thread 0x7f8b48ff9700 (LWP 2316) 0x00007f8ba846a054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
* 1 Thread 0x7f8baa8db700 (LWP 31615) 0x00007f8ba84640ad in pthread_join ()
from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007f8ba84640ad in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f8baa93c969 in slapd_daemon () at
../../%./servers/slapd/daemon.c:2929
#2 0x00007f8baa927733 in main (argc=6, argv=<value optimized out>) at
../../../servers/slapd/main.c:1012
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f8b48ff9700 (LWP 2316))]#0 0x00007f8ba846a054
in __lll_lock_wait (929 from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007f8ba846a054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f8ba8465388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f8ba8465257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f8baa4b96a0 in tlsm_session_accept_or_connect
(session=0x7f8b2c104110, is_accept=<value optimized out>) at tls_m.c:2656
#4 0x00007f8baa4b8787 in ldap_pvt_tls_accept (sb=0x7f8b2c104150,
ctx_arg=0x7f8bac6f8230) at tls2.c:425
#5 0x00007f8baa943f23 in connection_read (ctx=0x7f8b48ff8b70, argv=0x2f) at
../../../servers/slapd/connection.c:1372
#6 connection_read_thread (ctx=0x7f8b48ff8b70, argv=0x2f) at
../../../servers/slapd/connection.c:1284
#7 0x00007f8baa490a98 in ldap_int_thread_pool_wrapper (xpool=0x7f8bac5c8fb0) at
../../../libraries/libldap_r/tpool.c:688
#8 0x00007f8ba8463851 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f8ba7fa690d in clone () from /lib64/libc.so.6
(gdb) thread 14
[Switching to thread 14 (Thread 0x7f8b79809700 (LWP 31625))]#0
0x00007f8ba846a54d in read () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007f8ba846a54d in read () from /lib64/libpthread.so.0
#1 0x00007f8baa278b1c in sb_debug_read (sbiod=0x7f8b681def80,
buf=0x7f8b6819c830, len=5) at ../../../libraries/liblber/sockbuf.c:829
#2 0x00007f8baa4b9984 in tlsm_PR_Recv (fd=<value optimized out>,
buf=0x7f8b6819c830, len=5, flags=<value optimized out>, timeout=<value optimized
out>) at tls_m.c:3007
#3 0x00007f8ba946c1ed in ?? () from /usr/lib64/libssl3.so
#4 0000007f8ba9467480 in ?? () from /usr/lib64/libssl3.so
#5 0x00007f8ba9469ed2 in ?? () from /usr/lib64/libssl3.so
#6 0x00007f8ba9470135 in ?? () from /usr/lib64/libssl3.so
#7 0x00007f8ba947196f in SSL_ForceHandshake () from /usr/lib64/libssl3o%o
#8 0x00007f8baa4b96a8 in tlsm_session_accept_or_connect
(session=0x7f8b681b0f20, is_accept=<value optimized out>) at tls_m.c:2658
#9 0x00007f8baa4b82e2 in ldap_int_tls_connect (ld=0x7f8b681d9250, conn=<value
optimized out>) at tls2.c:362
#10 0x00007f8baa4b857inin ldap_int_tls_start (ld=0x7f8b681d9250,
conn=0x7f8b681b0ba0, srv=<value optimized out>) at tls2.c:860
#11 0x00007f8baa4b86ce in ldap_start_tls_s (ld=0x7f8b681d9250, serverctrls=0x0,
clientctrls=0x0) at tls2.c:1040
#12 0x00007f8baa937ff6 in slap_client_connect (ldp=0x7f8bac6d35a8,
sb=0x7f8bac6d3380) at ../../../servers/slapd/config.c:2012
#13 0x00007f8baa9ae3f5 in do_syncrep1 (ctx=<value optimized out>,
arg=0x7f8bac6cc430) at ../../../servers/slapd/syncrepl.c:613
#14 do_syncrepl (ctx=<value optimized out>, arg=0x7f8bac6cc430) at
../../../servers/slapd/syncrepl.c:1527
#15 0x00007f8baa490a98 in ldap_int_thread_pool_wrapper (xpool=0x7f8bac5c8fb0) at
../../../libraries/libldap_r/tpool.c:688
#16 0x00007f8ba8463851 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f8ba7fa690d in clone () from /lib64/libc.so.6
(gdb) thread 18
[Switching to thread 18 (Thread 0x7f8b7b80d700 (LWP 31620))]#0
0x00007f8ba7fa6f03 in epoll_wait () from /lib64/libc.so.6
(gdb) bt
#0 0x00007f8ba7fa6f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f8baa93d9e2 in slapd_daemon_task (ptr=<value optimized out>) at
../../../servers/slapd/daemon.c:2536
#2 0x00007f8ba8463851 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f8ba7fa690d in clone () from /lib64/libc.so.6
GDB on host2:
(gdb) info threads
7 Thread 0x7f5c942d5700 (LWP 3869) 0x00007f5cc0797f03 in epoll_wait () from
/lib64/libc.so.6
6 Thread 0x7f5c93ad4700 (LWP 3870) 0x00007f5cc0c5b54d in read () from
/lib64/libpthread.so.0
5 Thread 0x7f5c932d3700 (LWP 3871) 0x00007f5cc0c5b054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
4 Thread 0x7f5c92ad2700 (LWP 3872) 0x00007f5cc0c5843c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
3 Thread 0x7f5c922d1700 (LWP 3873) 0x00007f5cc0c5843c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
2 Thread 0x7f5c91ad0700 (LWP 3874) 0x00007f5cc0c5843c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 1 Thread 0x7f5cc30cc700 (LWP 3868) 0x00007f5cc0c550ad in pthread_join () from
/lib64/libpthread.so.0
(gdb) bt
#0 0x00007f5cc0c550ad in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f5cc312d969 in slapd_daemon () at
../../../servers/slapd/daemon.c:2929
#2 0x00007f5cc3118733 in main (argc=6, argv=<value optimized out>) at
../../../servers/slapd/main.c:1012
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f5c91ad0700 (LWP 3874))]#0 0x00007f5cc0c5843c
in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007f5cc0c5843c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007f5cc2c81af5 in ldap_int_thread_pool_wrapper (xpool=0x7f5cc3d18030) at
../../../libraries/libldap_r/tpool.c:675
#2 0x00007f5cc0c54851 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f5cc079790d in clone () from /lib64/libc.so.6
(gdb) thread 5
[Switching to thread 5 (Thread 0x7f5c932d3700 (LWP 3871))]#0 0x00007f5cc0c5b054
in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007f5cc0c5b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f5cc0c56388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f5cc0c56257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f5cc2caa6a0 in tlsm_session_accept_or_connect
(session=0x7f5c841165c0, is_accept=<value optimized out>) at tls_m.c:2656
#4 0x00007f5cc2ca9787 in ldap_pvt_tls_accept (sb=0x7f5c800008c0,
ctx_arg=0x7f5cc3e3b240) at tls2.c:425
#5 0x00007f5cc3134f23 in connection_read (ctx=0x7f5c932d2b70, argv=0x10) at
../../../servers/slapd/connection.c:1372
#6 connection_read_thread (ctx=0x7f5c932d2b70, argv=0x10) at
../../../servers/slapd/connection.c:1284
#7 0x00007f5cc2c81a98 in ldap_int_thread_pool_wrapper (xpool=0x7f5cc3d18030) at
../../../libraries/libldap_r%tptpool.c:688
#8 0x00007f5cc0c54851 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f5cc079790d in clone () from /lib64/libc.so.6
(gdb) thread 6
[Switching to thread 6 (Thread 0x7f5c93ad4700 (LWP 3870))]#0 0x00007f5cc0c5b54d
in read () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007f5cc0c5b54d in read () from /lib64/libpthread.so.0
#1 0x00007f5cc2a69b1c in sb_debug_read (sbiod=0x7f5c7c185e10,
buf=0x7f5c7c18a7f0, len=5) at ../../../libraries/liblber/sockbuf.c:829
#2 0x00007f5cc2caa984 in tlsm_PR_Recv (fd=<value optimized out>,
buf=0x7f5c7c18a7f0, len=5, flags=<value optimized out>, timeout=<value optimized
out>) at tls_m.c:3007
#3 0x00007f5cc1c5d1ed in ?? () from /usr/lib64/libssl3.so
#4 0x00007f5cc1c58480 in ?? () from /usr/lib64/libssl3.so
#5 0x00007f5cc1c5aed2 in ?? () from /usr/lib64/libssl3.so
#6 0x00007f5cc1c61135 in ?? () from /usr/lib64/libssl3.so
#7 0x00007f5cc1c6296f in SSL_ForceHandshake () from /usr/lib64/libssl3.so
#8 0x00007f5cc2caa6a8 in tlsm_session_accept_or_connect
(session=0x7f5c7c182100, is_accept=<value optimized out>) at tls_m.c:2658
#9 0x00007f5cc2ca92e2 in ldap_int_tls_connect (ld=0x7f5c7c100910, conn=<value
optimized out>) at tls2.c:362
#10 0x00007f5cc2ca9574 in ldap_int_tls_start (ld=0x7f5c7c100910,
conn=0x7f5c7c10a020, srv=<value optimized out>) at tls2.c:860
#11 0x00007f5cc2ca96ce in ldap_start_tls_s (ld=0x7f5c7c100910, serverctrls=0x0,
clientctrls=0x0) at tls2.c:1040
#12 0x00007f5cc3128ff6 in slap_client_connect (ldp=0x7f5cc3e22558,
sb=0x7f5cc3e22330) at ../../../servers/slapd/config.c:2012
#13 0x00007f5cc319f3f5 in do_syncrep1 (ctx=<value optimized out>,
arg=0x7f5cc3e1f410) at ../../../servers/slapd/syncrepl.c:613
#14 do_syncrepl (ctx=<value optimized out>, arg=0x7f5cc3e1f410) at
../../../servers/slapd/syncrepl.c:1527
#15 0x00007f5cc2c81a98 in ldap_int_thread_pool_wrapper (xpool=0x7f5cc3d18030) at
../../../libraries/libldap_r/tpool.c:688
#16 0x00007f5cc0c54851 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f5cc079790d in clone () from /lib64/libc.so.6
(gdb) thread 7
[Switching to thread 7 (Thread 0x7f5c942d5700 (LWP 3869))]#0 0x00007f5cc0797f03
in epoll_wait () from /lib64/libc.so.6
(gdb) bt
#0 0x00007f5cc0797f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f5cc312e9e2 in slapd_daemon_task (ptr=<value optimized out>) at
../../../servers/slapd/daemon.c:2536
#2 0x00007f5cc0c54851 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f5cc079790d in clone () from /lib64/libc.so.6
Relevant configuration information (from host2):
syncrepl rid=001
provider=ldap://host1/
type=refreshAndPersist
retry="10 +"
searchbase="dc=example,dc=com"
starttls=critical
tls_cacert=/etc/openldap/ssl-certs/cacert.pem
tls_cert=/etc/openldap/ssl-certs/host1.crt
tls_key=/etc/openldap/ssl-certs/host1.key
tls_reqcert=demand
bindmethod=sasl
saslmech=EXTERNAL
sizelimit=unlimited
keepalive="10:2:10"
logfilter="(&(objectClass=auditWriteObject)(reqResult=0))"
logbase="cn=accesslog"
syncdata=accesslog
Netstat of host1:
[root@host1 ~]# netstat -anp | grep slapd
tcp 0 0 0.0.0.0:389 0.0.0.0:*
LISTEN 31615/slapd
tcp 56 0 192.168.1.1:389 192.168.1.2:57276
ESTABLISHED 31615/slapd
tcp 0 0 192.168.1.1:45569 192.168.1.2:389
ESTABLISHED 31615/slapd
unix 2 [ ] STREAM CONNECTED 45346152 31615/slapd
unix 2 [ ] DGRAM 45346144 31615/slapd
Netstat of host2:
[root@host2 ~]# netstat -anp | grep slapd
tcp 0 0 0.0.0.0:389 0.0.0.0:*
LISTEN 3868/slapd
tcp 56 0 192.168.1.2:389 192.168.1.1:45569
ESTABLISHED 3868/slapd
tcp 0 0 192.168.1.2:57276 192.168.1.1:389
ESTABLISHED 3868/slapd
unix 2 [ ] STREAM CONNECTED 19144 3868/slapd
unix [ ] DGRAM 19136 3868/slapd
7 years, 6 months
Re: (ITS#8382) Activation of idlcache leads to wrong ldapsearch results
by fdopheide＠iponweb.net
Hi,
I am also adding the guide to replicate the error as plain text:=20
Step-by-step guide
The following guide was created on an Ubuntu Linux V. 14.04.3 LTS, but =
it should be executeable on an Debian based Linux operating systems. You =
should be logged in as root user or adopt all commands with sudo.
1. Add hostname
echo 'ldap-debug.example.com' > /etc/hostname && hostname `cat =
/etc/hostname` && hostname
echo '52.49.174.211 ldap-debug.example.com ldap-debug' >> /etc/hosts
2. Download latest Sources
wget =
http://www.openldap.org/software/download/OpenLDAP/openldap-release/openld=
ap-2.4.44.tgz
3. Validate Sources(compare md5 provided on website with generated md5 =
hash)
md5sum openldap-2.4.44.tgz
4. Make archive executable and extract sources and change into =
extracted directory
chmod +x openldap-2.4.44.tgz && tar xzf openldap-2.4.44.tgz && cd =
openldap-2.4.44
5. Fullfil most important software requirements (Dependencies)
apt-get update
apt-get install libdb-dev
apt-get install openssl
apt-get install gnutls-bin
apt-get install gcc
apt-get install make
6. Run configure with default settings and add a target dir only
./configure --prefix=3D/usr/local/openldap
7. Run make and install sources to target directory
make depend
make
make test
make install
8. Stop the daemon, because slapd was already started by make.
pkill slapd
9. Edit main config file in /usr/local/openldap/etc/openldap/slapd.conf =
and add the following lines:
include /usr/local/openldap/etc/openldap/schema/core.schema
pidfile /usr/local/openldap/var/run/slapd.pid
argsfile /usr/local/openldap/var/run/slapd.args
modulepath /usr/lib/ldap
moduleload back_hdb
backend hdb
database hdb
suffix "dc=3Dexample,dc=3Dcom"
rootdn "cn=3DManager,dc=3Dexample,dc=3Dcom"
rootpw secret
directory /var/lib/ldap
index objectClass eq
10. Call slappasswd and set rootpw to secret
slappasswd
11. Create data directory for hdb files
mkdir /var/lib/ldap
12. We will add a symbolic link to ease slapd commands
ln -s /usr/local/openldap/libexec/slapd /usr/local/bin/slapd
13. Add the path to OpenLDAP bin and sbin dirs and activate the new path
echo "export =
PATH=3D\$PATH:/usr/local/openldap/bin/:/usr/local/openldap/sbin/" >> =
/root/.bashrc && source /root/.bashrc
14. Start the slapd daemon with -f flag for location of the config file
slapd -f /usr/local/openldap/etc/openldap/slapd.conf
15. Create initial DIT file with an editor of your choice, i.e. vi =
/root/create_dit.ldif and copy paste the following lines into this file.
dn: dc=3Dexample,dc=3Dcom
changetype: add
objectClass: dcObject
objectClass: organization
objectClass: top
dc: example
o: example.com
=20
dn: ou=3DProjects,dc=3Dexample,dc=3Dcom
changetype: add
ou: Projects
objectClass: organizationalUnit
=20
dn: ou=3DTrash,dc=3Dexample,dc=3Dcom
changetype: add
ou: Trash
objectClass: organizationalUnit
=20
dn: ou=3Dtest1,ou=3DProjects,dc=3Dexample,dc=3Dcom
changetype: add
ou: test1
objectClass: organizationalUnit
=20
dn: ou=3Dtest2,ou=3Dtest1,ou=3DProjects,dc=3Dexample,dc=3Dcom
changetype: add
ou: test2
objectClass: organizationalUnit
16. Add the initial DIT
ldapadd -H 'ldap://localhost' -D 'cn=3DManager,dc=3Dexample,dc=3Dcom' =
-w secret -f /root/create_dit.ldif
17. Execute modrdn and assign directory Trash as a new superior of test1 =
directory
ldapmodrdn -x -w secret -D "cn=3DManager,dc=3Dexample,dc=3Dcom" -s =
"ou=3DTrash,dc=3Dexample,dc=3Dcom" -r =
"ou=3Dtest1,ou=3DProjects,dc=3Dexample,dc=3Dcom" "ou=3Dtest1"
18. Execute following ldapsearch and save the result somewhere
for i in {,{,{,ou=3Dtest2\,}ou=3Dtest1\,}ou=3DTrash\,}dc=3Dexample,dc=3D=
com; do echo -e "\n=3D=3D=3D=3D=3D> basedn: $i <=3D=3D=3D=3D=3D\n"; =
ldapsearch -LLL -s 'sub' -H 'ldap://localhost:389' -D =
'cn=3DManager,dc=3Dexample,dc=3Dcom' -w secret -b "$i" '(ou=3Dtest*)' =
dn; done
Result:
=3D=3D=3D=3D=3D> basedn: dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=3D
dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
=3D=3D=3D=3D=3D> basedn: ou=3DTrash,dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=
=3D
dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
=3D=3D=3D=3D=3D> basedn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom =
<=3D=3D=3D=3D=3D
dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
=3D=3D=3D=3D=3D> basedn: =
ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=3D
dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
19. We see until here everything is working as expected, but now we will =
enable cache and idlcache as recommended in OpenLDAP docs and =
Performance Guides and problems will appear soon.
echo -e "cachesize\t10000\nidlcachesize\t30000" >> =
/usr/local/openldap/etc/openldap/slapd.conf
20. Remove existing DIT
ldapdelete -H 'ldap://localhost' -D 'cn=3DManager,dc=3Dexample,dc=3Dco=
m' -w secret -r "dc=3Dexample,dc=3Dcom" -v
21. Restart daemon and recreate initial DIT
pkill slapd
slapd -f /usr/local/openldap/etc/openldap/slapd.conf
ldapadd -H 'ldap://localhost' -D 'cn=3DManager,dc=3Dexample,dc=3Dcom' =
-w secret -f /root/create_dit.ldif
22. Execute modrdn
ldapmodrdn -x -w secret -D "cn=3DManager,dc=3Dexample,dc=3Dcom" -s =
"ou=3DTrash,dc=3Dexample,dc=3Dcom" -r =
"ou=3Dtest1,ou=3DProjects,dc=3Dexample,dc=3Dcom" "ou=3Dtest1"
23. Check result again with ldapsearch
for i in {,{,{,ou=3Dtest2\,}ou=3Dtest1\,}ou=3DTrash\,}dc=3Dexample,dc=3D=
com; do echo -e "\n=3D=3D=3D=3D=3D> basedn: $i <=3D=3D=3D=3D=3D\n"; =
ldapsearch -LLL -s 'sub' -H 'ldap://localhost:389' -D =
'cn=3DManager,dc=3Dexample,dc=3Dcom' -w secret -b "$i" '(ou=3Dtest*)' =
dn; done
Result:
=3D=3D=3D=3D=3D> basedn: dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=3D
dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
=3D=3D=3D=3D=3D> basedn: ou=3DTrash,dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=
=3D
dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
=3D=3D=3D=3D=3D> basedn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom =
<=3D=3D=3D=3D=3D
dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
=3D=3D=3D=3D=3D> basedn: =
ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=3D
dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
24. Once again the result was fine and as expected. Keep in mind cache =
and idlcache were empty as we've restarted the daemon. Now we are =
performing same actions again while we have cached results.
25. Remove directories in ou =E2=80=9ETrash=E2=80=9C
ldapdelete -w secret -D "cn=3DManager,dc=3Dexample,dc=3Dcom" -r =
"ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom"
26. Add directories to ou =E2=80=9EProjects=E2=80=9C again
ldapmodify -D "cn=3DManager,dc=3Dexample,dc=3Dcom" -W
Type in password
Enter LDAP Password:
Copy paste following ldif and be sure emtpy lines are copied, too!
dn: ou=3Dtest1,ou=3DProjects,dc=3Dexample,dc=3Dcom
changetype: add
objectClass: organizationalUnit
=20
dn: ou=3Dtest2,ou=3Dtest1,ou=3DProjects,dc=3Dexample,dc=3Dcom
changetype: add
objectClass: organizationalUnit
27. Now we have same DIT structure again and can rerun modrdn command
ldapmodrdn -x -w secret -D "cn=3DManager,dc=3Dexample,dc=3Dcom" -s =
"ou=3DTrash,dc=3Dexample,dc=3Dcom" -r =
"ou=3Dtest1,ou=3DProjects,dc=3Dexample,dc=3Dcom" "ou=3Dtest1"
28. If we now check result with ldapsearch again the idlcache error will =
appear
for i in {,{,{,ou=3Dtest2\,}ou=3Dtest1\,}ou=3DTrash\,}dc=3Dexample,dc=3D=
com; do echo -e "\n=3D=3D=3D=3D=3D> basedn: $i <=3D=3D=3D=3D=3D\n"; =
ldapsearch -LLL -s 'sub' -H 'ldap://localhost:389' -D =
'cn=3DManager,dc=3Dexample,dc=3Dcom' -w secret -b "$i" '(ou=3Dtest*)' =
dn; done
Result:
=3D=3D=3D=3D=3D> basedn: dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=3D
dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
=3D=3D=3D=3D=3D> basedn: ou=3DTrash,dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=
=3D
dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
=3D=3D=3D=3D=3D> basedn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom =
<=3D=3D=3D=3D=3D
dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
=3D=3D=3D=3D=3D> basedn: =
ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=3D
dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
29. The search result of basedn: ou=3DTrash,dc=3Dexample,dc=3Dcom is =
wrong! The line of dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3D=
com is missing within the resultset.
30. After a restart of the daemon the missing line will appear again. =
Actually the one and only solution is to switch of idlcachesize and set =
it explicitely to "0"
> Am 07.03.2016 um 11:45 schrieb fdopheide(a)iponweb.net:
>=20
> Full_Name: Frank Dopheide
> Version: slapd 2.4.44 (Mar 3 2016 10:27:05)
> OS: Ubuntu 14.04.3 LTS
> URL: http://52.48.8.167/idlcache_error.pdf
> Submission from: (NULL) (87.139.29.59)
>=20
>=20
> Good morning Gentlemen,
>=20
> we've found a quite serious bug in slapd 2.4.44 on an Ubuntu 14.04.3 =
LTS server.
> As soon as we enable the idlcache the ldapsearch results will be wrong =
after
> moddn and modrdn commands. First result after idlcachesize activation =
is still
> correct as the cache is still empty and the daemon reads from db, but =
as soon as
> it starts reading from idlcache the latest modifications are missing =
until slapd
> will be restarted!
>=20
> We've created a small howto to replicatehe e error. The howto is =
available via
> http: http://52.48.8.167/idlcache_error.pdf
>=20
> Please let us now if you require any additional information!
>=20
> Thx,
> Frank
>=20
>=20
>=20
7 years, 6 months
(ITS#8383) msys build error "portable.h:1116:19: error: two or more data types in declaration specifiers"
by fixed-term.Asal.Mirzaieva＠boschrexroth.de
Full_Name: Asal Mirzaieva
Version: 2.4.44
OS: Windows 7
URL: http://paste.ubuntu.com/15326640/, http://paste.ubuntu.com/15326646/, http://paste.ubuntu.com/15326655/
Submission from: (NULL) (194.39.218.10)
When building openldap on windows, using msys and mingw (mingw32 v4.9.3) the
following error appears:
In file included from base64.c:38:0:../../include/portable.h:1116:19: error: two
or more data types in declaration specifiers
#define socklen_t int
Steps to reproduce the bug:
1. Install Mingw32 with mingw-get-setup.exe.
2. Launch msys.bat in msys folder to open msys console.
3. Download and install Spencer's regex in /usr/local
4. Download openldap-2.4.44
5. Unzip and untar the openldap and go to the openldap-2.4.44's directory
6. Run configure as follows
./configure CFLAGS="-I/usr/local/include/rxspencer" LDFLAGS=2-2-L/usr/local/lib"
LIBS="-static -lrxspencer" --enable-static --disable-shared --disable-slapd
--disable-bdb --disable-hdb
7. Run make depend
8. Run make
9. Get the error, that was described above.
I haven't tried to reproduce this bug with other PCs/OS versions/MinGW
versions.
Workaround: after commenting the line 1116 in include/portable.h the error was
gone and I was able to successfully build and use openldap. Well, "successfully"
means, I could link it statically and run ldap_initialize(), but not
ldap_sals_bind(), because I haven't installed SASL.
7 years, 6 months
(ITS#8382) Activation of idlcache leads to wrong ldapsearch results
by fdopheide＠iponweb.net
Full_Name: Frank Dopheide
Version: slapd 2.4.44 (Mar 3 2016 10:27:05)
OS: Ubuntu 14.04.3 LTS
URL: http://52.48.8.167/idlcache_error.pdf
Submission from: (NULL) (87.139.29.59)
Good morning Gentlemen,
we've found a quite serious bug in slapd 2.4.44 on an Ubuntu 14.04.3 LTS server.
As soon as we enable the idlcache the ldapsearch results will be wrong after
moddn and modrdn commands. First result after idlcachesize activation is still
correct as the cache is still empty and the daemon reads from db, but as soon as
it starts reading from idlcache the latest modifications are missing until slapd
will be restarted!
We've created a small howto to replicatehe e error. The howto is available via
http: http://52.48.8.167/idlcache_error.pdf
Please let us now if you require any additional information!
Thx,
Frank
7 years, 6 months
Re: (ITS#8380) Feature request: make a plugin like smbk5pwd for the HA1 and HA1b hashes used in DIGEST and HMAC
by daniel＠pocock.pro
On 06/03/16 23:02, Howard Chu wrote:
> Daniel Pocock wrote:
>>
>>
>> On 06/03/16 20:49, Howard Chu wrote:
>>> Daniel Pocock wrote:
>>>>
>>>>
>>>> On 06/03/16 20:00, Howard Chu wrote:
>>>>> daniel(a)pocock.pro wrote:
>>>>>> Full_Name: Daniel Pocock
>>>>>> Version:
>>>>>> OS: Debian
>>>>>> URL: ftp://ftp.openldap.org/incoming/
>>>>>> Submission from: (NULL) (2001:1620:b22::2042)
>>>>>>
>>>>>>
>>>>>> There are a few protocols that use a HA1[1] password hash, such as
>>>>>> HTTP
>>>>>> DIGEST[1], SIP DIGEST[2] and TURN[3] (which uses HMAC rather than
>>>>>> DIGEST)
>>>>>>
>>>>>> Is there a standard LDAP attribute name for storing a HA1 value or
>>>>>> should it be stored in a regular userPassword attribute as
>>>>>> described in
>>>>>> the manual[4]?
>>>>>
>>>>> The ITS is not for usage questions. You already asked this and were
>>>>> answered on the discussion mailing list.
>>>>>
>>>>> http://www.openldap.org/lists/openldap-technical/201507/msg00073.html
>>>>>
>>>>> There is nothing here that requires any OpenLDAP development activity.
>>>>> It's all already handled by the SASL Digest mechanism, as I already
>>>>> noted in the above email.
>>>>>
>>>>> Closing this ITS.
>>>>
>>>>
>>>> I didn't open this feature request to ask for somebody to implement it,
>>>> I'm simply trying to track a number of items that I'm working on
>>>> myself.
>>>> Normally I open a bug/feature request in anything I work on in case
>>>> somebody else wants to work on the same thing, it helps avoid
>>>> duplication.
>>>>
>>>> The email thread doesn't fully resolve the issue, it does appear to
>>>> require some plugin to be created for the server side, especially if
>>>> the
>>>> LDAP server doesn't keep plain text passwords. Given the fairly
>>>> generic
>>>> nature of the DIGEST algorithm, I also felt that when implemented, this
>>>> code should be contributed to the OpenLDAP repository and not hosted
>>>> elsewhere.
>>>
>>> Take the hint: RTF SASL Digest code. All the code you're asking for has
>>> already been implemented in Cyrus SASL and is of zero concern to
>>> OpenLDAP.
>>>
>>> The most important skill of a programmer is being able to *read* - not
>>> being able to write. Any fool can spew code.
>>>
>>
>> I'm not sure if you've seen my reply to the list, it looks like it got
>> stuck in moderation
>>
>> I understand your point about DIGEST and that may well work for HTTP and
>> SIP. TURN, however, uses HMAC-SHA1 and that involves sending a copy of
>> the entire message body to the authentication server for use in the
>> algorithm:
>> https://tools.ietf.org/html/rfc5766#page-51
>
> You continue to fail to read.
>
> Pasting this again
>
> http://www.openldap.org/lists/openldap-technical/201507/msg00073.html
>
> The required parameters cannot be passed in a Simple Bind request. The
> only way to be able to pass what you want and have the server
> authenticate it is to use a SASL mechanism. For TURN you may need to
> define your own SASL mechanism.
>
> None of this has anything to do with OpenLDAP.
I fully agree with that - the HMAC-SHA1 message verification for TURN,
like the DIGEST-MD5 code, would be in Cyrus SASL or in the TURN server
(LDAP client).
The OpenLDAP manual, s15.2.3 about DIGEST-MD5 explains "it needs access
to the plaintext password". The DIGEST-MD5 code that uses the password
is here:
https://cgit.cyrus.foundation/cyrus-sasl/tree/plugins/digestmd5.c#n440
and it takes the plaintext password and transforms it into a HA1 string.
Notice that:
HA1 = MD5(user:realm:password)
This is a repetitive calculation that does not involve the nonce, salt
or any other transient value. HA1 is always the same value for any
given user/realm/password permutation. Therefore, OpenLDAP could store
HA1 values instead of plaintext values.
In fact, Apache already does the same thing with a flat file maintained
by the htdigest utility:
$ htdigest -c /tmp/test.ha1 example.org daniel
instead of storing plaintext values in a file.
The HA1 values, not being salted, are still sensitive but some sites may
prefer to store the HA1 values instead of plaintext values.
So section 15.2.3 could actually be changed: "it needs access to the
plaintext password or a password that has already been hashed in HA1"
An alternative is that the TURN server process could bind to the LDAP
server and then query the HA1 attribute for any user or the HA1 values
could be periodically extract into a flat file similar to that created
by the htdigest utility.
I agree that people who have plaintext passwords don't need to store any
HA1 attribute, but if somebody does not have plaintext passwords and
does not want to have plaintext passwords, it appears that the HA1
hashing needs to be done in OpenLDAP on password changes.
This would also appear to require a change to the Cyrus code so that it
can accept either the plaintext or HA1 from the LDAP entry and the slapd
auxprop plugin would need to be able to return either attribute.
>
>> CRAM-MD5 from SASL does HMAC but it does not appear to transfer entire
>> message bodies in the manner required for TURN.
>>
>> DIGEST-MD5 and HMAC-SHA1 both use a HA1 value (or a cleartext password)
>> as the lowest common denominator but otherwise they are not the same.
>>
>>> Your mention of smbk5pwd is totally off base as well. The reason the
>>> smbk5pwd module was needed was because Samba 3 and the Kerberos5 KDC
>>> both stored their secrets in separate and incompatible formats but
>>> everyone wanted central coordinated administration for these separate
>>> attributes. If you're writing something from scratch there is no reason
>>> to use your own separate and incompatible attribute, and thus there is
>>> no reason to require any special synchronization or coordination.
>>>
>>
>> The reason I mentioned that is because it was the closest thing I could
>> find to the concept of storing multiple password hashes, but I'm not
>> locked in to that strategy. If there are cleartext passwords in LDAP
>> then they can be used for all of these algorithms. If the administrator
>> does not want to store cleartext values, however, then the salted
>> password strings used for UNIX logins are not interchangeable with HA1
>> hashed values, in that case, isn't it necessary to store and synchronize
>> multiple values, hashed with each algorithm?
>
> Once again you're back to software usage questions which is not what the
> ITS is for.
>
7 years, 6 months
