openldap-bugs January 2020

openldap-bugs@openldap.org

15 participants
38 discussions

Re: (ITS#9124) CVE-ID?
by michael＠stroeder.com 10 Jan '20

10 Jan '20

Stephan, regarding: https://www.openldap.org/its/index.cgi?findid=9124 Was there ever a CVE-Id assigned to this issue? I'd like to reference it in back-port patches for downstream packages. Ciao, Michael.

1 0

Re: (ITS#9098) assert fails in meta_back_search in some cases after reconnect
by maxime.besson＠worteks.com 10 Jan '20

10 Jan '20

Hi, After a couple of months without any issues using 2.4.48, we suddenly encountered a crash again, but this time on 2.4.48. It was the exact same symptom, and the same assert failing as in my original message. It appears that the issue happens a lot more rarely in 2.4.48 compared to 2.4.47, so ITS#8841 possibly had an effect on my issue, but did not solve it completely. I have a core dump I can run tests against, if you need. -- Maxime Besson Expert Infrastructure - Worteks maxime.besson(a)worteks.com

1 0

(ITS#9150) olcDbNoSync: FALSE has the same effect as olcDbNoSync: TRUE
by maxime.besson＠worteks.com 09 Jan '20

09 Jan '20

Full_Name: Maxime Besson Version: 2.4.48 OS: Debian Buster / CentOS7 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2a01:cb00:802:8400:2cbe:3c60:fca6:e50b) While migrating configuration from slapd.conf to cn=config, I noticed something pretty strange related to the dbnosync option The result of converting the following config: include ./core.schema pidfile ./slapd.1.pid argsfile ./slapd.1.args database mdb suffix "dc=example,dc=com" rootdn "cn=Manager,dc=example,dc=com" rootpw secret directory db.1.a index objectClass eq index cn,sn,uid pres,eq,sub index entryUUID,entryCSN eq Is (only interested in MDB config): # AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify. # CRC32 3df9c3d6 dn: olcDatabase={1}mdb objectClass: olcDatabaseConfig objectClass: olcMdbConfig olcDatabase: {1}mdb olcSuffix: dc=example,dc=com olcAddContentAcl: FALSE olcLastMod: TRUE olcMaxDerefDepth: 15 olcReadOnly: FALSE olcRootDN: cn=Manager,dc=example,dc=com olcRootPW:: c2VjcmV0 olcSyncUseSubentry: FALSE olcMonitoring: TRUE olcDbDirectory: db.1.a olcDbNoSync: FALSE olcDbIndex: objectClass eq olcDbIndex: entryUUID eq olcDbIndex: entryCSN eq olcDbIndex: cn pres,eq,sub olcDbIndex: uid pres,eq,sub olcDbIndex: sn pres,eq,sub olcDbMaxReaders: 0 olcDbMaxSize: 10485760 olcDbMode: 0600 olcDbSearchStack: 16 olcDbMaxEntrySize: 0 olcDbRtxnSize: 10000 As you can see, olcDbNoSync has the value FALSE, because there was no "dbnosync" option in slapd.conf, however, performance testing shows a dramatic increase in write speed after slapd.d migration. Which seems odd to me. It seems like the MDB database was not syncing to disk anymore despite olcDbNoSync being FALSE Removing olcDbNoSync: FALSE from cn=config did restore previous performance and syncing behavior Looking at servers/slapd/back-mdb/config.c:723 I found this: case MDB_DBNOSYNC: if ( c->value_int ) mdb->mi_dbenv_flags |= MDB_NOSYNC; else mdb->mi_dbenv_flags ^= MDB_NOSYNC; The else case seems wrong, I replaced it with case MDB_DBNOSYNC: if ( c->value_int ) mdb->mi_dbenv_flags |= MDB_NOSYNC; else mdb->mi_dbenv_flags &= ~MDB_NOSYNC; And it solves the issue on my system. Reproduced on master, and 2.4.48, on Debian

1 0

Re: (ITS#9146) incomplete initialization of sessionlog structure may cause session log to grow indefinitely
by hyc＠symas.com 08 Jan '20

08 Jan '20

maxime.besson(a)worteks.com wrote: > Full_Name: Maxime Besson > Version: 2.4.48 > OS: Amazon Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (92.184.104.114) Thanks for the report, fixed now in git master. > > > During a cloud provider migration, I have migrated an OpenLDAP instance using > MDB + Syncrepl (in mirror mode, with session log) from Ubuntu to Amazon Linux. > > When starting a server with no data, syncrepl kicks in and starts replicating > the other server's database (12G, 7M entries). > > In Ubuntu this phase takes about 25 minutes (with dbnosync). In Amazon Linux, > with the exact same cn=config and daemon options, it took 10 days on first try, > at 100% CPU consumption the whole time. > > I observed that the start of the import is pretty fast (data.mdb grows by > 10M/s) but import speed slows down rapidly to a crawl. > > After some investigation, I discovered that disabling the sessionlog brings the > import times to normal. > > Digging further, I found that most of the CPU time is spent in > syncprov_add_slog, and particularly this line: > > > /* Keep the list in csn order. */ > ... > for ( sep = &sl->sl_head; *sep; sep = &(*sep)->se_next ) { >>>>>>>> if ( ber_bvcmp( &se->se_csn, &(*sep)->se_csn ) < 0 ) { > se->se_next = *sep; > *sep = se; > break; > } > } > > The sessionlog appears to be growing endlessly: > > (gdb) print sl->sl_num > $1 = 681896 > (gdb) print sl->sl_size > $3 = 100 > > Digging around the source, it seems that the session log is supposed to be > cleaned up at the end of syncprov_add_slog, if it's not playing: > > if (!sl->sl_playing) { > while ( sl->sl_num > sl->sl_size ) { > ... > > However: > > (gdb) print sl->sl_playing > $4 = 543388517 > > This looked like an uninitialized value to me, and indeed, sp_cf_gen > doesn't seem to initialize this field: > > if ( !sl ) { > sl = ch_malloc( sizeof( sessionlog )); > sl->sl_mincsn = NULL; > sl->sl_sids = NULL; > sl->sl_num = 0; > sl->sl_numcsns = 0; > sl->sl_head = sl->sl_tail = NULL; > ldap_pvt_thread_mutex_init( &sl->sl_mutex ); > si->si_logs = sl; > } > > > I tried the following patch: > > --- openldap-2.4.48.orig/servers/slapd/overlays/syncprov.c 2019-07-23 > 16:46:22.000000000 +0200 > +++ openldap-2.4.48/servers/slapd/overlays/syncprov.c 2020-01-08 > 11:33:16.770110282 +0100 > @@ -3082,6 +3082,7 @@ > sl = ch_malloc( sizeof( sessionlog )); > sl->sl_mincsn = NULL; > sl->sl_sids = NULL; > + sl->sl_playing = 0; > sl->sl_num = 0; > sl->sl_numcsns = 0; > sl->sl_head = sl->sl_tail = NULL; > > > And it got rid of the issue. My guess is that most of the time, sl_playing > happens to be 0, but on Amazon Linux, for some reason (patched glibc?), the > sessionlog lands in the wrong chunk of uninitilized memory > > Additional info: > > # rpm -q glibc > glibc-2.26-32.amzn2.0.2.x86_64 > > # uname -a > Linux 4.14.152-127.182.amzn2.x86_64 #1 SMP Thu Nov 14 17:32:43 UTC 2019 x86_64 > x86_64 x86_64 GNU/Linux > > # /usr/local/openldap/libexec/slapd -V > @(#) $OpenLDAP: slapd 2.4.48 (Jan 8 2020 11:39:38) $ > > (LTB build) > > > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9147) syncrepl connexion leak when provider uses an expired certificate
by maxime.besson＠worteks.com 08 Jan '20

08 Jan '20

Full_Name: Maxime Besson Version: 2.4.48 OS: Debian Buster / CentOS7 URL: https://cloud.worteks.com/index.php/s/9CE6ALLaAfrxZW4/download Submission from: (NULL) (92.184.104.113) I have two OpenLDAP (2.4.48, Ubuntu) servers running with syncrepl in mirrormode. One of my server's X509 certificates has recently expired, and I noticed that while it was expired, the other node's connection count kept climbing until it a "Max open files" condition. It seems that when a Syncrepl consumer encounters a certificate error, the outgoing LDAP connection to the provider is never closed. Attached to this bug you will find a test case to reproduce this behavior # runs a provider with a bogus certificate # and a consumer with retry=3+ sh test.sh ... TLS certificate verification: Error, self signed certificate TLS trace: SSL3 alert write:fatal:unknown CA TLS trace: SSL_connect:error in error TLS: can't connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed (self signed certificate). 5e160966 slap_client_connect: URI=ldaps://127.0.1.1:6636/ DN="cn=manager,dc=example,dc=com" ldap_sasl_bind_s failed (-1) 5e160966 do_syncrepl: rid=001 rc -1 retrying ... and so on, every 3 seconds While the consumer retries, running lsof on its PID will reveal the connection leak: 42u TCP localhost:45964->localhost.lan:6636 (CLOSE_WAIT) 43u TCP localhost:45966->localhost.lan:6636 (CLOSE_WAIT) 44u TCP localhost:45968->localhost.lan:6636 (CLOSE_WAIT) 45u TCP localhost:45970->localhost.lan:6636 (CLOSE_WAIT) 46u TCP localhost:45972->localhost.lan:6636 (CLOSE_WAIT) 47u TCP localhost:45974->localhost.lan:6636 (CLOSE_WAIT) 48u TCP localhost:45976->localhost.lan:6636 (CLOSE_WAIT) 49u TCP localhost:45978->localhost.lan:6636 (CLOSE_WAIT) 50u TCP localhost:45980->localhost.lan:6636 (CLOSE_WAIT) 51u TCP localhost:45982->localhost.lan:6636 (CLOSE_WAIT) 52u TCP localhost:45984->localhost.lan:6636 (CLOSE_WAIT) 53u TCP localhost:45986->localhost.lan:6636 (CLOSE_WAIT) 54u TCP localhost:45988->localhost.lan:6636 (CLOSE_WAIT) 55u TCP localhost:45990->localhost.lan:6636 (CLOSE_WAIT) 56u TCP localhost:45992->localhost.lan:6636 (CLOSE_WAIT) 57u TCP localhost:45994->localhost.lan:6636 (CLOSE_WAIT) 58u TCP localhost:45996->localhost.lan:6636 (CLOSE_WAIT) 59u TCP localhost:45998->localhost.lan:6636 (CLOSE_WAIT) 60u TCP localhost:46000->localhost.lan:6636 (CLOSE_WAIT) 61u TCP localhost:46002->localhost.lan:6636 (CLOSE_WAIT) 62u TCP localhost:46004->localhost.lan:6636 (CLOSE_WAIT) 63u TCP localhost:46006->localhost.lan:6636 (CLOSE_WAIT) 64u TCP localhost:46008->localhost.lan:6636 (CLOSE_WAIT) Modifying the provider URL in slapd.2.conf with a wrong port causes the syncrepl consumer to fail and retry just as much, but without connection piling up in CLOSE_WAIT state. This is not a very critical issue because it only affects servers who are in already degraded condition (broken replication, invalid certificate on the provider) but I thought it still was worth reporting. I was able to reproduce this issue on the git master branch, on released 2.4.48, on Centos7 and on Debian Buster. OpenSSL version on the debian system: 1.1.1d-0+deb10u2

1 0

(ITS#9146) incomplete initialization of sessionlog structure may cause session log to grow indefinitely
by maxime.besson＠worteks.com 08 Jan '20

08 Jan '20

Full_Name: Maxime Besson Version: 2.4.48 OS: Amazon Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (92.184.104.114) During a cloud provider migration, I have migrated an OpenLDAP instance using MDB + Syncrepl (in mirror mode, with session log) from Ubuntu to Amazon Linux. When starting a server with no data, syncrepl kicks in and starts replicating the other server's database (12G, 7M entries). In Ubuntu this phase takes about 25 minutes (with dbnosync). In Amazon Linux, with the exact same cn=config and daemon options, it took 10 days on first try, at 100% CPU consumption the whole time. I observed that the start of the import is pretty fast (data.mdb grows by 10M/s) but import speed slows down rapidly to a crawl. After some investigation, I discovered that disabling the sessionlog brings the import times to normal. Digging further, I found that most of the CPU time is spent in syncprov_add_slog, and particularly this line: /* Keep the list in csn order. */ ... for ( sep = &sl->sl_head; *sep; sep = &(*sep)->se_next ) { >>>>>>> if ( ber_bvcmp( &se->se_csn, &(*sep)->se_csn ) < 0 ) { se->se_next = *sep; *sep = se; break; } } The sessionlog appears to be growing endlessly: (gdb) print sl->sl_num $1 = 681896 (gdb) print sl->sl_size $3 = 100 Digging around the source, it seems that the session log is supposed to be cleaned up at the end of syncprov_add_slog, if it's not playing: if (!sl->sl_playing) { while ( sl->sl_num > sl->sl_size ) { ... However: (gdb) print sl->sl_playing $4 = 543388517 This looked like an uninitialized value to me, and indeed, sp_cf_gen doesn't seem to initialize this field: if ( !sl ) { sl = ch_malloc( sizeof( sessionlog )); sl->sl_mincsn = NULL; sl->sl_sids = NULL; sl->sl_num = 0; sl->sl_numcsns = 0; sl->sl_head = sl->sl_tail = NULL; ldap_pvt_thread_mutex_init( &sl->sl_mutex ); si->si_logs = sl; } I tried the following patch: --- openldap-2.4.48.orig/servers/slapd/overlays/syncprov.c 2019-07-23 16:46:22.000000000 +0200 +++ openldap-2.4.48/servers/slapd/overlays/syncprov.c 2020-01-08 11:33:16.770110282 +0100 @@ -3082,6 +3082,7 @@ sl = ch_malloc( sizeof( sessionlog )); sl->sl_mincsn = NULL; sl->sl_sids = NULL; + sl->sl_playing = 0; sl->sl_num = 0; sl->sl_numcsns = 0; sl->sl_head = sl->sl_tail = NULL; And it got rid of the issue. My guess is that most of the time, sl_playing happens to be 0, but on Amazon Linux, for some reason (patched glibc?), the sessionlog lands in the wrong chunk of uninitilized memory Additional info: # rpm -q glibc glibc-2.26-32.amzn2.0.2.x86_64 # uname -a Linux 4.14.152-127.182.amzn2.x86_64 #1 SMP Thu Nov 14 17:32:43 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux # /usr/local/openldap/libexec/slapd -V @(#) $OpenLDAP: slapd 2.4.48 (Jan 8 2020 11:39:38) $ (LTB build)

1 0

Re: (ITS#6104) race condition with cancel operation
by ondra＠mistotebe.net 03 Jan '20

03 Jan '20

This might be related or the same issue. With ITS#9124 and ITS#9145 fixed, if you run: ldapsearch -E '!sync=rp' -E '!cancel' syncprov has freed the operation and cancel_extop() still tries to touch the same Operation pointer without checking it's still in the connection list. At least valgrind isn't happy. -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

1 0

(ITS#9145) ldap tools try to attach configured controls to cancel/abandon operations
by ondra＠openldap.org 03 Jan '20

03 Jan '20

Full_Name: Ondrej Kuznik Version: re24/master OS: URL: Submission from: (NULL) (146.90.154.72) Take this command for example: ldapsearch -E '!sync=rp' -E '!cancel' Here, tool_check_abandon() calls ldap_cancel_s() with sctrls == NULL, but ld->ld_sctrls contains the syncrepl control as a default so it gets used instead, same for abandon. This means the option is broken. tool_check_abandon() should pass an zero-length LDAPControl list instead. Patch coming shortly.

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs January 2020