openldap-bugs

openldap-bugs@openldap.org

1 participants
20963 discussions

(ITS#7165) Killing slap tools breaks the running mdb
by h.b.furuseth＠usit.uio.no 14 Feb '12

14 Feb '12

Full_Name: Hallvard B Furuseth Version: master, d861910f55cb1beb5e443caa7e961ed760074352 OS: Linux x86_64 URL: Submission from: (NULL) (195.1.106.125) Submitted by: hallvard Aborting an MDB process can break MDB if another MDB process is running, because then locks/reader table info are not reset. The problems below go away when the last slap process terminates so the lockfile can be reset. Sometimes kill -KILL is needed. ==== lock.conf ==== include servers/slapd/schema/core.schema database mdb suffix o=hi directory lock.dir (A) If a process holding a shared mutex dies, other processes' ops on the mutex can hang/fail: rm -rf lock.dir; mkdir lock.dir servers/slapd/slapd -f lock.conf -h ldapi://ldapi -d0 & MDB_KILL_R=true servers/slapd/slapd -Tcat -f lock.conf ldapsearch -xLLH ldapi://ldapi -b o=hi l=x 1.1 ^C (kills ldapsearch which is waiting for results) kill %% (fails to kill slapd) kill -KILL %% The Posix fix, robust mutexes, is outlined below. With _WIN32, check return code WAIT_ABANDONED. __APPLE__ Posix semaphores: Don't know. SysV semaphores have SEM_UNDO to cancel the process' effect if the process dies, but the peers do not seem to be informed that this happened. if ((rc = pthread_mutexattr_init( &mattr )) || (rc = pthread_mutexattr_setpshared(&mattr, PTHREAD_PROCESS_SHARED)) || (rc = pthread_mutexattr_setrobust( &mattr, PTHREAD_MUTEX_ROBUST )) || (rc = pthread_mutex_init(&mutex, &mattr ))) return rc; ... switch (pthread_mutex_lock(&mutex)) { case 0: break; case EOWNERDEAD: Repair the possibly half-updated data protected by <mutex>; if (successful repair) { pthread_mutex_consistent(&mutex); break; } /* With Posix mutexes, make future use of <mutex> return failure: */ pthread_mutex_unlock(&mutex); /* FALLTHRU */ default: return MDB_PANIC; } ... pthread_mutex_unlock(&mutex); BTW, the above also happens with MDB_KILL_W=true slapcat followed by ldapadd, dunno why slapcat uses a txn without MDB_TXN_RDONLY. In this case, raise(SIGINT) is sufficient. With MDB_KILL_R, that does not kill slapd and we need SIGKILL - it catches SIGINT I guess. Don't know if that difference is intentional. (B1) Repeated kill -KILL slap<tool> while slapd is running can use up the reader table since it does not clear its table slots, making the db unusable. (B2) slapcat report no error after it fails to read the db due to full reader table, it exits with success status. (B3) After one "kill -KILL slap<tool>" while slapd is running, I imagine the stale slot which does not go away can prevent freelist reuse so the db grows quickly. But I have not seen that. bash$ rm -rf lock.dir; mkdir lock.dir bash$ servers/slapd/slapd -f lock.conf -h ldapi://ldapi -d0 & bash$ for (( i=0; i<130; i++)) { MDB_KILL_UR=true servers/slapd/slapd -Tcat -f lock.conf } > /dev/null bash$ servers/slapd/slapd -Tcat -f lock.conf -d-1 2>cat.out (success result, no output, no errors in the log) bash$ ldapsearch -xLLH ldapi://ldapi -b o=hi l=x 1.1 version: 1 [R..RLOCKED..R1] Other (e.g., implementation specific) error (80) Additional information: internal error bash$ Maybe it's easier to have a per-process reader table after all, and a shared table with just the oldest txn per process. Or, each transaction chould have an exclusive file region lock. A writer could try to grab the oldest txn's region lock if that belongs to another process, and clear the txn it if successful. A reader, before giving up, would just search for another pid's txn whose lock it can grab. Except since the system reuses pids, the ID would either have to be more exclusive (e.g. pid + mdb_env_open() time) or mdb_env_open must clear away reader table entries with its own pid. Speaking of which, I don't see the use of the repeated getpid() calls. Should be enough to store the pid in the MDB_env. fcntl record locks do not survive fork(), so an MDB_env doesn't either. And the thread ID is unsed, and must be reset with memset, not 'mr_tid = 0' since pthread_t can be a struct. (C) Mentioned earlier: Startup can fail due to a race condition, when another process starts MDB at the same time and then aborts: - Process 1 starts MDB, gets write lock in mdb_env_setup_locks(). - Process 2 starts MDB, awaits read lock since write lock is taken. - Process 1 dies. - Process 2 gets the read lock and proceeds, but with an uninitialized lockfile since it expects that process 1 initialized it. Fix: An extra exclusive lock around the current lock setup code. kill slapd process, if any rm -rf lock.dir; mkdir lock.dir MDB_RACE=true servers/slapd/slapd -Tcat -f lock.conf & sleep 1; \ servers/slapd/slapd -Tcat -f lock.conf In addition, the above dumps core in mdb_env_close: for (i=0; i<env->me_txns->mti_numreaders; i++) where env->me_txns == (MDB_txninfo *) 0xffffffffffffffff. (D) And of course ^Z when holding a mutex can block other processes' threads, but I suppose BDB has the same problem. DB_SUSPEND=true servers/slapd/slapd -Tadd -f lock.conf < /dev/null servers/slapd/slapd -Tadd -f lock.conf < /dev/null (hanging...) ^C kill %% The fix I can see for now is "don't do that" in the doc. MDB could in addition use pthread_mutex_timedlock() so slapd at least can keep running and can be killed without kill -KILL. Final fix = a lock-free library, when a useful portable one emerges.

1 0

Re: ~*~ [spam] Re: (ITS#7143) Assertion error (crash); using relay backend and translucent ove
by hyc＠symas.com 14 Feb '12

14 Feb '12

Mattias Andersson wrote: > On Sat, 04 Feb 2012 01:59:49 +0100, Howard Chu<hyc(a)symas.com> wrote: >> OK, we need to know exactly what the difference is between the >> ldapsearch query and the browser query. Can you please run slapd -d7 and >> record first the ldapsearch query and then the browser query and post >> the output for each of those? > > Attached the output from the two queries. Thanks. The difference is that the softerra browser was searching with attrsOnly set true, so all of the returned attributes had no values. Setting this option in ldapsearch allowed the issue to be reproduced. This is now fixed in git master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7150) SEGFAULT in openldap-2.4.28 & 2.4.29
by hyc＠symas.com 14 Feb '12

14 Feb '12

David Coutadeur wrote: > > Many thanks for your rapid fix ! It seems to work better. Some problems > remain unfortunately : > > slapd freezes, after 10 to 15 rapid tests, interrupted by Ctrl-C after > 10 seconds (a test is 100 times 100 threads, each doing a bind, an add, > a modify, a delete, and a logout). > Memory is stable, but cpu is taken at 99,7% by slapd. Sounds like a different problem, which should obviously be handled in a different bug report. And again, please provide a gdb snapshot of all the active threads when the problem occurs. > > > > Le 03/02/2012 18:36, Howard Chu a écrit : >> Thanks. A fix for this is now in git master (5584e5b) >> >>> Le 03/02/2012 18:15, Howard Chu a écrit : >>>> david.coutadeur(a)linagora.com wrote: >>>>> I have reproduced the bug with a fresh new HEAD version yesterday, >>>>> including : >>>>> 02/02/2012 Howard Chu ITS#7146 fix prev commit >>>> >>>> Can you include a "bt full" gdb backtrace on this? >>>> >>>>> Le 02/02/2012 20:25, Michael Ströder a écrit : >>>>>> dcoutadeur(a)linagora.com wrote: >>>>>>> I have encountered a bug on openldap 2.4.28 and openldap 2.4.29 >>>>>>> (ie git >>>>>>> version) >>>>>> >>>>>> Which exact git version? Did you build RE24 from git already >>>>>> containing >>>>>> the fix for ITS#7113 (dup of ITS#6928)? >> > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7150) SEGFAULT in openldap-2.4.28 & 2.4.29
by david.coutadeur＠linagora.com 14 Feb '12

14 Feb '12

Many thanks for your rapid fix ! It seems to work better. Some problems remain unfortunately : slapd freezes, after 10 to 15 rapid tests, interrupted by Ctrl-C after 10 seconds (a test is 100 times 100 threads, each doing a bind, an add, a modify, a delete, and a logout). Memory is stable, but cpu is taken at 99,7% by slapd. Le 03/02/2012 18:36, Howard Chu a écrit : > Thanks. A fix for this is now in git master (5584e5b) > >> Le 03/02/2012 18:15, Howard Chu a écrit : >>> david.coutadeur(a)linagora.com wrote: >>>> I have reproduced the bug with a fresh new HEAD version yesterday, >>>> including : >>>> 02/02/2012 Howard Chu ITS#7146 fix prev commit >>> >>> Can you include a "bt full" gdb backtrace on this? >>> >>>> Le 02/02/2012 20:25, Michael Ströder a écrit : >>>>> dcoutadeur(a)linagora.com wrote: >>>>>> I have encountered a bug on openldap 2.4.28 and openldap 2.4.29 >>>>>> (ie git >>>>>> version) >>>>> >>>>> Which exact git version? Did you build RE24 from git already >>>>> containing >>>>> the fix for ITS#7113 (dup of ITS#6928)? >

1 0

(ITS#7164) Prevent change of TLS setting if invalid (Dynamic Config)
by nick＠eurobjects.com 13 Feb '12

13 Feb '12

Full_Name: Nikolaos Milas Version: 2.4.28 OS: CentOS 5.7 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (94.65.22.12) If, in a working OpenLDAP installation with dynamic configuration, we inadvertently change the value of: olcTLSCertificateKeyFile: /path/to/key.pem to some invalid value, like: olcTLSCertificateKeyFile: /path/to/non/existing/key.pem then OpenLDAP Server continues to work (and we see no error message whatsoever), but if it is stopped, it refuses to restart. In the logs, while OpenLDAP is starting, we see: Feb 11 16:20:44 vdev slapd[15272]: main: TLS init def ctx failed: -1 and then service is immediately stopped. OpenLDAP Server should check whether the new parameter value points to an existing file, and, if not, it should refuse to change the attribute value. For a more complete procedure, it could also check whether the indicated file includes a valid certificate, suitable for this parameter, and, again, if not, it should refuse to change the attribute value. Also, in case it's possible that one of the TLS File parameters gets an invalid value (e.g. through slapadd), then a meaningful message during startup failure would be helpful, e.g.: "olcTLSCertificateKeyFile: File /path/to/key.pem not found or contains invalid certificate."

1 0

Re: (ITS#7162)
by hyc＠symas.com 12 Feb '12

12 Feb '12

hyc(a)symas.com wrote: > Howard Chu wrote: >> michael(a)stroeder.com wrote: >>> FWIW: >>> With back-mdb I ran 64 iterations without this error. >>> Testing with back-bdb and back-hdb the failure occurs on my system after max. >>> 10 iterations. >> >> This sounds like it may be related to the issue I've been chasing down with >> ITS#7052. It appears that this bug was caused by the patch for ITS#6024. I'm >> about to revert that patch in master. Please re-test with d4b3c2af ... >> > Hm, no, that patch seemed to have no bearing here. But reverting ba4366eae > appears to clear things up. Now I just need to re-read test054 to see if this > is really what we want. > Just to sum up - this bug was a regression caused by ba4366eae. That patch was written in an attempt to resolve other bugs related to ITS#7052. But the actual cause of the remaining ITS#7052 issues was the ITS#6024 patch. Both of these patches have been reverted in git master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7162)
by hyc＠symas.com 12 Feb '12

12 Feb '12

Howard Chu wrote: > michael(a)stroeder.com wrote: >> FWIW: >> With back-mdb I ran 64 iterations without this error. >> Testing with back-bdb and back-hdb the failure occurs on my system after max. >> 10 iterations. > > This sounds like it may be related to the issue I've been chasing down with > ITS#7052. It appears that this bug was caused by the patch for ITS#6024. I'm > about to revert that patch in master. Please re-test with d4b3c2af ... > Hm, no, that patch seemed to have no bearing here. But reverting ba4366eae appears to clear things up. Now I just need to re-read test054 to see if this is really what we want. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

ITS#6024
by hyc＠symas.com 12 Feb '12

12 Feb '12

This patch has been partially reverted in master. The cookies are required for MMR loop detection. If it's confusing you when you analyze logs, too bad. Don't make "cosmetic" changes to working code. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7162)
by hyc＠symas.com 12 Feb '12

12 Feb '12

michael(a)stroeder.com wrote: > FWIW: > With back-mdb I ran 64 iterations without this error. > Testing with back-bdb and back-hdb the failure occurs on my system after max. > 10 iterations. This sounds like it may be related to the issue I've been chasing down with ITS#7052. It appears that this bug was caused by the patch for ITS#6024. I'm about to revert that patch in master. Please re-test with d4b3c2af ... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7162)
by michael＠stroeder.com 12 Feb '12

12 Feb '12

FWIW: With back-mdb I ran 64 iterations without this error. Testing with back-bdb and back-hdb the failure occurs on my system after max. 10 iterations.

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs