openldap-bugs

openldap-bugs@openldap.org

1 participants
20962 discussions

Re: (ITS#7364) mdb: clean up POSIX semaphores on environment close.
by h.b.furuseth＠usit.uio.no 06 Sep '12

06 Sep '12

Howard Chu writes: >h.b.furuseth(a)usit.uio.no wrote: >> This is worse with a database with is intended to be used by >> different users (A and B): > (...) >> mdb_env_setup_locks() can sem_unlink both semaphores before doing >> anything else, so that reopening a database as root will clean up. >> Drop the error checks of sem_unlink (so both get called), instead >> use O_EXCL in sem_open(,O_CREAT,,). Unless I'm missing something, >> the error checks just work like an emulation of O_EXCL anyway. > > The sem_unlink() and sem_open() sequence isn't ideal, certainly. I would > prefer to just use the existing semaphore. I just realized reusing the semaphores is a security problem: Anyone who can compute/read the semaphore names, can create the semaphores owned by themselves and then muck with them to sabotage someone else's mdb. I suppose the same goes for SysV semaphores. I don't know what the accepted way to deal with that is, but I suggest to unlink + O_EXCL as above for now, document the multi-uid restriction, and punt the rest until anyone who cares shows up. > Another possibility is to just use SysV semaphores instead of POSIX > semaphores. Then you can use the ipcs(1) command to manually cleanup. > (...) [later] > I'm also reminded that there is no defined behavior for SysV > semaphores in threads, they are only speciried for inter-process > synchronization. So forget that... I expect the work-around is to use them for IPC only, and wrap them in process-local mutexes in the MDB_env for thread sync. -- Hallvard

1 0

(ITS#7381) Dynamically added chaining configuration stops working after restart
by ian＠uns.ac.rs 05 Sep '12

05 Sep '12

Full_Name: Ivan Nejgebauer Version: 2.4.32 OS: CentOS 5 URL: ftp://ftp.openldap.org/incoming/ivannejgebauer-120905.tgz Submission from: (NULL) (2001:4170:2000:2:213:d3ff:fecf:b9ea) If a minimal chaining configuration is dynamically added to a consumer server in a provider/consumer pair running delta-syncrepl, it will stop working when the consumer is restarted. The configuration is modelled after the one used in test022 in the OpenLDAP source distribution: dn: olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config objectClass: olcOverlayConfig objectClass: olcChainConfig olcOverlay: {0}chain dn: olcDatabase={0}ldap,olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config objectClass: olcLDAPConfig objectClass: olcChainDatabase olcDatabase: {0}ldap olcDbIDAssertBind: bindmethod=simple binddn="cn=Manager,dc=example,dc=com" credentials="secret" mode=self olcDbURI: "ldap://localhost:6389" Scripts and data for replicating the issue are in the referenced archive. Quick summary: # make clean-all master replica # ./start-master.sh # ./start-replica.sh # ./replica-apply.sh pmod-add.ldif # returns referral # ./replica-conf.sh chain.ldif # adds chaining configuration # ./replica-apply.sh pmod-add.ldif # chains to master # ./stop-replica.sh # ./start-replica.sh # ./replica-apply.sh pmod-del.ldif # returns referral, but shouldn't I have tested both 2.4.32 and git master (54a111907b48900e1ce5de12a9d87818a334423d).

1 0

(ITS#7380) Open LDAP and Oracle Backend
by iliyas.nadaf＠gmail.com 04 Sep '12

04 Sep '12

Full_Name: iliyas Version: openldap-2.4.31 OS: solaris-x86 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (59.160.74.254) Hi, I need to develope "ladap client in C" with OpenLdap source and backend as Oracle database 11gr2. If it is possible suggest some resources. I have done it properly with BDB. Thanks, Iliyas

1 0

(ITS#7379) Open LDAP and Oracle Backend
by iliyas.nadaf＠gmail.com 04 Sep '12

04 Sep '12

1 0

Re: (ITS#7378) Slapd hangs on bdb write lock
by nikolai＠net24.co.nz 04 Sep '12

04 Sep '12

Thanks guys - I think we will look at going to MDB as well now. On 4/09/2012, at 8:42 AM, Quanah Gibson-Mount wrote: > --On Monday, September 03, 2012 6:35 PM +0000 hyc(a)symas.com wrote: > >> Nikolai Schupbach wrote: >>> Hi Howard, >>> >>> Thank you very much for the explanation. What BDB version would you >> recommend. Obviously I have quite a few options and would like to use a >> version that is known to be very solid. >> >> I believe 4.7.25 + all 4 of its official patches was pretty stable. >> http://www.oracle.com/technetwork/products/berkeleydb/patch-088170.html >> >> I've done limited testing with 4.8.30, 5.1.19, and 5.3.21. At this point >> I'm no longer tracking BDB revisions since MDB has superior performance >> while using 1/4 as much RAM and requiring no tuning. > > We've been using BDB 4.7.25+all 4 patches without issue for several years as well. However, I will also note that we are now switching over to MDB as well for our production services starting with OpenLDAP 2.4.32. > > --Quanah > > > -- > > Quanah Gibson-Mount > Sr. Member of Technical Staff > Zimbra, Inc > A Division of VMware, Inc. > -------------------- > Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#7378) Slapd hangs on bdb write lock
by quanah＠zimbra.com 03 Sep '12

03 Sep '12

--On Monday, September 03, 2012 6:35 PM +0000 hyc(a)symas.com wrote: > Nikolai Schupbach wrote: >> Hi Howard, >> >> Thank you very much for the explanation. What BDB version would you > recommend. Obviously I have quite a few options and would like to use a > version that is known to be very solid. > > I believe 4.7.25 + all 4 of its official patches was pretty stable. > http://www.oracle.com/technetwork/products/berkeleydb/patch-088170.html > > I've done limited testing with 4.8.30, 5.1.19, and 5.3.21. At this point > I'm no longer tracking BDB revisions since MDB has superior performance > while using 1/4 as much RAM and requiring no tuning. We've been using BDB 4.7.25+all 4 patches without issue for several years as well. However, I will also note that we are now switching over to MDB as well for our production services starting with OpenLDAP 2.4.32. --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#7378) Slapd hangs on bdb write lock
by hyc＠symas.com 03 Sep '12

03 Sep '12

Nikolai Schupbach wrote: > Hi Howard, > > Thank you very much for the explanation. What BDB version would you recommend. Obviously I have quite a few options and would like to use a version that is known to be very solid. I believe 4.7.25 + all 4 of its official patches was pretty stable. http://www.oracle.com/technetwork/products/berkeleydb/patch-088170.html I've done limited testing with 4.8.30, 5.1.19, and 5.3.21. At this point I'm no longer tracking BDB revisions since MDB has superior performance while using 1/4 as much RAM and requiring no tuning. > Sincerely, > Nikolai Schupbach > > On 3/09/2012, at 9:45 PM, Howard Chu wrote: > >> nikolai(a)net24.co.nz wrote: >>> Full_Name: Nikolai Schupbach >>> Version: 2.4.31 >>> OS: FreeBSD >>> URL: ftp://ftp.openldap.org/incoming/ >>> Submission from: (NULL) (202.78.158.60) >>> >>> >>> We are experiencing frequent hangs in slapd. Once hung we can continue to >>> connect, but all searches will just hang indefinitely until we kill -9 the slapd >>> process and restart it. The directory is used for mail routing and we have been >>> migrating to it from an existing directory server over the last 3 weeks - we >>> have noted the busier the directory becomes the more often it hangs (now once >>> every 2 days). >>> >>> We have one master and 10 syncrepl read only replicas - the master is used >>> mainly for writes and has not hung yet, but most of the replicas have hung at >>> least once. The replicas receive anywhere between 50 to 300 searches/sec, while >>> the master would only get 1/sec. There are 45k entries in the directory. >>> >>> We are running: >>> >>> FreeBSD 8.3/9.0 x64 >>> OpenLDAP 2.4.31 >>> Berkeley DB 4.6.21 >>> >>> The old directory we are migrating from has the same load and is also running >>> OpenLDAP, but has been rock solid for 5 years. It is running Berkeley DB 4.3.29 >>> and OpenLDAP 2.3.27. >>> >>> We have managed to collect db_stat lock information, which indicates the same >>> issue each time - a write lock on dn2id.bdb. >> >> It's more than that. Your db_stat shows that a single thread has 3 active >> transactions. This should never happen: >> >> 8000a85e dd= 0 locks held 2 write locks 0 pid/thread 88000/34386526336 >> 8000a85e READ 1 HELD 0xb19a8 len: 9 data: 40xa800000000000000 >> 8000a85e READ 1 HELD 0xb26c8 len: 9 data: 60xa800000000000000 >> 8000a85f dd= 0 locks held 8 write locks 4 pid/thread 88000/34386526336 >> 8000a85f READ 1 WAIT dn2id.bdb page 559 >> 8000a85f READ 1 HELD dn2id.bdb page 768 >> 8000a85f WRITE 2 HELD dn2id.bdb page 1362 >> 8000a85f READ 2 HELD dn2id.bdb page 1362 >> 8000a85f WRITE 2 HELD dn2id.bdb page 1353 >> 8000a85f READ 2 HELD dn2id.bdb page 1353 >> 8000a85f WRITE 2 HELD dn2id.bdb page 933 >> 8000a85f READ 1 HELD dn2id.bdb page 933 >> 8000a85f WRITE 4 HELD dn2id.bdb page 219 >> 80001047 dd=28 locks held 1 write locks 1 pid/thread 88000/34386526336 >> 80001047 WRITE 1 HELD dn2id.bdb page 559 >> >> I would first recommend changing from BDB 4.6.21 to some other version. There >> are no code paths in back-bdb where we would ever return without either >> committing or aborting the current transactions, so this appears to be a BDB >> bug, not an OpenLDAP bug. >> >>> We have also collected the backtrace for all the threads which I have uploaded >>> to: >>> >>> ftp://ftp.openldap.org/incoming/nikolai-gdb-120902.txt >>> >>> The full db_stat output is located at: >>> >>> ftp://ftp.openldap.org/incoming/nikolai-dbstat-120902.txt >> >> -- >> -- Howard Chu >> CTO, Symas Corp. http://www.symas.com >> Director, Highland Sun http://highlandsun.com/hyc/ >> Chief Architect, OpenLDAP http://www.openldap.org/project/ > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7378) Slapd hangs on bdb write lock
by nikolai＠net24.co.nz 03 Sep '12

03 Sep '12

Hi Howard, Thank you very much for the explanation. What BDB version would you recommend. Obviously I have quite a few options and would like to use a version that is known to be very solid. Sincerely, Nikolai Schupbach On 3/09/2012, at 9:45 PM, Howard Chu wrote: > nikolai(a)net24.co.nz wrote: >> Full_Name: Nikolai Schupbach >> Version: 2.4.31 >> OS: FreeBSD >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (202.78.158.60) >> >> >> We are experiencing frequent hangs in slapd. Once hung we can continue to >> connect, but all searches will just hang indefinitely until we kill -9 the slapd >> process and restart it. The directory is used for mail routing and we have been >> migrating to it from an existing directory server over the last 3 weeks - we >> have noted the busier the directory becomes the more often it hangs (now once >> every 2 days). >> >> We have one master and 10 syncrepl read only replicas - the master is used >> mainly for writes and has not hung yet, but most of the replicas have hung at >> least once. The replicas receive anywhere between 50 to 300 searches/sec, while >> the master would only get 1/sec. There are 45k entries in the directory. >> >> We are running: >> >> FreeBSD 8.3/9.0 x64 >> OpenLDAP 2.4.31 >> Berkeley DB 4.6.21 >> >> The old directory we are migrating from has the same load and is also running >> OpenLDAP, but has been rock solid for 5 years. It is running Berkeley DB 4.3.29 >> and OpenLDAP 2.3.27. >> >> We have managed to collect db_stat lock information, which indicates the same >> issue each time - a write lock on dn2id.bdb. > > It's more than that. Your db_stat shows that a single thread has 3 active > transactions. This should never happen: > > 8000a85e dd= 0 locks held 2 write locks 0 pid/thread 88000/34386526336 > 8000a85e READ 1 HELD 0xb19a8 len: 9 data: 40xa800000000000000 > 8000a85e READ 1 HELD 0xb26c8 len: 9 data: 60xa800000000000000 > 8000a85f dd= 0 locks held 8 write locks 4 pid/thread 88000/34386526336 > 8000a85f READ 1 WAIT dn2id.bdb page 559 > 8000a85f READ 1 HELD dn2id.bdb page 768 > 8000a85f WRITE 2 HELD dn2id.bdb page 1362 > 8000a85f READ 2 HELD dn2id.bdb page 1362 > 8000a85f WRITE 2 HELD dn2id.bdb page 1353 > 8000a85f READ 2 HELD dn2id.bdb page 1353 > 8000a85f WRITE 2 HELD dn2id.bdb page 933 > 8000a85f READ 1 HELD dn2id.bdb page 933 > 8000a85f WRITE 4 HELD dn2id.bdb page 219 > 80001047 dd=28 locks held 1 write locks 1 pid/thread 88000/34386526336 > 80001047 WRITE 1 HELD dn2id.bdb page 559 > > I would first recommend changing from BDB 4.6.21 to some other version. There > are no code paths in back-bdb where we would ever return without either > committing or aborting the current transactions, so this appears to be a BDB > bug, not an OpenLDAP bug. > >> We have also collected the backtrace for all the threads which I have uploaded >> to: >> >> ftp://ftp.openldap.org/incoming/nikolai-gdb-120902.txt >> >> The full db_stat output is located at: >> >> ftp://ftp.openldap.org/incoming/nikolai-dbstat-120902.txt > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7364) mdb: clean up POSIX semaphores on environment close.
by h.b.furuseth＠usit.uio.no 03 Sep '12

03 Sep '12

Howard Chu writes: > h.b.furuseth(a)usit.uio.no wrote: >> Reopening this. >> >> >> This is worse with a database with is intended to be used by >> different users (A and B): > > This is pretty much never a use case we would worry about. In most > applications, a single userID creates and operates on the DB. I'm fine with just documenting that as a restriction on some systems. If not: >> (...) >> The work-around I can think of is a "multi-uid" mode which instead >> resets the semaphore with sem_post() if sem_getvalue() returns 0. >> I don't know how ugly that is considered to be. Could ask >> comp.programming.unix, or check what Berkeley DB does. > > That would be OK in general. It still leaves the question of how to remove the > semaphore if the DB is being destroyed. But it's probably not worth the > trouble to worry about this so much. Those OSes should just get their act > together and support the POSIX process-shared mutexes. > >> This mode >> should use mode 0666 for the semaphores (temporarily setting umask >> 0, yuck), > > Definitely not. The caller specifies a mode; if they want 0666 they > should configure it as such. 0666 would likely be wrong for the database file(s). But this flag could just as well consist of specifying a mode parameter for the semaphores. It's a threaded library doing umask() I dislike. And, as you say, the need to remove the semaphore afterwards. >> or it should not sem_unlink() since next user may create >> the semaphores with a group which gives the wrong users access. > > Same as today where running slapadd with the wrong uid causes trouble > for the following slapd. The answer is obvious: use the right uid when > accessing the DB. We're talking about the case where there is no single "right" uid. Not relevant for slapd, only libmdb by itself. >> Other matters with the current implementation - I'll patch these: >> >> mdb_env_excl_lock() need not retry getting a non-exclusive lock when >> closing. mdb_env_close() can pass *excl = -1 to tell it not to. >> >> mdb_env_setup_locks() can sem_unlink both semaphores before doing >> anything else, so that reopening a database as root will clean up. >> Drop the error checks of sem_unlink (so both get called), instead >> use O_EXCL in sem_open(,O_CREAT,,). Unless I'm missing something, >> the error checks just work like an emulation of O_EXCL anyway. > > The sem_unlink() and sem_open() sequence isn't ideal, certainly. I would > prefer to just use the existing semaphore. ...followed by 'if (sem_getvalue() shows 0) sem_post()' as above, then. -- Hallvard

1 0

Re: (ITS#7378) Slapd hangs on bdb write lock
by hyc＠symas.com 03 Sep '12

03 Sep '12

michael(a)stroeder.com wrote: > This is a cryptographically signed message in MIME format. > > --------------ms080100030105010600070605 > Content-Type: text/plain; charset=ISO-8859-1 > Content-Transfer-Encoding: quoted-printable > > A couple of days ago I had a hang with OpenLDAP 2.4.32 / back-hdb running= > on > Debian Squeeze, self-compiled against BDB 4.8.30. It seemed Database was > locked as restarting slapd of even rebooting OS did not help. Unfortunate= > ly I > had to bring up the system as fast as possible and could not examine the = > problem. db_recover will always return the DB to a usable state and reset any DB locks. (It completely deletes the lock region, so there cannot be any stale locks after it runs.) > The system has only 200 entries and not much load yet. I had renamed entr= > ies > with web2ldap when all 4 masters (4-way MMR) locked up one after the othe= > r. > So there seem to be lockup problems in 2.4.32. The only way to know if you're seeing the same problem as the original poster is if you provide db_stat -CA and gdb trace output, like the original poster did. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs