openldap-devel January 2013

openldap-devel@openldap.org

7 participants
8 discussions

Re: slapd deadlock bug ITS#7296
by Howard Chu 31 Jan '13

31 Jan '13

Richard Silverman wrote: > FYI: we have just observed another slapd deadlock with the same signature > as the older one I indicated above, rather than the newer one which > started this thread (all worker threads blocked on condition waits for > database rather than network access). The F5 load balancer is now not > involved. Again, this is RHEL 6.3 using this package: > > openldap-servers-2.4.23-26.el6_3.2.x86_64 > > I am including a stack dump and gdb trace of all threads below. This is most likely ITS#7222, your release is too old to contain the fix for this. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

RE24 Testing call #2 (OpenLDAP 2.4.34)
by Quanah Gibson-Mount 31 Jan '13

31 Jan '13

Current RE24 is ready for testing for the 2.4.34 release. Please note that if you are using back-mdb, any existing database *must* be reloaded via slapcat/slapadd if you are going to test it against this build. Thanks! --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

2 1

MDB mapsize changes
by Hallvard Breien Furuseth 17 Jan '13

17 Jan '13

Current mdb.master can change MDB_meta.mm_mapsize when another process has the database open. That can break when one MDB_env grows the DB file beyond another MDB_env's me_mapsize. Coexisting MDB_env's must use the same mapsize. Any mm_mapsize change must be written while mdb_env_open() has the exlusive lock. For writing the mapsize: - I presume the cleanest way wouldbe to write it to the not-current meta page. But mdb_page_alloc() and mdb_txn_commit()'s "Delete IDLs" loop look like they expect each recent txnid to have a freelist entry. Is that right? page_alloc uses MDB_SET, should maybe be MDB_SET_RANGE. I think older libmdb's seeing such a missing entry would grow the map instead of grabbing the next freelist entry. - Or overwrite (current meta).mm_mapsize, but then I don't know what to do at a failed write. env_open cannot get around that by writing back an older txnid like mdb_env_write_meta() does. Maybe just try to write back the old mapsize before returning failure. BTW, write_meta with WRITEMAP should undo the change if msync fails. - Or do mdb_txn_begin(); mdb_txn_commit(); with the new mm_mapsize and a dummy freelist entry. Simple enough, if excessive. As for determining the mapsize, I think it goes something like branch "mdb/mapsize" in <http://folk.uio.no/hbf/OpenLDAP/openldap.git>. -- Hallvard

1 0

Avoiding read() of meta pages (was: Fwd: adding support to mdb)
by Hallvard Breien Furuseth 15 Jan '13

15 Jan '13

In openldap-technical, Quanah Gibson-Mount wrote: > As per Howard Chu (Author of MDB, Primary OpenLDAP Developer): > > ------------------------------------------------------------- > Full details are in the paper. > http://www.openldap.org/pub/hyc/mdm-paper.pdf > > MDB assumes a unified buffer cache. See section 3.1, references 17, 18, and > 19. > > Note that this requirement can be relaxed in the current version of the > library. If you create the environment with the MDB_WRITEMAP option then > all reads and writes are performed using mmap, so the file buffer cache is > irrelevant. Of course then you lose the protection that the read-only map > offers. That's not quite true. mdb_env_open() does a read() of the meta pages. I presume this can only be a problem when other processes have the database open? In that situation, I think the read() can be avoided by maintaining a copy of the relevant MDB_meta information in the lock file. The read() only needs enough info to know how to map the data file. This would require a version increase for the lock file and programs using it, but not for the database file. -- Hallvard

2 2

ITS#7473 back-mdb search
by Howard Chu 13 Jan '13

13 Jan '13

/* There are two approaches to the search function: * 1) walk the list of filter candidates, and see which are in scope. * 2) walk the scope tree, and see which are in the filter candidates. * (1) is faster if the filter candidate list is smaller than the scope tree, * and vice versa. Currently only (1) is implemented. * * We don't know the actual size of the subtree, we only know the count of * onelevel children. For subtree searches we take a guess. * * Due to the fact pagedResults cookie is only big enough to store a single * entryID, if pagedResults is in use we will always do (1). For (2) to work * reliably we need to use the parent's entryID, to avoid losing our place * if the target entry is moved or deleted. We'd also need to store the * count of where we were under the parent. I.e., we need the cookie to be * two words long. We can examine that in the future, but the pagedResults * cookie definition is global to all of slapd so the change would impact * other backends and overlays. */ We could alter the dn2id index format, and maintain a numSubordinates counter there. That would avoid the need to guess. Otherwise I figure we fudge a guess based on the number of onelevel children and the depth of the baseDN (relative to the suffix). I.e., the longer the DN, the smaller the guess, relative to the total number of entries in the DB. Naturally I'd prefer to avoid altering the dn2id index format. Suggestions? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

3 5

connection management changes
by Howard Chu 09 Jan '13

09 Jan '13

Currently we have a single select(or epoll) loop in daemon.c that lists for all readable and writable sockets, then passes events off to the thread pool for processing. We listen for writable sockets if a write attempt returns incomplete. There's a pair of mutexes and condition variables used to synch up here between the writing threads and the listener thread. It's quite a lot of lock overhead. As far as I can tell the main reason we do this is so that we can stop a writer thread on demand instead of having it just block forever in write(). We could make the listener's job a lot easier if we only have it listen for readable sockets, and make each writer thread do its own poll. It would need to poll on two descriptors - the one it's waiting to write on, and a pipe used by the listener to terminate the poll. That pipe could be signalled by e.g. the listener writing a byte to it; all writer threads could poll for its read status. (One question here - if multiple threads are polling the same descriptor, do they all receive the wakeup event?) -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

3 4

send_ldap_ber mutex overhead
by Howard Chu 07 Jan '13

07 Jan '13

The attached patch avoids grabbing conn->c_mutex unnecessarily. Testing on my laptop with test008 and SLAPD_DEBUG=0 shows a 33% speedup. Before moving ahead with this patch, I'd like to verify that ITS#5835 doesn't reappear. Anyone able to test and report back? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

Re: slapd deadlock bug ITS#7296
by Howard Chu 02 Jan '13

02 Jan '13

Richard Silverman wrote: > I’ve done this, and it may well be epoll-specific; the test has now run > over twice as long as the longest it has ever required to produce the > deadlock. With this sort of bug no amount of waiting would make me sure, > but it seems likely. I’ll leave it running. > > epoll(7) specifically mentions the possibility of epoll_wait hanging even > though there is outstanding unread data on a socket, when using > edge-triggered operation, and I notice in daemon.c that you switch to > edge-triggered mode in the event that the client closes the connection (at > least that’s what the comment suggests): > > /* Don't keep reporting the hangup > */ > if ( SLAP_SOCK_IS_ACTIVE( tid, fd )) { > SLAP_EPOLL_SOCK_SET( tid, fd, EPOLLET ); > } > > Perhaps related? Indeed. Seems like ITS#5886 has resurfaced. You can get some insight into this looking at about commit 96192064f3a3daea994eb8293f0413def5379958 forward. I don't have time to dig further into it at the moment, Christmas dinner(s) calling... > >> Also, what kernel version(s) are you testing on? > > 2.6.32 (Red Hat Enterprise Linux) I see a lot of Linux-kernel email traffic about epoll bugs as well, but not sure which are relevant to this version. Just something to stay aware of. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 3

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel January 2013