openldap-devel

openldap-devel@openldap.org

1425 discussions

LMDB dead process detection
by Howard Chu 18 Jul '13

18 Jul '13

There's been a long-running discussion about the need to have APIs in liblmdb for displaying the reader table and clearing out stale slots. Quite a few open questions on the topic: 1) What should the API look like for examining the table? My initial instinct is to provide an iterator function that returns info about the next slot each time it's called. Not sure that this is necessary or most convenient though. Another possibility is just a one-shot function that walks the table itself and dumps the output as a formatted string to stdout, stderr, or a custom output callback. 2) What should APIs look like for clearing out a stale slot? Should it just be implicit inside the library, with no externally visible API? I.e., should the library periodically check on its own, with no outside intervention? Or should there be an API that lets a user explicitly request a particular slot to be freed? This latter sounds pretty dangerous, since freeing a slot that's actually still in use would allow a reader's view of the DB to be corrupted. 3) What approach should be used for automatic detection of stale slots? Currently we record the process ID and thread ID of a reader in the table. It's not clear to me that the thread ID has anything more than informational value. Since we register a per-thread destructor for slots, exiting threads should never be leaving stale slots in the first place. I'm also not sure that there are good APIs for an outside caller to determine the liveness of a given thread ID. The process ID is also prone to wraparound; it's still very common for Linux systems to use 15 bit process IDs. So just checking that a pid is still alive doesn't guarantee that it's the same process that was using an LMDB environment at any point in time. We have two main approaches to work around this latter issue: A) set a byte range lock for every process attached to the environment. This is what slapd's alock.c already does, which is used with BDB- and LDBM- based backends. This is fairly portable code, and has the desirable property that file locks automatically go away when a process exits. But: a) On Windows, the OS can take several minutes to clean up the locks of an exited process. So just checking for presence of a lock could erroneously consider a process to be alive long after it had actually died. b) file lock syscalls are fairly slow to execute. If we are checking liveness frequently, there will be a noticeable performance hit. Their performance also degrades exponentially with the number of processes locking concurrently, and degrades further still if networked filesystems are involved. c) This approach won't tell us if a process is in Zombie state. B) check process ID and process start time. This appears to be a fairly reliable approach, and reasonably fast, but there is no POSIX standard API for obtaining this process information. Methods for obtaining the info are fairly well documented across a variety of platforms (AIX, HPUX, multiple BSDs, Linux, Solaris, etc.) but they are all different. It appears that we can implement this compactly for each of the systems, but it means carrying around a dozen or so different implementations. Also, assuming we want to support shared LMDB access across NFS (as discussed in an earlier thread), it seems we're going to have to use a lock-based solution anyway, since process IDs won't be meaningful across host boundaries. We can implement approach (A) fairly easily, with no major repercussions. For (B) we would need to add a field to the reader table records to store the process start time. (Thus a lockfile format change.) (note: performance of fcntl locks vs checking process start time was measured with some simple code on my laptop running Linux. These functions are all highly OS-dependent, so the perf ratios may vary quite a lot from system to system.) The relative performance may not even be an issue in general, since we would only need to trigger a scan if a writer actually finds that some reader txn is preventing it from using free pages from the freeDB. Most of the time this wouldn't be happening. But if there were a legitimate long running read txn (e.g., for mdb_env_copy) we may find ourselves checking fairly often. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

BerkeleyDB support EOL?
by Gavin Henry 15 Jul '13

15 Jul '13

Hi all, Hope every one is good? What's the timescale for dropping bdb and defaulting to lmdb? Kurt, what ever happened to that Google server? Thanks. -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry(a)suretec.co.uk Open Source. Open Solutions(tm). http://www.suretecsystems.com/ Suretec Systems is a limited company registered in Scotland. Registered number: SC258005. Registered office: 24 Cormack Park, Rothienorman, Inverurie, Aberdeenshire, AB51 8GL. Subject to disclaimer at http://www.suretecgroup.com/disclaimer.html Do you know we have our own VoIP provider called SureVoIP? See http://www.surevoip.co.uk Did you see our API? http://www.surevoip.co.uk/api

2 2

LMDB proposed changes
by Howard Chu 12 Jul '13

12 Jul '13

Summarizing some discussions from IRC... The hardcoded limit on the size of the dirty page list in a transaction is a problem, there should not be limits on the effective size of a transaction. The plan is to change LMDB's disk page format to include the txnID in the page header. This way, when the dirty page list gets full we can flush it to disk without losing track of which pages were dirtied. Then if a subsequent access in the same txn revisits one of these pages, when we read it back from the DB we'll know that it came from the current txn and doesn't need to be copied again before making further modifications. The P_DIRTY bit in the page header will no longer be needed - if the txnID matches, the page can be used directly. If not, the page is clean and a new page must be allocated before writing. For WRITEMAP mode the dirty page list can be completely eliminated, the only reason we keep it now is to know which pages' P_DIRTY bit we need to clear at commit time. Increasing the size of the page header by 8 bytes is a bit annoying, this will require a full slapcat/slapadd reload of existing back-mdb databases. It would be nice if we can avoid this but I don't see how. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 1

NFS-shared LMDB?
by Howard Chu 10 Jul '13

10 Jul '13

It occurs to me that there is the potential to support an interesting use case with LMDB when the database resides on remote shared storage. In the context of slapd, you could run multiple read-only slapds concurrent with a single read-write slapd on a single database. The current liblmdb would need a couple small modifications to make this safe - an option to use fcntl(LOCK) when obtaining a reader slot, and an msync() when writing to a reader slot, to force reader lock table changes back to the server before progressing on a read txn. With an appropriate sharding director (like the feature recently added to back-meta) you could arrange so that each slapd instance serves reads for a distinct portion of the overall database. Then each host's memory would be caching a distinct set of data, maximizing cache effectiveness. The DB size could then grow arbitrarily large, and you simply add more machines/RAM/slapds as needed to keep serving from cache. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

3 8

Time fo another RE24 testing call (OpenLDAP 2.4.36)?
by Michael Ströder 04 Jul '13

04 Jul '13

HI! What are plans to cut a 2.4.36 release? Given the fact that there are important changes in RE24 for back-mdb what's the plan to release them? Ciao, Michael.

2 1

Re: openldap.git branch mdb.master updated. 21da623bf40dc21f89c3172c523d094075e0824b
by Hallvard Breien Furuseth 02 May '13

02 May '13

openldap-commit2devel(a)OpenLDAP.org writes: > Allow reading freelist while working on it > > The circular dependency issues appear to have been resolved. > Still, need to watch closely, maybe revert this change if > problems arise. IIRC this could get page-hungry when I tried that with a fragented DB, and with large changes which needed overflow pages and used them up for non-freelist items. One of those big LDIFs at ada, I think. Something like: Commit wants a big enough range for an overflow page for mt_free_pgs, maybe finds and allocates one, but the freelist changes eat too many freelist items and will need a bigger page one later. It doesn't free the useless overflow page it already allocated. Repeat. Anyway, I think this change needs code to give pages back to me_pghead, or revert to an earlier {me_pglast, me_pghead}, or something like that. Or maybe someday clean up nested txns, support them with WRITEMAP, and use those for the freelist loop since they support reverting a change. -- Hallvard

1 0

Current master broken with back-mdb
by Quanah Gibson-Mount 07 Apr '13

07 Apr '13

Just FYI, current master is utterly broken with back-mdb. Most sub-databases are missing. mdb database with 2.4.35: zimbra@zre-ldap002:~/data/ldap/mdb/db$ mdb_stat -eaf . | grep "Status of" Status of Main DB Status of DKIMIdentity Status of DKIMSelector Status of ad2i Status of cn Status of displayName Status of dn2i Status of entryCSN Status of entryUUID Status of givenName Status of id2e Status of mail Status of objectClass Status of sn Status of uid Status of zimbraACE Status of zimbraAuthKerberos5Realm Status of zimbraCalResBuilding Status of zimbraCalResCapacity Status of zimbraCalResFloor Status of zimbraCalResRoom Status of zimbraCalResSite Status of zimbraDomainAliasTargetId Status of zimbraDomainName Status of zimbraForeignPrincipal Status of zimbraId Status of zimbraMailAlias Status of zimbraMailCatchAllAddress Status of zimbraMailDeliveryAddress Status of zimbraMailForwardingAddress Status of zimbraMailHost Status of zimbraMailTransport Status of zimbraMemberOf Status of zimbraShareInfo Status of zimbraSharedItem Status of zimbraUCServiceId Status of zimbraVirtualHostname Status of zimbraVirtualIPAddress Status of zimbraYahooId mdb database with master: zimbra@zre-ldap001:~/data/ldap/mdb/db$ mdb_stat -eaf . | grep "Status of" Status of Main DB Status of DKIMIdentity Status of DKIMSelector Status of ad2i Status of cn --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

2 3

lmdb.h defining mode_t
by Hallvard Breien Furuseth 31 Mar '13

31 Mar '13

openldap-commit2devel(a)OpenLDAP.org writes: > commit 0c568117cea6764c3f12a0d69ddf54ee467d2dec > Author: Howard Chu <hyc(a)symas.com> > Date: Sun Mar 31 08:10:43 2013 -0800 > > MSVC doesn't define mode_t Nor should ldmb.h. That can break if it meets a header from another package doing almost the same thing. E.g. #define mode_t unsigned int. Try: #ifndef _MSC_VER typedef mode_t mdb_mode_t; #else typedef int mdb_mode_t; #endif ... int mdb_env_open(... mdb_mode_t mode); -- Hallvard

1 0

RE24 Testing call #1 (OpenLDAP 2.4.35)
by Quanah Gibson-Mount 29 Mar '13

29 Mar '13

Current RE24 is ready for testing for the 2.4.35 release. Thanks! --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

2 1

Re: Notes
by Howard Chu 11 Mar '13

11 Mar '13

Brett Maxfield wrote: > On 12/03/2013, at 8:04 AM, Howard Chu <hyc(a)symas.com> wrote: > >> We're talking about deprecating the BDB-based backends Real Soon Now. I suppose this should be done in OpenLDAP 2.5, but we've pulled a lot into 2.4 that we had originally slated for 2.5, which keeps pushing any actual 2.5 work further into the future as we keep extending 2.4. > > I imagine dropping bdb completely will leave a multitude of systems which apart from suffering from distribution lag, will now suffer additionally from end of line itus, as they will be unable to patch to the newest minor version if bdb is dropped entirely in some future 2.4.x version.. > > I'd say deprecate it in 2.4.x great, but dont drop it completely until > 2.5. If you are in a rush, drop bdb backend from the default configure options to get the point across, or print a warning or some such notice in configure, so package maintainers get the hint before 2.6 is released.. > > Given users have the option to drop bdb today, why force the issue? > > Most people don't expect huge changes in behavior, for only minor version releases. > > If 2.4.x is stable and feature complete / no new features, then at some point there will be a natural finish, when there will be few changes for 2.4.x right? Bug fix only tree for 2.4 would leave resources to work on 2.5 release ? I only said "deprecating", I didn't say anything about dropping. back-bdb has already been turned off by default in HEAD. Looks like it's still on by default in RE24, that seems to be an oversight since we'd been recommending back-hdb for a long time already. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel