openldap-devel

openldap-devel@openldap.org

1418 discussions

RE24 Testing call #1 (OpenLDAP 2.4.36)
by Quanah Gibson-Mount 31 Jul '13

31 Jul '13

Current RE24 is ready for testing for the 2.4.36 release. Thanks! --Quanah -- Quanah Gibson-Mount Lead Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

2 2

Clusters and scaling, again
by Howard Chu 25 Jul '13

25 Jul '13

It's been a little over 4 years since we worked with MySQL using their NDBCluster engine to develop back-ndb, in the hopes of developing a backend that could leverage horizontal scaling systems. After Oracle acquired MySQL (via acquisition of Sun) they terminated the project. NDB had some promise, although reports from the field indicate that folks felt it was too difficult to configure and deploy. It appears that NDB has not gained much traction anywhere in the past few years, and with Oracle cutting off our access to MySQL/NDB developers, back-ndb was effectively killed. In OpenLDAP 2.5 we'll be trying again, this time using HyperDex. http://hyperdex.org. HyperDex has a number of desirable properties - trivial deployment, automatic self-balancing of data, guaranteed data durability, configurable fault tolerance... There is an extension for it providing distributed transaction support as well (which I haven't looked into yet). It's also quite fast, much faster than any of the other commonly known NoSQL systems out there. It's also reliable by design, unlike systems like MongoDB (whose only consistency is that it consistently loses data...) I've been playing with it off and on for the past several months. I've also written an LMDB backend for it, available on github. https://github.com/hyc/HyperDex/tree/lmdb So using HyperDex for an OpenLDAP backend has some greater appeal, because it would be OpenLDAP code at both ends of the stack. If you're interested in working with NoSQL and large scale databases with OpenLDAP you should definitely take a look at this. I suspect it will be easier to write a slapd backend for this API than for NDB, particularly since it's free of the rigid column layouts that NDB/MySQL required. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Please test back-mdb in main
by Quanah Gibson-Mount 18 Jul '13

18 Jul '13

The work on large transactions, nested transactions, and reader tables in lmdb has come to a point where we'd like to confirm that people do not find issues when running slapd-mdb before rolling it into RE24. If people could download and build current master of OpenLDAP and do whatever tests against slapd-mdb you can think of in addition to the test suite, particularly across multiple OSes, that would be much appreciated. Thanks, Quanah -- Quanah Gibson-Mount Lead Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

LMDB dead process detection
by Howard Chu 18 Jul '13

18 Jul '13

There's been a long-running discussion about the need to have APIs in liblmdb for displaying the reader table and clearing out stale slots. Quite a few open questions on the topic: 1) What should the API look like for examining the table? My initial instinct is to provide an iterator function that returns info about the next slot each time it's called. Not sure that this is necessary or most convenient though. Another possibility is just a one-shot function that walks the table itself and dumps the output as a formatted string to stdout, stderr, or a custom output callback. 2) What should APIs look like for clearing out a stale slot? Should it just be implicit inside the library, with no externally visible API? I.e., should the library periodically check on its own, with no outside intervention? Or should there be an API that lets a user explicitly request a particular slot to be freed? This latter sounds pretty dangerous, since freeing a slot that's actually still in use would allow a reader's view of the DB to be corrupted. 3) What approach should be used for automatic detection of stale slots? Currently we record the process ID and thread ID of a reader in the table. It's not clear to me that the thread ID has anything more than informational value. Since we register a per-thread destructor for slots, exiting threads should never be leaving stale slots in the first place. I'm also not sure that there are good APIs for an outside caller to determine the liveness of a given thread ID. The process ID is also prone to wraparound; it's still very common for Linux systems to use 15 bit process IDs. So just checking that a pid is still alive doesn't guarantee that it's the same process that was using an LMDB environment at any point in time. We have two main approaches to work around this latter issue: A) set a byte range lock for every process attached to the environment. This is what slapd's alock.c already does, which is used with BDB- and LDBM- based backends. This is fairly portable code, and has the desirable property that file locks automatically go away when a process exits. But: a) On Windows, the OS can take several minutes to clean up the locks of an exited process. So just checking for presence of a lock could erroneously consider a process to be alive long after it had actually died. b) file lock syscalls are fairly slow to execute. If we are checking liveness frequently, there will be a noticeable performance hit. Their performance also degrades exponentially with the number of processes locking concurrently, and degrades further still if networked filesystems are involved. c) This approach won't tell us if a process is in Zombie state. B) check process ID and process start time. This appears to be a fairly reliable approach, and reasonably fast, but there is no POSIX standard API for obtaining this process information. Methods for obtaining the info are fairly well documented across a variety of platforms (AIX, HPUX, multiple BSDs, Linux, Solaris, etc.) but they are all different. It appears that we can implement this compactly for each of the systems, but it means carrying around a dozen or so different implementations. Also, assuming we want to support shared LMDB access across NFS (as discussed in an earlier thread), it seems we're going to have to use a lock-based solution anyway, since process IDs won't be meaningful across host boundaries. We can implement approach (A) fairly easily, with no major repercussions. For (B) we would need to add a field to the reader table records to store the process start time. (Thus a lockfile format change.) (note: performance of fcntl locks vs checking process start time was measured with some simple code on my laptop running Linux. These functions are all highly OS-dependent, so the perf ratios may vary quite a lot from system to system.) The relative performance may not even be an issue in general, since we would only need to trigger a scan if a writer actually finds that some reader txn is preventing it from using free pages from the freeDB. Most of the time this wouldn't be happening. But if there were a legitimate long running read txn (e.g., for mdb_env_copy) we may find ourselves checking fairly often. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

BerkeleyDB support EOL?
by Gavin Henry 15 Jul '13

15 Jul '13

Hi all, Hope every one is good? What's the timescale for dropping bdb and defaulting to lmdb? Kurt, what ever happened to that Google server? Thanks. -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry(a)suretec.co.uk Open Source. Open Solutions(tm). http://www.suretecsystems.com/ Suretec Systems is a limited company registered in Scotland. Registered number: SC258005. Registered office: 24 Cormack Park, Rothienorman, Inverurie, Aberdeenshire, AB51 8GL. Subject to disclaimer at http://www.suretecgroup.com/disclaimer.html Do you know we have our own VoIP provider called SureVoIP? See http://www.surevoip.co.uk Did you see our API? http://www.surevoip.co.uk/api

2 2

LMDB proposed changes
by Howard Chu 12 Jul '13

12 Jul '13

Summarizing some discussions from IRC... The hardcoded limit on the size of the dirty page list in a transaction is a problem, there should not be limits on the effective size of a transaction. The plan is to change LMDB's disk page format to include the txnID in the page header. This way, when the dirty page list gets full we can flush it to disk without losing track of which pages were dirtied. Then if a subsequent access in the same txn revisits one of these pages, when we read it back from the DB we'll know that it came from the current txn and doesn't need to be copied again before making further modifications. The P_DIRTY bit in the page header will no longer be needed - if the txnID matches, the page can be used directly. If not, the page is clean and a new page must be allocated before writing. For WRITEMAP mode the dirty page list can be completely eliminated, the only reason we keep it now is to know which pages' P_DIRTY bit we need to clear at commit time. Increasing the size of the page header by 8 bytes is a bit annoying, this will require a full slapcat/slapadd reload of existing back-mdb databases. It would be nice if we can avoid this but I don't see how. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 1

NFS-shared LMDB?
by Howard Chu 10 Jul '13

10 Jul '13

It occurs to me that there is the potential to support an interesting use case with LMDB when the database resides on remote shared storage. In the context of slapd, you could run multiple read-only slapds concurrent with a single read-write slapd on a single database. The current liblmdb would need a couple small modifications to make this safe - an option to use fcntl(LOCK) when obtaining a reader slot, and an msync() when writing to a reader slot, to force reader lock table changes back to the server before progressing on a read txn. With an appropriate sharding director (like the feature recently added to back-meta) you could arrange so that each slapd instance serves reads for a distinct portion of the overall database. Then each host's memory would be caching a distinct set of data, maximizing cache effectiveness. The DB size could then grow arbitrarily large, and you simply add more machines/RAM/slapds as needed to keep serving from cache. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

3 8

Time fo another RE24 testing call (OpenLDAP 2.4.36)?
by Michael Ströder 04 Jul '13

04 Jul '13

HI! What are plans to cut a 2.4.36 release? Given the fact that there are important changes in RE24 for back-mdb what's the plan to release them? Ciao, Michael.

2 1

Re: openldap.git branch mdb.master updated. 21da623bf40dc21f89c3172c523d094075e0824b
by Hallvard Breien Furuseth 02 May '13

02 May '13

openldap-commit2devel(a)OpenLDAP.org writes: > Allow reading freelist while working on it > > The circular dependency issues appear to have been resolved. > Still, need to watch closely, maybe revert this change if > problems arise. IIRC this could get page-hungry when I tried that with a fragented DB, and with large changes which needed overflow pages and used them up for non-freelist items. One of those big LDIFs at ada, I think. Something like: Commit wants a big enough range for an overflow page for mt_free_pgs, maybe finds and allocates one, but the freelist changes eat too many freelist items and will need a bigger page one later. It doesn't free the useless overflow page it already allocated. Repeat. Anyway, I think this change needs code to give pages back to me_pghead, or revert to an earlier {me_pglast, me_pghead}, or something like that. Or maybe someday clean up nested txns, support them with WRITEMAP, and use those for the freelist loop since they support reverting a change. -- Hallvard

1 0

Current master broken with back-mdb
by Quanah Gibson-Mount 08 Apr '13

08 Apr '13

Just FYI, current master is utterly broken with back-mdb. Most sub-databases are missing. mdb database with 2.4.35: zimbra@zre-ldap002:~/data/ldap/mdb/db$ mdb_stat -eaf . | grep "Status of" Status of Main DB Status of DKIMIdentity Status of DKIMSelector Status of ad2i Status of cn Status of displayName Status of dn2i Status of entryCSN Status of entryUUID Status of givenName Status of id2e Status of mail Status of objectClass Status of sn Status of uid Status of zimbraACE Status of zimbraAuthKerberos5Realm Status of zimbraCalResBuilding Status of zimbraCalResCapacity Status of zimbraCalResFloor Status of zimbraCalResRoom Status of zimbraCalResSite Status of zimbraDomainAliasTargetId Status of zimbraDomainName Status of zimbraForeignPrincipal Status of zimbraId Status of zimbraMailAlias Status of zimbraMailCatchAllAddress Status of zimbraMailDeliveryAddress Status of zimbraMailForwardingAddress Status of zimbraMailHost Status of zimbraMailTransport Status of zimbraMemberOf Status of zimbraShareInfo Status of zimbraSharedItem Status of zimbraUCServiceId Status of zimbraVirtualHostname Status of zimbraVirtualIPAddress Status of zimbraYahooId mdb database with master: zimbra@zre-ldap001:~/data/ldap/mdb/db$ mdb_stat -eaf . | grep "Status of" Status of Main DB Status of DKIMIdentity Status of DKIMSelector Status of ad2i Status of cn --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

2 3

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel