openldap-devel

openldap-devel@openldap.org

1416 discussions

Re: RE24 testing call (2.4.27 official call #2)
by Marco Pizzoli 21 Nov '11

21 Nov '11

All tests passed with RE24 from 18Nov on RHEL6.1-64bit, compiled against BDB 5.2.36. Marco On Thu, Nov 10, 2011 at 4:05 AM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote: > Please test latest RE24. Thanks! > > --Quanah > > -- > > Quanah Gibson-Mount > Sr. Member of Technical Staff > Zimbra, Inc > A Division of VMware, Inc. > -------------------- > Zimbra :: the leader in open source messaging and collaboration > > -- _________________________________________ Non è forte chi non cade, ma chi cadendo ha la forza di rialzarsi. Jim Morrison

1 0

What to do with test058?
by Ralf Haferkamp 14 Nov '11

14 Nov '11

Hi, I wonder how we should go forward with test058-syncrepl-asymmetric. If I understand the test (and the comments in syncrepl.c) correctly the setup it is testing is unsupported in slapd, am I right? (Which is the reason why test failures are ignored currently, I guess) On the provider the syncprov overlay is setup for the glued database and the consumer's syncrepl clients (also glued) are configured on the individual child databases. Additionally the consumers have configure the the syncprov overlay also of the topmost database. AFAICS all of the failures that test058 reports are related to this unsupported config. Is anybody planning to work on adding support for such a setup. Or should we just rework the test to a supported scenario (or even remove it completely)? Ralf

2 2

MDB name
by Howard Chu 13 Nov '11

13 Nov '11

What's in a name... Apparently "MDB" is already a well-known suffix for Microsoft Access database files. "mdb" is also the name of a debugger tool in Solaris. To avoid confusion with those unrelated items, it might be a good idea to rename our MDB to something else. Now, before the next OpenLDAP release is published. Comments? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

4 6

drop "stable"
by Michael Ströder 13 Nov '11

13 Nov '11

HI! Many people are downloading and compiling "stable" releases from http://www.openldap.org/software/download/ which is IMO problematic: Most developers consider those "stable" releases to be rather outdated since lots of fixes are missing and they don't want to get bothered with old issues (which I personally perfectly understand). ITS filed against those "stable" versions are most times answered like "get a more recent release". There is no well-defined process how to determine whether a release has reached "stable" status. => My suggestion: Simply drop "stable" completely and list only the releases with version number without further qualification. Today most people can deal with version numbers quite well. Ciao, Michael.

4 4

RE24 testing (pre-testing on 2.4.27)
by Quanah Gibson-Mount 11 Nov '11

11 Nov '11

Tomorrow I'll be working on merging in libmdb/back-mdb. If anyone would like to do a basic sanity test against current RE24, that would be helpful. It has all of the planned fixes for 2.4.27 except ITS#7025 which Howard is still working on. Thanks! --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

6 29

RE24 testing call (2.4.27 official call #1)
by Quanah Gibson-Mount 11 Nov '11

11 Nov '11

Please test RE24. Thanks! --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

6 16

RE24 testing call (2.4.27 official call #2)
by Quanah Gibson-Mount 10 Nov '11

10 Nov '11

Please test latest RE24. Thanks! --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

3 3

Note on Cyrus-SASL when testing
by Quanah Gibson-Mount 07 Nov '11

07 Nov '11

Latest cyrus-sasl 2.1.25 introduces a new mechanism called SCRAM. Unlike the other mechanisms, SCRAM generates entropy as soon as the cyrus-sasl library is loaded. This is a problem because cyrus-sasl defaults to using /dev/random instead of /dev/urandom when it loads up. I was having numerous test failures until gdb'ing slapd and finding it was stuck initializing SCRAM while starting up. If you rebuild cyrus-sasl to use /dev/urandom (They thankfully have a configure switch for it), things'll work a lot more smoothly. --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

2 1

Re: (b68fa5 don't flush entries until after cleanup callbacks)
by Hallvard B Furuseth 01 Nov '11

01 Nov '11

openldap-commit2devel(a)OpenLDAP.org writes: > commit b68fa5ecd73037ebd436a2663003f544d482f71e > Author: Howard Chu <hyc(a)openldap.org> > Date: Tue Nov 1 13:17:06 2011 -0700 > > ITS#6981 don't flush entries until after cleanup callbacks This sounds like it breaks REP_ENTRY_MUSTRELEASE's original purpose: commit b92387bb1caf75bac5bf308d1780204a36ceb5dc Author: Howard Chu <hyc(a)openldap.org> Date: Sun Apr 24 22:21:57 2005 +0000 ITS#3671 add REP_ENTRY_MUSTRELEASE flag, release the entry before attempting to write the response on the network. (This should probably be the default behavior anyway, since network hangs of this sort can cause problems for any backend that has entry locks.) I suppose the solution ought to be the opposite: Move the cleanup call too above send_ldap_ber(). Dunno how feasible that is. -- Hallvard

2 2

back-mdb status
by Howard Chu 20 Oct '11

20 Oct '11

A bit of a summary of how the backend is shaping up. I've been testing with a variety of synthetic LDIFs as well as an actual application database (Zimbra accounts). I noted before that back-mdb's write speeds on disk are quite slow. This is because a lot of its writes will be to random disk pages, and also the data writes in a transaction commit are followed by a meta page write, which always involves a seek to page 0 or page 1 of the DB file. For slapadd -q this effect can be somewhat hidden because the writes are done with MDB_NOSYNC specified, so no explicit flushes are performed. In my current tests with synchronous writes, back-mdb is one half the speed of back-bdb/hdb. (Even in fully synchronous mode, BDB only writes its transaction logs synchronously, and those are always sequential writes so there's no seek overhead to deal with.) With that said, slapadd -q for a 3.2M entry database on a tmpfs: back-hdb: real 75m32.678s user 84m31.733s sys 1m0.316s back-mdb: real 63m51.048s user 50m23.125s sys 13m27.958s For back-hdb, BDB was configured with a 32GB environment cache. The resulting DB directory consumed 14951004KB including data files and environment files. For back-mdb, MDB was configured with a 32GB mapsize. The resulting DB directory consumed 18299832KB. The input LDIF was 2.7GB, and there were 29 attributes indexed. Currently MDB is somewhat wasteful with space when dealing with the sorted-duplicate databases that are used for indexing, there's definitely room for improvement here. Also this slapadd was done with tool-threads set to 1, because back-mdb only allows one writer at a time anyway. There is also obviously room for improvement here, in terms of a bulk-loading API for the MDB library. With the DB loaded, the time to execute a search that scans every entry in the DB was performed against each server. Initially back-hdb was only configured with a cachesize of 10000 and IDLcachesize of 10000. It was tested again using a cachesize of 5,000,000 (which is more than was needed since the DB only contained 3,200,100 entries). In each configuration a search was performed twice - once to measure the time to go from an empty cache to a fully primed cache, and again to measure the time for the fully cached search. first second slapd size back-hdb, 10K cache 3m6.906s 1m39.835s 7.3GB back-hdb, 5M cache 3m12.596s 0m10.984s 46.8GB back-mdb 0m19.420s 0m16.625s 7.0GB Next, the time to execute multiple instances of this search was measured, using 2, 4, 8, and 16 ldapsearch instances running concurrently. average result time 2 4 8 16 back-hdb, 5M 0m14.147s 0m17.384s 0m45.665s 17m15.114s back-mdb 0m16.701s 0m16.688s 0m16.621 0m16.955s I don't recall doing this test against back-hdb on ada.openldap.org before, certainly the total blowup at 16 searches was unexpected. But as you can see, with no read locks in back-mdb, search performance is pretty much independent of load. At 16 threads back-mdb slowed down measurably, but that's understandable given that the rest of the system still needed CPU cycles here and there. Otherwise, slapd was running at 1600% CPU the entire time. For back-hdb, slapd maxed out at 1467% CPU, the lock overhead drove it into the ground. So far I'm pretty pleased with the results; for the most part back-mdb is delivering on what I expected. Decoding each entry every time is a bit of a slowdown, compared to having entries fully cached. But the cost disappears as soon as you get more than a couple requests running at once. Overall I believe it proves the basic philosophy - in this day and age, it's a waste of application developers' time to incorporate a caching layer into their own code. The OS already does it and does it well. Give yourself as transparent a path as possible between RAM and disk using mmap, and don't fuss with it any further. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 10

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel