openldap-devel January 2010

openldap-devel@openldap.org

10 participants
9 discussions

OpenLDAP 2.5
by Howard Chu 28 Apr '10

28 Apr '10

With 2.4.21 out, and hopefully stable enough to promote to the next Stable release, it's time to feature-freeze 2.4 and prepare for the 2.5 branch. As I already announced to the OpenLDAP-Committers, we're also planning to switch from CVS to GIT in mid-January. Commits for 2.5 will begin after we've settled into GIT. We haven't really laid out a formal roadmap for 2.5 yet, but I think most of it has been discussed here or in Development ITSs already. I would like to be able to resolve all outstanding Development ITSs - we will either implement them or reject/close them. There are 42 outstanding at the moment. Likewise for all outstanding ITSs in Software Bugs - many of them have been deferred because a proper fix would require invasive changes to large parts of the code base. There are 26 outstanding. With 2.5 beginning we are free to make these large scale changes. We should also walk thru the Software Enhancement requests and decide which to accept and which to reject. Currently there are 37 outstanding. I also have a number of specific areas I want to see worked on; some of these are included in the above ITSs but I'll outline them here: syncrepl config - this is pretty unwieldy already; syncrepl needs to be moved outside of the slapd core and into an overlay. That will allow us a lot more flexibility in configuring while also eliminating a lot of redundant parsing code. suffixmassage - we at the very least need to be able to point a consumer at some non-homogeneous suffix in the provider. We may go for complete librewrite support as well, although at this point I don't see as strong a need. config TLS certs and keys should be stored as LDAP attributes, not pointers to filesystem locations. This is a prereq to using some of the dynamic cert generation features of the CA overlay. (This can be troublesome as there may not be clean APIs for reading certs from memory in all of the TLS APIs we support.) Disabling individual config attribute values and entries. At the moment I'm thinking of adding an ";x-disabled" tag to those values. back-mdb Using a single-level store for Entries will impact all of the schema engine as well. I think the simplest solution here is going to be using an mmap'd file for all of the schema elements. The actual design of back-mdb still needs to be defined in several areas. The single-level store approach exposes us to some new failure modes that the current multi-level backends don't have. (E.g., corruptions due to bad RAM / wild pointer writes are very likely to get persisted on disk, implicitly.) The solution I'm considering is based on a mirroring strategy. Every database will be stored twice on disk: once in the file that is actively mmap'd into the process, and once in a write-only file. On every intentional update of a memory page, we will also store a checksum of the page, and manually write the page to the mirror. If we detect a checksum failure on any in-memory page we can still retrieve a valid copy from the mirror file. This of course doubles our potential I/O load, but I don't believe it's any worse than the load from performing write-ahead logging on a traditional database. (And yes, mirroring will take the place of writing transaction log files.) Some of these same considerations apply to the schema storage, but not entirely. At runtime, the schema is effectively read-only. When we do dynamic schema changes thru cn=config, all other threads are suspended. For the mmap purposes, we can mark all of the schema pages as read-only during runtime, and only make them read-write when cn=config is actually trying to perform an update. As such, the only sticky issue is dealing with changes made to the back-config internal files by plain text editors and such. These are the things I'm interested in. But as always, this Project is driven forward by the particular interests of each individual contributor. If you have other ideas you want to pursue, speak up. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

9 13

Providing LDIF support at the LDAP level
by Rich Megginson 29 Jan '10

29 Jan '10

This patch provides low level LDIF support in libldif and high level support in libldap. http://rmeggins.fedorapeople.org/newldif.patch http://rmeggins.fedorapeople.org/ldifutil.c The libldap support is in the form of LDAP_F( int ) ldap_parse_ldif_record LDAP_P(( char *rbuf, int linenum, LDIFRecord *lr, const char *errstr, unsigned int flags, void *ctx )); Where rbuf and linenum come from ldif_read_record; lr is new (see below); errstr is the string prefix to print for error/debug messages (usually the program name e.g. "ldapmodify"), flags are one of these: #define LDIF_DEFAULT_ADD 0x01 /* if changetype missing, assume LDAP_ADD */ #define LDIF_ENTRIES_ONLY 0x02 /* ignore changetypes other than add */ #define LDIF_NO_CONTROLS 0x04 /* ignore control specifications */ and ctx is the memory context (for ber_*alloc_x) LDIFRecord returns all of the data from the LDIF record, regardless of what type of operation it is. The lr_op field is used to determine the type of operation, and the other fields return the operation specific data. There is also a section of several data fields used by the internal implementation (lifted almost verbatim from ldapmodify.c). This is cleaned up properly by ldap_ldif_record_done() If the LDIF record in rbuf contains no actual record (e.g. the version: field, comments only, etc.) then the return value of ldap_parse_ldif_record() will be 0, but the lr_op field will also be 0.

2 4

Source Code Visualization with Gource. | michaeldehaan.net
by Gavin Henry 20 Jan '10

20 Jan '10

Something we should do when we are on git! http://michaeldehaan.net/2010/01/15/source-code-visualization-with-gource/ -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry(a)suretecsystems.com Open Source. Open Solutions(tm). http://www.suretecsystems.com/ Suretec Systems is a limited company registered in Scotland. Registered number: SC258005. Registered office: 24 Cormack Park, Rothienorman, Inverurie, Aberdeenshire, AB51 8GL. Subject to disclaimer at http://www.suretecgroup.com/disclaimer.html

3 9

tab-width 4 (was: commit: ldap/servers/slapd/back-ldif ldif.c)
by Hallvard B Furuseth 18 Jan '10

18 Jan '10

Please use tab-width 4 like most of the rest of the code. Unless you are in a tab-width 8 file, anyway. This mixture of tab-width 4 and 8 keeps misaligning more and more of the code, one line here and one there over the years. In this case the 'struct ldif_tool' contents, which was aligned until now. Look at it with 'less -x4' or something if you haven't done so. Same with slap.h, which still does look better with tab-width 4 than 8, despite having been mixed up quite a bit by now. -- Hallvard

1 0

Re: OpenLDAP 2.5
by Kurt Zeilenga 14 Jan '10

14 Jan '10

On Jan 14, 2010, at 1:57 AM, Simon Wilkinson wrote: > > On 14 Jan 2010, at 02:24, Kurt Zeilenga wrote: >> >>>> You can't do that with a git merge, >> >> Yes you can. We do in at my day job all the time. >> >>>> because the commit on HEAD includes >>>> the history of everything else on head, which you definitely don't >>>> want to pull into the branch. >> >> Individuals commit on their own local clones, and use 'git push' onto the project's trunk. Release engineers can then cherry-pick the trunk patch onto their local clone of the release branch and push to the project's release branch. > > That's a cherry-pick, rather than a merge. Yes, that's what I said. > There's an important difference here - cherry-picks create a completely new object, where as a merge causes the same object to be applied to both branches. Merges make it much easier to see visually what's been applied where, but they do require that you commit to branch, then merge to trunk. Oh, and they only work at all if there's no back, or forwards, porting required. As the branches will diverge, the process needs to account for porting issues. > >>>> I'd also strongly recommend that you take a look at Shawn's "gerrit" >>>> tool. This provides a code review tool for patch submission, and >>>> hugely simplifies the process of controlling push access to a git >>>> tree, and reviewing the changes that you accept. >> >> We've never had any restrictions on commits. > > Presumably you don't allow anyone on the internet commit access to the repository! Never had any restrictions enforced by the repo software (e.g., CVS). We only have system access restrictions, enforced today by who has ssh access or not. > That's the big advantage with gerrit - that it provides a way of managing internet wide commit access. Developers set up ssh keys through a web interface, and can then be given commit access to a project. You can let trusted developers push straight into the tree, and require everyone else to use code review, or you can mandate review for everyone. At OpenAFS, we've found that using gerrit has really increased the pool of people we get code submissions from, as it makes it trivially easy for them to push changes to us. It also massively simplifies the process of reviewing which changes should and shouldn't go in. See http://gerrit.openafs.org/ for our instance, or http://review.source.android.com for the Android one. I don't see any reason, at least at first, manage access as we do with our CVS repository. Committers get ssh access to the main repo, no git-enforced restrictions. The world gets anonymous access to a read-only clone. > >>>> I'm happy to talk to you about OpenAFS's experiences in all of this, >>>> if you'd like more details. >>> >>> Absolutely, thanks for the offer. Up till now my use of git has been extremely >>> basic. It wouldn't bother me too much to continue the CVS-style workflow we've >>> had, >> >> I would strong recommend that one should not change both repo software and engineering process at the same time. First, switch repo software. Learn it. Then, if consider changes the workflow. > > Yes - that was pretty much the approach we decided upon. We've kept our cherry-pick based workflow from CVS. The big git change has been that rather than patches going into a bug tracking system for consideration, and then being pulled out of there and pushed into CVS by a small set of commiters, changes now get pushed directly into gerrit by their submitters. This means that, for the commiters, the process of applying the change is a button press or two, which has hugely speeded up our rate of change. The least change to our processes would be for the random developers (not committers) to submit git patches to our issue tracker which our committers can, upon acceptance, git-am onto their trunk clone and push from their. This is actually just a submission format change, not a change in processes or work-flow. -- Kurt

1 0

Re: OpenLDAP 2.5
by Howard Chu 13 Jan '10

13 Jan '10

Simon Wilkinson wrote: > > On 5 Jan 2010, at 18:33, Quanah Gibson-Mount wrote: > >> --On Tuesday, December 22, 2009 7:40 PM -0800 Howard Chu <hyc(a)symas.com >>> wrote: >> >>> With 2.4.21 out, and hopefully stable enough to promote to the next >>> Stable release, it's time to feature-freeze 2.4 and prepare for the >>> 2.5 >>> branch. As I already announced to the OpenLDAP-Committers, we're also >>> planning to switch from CVS to GIT in mid-January. Commits for 2.5 >>> will >>> begin after we've settled into GIT. >> >> Yeah! One question from the RE side, is how to best handle 2.4 >> fixes with HEAD getting further & further apart. > > You also want to consider how git's tools for branching and merging > fit together. We went through this with OpenAFS last year, guided by > some very helpful conversations with Shawn Pearce. Basically, the > fundamental question is whether you want a merge based workflow, one > based around cherry-picks, or some hybrid version of the two. > > If you have a merge based workflow, then the model is that you make > that change on a branch, and then merge that branch into the head. > OpenAFS (and OpenLDAP, by the sounds of things) had a model where we > committed to HEAD, and then 'pulled-up' to the relevant branches. You > can't do that with a git merge, because the commit on HEAD includes > the history of everything else on head, which you definitely don't > want to pull into the branch. So, we ended up deciding to cherry-pick > pullups. This makes it harder to tell whether we've missed anything > that should be pulled up, but "git cherry" helps ease that pain, and > it means we keep our existing workflow. > > I'd also strongly recommend that you take a look at Shawn's "gerrit" > tool. This provides a code review tool for patch submission, and > hugely simplifies the process of controlling push access to a git > tree, and reviewing the changes that you accept. > > I'm happy to talk to you about OpenAFS's experiences in all of this, > if you'd like more details. Absolutely, thanks for the offer. Up till now my use of git has been extremely basic. It wouldn't bother me too much to continue the CVS-style workflow we've had, but if there are better ways that involve less manual labor, we certainly want to know. (Though frankly, the fact that git is faster and works from a complete local repo was convincing enough for me. Being able to perform local commits to save and backtrack my work-in-progress without requiring network access is already a huge gain.) -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

Re: commit: ldap/servers/slapd sasl.c
by Hallvard B Furuseth 06 Jan '10

06 Jan '10

rein(a)OpenLDAP.org writes: > ITS#6441 cyrus-sasl 2.1.24 auxprop_lookup plugin returns status. > > +#if SASL_VERSION_FULL >= 0x020118 If it's for 2.1.24, checking for 2.1.18 looks somewhat suboptimal. -- Hallvard

2 2

ACL check parameters
by Howard Chu 05 Jan '10

05 Jan '10

oprofile shows quite a bit more overhead in 2.4's ACL processing vs 2.3's. I'm thinking of streamlining things a bit, along these lines. Any thoughts? This acl.c compiles, but a fair amount of slapd code needs to be altered to use it. I guess if we commit and move forward on it it will have to be for 2.5. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

3 7

Fwd: Re: (ITS#6194) Patch - Enhancement - provide LDIF support as libldif
by Howard Chu 05 Jan '10

05 Jan '10

So, what do folks think about reviving libldif as an exported piece of the distribution? -------- Original Message -------- Subject: Re: (ITS#6194) Patch - Enhancement - provide LDIF support as libldif Date: Thu, 17 Dec 2009 03:46:08 GMT From: hyc(a)symas.com To: openldap-its(a)openldap.org rmeggins(a)redhat.com wrote: > I think this could be accomplished in one of two ways: > 1) Just have libldif return lists of struct berval* for the various data > parsed. The caller would be responsible for turning these into LDAPMod > or LDAPControl structures - the advantage is that libldif doesn't have > to know about any of these higher level structures > 2) Have libldif create LDAPMod and LDAPControl - I think this could be > accomplished by having ldif.c #include<ldap.h> to pull in the > definitions of LDAPMod and LDAPControl - would this be ok? Let's move this discussion to the openldap-devel mailing list. I'm thinking (2) is OK but I'd like to hear from other developers / potential users of this library. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

4 4

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel January 2010