openldap-devel August 2010

openldap-devel@openldap.org

9 participants
5 discussions

OpenLDAP git repo?
by Volker Lendecke 29 Mar '11

29 Mar '11

Hi! Out of several rumors I've heard that OpenLDAP converted it source repo to git. Is that true? www.openldap.org still speaks about cvs. If you changed to git, is the repo publically available? Thanks, Volker

5 12

Re: Proposed patch to support: draft-zeilenga-ldap-c-api-concurrency
by masarati＠aero.polimi.it 28 Sep '10

28 Sep '10

> Doug, > > a list of relatively quick comments, as I didn't go into too much detail > in the analysis of each bit of your work. > > I think your patch contains at least three levels of modifications; they > all make sense separately and incrementally, so I think we should revise > them separately. > > 1) "cosmetic" cleanup (like wrapping locks in specific macros); they do > not strictly relate to the support of > draft-zeilenga-ldap-c-api-concurrency (but they make sense, and I like > them); they should be considered first, regardless of the rest (I note > that according to the way you define LDAP_NEXT_MSGID when #ifdef > LDAP_R_COMPILE, you don't need to #ifdef its definition :) > > 2) a set of issues that fix the concurrent behavior of libldap relatively > irrespective of the support of draft-zeilenga-ldap-c-api-concurrency (see > for example the private discussion we had on fixing concurrent access to > lconn_refcnt). > > 3) specific support to draft-zeilenga-ldap-c-api-concurrency, including > but not limited to the addition of ldap_dup(), ldap_destroy() and related > stuff. > > As you may have understood, I'm especially interested in (2), because it > fixes a number of potential (or actual) issues in OpenLDAP code that uses > libldap concurrently, like the proxy backends. So far, that code used > libldap taking into account known weaknesses from a concurrency > standpoint. For example, proxy backends protect handlers until a bind > succeeds, to prevent the undesired simultaneous creation of default > connections. However, some issues may slip in (as I believe is the case > of ITS#6574). > > One thing I'd like to ask is: you introduce a few additional mutexes: > > ldapoptions -> ldo_mutex > > ldapcommon -> ldc_msgid_mutex, ldc_abandon_mutex > > in addition of the already existing > > ldap -> ld_conn_mutex, ld_req_mutex, ld_res_mutex > > that move to ldap_common. > > My concern is: can you guarantee that the occurrences of locking/unlocking > those additional mutexes, combined to the existing ones, do not result in > deadlocks? I mean: did you explicitly check all possible logical paths or > so, or take measures to avoid this possibility? Another mostly "style" comment: I think you shouldn't be polluting libldap's space with thread implementation specific details, like init.c: #if defined(HAVE_THR) #elif defined(HAVE_PTHREADS) PTHREAD_MUTEX_INITIALIZER #if !(defined(HAVE_THR) || defined(HAVE_PTHREADS)) and so. I understand PTHREAD_MUTEX_INITIALIZER is handy, but the whole thing should remain confined in libldap_r specific files. Also, but since it's a test tool it is not a priority, slapd-mtread.c should use libldap_r and hide thread implementation details under OpenLDAP's portable interface. p.

2 5

Proposed patch to support: draft-zeilenga-ldap-c-api-concurrency
by Doug Leavitt 23 Aug '10

23 Aug '10

I would like to propose a patch to the OpenLDAP C APIs to support the operation thread safe features described in the draft-zeilenga-ldap-c-api-concurrency drafts. The enhancements add upwardly compatible interfaces and should have no behavioral regressions w.r.t. any existing libldap interfaces. The new concurrency APIs include: ldap_dup and ldap_destroy They are described in more detail in the concurrency draft and in the applicable man paged. The related feature options are also provided. Additionally I would like to propose 1 new public error API ldap_get_lderror which returns all three ldap_get_option results: LDAP_OPT_MATCHED_DN LDAP_OPT_DIAGNOSTIC_MESSAGE LDAP_OPT_RESULT_CODE in a single API, and 1 new depreciated public API: ldap_get_lderrno which performs almost the same function as ldap_get_error but does so in a mt-unsafe manner, but that is compatible with Mozilla's libldap API of the same name. The point of the two new error APIs is to provide a transition path from Mozilla libldap users that still use ldap_get_lderrno to either a similar API that obeys OpenLDAP like practices (ldap_get_error) or as a transition until any existing code can be converted away from ldap_get_lderrno to just using ldap_get_options as needed. The patch includes a new test case to test the new functionality and man pages. The new test case needs additional work for some of the threading models. Any input here is requested, since I do not have testing resources for all the threading combination and do not have the ability to test the Windows model. I am looking forward to any and all comments. I plan to submit a formal ITS with a finalized version of changes once any and all issues are thrashed out and resolved to a level that they are acceptable for integration. Just for the record: This patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Oracle Corporation.Oracle Corporation has not assigned rights and/or interest in this work to any party. I, Douglas Leavitt am authorized by Oracle Corporation, my employer, to release this work under the following terms. The initial version of my proposed patch can be found here: http://cr.opensolaris.org/~djl/openldap-codereview This includes a diff -U of the full patch (/diffU) a webrev diff (/webrev) for on-line review and a tar ball image of the OpenLDAP HEAD as of 8/13/10 with the current patch applied. Thank you for you consideration, and I look forward to any and all comments. Doug Leavitt

4 8

Re: commit: ldap/servers/slapd bconfig.c daemon.c proto-slap.h
by Howard Chu 16 Aug '10

16 Aug '10

hyc(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/servers/slapd > > Modified Files: > bconfig.c 1.413 -> 1.414 > daemon.c 1.444 -> 1.445 > proto-slap.h 1.802 -> 1.803 > > Log Message: > Add support for multiple listener threads. Lightly tested on Linux, > Winsock needs a couple more tweaks Well, it doesn't look like this patch caused any harm for the default case. I'm only seeing about a 10% gain in throughput using two listener threads on a 16 core machine. Not earth-shattering, not bad. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 8

Re: (ITS#3665) Multi-Listener Thread Support
by Howard Chu 06 Aug '10

06 Aug '10

Revisiting an old thread... http://www.openldap.org/lists/openldap-devel/200504/msg00066.html I'm definitely seeing our current listener running out of steam on servers with more than 12 or so cores, and some work in this direction will definitely help. First it would be a good idea to classify all of the tasks that the current listener manages, before deciding how to divide them among multiple threads. The listener is responsible for many events right now: signal/shutdown processing idle timeout checks write timeout checks runqueue scheduling listener socket events threadpool pauses client socket read events client socket write events Splitting the client socket handling across multiple threads will bring the greatest improvement in scalability. Just need to check our thinking and make sure the remaining division of labor still makes sense. There are two cases that annoy me in our current design - why don't we just dedicate a thread to each listener socket, and let it block on accept() ? That would eliminate a bit of churn in the current select() workload. Likewise, why don't we just let writer threads block in write(), instead of having them ask the listener to listen for writability on their socket? Or, if we're using non-blocking sockets, why don't we let the writer threads block in their own select call, instead of relying on the central thread to do the select and re-dispatch? The first, obvious answer is this: when threads are blocked in system calls like accept(), we can't simply wake them up again for shutdown events or other situations. I believe the obvious fix here is to use select() in each thread, waiting for both the target fd and the wake_sds fd which is written to whenever a signal is caught. Off the top of my head I'm not sure, when several threads are selecting on the same fd, if they all receive a Readable event or if only one of them will. Anyone know? I don't think the remaining tasks really involve much overhead. So ideally, we can handle the idle/write timeout and runqueue scheduling in a single thread, which can also be responsible for the main signal/shutdown processing. The listener sockets can each have their own dedicated thread, selecting on their listener socket and the signal fd. A small number of threads can handle the bulk of the client socket events. We would continue to use a power-of-two descriptor table, and a power-of-two set of threads here. The division of labor would simply be thread# = FD % number of threads (There's no need to split the current connection table into multiple arrays, we just divvy things up so that a given thread only accesses its own slots in the array.) Comments? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

6 10

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel August 2010