openldap-devel

openldap-devel@openldap.org

2 participants
1419 discussions

common pessimization: ber_mem*_x(...,NULL)
by Hallvard B Furuseth 10 Sep '10

10 Sep '10

The OpenLDAP code is a bit too clever sometimes. If someone has too much time on their hands, it could be useful to look for places where the code tries and fails to be more clever than the compiler, and simplify that. E.g. compilers are commonly very good at optimizing array operations - i.e. when a function takes a pointer which it *does not modify* and walks an index up and down the pointed-to array. However once the function starts doing pointer games, a number of compilers seem to be scared away from array optimizations and instead just do plain pointer operations. Another example: OpenLDAP has a fair number of calls like ber_memfree_x(ptr, NULL) which I guess is intended as an optimization - it avoids the ber_memfree(ptr) wrapper call. Actually that's a pessimization if liblber/memory.c is optimized to inline ber_memfree_x() into ber_memfree(): ber_memfree_x() cannot be optimized on ctx==NULL. For an optimization, assuming it makes any difference, it would make more sense remove the _x from the call and inline by hand some of the ber_*_x() functions in memory.c. gcc even has some builtins that can be use to turn ber_foo_x(bar,NULL) calls into ber_foo(bar). Someething like this: #ifdef __GNUC__ # define ber_memfree_x(ptr, ctx) \ ((__builtin_constant_p(ctx) && ((ctx) == NULL)) \ ? ber_memfree(ptr) : (ber_memfree_x)(ptr, ctx)) #endif That'll also help code like this: void foo(void *bar) { foo_x(bar, NULL); } void foo_x(void *bar, void *ctx) { ber_memfree_x(bar, ctx); } If foo_x() is into foo(), that will turn ber_memfree_x into ber_memfree. -- Hallvard

1 0

Proposed patch to support: draft-zeilenga-ldap-c-api-concurrency
by Doug Leavitt 23 Aug '10

23 Aug '10

I would like to propose a patch to the OpenLDAP C APIs to support the operation thread safe features described in the draft-zeilenga-ldap-c-api-concurrency drafts. The enhancements add upwardly compatible interfaces and should have no behavioral regressions w.r.t. any existing libldap interfaces. The new concurrency APIs include: ldap_dup and ldap_destroy They are described in more detail in the concurrency draft and in the applicable man paged. The related feature options are also provided. Additionally I would like to propose 1 new public error API ldap_get_lderror which returns all three ldap_get_option results: LDAP_OPT_MATCHED_DN LDAP_OPT_DIAGNOSTIC_MESSAGE LDAP_OPT_RESULT_CODE in a single API, and 1 new depreciated public API: ldap_get_lderrno which performs almost the same function as ldap_get_error but does so in a mt-unsafe manner, but that is compatible with Mozilla's libldap API of the same name. The point of the two new error APIs is to provide a transition path from Mozilla libldap users that still use ldap_get_lderrno to either a similar API that obeys OpenLDAP like practices (ldap_get_error) or as a transition until any existing code can be converted away from ldap_get_lderrno to just using ldap_get_options as needed. The patch includes a new test case to test the new functionality and man pages. The new test case needs additional work for some of the threading models. Any input here is requested, since I do not have testing resources for all the threading combination and do not have the ability to test the Windows model. I am looking forward to any and all comments. I plan to submit a formal ITS with a finalized version of changes once any and all issues are thrashed out and resolved to a level that they are acceptable for integration. Just for the record: This patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Oracle Corporation.Oracle Corporation has not assigned rights and/or interest in this work to any party. I, Douglas Leavitt am authorized by Oracle Corporation, my employer, to release this work under the following terms. The initial version of my proposed patch can be found here: http://cr.opensolaris.org/~djl/openldap-codereview This includes a diff -U of the full patch (/diffU) a webrev diff (/webrev) for on-line review and a tar ball image of the OpenLDAP HEAD as of 8/13/10 with the current patch applied. Thank you for you consideration, and I look forward to any and all comments. Doug Leavitt

4 8

Re: commit: ldap/servers/slapd bconfig.c daemon.c proto-slap.h
by Howard Chu 16 Aug '10

16 Aug '10

hyc(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/servers/slapd > > Modified Files: > bconfig.c 1.413 -> 1.414 > daemon.c 1.444 -> 1.445 > proto-slap.h 1.802 -> 1.803 > > Log Message: > Add support for multiple listener threads. Lightly tested on Linux, > Winsock needs a couple more tweaks Well, it doesn't look like this patch caused any harm for the default case. I'm only seeing about a 10% gain in throughput using two listener threads on a 16 core machine. Not earth-shattering, not bad. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 8

Re: (ITS#3665) Multi-Listener Thread Support
by Howard Chu 07 Aug '10

07 Aug '10

Revisiting an old thread... http://www.openldap.org/lists/openldap-devel/200504/msg00066.html I'm definitely seeing our current listener running out of steam on servers with more than 12 or so cores, and some work in this direction will definitely help. First it would be a good idea to classify all of the tasks that the current listener manages, before deciding how to divide them among multiple threads. The listener is responsible for many events right now: signal/shutdown processing idle timeout checks write timeout checks runqueue scheduling listener socket events threadpool pauses client socket read events client socket write events Splitting the client socket handling across multiple threads will bring the greatest improvement in scalability. Just need to check our thinking and make sure the remaining division of labor still makes sense. There are two cases that annoy me in our current design - why don't we just dedicate a thread to each listener socket, and let it block on accept() ? That would eliminate a bit of churn in the current select() workload. Likewise, why don't we just let writer threads block in write(), instead of having them ask the listener to listen for writability on their socket? Or, if we're using non-blocking sockets, why don't we let the writer threads block in their own select call, instead of relying on the central thread to do the select and re-dispatch? The first, obvious answer is this: when threads are blocked in system calls like accept(), we can't simply wake them up again for shutdown events or other situations. I believe the obvious fix here is to use select() in each thread, waiting for both the target fd and the wake_sds fd which is written to whenever a signal is caught. Off the top of my head I'm not sure, when several threads are selecting on the same fd, if they all receive a Readable event or if only one of them will. Anyone know? I don't think the remaining tasks really involve much overhead. So ideally, we can handle the idle/write timeout and runqueue scheduling in a single thread, which can also be responsible for the main signal/shutdown processing. The listener sockets can each have their own dedicated thread, selecting on their listener socket and the signal fd. A small number of threads can handle the bulk of the client socket events. We would continue to use a power-of-two descriptor table, and a power-of-two set of threads here. The division of labor would simply be thread# = FD % number of threads (There's no need to split the current connection table into multiple arrays, we just divvy things up so that a given thread only accesses its own slots in the array.) Comments? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

6 10

Back-null cn=config support (used to be in ITS#5330)
by Ondrej Kuznik 27 Jul '10

27 Jul '10

Hi, if I revive ITS#5330 or create another one to add back-config support for the null database would it be accepted? I have the patch ready (actually two versions of it as I do not know which approach is more acceptable). Regards, Ondrej Kuznik This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.

1 0

Re: More about libldap locking
by masarati＠aero.polimi.it 26 Jul '10

26 Jul '10

[diverted from -technical] > masarati(a)aero.polimi.it wrote: >> I'm having trouble in determining what mutex is related to what data in >> libldap's structures. For example, as far as I understand, lconn_refcnt >> in LDAPConn is tested and even modified without expressly locking any >> mutex. Logically, it seems to me that ld_conn_mutex should be the one. >> Could anybody please confirm this or provide insight? > > I think this is a -devel thread. > > The last time I looked, ld_res_mutex is always held when touching > lconn_refcnt. That was before your most recent commits. Thanks. Just wanted to note that I didn't change anything so far, except for checking mutex ownership when #ifdef LDAP_THREAD_DEBUG. Keep investigating... p.

1 0

Openldap replication over XMPP
by kalyan 25 Jul '10

25 Jul '10

Hi, I would like to share some simple idea and find out similar interest. This is about the ldap replication using the existing xmpp network. Every server is considered as a single user entity. There is no modes like refresh or refreshonly or delete phase or current phase instead every servers acts as receiver or sender at the same time. Its also easily possible to make roles of the server to read only or master of that replica and changing them on the fly. This idea of replicating ldap data over xmpp is inspired from Isode's sodium sync where the replication is done through series of mail exchange and XEP-0072 soap over xmpp protocol. The rough idea is, Each server will have its own jid (Jabber id) and its password mentioned in slapd.conf. When the server comes if possible or if replication is required it logins to the jabber server like a normal user. The server becomes active. In slapd.conf each database will mention its replicated or not. After server logins to the jabber server, during db_open the server will join the MUC (multi user chat). The room name is same as the suffix of the database. The other servers who has the same copy of the database will join the same MUC. Then Each server will negotiate with their contextCSN? to find out the need to pull the data. (It needs a better way). The MUC can be used as a command stream to other servers. It would help to find out which servers are online right now which can do replication or which servers are busy right now and cant do replication. (I don't know yet on how to figure out the offline servers. We can have an attribute which holds the list of JID of the servers who are all holding the copy of the DIT and this attribute can be synced like any other attribute and the servers from the list that not appearing as online can be considered as offline servers). Now we negotiated, we can send receive signals with the other server. Though its not technically very clear, its possible to achieve maximum things. The real data transfer can be done multiple ways. Like the sodium sync we can generate the ldif file and do a jingle file transfer to other servers. Or during the onreponse event catch the event and if its add/modify/delete/moddn generate the changes as xml elements for each entry and either send it through the MUC itself or each server opens a pubsub (publish-subscribe) node and publish their changes. The other remaining server who are all holding the copy will listen to the node get their updates. All the server plays the role of receiver and sender simultaneously. Once the offline servers are identified the updates can be sent over offline message so that when the server comes up they need not pull everything, its just has to process the offline messages. (Only the required content of Modifications* strcuture orm_modlist or ora_modlist is sent) MUC also has the role of Administrator, Owner, Participant, based on the server roles we can decide server holds master copy or read write copy but again this needs to be stored insome attribute. This helps regulating the changes from one server to another, for instance the updates from read only server can be denied. Change of the role on the fly might be possible. Since its all either xml or ldif data the other directory server can talk to openldap as well. The next problem is schema, we could sync the entire cn=subschema as well. Or whenever the updates goes for an entry, the corresponding schema information also can be sent over for verification. Or simply the receiving server tries to update their DIT and ignore if there is a failure? This is it for now. There are lot of place it needs a decision. A XEP is needed to define the format of each command and data. I do have some working code (may be the worst code). I am writing it as a overlay per database basis. It logins to the xmpp server and join the MUC and creates the pubsub node and other server starts listening to the node. Right now I am trying to convert the data to xml and publish on the node. I wanted to find out does this idea make any sense at all to someone. I hope someone finds it interesting. :) I know I have missed a lot of technical points, its because I dont know them yet deep enough or I haven't thought about it. It would be nice if the discussion could help brushing those gray areas. Thanks, -Kalyan

7 11

Fwd: RE24 (2.4.23) testing call #2
by Emmanuel Lecharny 22 Jul '10

22 Jul '10

Forwarded to the dev list... ---------- Forwarded message ---------- From: Emmanuel Lecharny <elecharny(a)apache.org> Date: Thu, Jul 22, 2010 at 6:10 PM Subject: Re: RE24 (2.4.23) testing call #2 To: Quanah Gibson-Mount <quanah(a)zimbra.com> On Sun, Jun 13, 2010 at 7:39 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote: > This should be the final testing call prior to release. Only change is a > fix to slapo-refint for ITS#6572. > > May be a bit late, but ... ... Using ldapsearch to check that entry was deleted on central master... ERROR: Entry not removed on central master! Race error found after 4 of 10 iterations Found 3 errors >>>>>> Exiting with a false success status for now >>>>> ./scripts/test058-syncrepl-asymmetric completed OK for hdb. make[2]: Leaving directory `/home/openldap/install/openldap-2.4.23/tests' make[1]: Leaving directory `/home/openldap/install/openldap-2.4.23/tests' ~/install/openldap-2.4.23$ uname -a Linux pajbam-desktop 2.6.32-23-generic #37-Ubuntu SMP Fri Jun 11 07:54:58 UTC 2010 i686 GNU/Linux -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com

1 0

double close socket in libldap
by Test Seven 10 Jul '10

10 Jul '10

We sometimes experience an invalid close socket call inside ldap client library. (Why do we care? It signals a bug that may disrupt an innocent thread if the innocent thread happens to open a socket with the same descriptor number as the buggy thread is trying to close.) We tracked down that the same socket is closed twice in rapid succession. Do you know how could it happen that sb_iod and sb_iod->next share the same sbiod_sb? Below: callstack, data of LDAP* conn, relevant code, description of the data and code, workaround. my_app_1!closesocket my_app_1!sb_stream_close+0x21 my_app_1!ber_int_sb_close+0x1e my_app_1!ldap_free_connection+0x28d my_app_1!ldap_ld_free+0x51 +0x000 conn : 0x0523fa80 ldap +0x000 ld_sb : 0x0526b3b8 sockbuf +0x000 sb_opts : lber_options +0x000 lbo_valid : 0n3 +0x002 lbo_options : 0 +0x004 lbo_debug : 0n0 +0x008 sb_iod : 0x05259dd8 sockbuf_io_desc +0x000 sbiod_level : 0n10 +0x004 sbiod_sb : 0x0526b3b8 sockbuf +0x000 sb_opts : lber_options +0x000 lbo_valid : 0n3 +0x002 lbo_options : 0 +0x004 lbo_debug : 0n0 +0x008 sb_iod : 0x05259dd8 sockbuf_io_desc +0x000 sbiod_level : 0n10 +0x004 sbiod_sb : 0x0526b3b8 sockbuf +0x008 sbiod_io : 0x0091ef88 sockbuf_io +0x00c sbiod_pvt : (null) +0x010 sbiod_next : 0x0520c380 sockbuf_io_desc +0x00c sb_fd : 0x1534 +0x010 sb_max_incoming : 0 +0x014 sb_trans_needs_read : 0y0 +0x014 sb_trans_needs_write : 0y0 +0x008 sbiod_io : 0x0091ef88 sockbuf_io +0x000 sbi_setup : 0x006ace40 int my_app_1!sb_fd_setup+0 +0x004 sbi_remove : (null) +0x008 sbi_ctrl : 0x0052cdd0 int my_app_1!mbufFlush+0 +0x00c sbi_read : 0x006acdd0 long my_app_1!sb_stream_read+0 +0x010 sbi_write : 0x006acdf0 long my_app_1!sb_stream_write+0 +0x014 sbi_close : 0x006ace10 int my_app_1!sb_stream_close+0 +0x00c sbiod_pvt : (null) +0x010 sbiod_next : 0x0520c380 sockbuf_io_desc +0x000 sbiod_level : 0n10 +0x004 sbiod_sb : 0x0526b3b8 sockbuf +0x000 sb_opts : lber_options +0x008 sb_iod : 0x05259dd8 sockbuf_io_desc +0x00c sb_fd : 0x1534 +0x010 sb_max_incoming : 0 +0x014 sb_trans_needs_read : 0y0 +0x014 sb_trans_needs_write : 0y0 +0x008 sbiod_io : 0x0091ef88 sockbuf_io +0x000 sbi_setup : 0x006ace40 int my_app_1!sb_fd_setup+0 +0x004 sbi_remove : (null) +0x008 sbi_ctrl : 0x0052cdd0 int my_app_1!mbufFlush+0 +0x00c sbi_read : 0x006acdd0 long my_app_1!sb_stream_read+0 +0x010 sbi_write : 0x006acdf0 long my_app_1!sb_stream_write+0 +0x014 sbi_close : 0x006ace10 int my_app_1!sb_stream_close+0 +0x00c sbiod_pvt : (null) +0x010 sbiod_next : (null) +0x00c sb_fd : 0x1534 +0x010 sb_max_incoming : 0 +0x014 sb_trans_needs_read : 0y0 +0x014 sb_trans_needs_write : 0y0 (etc.) ber_int_sb_close(): p = sb->sb_iod; while ( p ) { if ( p->sbiod_io->sbi_close && p->sbiod_io->sbi_close( p ) < 0 ) { return -1; } p = p->sbiod_next; } sb_stream_close( Sockbuf_IO_Desc *sbiod ): tcp_close( sbiod->sbiod_sb->sb_fd ); So the socket number 0x1534 was closed twice in rapid succession, because there are 2 linked list nodes, both with the same sb_fd (actually, both sbiod_sb point to 0x0526b3b8 sockbuf, which is their "parent" a.k.a. conn->ld_sb, i.e. conn->ld_sb->sb_iod->sbiod_sb == conn->ld_sb and also == conn->ld_sb->sb_iod->sbiod_next->sbiod_sb). The question is how/when the sbiod_next was added. A dirty workaround would be to detect this special case in ber_int_sb_close(), but I'd strongly prefer to fix the problem. p = sb->sb_iod; while ( p ) { if ( p->sbiod_io->sbi_close && p->sbiod_io->sbi_close( p ) < 0 ) { return -1; } + // workaround for double-closesocket: + while (p->sbiod_next // there's another sbiod + && (p->sbiod_next->sbiod_sb == p->sbiod_sb) // points to the same socket + && (p->sbiod_next->sbiod_io == p->sbiod_io) // same functions used + && (p->sbiod_io->sbi_close == sb_stream_close) // tcp + ) { + p = p->sbiod_next; // skip, don't close it again + } + // end of workaround p = p->sbiod_next; }

2 1

Re: commit: ldap/servers/slapd extended.c proto-slap.h
by Pierangelo Masarati 02 Jul '10

02 Jul '10

ando(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/servers/slapd > > Modified Files: > extended.c 1.100 -> 1.101 > proto-slap.h 1.799 -> 1.800 > > Log Message: > implement unload_extop for symmetry (needs test) Quick comment for future reference: I've left the "flags" argument, although unused, as we might need it in the future for special purposes; also, when changing (at least minor) version, we could rename load_extop2() as load_extop(), getting rid of the macro with the same name, or take the opportunity to deprecate those names and implement the equivalent extop_register()/extop_unregister(), for consistency with other features. p.

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel