openldap-devel

openldap-devel@openldap.org

1416 discussions

Re: (ITS#3665) Multi-Listener Thread Support
by Howard Chu 06 Aug '10

06 Aug '10

Revisiting an old thread... http://www.openldap.org/lists/openldap-devel/200504/msg00066.html I'm definitely seeing our current listener running out of steam on servers with more than 12 or so cores, and some work in this direction will definitely help. First it would be a good idea to classify all of the tasks that the current listener manages, before deciding how to divide them among multiple threads. The listener is responsible for many events right now: signal/shutdown processing idle timeout checks write timeout checks runqueue scheduling listener socket events threadpool pauses client socket read events client socket write events Splitting the client socket handling across multiple threads will bring the greatest improvement in scalability. Just need to check our thinking and make sure the remaining division of labor still makes sense. There are two cases that annoy me in our current design - why don't we just dedicate a thread to each listener socket, and let it block on accept() ? That would eliminate a bit of churn in the current select() workload. Likewise, why don't we just let writer threads block in write(), instead of having them ask the listener to listen for writability on their socket? Or, if we're using non-blocking sockets, why don't we let the writer threads block in their own select call, instead of relying on the central thread to do the select and re-dispatch? The first, obvious answer is this: when threads are blocked in system calls like accept(), we can't simply wake them up again for shutdown events or other situations. I believe the obvious fix here is to use select() in each thread, waiting for both the target fd and the wake_sds fd which is written to whenever a signal is caught. Off the top of my head I'm not sure, when several threads are selecting on the same fd, if they all receive a Readable event or if only one of them will. Anyone know? I don't think the remaining tasks really involve much overhead. So ideally, we can handle the idle/write timeout and runqueue scheduling in a single thread, which can also be responsible for the main signal/shutdown processing. The listener sockets can each have their own dedicated thread, selecting on their listener socket and the signal fd. A small number of threads can handle the bulk of the client socket events. We would continue to use a power-of-two descriptor table, and a power-of-two set of threads here. The division of labor would simply be thread# = FD % number of threads (There's no need to split the current connection table into multiple arrays, we just divvy things up so that a given thread only accesses its own slots in the array.) Comments? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

6 10

Back-null cn=config support (used to be in ITS#5330)
by Ondrej Kuznik 27 Jul '10

27 Jul '10

Hi, if I revive ITS#5330 or create another one to add back-config support for the null database would it be accepted? I have the patch ready (actually two versions of it as I do not know which approach is more acceptable). Regards, Ondrej Kuznik This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.

1 0

Re: More about libldap locking
by masarati＠aero.polimi.it 26 Jul '10

26 Jul '10

[diverted from -technical] > masarati(a)aero.polimi.it wrote: >> I'm having trouble in determining what mutex is related to what data in >> libldap's structures. For example, as far as I understand, lconn_refcnt >> in LDAPConn is tested and even modified without expressly locking any >> mutex. Logically, it seems to me that ld_conn_mutex should be the one. >> Could anybody please confirm this or provide insight? > > I think this is a -devel thread. > > The last time I looked, ld_res_mutex is always held when touching > lconn_refcnt. That was before your most recent commits. Thanks. Just wanted to note that I didn't change anything so far, except for checking mutex ownership when #ifdef LDAP_THREAD_DEBUG. Keep investigating... p.

1 0

Openldap replication over XMPP
by kalyan 25 Jul '10

25 Jul '10

Hi, I would like to share some simple idea and find out similar interest. This is about the ldap replication using the existing xmpp network. Every server is considered as a single user entity. There is no modes like refresh or refreshonly or delete phase or current phase instead every servers acts as receiver or sender at the same time. Its also easily possible to make roles of the server to read only or master of that replica and changing them on the fly. This idea of replicating ldap data over xmpp is inspired from Isode's sodium sync where the replication is done through series of mail exchange and XEP-0072 soap over xmpp protocol. The rough idea is, Each server will have its own jid (Jabber id) and its password mentioned in slapd.conf. When the server comes if possible or if replication is required it logins to the jabber server like a normal user. The server becomes active. In slapd.conf each database will mention its replicated or not. After server logins to the jabber server, during db_open the server will join the MUC (multi user chat). The room name is same as the suffix of the database. The other servers who has the same copy of the database will join the same MUC. Then Each server will negotiate with their contextCSN? to find out the need to pull the data. (It needs a better way). The MUC can be used as a command stream to other servers. It would help to find out which servers are online right now which can do replication or which servers are busy right now and cant do replication. (I don't know yet on how to figure out the offline servers. We can have an attribute which holds the list of JID of the servers who are all holding the copy of the DIT and this attribute can be synced like any other attribute and the servers from the list that not appearing as online can be considered as offline servers). Now we negotiated, we can send receive signals with the other server. Though its not technically very clear, its possible to achieve maximum things. The real data transfer can be done multiple ways. Like the sodium sync we can generate the ldif file and do a jingle file transfer to other servers. Or during the onreponse event catch the event and if its add/modify/delete/moddn generate the changes as xml elements for each entry and either send it through the MUC itself or each server opens a pubsub (publish-subscribe) node and publish their changes. The other remaining server who are all holding the copy will listen to the node get their updates. All the server plays the role of receiver and sender simultaneously. Once the offline servers are identified the updates can be sent over offline message so that when the server comes up they need not pull everything, its just has to process the offline messages. (Only the required content of Modifications* strcuture orm_modlist or ora_modlist is sent) MUC also has the role of Administrator, Owner, Participant, based on the server roles we can decide server holds master copy or read write copy but again this needs to be stored insome attribute. This helps regulating the changes from one server to another, for instance the updates from read only server can be denied. Change of the role on the fly might be possible. Since its all either xml or ldif data the other directory server can talk to openldap as well. The next problem is schema, we could sync the entire cn=subschema as well. Or whenever the updates goes for an entry, the corresponding schema information also can be sent over for verification. Or simply the receiving server tries to update their DIT and ignore if there is a failure? This is it for now. There are lot of place it needs a decision. A XEP is needed to define the format of each command and data. I do have some working code (may be the worst code). I am writing it as a overlay per database basis. It logins to the xmpp server and join the MUC and creates the pubsub node and other server starts listening to the node. Right now I am trying to convert the data to xml and publish on the node. I wanted to find out does this idea make any sense at all to someone. I hope someone finds it interesting. :) I know I have missed a lot of technical points, its because I dont know them yet deep enough or I haven't thought about it. It would be nice if the discussion could help brushing those gray areas. Thanks, -Kalyan

7 11

Fwd: RE24 (2.4.23) testing call #2
by Emmanuel Lecharny 22 Jul '10

22 Jul '10

Forwarded to the dev list... ---------- Forwarded message ---------- From: Emmanuel Lecharny <elecharny(a)apache.org> Date: Thu, Jul 22, 2010 at 6:10 PM Subject: Re: RE24 (2.4.23) testing call #2 To: Quanah Gibson-Mount <quanah(a)zimbra.com> On Sun, Jun 13, 2010 at 7:39 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote: > This should be the final testing call prior to release. Only change is a > fix to slapo-refint for ITS#6572. > > May be a bit late, but ... ... Using ldapsearch to check that entry was deleted on central master... ERROR: Entry not removed on central master! Race error found after 4 of 10 iterations Found 3 errors >>>>>> Exiting with a false success status for now >>>>> ./scripts/test058-syncrepl-asymmetric completed OK for hdb. make[2]: Leaving directory `/home/openldap/install/openldap-2.4.23/tests' make[1]: Leaving directory `/home/openldap/install/openldap-2.4.23/tests' ~/install/openldap-2.4.23$ uname -a Linux pajbam-desktop 2.6.32-23-generic #37-Ubuntu SMP Fri Jun 11 07:54:58 UTC 2010 i686 GNU/Linux -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com

1 0

double close socket in libldap
by Test Seven 10 Jul '10

10 Jul '10

We sometimes experience an invalid close socket call inside ldap client library. (Why do we care? It signals a bug that may disrupt an innocent thread if the innocent thread happens to open a socket with the same descriptor number as the buggy thread is trying to close.) We tracked down that the same socket is closed twice in rapid succession. Do you know how could it happen that sb_iod and sb_iod->next share the same sbiod_sb? Below: callstack, data of LDAP* conn, relevant code, description of the data and code, workaround. my_app_1!closesocket my_app_1!sb_stream_close+0x21 my_app_1!ber_int_sb_close+0x1e my_app_1!ldap_free_connection+0x28d my_app_1!ldap_ld_free+0x51 +0x000 conn : 0x0523fa80 ldap +0x000 ld_sb : 0x0526b3b8 sockbuf +0x000 sb_opts : lber_options +0x000 lbo_valid : 0n3 +0x002 lbo_options : 0 +0x004 lbo_debug : 0n0 +0x008 sb_iod : 0x05259dd8 sockbuf_io_desc +0x000 sbiod_level : 0n10 +0x004 sbiod_sb : 0x0526b3b8 sockbuf +0x000 sb_opts : lber_options +0x000 lbo_valid : 0n3 +0x002 lbo_options : 0 +0x004 lbo_debug : 0n0 +0x008 sb_iod : 0x05259dd8 sockbuf_io_desc +0x000 sbiod_level : 0n10 +0x004 sbiod_sb : 0x0526b3b8 sockbuf +0x008 sbiod_io : 0x0091ef88 sockbuf_io +0x00c sbiod_pvt : (null) +0x010 sbiod_next : 0x0520c380 sockbuf_io_desc +0x00c sb_fd : 0x1534 +0x010 sb_max_incoming : 0 +0x014 sb_trans_needs_read : 0y0 +0x014 sb_trans_needs_write : 0y0 +0x008 sbiod_io : 0x0091ef88 sockbuf_io +0x000 sbi_setup : 0x006ace40 int my_app_1!sb_fd_setup+0 +0x004 sbi_remove : (null) +0x008 sbi_ctrl : 0x0052cdd0 int my_app_1!mbufFlush+0 +0x00c sbi_read : 0x006acdd0 long my_app_1!sb_stream_read+0 +0x010 sbi_write : 0x006acdf0 long my_app_1!sb_stream_write+0 +0x014 sbi_close : 0x006ace10 int my_app_1!sb_stream_close+0 +0x00c sbiod_pvt : (null) +0x010 sbiod_next : 0x0520c380 sockbuf_io_desc +0x000 sbiod_level : 0n10 +0x004 sbiod_sb : 0x0526b3b8 sockbuf +0x000 sb_opts : lber_options +0x008 sb_iod : 0x05259dd8 sockbuf_io_desc +0x00c sb_fd : 0x1534 +0x010 sb_max_incoming : 0 +0x014 sb_trans_needs_read : 0y0 +0x014 sb_trans_needs_write : 0y0 +0x008 sbiod_io : 0x0091ef88 sockbuf_io +0x000 sbi_setup : 0x006ace40 int my_app_1!sb_fd_setup+0 +0x004 sbi_remove : (null) +0x008 sbi_ctrl : 0x0052cdd0 int my_app_1!mbufFlush+0 +0x00c sbi_read : 0x006acdd0 long my_app_1!sb_stream_read+0 +0x010 sbi_write : 0x006acdf0 long my_app_1!sb_stream_write+0 +0x014 sbi_close : 0x006ace10 int my_app_1!sb_stream_close+0 +0x00c sbiod_pvt : (null) +0x010 sbiod_next : (null) +0x00c sb_fd : 0x1534 +0x010 sb_max_incoming : 0 +0x014 sb_trans_needs_read : 0y0 +0x014 sb_trans_needs_write : 0y0 (etc.) ber_int_sb_close(): p = sb->sb_iod; while ( p ) { if ( p->sbiod_io->sbi_close && p->sbiod_io->sbi_close( p ) < 0 ) { return -1; } p = p->sbiod_next; } sb_stream_close( Sockbuf_IO_Desc *sbiod ): tcp_close( sbiod->sbiod_sb->sb_fd ); So the socket number 0x1534 was closed twice in rapid succession, because there are 2 linked list nodes, both with the same sb_fd (actually, both sbiod_sb point to 0x0526b3b8 sockbuf, which is their "parent" a.k.a. conn->ld_sb, i.e. conn->ld_sb->sb_iod->sbiod_sb == conn->ld_sb and also == conn->ld_sb->sb_iod->sbiod_next->sbiod_sb). The question is how/when the sbiod_next was added. A dirty workaround would be to detect this special case in ber_int_sb_close(), but I'd strongly prefer to fix the problem. p = sb->sb_iod; while ( p ) { if ( p->sbiod_io->sbi_close && p->sbiod_io->sbi_close( p ) < 0 ) { return -1; } + // workaround for double-closesocket: + while (p->sbiod_next // there's another sbiod + && (p->sbiod_next->sbiod_sb == p->sbiod_sb) // points to the same socket + && (p->sbiod_next->sbiod_io == p->sbiod_io) // same functions used + && (p->sbiod_io->sbi_close == sb_stream_close) // tcp + ) { + p = p->sbiod_next; // skip, don't close it again + } + // end of workaround p = p->sbiod_next; }

2 1

Re: commit: ldap/servers/slapd extended.c proto-slap.h
by Pierangelo Masarati 02 Jul '10

02 Jul '10

ando(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/servers/slapd > > Modified Files: > extended.c 1.100 -> 1.101 > proto-slap.h 1.799 -> 1.800 > > Log Message: > implement unload_extop for symmetry (needs test) Quick comment for future reference: I've left the "flags" argument, although unused, as we might need it in the future for special purposes; also, when changing (at least minor) version, we could rename load_extop2() as load_extop(), getting rid of the macro with the same name, or take the opportunity to deprecate those names and implement the equivalent extop_register()/extop_unregister(), for consistency with other features. p.

2 1

Re: commit: ldap/servers/slapd controls.c proto-slap.h
by Hallvard B Furuseth 01 Jul '10

01 Jul '10

ralf(a)OpenLDAP.org writes: > controls.c 1.212 -> 1.213 > new call unregister_supported_control(), will be > needed for cn=config delete support Gcc complains about at this line: slap_known_controls[i++] = slap_known_controls[i]; because you are mixing read and write of 'i'. Use slap_known_controls[i] = slap_known_controls[i]; i++; or i++; slap_known_controls[i] = slap_known_controls[i]; -- Hallvard

2 1

"fixing" syncrepl with GSSAPI (MIT kerberos)
by Ralf Haferkamp 23 Jun '10

23 Jun '10

Hi, most of you probably know the issues with using syncrepl with SASL/gssapi when built against MIT Kerberos. Is cause of the problem is also well know. MIT's gssapi implementation will not encode packages for established connection anymore once the ticket is expired. Once this happened any connected syncrepl consumer will just hang forever. I know there have been a lot of discussion in the past on wether Heimdal's of MIT's approach is correct. And I don't want to start yet another one. (It seems even that MIT will switch to the Heimdal behavior with future releases: http://krbdev.mit.edu/rt/Ticket/Display.html?id=6739 ) But to fixing the problem with current releases seems to be pretty easy. At least if I didn't overlook something. If we'd just close the syncrepl connection once the provider fails to send a message to the consumer, we consumer's retry mechanmis can try to reestablish the connection (this will succeed once the tickets have be refreshed by some external tool). The basic functionality is there already it seems. send_ldap_ber() calls connection_closing() when ber_flush fails (which happens when sasl_encode() fails because the ticket expired). The only thing that's missing seems to be to actually close the connection in the syncprov overlay after syncprov_sendresp() failed. For that to happened we'd need to export connection_close() to have it available in syncprov.c. Did I overlook something? Would anybody object if I'd commit the required changes? -- regards, Ralf

3 4

Re: commit: ldap/servers/slapd syncrepl.c
by Jonathan Clarke 21 Jun '10

21 Jun '10

On 19/06/2010 03:15, hyc(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/servers/slapd > > Modified Files: > syncrepl.c 1.510 -> 1.511 > > Log Message: > Add suffixmassage processing I just gave this a try, configuring one slapd to replicate it's config database from another, using this syncrepl statement: > olcSyncrepl: {0}rid=001 provider=ldap://localhost:33389/ searchbase="cn=config,ou=configrepl,dc=my-domain,dc=com" suffixmassage="cn=config" It performs a search on the remote database OK, but then fails with: > syncrepl_message_to_entry: rid=001 mods check (creatorsName: value #0 invalid per syntax) > *** glibc detected *** /home/jclarke/COMMUNAUTES/OpenLDAP/openldap-RE24/servers/slapd/slapd: double free or corruption (fasttop): 0x0000000000d4af70 *** The "double free or corruption" comes from the call to slap_mods_free in do_syncrep2, around line 975. No time to investigate more right now, sorry... I did just apply your two patches to syncrepl.c to RE24 in order to test. Does this depend on something else I'm missing? Jonathan -- -------------------------------------------------------------- Jonathan Clarke - jonathan(a)phillipoux.net -------------------------------------------------------------- Ldap Synchronization Connector (LSC) - http://lsc-project.org --------------------------------------------------------------

2 3

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel