OpenLDAP git repo?
by Volker Lendecke
Hi!
Out of several rumors I've heard that OpenLDAP converted it
source repo to git. Is that true? www.openldap.org still
speaks about cvs. If you changed to git, is the repo
publically available?
Thanks,
Volker
12 years, 2 months
Re: (ITS#3665) Multi-Listener Thread Support
by Howard Chu
Revisiting an old thread...
http://www.openldap.org/lists/openldap-devel/200504/msg00066.html
I'm definitely seeing our current listener running out of steam on servers
with more than 12 or so cores, and some work in this direction will definitely
help. First it would be a good idea to classify all of the tasks that the
current listener manages, before deciding how to divide them among multiple
threads.
The listener is responsible for many events right now:
signal/shutdown processing
idle timeout checks
write timeout checks
runqueue scheduling
listener socket events
threadpool pauses
client socket read events
client socket write events
Splitting the client socket handling across multiple threads will bring the
greatest improvement in scalability. Just need to check our thinking and make
sure the remaining division of labor still makes sense.
There are two cases that annoy me in our current design - why don't we just
dedicate a thread to each listener socket, and let it block on accept() ? That
would eliminate a bit of churn in the current select() workload.
Likewise, why don't we just let writer threads block in write(), instead of
having them ask the listener to listen for writability on their socket? Or, if
we're using non-blocking sockets, why don't we let the writer threads block in
their own select call, instead of relying on the central thread to do the
select and re-dispatch?
The first, obvious answer is this: when threads are blocked in system calls
like accept(), we can't simply wake them up again for shutdown events or other
situations. I believe the obvious fix here is to use select() in each thread,
waiting for both the target fd and the wake_sds fd which is written to
whenever a signal is caught. Off the top of my head I'm not sure, when several
threads are selecting on the same fd, if they all receive a Readable event or
if only one of them will. Anyone know?
I don't think the remaining tasks really involve much overhead. So ideally, we
can handle the idle/write timeout and runqueue scheduling in a single thread,
which can also be responsible for the main signal/shutdown processing.
The listener sockets can each have their own dedicated thread, selecting on
their listener socket and the signal fd.
A small number of threads can handle the bulk of the client socket events. We
would continue to use a power-of-two descriptor table, and a power-of-two set
of threads here. The division of labor would simply be
thread# = FD % number of threads
(There's no need to split the current connection table into multiple arrays,
we just divvy things up so that a given thread only accesses its own slots in
the array.)
Comments?
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
12 years, 10 months
Back-null cn=config support (used to be in ITS#5330)
by Ondrej Kuznik
Hi,
if I revive ITS#5330 or create another one to add back-config support
for the null database would it be accepted? I have the patch ready
(actually two versions of it as I do not know which approach is more
acceptable).
Regards,
Ondrej Kuznik
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.
12 years, 10 months
Re: More about libldap locking
by masarati＠aero.polimi.it
[diverted from -technical]
> masarati(a)aero.polimi.it wrote:
>> I'm having trouble in determining what mutex is related to what data in
>> libldap's structures. For example, as far as I understand, lconn_refcnt
>> in LDAPConn is tested and even modified without expressly locking any
>> mutex. Logically, it seems to me that ld_conn_mutex should be the one.
>> Could anybody please confirm this or provide insight?
>
> I think this is a -devel thread.
>
> The last time I looked, ld_res_mutex is always held when touching
> lconn_refcnt. That was before your most recent commits.
Thanks. Just wanted to note that I didn't change anything so far, except
for checking mutex ownership when #ifdef LDAP_THREAD_DEBUG. Keep
investigating...
p.
12 years, 10 months
Openldap replication over XMPP
by kalyan
Hi,
I would like to share some simple idea and find out similar interest.
This is about the ldap replication using the existing xmpp network.
Every server is considered as a single user entity. There is no modes
like refresh or refreshonly or delete phase or current phase instead
every servers acts as receiver or sender at the same time. Its also
easily possible to make roles of the server to read only or master of
that replica and changing them on the fly.
This idea of replicating ldap data over xmpp is inspired from Isode's
sodium sync where the replication is done through series of mail
exchange and XEP-0072 soap over xmpp protocol.
The rough idea is,
Each server will have its own jid (Jabber id) and its password mentioned
in slapd.conf. When the server comes if possible or if replication is
required it logins to the jabber server like a normal user. The server
becomes active.
In slapd.conf each database will mention its replicated or not. After
server logins to the jabber server, during db_open
the server will join the MUC (multi user chat). The room name is same as
the suffix of the database. The other servers who has the same copy of
the database will join the same MUC.
Then Each server will negotiate with their contextCSN? to find out the
need to pull the data. (It needs a better way). The MUC can be used as a
command stream to other servers. It would help to find out which servers
are online right now which can do replication or which servers are busy
right now and cant do replication. (I don't know yet on how to figure out
the offline servers. We can have an attribute which holds the list of JID
of the servers who are all holding the copy of the DIT and this attribute
can be synced like any other attribute and the servers from the list that
not appearing as online can be considered as offline servers).
Now we negotiated, we can send receive signals with the other server.
Though its not technically very clear, its possible to achieve maximum
things. The real data transfer can be done multiple ways. Like the
sodium sync we can generate the ldif file and do a jingle file transfer
to other servers. Or during the onreponse event catch the event and if its
add/modify/delete/moddn generate the changes as xml elements for each
entry and either send it through the MUC itself or each server opens a pubsub
(publish-subscribe) node and publish their changes. The other remaining
server who are all holding the copy will listen to the node get their updates. All the
server plays the role of receiver and sender simultaneously.
Once the offline servers are identified the updates can be sent over
offline message so that when the server comes up they need not pull
everything, its just has to process the offline messages.
(Only the required content of Modifications* strcuture orm_modlist
or ora_modlist is sent)
MUC also has the role of Administrator, Owner, Participant, based on the
server roles we can decide server holds master copy or read write copy
but again this needs to be stored insome attribute. This helps regulating
the changes from one server to another, for instance the updates from read
only server can be denied. Change of the role on the fly might be possible.
Since its all either xml or ldif data the other directory server can
talk to openldap as well. The next problem is schema, we could sync the
entire cn=subschema as well. Or whenever the updates goes for an entry,
the corresponding schema information also can be sent over for verification.
Or simply the receiving server tries to update their DIT and ignore if there
is a failure?
This is it for now.
There are lot of place it needs a decision. A XEP is needed to define
the format of each command and data. I do have some working code (may be
the worst code). I am writing it as a overlay per database basis.
It logins to the xmpp server and join the MUC and creates the pubsub
node and other server starts listening to the node. Right now I am
trying to convert the data to xml and publish on the node.
I wanted to find out does this idea make any sense at all to someone. I
hope someone finds it interesting. :)
I know I have missed a lot of technical points, its because I dont know
them yet deep enough or I haven't thought about it. It would be nice if the
discussion could help brushing those gray areas.
Thanks,
-Kalyan
12 years, 10 months
Fwd: RE24 (2.4.23) testing call #2
by Emmanuel Lecharny
Forwarded to the dev list...
---------- Forwarded message ----------
From: Emmanuel Lecharny <elecharny(a)apache.org>
Date: Thu, Jul 22, 2010 at 6:10 PM
Subject: Re: RE24 (2.4.23) testing call #2
To: Quanah Gibson-Mount <quanah(a)zimbra.com>
On Sun, Jun 13, 2010 at 7:39 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote:
> This should be the final testing call prior to release. Only change is a
> fix to slapo-refint for ITS#6572.
>
>
May be a bit late, but ...
...
Using ldapsearch to check that entry was deleted on central master...
ERROR: Entry not removed on central master!
Race error found after 4 of 10 iterations
Found 3 errors
>>>>>> Exiting with a false success status for now
>>>>> ./scripts/test058-syncrepl-asymmetric completed OK for hdb.
make[2]: Leaving directory `/home/openldap/install/openldap-2.4.23/tests'
make[1]: Leaving directory `/home/openldap/install/openldap-2.4.23/tests'
~/install/openldap-2.4.23$ uname -a
Linux pajbam-desktop 2.6.32-23-generic #37-Ubuntu SMP Fri Jun 11 07:54:58
UTC 2010 i686 GNU/Linux
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
12 years, 10 months
double close socket in libldap
by Test Seven
We sometimes experience an invalid close socket call inside ldap
client library. (Why do we care? It signals a bug that may disrupt an
innocent thread if the innocent thread happens to open a socket with the
same descriptor number as the buggy thread is trying to close.) We
tracked down that the same socket is closed twice in rapid succession.
Do you know how could it happen that sb_iod and sb_iod->next share the
same sbiod_sb?
Below: callstack, data of LDAP* conn, relevant code, description of the
data and code, workaround.
my_app_1!closesocket
my_app_1!sb_stream_close+0x21
my_app_1!ber_int_sb_close+0x1e
my_app_1!ldap_free_connection+0x28d
my_app_1!ldap_ld_free+0x51
+0x000 conn : 0x0523fa80 ldap
+0x000 ld_sb : 0x0526b3b8 sockbuf
+0x000 sb_opts : lber_options
+0x000 lbo_valid : 0n3
+0x002 lbo_options : 0
+0x004 lbo_debug : 0n0
+0x008 sb_iod : 0x05259dd8 sockbuf_io_desc
+0x000 sbiod_level : 0n10
+0x004 sbiod_sb : 0x0526b3b8 sockbuf
+0x000 sb_opts : lber_options
+0x000 lbo_valid : 0n3
+0x002 lbo_options : 0
+0x004 lbo_debug : 0n0
+0x008 sb_iod : 0x05259dd8 sockbuf_io_desc
+0x000 sbiod_level : 0n10
+0x004 sbiod_sb : 0x0526b3b8 sockbuf
+0x008 sbiod_io : 0x0091ef88 sockbuf_io
+0x00c sbiod_pvt : (null)
+0x010 sbiod_next : 0x0520c380 sockbuf_io_desc
+0x00c sb_fd : 0x1534
+0x010 sb_max_incoming : 0
+0x014 sb_trans_needs_read : 0y0
+0x014 sb_trans_needs_write : 0y0
+0x008 sbiod_io : 0x0091ef88 sockbuf_io
+0x000 sbi_setup : 0x006ace40 int
my_app_1!sb_fd_setup+0
+0x004 sbi_remove : (null)
+0x008 sbi_ctrl : 0x0052cdd0 int
my_app_1!mbufFlush+0
+0x00c sbi_read : 0x006acdd0
long my_app_1!sb_stream_read+0
+0x010 sbi_write : 0x006acdf0
long my_app_1!sb_stream_write+0
+0x014 sbi_close : 0x006ace10 int
my_app_1!sb_stream_close+0
+0x00c sbiod_pvt : (null)
+0x010 sbiod_next : 0x0520c380 sockbuf_io_desc
+0x000 sbiod_level : 0n10
+0x004 sbiod_sb : 0x0526b3b8 sockbuf
+0x000 sb_opts : lber_options
+0x008 sb_iod : 0x05259dd8 sockbuf_io_desc
+0x00c sb_fd : 0x1534
+0x010 sb_max_incoming : 0
+0x014 sb_trans_needs_read : 0y0
+0x014 sb_trans_needs_write : 0y0
+0x008 sbiod_io : 0x0091ef88 sockbuf_io
+0x000 sbi_setup :
0x006ace40 int my_app_1!sb_fd_setup+0
+0x004 sbi_remove : (null)
+0x008 sbi_ctrl :
0x0052cdd0 int my_app_1!mbufFlush+0
+0x00c sbi_read :
0x006acdd0 long my_app_1!sb_stream_read+0
+0x010 sbi_write :
0x006acdf0 long my_app_1!sb_stream_write+0
+0x014 sbi_close :
0x006ace10 int my_app_1!sb_stream_close+0
+0x00c sbiod_pvt : (null)
+0x010 sbiod_next : (null)
+0x00c sb_fd : 0x1534
+0x010 sb_max_incoming : 0
+0x014 sb_trans_needs_read : 0y0
+0x014 sb_trans_needs_write : 0y0
(etc.)
ber_int_sb_close():
p = sb->sb_iod;
while ( p ) {
if ( p->sbiod_io->sbi_close && p->sbiod_io->sbi_close( p ) < 0 ) {
return -1;
}
p = p->sbiod_next;
}
sb_stream_close( Sockbuf_IO_Desc *sbiod ):
tcp_close( sbiod->sbiod_sb->sb_fd );
So the socket number 0x1534 was closed twice in rapid succession,
because there are 2 linked list nodes, both with the same sb_fd
(actually, both sbiod_sb point to 0x0526b3b8 sockbuf, which is their
"parent" a.k.a. conn->ld_sb, i.e.
conn->ld_sb->sb_iod->sbiod_sb == conn->ld_sb
and also == conn->ld_sb->sb_iod->sbiod_next->sbiod_sb).
The question is how/when the sbiod_next was added.
A dirty workaround would be to detect this special case in
ber_int_sb_close(), but I'd strongly prefer to fix the problem.
p = sb->sb_iod;
while ( p ) {
if ( p->sbiod_io->sbi_close && p->sbiod_io->sbi_close(
p ) < 0 ) {
return -1;
}
+ // workaround for double-closesocket:
+ while (p->sbiod_next
// there's another sbiod
+ && (p->sbiod_next->sbiod_sb == p->sbiod_sb) // points to the same
socket
+ && (p->sbiod_next->sbiod_io == p->sbiod_io) // same functions used
+ && (p->sbiod_io->sbi_close == sb_stream_close) // tcp
+ ) {
+ p = p->sbiod_next; // skip, don't close it
again
+ }
+ // end of workaround
p = p->sbiod_next;
}
12 years, 10 months
Re: commit: ldap/servers/slapd extended.c proto-slap.h
by Pierangelo Masarati
ando(a)OpenLDAP.org wrote:
> Update of /repo/OpenLDAP/pkg/ldap/servers/slapd
>
> Modified Files:
> extended.c 1.100 -> 1.101
> proto-slap.h 1.799 -> 1.800
>
> Log Message:
> implement unload_extop for symmetry (needs test)
Quick comment for future reference: I've left the "flags" argument,
although unused, as we might need it in the future for special purposes;
also, when changing (at least minor) version, we could rename
load_extop2() as load_extop(), getting rid of the macro with the same
name, or take the opportunity to deprecate those names and implement the
equivalent extop_register()/extop_unregister(), for consistency with
other features.
p.
12 years, 11 months
Re: commit: ldap/servers/slapd controls.c proto-slap.h
by Hallvard B Furuseth
ralf(a)OpenLDAP.org writes:
> controls.c 1.212 -> 1.213
> new call unregister_supported_control(), will be
> needed for cn=config delete support
Gcc complains about at this line:
slap_known_controls[i++] = slap_known_controls[i];
because you are mixing read and write of 'i'. Use
slap_known_controls[i] = slap_known_controls[i];
i++;
or
i++;
slap_known_controls[i] = slap_known_controls[i];
--
Hallvard
12 years, 11 months
