contextCSN of subordinate syncrepl DBs
by Rein Tollevik
I've been trying to figure out why syncrepl used on a backend that is
subordinate to a glue database with the syncprov overlay should save the
contextCSN in the suffix of the glue database rather than the suffix of
the backend where syncrepl is used. But all I come up with are reasons
why this should not be the case. So, unless anyone can enlighten me as
to what I'm missing, I suggest that this be changed.
The problem with the current design is that it makes it impossible to
reliably replicate more than one subordinate db from the same remote
server, as there are now race conditions where one of the subordinate
backends could save an updated contextCSN value that is picked up by the
other before it has finished its synchronization. An example of a
configuration where more than one subordinate db replicated from the
same server might be necessary is the central master described in my
previous posting in
http://www.openldap.org/lists/openldap-devel/200806/msg00041.html
My idea as to how this race condition could be verified was to add
enough entries to one of the backends (while the consumer was stopped)
to make it possible to restart the consumer after the first backend had
saved the updated contextCSN but before the second has finished its
synchronization. But I was able to produce it by simply add or delete
of an entry in one of the backends before starting the consumer. Far to
often was the backend without any changes able to pick up and save the
updated contextCSN from the producer before syncrepl on the second
backend fetched its initial value. I.e it started with an updated
contextCSN and didn't receive the changes that had taken place on the
producer. If syncrepl stored the values in the suffix of their own
database then they wouldn't interfere with each other like this.
There is a similar problem in syncprov, as it must use the lowest
contextCSN value (with a given sid) saved by the syncrepl backends
configured within the subtree where syncprov is used. But to do that it
also needs to distinguish the contextCSN values of each syncrepl
backend, which it can't do when they all save them in the glue suffix.
This also implies that syncprov must ignore contextCSN updates from
syncrepl until all syncrepl backends has saved a value, and that
syncprov on the provider must send newCookie sync info messages when it
updates its contextCSN value when the changed entry isn't being
replicated to a consumer. I.e as outlined in the message referred to above.
Neither of these changes should interfere with ordinary multi-master
configurations where syncrepl and syncprov are both use on the same
(glue) database.
I'll volunteer to implement and test the necessary changes if this is
the right solution. But to know whether my analysis is correct or not I
need feedback. So, comments please?
--
Rein Tollevik
Basefarm AS
13 years, 6 months
dITStructureRules/nameForms in subschema subentry for informational purpose
by Michael Ströder
HI!
Discussed this very briefly with Howard at LDAPcon 2007 based on an idea
of Steve:
Support for dITStructureRules and nameForms is still in OpenLDAP's TODO.
In the meanwhile slapd could accept definitions for both in slapd.conf
and simply pass them on to a schema-aware LDAP client for informational
purpose without enforcing them. Same function like rootDSE <file> in
slapd.conf.
Opinions?
Ciao, Michael.
--
Michael Ströder
E-Mail: michael(a)stroeder.com
http://www.stroeder.com
14 years, 2 months
slapd.conf vs slapd.d in the docs
by Gavin Henry
Hi all,
I'm just wondering how we move forward with the docs and the config
backend. Should we document examples with both, or only slapd.d or
move example config to the appendix and have an appendix for each
format so it doesn't clutter the guide.
My feeling is the last option is best. Thoughts?
Gavin.
--
Sent from my mobile device
http://www.suretecsystems.com/services/openldap/
14 years, 4 months
hide attribute
by manu＠netbsd.org
Hello
Many badly designed software fetch all attribute when looking up an user
in the directory, instead of just fetching the one they are interested
in.
My user objects have jpegPhoto attribute, which get fetched with the
whole user object. jpegPhoto are big, so this cause unnescesary load on
the network and LDAP servers and it slows down login process on the bad
software.
Setting up ACL to deny read access to jpegPhoto is not always feasible,
nor it is easily maintainable.
A nicer approach would probably to have a hidden jpegPhoto: it would not
be sent to a client requesting all attributes, but a client explicitely
requesting a set of attribute including jpegPhoto would get it.
AFAIK, there is no way to do that for now. Am I right?
I suspect an overlay would be the right way of implementing it
(slapo-cloak?). Would it be of enough interest to go into
server/slapd/overlays ? If it does, I will contribute it.
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu(a)netbsd.org
14 years, 5 months
Syntax/matching rule implementation - floating point numbers
by Hallvard B Furuseth
Here are some notes about how slapd implements a syntax/matching rule.
I'm moving this to openldap-devel, it seems to belong there by now.
A slapd search first uses any indexes to reduce the number of entries
to examine, and then it checks these entries against the filter.
An index entry contains the indexed value translated to a string which
can be compared with simple memcmp(), and a list of entry IDs for the
entries that contain that value. So e.g. caseIgnoreMatch translates
upper- and lowercase characters to the same character in the index.
If (x < y) implies (memcmp(indexed(x), indexed(y)) < 0), then ORDERING
match can use indexing - the 'eq' index. Examples include the 'integer'
indexing format since OpenLDAP 2.4.7, and 'generalizedTime'.
Index keys should not be large, so string indexing stores a hash of the
indexed value while integer and generalizedTime use binary formats.
The integer index format is quite different from normal binary integer
formats, to make ORDERING (i.e. memcmp()) work.
You may want to normalize or prettify values received over the protocol
before storing them, e.g. store incoming "nan" as "NaN".
For syntax/matching rule examples see schema_init.c - for syntax 'foo':
syntax_defs[] DESC 'Foo', mrule_defs[] NAME 'fooMatch','fooOrderingMatch'.
fooValidate() checks the syntax of an attribute value.
fooMatch() implements the EQUALITY and ORDERING rules.
fooIndexer(), fooFilter() make the syntax indexable.
fooNormalize() normalizes values, so they can be compared just with
memcmp().
fooPretty() translates to a nicer to deal with but not normalized
value, e.g. with DN syntax CN=foo\,bar ==> cn=foo\2Cbar.
For a non-normalized syntax, look at 'foo'='integer'.
For a normalized index, look at 'foo'='generalizedTime'.
Note that having normalized the value, the 'generalizedTimeMatch' rule
in mrule_defs[] simply uses octetStringMatch to compare instead of a
more complex rule. 'generalizedTimeOrderingMatch' cannot do that though.
If you go for a binary attribute syntax like some IEEE format, ask
someone else for advise:-) There is also a distinction between whether
such a syntax is transferred with the ";binary" option. (I think
certificates must and are transferred as ASN.1 BER values. OTOH
Octet String can contain any set of bytes, but is not ;binary.)
Regarding a syntax for 'Real':
Look at ITS#745 <http://www.openldap.org/its/?findid=745>. Nothing
came of it, but maybe the author is around.
NaN is a problem. NaN in math does not compare equal to itself, but
such behavior would be messy for LDAP values. I haven't tested, but I
think you could not have a multi-valued real attribute which contains
NaN and some other number: When storing a value, LDAP uses the equality
matching rule to check if you are trying to store a duplicate of an
existing value.
OTOH I expect realOrderingMatch(NaN, anything)=Undefined is OK, but I
haven't tested that either. (Remember that LDAP filters have
three-valued logic: A compare can return True, False, Undefined. Or
an error, which isn't quite the same.)
If you make it defined, and if current math practice doesn't say
anything, I guess NaN should be larger than other values (except Inf?)
to match Server Side Sorting (RFC 2891)'s treatment of absent values.
OpenLDAP doesn't implement server side sorting, but people keep asking
so maybe it will someday.
--
Hallvard
14 years, 5 months
Re: commit: ldap/servers/slapd syncrepl.c
by hyc＠symas.com
If anyone has some free time it would be a good idea to extend
tests 17, 18, and 19 for these cases.
Modify an entry (some non-RDN attribute) and then rename the entry.
We should test Refresh when both ops have already occurred to the
entry, as well as Refresh when each op occurs in a separate refresh pass.
And as noted in -bugs, we should test with an entry using a single-valued RDN attribute.
On Wed, Dec 03, 2008 at 03:49:54AM +0000, hyc(a)OpenLDAP.org wrote:
> Update of /repo/OpenLDAP/pkg/ldap/servers/slapd
>
> Modified Files:
> syncrepl.c 1.431 -> 1.432
>
> Log Message:
> More for prev commit - if doing both modrdn and modify at once, only
> set the CSN on the modify op.
>
>
> CVS Web URLs:
> http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/
> http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/syncrepl.c
>
> Changes are generally available on cvs.openldap.org (and CVSweb)
> within 30 minutes of being committed.
14 years, 6 months
OpenLDAP stuck deep in BDB?
by Quanah Gibson-Mount
I've seen this now with two different masters, using BDB 4.2.52 + patches,
which we long considered stable:
Thread 3 (Thread 1115703648 (LWP 31507)):
#0 0x0000003fca90bd9c in __pread_nocancel () from
/lib64/tls/libpthread.so.0
#1 0x0000002a97348edc in __os_io (dbenv=0xdc3100, op=1, fhp=0xf7d2d0,
pgno=71531, pagesize=4096, buf=0x2cb08f8cf0 "_3", niop=0x42681b70) at
../dist/../os/os_rw.c:55
#2 0x0000002a973405ea in __memp_pgread (dbmfp=0xeb5780,
mutexp=0x2cbba53150, bhp=0x2cb08f8c58, can_create=0) at
../dist/../mp/mp_bh.c:219
#3 0x0000002a973413f7 in __memp_fget (dbmfp=0xeb5780, pgnoaddr=0x42681cc4,
flags=0, addrp=0x42681cc8) at ../dist/../mp/mp_fget.c:580
#4 0x0000002a972d18b3 in __bam_search (dbc=0xdb1dc40, root_pgno=Variable
"root_pgno" is not available.
) at ../dist/../btree/bt_search.c:307
#5 0x0000002a972c6e27 in __bam_c_search (dbc=0xdb1dc40, root_pgno=0,
key=0x42681f40, flags=28, exactp=0x42681df4) at
../dist/../btree/bt_cursor.c:2547
#6 0x0000002a972c7b7e in __bam_c_get (dbc=0xdb1dc40, key=0x42681f40,
data=0x42681f20, flags=28, pgnop=0x42681e64) at
../dist/../btree/bt_cursor.c:962
#7 0x0000002a9730f51b in __db_c_get (dbc_arg=0x28ea700, key=0x42681f40,
data=0x42681f20, flags=28) at ../dist/../db/db_cam.c:643
#8 0x0000002a973177d7 in __db_c_get_pp (dbc=0x28ea700, key=0x42681f40,
data=0x42681f20, flags=28) at ../dist/../db/db_iface.c:1836
#9 0x0000002a97183dc2 in bdb_dn2id (op=0x428025e0, dn=0x42682028,
ei=0x42682010, locker=44, lock=0x42681fd0) at dn2id.c:315
#10 0x0000002a971896fd in bdb_cache_find_ndn (op=0x428025e0, locker=44,
ndn=0x291c370, res=0x42682310) at cache.c:341
#11 0x0000002a97189e39 in bdb_cache_find_id (op=0x428025e0, tid=0x0,
id=25145, eip=0x42682310, islocked=0, locker=44, lock=0x42682250) at
cache.c:716
#12 0x0000002a9717b1c7 in bdb_search (op=0x428025e0, rs=0x42802570) at
search.c:696
#13 0x0000002a975921eb in syncprov_findcsn (op=0x5ccd900,
mode=FIND_PRESENT) at syncprov.c:681
#14 0x0000002a9759690a in syncprov_op_search (op=0x5ccd900, rs=0x42803d60)
at syncprov.c:2074
#15 0x000000000049b935 in overlay_op_walk (op=0x5ccd900, rs=0x42803d60,
which=op_search, oi=0xded8c0, on=0xded700) at backover.c:640
#16 0x000000000049bb91 in over_op_func (op=0x5ccd900, rs=0x42803d60,
which=op_search) at backover.c:702
#17 0x000000000049bc27 in over_op_search (op=0x5ccd900, rs=0x42803d60) at
backover.c:724
#18 0x000000000042f1d8 in fe_op_search (op=0x5ccd900, rs=0x42803d60) at
search.c:355
#19 0x000000000042ecac in do_search (op=0x5ccd900, rs=0x42803d60) at
search.c:217
#20 0x000000000042bd9c in connection_operation (ctx=0x42803e90,
arg_v=0x5ccd900) at connection.c:1133
#21 0x000000000042c28c in connection_read_thread (ctx=0x42803e90,
argv=0x1e) at connection.c:1261
---Type <return> to continue, or q <return> to quit---
#22 0x0000002a956c7c77 in ldap_int_thread_pool_wrapper (xpool=0x8a1f00) at
tpool.c:478
#23 0x0000003fca90610a in start_thread () from /lib64/tls/libpthread.so.0
#24 0x0000003fca0c68c3 in clone () from /lib64/tls/libc.so.6
#25 0x0000000000000000 in ?? ()
Has anyone else seen anything like this? It seems there's possibly a bug
inside BDB 4.2.52 that gets hit occasionally.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
14 years, 6 months
ITS#5835 insights?
by Quanah Gibson-Mount
Does anyone have any thoughts on ITS#5835?
The server keeps hitting the assert in connection.c:
static void
connection_close( Connection *c )
{
ber_socket_t sd = AC_SOCKET_INVALID;
assert( connections != NULL );
assert( c != NULL );
/* ITS#4667 we may have gotten here twice */
if ( c->c_conn_state == SLAP_C_INVALID )
return;
assert( c->c_struct_state == SLAP_C_USED );
--> assert( c->c_conn_state == SLAP_C_CLOSING );
Which was called by connections_timeout_idle in connection.c:
if( difftime( c->c_activitytime+global_idletimeout, now) <
0 ) {
/* close it */
connection_closing( c, "idletimeout" );
--> connection_close( c );
i++;
With the modified debug logging Howard checked in, I see the following
steps taking place:
Dec 2 03:06:05 ldap slapd[8292]: connection_closing: readying conn=3049
sd=28 for close
so we know it made it into connection_closing.
Then we see:
Dec 2 03:06:05 ldap slapd[8292]: connection_resched: attempting closing
conn=3049 sd=28
which means it made it to connection_resched, which calls connection_close:
Debug( LDAP_DEBUG_CONNS, "connection_resched: "
"attempting closing conn=%lu sd=%d\n",
conn->c_connid, sd, 0 );
connection_close( conn );
Doesn't this directly conflict with line 242 above, which also calls
connection_close on the connection?
I also see this comment in the connection_closing code:
/* ITS#4667 this may allow another thread to drop
into
* connection_resched / connection_close before we
* finish, but that's OK.
*/
I'm honestly not sure this is really "OK". The only place I see
connection_resched called is from connection_operation, and I don't see
exactly how I'ved ended up in there in this code path.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
14 years, 6 months
