openldap-devel December 2008

openldap-devel@openldap.org

10 participants
11 discussions

contextCSN of subordinate syncrepl DBs
by Rein Tollevik 22 Nov '09

22 Nov '09

I've been trying to figure out why syncrepl used on a backend that is subordinate to a glue database with the syncprov overlay should save the contextCSN in the suffix of the glue database rather than the suffix of the backend where syncrepl is used. But all I come up with are reasons why this should not be the case. So, unless anyone can enlighten me as to what I'm missing, I suggest that this be changed. The problem with the current design is that it makes it impossible to reliably replicate more than one subordinate db from the same remote server, as there are now race conditions where one of the subordinate backends could save an updated contextCSN value that is picked up by the other before it has finished its synchronization. An example of a configuration where more than one subordinate db replicated from the same server might be necessary is the central master described in my previous posting in http://www.openldap.org/lists/openldap-devel/200806/msg00041.html My idea as to how this race condition could be verified was to add enough entries to one of the backends (while the consumer was stopped) to make it possible to restart the consumer after the first backend had saved the updated contextCSN but before the second has finished its synchronization. But I was able to produce it by simply add or delete of an entry in one of the backends before starting the consumer. Far to often was the backend without any changes able to pick up and save the updated contextCSN from the producer before syncrepl on the second backend fetched its initial value. I.e it started with an updated contextCSN and didn't receive the changes that had taken place on the producer. If syncrepl stored the values in the suffix of their own database then they wouldn't interfere with each other like this. There is a similar problem in syncprov, as it must use the lowest contextCSN value (with a given sid) saved by the syncrepl backends configured within the subtree where syncprov is used. But to do that it also needs to distinguish the contextCSN values of each syncrepl backend, which it can't do when they all save them in the glue suffix. This also implies that syncprov must ignore contextCSN updates from syncrepl until all syncrepl backends has saved a value, and that syncprov on the provider must send newCookie sync info messages when it updates its contextCSN value when the changed entry isn't being replicated to a consumer. I.e as outlined in the message referred to above. Neither of these changes should interfere with ordinary multi-master configurations where syncrepl and syncprov are both use on the same (glue) database. I'll volunteer to implement and test the necessary changes if this is the right solution. But to know whether my analysis is correct or not I need feedback. So, comments please? -- Rein Tollevik Basefarm AS

2 6

dITStructureRules/nameForms in subschema subentry for informational purpose
by Michael Ströder 21 Mar '09

21 Mar '09

HI! Discussed this very briefly with Howard at LDAPcon 2007 based on an idea of Steve: Support for dITStructureRules and nameForms is still in OpenLDAP's TODO. In the meanwhile slapd could accept definitions for both in slapd.conf and simply pass them on to a schema-aware LDAP client for informational purpose without enforcing them. Same function like rootDSE <file> in slapd.conf. Opinions? Ciao, Michael. -- Michael Ströder E-Mail: michael(a)stroeder.com http://www.stroeder.com

2 4

slapd.conf vs slapd.d in the docs
by Gavin Henry 12 Jan '09

12 Jan '09

Hi all, I'm just wondering how we move forward with the docs and the config backend. Should we document examples with both, or only slapd.d or move example config to the appendix and have an appendix for each format so it doesn't clutter the guide. My feeling is the last option is best. Thoughts? Gavin. -- Sent from my mobile device http://www.suretecsystems.com/services/openldap/

5 5

Re: commit: ldap/servers/slapd syncrepl.c
by Rein Tollevik 29 Dec '08

29 Dec '08

hyc(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/servers/slapd > > Modified Files: > syncrepl.c 1.434 -> 1.435 > > Log Message: > ITS#5866 skip CSN check for glue entries > > > CVS Web URLs: > http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/ > http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/syncrepl.c This patch adds test for "!is_entry_glue( rs->sr_entry )" twice, both in the outer and the inner if condition. A suspect it should only have been in the inner condition? Rein

2 1

ITS#5875: build fix for sparc64/gcc-3.3
by manu＠netbsd.org 29 Dec '08

29 Dec '08

Hello If it does not raise any concern, I will commit this patch, which fixes the build on sparc64/gcc-3.3. It has been tested in NetBSD package system for 5 years, so that looks quite safe... -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz manu(a)netbsd.org

2 1

hide attribute
by manu＠netbsd.org 25 Dec '08

25 Dec '08

Hello Many badly designed software fetch all attribute when looking up an user in the directory, instead of just fetching the one they are interested in. My user objects have jpegPhoto attribute, which get fetched with the whole user object. jpegPhoto are big, so this cause unnescesary load on the network and LDAP servers and it slows down login process on the bad software. Setting up ACL to deny read access to jpegPhoto is not always feasible, nor it is easily maintainable. A nicer approach would probably to have a hidden jpegPhoto: it would not be sent to a client requesting all attributes, but a client explicitely requesting a set of attribute including jpegPhoto would get it. AFAIK, there is no way to do that for now. Am I right? I suspect an overlay would be the right way of implementing it (slapo-cloak?). Would it be of enough interest to go into server/slapd/overlays ? If it does, I will contribute it. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz manu(a)netbsd.org

3 10

Syntax/matching rule implementation - floating point numbers
by Hallvard B Furuseth 17 Dec '08

17 Dec '08

Here are some notes about how slapd implements a syntax/matching rule. I'm moving this to openldap-devel, it seems to belong there by now. A slapd search first uses any indexes to reduce the number of entries to examine, and then it checks these entries against the filter. An index entry contains the indexed value translated to a string which can be compared with simple memcmp(), and a list of entry IDs for the entries that contain that value. So e.g. caseIgnoreMatch translates upper- and lowercase characters to the same character in the index. If (x < y) implies (memcmp(indexed(x), indexed(y)) < 0), then ORDERING match can use indexing - the 'eq' index. Examples include the 'integer' indexing format since OpenLDAP 2.4.7, and 'generalizedTime'. Index keys should not be large, so string indexing stores a hash of the indexed value while integer and generalizedTime use binary formats. The integer index format is quite different from normal binary integer formats, to make ORDERING (i.e. memcmp()) work. You may want to normalize or prettify values received over the protocol before storing them, e.g. store incoming "nan" as "NaN". For syntax/matching rule examples see schema_init.c - for syntax 'foo': syntax_defs[] DESC 'Foo', mrule_defs[] NAME 'fooMatch','fooOrderingMatch'. fooValidate() checks the syntax of an attribute value. fooMatch() implements the EQUALITY and ORDERING rules. fooIndexer(), fooFilter() make the syntax indexable. fooNormalize() normalizes values, so they can be compared just with memcmp(). fooPretty() translates to a nicer to deal with but not normalized value, e.g. with DN syntax CN=foo\,bar ==> cn=foo\2Cbar. For a non-normalized syntax, look at 'foo'='integer'. For a normalized index, look at 'foo'='generalizedTime'. Note that having normalized the value, the 'generalizedTimeMatch' rule in mrule_defs[] simply uses octetStringMatch to compare instead of a more complex rule. 'generalizedTimeOrderingMatch' cannot do that though. If you go for a binary attribute syntax like some IEEE format, ask someone else for advise:-) There is also a distinction between whether such a syntax is transferred with the ";binary" option. (I think certificates must and are transferred as ASN.1 BER values. OTOH Octet String can contain any set of bytes, but is not ;binary.) Regarding a syntax for 'Real': Look at ITS#745 <http://www.openldap.org/its/?findid=745>. Nothing came of it, but maybe the author is around. NaN is a problem. NaN in math does not compare equal to itself, but such behavior would be messy for LDAP values. I haven't tested, but I think you could not have a multi-valued real attribute which contains NaN and some other number: When storing a value, LDAP uses the equality matching rule to check if you are trying to store a duplicate of an existing value. OTOH I expect realOrderingMatch(NaN, anything)=Undefined is OK, but I haven't tested that either. (Remember that LDAP filters have three-valued logic: A compare can return True, False, Undefined. Or an error, which isn't quite the same.) If you make it defined, and if current math practice doesn't say anything, I guess NaN should be larger than other values (except Inf?) to match Server Side Sorting (RFC 2891)'s treatment of absent values. OpenLDAP doesn't implement server side sorting, but people keep asking so maybe it will someday. -- Hallvard

2 5

Re: commit: ldap/servers/slapd syncrepl.c
by hyc＠symas.com 03 Dec '08

03 Dec '08

If anyone has some free time it would be a good idea to extend tests 17, 18, and 19 for these cases. Modify an entry (some non-RDN attribute) and then rename the entry. We should test Refresh when both ops have already occurred to the entry, as well as Refresh when each op occurs in a separate refresh pass. And as noted in -bugs, we should test with an entry using a single-valued RDN attribute. On Wed, Dec 03, 2008 at 03:49:54AM +0000, hyc(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/servers/slapd > > Modified Files: > syncrepl.c 1.431 -> 1.432 > > Log Message: > More for prev commit - if doing both modrdn and modify at once, only > set the CSN on the modify op. > > > CVS Web URLs: > http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/ > http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/syncrepl.c > > Changes are generally available on cvs.openldap.org (and CVSweb) > within 30 minutes of being committed.

2 1

OpenLDAP stuck deep in BDB?
by Quanah Gibson-Mount 03 Dec '08

03 Dec '08

I've seen this now with two different masters, using BDB 4.2.52 + patches, which we long considered stable: Thread 3 (Thread 1115703648 (LWP 31507)): #0 0x0000003fca90bd9c in __pread_nocancel () from /lib64/tls/libpthread.so.0 #1 0x0000002a97348edc in __os_io (dbenv=0xdc3100, op=1, fhp=0xf7d2d0, pgno=71531, pagesize=4096, buf=0x2cb08f8cf0 "_3", niop=0x42681b70) at ../dist/../os/os_rw.c:55 #2 0x0000002a973405ea in __memp_pgread (dbmfp=0xeb5780, mutexp=0x2cbba53150, bhp=0x2cb08f8c58, can_create=0) at ../dist/../mp/mp_bh.c:219 #3 0x0000002a973413f7 in __memp_fget (dbmfp=0xeb5780, pgnoaddr=0x42681cc4, flags=0, addrp=0x42681cc8) at ../dist/../mp/mp_fget.c:580 #4 0x0000002a972d18b3 in __bam_search (dbc=0xdb1dc40, root_pgno=Variable "root_pgno" is not available. ) at ../dist/../btree/bt_search.c:307 #5 0x0000002a972c6e27 in __bam_c_search (dbc=0xdb1dc40, root_pgno=0, key=0x42681f40, flags=28, exactp=0x42681df4) at ../dist/../btree/bt_cursor.c:2547 #6 0x0000002a972c7b7e in __bam_c_get (dbc=0xdb1dc40, key=0x42681f40, data=0x42681f20, flags=28, pgnop=0x42681e64) at ../dist/../btree/bt_cursor.c:962 #7 0x0000002a9730f51b in __db_c_get (dbc_arg=0x28ea700, key=0x42681f40, data=0x42681f20, flags=28) at ../dist/../db/db_cam.c:643 #8 0x0000002a973177d7 in __db_c_get_pp (dbc=0x28ea700, key=0x42681f40, data=0x42681f20, flags=28) at ../dist/../db/db_iface.c:1836 #9 0x0000002a97183dc2 in bdb_dn2id (op=0x428025e0, dn=0x42682028, ei=0x42682010, locker=44, lock=0x42681fd0) at dn2id.c:315 #10 0x0000002a971896fd in bdb_cache_find_ndn (op=0x428025e0, locker=44, ndn=0x291c370, res=0x42682310) at cache.c:341 #11 0x0000002a97189e39 in bdb_cache_find_id (op=0x428025e0, tid=0x0, id=25145, eip=0x42682310, islocked=0, locker=44, lock=0x42682250) at cache.c:716 #12 0x0000002a9717b1c7 in bdb_search (op=0x428025e0, rs=0x42802570) at search.c:696 #13 0x0000002a975921eb in syncprov_findcsn (op=0x5ccd900, mode=FIND_PRESENT) at syncprov.c:681 #14 0x0000002a9759690a in syncprov_op_search (op=0x5ccd900, rs=0x42803d60) at syncprov.c:2074 #15 0x000000000049b935 in overlay_op_walk (op=0x5ccd900, rs=0x42803d60, which=op_search, oi=0xded8c0, on=0xded700) at backover.c:640 #16 0x000000000049bb91 in over_op_func (op=0x5ccd900, rs=0x42803d60, which=op_search) at backover.c:702 #17 0x000000000049bc27 in over_op_search (op=0x5ccd900, rs=0x42803d60) at backover.c:724 #18 0x000000000042f1d8 in fe_op_search (op=0x5ccd900, rs=0x42803d60) at search.c:355 #19 0x000000000042ecac in do_search (op=0x5ccd900, rs=0x42803d60) at search.c:217 #20 0x000000000042bd9c in connection_operation (ctx=0x42803e90, arg_v=0x5ccd900) at connection.c:1133 #21 0x000000000042c28c in connection_read_thread (ctx=0x42803e90, argv=0x1e) at connection.c:1261 ---Type <return> to continue, or q <return> to quit--- #22 0x0000002a956c7c77 in ldap_int_thread_pool_wrapper (xpool=0x8a1f00) at tpool.c:478 #23 0x0000003fca90610a in start_thread () from /lib64/tls/libpthread.so.0 #24 0x0000003fca0c68c3 in clone () from /lib64/tls/libc.so.6 #25 0x0000000000000000 in ?? () Has anyone else seen anything like this? It seems there's possibly a bug inside BDB 4.2.52 that gets hit occasionally. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

ITS#5835 insights?
by Quanah Gibson-Mount 02 Dec '08

02 Dec '08

Does anyone have any thoughts on ITS#5835? The server keeps hitting the assert in connection.c: static void connection_close( Connection *c ) { ber_socket_t sd = AC_SOCKET_INVALID; assert( connections != NULL ); assert( c != NULL ); /* ITS#4667 we may have gotten here twice */ if ( c->c_conn_state == SLAP_C_INVALID ) return; assert( c->c_struct_state == SLAP_C_USED ); --> assert( c->c_conn_state == SLAP_C_CLOSING ); Which was called by connections_timeout_idle in connection.c: if( difftime( c->c_activitytime+global_idletimeout, now) < 0 ) { /* close it */ connection_closing( c, "idletimeout" ); --> connection_close( c ); i++; With the modified debug logging Howard checked in, I see the following steps taking place: Dec 2 03:06:05 ldap slapd[8292]: connection_closing: readying conn=3049 sd=28 for close so we know it made it into connection_closing. Then we see: Dec 2 03:06:05 ldap slapd[8292]: connection_resched: attempting closing conn=3049 sd=28 which means it made it to connection_resched, which calls connection_close: Debug( LDAP_DEBUG_CONNS, "connection_resched: " "attempting closing conn=%lu sd=%d\n", conn->c_connid, sd, 0 ); connection_close( conn ); Doesn't this directly conflict with line 242 above, which also calls connection_close on the connection? I also see this comment in the connection_closing code: /* ITS#4667 this may allow another thread to drop into * connection_resched / connection_close before we * finish, but that's OK. */ I'm honestly not sure this is really "OK". The only place I see connection_resched called is from connection_operation, and I don't see exactly how I'ved ended up in there in this code path. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel December 2008