openldap-devel September 2008

openldap-devel@openldap.org

16 participants
27 discussions

contextCSN of subordinate syncrepl DBs
by Rein Tollevik 22 Nov '09

22 Nov '09

I've been trying to figure out why syncrepl used on a backend that is subordinate to a glue database with the syncprov overlay should save the contextCSN in the suffix of the glue database rather than the suffix of the backend where syncrepl is used. But all I come up with are reasons why this should not be the case. So, unless anyone can enlighten me as to what I'm missing, I suggest that this be changed. The problem with the current design is that it makes it impossible to reliably replicate more than one subordinate db from the same remote server, as there are now race conditions where one of the subordinate backends could save an updated contextCSN value that is picked up by the other before it has finished its synchronization. An example of a configuration where more than one subordinate db replicated from the same server might be necessary is the central master described in my previous posting in http://www.openldap.org/lists/openldap-devel/200806/msg00041.html My idea as to how this race condition could be verified was to add enough entries to one of the backends (while the consumer was stopped) to make it possible to restart the consumer after the first backend had saved the updated contextCSN but before the second has finished its synchronization. But I was able to produce it by simply add or delete of an entry in one of the backends before starting the consumer. Far to often was the backend without any changes able to pick up and save the updated contextCSN from the producer before syncrepl on the second backend fetched its initial value. I.e it started with an updated contextCSN and didn't receive the changes that had taken place on the producer. If syncrepl stored the values in the suffix of their own database then they wouldn't interfere with each other like this. There is a similar problem in syncprov, as it must use the lowest contextCSN value (with a given sid) saved by the syncrepl backends configured within the subtree where syncprov is used. But to do that it also needs to distinguish the contextCSN values of each syncrepl backend, which it can't do when they all save them in the glue suffix. This also implies that syncprov must ignore contextCSN updates from syncrepl until all syncrepl backends has saved a value, and that syncprov on the provider must send newCookie sync info messages when it updates its contextCSN value when the changed entry isn't being replicated to a consumer. I.e as outlined in the message referred to above. Neither of these changes should interfere with ordinary multi-master configurations where syncrepl and syncprov are both use on the same (glue) database. I'll volunteer to implement and test the necessary changes if this is the right solution. But to know whether my analysis is correct or not I need feedback. So, comments please? -- Rein Tollevik Basefarm AS

2 6

dITStructureRules/nameForms in subschema subentry for informational purpose
by Michael Ströder 21 Mar '09

21 Mar '09

HI! Discussed this very briefly with Howard at LDAPcon 2007 based on an idea of Steve: Support for dITStructureRules and nameForms is still in OpenLDAP's TODO. In the meanwhile slapd could accept definitions for both in slapd.conf and simply pass them on to a schema-aware LDAP client for informational purpose without enforcing them. Same function like rootDSE <file> in slapd.conf. Opinions? Ciao, Michael. -- Michael Ströder E-Mail: michael(a)stroeder.com http://www.stroeder.com

2 4

Enforcing attribute ACL on add operations
by manu＠netbsd.org 04 Oct '08

04 Oct '08

Hello Right now, slapd ignore attribute ACL when performing an add operation. If you have privilegied users that can add entries, it means that you can prevent them from modifying attributes in existing entries, but you cannot prevent them from adding an entry with a read-only attribute. The problem can be interesting with an attribute such as authzTo, where the whole access control can be circumvented by any user that can create an entry in the tree. IMO this behavior was not intended, but if it was, then it should be clearly documented. Below is a patch that cause attribute ACL to be checked for add operations. It is done in the backend, so if it is acceptable, then I will have to do it for other backends. I wonder if the modrdn operation shoulnd't be subject to the same sanity checks. Any thought? Does it look right? diff -U2 -r1.174 add.c --- servers/slapd/back-bdb/add.c 26 Aug 2008 23:45:35 -0000 1.174 +++ servers/slapd/back-bdb/add.c 27 Sep 2008 15:54:58 -0000 @@ -300,4 +300,22 @@ } + /* + * Check ACL for attribute write access + */ + if (!acl_check_modlist(op, oe, op->ora_modlist)) { + switch( opinfo.boi_err ) { + case DB_LOCK_DEADLOCK: + case DB_LOCK_NOTGRANTED: + goto retry; + } + + Debug( LDAP_DEBUG_TRACE, + LDAP_XSTRING(bdb_add) ": no write access to attribute\n", + 0, 0, 0 ); + rs->sr_err = LDAP_INSUFFICIENT_ACCESS; + rs->sr_text = "no write access to attribute"; + goto return_results;; + } + if ( eid == NOID ) { rs->sr_err = bdb_next_id( op->o_bd, &eid ); -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz manu(a)netbsd.org

4 22

Fwd: RE24 testing
by Brett ＠Google 03 Oct '08

03 Oct '08

On Thu, Sep 11, 2008 at 2:33 AM, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > Please test RE24. > > Thanks! > > --Quanah on suse 10.2 SMP / xen kernel, ie : Linux suse 2.6.22.18-0.2-xen #1 SMP 2008-06-09 13:53:20 +0200 x86_64 x86_64 x86_64 GNU/Linux test failed - producer and consumer2 databases differ >>>>> ./scripts/test050-syncrepl-multimaster failed (exit 1) make[2]: *** [bdb-yes] Error 1 make[2]: Leaving directory `/home/maxfieb/keep-ldap/openldap-src-RE24/tests' make[1]: *** [test] Error 2 make[1]: Leaving directory `/home/maxfieb/keep-ldap/openldap-src-RE24/tests' make: *** [test] Error 2 compiling against bdb 4.7.25.NC (no patched applied) bdb options : ../dist/configure --disable-java --enable-posixmutexes --enable-largefile --enable-cxx --disable-compat185 make make install openldap configure : CFLAGS="-g -g3 -ggdb -ggdb3" CPPFLAGS="-I/usr/local/BerkeleyDB.4.7/include" \ LDFLAGS="-L/usr/local/BerkeleyDB.4.7/lib -R/usr/local/BerkeleyDB.4.7/lib" \ ./configure --enable-slapd --enable-backends --enable-shared \ --enable-overlays --disable-ipv6 --disable-sql --disable-ndb --with-threads \ --enable-ssl --enable-collect --enable-debug building HEAD with same openldap configure options and same bdb binaries above, all tests succeed (albeit much more slowly.. perhaps DB_CONFIG needs tweaking in head ?).. Cheers Brett

4 9

Please test RE24
by Quanah Gibson-Mount 03 Oct '08

03 Oct '08

Please test RE24. Issues remain with test050, but are not considered release blocking as they are not a regression. Thanks, Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

7 18

Re: commit: ldap/servers/slapd/overlays ppolicy.c
by Pierangelo Masarati 02 Oct '08

02 Oct '08

ando(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/servers/slapd/overlays > > Modified Files: > ppolicy.c 1.116 -> 1.117 > > Log Message: > allow controls deletion to be deferred; pass response controls back to frontend in extops (ITS#5711) Please disregard, this patch is broken. A fix will appear shortly. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ----------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Fax: +39 0382 476497 Email: ando(a)sys-net.it -----------------------------------

3 3

Re: Please test RE24
by ghenry＠OpenLDAP.org 30 Sep '08

30 Sep '08

----- "Quanah Gibson-Mount" <quanah(a)zimbra.com> wrote: > Please test RE24. Issues remain with test050, but are not considered > > release blocking as they are not a regression. Only test50 fails for me on dual core 386 with 4.7.25 bdb. -- Kind Regards, Gavin Henry. OpenLDAP Engineering Team. E ghenry(a)OpenLDAP.org Community developed LDAP software. http://www.openldap.org/project/

1 0

Use of oidValidate() in schema-related attrs?
by Pierangelo Masarati 27 Sep '08

27 Sep '08

We currently use oidValidate() in the hardcoded definition of attributes like dITContentRules, matchingRules, attributeTypes, objectClasses, nameForms, matchingRuleUse. However, this only looks correct as a validator for an assertion value of the related syntaxes, while it fails for valid attribute values. I understand this is a moot point, since we do not allow direct writing of cn=subschema and subschema subentries in general. However this causes, for example, an incorrect error to be returned when one attempts to write, for example, cn=subschema. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ----------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Fax: +39 0382 476497 Email: ando(a)sys-net.it -----------------------------------

1 0

New test054-syncrepl-asymmetric
by Rein Tollevik 25 Sep '08

25 Sep '08

I have created a test script that shows the contextCSN propagation problems we have, as well as a patch that fixes most of them. But before I commit anything or files more ITSs I'd like to receive comments. The test script and patch can be found at: ftp://ftp.openldap.org/incoming/test054-syncrepl-asymmetric ftp://ftp.openldap.org/incoming/test054-syncrepl-asymmetric.patch The comments in the scripts should hopefully explain the configuration. The patch adds support for the newCookie sync info protocol messages, and sends them when the contextCSN is being updated without any changes being sent from the syncprov provider (i.e when the entries that changed didn't match the syncrepl filter and/or was forbidden by ACL rules). The queuing of the CSNs done in syncrepl_updateCookie should not be necessary with this patch, it should be removed after the patch has been committed. The bug reported in ITS#5710 is also worked around by this patch, but defining SLAP_SYNC_UPDATE_MSGID with an unique value would be better. This test requires the proposed patch to fix ITS#5572 to be applied. Dynamically creating the configuration doesn't work as expected without it, and the number of errors reported by the script increases from 10 to 25. Mostly due to the replication starting in an unexpected state, and several restarts and writes are needed before an expected state is reached. This patch can be found at: ftp://ftp.openldap.org/incoming/ITS5572.patch The test script finds 10 errors (assuming the race condition discussed below is actually hit) after the ITS#5572 patch has been applied With both patches I'm left with 3 errors, all related to the race condition. Howard, you'll probably object to the rootdn usage in this test. But as of now there is no way I know of that allows me to create the layout I need without it. Extending syncrepl with the ability to exclude a list of dn subtrees from its control could be the solution. But in that case I assume it would be better to have a list of URIs (evaluated locally) that defines the entries to exclude. Which again opens a can of new questions as to when the URIs should be evaluated. For add and delete it should be pretty obvious, but what about modify? Test against the entry as it is before the change, would be after or both? Hmm, the ability to exclude a list of URIs from syncrepl control could be (ab)used to merge some type of entries from one server, other types from other servers, if anyone should wish to do such weired things.. The errors remaining after these two patches have been applied are due to the race condition that arise if more then one subordinate database replicates from the same provider. I have a clear idea as to how this race can be eliminated, it must be addressed in both syncrepl and syncprov: In syncrepl it should be fairly easy, as storing the contextCSN values in the suffix of the database where syncrepl is configured rather than the suffix of the glue entry should be sufficient. Syncprov needs a bit more work. Whenever it detects that syncrepl updates the contextCSN value it must use the *lowest* contextCSN value for any given sid stored in all the databases within its context. And no updates must be accepted from syncrepl until a contextCSN value has been stored in the suffix dn of all databases (within its context) where syncrepl has been enabled. Assuming syncrepl cannot be used on both a superior and a subordinate database (which could no be true if the ability to exclude something from its control is introduced) it should not be required to make these extra tests when syncrepl updates the contextCSN in the suffix of the glue database itself (i.e when syncrepl and syncprov are configured in the same database). The sending of newCookie messages should cause the contextCSN values to be updated to the newest values fairly quickly. Actually, there could be another problem buried here as well, as the consumers really need a full resynchronization after new subordinate syncrepl consumers are added on their provider (at least if the new consumer replicates from the same server as an existing). The easiest fix would be to require the clients to do a full resync whenever the config of the provider is changed, which I find a bit drastic. Noting it in the doc. should hopefully be sufficient. Rein Tollevik Basefarm AS

1 0

Re: commit: ldap/servers/slapd/back-bdb add.c back-bdb.h bind.c cache.c compare.c config.c delete.c dn2entry.c dn2id.c filterindex.c id2entry.c idl.c init.c key.c modify.c modrdn.c proto-bdb.h referral.c search.c tools.c
by Howard Chu 25 Sep '08

25 Sep '08

hyc(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/servers/slapd/back-bdb > > Modified Files: > add.c 1.173 -> 1.174 > back-bdb.h 1.173 -> 1.174 > bind.c 1.54 -> 1.55 > cache.c 1.180 -> 1.181 > compare.c 1.57 -> 1.58 > config.c 1.107 -> 1.108 > delete.c 1.172 -> 1.173 > dn2entry.c 1.35 -> 1.36 > dn2id.c 1.156 -> 1.157 > filterindex.c 1.75 -> 1.76 > id2entry.c 1.86 -> 1.87 > idl.c 1.134 -> 1.135 > init.c 1.291 -> 1.292 > key.c 1.24 -> 1.25 > modify.c 1.176 -> 1.177 > modrdn.c 1.200 -> 1.201 > proto-bdb.h 1.159 -> 1.160 > referral.c 1.50 -> 1.51 > search.c 1.265 -> 1.266 > tools.c 1.127 -> 1.128 > > Log Message: > Use read-only txn's instead of read lockers. Support BDB 4.4-4.7 Please test... At the moment I've only run it with BDB 4.7.25. I'll be testing with older releases over the next couple of days. In particular, take note of any LDAP_OTHER/Internal Error results in the test suite. Right now it passes all tests for me on bdb and hdb. If push comes to shove we can probably create a new BerkeleyDB4.2.patch file to get 4.2 working again, but at this point most people should already be on something newer, and people stuck on 4.2 probably won't compile their own anyway. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 4

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel September 2008