openldap-devel June 2008

openldap-devel@openldap.org

19 participants
18 discussions

contextCSN propagation problems
by Rein Tollevik 10 Jun '08

10 Jun '08

The worst problems we had after upgrading to 2.4.x seems to be over now, and the replication appear to work (at least mostly) as it should. One problem that still remains is that the contextCSN attributes themselves don't propagate as I wish they would. Before I start coding or filing more ITSes I would like to have comments as to whether my analysis of the problem seems correct or not. First, a description of our configuration is needed. I think a test script for this configuration would be nice to have, so I'm going to create that before I do anything else. I currently think of it as syncrepl-asymmetric, but suggestions for better names are welcome. Now the configuration, which is centered around a central master server. It has a glued database with a set of subordinates. The central master is the (sole) master for most of these subordinates, but there are also a set of remote site-masters that each has one subordinate database that they are the master for and that is replicated back to the central master. The site-masters has a similar glued configuration, and they replicates the glue entry from the central master. Different rootdn values on the subordinates managed by syncrepl and the one the site-master is the master for prevents syncrepl from wiping out the content of that database during the present phase. None of the site-masters receives all the subordinate databases the central master has, and which they receive varies. This is controlled by acl rules on the central master. On each of the sites (including where the central master is) there are search-only servers that replicates the glue suffix from their site-master (or the central master). The search servers has a single database (for historical reasons), but their layout shouldn't matter very much. All of the master servers uses syncprov on the glue database, and everyone except the central master uses syncrepl on the glue database. The central master cannot use it there, as that would have caused it to wipe out those subordinates that aren't on the site-master it replicates from during the refresh phase. Different serverIDs are used on the master servers, so an updated contextCSN set should include as many values as there are master servers. Now to the problem. If a modification is made (on the central master) to a subordinate database that isn't replicated to one of its consumers it will not receive the updated contextCSN value from the central master. Which means that I cannot monitor the contextCSN values to verify that the replication is working as it should. And the consumers will (after a restart) present an outdated contextCSN set during the refresh phase, even though their database content is up to date. But this will be corrected when/if an update is made to a subordinate db that is replicated to the consumer. A worse problem is when the modifications is made to a subordinate db the central master replicates from one of the site-masters. In this case there will never be any updates from that site-master that should be propagated. So the consumers will, until it restarts, be stuck with the contextCSN value matching that remote site-master which it received after the present phase. Or, oh well, that is not quite true... Due to a bug in syncprov it fails (on the central master) to detect and filter out the contextCSN update from syncrepl on a subordinate db. Which means that the central master will send an update of the glue entry to its consumers, so that the value is propagated from the central master to its immediate consumers. But it will not propagate further from the site-masters to the search servers on their site. As this bug makes things work at least partly as I wish I'm a bit reluctant to have it fixed yet... So far my proposed solution to this problem is that syncprov_matchops() should, when a modification fails to match the test_filter() nor is a deleted entry, send a sync info protocol message with the updated contextCSN in the newCookie field to its consumers. Does this sound like a valid solution? There seem to be support in syncrepl.c for receiving these messages, and in syncprov.c for sending them. But it never actually does it as far as I can see. -- Rein Tollevik Basefarm AS

1 0

demand slapindex after slapd.conf += index
by Hallvard B Furuseth 10 Jun '08

10 Jun '08

A suggestion for a vain attempt to kill a FAQ: To teach people to run sbin/slapindex after adding an index to slapd.conf, slapd can refuse to start if an index file is missing but dn2id.bdb exists. Same with other slap tools that need to open/create the index file. (At least slapadd, I don't know if there are others.) slapindex and slapcat, at least, will need to keep working. If database recovery can sabotage this feature or vice versa, recovery will of course have to take precedence. -- Hallvard

2 2

RE24 testing (again)
by Quanah Gibson-Mount 06 Jun '08

06 Jun '08

Pulled in a few more things, please re-sync and test. These were the last outstanding changes known for 2.4.10. Thanks! --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

7 6

Re: RE24 testing (again)
by Raphaël Ouazana-Sustowski 03 Jun '08

03 Jun '08

Hi, Le Mar 3 juin 2008 10:37, Jonathan Clarke a écrit : > Gavin Henry a écrit : >> <quote who="Howard Chu"> >>> Quanah Gibson-Mount wrote: >>>> Pulled in a few more things, please re-sync and test. These were the >>>> last >>>> outstanding changes known for 2.4.10. >>> All tests OK here. OpenSuSE 10.2, x86-64 >> >> Likewise. Fedora 8/9 i386/x86_64, Centos 5.2 i386/x86_64 and Debian Etch >> (4.0) i386/x86_64 > > Also OK for me on Ubuntu i386 and Debian x86_64. Idem here on Ubuntu/i386/dual core. Note that this time test050 (on hdb) ran more then 200 times without any problem. Regards, Raphaël Ouazana.

1 0

AVL -> T-trees
by Howard Chu 02 Jun '08

02 Jun '08

Taking a cue from our MySQL friends - MySQL uses T-trees for their in-memory structures. These are balanced trees, like AVL trees. But instead of just one data item per tree node, they have N items per node. (Presumably N is a compile-time constant.) The advantage to using T-trees is that inserts and deletes have less impact on the overall tree, thus minimizing the need for rebalancing. I would expect that they perform about as well as AVL trees for lookups. Anyone interested in experimenting here and reporting on the relative performance? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

3 4

Re: AVL -> T-trees
by Howard Chu 02 Jun '08

02 Jun '08

Simon Riggs wrote: > Hi Gavin, > > PostgreSQL uses B-trees as the main index type, though also supports > GIST, GIN and Hash index types. We avoid re-balancing the trees at > delete time, which is a high source of contention. > > MySQL, to my knowledge, doesn't support T-Trees? > http://dev.mysql.com/doc/refman/5.0/en/create-index.html As I noted to Gavin, that's a different thing entirely. Our on-disk indexing uses B-trees as well, but our in-memory caches use AVL trees. I would love to try a B-link tree implementation for our on-disk data, but that's yet another story... > T-Trees are a class of index known to be cache conscious, but they also > perform poorly when I/O is involved. Selecting that would mean you'd > need to know in advance the memory allocation given to your server, > which may not be as much as you want and can change over time. You can > have non-persistent indexes, though rebuilding indexes at startup can be > annoying also. I/O is not a consideration for this purpose. > Does OpenLDAP experience a high insert/delete rate? I would have > expected most uses to be mostly read-only, percentage-wise, but I guess > you'll set me straight. Yes, there is a high rate of insert/deletes into the caches when the cache is smaller than the working set. Since T-trees allow more data items to be stored per tree node, that also means we could have less overhead in tree link pointers. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Please retest RE24
by Quanah Gibson-Mount 02 Jun '08

02 Jun '08

Two new bug fixes for RE24 committed today. Please retest. Hopefully this'll be all for 2.4.10. Thanks!! --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

5 4

Re: commit: ldap/contrib/slapd-modules/nssov - New directory
by Howard Chu 02 Jun '08

02 Jun '08

hyc(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/contrib/slapd-modules/nssov > Log Message: > Directory /repo/OpenLDAP/pkg/ldap/contrib/slapd-modules/nssov added to the repository I've imported a complete copy of Arthur de Jong's nss-ldapd-0.6.2 with this overlay. The overlay implements a listener inside slapd that speaks the same protocol as nss-ldapd. As such, it replaces the server side of his package (nslcd). You still need to build and install his client side though (nss). It passes the majority of the "make check" tests in the nss-ldapd/tests directory on my OpenSuSE system. The ones that fail appear to be incorrectly written tests. Unfortunately those tests are heavily dependent on your system's nsswitch.conf and the other databases; they ought to be cleaned up to be completely self-contained. The point of all this: the nss-ldapd approach avoids the issue of polluting the user space with libldap's symbols, by sending all requests thru a small nss stub. This stub sends requests (using a very simple protocol) over a Unix Domain socket to some other server which actually processes the requests. With the original nss-ldapd, an nslcd daemon listening on that socket then uses libldap to contact whatever LDAP server was configured. With this overlay, slapd itself answers the nss requests. On a host with the master database, this avoids an unnecessary context switch if nothing else. Even on hosts without the full database, this approach opens up the possibility of using pcache to perform intelligent caching of nss data, as well as using syncrepl to keep information current. I've also discussed with Arthur some directions for improvement in the base nss code. I may push some of those into here later. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel June 2008