Hello.
I'm trying to follow change propagations errors using delta-synrepl in search & persist mode, by using an accesslog overlay both on master and on slaves. I've a few questions for which I didn't found answer in the documentationn.
First, I noticed than changes on the slaves seems to be written from rootdn, not from the dn declared in the syncrepl directive, which seems to be only used on the master, meaning I don't need any specific ACL on the slave. Is this correct ?
Second, I have several slaves synchronizing on the same master. From slapd.conf man page, rid is supposed to be unique in the consumer only, meaning all my slaves can safely use the same rid (easier for maintaining centralized configurations). Is this correct interpretation ?
Third, I noticed a lot of errors correction for syncrepl in openldap changelog. As I can't easily change installed versions (our policy is to stick with our distribution provided package, meaning a mix of 2.3.27 and 2.3.34), am I correct assuming 'refresh only' mode is less fragile than 'refresh & persist' mode, and than total synchronisation is also less fragile than delta synchronisation, if I need to fallback on a safer mode ?
Guillaume Rousse a écrit :
Hello.
I'm trying to follow change propagations errors using delta-synrepl in search & persist mode, by using an accesslog overlay both on master and on slaves.
About the error themselves, I clearly see changes commited in the master (version 2.3.27) not being propagated to one of the slave (version 2.3.27 also), whereas propagated to the second slave (version 2.3.34), by examining the access log on both slaves.
In case it matters, the master configuration is
# log database database bdb suffix "cn=log" rootdn "cn=root,cn=log" directory /var/lib/ldap/log index entryCSN,objectClass,reqEnd,reqResult,reqStart eq
overlay syncprov syncprov-nopresent TRUE syncprov-reloadhint TRUE
# needed for replication overlay syncprov syncprov-checkpoint 100 10 syncprov-sessionlog 100
# access logs overlay accesslog logdb cn=log logops writes logsuccess TRUE # scan the accesslog DB every day, and purge entries older than 7 days logpurge 07+00:00 01+00:00
And the slaves configuration is: syncrepl rid=123 provider=ldaps://ldap.futurs.inria.fr type=refreshAndPersist logbase="cn=log" logfilter="(&(objectClass=auditWriteObject)(reqResult=0))" syncdata=accesslog searchbase="dc=futurs,dc=inria,dc=fr" scope=sub schemachecking=off bindmethod=simple binddn="cn=syncuser,dc=futurs,dc=inria,dc=fr" credentials=xxxxxxxxxx
--On Friday, October 05, 2007 4:20 PM +0200 Guillaume Rousse Guillaume.Rousse@inria.fr wrote:
Guillaume Rousse a écrit :
Hello.
I'm trying to follow change propagations errors using delta-synrepl in search & persist mode, by using an accesslog overlay both on master and on slaves.
About the error themselves, I clearly see changes commited in the master (version 2.3.27) not being propagated to one of the slave (version 2.3.27 also), whereas propagated to the second slave (version 2.3.34), by examining the access log on both slaves.
See the fixes since 2.3.27, and see my earlier reply about the major flaws in your sites policies.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount a écrit :
--On Friday, October 05, 2007 4:20 PM +0200 Guillaume Rousse Guillaume.Rousse@inria.fr wrote:
Guillaume Rousse a écrit :
Hello.
I'm trying to follow change propagations errors using delta-synrepl in search & persist mode, by using an accesslog overlay both on master and on slaves.
About the error themselves, I clearly see changes commited in the master (version 2.3.27) not being propagated to one of the slave (version 2.3.27 also), whereas propagated to the second slave (version 2.3.34), by examining the access log on both slaves.
See the fixes since 2.3.27, and see my earlier reply about the major flaws in your sites policies.
I saw, that's also why I'm trying to identify what's the exact problem I'm facing, which is quite difficult (I just found log level 16384 seems to be what i need, tough). I'd rather not upgrade just because 'it might eventually solve the issue'.
And you didn't replied to my actual question: among the four kind of syncrepl usage (delta vs total, search and persist vs refresh only), are some of them more robust than others ?
--On Tuesday, October 09, 2007 9:30 AM +0200 Guillaume Rousse Guillaume.Rousse@inria.fr wrote:
I saw, that's also why I'm trying to identify what's the exact problem I'm facing, which is quite difficult (I just found log level 16384 seems to be what i need, tough). I'd rather not upgrade just because 'it might eventually solve the issue'.
And you didn't replied to my actual question: among the four kind of syncrepl usage (delta vs total, search and persist vs refresh only), are some of them more robust than others ?
delta-syncrepl has been the most stable for me, although I believe the final issue I had seen in normal syncrepl will be fixed in the upcoming 2.3.39 release. What I'm trying to point out to you, is that several *known problems* with both syncrepl and delta-syncrepl have been fixed since the 2.3.27 release. No developer is going to want to waste the time trying to help you with an issue that's likely already solved, and certainly the symptoms you've described remind me of issues I had in past releases.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount skrev, on 09-10-2007 11:06:
[...]
And you didn't replied to my actual question: among the four kind of syncrepl usage (delta vs total, search and persist vs refresh only), are some of them more robust than others ?
delta-syncrepl has been the most stable for me, although I believe the final issue I had seen in normal syncrepl will be fixed in the upcoming 2.3.39 release. What I'm trying to point out to you, is that several *known problems* with both syncrepl and delta-syncrepl have been fixed since the 2.3.27 release. No developer is going to want to waste the time trying to help you with an issue that's likely already solved, and certainly the symptoms you've described remind me of issues I had in past releases.
Using my newly repaired crystal ball, I'd guess that many people doggedly holding on to 2.3.27 are most likely using RHEL5 or CentOS5 and scared of going outside of the latters' update frames.
Whilst Red Hat claims to backport version improvements it's pretty obvious that its policy for OpenLDAP is confined to fixing security bugs. When RHEL5 was introduced with OL 2.3.27 it was a relatively recent release and I decided to try it instead of Buchan's up to date stuff. It's basically flawed in that it doesn't allow the module flexibility that Buchan's does and maybe in other ways. Certainly delta syncrepl doesn't work as it is supposed to.
My high school site runs an OL 2.3.33 delta syncrepl provider and 3 2.3.37 refresh and persist consumers, it's utterly stable 24x7 (highest uptime is 17 days because of new-kernel reboots but with the obsolete RHEL4 there were uptimes of scores of days) and does everything it's supposed to, including chaining from one of the slaves. Yes, there's a reason I'm sticking to 2.3.33 on the provider for the time being.
Best,
--Tonni
Tony Earnshaw skrev, on 09-10-2007 13:30:
[...]
My high school site runs an OL 2.3.33 delta syncrepl provider and 3 2.3.37 refresh and persist consumers, it's utterly stable 24x7 (highest uptime is 17 days because of new-kernel reboots but with the obsolete RHEL4 there were uptimes of scores of days) and does everything it's supposed to, including chaining from one of the slaves. Yes, there's a reason I'm sticking to 2.3.33 on the provider for the time being.
Sorry, yes, all Buchan's OL.
--Tonni
--On Tuesday, October 09, 2007 1:30 PM +0200 Tony Earnshaw tonni@hetnet.nl wrote:
My high school site runs an OL 2.3.33 delta syncrepl provider and 3 2.3.37 refresh and persist consumers, it's utterly stable 24x7 (highest uptime is 17 days because of new-kernel reboots but with the obsolete RHEL4 there were uptimes of scores of days) and does everything it's supposed to, including chaining from one of the slaves. Yes, there's a reason I'm sticking to 2.3.33 on the provider for the time being.
While I'm glad it is working for you, I ran into problems using delta-syncrepl in all releases prior to 2.3.35, and even then, I had to substantially patch it with bits from 2.3.36, just the way in which they'd show up differed. Hence things like ITS#4904, ITS#4813, ITS#4720, ITS#4977 all of which were fixed after 2.3.33. So how robust your setup actually is vs your perception can be very different things. I.e., you knowingly put yourself and your community at risk by running the release you are using.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount skrev, on 09-10-2007 18:56:
[...]
While I'm glad it is working for you, I ran into problems using delta-syncrepl in all releases prior to 2.3.35, and even then, I had to substantially patch it with bits from 2.3.36, just the way in which they'd show up differed. Hence things like ITS#4904, ITS#4813, ITS#4720, ITS#4977 all of which were fixed after 2.3.33. So how robust your setup actually is vs your perception can be very different things. I.e., you knowingly put yourself and your community at risk by running the release you are using.
If it were not working, my head would be able to be compared with that of Johannes the Baptist, i.e. severed upon a silver platter.
My head is still in its place.
--Tonni
Tony Earnshaw Email: tonni at hetnet dot nl
Quanah Gibson-Mount skrev, on 09-10-2007 18:56:
[...}
While I'm glad it is working for you, I ran into problems using delta-syncrepl in all releases prior to 2.3.35, and even then, I had to substantially patch it with bits from 2.3.36, just the way in which they'd show up differed. Hence things like ITS#4904, ITS#4813, ITS#4720, ITS#4977 all of which were fixed after 2.3.33. So how robust your setup actually is vs your perception can be very different things. I.e., you knowingly put yourself and your community at risk by running the release you are using.
"Yes, there's a reason I'm sticking to 2.3.33 on the provider for the time being."
Now what could that be? (Hint: I've written about it before, but all you could comment was, that you required proof. Whilst I'm running a production machine, required to be up 24x7x52).
--Tonni
--On October 9, 2007 8:44:51 PM +0200 Tony Earnshaw tonni@hetnet.nl wrote:
Quanah Gibson-Mount skrev, on 09-10-2007 18:56:
[...}
While I'm glad it is working for you, I ran into problems using delta-syncrepl in all releases prior to 2.3.35, and even then, I had to substantially patch it with bits from 2.3.36, just the way in which they'd show up differed. Hence things like ITS#4904, ITS#4813, ITS#4720, ITS#4977 all of which were fixed after 2.3.33. So how robust your setup actually is vs your perception can be very different things. I.e., you knowingly put yourself and your community at risk by running the release you are using.
"Yes, there's a reason I'm sticking to 2.3.33 on the provider for the time being."
Now what could that be? (Hint: I've written about it before, but all you could comment was, that you required proof. Whilst I'm running a production machine, required to be up 24x7x52).
Which is what I was running as well. And it wasn't working. I'm simply pointing out that although it is working for you at *this* moment, there's no guarantee it'll work in the next, because it could hit any of those known issues at any given time. You are, essentially, playing russian roulette. But, as you note, it is your head and their platter. ;)
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount skrev, on 09-10-2007 21:22:
[...]
"Yes, there's a reason I'm sticking to 2.3.33 on the provider for the time being."
Now what could that be? (Hint: I've written about it before, but all you could comment was, that you required proof. Whilst I'm running a production machine, required to be up 24x7x52).
Which is what I was running as well. And it wasn't working. I'm simply pointing out that although it is working for you at *this* moment, there's no guarantee it'll work in the next, because it could hit any of those known issues at any given time. You are, essentially, playing russian roulette. But, as you note, it is your head and their platter. ;)
It's fall vacation for the school next week, I'll try OL 2.3.38 then - we're also going to implement ppolicy (got the latter working perfectly on my FC6 test machine). The beauty of rpms is that it's a matter of seconds to revert and completely cleanly if something goes wrong with a new software version. I'll post what happens ...
--Tonni
On 10/10/07, Tony Earnshaw tonni@hetnet.nl wrote:
It's fall vacation for the school next week, I'll try OL 2.3.38 then - we're also going to implement ppolicy (got the latter working perfectly on my FC6 test machine). The beauty of rpms is that it's a matter of seconds to revert and completely cleanly if something goes wrong with a new software version. I'll post what happens ...
Provided that it still uses the same version of BDB :)
Steph
On 10/9/07, Guillaume Rousse Guillaume.Rousse@inria.fr wrote:
And you didn't replied to my actual question: among the four kind of syncrepl usage (delta vs total, search and persist vs refresh only), are some of them more robust than others ?
Hello,
There is no need to answer that way. There is a usual reply here which goes by : you should always upgrade to the most recent. In your case, down the 2.3 road, that'd be 2.3.38.
Now from experience, having the same patch level of master/slaves keep me from more headaches. I know this is not what you want to do, but beggars can't be choosers. If time allows, you could a test server with the latest version to see if it does take your changes right, bet it actually will.
There have been a few posts on the list about differences in syncrepl full vs delta, you might want to take a look at the archives.
As for which, it all depends what you use on the side, if you use Samba (as we do), it is recommended to actually use delta.
Steph
--On Friday, October 05, 2007 9:53 AM +0200 Guillaume Rousse Guillaume.Rousse@inria.fr wrote:
First, I noticed than changes on the slaves seems to be written from rootdn, not from the dn declared in the syncrepl directive, which seems to be only used on the master, meaning I don't need any specific ACL on the slave. Is this correct ?
Right, the DN in the syncrepl directive is purely for how it performs its searches on the master.
Second, I have several slaves synchronizing on the same master. From slapd.conf man page, rid is supposed to be unique in the consumer only, meaning all my slaves can safely use the same rid (easier for maintaining centralized configurations). Is this correct interpretation ?
Correct.
Third, I noticed a lot of errors correction for syncrepl in openldap changelog. As I can't easily change installed versions (our policy is to stick with our distribution provided package, meaning a mix of 2.3.27 and 2.3.34), am I correct assuming 'refresh only' mode is less fragile than 'refresh & persist' mode, and than total synchronisation is also less fragile than delta synchronisation, if I need to fallback on a safer mode ?
Nope. And, by the way, I'd seriously examine your policy, it is mightily flawed. Distro versions are almost never meant for running OpenLDAP as a server, but for providing the client libraries. You are only setting yourself up to be shot in the foot by following your current policy. A wiser choice would be to do something like use the pre-compiled releases from Symas (http://www.symas.com/) or if you are using RedHat or CentOS, Buchan Milne's pre-compiled packages
http://staff.telkomsa.net/packages/
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
openldap-software@openldap.org
-
FRLinux -
Guillaume Rousse -
Quanah Gibson-Mount -
Tony Earnshaw