Dear list!
We found one issue with a quite simple syncrepl replication; we have one provider and just one consumer. The consumer got set up about a week ago and loaded its initial database content from the provider. Then - after a couple of days - one object got deleted on the provider. So the consumer had to perform that delete operation as well; which is what it did fine. No problem so far.
Today we compared the contextCSN values for the provider and the consumer and first thought we may have a problem, actually, because we found the contextCSN on the consumer to be newer than on the provider. Here are the actual values:
consumer
contextCSN: 20111216215531.923716Z#000000#003#000000
provider
contextCSN: 20110124140058.246484Z#000000#003#000000
Looking twice, we found out that on the provider we got the above contextCSN value (the one which is too small) using slapcat. If we ask the provider using ldapsearch it actually reports the expectec contextCSN value, i.e. the same one which we can see on the consumer. (Note: The above value on the consumer has been taken from slapcat as well.
In other words: The provider's latest contextCSN value has not been written to the disk until now; actually 4 days since the change.
I guess if the power would fail now for example, the provider would load it's context CSN from the database, reading the lower value, and thus replication would get seriously out of sync, wouldn't it.
Also we found that the result of the operation which increased the contextCSN (deleting an object on the provider) has been written to the disk, in contrast to the contextCSN value.
Is that epxected behaviour or would we need to configure the flushing to the disk?
We're using OpenLDAP 2.4.23 on Debian Linux. The backend is a back-hdb.
Regards, Torsten
--On Tuesday, December 20, 2011 8:39 PM +0000 "Torsten Schlabach (Tascel eG)" tschlabach@tascel.net wrote:
We're using OpenLDAP 2.4.23 on Debian Linux. The backend is a back-hdb.
Upgrade.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
--On Tuesday, December 20, 2011 12:54 PM -0800 Quanah Gibson-Mount quanah@zimbra.com wrote:
--On Tuesday, December 20, 2011 8:39 PM +0000 "Torsten Schlabach (Tascel eG)" tschlabach@tascel.net wrote:
We're using OpenLDAP 2.4.23 on Debian Linux. The backend is a back-hdb.
Upgrade.
A more complete answer (other than you really do need to upgrade, and for sanity, rebuild the OpenLDAP you have against OpenSSL), is that you have syncprov checkpointing on. So the CSN is only updated in the underlying DB based off the period of time you specified to the overlay. Thus when slapd is running, ldapsearch is always going to give you the correct answer. slapcat may or may not. If you stop slapd, slapcat will.
From the manual page:
syncprov-checkpoint <ops> <minutes> After a write operation has succeeded, write the contextCSN to the underlying database if <ops> write operations or more than <minutes> time have passed since the last checkpoint. Checkpointing is disabled by default.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Hi Quanah!
First of all: Thanks for answering the original question. I was wondering first why changes to the database content as such have been persisted and the new contextCSN hasn't until I found that this syncprov-checkpoint thing actually belongs to the syncprov overlay. So I guess it makes sense to switch that on as otherwise - for example after a power failure - I would have lost information, wouldn't I?
Now please allow me some remarks about the part of your answer which I did expect and where you met my expectation straight:
We're using OpenLDAP 2.4.23 on Debian Linux. The backend is a
back-hdb.
Upgrade.
I understand that the behavior which I asked about originally is NOT a bug but "works as designed", so why should I upgrade then?
Actually, yes, newer is better most of the time in OSS land, but still, in a production system IMO you cannot upgrade your software every week, can you?
The other part of the problem is that OpenLDAP is releasing a lot faster than most distros manage to update their packages. For example, in Debian right now they are working on getting 2.4.25 into "experimental" while you're at 2.4.28 already. On average, Debian as well as Ubuntu and others are frequently 2-3 releases behind the current one.
I by no means want to blame the OpenLDAP team for releasing, so maybe I am preaching to the wrong audience here, but I want to run my productive systems with packaged software, not with software which I compiled on the fly from a tarball. So it would be nice if the Debian team would just be faster to package OpenLDAP and maybe move it to the volatile section of their distribution.
Another alternative would of course be a PPA or any other mechanism where the OpenLDAP team would release binary packages for Debian (and possibly other distros as well) right with every new release. Or does something like that exist somewhere and I just did not yet find it?
Regards, Torsten
On Tue, 20 Dec 2011 17:01:19 -0800, Quanah Gibson-Mount quanah@zimbra.com wrote:
--On Tuesday, December 20, 2011 12:54 PM -0800 Quanah Gibson-Mount quanah@zimbra.com wrote:
--On Tuesday, December 20, 2011 8:39 PM +0000 "Torsten Schlabach
(Tascel
eG)" tschlabach@tascel.net wrote:
We're using OpenLDAP 2.4.23 on Debian Linux. The backend is a
back-hdb.
Upgrade.
A more complete answer (other than you really do need to upgrade, and
for
sanity, rebuild the OpenLDAP you have against OpenSSL), is that you have
syncprov checkpointing on. So the CSN is only updated in the underlying DB based off the period of time you specified to the overlay. Thus when slapd is running, ldapsearch is always going to give you the correct answer. slapcat may or may not. If you stop slapd, slapcat will.
From the manual page:
syncprov-checkpoint <ops> <minutes> After a write operation has succeeded, write the contextCSN to the underlying database if <ops> write operations or more than <minutes>
time
have passed since the last checkpoint. Checkpointing is disabled by default.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc.
Zimbra :: the leader in open source messaging and collaboration
Torsten Schlabach (Tascel eG) wrote:
Hi Quanah!
First of all: Thanks for answering the original question. I was wondering first why changes to the database content as such have been persisted and the new contextCSN hasn't until I found that this syncprov-checkpoint thing actually belongs to the syncprov overlay. So I guess it makes sense to switch that on as otherwise - for example after a power failure - I would have lost information, wouldn't I?
No, of course not. If there was a danger of data loss we certainly would not leave that as the default behavior.
In fact the syncprov overlay always reads the largest entryCSN out of the database at startup time anyway, so even if it didn't save the contextCSN before stopping, it will be up to date on every startup. Checkpointing the CSN is only an optimization, it will reduce the length of the search time on the next startup.
--On Wednesday, December 21, 2011 4:36 PM +0000 "Torsten Schlabach (Tascel eG)" tschlabach@tascel.net wrote:
We're using OpenLDAP 2.4.23 on Debian Linux. The backend is a
back-hdb.
Upgrade.
Actually, yes, newer is better most of the time in OSS land, but still, in a production system IMO you cannot upgrade your software every week, can you?
Certainly no one is asking that be done. In fact, OpenLDAP has a fairly slow release cycle. However, it also can be expected to update periodically.
The other part of the problem is that OpenLDAP is releasing a lot faster than most distros manage to update their packages. For example, in Debian right now they are working on getting 2.4.25 into "experimental" while you're at 2.4.28 already. On average, Debian as well as Ubuntu and others are frequently 2-3 releases behind the current one.
This question has come up so often that the Debian maintainers actually contributed an FAQ on the topic: http://www.openldap.org/faq/data/cache/1456.html
You may find it helpful. Debian also has the complication that they want to build OpenLDAP against GnuTLS, which has caused all sorts of nasty problems, and GnuTLS remains an additional security risk as well due the the way in which it is coded. When using Debian/Ubuntu, I always advise people to (a) build their own packages and (b) ensure they are linked against OpenSSL instead of GnuTLS.
I would also strongly advise you to read the change history between 2.4.23 and at least 2.4.26.http://www.openldap.org/software/release/changes.html
And yes, I personally build and upgrade OpenLDAP for the various production environments I've worked in, both at my previous job and my current job. I also don't build out every release. There are clearly times when it isn't necessary. However, in Debian's case, neither 2.4.23 or 2.4.25 are what I would consider suitable releases for a production OpenLDAP installation. 2.4.26 has been quite good for me, and I'll be moving to 2.4.28 in the near future.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Howard!
In fact the syncprov overlay always reads the largest entryCSN out of
the
database at startup time anyway, so even if it didn't save the
contextCSN
before stopping, it will be up to date on every startup.
I wonder how that works if the last operation before the unclean shutdown has been a delete operation?
IIUC, then on a delete, the contextCSN is updated but you would have to read the deleted entry upon the next startup to get the contextCSN back, wouldn't you?
I didn't actually try it, but I bet that in the discussed scenario, i.e.:
- delete an entry from master - replicate deletion to slaves - master crashes - master is restarted
you would end up with a "slave newer than master" kind of situation, which IIUC, is considered an error, isn't it?
Or do you keep a log of operations (which is persisted immediately / timely) which syncprov may read also upon startup to learn about the delete operation and the resulting contextCSN? If you just scan for the still existing entry with the largest entryCSN number, you will detinitely end up with a to low contextCSN of the master in case of the above scenario IMO.
Regards, Torsten
On Wed, 21 Dec 2011 10:29:08 -0800, Howard Chu hyc@symas.com wrote:
Torsten Schlabach (Tascel eG) wrote:
Hi Quanah!
First of all: Thanks for answering the original question. I was
wondering
first why changes to the database content as such have been persisted
and
the new contextCSN hasn't until I found that this syncprov-checkpoint thing actually belongs to the syncprov overlay. So I guess it makes sense to switch that on as otherwise - for example after a power failure - I
would
have lost information, wouldn't I?
No, of course not. If there was a danger of data loss we certainly would not leave that as the default behavior.
In fact the syncprov overlay always reads the largest entryCSN out of
the
database at startup time anyway, so even if it didn't save the
contextCSN
before stopping, it will be up to date on every startup. Checkpointing
the
CSN is only an optimization, it will reduce the length of the search time on the next startup.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Torsten Schlabach (Tascel eG) wrote:
Howard!
In fact the syncprov overlay always reads the largest entryCSN out of
the
database at startup time anyway, so even if it didn't save the
contextCSN
before stopping, it will be up to date on every startup.
I wonder how that works if the last operation before the unclean shutdown has been a delete operation?
IIUC, then on a delete, the contextCSN is updated but you would have to read the deleted entry upon the next startup to get the contextCSN back, wouldn't you?
I didn't actually try it, but I bet that in the discussed scenario, i.e.:
- delete an entry from master
- replicate deletion to slaves
- master crashes
- master is restarted
you would end up with a "slave newer than master" kind of situation, which IIUC, is considered an error, isn't it?
Or do you keep a log of operations (which is persisted immediately / timely) which syncprov may read also upon startup to learn about the delete operation and the resulting contextCSN? If you just scan for the still existing entry with the largest entryCSN number, you will detinitely end up with a to low contextCSN of the master in case of the above scenario IMO.
In regular syncrepl there is no persistent log. In delta-syncrepl yes, there is a persistent log.
The situation you describe would have caused problems for regular syncrepl, but in fact we've got a workaround for that as well.
Regards, Torsten
On Wed, 21 Dec 2011 10:29:08 -0800, Howard Chuhyc@symas.com wrote:
Torsten Schlabach (Tascel eG) wrote:
Hi Quanah!
First of all: Thanks for answering the original question. I was
wondering
first why changes to the database content as such have been persisted
and
the new contextCSN hasn't until I found that this syncprov-checkpoint thing actually belongs to the syncprov overlay. So I guess it makes sense to switch that on as otherwise - for example after a power failure - I
would
have lost information, wouldn't I?
No, of course not. If there was a danger of data loss we certainly would not leave that as the default behavior.
In fact the syncprov overlay always reads the largest entryCSN out of
the
database at startup time anyway, so even if it didn't save the
contextCSN
before stopping, it will be up to date on every startup. Checkpointing
the
CSN is only an optimization, it will reduce the length of the search time on the next startup.
openldap-technical@openldap.org
-
Howard Chu -
Quanah Gibson-Mount -
Torsten Schlabach (Tascel eG)