Dear Folks,
our old backup system for our LDAP clusters using slurpd is as follows:
1. A dedicated slave (we call it a "sync") has a replogfile configured. 2. A cron job moves the replog file to a backup directory every 15 minutes. 3. Another cron job runs every 24 hours, which: a. Stops slapd. b. slapcats to a file which is compressed with gzip. c. starts slapd. 4. Another system backs up the backup directory.
Now we like text formatted backups, and I need to implement a similar or better system with syncrepl.
Here are my questions:
1. Would the auditlog overlay be an appropriate replacement for the replogfile? 2. Can we run this on the master without stopping slapd? 3. Would we be able to accurately reconstruct the data using the slapcat and periodic copies of auditlog files?
I will write software to fully automate whatever scheme we determine to be the most appropriate.
--On Thursday, June 24, 2010 12:19 PM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote:
Dear Folks,
our old backup system for our LDAP clusters using slurpd is as follows:
Just use slapcat. auditlog is not appropriate.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Dear Quanah,
On 23/06/10 21:46 -0700, Quanah Gibson-Mount wrote:
--On Thursday, June 24, 2010 12:19 PM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote:
Dear Folks,
our old backup system for our LDAP clusters using slurpd is as follows:
Just use slapcat.
# du -sh /var/lib/ldap 7.4G /var/lib/ldap
Slapcatting this lot every 15 minutes is likely to make a machine less capable of doing other work.
auditlog is not appropriate.
Please can you explain why this is so?
Do you think we are wrong now to be using the replogfile to achieve what is in effect an incremental textual backup every fifteen minutes?
Can you suggest any reasonable alternative to auditlog of achieving the ability to reconstruct the LDAP databases from textual LDIF files losing a maximum of fifteen minutes of data?
--On Friday, June 25, 2010 9:58 AM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote:
Dear Quanah,
On 23/06/10 21:46 -0700, Quanah Gibson-Mount wrote:
--On Thursday, June 24, 2010 12:19 PM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote:
Dear Folks,
our old backup system for our LDAP clusters using slurpd is as follows:
Just use slapcat.
# du -sh /var/lib/ldap 7.4G /var/lib/ldap
Slapcatting this lot every 15 minutes is likely to make a machine less capable of doing other work.
Slapcat doesn't take a lot of resources, in my experience, and the generated LDIF is often significantly smaller than the actual database size. In fact I don't think I've ever seen an LDIF that's larger than the DB. Note that the database has files for any number of things (DB environment, indices, etc). Which is part of why there is no exact correlation between slapcat'd LDIF and the size of the DB directory.
auditlog is not appropriate.
Please can you explain why this is so?
Auditlog is a poorly written overlay that hopefully will be deleted in the future. It kills perf, and in general is not recommended for use. If you want a record of changes, I suggest you look at the accesslog overlay.
Do you think we are wrong now to be using the replogfile to achieve what is in effect an incremental textual backup every fifteen minutes?
I would suggest you have multiple replicas, thus being able to restore any given server from any of the other existing replicas. If you have a disaster that wipes out every replica, you're in a world of hurt as it is. Although I've used slapcat's for DR purposes, I've never needed it, as I simply used one of my other servers to restore a lost database.
One of the advantages of syncrepl (Although I use and prefer delta-syncrepl, since syncrepl has not proven reliable in my experience) is that it can "catch up" from any given point with the data from an LDIF file or a copy of another replica's DB, if it is behind the master.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Dear Quanah,
Thanks again for taking the time to write a detailed and helpful reply.
On 25/06/10 10:41 -0700, Quanah Gibson-Mount wrote:
--On Friday, June 25, 2010 9:58 AM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote:
On 23/06/10 21:46 -0700, Quanah Gibson-Mount wrote:
--On Thursday, June 24, 2010 12:19 PM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote: auditlog is not appropriate.
Please can you explain why this is so?
Auditlog is a poorly written overlay that hopefully will be deleted in the future. It kills perf, and in general is not recommended for use. If you want a record of changes, I suggest you look at the accesslog overlay.
1. Can we use an LDIF backend with the auditlog overlay?
2. Has anyone written a program to convert the auditlog data to an LDIF format that can be applied with ldapmodify?
One of the advantages of syncrepl (Although I use and prefer delta-syncrepl, since syncrepl has not proven reliable in my experience)
We are avoiding delta-syncrepl because, for reliability, we want to use mirror-mode, and at http://www.openldap.org/doc/admin24/replication.html#MirrorMode%20replicatio... under section 18.2.3.2. Arguments against MirrorMode is written:
"Delta-Syncrepl is not yet supported"
3. Does this mean that we are aiming for a less reliable replication with syncrepl that with delta-syncrepl?
4. Is there any prospect of delta-syncrepl being supported with mirror-mode?
We will use something like heartbeat or corosync/pacemaker to limit writes to one of the two mirror-mode masters.
--On Monday, June 28, 2010 11:42 AM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote:
Dear Quanah,
Thanks again for taking the time to write a detailed and helpful reply.
On 25/06/10 10:41 -0700, Quanah Gibson-Mount wrote:
--On Friday, June 25, 2010 9:58 AM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote:
On 23/06/10 21:46 -0700, Quanah Gibson-Mount wrote:
--On Thursday, June 24, 2010 12:19 PM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote: auditlog is not appropriate.
Please can you explain why this is so?
Auditlog is a poorly written overlay that hopefully will be deleted in the future. It kills perf, and in general is not recommended for use. If you want a record of changes, I suggest you look at the accesslog overlay.
- Can we use an LDIF backend with the auditlog overlay?
Why are you stuck on auditlog? As I said, don't use it.
- Has anyone written a program to convert the auditlog data to an LDIF format that can be applied with ldapmodify?
No idea. As I said, it's not really a usable piece of software.
One of the advantages of syncrepl (Although I use and prefer delta-syncrepl, since syncrepl has not proven reliable in my experience)
We are avoiding delta-syncrepl because, for reliability, we want to use mirror-mode, and at http://www.openldap.org/doc/admin24/replication.html#MirrorMode%20replica tion under section 18.2.3.2. Arguments against MirrorMode is written:
"Delta-Syncrepl is not yet supported"
- Does this mean that we are aiming for a less reliable replication with syncrepl that with delta-syncrepl?
You can still enable the accesslog overlay regardless of what replication mechanism is used between the mirrors. As for the reliability, as I said, in my experience, delta-syncrepl has been reliable.
- Is there any prospect of delta-syncrepl being supported with mirror-mode?
Yes, that's a future request.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Dear Quanah,
On 29/06/10 08:51 -0700, Quanah Gibson-Mount wrote:
--On Monday, June 28, 2010 11:42 AM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote:
On 25/06/10 10:41 -0700, Quanah Gibson-Mount wrote:
--On Friday, June 25, 2010 9:58 AM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote:
On 23/06/10 21:46 -0700, Quanah Gibson-Mount wrote:
Please can you explain why this is so?
Auditlog is a poorly written overlay that hopefully will be deleted in the future. It kills perf, and in general is not recommended for use. If you want a record of changes, I suggest you look at the accesslog overlay.
- Can we use an LDIF backend with the auditlog overlay?
Why are you stuck on auditlog? As I said, don't use it.
Sorry, I was too tired; I mean: "1. Can we use an LDIF backend with the accesslog overlay?"
- Has anyone written a program to convert the auditlog data to an LDIF format that can be applied with ldapmodify?
No idea. As I said, it's not really a usable piece of software.
Similarly, I meant: "2. Has anyone written a program to convert the accesslog data to an LDIF format that can be applied with ldapmodify?"
One of the advantages of syncrepl (Although I use and prefer delta-syncrepl, since syncrepl has not proven reliable in my experience)
We are avoiding delta-syncrepl because, for reliability, we want to use mirror-mode, and at http://www.openldap.org/doc/admin24/replication.html#MirrorMode%20replicatio... under section 18.2.3.2. Arguments against MirrorMode is written:
"Delta-Syncrepl is not yet supported"
- Does this mean that we are aiming for a less reliable replication
with syncrepl that with delta-syncrepl?
You can still enable the accesslog overlay regardless of what replication mechanism is used between the mirrors.
Yes, I understand that.
As for the reliability, as I said, in my experience, delta-syncrepl has been reliable.
But has plain syncrepl been unreliable? If so, in what way?
- Is there any prospect of delta-syncrepl being supported with mirror-mode?
Yes, that's a future request.
Would you suggest we avoid using mirror mode so that we can use delta-syncrepl in order to achieve more reliable replication?
--On Wednesday, June 30, 2010 2:43 PM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote:
- Can we use an LDIF backend with the auditlog overlay?
Why are you stuck on auditlog? As I said, don't use it.
Sorry, I was too tired; I mean: "1. Can we use an LDIF backend with the accesslog overlay?"
Hm, that I don't know, I've never tried it. Nor would I plan on it. I'd just slapcat it if I wanted it in LDIF format.
- Has anyone written a program to convert the auditlog data to an LDIF format that can be applied with ldapmodify?
No idea. As I said, it's not really a usable piece of software.
Similarly, I meant: "2. Has anyone written a program to convert the accesslog data to an LDIF format that can be applied with ldapmodify?"
Not that I'm aware of. You could, of course, write your own delta-syncrepl client to write out any changes it received in whatever format you wanted to whatever file you wanted. Probably fairly trivial to do with Perl and Net::LDAPapi.
I discussed auditlog with Howard yesterday, and he says it is not so bad as I make out. So if you decide to go that route, let us know how it works out.
As for the reliability, as I said, in my experience, delta-syncrepl has been reliable.
But has plain syncrepl been unreliable? If so, in what way?
Read the change logs for the OpenLDAP releases. Except for a minor edge issue fixed in 2.4.22 that took some 3+ years to find, the last time there was a fix for delta-syncrepl was June 8, 2008. Compare that to the fixes for normal syncrepl.
- Is there any prospect of delta-syncrepl being supported with mirror-mode?
Yes, that's a future request.
Would you suggest we avoid using mirror mode so that we can use delta-syncrepl in order to achieve more reliable replication?
That's a decision I can't make for you. I like the idea of a hot standby master, but one could always promote a replica to a master as well, it just takes a bit longer.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
openldap-technical@openldap.org
-
Nick Urbanik -
Quanah Gibson-Mount