Dear Howard,

Thank you for taking the time to answer.

On 22/06/10 07:34 -0700, Howard Chu wrote:

Nick Urbanik wrote:

...

Dear Folks,

With slurpd, copying a tree from one slave to another was like this:

1. stop slapd on both slaves.
2. netcat the directory across from one slave to the other.
3. stop slurpd on master
4. edit slurpd.status to make the time and replication number match by copying that for the source to that for the destination slave.
5. start everything back up.

My question with syncrepl is:

How do I copy the database for a tree from one consumer to another consumer (of the same producer) so that the newly copied replica knows where its replication should continue from?

Is the state for replication of the database stored in the contextCSN of the suffix entry?

If so, does that mean that with syncrepl, the above operation is reduced to the following three steps?

1. Stop slapd on both consumers.
2. Netcat the database from one to the other.
3. start both consumers.

The procedures you described would only have worked in the past if all of the servers had identical architecture.

Yes, the slaves are in a cluster of eight, and are all the same architecture.

The only supported means for copying databases is using slapcat and slapadd, which use LDIF and are architecture-independent.

But in the case where the consumers are all the same architecture, do you see a problem with making binary copies of the database?

There is no need to stop the first consumer when running slapcat.

This puzzles me; with hundreds of updates happening per minute, some changes will be missed during the slapcat. Will syncrepl cope and make the new slave identical to the master?

I quote from http://www.openldap.org/doc/admin24/maintenance.html#Directory%20Backups

Slapcat can be run while slapd is active. However, one runs the risk of an inconsistent database- not from the point of slapd, but from the point of the applications using LDAP. For example, if a provisioning application performed tasks that consisted of several LDAP operations, and the slapcat took place concurrently with those operations, then there might be inconsistencies in the LDAP database from the point of view of that provisioning application and applications that depended on it. One must, therefore, be convinced something like that won't happen.

Nick Urbanik wrote:

Dear Howard,

Thank you for taking the time to answer.

On 22/06/10 07:34 -0700, Howard Chu wrote:

...

Nick Urbanik wrote:

...

Dear Folks,

With slurpd, copying a tree from one slave to another was like this:

1. stop slapd on both slaves.
2. netcat the directory across from one slave to the other.
3. stop slurpd on master
4. edit slurpd.status to make the time and replication number match by copying that for the source to that for the destination slave.
5. start everything back up.

My question with syncrepl is:

How do I copy the database for a tree from one consumer to another consumer (of the same producer) so that the newly copied replica knows where its replication should continue from?

Is the state for replication of the database stored in the contextCSN of the suffix entry?

If so, does that mean that with syncrepl, the above operation is reduced to the following three steps?

1. Stop slapd on both consumers.
2. Netcat the database from one to the other.
3. start both consumers.

The procedures you described would only have worked in the past if all of the servers had identical architecture.

Yes, the slaves are in a cluster of eight, and are all the same architecture.

...

The only supported means for copying databases is using slapcat and slapadd, which use LDIF and are architecture-independent.

But in the case where the consumers are all the same architecture, do you see a problem with making binary copies of the database?

...

There is no need to stop the first consumer when running slapcat.

This puzzles me; with hundreds of updates happening per minute, some changes will be missed during the slapcat. Will syncrepl cope and make the new slave identical to the master?

I quote from http://www.openldap.org/doc/admin24/maintenance.html#Directory%20Backups

Slapcat can be run while slapd is active. However, one runs the risk
of an inconsistent database- not from the point of slapd, but from the
point of the applications using LDAP. For example, if a provisioning
application performed tasks that consisted of several LDAP operations,
and the slapcat took place concurrently with those operations, then
there might be inconsistencies in the LDAP database from the point of
view of that provisioning application and applications that depended
on it. One must, therefore, be convinced something like that won't
happen.

The note here says you must convince yourself...

As already noted in other posts, the contextCSN is recorded in the suffix entry of the DB, and all replication state is based on that contextCSN. This is always the first entry that slapcat will output. As such, the contextCSN in this LDIF represents the state of that database when the slapcat began. It is possible for updates to occur while slapcat is running that will alter its output, of course. But when the consumer connects to the provider, it will sync up on everything based on the contextCSN.

Dear Quanah,

Thank you again for your very helpful replies.

On 22/06/10 07:57 -0700, Quanah Gibson-Mount wrote:

--On Tuesday, June 22, 2010 12:24 PM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote:

...

If so, does that mean that with syncrepl, the above operation is reduced to the following three steps?

1. Stop slapd on both consumers.
2. Netcat the database from one to the other.
3. start both consumers.

The officially supported method is slapcat/slapadd. But as long as your architecture is the same, yes, the above steps are fine, although I use scp rather than netcat to copy the db.

When the database is many gigabytes in size, netcat is fast; I get impatient :-)

You failed to note running db_recover before copying the databases, which I would recommend (to force writing out any pending data in the BDB env).

Thank you very much for this! I was unaware of the need for this. I will investigate it further.

--On Tuesday, June 22, 2010 8:05 AM -0700 Chris Jacobs Chris.Jacobs@apollogrp.edu wrote:

Nick,

I've done this a number of times - Dev on an 'in production' ldap infrastructure. I've managed to get the boxes out of sync a number of times.

1. Stop slapd.
2. Delete the contents of the db dir
3. Copy a db-config into it.
4. Startup slapd.

The full tree is replicated pretty quickly. Try it out, you'll see. It take less then half a minute to replicate a tree with approx 800 entries (very rough estimate) even on slaves in AZ and masters in WA.

I'm sure it does work fine for such a tiny database. Get a large database (20 million, 200 million etc), it's not worth the time to do it that way.

--Quanah

--

Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

Dear Chris,

Thank you for taking the time to offer your experience.

On 22/06/10 08:05 -0700, Chris Jacobs wrote:

Nick,

I've done this a number of times - Dev on an 'in production' ldap infrastructure. I've managed to get the boxes out of sync a number of times.

1. Stop slapd.
2. Delete the contents of the db dir
3. Copy a db-config into it.
4. Startup slapd.

The full tree is replicated pretty quickly. Try it out, you'll see. It take less then half a minute to replicate a tree with approx 800 entries (very rough estimate) even on slaves in AZ and masters in WA.

Yes, I tried this out in our dev environment; took a couple of hours.

Most of our directories have in the order of a million entries, some many more: # zgrep '^dn:' 201006230300.gz | wc -l 3377876

--On Tuesday, June 22, 2010 5:11 PM +0200 Jonathan Clarke jonathan@phillipoux.net wrote:

I would instead recommend doing a slapcat on one consumer, copying over the file to second consumer, slapadd, then starting that consumer.

This depends a lot on the size of the DB. I work with some in excess of 60GB. The DB copy would be a lot quicker, period. It's all a matter of doing what is best for a given setup.

--Quanah

--

Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration