Hello, I am setting up n-way multi-provider replication.
Should we expect a full sync when we setup sync for the first time on a new provider? Can we load a backup data.mdb file first and have deltasync run from there to prevent a full sync (i.e. to save time)? What is the expected behavior/best practice?
Also, I have some questions around mdb_copy. Is there any concern using the -c flag for compacting the data.mdb file to be used for restore (or new n-way provider if that works)? I notice the compacted file is smaller on disk than the original data.mdb (due to the removed freed pages). I assume the data sets are still the same? Are there tools to compare the two?
And last question, I am using mdb_copy to create a backup but standard linux cp to put the new file into /var/lib/ldap/data.mdb. Will this cause any issues or is it recommended to use mdb_copy to also move the data.mdb file into place?
Thanks! Paul
--On Wednesday, April 7, 2021 8:02 PM +0000 paul.jc@yahoo.com wrote:
Hello, I am setting up n-way multi-provider replication.
Should we expect a full sync when we setup sync for the first time on a new provider? Can we load a backup data.mdb file first and have deltasync run from there to prevent a full sync (i.e. to save time)? What is the expected behavior/best practice?
Given the known issues in standard syncrepl it is ill advised to do a full sync. Additionaly it's a waste of time. ;)
The best practice is to either pre-load the DB via slapadd or provide it a binary copy of the db from mdb_copy -c.
Also, I have some questions around mdb_copy. Is there any concern using the -c flag for compacting the data.mdb file to be used for restore (or new n-way provider if that works)? I notice the compacted file is smaller on disk than the original data.mdb (due to the removed freed pages). I assume the data sets are still the same? Are there tools to compare the two?
You should use mdb_copy -c. All it does is de-fragment the database as much as possible.
And last question, I am using mdb_copy to create a backup but standard linux cp to put the new file into /var/lib/ldap/data.mdb. Will this cause any issues or is it recommended to use mdb_copy to also move the data.mdb file into place?
You can just use cp after you've made the initial copy.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Quanah Gibson-Mount wrote:
--On Wednesday, April 7, 2021 8:02 PM +0000 paul.jc(a)yahoo.com wrote:
Hello, I am setting up n-way multi-provider replication.
Should we expect a full sync when we setup sync for the first time on a new provider? Can we load a backup data.mdb file first and have deltasync run from there to prevent a full sync (i.e. to save time)? What is the expected behavior/best practice?
Given the known issues in standard syncrepl it is ill advised to do a full sync. Additionaly it's a waste of time. ;)
The best practice is to either pre-load the DB via slapadd or provide it a binary copy of the db from mdb_copy -c.
Also, I have some questions around mdb_copy. Is there any concern using the -c flag for compacting the data.mdb file to be used for restore (or new n-way provider if that works)? I notice the compacted file is smaller on disk than the original data.mdb (due to the removed freed pages). I assume the data sets are still the same? Are there tools to compare the two?
You should use mdb_copy -c. All it does is de-fragment the database as much as possible.
And last question, I am using mdb_copy to create a backup but standard linux cp to put the new file into /var/lib/ldap/data.mdb. Will this cause any issues or is it recommended to use mdb_copy to also move the data.mdb file into place?
You can just use cp after you've made the initial copy.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Quanah Gibson-Mount wrote:
--On Wednesday, April 7, 2021 8:02 PM +0000 paul.jc(a)yahoo.com wrote:
Hello, I am setting up n-way multi-provider replication.
Should we expect a full sync when we setup sync for the first time on a new provider? Can we load a backup data.mdb file first and have deltasync run from there to prevent a full sync (i.e. to save time)? What is the expected behavior/best practice?
Given the known issues in standard syncrepl it is ill advised to do a full sync. Additionaly it's a waste of time. ;)
The best practice is to either pre-load the DB via slapadd or provide it a binary copy of the db from mdb_copy -c.
Also, I have some questions around mdb_copy. Is there any concern using the -c flag for compacting the data.mdb file to be used for restore (or new n-way provider if that works)? I notice the compacted file is smaller on disk than the original data.mdb (due to the removed freed pages). I assume the data sets are still the same? Are there tools to compare the two?
You should use mdb_copy -c. All it does is de-fragment the database as much as possible.
And last question, I am using mdb_copy to create a backup but standard linux cp to put the new file into /var/lib/ldap/data.mdb. Will this cause any issues or is it recommended to use mdb_copy to also move the data.mdb file into place?
You can just use cp after you've made the initial copy.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Quanah Gibson-Mount wrote:
--On Wednesday, April 7, 2021 8:02 PM +0000 paul.jc(a)yahoo.com wrote:
Hello, I am setting up n-way multi-provider replication.
Should we expect a full sync when we setup sync for the first time on a new provider? Can we load a backup data.mdb file first and have deltasync run from there to prevent a full sync (i.e. to save time)? What is the expected behavior/best practice?
Given the known issues in standard syncrepl it is ill advised to do a full sync. Additionaly it's a waste of time. ;)
The best practice is to either pre-load the DB via slapadd or provide it a binary copy of the db from mdb_copy -c.
Also, I have some questions around mdb_copy. Is there any concern using the -c flag for compacting the data.mdb file to be used for restore (or new n-way provider if that works)? I notice the compacted file is smaller on disk than the original data.mdb (due to the removed freed pages). I assume the data sets are still the same? Are there tools to compare the two?
You should use mdb_copy -c. All it does is de-fragment the database as much as possible.
And last question, I am using mdb_copy to create a backup but standard linux cp to put the new file into /var/lib/ldap/data.mdb. Will this cause any issues or is it recommended to use mdb_copy to also move the data.mdb file into place?
You can just use cp after you've made the initial copy.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Quanah Gibson-Mount wrote:
--On Wednesday, April 7, 2021 8:02 PM +0000 paul.jc(a)yahoo.com wrote:
Hello, I am setting up n-way multi-provider replication.
Should we expect a full sync when we setup sync for the first time on a new provider? Can we load a backup data.mdb file first and have deltasync run from there to prevent a full sync (i.e. to save time)? What is the expected behavior/best practice?
Given the known issues in standard syncrepl it is ill advised to do a full sync. Additionaly it's a waste of time. ;)
The best practice is to either pre-load the DB via slapadd or provide it a binary copy of the db from mdb_copy -c.
Also, I have some questions around mdb_copy. Is there any concern using the -c flag for compacting the data.mdb file to be used for restore (or new n-way provider if that works)? I notice the compacted file is smaller on disk than the original data.mdb (due to the removed freed pages). I assume the data sets are still the same? Are there tools to compare the two?
You should use mdb_copy -c. All it does is de-fragment the database as much as possible.
And last question, I am using mdb_copy to create a backup but standard linux cp to put the new file into /var/lib/ldap/data.mdb. Will this cause any issues or is it recommended to use mdb_copy to also move the data.mdb file into place?
You can just use cp after you've made the initial copy.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Quanah Gibson-Mount wrote:
--On Wednesday, April 7, 2021 8:02 PM +0000 paul.jc(a)yahoo.com wrote:
Hello, I am setting up n-way multi-provider replication.
Should we expect a full sync when we setup sync for the first time on a new provider? Can we load a backup data.mdb file first and have deltasync run from there to prevent a full sync (i.e. to save time)? What is the expected behavior/best practice?
Given the known issues in standard syncrepl it is ill advised to do a full sync. Additionaly it's a waste of time. ;)
The best practice is to either pre-load the DB via slapadd or provide it a binary copy of the db from mdb_copy -c.
Also, I have some questions around mdb_copy. Is there any concern using the -c flag for compacting the data.mdb file to be used for restore (or new n-way provider if that works)? I notice the compacted file is smaller on disk than the original data.mdb (due to the removed freed pages). I assume the data sets are still the same? Are there tools to compare the two?
You should use mdb_copy -c. All it does is de-fragment the database as much as possible.
And last question, I am using mdb_copy to create a backup but standard linux cp to put the new file into /var/lib/ldap/data.mdb. Will this cause any issues or is it recommended to use mdb_copy to also move the data.mdb file into place?
You can just use cp after you've made the initial copy.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Quanah Gibson-Mount wrote:
--On Wednesday, April 7, 2021 8:02 PM +0000 paul.jc(a)yahoo.com wrote:
Hello, I am setting up n-way multi-provider replication.
Should we expect a full sync when we setup sync for the first time on a new provider? Can we load a backup data.mdb file first and have deltasync run from there to prevent a full sync (i.e. to save time)? What is the expected behavior/best practice?
Given the known issues in standard syncrepl it is ill advised to do a full sync. Additionaly it's a waste of time. ;)
The best practice is to either pre-load the DB via slapadd or provide it a binary copy of the db from mdb_copy -c.
Also, I have some questions around mdb_copy. Is there any concern using the -c flag for compacting the data.mdb file to be used for restore (or new n-way provider if that works)? I notice the compacted file is smaller on disk than the original data.mdb (due to the removed freed pages). I assume the data sets are still the same? Are there tools to compare the two?
You should use mdb_copy -c. All it does is de-fragment the database as much as possible.
And last question, I am using mdb_copy to create a backup but standard linux cp to put the new file into /var/lib/ldap/data.mdb. Will this cause any issues or is it recommended to use mdb_copy to also move the data.mdb file into place?
You can just use cp after you've made the initial copy.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Quanah Gibson-Mount wrote:
--On Wednesday, April 7, 2021 8:02 PM +0000 paul.jc(a)yahoo.com wrote:
Hello, I am setting up n-way multi-provider replication.
Should we expect a full sync when we setup sync for the first time on a new provider? Can we load a backup data.mdb file first and have deltasync run from there to prevent a full sync (i.e. to save time)? What is the expected behavior/best practice?
Given the known issues in standard syncrepl it is ill advised to do a full sync. Additionaly it's a waste of time. ;)
The best practice is to either pre-load the DB via slapadd or provide it a binary copy of the db from mdb_copy -c.
Also, I have some questions around mdb_copy. Is there any concern using the -c flag for compacting the data.mdb file to be used for restore (or new n-way provider if that works)? I notice the compacted file is smaller on disk than the original data.mdb (due to the removed freed pages). I assume the data sets are still the same? Are there tools to compare the two?
You should use mdb_copy -c. All it does is de-fragment the database as much as possible.
And last question, I am using mdb_copy to create a backup but standard linux cp to put the new file into /var/lib/ldap/data.mdb. Will this cause any issues or is it recommended to use mdb_copy to also move the data.mdb file into place?
You can just use cp after you've made the initial copy.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Hi Quanah & Paul
I am finding myself with similar questions. I have a single provider (provider1) setup with consumers replicating via delta sync. I want to get to an n-way setup as described in the open ldap documentation where I end up with provider1, provider2, and provider3.
1. Do providers need to use delta-sync between themselves in n-way if the consumers are? Do we need to use delta sync for both the config replication and the data replication? So I need two access logs now for both config db and data db? 2. Is copying the mdb binary the best way to provision provider2 and provider3? Why does slapadd support a -S and a -w flag which presumably sets things up for replication properly (serverIDs, CSN calculation), yet we can just copy the mdb binary to the new provider2 and provider3? Why does no serverID or contextCSN need to be considered when copying mdb_copy mdb binaries between providers? 3. I've taken a stab at provisioning new providers for n-way without delta-sync between providers. In some instances I see the new providers undergo a full data sync, in other instances they sometimes seem to realize they are already caught up. Would using delta sync solve this phenomenon? Further, my consumers start panicking and losing 'delta sync' and say they go back to a full sync, however things never 'catch up'. Simply restarting slapd seemed to resolve that. Do you guys find that you often need to restart slapd? 4. Section 18.2.1 in 2.4 Admin Guide explains "Note: since the database state is stored in both the changelog DB and the main DB on the provider, it is important to backup/restore both the changelog DB and the main DB using slapcat/slapadd when restoring a DB or copying it to another machine." Section 18.3.2.2 in 2.4 Admin Guide explains "Note: An accesslog database is unique to a given provider. It should never be replicated."
If I just add an mdb binary to my new providers, should I include the access log from the binary it was copied from? I am uncertain when I need to include an access log or not. Seems like the documents say you need it and other resources online think you can start with an empty access log.
Appreciate your consideration. Thomas
--On Thursday, April 8, 2021 8:15 PM +0000 thomaswilliampritchard@gmail.com wrote: Hi Thomas,
You really need to fix whatever email client you're using so that it doesn't do 8+ copies of what you're replying to. I nearly discarded this as spam.
I am finding myself with similar questions. I have a single provider (provider1) setup with consumers replicating via delta sync. I want to get to an n-way setup as described in the open ldap documentation where I end up with provider1, provider2, and provider3.
- Do providers need to use delta-sync between themselves in n-way if the
consumers are? Do we need to use delta sync for both the config replication and the data replication? So I need two access logs now for both config db and data db?
a) You should always use delta-sync and not standard syncrepl for the primary database replication.
b) I would strongly advise against using cn=config replication in 2.4, there are multiple known bugs with it.
c) I would not use delta-syncrepl for cn=config replication in 2.4, there are open bugs with it.
- Is copying the mdb binary the best way to
provision provider2 and provider3? Why does slapadd support a -S and a -w flag which presumably sets things up for replication properly (serverIDs, CSN calculation), yet we can just copy the mdb binary to the new provider2 and provider3?
You're misunderstanding the purpose of -S and -w.
The -S flag is generally for raw LDIF first time imports that have never been a part of directory server, and so lack the entryCSN attribute. The -S flag does nothing if an entryCSN value already exists on each entry. It's essentially a way to seed a raw LDIF database for use with replication.
The -w flag specifically updates the contextCSN value(s) on import of an LDIF file. If one was importing a raw LDIF database that had never been used before, it would make sense to use -w -S # for that import. It can be an extremely bad idea to use -w in most cases.
Generally, one should never be using -S or -w with backups generated via slapcat.
Why does no serverID or contextCSN need to be considered when copying mdb_copy mdb binaries between providers?
The primary database is not server specific. Can you imagine what a nightmare it would be if they were?
- I've
taken a stab at provisioning new providers for n-way without delta-sync between providers.
This is a exceptionally bad idea. It's generally unsafe to use standard syncrepl with a primary database in OpenLDAP 2.4 due to fundamental flaws in how it behaves that aren't addressed until OpenLDAP 2.5. Even then, I'd still strongly advise against using standard syncrepl. It has no real benefit and numerous drawbacks.
In some instances I see the new providers undergo a full data sync, in other instances they sometimes seem to realize they are already caught up.
This would indicate to me that you're following a flawed process.
Would using delta sync solve this phenomenon?
No idea, given the above.
Further, my consumers start panicking and losing 'delta sync' and say they go back to a full sync, however things never 'catch up'. Simply restarting slapd seemed to resolve that. Do you guys find that you often need to restart slapd?
No. You've failed to provide any real useful information as well, i.e.:
a) Whatever your process is b) What version of OpenLDAP you have deployed
If I just add an mdb binary to my new providers, should I include the access log from the binary it was copied from? I am uncertain when I need to include an access log or not. Seems like the documents say you need it and other resources online think you can start with an empty access log.
You can use mdb_copy for the primary DB to a new provider. As clearly noted in the documentation provided, an accesslog DB is provider specific, so you would have a new provider initialize its own accesslog DB. Additionally, an accesslog DB is only safe to restore with an exact point in time copy of the primary DB.
Generally the accesslog DB is only useful as a backup if:
a) slapd is stopped b) primary db is slapcat'd c) accesslog db is slapcat'd d) slapd is started
That gives you a very specific point in time restoration point. accesslog DB exports can also be useful when debugging replication related issues.
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
openldap-technical@openldap.org
-
paul.jc＠yahoo.com -
Quanah Gibson-Mount -
thomaswilliampritchard＠gmail.com