I have implemented a multi-master two node ldap with openldap 2.4.22 and Berkely DB 4.8.26 on Redhat enterprise 5.4 with several readonly replicas off of the masters.
I have a need to add several optional attributes to a schema and probably should upgrade to 2.4.24 as well. If this was a single-master server, it would be easy to do; just slapcat the ldap store, update software, change schema, slapadd the ldap store back, and resume slapd.
I'm not sure how to do that with multi-master. One reason for using multi-master was if one master was down, the other would keep running. One should be able to upgrade one server, have it catch up with the changes that the other master had done while the first master is down and then repeat for the 2nd master. Is this possible? Has anyone done this and how was it done?
I know in the near future, a high-level branch on my DIT will be purged and bulk reloaded. I have tested the load with a test setup of multi-master ldap. If I do it via ldapadd, it takes over 6 hours to load. With slapadd (and slapd down) it only takes 25 minutes plus the time for the other master to get up-to-date. Is there any way that I can speed-up the update with ldapadd? I have pieces of my slapd.conf for the 1st master at the end of this email.
Slapadd has two options that appear to be needed when dealing with multi-master or replicate nodes. The first is the "-S sid" option, the second is "-w". I'm a little confused what is used where. If you are doing a dump and restore operation (slapcat, delete database, slapadd) the only option you need is the "-w" option? If you are adding new entries offline then do both options need to be specified?
Is there a multi-master best practice quide somewhere?
Thanks for any help,
============== slapd.conf extract ==============: ServerID 001 pidfile /usr/local/var/run/slapd.pid argsfile /usr/local/var/run/slapd.args sizelimit unlimited moduleload ppolicy.1a
database bdb suffix "dc=htc,dc=com" rootdn "cn=Manager,dc=htc,dc=com" rootpw XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx
directory /usr/local/var/openldap-data cachesize 50000 dncachesize 50000 idlcachesize 150000 checkpoint 1024 5 dbconfig set_cachesize 0 268435456 1 dbconfig set_lg_bsize 2097152 dbconfig set_lg_regionmax 262144 dbconfig set_flags DB_LOG_AUTOREMOVE monitoring on
index objectClass eq,pres index ou,cn,mail,surname,givenname eq,pres,sub index uidNumber,gidNumber,loginShell eq,pres index uid,memberUid eq,pres,sub index nisMapName,nisMapEntry eq,pres,sub index entryCSN eq index entryUUID eq index uniqueIdentifier eq
overlay ppolicy ppolicy_default "cn=default,ou=policies,dc=htc,dc=com"
syncrepl rid=002 provider=ldap://vmldapdev2.htc.external:389 type=refreshAndPersist retry="5 5 300 +" searchbase="dc=htc,dc=com" attrs="*,+" bindmethod=simple binddn="uid=vmldapdev1,ou=replicants,ou=admin,dc=htc,dc=com" credentials=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
mirrormode TRUE
overlay syncprov syncprov-checkpoint 1000 1
database monitor
********************************************************************** HTC Disclaimer: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. **********************************************************************
----- "Mike Cannady" mike.cannady@htcinc.net wrote:
I have implemented a multi-master two node ldap with openldap 2.4.22 and Berkely DB 4.8.26 on Redhat enterprise 5.4 with several readonly replicas off of the masters.
I have a need to add several optional attributes to a schema and probably should upgrade to 2.4.24 as well. If this was a single-master server, it would be easy to do; just slapcat the ldap store, update software, change schema, slapadd the ldap store back, and resume slapd.
Why would you need to slapcat/slapadd to "add several optional attributes"
I'm not sure how to do that with multi-master. One reason for using multi-master was if one master was down, the other would keep running. One should be able to upgrade one server, have it catch up with the changes that the other master had done while the first master is down and then repeat for the 2nd master.
Well, it would apply if you weren't modifying data offline on the 1st master.
Is this possible? Has anyone done this and how was it done?
I know in the near future, a high-level branch on my DIT will be purged and bulk reloaded.
I can't think of a strategy where a bulk load will have neither: -write downtime -inconsistency (changes made in the window between the bulk generation and the startup of the server after import will be lost)
You aren't clear which of these you want/prefer/require.
I have tested the load with a test setup of multi-master ldap. If I do it via ldapadd, it takes over 6 hours to load. With slapadd (and slapd down) it only takes 25 minutes plus the time for the other master to get up-to-date.
What is tool-threads set to? Which interesting slapadd options (e.g. -q) did you use?
Is there any way that I can speed-up the update with ldapadd?
ldapadd will never be as fast as slapadd.
I have pieces of my slapd.conf for the 1st master at the end of this email.
Slapadd has two options that appear to be needed when dealing with multi-master or replicate nodes. The first is the "-S sid" option, the second is "-w". I'm a little confused what is used where. If you are doing a dump and restore operation (slapcat, delete database, slapadd) the only option you need is the "-w" option? If you are adding new entries offline then do both options need to be specified?
Adding, *or* modifying.
Is there a multi-master best practice quide somewhere?
A good start is to never lie to slapd. If you have changed the contents of an entry, the entryCSN should not be retained.
I also prefer to avoid non-restore bulk-loading.
Regards, Buchan
-----Original Message----- From: Buchan Milne [mailto:bgmilne@staff.telkomsa.net] Sent: Friday, March 25, 2011 6:17 AM To: Cannady, Mike Cc: openldap-technical@openldap.org Subject: Re: Multimaster ldap related questions
----- "Mike Cannady" mike.cannady@htcinc.net wrote:
I have implemented a multi-master two node ldap with openldap 2.4.22 and Berkely DB 4.8.26 on Redhat enterprise 5.4 with several readonly replicas off of the masters.
I have a need to add several optional attributes to a schema and probably should upgrade to 2.4.24 as well. If this was a single-master server, it would be easy to do; just slapcat the ldap store, update software, change schema, slapadd the ldap store back, and resume slapd.
Why would you need to slapcat/slapadd to "add several optional
attributes"
While testing the 2 node multi-master nodes, I did identical changes (added an optional attribute to a schema) by stopping the 2 slapd daemonds, changed schema, started the daemonds. After a lot of adding and deleting of info out of the stores, it was apparent that something was wrong with the data and the syncs. I stopped both daemons, did the dump and restore on one and purge the database on the other and started them back up. I didn't have any problems after that. I know I didn't have issues prior to the changes, so I assumed the S.O.P. for schema changes is dump, change, restore.
Are you indicating that isn't the case? What about newly defined required attributes in a schema or one that was optional and now required where all the instances have the optional already specified.
I'm not sure how to do that with multi-master. One reason for using multi-master was if one master was down, the other would keep running. One should be able to upgrade one server, have it catch up with the changes that the other master had done while the first master is
down
and then repeat for the 2nd master.
Well, it would apply if you weren't modifying data offline on the 1st master.
Is this possible? Has anyone done this and how was it done?
I know in the near future, a high-level branch on my DIT will be purged and bulk reloaded.
I can't think of a strategy where a bulk load will have neither: -write downtime -inconsistency (changes made in the window between the bulk generation
and
the startup of the server after import will be lost)
You aren't clear which of these you want/prefer/require.
I want to minimize the time that the data in the store is unavailable. If the delete and bulk-load take 10 hours to do online, then the service is effectively down for 10 hours. If the offline version of dump, delete what is to be deleted, slapadd the results, and then slapadd the new info takes only 30-60 minutes. The offline would be the method I choose based on time. The amount of time was my next question:
I have tested the load with a test setup of multi-master ldap. If I do it via ldapadd, it takes over 6 hours to load. With slapadd (and slapd down) it only takes 25 minutes plus the time for the other master to get up-to-date.
What is tool-threads set to? Which interesting slapadd options (e.g.
-q)
did you use?
Tool-threads is not specified, so I guess its one. The test box is a single hyperthreaded cpu.
Slapadd command (for master # 1) slapadd -c -f ldapfile -n 1 -S 001 -w
Is there any way that I can speed-up the update with ldapadd?
ldapadd will never be as fast as slapadd.
Granted. What I'm asking is there a way to speed up slapd for the duration of the mass deletes and bulk loads. In normal circumstances, the syncs and such would be OK for normal processing; but, with bulk changes, I would like have the daemon run in such a way that syncs are not done and everything is done in memory. My tests seem to indicate that disk I/O is the bottleneck. I know if there is a failure during this time, the database may be corrupted, but this OK for the bulk change duration.
I have pieces of my slapd.conf for the 1st master at the end of this email.
Slapadd has two options that appear to be needed when dealing with multi-master or replicate nodes. The first is the "-S sid" option, the second is "-w". I'm a little confused what is used where. If you are doing a dump and restore operation (slapcat, delete database, slapadd) the only option you need is the "-w" option? If you are adding new entries offline then do both options need to be specified?
Adding, *or* modifying.
I don't understand this answer.
Is there a multi-master best practice quide somewhere?
A good start is to never lie to slapd. If you have changed the
contents of
an entry, the entryCSN should not be retained.
I also prefer to avoid non-restore bulk-loading.
Me too, but if it is a decision of 10 hours vs. 1 hour, the 1 hour will win.
Regards, Buchan
Thanks,
Mike
********************************************************************** HTC Disclaimer: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. **********************************************************************
2011/3/25 Cannady, Mike mike.cannady@htcinc.net
-----Original Message----- From: Buchan Milne [mailto:bgmilne@staff.telkomsa.net] Sent: Friday, March 25, 2011 6:17 AM To: Cannady, Mike Cc: openldap-technical@openldap.org Subject: Re: Multimaster ldap related questions
----- "Mike Cannady" mike.cannady@htcinc.net wrote:
I have implemented a multi-master two node ldap with openldap 2.4.22 and Berkely DB 4.8.26 on Redhat enterprise 5.4 with several readonly replicas off of the masters.
I have a need to add several optional attributes to a schema and probably should upgrade to 2.4.24 as well. If this was a single-master server, it would be easy to do; just slapcat the ldap store, update software, change schema, slapadd the ldap store back, and resume slapd.
Why would you need to slapcat/slapadd to "add several optional
attributes"
While testing the 2 node multi-master nodes, I did identical changes (added an optional attribute to a schema) by stopping the 2 slapd daemonds, changed schema, started the daemonds. After a lot of adding and deleting of info out of the stores, it was apparent that something was wrong with the data and the syncs. I stopped both daemons, did the dump and restore on one and purge the database on the other and started them back up. I didn't have any problems after that. I know I didn't have issues prior to the changes, so I assumed the S.O.P. for schema changes is dump, change, restore.
Are you indicating that isn't the case? What about newly defined required attributes in a schema or one that was optional and now required where all the instances have the optional already specified.
I'm not sure how to do that with multi-master. One reason for using multi-master was if one master was down, the other would keep running. One should be able to upgrade one server, have it catch up with the changes that the other master had done while the first master is
down
and then repeat for the 2nd master.
Well, it would apply if you weren't modifying data offline on the 1st master.
Is this possible? Has anyone done this and how was it done?
I know in the near future, a high-level branch on my DIT will be purged and bulk reloaded.
I can't think of a strategy where a bulk load will have neither: -write downtime -inconsistency (changes made in the window between the bulk generation
and
the startup of the server after import will be lost)
You aren't clear which of these you want/prefer/require.
I want to minimize the time that the data in the store is unavailable. If the delete and bulk-load take 10 hours to do online, then the service is effectively down for 10 hours. If the offline version of dump, delete what is to be deleted, slapadd the results, and then slapadd the new info takes only 30-60 minutes. The offline would be the method I choose based on time. The amount of time was my next question:
I have tested the load with a test setup of multi-master ldap. If I do it via ldapadd, it takes over 6 hours to load. With slapadd (and slapd down) it only takes 25 minutes plus the time for the other master to get up-to-date.
What is tool-threads set to? Which interesting slapadd options (e.g.
-q)
did you use?
Tool-threads is not specified, so I guess its one. The test box is a single hyperthreaded cpu.
Slapadd command (for master # 1) slapadd -c -f ldapfile -n 1 -S 001 -w
Is there any way that I can speed-up the update with ldapadd?
ldapadd will never be as fast as slapadd.
Granted. What I'm asking is there a way to speed up slapd for the duration of the mass deletes and bulk loads. In normal circumstances, the syncs and such would be OK for normal processing; but, with bulk changes, I would like have the daemon run in such a way that syncs are not done and everything is done in memory. My tests seem to indicate that disk I/O is the bottleneck. I know if there is a failure during this time, the database may be corrupted, but this OK for the bulk change duration.
Hello,
I have a machine that was time consuming to restore a base ldif with "slapadd.
My problem was the I / O (HD very slow).
Resolved with a small shell:
#!/bin/bash
/etc/init.d/slapd stop slapcat /tmp/backup.ldif
dir_ldap="/var/lib/ldap"
rm -f /var/lib/ldap/{alock,__db.*,*bdb,log.*}
mv $dir_ldap $dir_ldap.real mkdir $dir_ldap mount -t tmpfs tmpfs $dir_ldap -o nr_inodes=3000000,size=4G
cp $dir_ldap.real/DB_CONFIG $dir_ldap
slapadd -c -q -w -b "dc=example,dc=com" -l /tmp/backup.ldif rsync -a --delete $dir_ldap/ $dir_ldap.real/ umount $dir_ldap rmdir $dir_ldap mv $dir_ldap.real $dir_ldap chown openldap:openldap $dir_ldap -R
/etc/init.d/slapd start
Thus, the I / O during the slapadd no longer exists, because all files are in memory. The time difference with "tmpfs" is striking.
Regards, Jarbas
I have pieces of my slapd.conf for the 1st master at the end of this email.
Slapadd has two options that appear to be needed when dealing with multi-master or replicate nodes. The first is the "-S sid" option, the second is "-w". I'm a little confused what is used where. If you are doing a dump and restore operation (slapcat, delete database, slapadd) the only option you need is the "-w" option? If you are adding new entries offline then do both options need to be specified?
Adding, *or* modifying.
I don't understand this answer.
Is there a multi-master best practice quide somewhere?
A good start is to never lie to slapd. If you have changed the
contents of
an entry, the entryCSN should not be retained.
I also prefer to avoid non-restore bulk-loading.
Me too, but if it is a decision of 10 hours vs. 1 hour, the 1 hour will win.
Regards, Buchan
Thanks,
Mike
HTC Disclaimer: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you.
openldap-technical@openldap.org
-
Buchan Milne -
Cannady, Mike -
Jarbas Peixoto Júnior