-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Dear list users,
I have a master openldap 2.3 server which is replicated via syncrepl on half a dozen other servers (2.3 too). Due to a legacy application from which I have to import passwords once a day, the master server is stopped, erased and re-built from scratch once every day...
I have noticed that all the replicas have recovered only a partial subset of the entries and the strange thing is that all the replicas have the same subset. They are all missing a few hundred entries.
When I stop the replicas, erase their data and start them anew, they replicate just fine and are consistent with the master server.
I was wondering : could this be due to the fact that the replicas have problems erasing the old entries and replacing them with the new set of entries? Would increasing syncprov-checkpoint <ops> and syncprov-sessionlog <size> values improve the situation? I also recall reading something about a specific configuration directive to improve delete replication but I have a feeling it was 2.4 specific...
Thanks,
Cheers - -- Oliver Henriot B.Sc. Ph.D. | Technicien de Maintenance Moyens Informatiques et Multimédia | UMS MI2S | http://mi2s.imag.fr/ Domaine universitaire BP53 | 38041 Grenoble cedex 9 | France tel.: +33 4 76 51 43 48 | fax: +33 4 76 51 47 15
Oliver Henriot wrote:
I have a master openldap 2.3 server which is replicated via syncrepl on half a dozen other servers (2.3 too).
Please tell us the exact version.
Due to a legacy application from which I have to import passwords once a day, the master server is stopped, erased and re-built from scratch once every day...
Why don't you just reset the passwords?
Ciao, Michael.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Dear Michael and list users,
Dans sa grande sagesse, Michael Ströder a écrit, le 30.03.2009 15:43 :
Oliver Henriot wrote:
I have a master openldap 2.3 server which is replicated via syncrepl on half a dozen other servers (2.3 too).
Please tell us the exact version.
2.3.27-8 on Centos5.2
Due to a legacy application from which I have to import passwords once a day, the master server is stopped, erased and re-built from scratch once every day...
Why don't you just reset the passwords?
I'd love to, I'd really love to. The problem is : this legacy application is used by our clients so we can't remove it right now and the person who developped it never documented it and has left the team. So I'm stuck with it until all my clients migrate to the new application. I'm hoping this won't be too long but in the mean time I have to make do with it.
Best regards,
Ciao, Michael.
- -- Oliver Henriot B.Sc. Ph.D. | Technicien de Maintenance Moyens Informatiques et Multimédia | UMS MI2S | http://mi2s.imag.fr/ Domaine universitaire BP53 | 38041 Grenoble cedex 9 | France tel.: +33 4 76 51 43 48 | fax: +33 4 76 51 47 15
Oliver Henriot wrote:
Dear Michael and list users,
Dans sa grande sagesse, Michael Ströder a écrit, le 30.03.2009 15:43 :
Oliver Henriot wrote:
I have a master openldap 2.3 server which is replicated via syncrepl on half a dozen other servers (2.3 too).
Please tell us the exact version.
2.3.27-8 on Centos5.2
Note that the latest release in the 2.3 series was 2.3.43.
CHANGES in CVS branch OPENLDAP_REL_ENG_2_3 even mentions "2.3.44 Engineering" with a fix. If you go through CHANGES you will see many syncrepl-related fixes mentioned after 2.3.27. So you definitely should use a more recent version. (Yes, we all know the arguments for using the distro packages. This has been discussed before...)
Due to a legacy application from which I have to import passwords once a day, the master server is stopped, erased and re-built from scratch once every day...
Why don't you just reset the passwords?
I'd love to, I'd really love to. The problem is : this legacy application is used by our clients so we can't remove it right now and the person who developped it never documented it and has left the team. So I'm stuck with it until all my clients migrate to the new application. I'm hoping this won't be too long but in the mean time I have to make do with it.
Well, dropping the whole DB resetting all passwords is weird anyway. Why are you doing replication at all. I'd have a look at this sync implementation. It can't be too hard to understand it if you have the source or at least have access to the data model of the source database.
Ciao, Michael.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Dans sa grande sagesse, Michael Ströder a écrit, le 30.03.2009 16:55 :
Oliver Henriot wrote:
Dear Michael and list users,
Dans sa grande sagesse, Michael Ströder a écrit, le 30.03.2009 15:43 :
Oliver Henriot wrote:
I have a master openldap 2.3 server which is replicated via syncrepl on half a dozen other servers (2.3 too).
Please tell us the exact version.
2.3.27-8 on Centos5.2
Note that the latest release in the 2.3 series was 2.3.43.
CHANGES in CVS branch OPENLDAP_REL_ENG_2_3 even mentions "2.3.44 Engineering" with a fix. If you go through CHANGES you will see many syncrepl-related fixes mentioned after 2.3.27. So you definitely should use a more recent version. (Yes, we all know the arguments for using the distro packages. This has been discussed before...)
OK, I'll consider that path. Thanks for the advice.
Due to a legacy application from which I have to import passwords once a day, the master server is stopped, erased and re-built from scratch once every day...
Why don't you just reset the passwords?
I'd love to, I'd really love to. The problem is : this legacy application is used by our clients so we can't remove it right now and the person who developped it never documented it and has left the team. So I'm stuck with it until all my clients migrate to the new application. I'm hoping this won't be too long but in the mean time I have to make do with it.
Well, dropping the whole DB resetting all passwords is weird anyway. Why are you doing replication at all. I'd have a look at this sync implementation. It can't be too hard to understand it if you have the source or at least have access to the data model of the source database.
I agree it's very weird. The way many things function here are, to put it mildly, very weird...
Thanks for the info.
Best regards,
Ciao, Michael.
- -- Oliver Henriot B.Sc. Ph.D. | Technicien de Maintenance Moyens Informatiques et Multimédia | UMS MI2S | http://mi2s.imag.fr/ Domaine universitaire BP53 | 38041 Grenoble cedex 9 | France tel.: +33 4 76 51 43 48 | fax: +33 4 76 51 47 15
Oliver Henriot wrote:
Dans sa grande sagesse, Michael Ströder a écrit, le 30.03.2009 16:55 :
Oliver Henriot wrote:
Dear Michael and list users,
Dans sa grande sagesse, Michael Ströder a écrit, le 30.03.2009 15:43 :
Oliver Henriot wrote:
I have a master openldap 2.3 server which is replicated via syncrepl on half a dozen other servers (2.3 too).
Please tell us the exact version.
2.3.27-8 on Centos5.2
Note that the latest release in the 2.3 series was 2.3.43.
CHANGES in CVS branch OPENLDAP_REL_ENG_2_3 even mentions "2.3.44 Engineering" with a fix. If you go through CHANGES you will see many syncrepl-related fixes mentioned after 2.3.27. So you definitely should use a more recent version. (Yes, we all know the arguments for using the distro packages. This has been discussed before...)
OK, I'll consider that path. Thanks for the advice.
While you're at it you might want to consider upgrading to 2.4.x. The upcoming 2.4.16 will also contain many important fixes.
But first you should seriously consider how to re-engineer the password sync. IMO that can't be too hard.
Ciao, Michael.
On Monday 30 March 2009 16:55:54 Michael Ströder wrote:
Oliver Henriot wrote:
Dear Michael and list users,
Dans sa grande sagesse, Michael Ströder a écrit, le 30.03.2009 15:43 :
Oliver Henriot wrote:
I have a master openldap 2.3 server which is replicated via syncrepl on half a dozen other servers (2.3 too).
Please tell us the exact version.
2.3.27-8 on Centos5.2
Note that the latest release in the 2.3 series was 2.3.43.
RHEL 5.3 ships more adequate 2.3.43 packages than they have in the past (most overlays are now enabled, but shipped in a subpackage for some unknown reason).
Since Centos 5.3 was recently released, there is no longer an excuse for running a non-current 2.3 on Centos 5 or RHEL5.
CHANGES in CVS branch OPENLDAP_REL_ENG_2_3 even mentions "2.3.44 Engineering" with a fix.
But, whether another 2.3 release will be made is a different question ...
If you go through CHANGES you will see many syncrepl-related fixes mentioned after 2.3.27. So you definitely should use a more recent version. (Yes, we all know the arguments for using the distro packages. This has been discussed before...)
Regards, Buchan
Oliver Henriot wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Dear list users,
I have a master openldap 2.3 server which is replicated via syncrepl on half a dozen other servers (2.3 too). Due to a legacy application from which I have to import passwords once a day, the master server is stopped, erased and re-built from scratch once every day...
I have noticed that all the replicas have recovered only a partial subset of the entries and the strange thing is that all the replicas have the same subset. They are all missing a few hundred entries.
When I stop the replicas, erase their data and start them anew, they replicate just fine and are consistent with the master server.
I was wondering : could this be due to the fact that the replicas have problems erasing the old entries and replacing them with the new set of entries?
No.
Would increasing syncprov-checkpoint<ops> and syncprov-sessionlog<size> values improve the situation?
No.
I also recall reading something about a specific configuration directive to improve delete replication but I have a feeling it was 2.4 specific...
What you're doing is unsupported. Syncrepl only works if all of the changes to the database are visible to the consumers, so that the before and after state can be determined. When you completely delete the provider DB and re-import it while the provider is shutdown there's no way for the consumers to track this, and after the delete there's no longer any record of the old contents on the provider.
If you're going to completely delete your database on the provider and expect the consumers to notice this, you must start the provider with an empty database so the consumers see the empty state and empty themselves accordingly. Then re-populate the provider and the consumers will likewise. All in all this is not a good way to operate your database...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Dear Howard and list users,
Dans sa grande sagesse, Howard Chu a écrit, le 30.03.2009 22:46 :
Oliver Henriot wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Dear list users,
I have a master openldap 2.3 server which is replicated via syncrepl on half a dozen other servers (2.3 too). Due to a legacy application from which I have to import passwords once a day, the master server is stopped, erased and re-built from scratch once every day...
I have noticed that all the replicas have recovered only a partial subset of the entries and the strange thing is that all the replicas have the same subset. They are all missing a few hundred entries.
When I stop the replicas, erase their data and start them anew, they replicate just fine and are consistent with the master server.
I was wondering : could this be due to the fact that the replicas have problems erasing the old entries and replacing them with the new set of entries?
No.
OK
Would increasing syncprov-checkpoint<ops> and syncprov-sessionlog<size> values improve the situation?
No.
OK
I also recall reading something about a specific configuration directive to improve delete replication but I have a feeling it was 2.4 specific...
What you're doing is unsupported. Syncrepl only works if all of the changes to the database are visible to the consumers, so that the before and after state can be determined. When you completely delete the provider DB and re-import it while the provider is shutdown there's no way for the consumers to track this, and after the delete there's no longer any record of the old contents on the provider.
If you're going to completely delete your database on the provider and expect the consumers to notice this, you must start the provider with an empty database so the consumers see the empty state and empty themselves accordingly. Then re-populate the provider and the consumers will likewise. All in all this is not a good way to operate your database...
Yes, I agree it's not good. Once I'll have got rid of the old system, I won't have to do this any more.
I had envisaged doing an ldapdelete of all entries on the master before shutting it down, but if just starting it up empty before importing the data will suffice, that will be just fine until the old system is removed.
Thank you very much for your help.
Best regards, - -- Oliver Henriot B.Sc. Ph.D. | Technicien de Maintenance Moyens Informatiques et Multimédia | UMS MI2S | http://mi2s.imag.fr/ Domaine universitaire BP53 | 38041 Grenoble cedex 9 | France tel.: +33 4 76 51 43 48 | fax: +33 4 76 51 47 15
Oliver Henriot wrote:
I had envisaged doing an ldapdelete of all entries on the master before shutting it down, but if just starting it up empty before importing the data will suffice, that will be just fine until the old system is removed.
Still the replication attributes (contextCSN, entryCSN) aren't there anymore after deleting the DB on the master. And deleting all the entries via LDAP and then fully re-add them on the replicas is not a good approach anyway regarding performance. Well, it might work with just a couple of test entries...
Ciao, Michael.
On Monday 30 March 2009 12:48:35 Oliver Henriot wrote:
Dear list users,
I have a master openldap 2.3 server which is replicated via syncrepl on half a dozen other servers (2.3 too). Due to a legacy application from which I have to import passwords once a day, the master server is stopped, erased and re-built from scratch once every day...
If you are going to rebuild the master's database, you should rebuild the databases on the slaves as well.
For instance, you could restart slapd with the csn cookie reset after the import has been completed on the master, or do the slapadd on the slaves as well.
However, this kind of legacy solution belongs in a different century ... (one in which there was a concept of "after-hours").
Regards, Buchan
openldap-technical@openldap.org
-
Buchan Milne -
Howard Chu -
Michael Ströder -
Oliver Henriot