Hello,
Under typical circumstances we run a config database and have a single application database for ldap data. We run consumers replicating from providers where they replicate the entire application database. We detect delayed replication by querying consumer CSNs and comparing them to the provider CSNs to determine if any consumers have delayed replication through a health check script that executes every few minutes.
For one particular use case we replicate a subset of the application database, but our replication check cannot work for this use case. It appears because we're not replicating the entire database we do not copy the provider CSNs over to the consumers (which live at the base level, and we're copying a DIT below that level) and therefore I cannot ask the consumers what their 'latest csn' is since they do not seem to have one.
Is there any recommended way I can check that the consumer replication is functioning properly when replicating a subset of the provider database?
Thanks for the consideration. Tom
On 12/12/22 16:47, thomaswilliampritchard@gmail.com wrote:
For one particular use case we replicate a subset of the application database, but our replication check cannot work for this use case.
Partial replication is somewhat tricky because the highest entryCSN value seen in a replicated entry might not the same like the highest entryCSN value in the full data set (the latter stored as contextCSN at the provider).
So from my understanding just comparing contextCSN values will not work.
Since release 2.5 there's more replication info in back-monitor. Didn't have the time to adapt my own monitoring checks to this though.
Ciao, Michael.
On Mon, Dec 12, 2022 at 04:58:44PM +0100, Michael Ströder wrote:
On 12/12/22 16:47, thomaswilliampritchard@gmail.com wrote:
For one particular use case we replicate a subset of the application database, but our replication check cannot work for this use case.
Partial replication is somewhat tricky because the highest entryCSN value seen in a replicated entry might not the same like the highest entryCSN value in the full data set (the latter stored as contextCSN at the provider).
So from my understanding just comparing contextCSN values will not work.
Comparing entryCSN values will not work because of what you note, but if using OpenLDAP's own syncrepl (olcSyncrepl), contextCSNs should work just fine as they record whatever has been passed in the cookies. It might be necessary to retrieve if from the right place (manageDSAit on the suffix entry or reading the subentry if it was necessary to store contextCSN there).
As usual I will note that you don't want to use contextCSNs for anything apart from being able to tell whether the contextCSN sets are up to date or not. Especially avoid guessing "replication delay" from its differences, doing so will yield meaningless values more often than you think. If you want to measure replication delay, use a time-series DB or syncmonitor[0].
Since release 2.5 there's more replication info in back-monitor. Didn't have the time to adapt my own monitoring checks to this though.
[0]. https://git.openldap.org/openldap/syncmonitor
Regards,
thomaswilliampritchard@gmail.com wrote:
Hello,
Under typical circumstances we run a config database and have a single application database for ldap data. We run consumers replicating from providers where they replicate the entire application database. We detect delayed replication by querying consumer CSNs and comparing them to the provider CSNs to determine if any consumers have delayed replication through a health check script that executes every few minutes.
For one particular use case we replicate a subset of the application database, but our replication check cannot work for this use case. It appears because we're not replicating the entire database we do not copy the provider CSNs over to the consumers (which live at the base level, and we're copying a DIT below that level) and therefore I cannot ask the consumers what their 'latest csn' is since they do not seem to have one.
Is there any recommended way I can check that the consumer replication is functioning properly when replicating a subset of the provider database?
The consumer still tracks the provider's contextCSN and still stores it in the DIT's context entry. If you slapcat the consumer DB you'll see it.
Thanks for the consideration. Tom
thomaswilliampritchard@gmail.com schrieb am 12.12.2022 um 16:47 in Nachricht
20221212154750.5262.89868@hypatia.openldap.org:
Hello,
Under typical circumstances we run a config database and have a single application database for ldap data. We run consumers replicating from providers where they replicate the entire application database. We detect delayed replication by querying consumer CSNs and comparing them to the provider CSNs to determine if any consumers have delayed replication through a health check script that executes every few minutes.
For one particular use case we replicate a subset of the application database, but our replication check cannot work for this use case. It appears because we're not replicating the entire database we do not copy the provider CSNs over to the consumers (which live at the base level, and we're copying a DIT below that level) and therefore I cannot ask the consumers what their 'latest csn' is since they do not seem to have one.
Is there any recommended way I can check that the consumer replication is functioning properly when replicating a subset of the provider database?
My guess is to compute the maximum of all replicated EntryCSNs on the provider side and compare them with the maximum of the same entries on the consumer side. Or have a test entry you "touch" and compare the EntryCSN of that test entry.
Thanks for the consideration. Tom
openldap-technical@openldap.org
-
Howard Chu -
Michael Ströder -
Ondřej Kuzník -
thomaswilliampritchard＠gmail.com -
Ulrich Windl