Hi Friends,
Is there a hard limit on the number of consumers that can replicate from a provider?
I have a 3 provider master multi master configuration where each consumer is aware of all 3 providers and replicate with refreshAndPersist. Over time we find that the consumers stop getting updates from providers. We enabled keepalive on our replication configs and see that keepalive takes effect. After the idle timer expires keepalive seems to think the connection is still active. Is it possible the providers simply stop pushing changes to some consumers, or fail to push to some at a high number of consumers? What would be the potential failures modes of providers who are expected to push to a high number of consumers?
Thanks for the considerations, Tom
--On Wednesday, July 31, 2024 10:33 PM +0000 thomaswilliampritchard@gmail.com wrote:
Hi Friends,
Is there a hard limit on the number of consumers that can replicate from a provider?
I have a 3 provider master multi master configuration where each consumer is aware of all 3 providers and replicate with refreshAndPersist. Over time we find that the consumers stop getting updates from providers. We enabled keepalive on our replication configs and see that keepalive takes effect. After the idle timer expires keepalive seems to think the connection is still active. Is it possible the providers simply stop pushing changes to some consumers, or fail to push to some at a high number of consumers? What would be the potential failures modes of providers who are expected to push to a high number of consumers?
OpenLDAP version? How many consumers do you have? Is there a load balancer or other network appliance in between the systems that affects network traffic?
--Quanah
We have 20 consumers distributed across 4 regions. The multi master providers are in region A, and we have 8 consumers in region A, 4 consumers in region B, 4 consumers in region C, and 4 consumers in region D.
The consumers located in the same region as providers, region A, cross availability zones but we do not have a proxy between consumers and providers, they dial directly to providers. The consumers located in regions B, C, and D have to go through a transit gateway but there is no proxy applications otherwise. Oddly enough we see the most issues with the consumers in region A, so I do not believe the transit gateways are at fault.
--On Friday, August 2, 2024 5:05 AM +0000 thomaswilliampritchard@gmail.com wrote:
We have 20 consumers distributed across 4 regions. The multi master providers are in region A, and we have 8 consumers in region A, 4 consumers in region B, 4 consumers in region C, and 4 consumers in region D.
The consumers located in the same region as providers, region A, cross availability zones but we do not have a proxy between consumers and providers, they dial directly to providers. The consumers located in regions B, C, and D have to go through a transit gateway but there is no proxy applications otherwise. Oddly enough we see the most issues with the consumers in region A, so I do not believe the transit gateways are at fault.
Again, what verison of OpenLDAP are you running?
Additionally, what is your replication configuration? syncrepl? delta-syncrepl?
Thanks!
--Quanah
OpenLDAP 2.4.56 (we are working to upgrade to OpenLDAP 2.5 LTS)
We are configured with delta-syncrepl.
--On Friday, August 2, 2024 9:15 PM +0000 thomaswilliampritchard@gmail.com wrote:
OpenLDAP 2.4.56 (we are working to upgrade to OpenLDAP 2.5 LTS)
Known issue with OpenLDAP 2.4 series.
We are configured with delta-syncrepl.
I would recommend upgrading to the latest OpenLDAP 2.6 release and switching to standard syncrepl. If you don't want to build OpenLDAP yourself, the Symas packages are what I use in production.
--Quanah
Thanks for the recommendations.
I was under the impression 2.5 was the LTS version, but I do see others are often recommended to 2.6. Why 2.6 over the long term support 2.5?
Also, why the standard syncrepl recommendation? Delta syncrepl style approach seems to be along the typical industry standard sync strategy for databases like mongodb as an example with their "oplog". From what I understand the accesslog in OpenLDAP is analogous to the oplog from mongodb. Is the OpenLDAP project discontinuing Delta Syncrepl support?
--On Sunday, August 4, 2024 7:15 PM +0000 thomaswilliampritchard@gmail.com wrote:
Thanks for the recommendations.
I was under the impression 2.5 was the LTS version, but I do see others are often recommended to 2.6. Why 2.6 over the long term support 2.5?
The additional fixes and features in 2.6. At least in my case, the following items:
* Additional fixes to replication * Ability to bypass syslog to log directly to file (Significant perf increase) * slapo-memberof no longer deprecated & supports replication
I've been using it in production for a few years now, no issue.
Also, why the standard syncrepl recommendation? Delta syncrepl style approach seems to be along the typical industry standard sync strategy for databases like mongodb as an example with their "oplog". From what I understand the accesslog in OpenLDAP is analogous to the oplog from mongodb. Is the OpenLDAP project discontinuing Delta Syncrepl support?
See Ondrej's answer.
--Quanah
--On Tuesday, August 6, 2024 1:57 AM +0000 thomaswilliampritchard@gmail.com wrote:
See Ondrej's answer.
--Quanah
Forgive me, I do not see a response from an Ondrej.
--Quanah
openldap-technical@openldap.org
-
Quanah Gibson-Mount -
thomaswilliampritchard＠gmail.com