...

We have an aggressive target to meet - updates out in 5 minutes. Updates are small, but alarm bells are now ringing in my head about OpenLDAP's ability to deliver this.

This doesn't address the rest of your question, but if you need quick updates, you should be using delta-syncrepl rather than syncrepl for its smaller change footprint. It's also been substantially more reliable than syncrepl in my experience.

--Quanah

Thanks Quanah, but I don't see how that addresses any of my questions. We don't need delta-syncrepl because we are not changing individual attributes. We typically add or delete entire objects.

Mark.

--------------------------------------------------------------------------------

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.

Hi Quanah,

Are you sure your issues with syncrepl aren't specific to Zimbra? When I ran the Zimbra at Homestead/Intuit we saw syncrepl issues, but I have not seen those issues in non-Zimbra LDAP clusters.

-----Original Message----- From: openldap-technical [mailto:openldap-technical-bounces@openldap.org] On Behalf Of Quanah Gibson-Mount Sent: Friday, November 20, 2015 11:31 AM To: Bannister, Mark; openldap-technical@openldap.org Subject: RE: Getting around the single-threaded syncrepl model?

--On Friday, November 20, 2015 4:23 PM +0000 "Bannister, Mark" Mark.Bannister@morganstanley.com wrote:

Thanks Quanah, but I don't see how that addresses any of my questions. We don't need delta-syncrepl because we are not changing individual attributes. We typically add or delete entire objects.

Then you are correct, you don't need it for the reduced bandwidth. But you may still want it given it's been reliable for replication, which has not been my experience with syncrepl.

--Quanah

--

Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

...

--On Friday, November 20, 2015 6:31 PM +0000 Albert Braden abraden@about.com wrote:

...

Hi Quanah,

Are you sure your issues with syncrepl aren't specific to Zimbra? When I ran the Zimbra at Homestead/Intuit we saw syncrepl issues, but I have not seen those issues in non-Zimbra LDAP clusters.

Quite certain. For one thing, I file very few syncrepl related ITSes, since I don't use it, but many are filed... For another, while I have encountered a very few issues with delta-syncrepl, the majority of issues I find w/ Zimbra and replication are related to syncrepl when it has to be used in an initial or fallback scenario.

What sort of issues?

And here's some quick stats: $ grep syncrepl CHANGES | grep -v delta | wc -l 76

$ grep delta-syncrepl CHANGES | wc -l 7

Or syncrepl has had 10x times the issues.

Or syncrepl has 10x the users, more eyes spot more bugs, and is now more stable because it has had more fixes? I'm just speculating here, but I wouldn't be more confident in product Y because it is mentioned less in the change log than product X.

However, this is a tangent, and still does not help me with my original question. Correct me if I'm wrong, but delta-syncrepl is also going to be running in a serial fashion on the master server, and so, if I'm replicating out to hundreds of replicas, high network latency or timeouts on a single replica would still have a big impact on my ability to replicate everywhere within 5 minutes.

Syncrepl/delta-syncrepl doesn't make a difference here. Unless someone has a better suggestion, as I don't suppose OpenLDAP is going to be fixed to do this in parallel anytime soon, I will go ahead and launch multiple replica processes on the master server as I proposed at the beginning of the thread, and then balance the load across these processes to reduce the impact of any one replica going slow.

(Or investigate other directory server products that can replicate in parallel ...)

Mark.

--------------------------------------------------------------------------------

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.

On Mon, 23 Nov 2015 12:25:36 +0000 Howard Chu wrote:

Bannister, Mark wrote:

...

...

...

--On Friday, November 20, 2015 6:31 PM +0000 Albert Braden abraden@about.com wrote:

...

Hi Quanah,

Are you sure your issues with syncrepl aren't specific to Zimbra? When I ran the Zimbra at Homestead/Intuit we saw syncrepl issues, but I have not seen those issues in non-Zimbra LDAP clusters.

Quite certain. For one thing, I file very few syncrepl related ITSes, since I don't use it, but many are filed... For another, while I have encountered a very few issues with delta-syncrepl, the majority of issues I find w/ Zimbra and replication are related to syncrepl when it has to be used in an initial or fallback scenario.

What sort of issues?

...

And here's some quick stats: $ grep syncrepl CHANGES | grep -v delta | wc -l 76

$ grep delta-syncrepl CHANGES | wc -l 7

Or syncrepl has had 10x times the issues.

Or syncrepl has 10x the users, more eyes spot more bugs, and is now more stable because it has had more fixes? I'm just speculating here, but I wouldn't be more confident in product Y because it is mentioned less in the change log than product X.

However, this is a tangent, and still does not help me with my original question. Correct me if I'm wrong, but delta-syncrepl is also going to be running in a serial fashion on the master server, and so, if I'm replicating out to hundreds of replicas, high network latency or timeouts on a single replica would still have a big impact on my ability to replicate everywhere within 5 minutes.

Syncrepl/delta-syncrepl doesn't make a difference here. Unless someone has a better suggestion, as I don't suppose OpenLDAP is going to be fixed to do this in parallel anytime soon, I will go ahead and launch multiple replica processes on the master server as I proposed at the beginning of the thread, and then balance the load across these processes to reduce the impact of any one replica going slow.

Certainly won't be changing in 2.4. Very likely to change in 2.5.

https://www.openldap.org/its/index.cgi/Incoming?id=8227

...

(Or investigate other directory server products that can replicate in parallel ...)

Have fun investigating.

I don't know if this is of any use to anyone. There doesn't seem to be a whole lot of information available.

https://technet.microsoft.com/en-us/library/cc755994%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396

...

...

And here's some quick stats: $ grep syncrepl CHANGES | grep -v delta | wc -l 76

$ grep delta-syncrepl CHANGES | wc -l 7

Or syncrepl has had 10x times the issues.

Or syncrepl has 10x the users, more eyes spot more bugs, and is now more stable because it has had more fixes? I'm just speculating here, but I wouldn't be more confident in product Y because it is mentioned less in the change log than product X.

Nope. I've been using syncrepl since it was first released in 2.2. I switched over the production ldap servers I was running @ Stanford to syncrepl when 2.3 came out, and found a really vexing problem when mass updates occurred (primarily at quarter end when we'd receive 10s of thousands of updates due to class enrollment changes). While 1-2 of our 6 replicas would stay up to date, the other 4 would fall hours or days behind. I.e., I'm quite familiar with the issue you are discussing, because I've been experiencing it for over a decade. Out of this, delta-syncrepl was born. I've been using it steadily now for over a decade. I switched Zimbra to it back in 2007, and it is deployed at customers world wide, from insallations with 10-20 users to installations with many millions of users. Has there been the occassional issue? Yes. But again, the vast majority of problems I encounter when investigating problems around replication come from syncrepl fallback. There is a very significant fix coming out in 2.4.43 that was found by Zimbra with syncrepl refresh when it is interrupted during the refresh phase (ITS#8281).

So, when I talk about replication and reliability in relation to OpenLDAP, this is something I've been working with since the early days of OpenLDAP 2.1. It's something that as a part of my job, I have to take with the utmost seriousness.

Now, you can ignore my advice if you wish, it doesn't matter to me one way or the other. But to me, mission critical systems must be as reliable as possible, and I've only found that to be the case with openldap replication when I deploy delta-syncrepl as the primary replication mechanism.

Thanks for your insight, Quanah, and this is a much more substantial argument now than doing 'grep -c' on a changelog, and I appreciate the time you spent to make that case.

If we're primarily adding or deleting entire objects, however, not modifying single attributes, won't delta-syncrepl just fall back to syncrepl anyway?

And this still doesn't change the fact that I'm now going to have to spin up 10-20 LDAP replicas on a single machine to get around the single-threaded replication engine problem.

Mark.

--------------------------------------------------------------------------------

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.

Quanah Gibson-Mount wrote:

All write ops (add/mod/rename/delete etc) get stored in the accesslog db. Fallback only occurs when the change has been expired out of the accesslog db.

One of the caveats with delta-syncrepl is that partial replication based on ACLs gets much more complicated.

In case of OATH-LDAP I don't want to replicate attribute 'oathSecret' to read-only consumers because it's of no use there anyway. So I have separate groups for providers and all replicas applying different ACLs to them.

Defining ACLs for distinct values in accesslog's 'reqMod' attribute is not impossible but more complex. Especially I expect it's not tested with delta-syncrepl at all so far.

Ciao, Michael.

Hi Michael,

...

If we're primarily adding or deleting entire objects, however, not modifying single attributes, won't delta-syncrepl just fall back to syncrepl anyway?

Can't you change your modification process to modify entries instead of deleting and re-adding them?

I didn't say we deleted and re-added stuff. We will modify if the object needs modifying, but most of our operations involve either adding brand new objects, or deleting old ones.

...

And this still doesn't change the fact that I'm now going to have to spin up 10-20 LDAP replicas on a single machine to get around the single-threaded replication engine problem.

This is the concept of "replication hubs". You can do this to spread the syncrepl load.

Ah, "replication hubs", yes. I'd rather not have to go down this route unless I have to, but at the moment it seems there is little choice.

Mark.

--------------------------------------------------------------------------------

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.