--On Monday, March 18, 2019 12:21 PM +0100 "Ángel L. Mateo" <amateo(a)um.es&gt; wrote: Unfortunately, you've not really provided any actionable information, so it's hard to say. I can say that I've had MMR deployments for years, using delta-syncrepl without delay issues like you are reporting. There are several examples of setting up N-way delta-syncrepl MMR servers in the tests/data/regressions area, for example the regression tests for ITS8444 and ITS8800. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

On 3/21/19 8:22 AM, Ángel L. Mateo wrote: How do you detect this? Ciao, Michael.

On 3/22/19 7:54 AM, Angel L. Mateo wrote: This is most times caused by an OpenLDAP bug. I see this quite often with MMR providers even though the entries have been correctly replicated to the other providers. Hence I asked for your detection method. I've double-checked the code of my monitoring script very often! And I'm not the only one seeing this false alarm in the monitoring. E.g. two guys approached me after my OpenLDAP lightning monitoring talk at FOSDEM reporting the same issue. And they use another monitoring tool. So please check whether changes were correctly replicated instead. Yes, that's nearly impossible if you have many changes on many entries. I've considered to search the highest entryCSN value per provider ID (server-side sorting on entryCSN, search limit 1) to compare it against its accompanying contextCSN value. But during first superficial tests I only got strange results. I have to investigate further before I can come up with detailed results. Ciao, Michael.

--On Thursday, March 21, 2019 9:22 AM +0100 "Ángel L. Mateo" <amateo(a)um.es&gt; wrote: I think one of the key differences to understand between syncrepl and delta-syncrepl is that syncrepl will simply overwrite entries that have discrepencies while delta-syncrepl will go into REFRESH mode. Since you started out with an OpenLDAP version that has numerous replication bugs, it may be your masters are not in accordance with one another, and the "delays" you are seeing are actually systems going into refresh. The first thing you should look at is if there are any instances of REFRESH in your logs, something along the lines of: do_syncrep2: rid=001 delta-sync lost sync, switching to REFRESH If you see that, the issue isn't delayed replication, the issue is that your databases are out of sync. If you don't see instances of REFRESH mode being triggered in your log, we can discuss further what the logs indicate and other things that may need looking at. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

--On Friday, March 22, 2019 9:01 AM +0100 "Angel L. Mateo" <amateo(a)um.es&gt; wrote: One of your quoted syncrepl stanzas clearly shows delta-syncrepl. I cannot magically intuit which version you're having problems with. olcSyncrepl: {1}rid=42 provider=ldap://canis42.um.es binddn=<repl user dn> bindmethod=simple credentials=XXXXXXXX searc hbase=dc=Telematica type=refreshAndPersist retry="30 +" timeout=1 logbase=" cn=log" logfilter="(&(objectClass=auditWriteObject)(reqResult=0))" schemaCh ecking=on syncdata=accesslog exattrs="pwdFailureTime" ^^^^^^^^^^^^^^^^ So that's not the same thing, and since you're not using delta-syncrepl you wouldn't find the string I was asking about. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

--On Monday, March 25, 2019 9:34 AM +0100 "Ángel L. Mateo" <amateo(a)um.es&gt; wrote: None of your configurations set a keepalive. If you have any sort of load balancer/switch in between, they will often cut idle connections, and replication will stop until slapd is restarted. I recommend always defining a keepalive interval in the syncrepl stanza to avoid these types of issues. keepalive=240:10:30 usually works well, unless the device is configured to close idle connections at less than a 5 minute interval. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

Am 26.03.19 um 07:58 schrieb Ulrich Windl: Any statefull firewall will reset/drop/delete/discard a connections state information from it's internal connection table after some time if no traffic was seen (for example for 5 minutes) Andreas

--On Tuesday, March 26, 2019 8:58 AM +0100 Ulrich Windl <Ulrich.Windl(a)rz.uni-regensburg.de&gt; wrote: Hello, You're understanding is generally incorrect. The paramater is called "keep alive" and not "kill dead" for a reason. When using refreshAndPersist connections in an environment where you have a traffic device (such as an F5), the devices are often configured to terminate idle connections after some period of time (which often defaults to 5 minutes). By configuring a keepalive in the syncrepl stanza, the connection appears "alive" to the traffic device even if there is no replication traffic occurring. Thus "keep alive". There is a *secondary* benefit, in that it does also allow slapd to detect if a connection was severed, and so re-establish a new connection in that case, but the primary purpose is to keep it from ever getting severed in the first place. If you have no "keepalive" set, and the traffic device severs the connection, slapd cannot detect this, and will think it's still connected and no replication will occur until slapd is restarted. I.e., the primary purpose is to keep the connection alive. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

--On Wednesday, March 27, 2019 9:24 AM +0100 "Ángel L. Mateo" <amateo(a)um.es&gt; wrote: Yeah, it's more of a safeguard to ensure one avoids those types of issues. As for your current situation, I did have a customer the other day who was reporting multi-hour replication delays. But when I looked into it, the issue was not unexpected. I don't know if this has any relevance to your situation, but here is what they were doing. They had a 4-way MMR setup. They were doing 50k ADDs to each of the masters. This is one of the worst things one could do, becuase it will massively slow down replication and cause all the servers to get behind due to the contention caused by the massive number of ADD ops occurring on each server while also trying to replicate the ADDs from the other servers. I.e., the best route would be to simply ADD all 150k entries to a single member node and let them replicate out from there. You've noted before wanting some information on what replication lines mean. It's hard to answer that exactly, as what's important to examine and what isn't important can be difficult to determine without looking at the actual logs and having some knowledge of what's going on, but this may be of some help. Here are some different sets of logs and scenarios. All logs, etc, are based on delta-syncrepl being in use. Scenario #1: Skipping an change that's too far in the past. This should have been processed already from the original change on the original master (SID 066). This change request came from SID 0c9 Mar 25 12:19:23 master1 slapd[29939]: do_syncrep2: rid=201 cookie=rid=201,sid=0c9,csn=20190325191922.072386Z#000000#066#000000 Mar 25 12:19:23 master1 slapd[29939]: do_syncrep2: rid=201 CSN too old, ignoring 20190325191922.072386Z#000000#066#000000 (reqStart=20190325191922.000185Z,cn=deltalog) Scenario #2: Applying a change that originated on another server: Mar 25 12:19:23 master1 slapd[29939]: do_syncrep2: rid=102 cookie=rid=102,sid=066,csn=20190325191923.177761Z#000000#066#000000 Mar 25 12:19:23 master1 slapd[29939]: slap_queue_csn: queueing 0x7f9ab88ce0f0 20190325191923.177761Z#000000#066#000000 Mar 25 12:19:23 master1 slapd[29939]: syncrepl_message_to_op: rid=102 tid bcffc700 Mar 25 12:19:23 master1 slapd[29939]: slap_queue_csn: queueing 0x7f9ab842eda0 20190325191923.177761Z#000000#066#000000 Mar 25 12:19:23 master1 slapd[29939]: syncprov_sendresp: to=12d, cookie=rid=101,sid=065,csn=20190325191923.177761Z#000000#066#000000 Mar 25 12:19:23 master1 slapd[29939]: syncprov_matchops: skipping original sid 066 Mar 25 12:19:23 master1 slapd[29939]: slap_graduate_commit_csn: removing 0x7f9ab842eda0 20190325191923.177761Z#000000#066#000000 Mar 25 12:19:23 master1 slapd[29939]: slap_graduate_commit_csn: removing 0x7f9ab88ce0f0 20190325191923.177761Z#000000#066#000000 Mar 25 12:19:23 master1 slapd[29939]: syncprov_sendresp: to=0c9, cookie=rid=101,sid=065,csn=20190325191923.177761Z#000000#066#000000 Mar 25 12:19:23 master1 slapd[29939]: syncrepl_message_to_op: rid=102 be_add cn=QA_MACHINE_9F06E7769C9B49B49D737414662C9089,ou=machines,dc=example,dc=com (0) Mar 25 12:19:23 master1 slapd[29939]: slap_queue_csn: queueing 0x7f9ab8a3b930 20190325191923.177761Z#000000#066#000000 Mar 25 12:19:23 master1 slapd[29939]: slap_graduate_commit_csn: removing 0x7f9ab8a3b930 20190325191923.177761Z#000000#066#000000 So in the above, this master (SID065) got a change from another master (SID066). It was not a change it had received yet from 066, so it applies it. It skips sending this change back to the originating server (Thus the "skipping original sid 066" line). It forwards the change to its other listeners (0c9, 12d) in case they do not have a direct link to 066, and commits the change locally. Scenario #3: Originating a change: Mar 25 03:58:24 master1 slapd[29939]: conn=5788689 op=75 ADD dn="cn=ATOM Virtual:843a4b737992,ou=identities,dc=example,dc=com" Mar 25 03:58:24 master1 slapd[29939]: slap_queue_csn: queueing 0x7f9ab053ed40 20190325105824.517526Z#000000#065#000000 Mar 25 03:58:24 master1 slapd[29939]: slap_queue_csn: queueing 0x7f9ab106c0f0 20190325105824.517526Z#000000#065#000000 Mar 25 03:58:24 master1 slapd[29939]: syncprov_sendresp: to=12d, cookie=rid=101,sid=065,csn=20190325105824.517526Z#000000#065#000000 Mar 25 03:58:24 master1 slapd[29939]: syncprov_sendresp: to=0c9, cookie=rid=101,sid=065,csn=20190325105824.517526Z#000000#065#000000 Mar 25 03:58:24 master1 slapd[29939]: syncprov_sendresp: to=066, cookie=rid=101,sid=065,csn=20190325105824.517526Z#000000#065#000000 Mar 25 03:58:24 master1 slapd[29939]: slap_graduate_commit_csn: removing 0x7f9ab106c0f0 20190325105824.517526Z#000000#065#000000 Mar 25 03:58:24 master1 slapd[29939]: conn=5788689 op=75 RESULT tag=105 err=0 text= Mar 25 03:58:24 master1 slapd[29939]: slap_graduate_commit_csn: removing 0x7f9ab053ed40 20190325105824.517526Z#000000#065#000000 So in the above, the master originating this change (065) creates the relevant CSN and forwards this change to its listeners (SIDs 12d, 0c9, 066), commits, and cleans up. To read a CSN value: 20190325105824.517526Z#000000#065#000000 The first part is the time (Zulu) that the change occured. So in above case, the change occurred at: 2019/03/25 @ 10:58:24.517526 seconds Zulu time (Server time as we see above was 03:58:24), so this server is 7 hours behind Zulu time. The server where the modification was made was the one with a SID value of 065. If you're looking at the contextCSN values on a server, it essentially tells you the last change it processed from each master that it talks to (and its own current change value). If the values aren't matching between servers, then you need to see if those servers are still processing changes from the other masters. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

--On Monday, April 01, 2019 9:40 AM +0200 "Ángel L. Mateo" <amateo(a)um.es&gt; wrote: Hi, Unfortunately there's been near zero information provided that's actionable on this end, so unless you're willing to be more forthcoming, there's not really any additional assistance I can provide. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

--On Thursday, April 04, 2019 2:53 PM +0200 "Ángel L. Mateo" <amateo(a)um.es&gt; wrote: This indicates that there is an "idletimeout" directive configured and that the "limits" directive has not been configured for the replication DN to by pass that limit. The replication DN generally should not be subject to time or size limits. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

El 4/4/19 a las 16:32, Quanah Gibson-Mount escribió: I have an idletimeout directive: dn: cn=config olcIdleTimeout: 600 but I also have the lits directive for replica user: olcLimits: {6}dn.exact=cn=Replica,ou=People,ou=Management,o=SlapdRoot size.hard=unlimited size.soft=unlimited time.hard=unlimited time.soft=unlimited so, according to Howard Chu, its must be another reason. -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868889150 Fax: 868888337

El 5/4/19 a las 15:23, Quanah Gibson-Mount escribió: The only difference between old servers and the new one, besides openldap version, is the version of the OS (ubuntu 14.04 vs ubuntu 18.04). I'm looking form something in the os closing idle connections, but I can't find it. -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868889150 Fax: 868888337

--On Friday, March 22, 2019 10:55 AM +0100 "Angel L. Mateo" <amateo(a)um.es&gt; wrote: How are you determining that you have a delay? --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

--On Friday, March 22, 2019 8:35 PM +0100 Michael Ströder <michael(a)stroeder.com&gt; wrote: It's not a bug. syncprov-checkpoint <ops> <minutes> After a write operation has succeeded, write the contextCSN to the underlying database if <ops> write operations or more than <minutes> time have passed since the last checkpoint. Checkpointing is disabled by default. Note that he has a 10 minute interval set on the checkpoint: olcSpCheckpoint: 100 10 --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

--On Friday, March 22, 2019 10:05 PM +0100 Michael Ströder <michael(a)stroeder.com&gt; wrote: I would suggest creating a repeatable test case and filing an ITS then, since one generally does need to be able to validate existing contextCSNs across a cluster. ;) --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

El 22/3/19 a las 19:23, Quanah Gibson-Mount escribió: I manually get changes from accesslog database (of provider ldap server) and look for them in the consumer ldap. -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868889150 Fax: 868888337