Another instance:
Yes, TCP keepalive is enabled. 

aaa-prod-aws-7:1636
# requesting: contextCSN
contextCSN: 20250101065905.147164Z#000000#000#000000

aaa-prod-aws-7:2636
# requesting: contextCSN
contextCSN: 20250102140005.217756Z#000000#000#000000


dn: cn=Consumer 147,cn=Database 1,cn=Databases,cn=Monitor
objectClass: olmSyncReplInstance
cn: Consumer 147

Consumer logs:

Jan  1 01:59:12 aaa-prod-aws-7 slapd[1161089]: do_syncrep2: rid=147 cookie=rid=147,csn=20250101065905.147164Z#000000#000#000000
Jan  1 01:59:12 aaa-prod-aws-7 slapd[1161089]: syncrepl_entry: rid=147 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20250101065905.147164Z#000000#000#000000 tid 0x7f8e4a5fd640
Jan  1 01:59:12 aaa-prod-aws-7 slapd[1161089]: slap_queue_csn: queueing 0x7f8e40152360 20250101065905.147164Z#000000#000#000000
Jan  1 01:59:12 aaa-prod-aws-7 slapd[1161089]: slap_graduate_commit_csn: removing 0x7f8e40152360 20250101065905.147164Z#000000#000#000000
Jan  1 01:59:12 aaa-prod-aws-7 slapd[1161089]: slap_queue_csn: queueing 0x7f8e4036c6c0 20250101065905.147164Z#000000#000#000000
Jan  1 01:59:12 aaa-prod-aws-7 slapd[1161089]: slap_graduate_commit_csn: removing 0x7f8e4036c6c0 20250101065905.147164Z#000000#000#000000

(Nothing after the above is logged regarding replication)

Master:

Jan  1 01:59:05 aaa-prod-master-1 slapd[3281130]: conn=1034 op=1 syncprov_sendresp: cookie=rid=147,csn=20250101065905.124585Z#000000#000#000000
Jan  1 01:59:05 aaa-prod-master-1 slapd[3281130]: conn=1034 op=1 syncprov_sendresp: cookie=rid=147,csn=20250101065905.147164Z#000000#000#000000

(Nothing after the above for rid=147)

After restarting Consumer:

Jan  2 09:29:54 aaa-prod-aws-7 slapd[2750000]: slap_get_csn: conn=-1 op=0 generated new csn=20250102142954.929120Z#000000#000#000000 manage=
0
Jan  2 09:29:55 aaa-prod-aws-7 slapd[2750000]: do_syncrep1: rid=147 starting refresh (sending cookie=rid=147,csn=20250101065905.147164Z#0000
00#000#000000)
Jan  2 09:29:59 aaa-prod-aws-7 slapd[2750000]: syncrepl_entry: rid=147 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_ADD) csn=(none) tid 0x7f3288dfd640
Jan  2 09:29:59 aaa-prod-aws-7 slapd[2750000]: syncrepl_entry: rid=147 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_ADD) csn=(none) tid 0x7f3288dfd640

Thanks,
Suresh

On Thu, Jan 2, 2025 at 6:34 AM Ondřej Kuzník <ondra@mistotebe.net> wrote:
On Sun, Dec 29, 2024 at 01:28:42PM -0500, Suresh Veliveli wrote:
> Another instance where replication is stuck and not recovering.
>
> # requesting: contextCSN
> contextCSN: *20241229135907.725117Z#000000#000#000000*
> aaa-prod-aws-10:2636
> # requesting: contextCSN
> contextCSN:* 20241228185913.665451Z#000000#000#000000*
>
> *Log info:*
> Dec 28 13:59:21 aaa-prod-aws-10 slapd[1161864]: do_syncrep2: rid=650
> cookie=rid=650,csn=20241228185913.665451Z#000000#000#000000
> Dec 28 13:59:21 aaa-prod-aws-10 slapd[1161864]: syncrepl_entry: rid=650
> LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY)
> csn=20241228185913.665451Z#000000#000#000000 tid 0x7f26ee5fd640
> Dec 28 13:59:21 aaa-prod-aws-10 slapd[1161864]: slap_queue_csn: queueing
> 0x7f26e0dcee50 20241228185913.665451Z#000000#000#000000
> Dec 28 13:59:21 aaa-prod-aws-10 slapd[1161864]: slap_graduate_commit_csn:
> removing 0x7f26e0dcee50 20241228185913.665451Z#000000#000#000000
> Dec 28 13:59:21 aaa-prod-aws-10 slapd[1161864]: slap_queue_csn: queueing
> 0x7f26e0f34360 20241228185913.665451Z#000000#000#000000
> Dec 28 13:59:21 aaa-prod-aws-10 slapd[1161864]: slap_graduate_commit_csn:
> removing 0x7f26e0f34360 20241228185913.665451Z#000000#000#000000
>
> Nothing gets logged about replication after the above.
>
> Am I missing something?

Hi Suresh,
anything in the provider logs around that time? All consumers messages
will be tagged with a specific "conn=xxx op=yyy" which you can discover
e.g. by looking for the cookie it sends at the beginning of the session.

Couple of other questions:
- is the TCP connection alive as far as the OS is concerned (I see in
  the thread you've confirmed TCP keepalive is enabled, correct?)
- could you post the cn=monitor info for the consumer? The objectclass
  to look for is olmSyncReplInstance

Thanks,

--
Ondřej Kuzník
Senior Software Engineer
Symas Corporation                       http://www.symas.com
Packaged, certified, and supported LDAP solutions powered by OpenLDAP


--
Suresh Veliveli
Sr. UNIX Systems Engineer
Georgetown University
University Information Services | Security Infrastructure and Policy-Identity and Collaboration
202-262-6676 (cell) | 202-687-3108 (work)