We've started experiencing syncrepl replication issues recently that look like this in the logs:
syncrepl_entry: rid=011 be_search (0) syncrepl_entry: rid=011 uid=example,ou=People,dc=foo,dc=bar syncrepl_null_callback : error code 0x7a syncrepl_entry: rid=011 be_modify uid=example,ou=People,dc=foo,dc=bar (122)
We have a 3 node multi-master/provider cluster that we recently upgraded from 2.6.10 to 2.6.13. We've been using the Symas packages, on Debian 12, FWIW.
Our config uses simple syncrepl (not delta-syncrepl). We replicate both cn=config as well as our main base dn.
Upon upgrading to 2.6.13, we faced an initial hiccup because the symas packages transitioned to use the OS provided cyrus sasl libraries, which meant the sasl config file changed locations. Once we sorted that out things seemed to be functioning properly. I should have checked the changelog on the package.
As a practice we make all ldap entry updates/additions/modifications via one designated "primary" cluster member. My initial attempt to resolve the issue involved reloading the db on the two "non-primary" nodes from a slapcat from the "primary" node. This seemed to fix the replication issue (I could make changes via our primary node and they'd replicate properly to the other nodes), however a day later I'm noticing that replication has stalled out again.
Does this sort of issue ring any bells for anyone? At first glance I don't see anything in the CHANGES file between v2.6.10 and v2.6.13 that would trigger this issue, but I'm certainly not an expert. Our cluster has been performing flawlessly with the 2.6 branch for several years now, so this is a bit disconcerting.
Ben
Today, at any rate, simply restarting slapd on the two non-primary cluster nodes got replication moving again (that didn't work yesterday, so I had to reload the db from a slapcat dump, as mentioned in my original message). Guess I'll see if it comes back again tomorrow.
Ben
On Wed, Jul 1, 2026 at 2:39 PM Ben Poliakoff benp@reed.edu wrote:
We've started experiencing syncrepl replication issues recently that look like this in the logs:
syncrepl_entry: rid=011 be_search (0) syncrepl_entry: rid=011 uid=example,ou=People,dc=foo,dc=bar syncrepl_null_callback : error code 0x7a syncrepl_entry: rid=011 be_modify uid=example,ou=People,dc=foo,dc=bar (122)
We have a 3 node multi-master/provider cluster that we recently upgraded from 2.6.10 to 2.6.13. We've been using the Symas packages, on Debian 12, FWIW.
Our config uses simple syncrepl (not delta-syncrepl). We replicate both cn=config as well as our main base dn.
Upon upgrading to 2.6.13, we faced an initial hiccup because the symas packages transitioned to use the OS provided cyrus sasl libraries, which meant the sasl config file changed locations. Once we sorted that out things seemed to be functioning properly. I should have checked the changelog on the package.
As a practice we make all ldap entry updates/additions/modifications via one designated "primary" cluster member. My initial attempt to resolve the issue involved reloading the db on the two "non-primary" nodes from a slapcat from the "primary" node. This seemed to fix the replication issue (I could make changes via our primary node and they'd replicate properly to the other nodes), however a day later I'm noticing that replication has stalled out again.
Does this sort of issue ring any bells for anyone? At first glance I don't see anything in the CHANGES file between v2.6.10 and v2.6.13 that would trigger this issue, but I'm certainly not an expert. Our cluster has been performing flawlessly with the 2.6 branch for several years now, so this is a bit disconcerting.
Ben
-- Ben Poliakoff benp@reed.edu - Technology Infrastructure Services - Reed College
Hi!
Maybe explain how you did “reload” your DB exactly (just in case).
Kind regards, Ulrich Windl
From: Ben Poliakoff benp@reed.edu Sent: Wednesday, July 1, 2026 11:40 PM To: openldap-technical@openldap.org Subject: [EXT] Issues with syncrepl replication potentially since update from 2.6.10 to 2.6.13
We've started experiencing syncrepl replication issues recently that look like this in the logs:
syncrepl_entry: rid=011 be_search (0) syncrepl_entry: rid=011 uid=example,ou=People,dc=foo,dc=bar syncrepl_null_callback : error code 0x7a syncrepl_entry: rid=011 be_modify uid=example,ou=People,dc=foo,dc=bar (122)
We have a 3 node multi-master/provider cluster that we recently upgraded from 2.6.10 to 2.6.13. We've been using the Symas packages, on Debian 12, FWIW.
Our config uses simple syncrepl (not delta-syncrepl). We replicate both cn=config as well as our main base dn.
Upon upgrading to 2.6.13, we faced an initial hiccup because the symas packages transitioned to use the OS provided cyrus sasl libraries, which meant the sasl config file changed locations. Once we sorted that out things seemed to be functioning properly. I should have checked the changelog on the package.
As a practice we make all ldap entry updates/additions/modifications via one designated "primary" cluster member. My initial attempt to resolve the issue involved reloading the db on the two "non-primary" nodes from a slapcat from the "primary" node. This seemed to fix the replication issue (I could make changes via our primary node and they'd replicate properly to the other nodes), however a day later I'm noticing that replication has stalled out again.
Does this sort of issue ring any bells for anyone? At first glance I don't see anything in the CHANGES file between v2.6.10 and v2.6.13 that would trigger this issue, but I'm certainly not an expert. Our cluster has been performing flawlessly with the 2.6 branch for several years now, so this is a bit disconcerting.
Ben
-- Ben Poliakoff <benp@reed.edumailto:benp@reed.edu> - Technology Infrastructure Services - Reed College
The "reload" process that I've followed is the same sort of thing that I'd do if I were standing up a new cluster node:
1. grab a backup from the primary node (slapcat output of the base DN for the directory) 2. shut down slapd on the node that's to be "reloaded" 3. move aside the data.mdb and lock.mdb files on the node that's being "reloaded" 4. repopulate the db on the node that's being "reloaded" via slapadd, using the backup slapcat dump mentioned above 5. start slapd
After the above process, the "reloaded" node generally catches up pretty quickly with the primary node (observed by watching contextCSNs sync up across the cluster). I haven't had to reload them again, since I wrote my initial post to this list, but I have had to restart slapd on the non-primary nodes twice, after watching contextCSNs go out of sync.
All of this is a new issue for me, we've run openldap in a variety of configurations, and this particular configuration has been very reliable for 3 or 4 years now, so I'm puzzled.
Ben
On Thu, Jul 2, 2026 at 11:55 PM Windl, Ulrich u.windl@ukr.de wrote:
Hi!
Maybe explain how you did “reload” your DB exactly (just in case).
Kind regards,
Ulrich Windl
*From:* Ben Poliakoff benp@reed.edu *Sent:* Wednesday, July 1, 2026 11:40 PM *To:* openldap-technical@openldap.org *Subject:* [EXT] Issues with syncrepl replication potentially since update from 2.6.10 to 2.6.13
We've started experiencing syncrepl replication issues recently that look like this in the logs:
syncrepl_entry: rid=011 be_search (0) syncrepl_entry: rid=011 uid=example,ou=People,dc=foo,dc=bar syncrepl_null_callback : error code 0x7a syncrepl_entry: rid=011 be_modify uid=example,ou=People,dc=foo,dc=bar (122)
We have a 3 node multi-master/provider cluster that we recently upgraded from 2.6.10 to 2.6.13.
We've been using the Symas packages, on Debian 12, FWIW.
Our config uses simple syncrepl (not delta-syncrepl). We replicate both cn=config as well as our main base dn.
Upon upgrading to 2.6.13, we faced an initial hiccup because the symas packages transitioned to use the OS provided cyrus sasl libraries, which meant the sasl config file changed locations. Once we sorted that out things seemed to be functioning properly. I should have checked the changelog on the package.
As a practice we make all ldap entry updates/additions/modifications via one designated "primary" cluster member. My initial attempt to resolve the issue involved reloading the db on the two "non-primary" nodes from a slapcat from the "primary" node. This seemed to fix the replication issue (I could make changes via our primary node and they'd replicate properly to the other nodes), however a day later I'm noticing that replication has stalled out again.
Does this sort of issue ring any bells for anyone? At first glance I don't see anything in the CHANGES file between v2.6.10 and v2.6.13 that would trigger this issue, but I'm certainly not an expert. Our cluster has been performing flawlessly with the 2.6 branch for several years now, so this is a bit disconcerting.
Ben
--
Ben Poliakoff benp@reed.edu - Technology Infrastructure Services - Reed College
Full commands.
On Mon, Jul 6, 2026, 18:24 Ben Poliakoff benp@reed.edu wrote:
The "reload" process that I've followed is the same sort of thing that I'd do if I were standing up a new cluster node:
- grab a backup from the primary node (slapcat output of the base DN for
the directory) 2. shut down slapd on the node that's to be "reloaded" 3. move aside the data.mdb and lock.mdb files on the node that's being "reloaded" 4. repopulate the db on the node that's being "reloaded" via slapadd, using the backup slapcat dump mentioned above 5. start slapd
After the above process, the "reloaded" node generally catches up pretty quickly with the primary node (observed by watching contextCSNs sync up across the cluster). I haven't had to reload them again, since I wrote my initial post to this list, but I have had to restart slapd on the non-primary nodes twice, after watching contextCSNs go out of sync.
All of this is a new issue for me, we've run openldap in a variety of configurations, and this particular configuration has been very reliable for 3 or 4 years now, so I'm puzzled.
Ben
On Thu, Jul 2, 2026 at 11:55 PM Windl, Ulrich u.windl@ukr.de wrote:
Hi!
Maybe explain how you did “reload” your DB exactly (just in case).
Kind regards,
Ulrich Windl
*From:* Ben Poliakoff benp@reed.edu *Sent:* Wednesday, July 1, 2026 11:40 PM *To:* openldap-technical@openldap.org *Subject:* [EXT] Issues with syncrepl replication potentially since update from 2.6.10 to 2.6.13
We've started experiencing syncrepl replication issues recently that look like this in the logs:
syncrepl_entry: rid=011 be_search (0) syncrepl_entry: rid=011 uid=example,ou=People,dc=foo,dc=bar syncrepl_null_callback : error code 0x7a syncrepl_entry: rid=011 be_modify uid=example,ou=People,dc=foo,dc=bar (122)
We have a 3 node multi-master/provider cluster that we recently upgraded from 2.6.10 to 2.6.13.
We've been using the Symas packages, on Debian 12, FWIW.
Our config uses simple syncrepl (not delta-syncrepl). We replicate both cn=config as well as our main base dn.
Upon upgrading to 2.6.13, we faced an initial hiccup because the symas packages transitioned to use the OS provided cyrus sasl libraries, which meant the sasl config file changed locations. Once we sorted that out things seemed to be functioning properly. I should have checked the changelog on the package.
As a practice we make all ldap entry updates/additions/modifications via one designated "primary" cluster member. My initial attempt to resolve the issue involved reloading the db on the two "non-primary" nodes from a slapcat from the "primary" node. This seemed to fix the replication issue (I could make changes via our primary node and they'd replicate properly to the other nodes), however a day later I'm noticing that replication has stalled out again.
Does this sort of issue ring any bells for anyone? At first glance I don't see anything in the CHANGES file between v2.6.10 and v2.6.13 that would trigger this issue, but I'm certainly not an expert. Our cluster has been performing flawlessly with the 2.6 branch for several years now, so this is a bit disconcerting.
Ben
--
Ben Poliakoff benp@reed.edu - Technology Infrastructure Services - Reed College
1. systemctl stop symas-openldap-server (stops slapd) 2. within the db directory (in my case /var/symas/openldap-data): mv data.mdb data.mdb.prev; mv lock.mdb lock.mdb.prev 3. $SLAPADD -b $BASE -F $SLAPD_D -v < /path/to/slapcat-dump-file 4. systemctl start symas-openldap-server
Ben
On Mon, Jul 6, 2026 at 1:13 PM Erik de Waard erikdewaard@gmail.com wrote:
Full commands.
On Mon, Jul 6, 2026, 18:24 Ben Poliakoff benp@reed.edu wrote:
The "reload" process that I've followed is the same sort of thing that I'd do if I were standing up a new cluster node:
- grab a backup from the primary node (slapcat output of the base DN for
the directory) 2. shut down slapd on the node that's to be "reloaded" 3. move aside the data.mdb and lock.mdb files on the node that's being "reloaded" 4. repopulate the db on the node that's being "reloaded" via slapadd, using the backup slapcat dump mentioned above 5. start slapd
After the above process, the "reloaded" node generally catches up pretty quickly with the primary node (observed by watching contextCSNs sync up across the cluster). I haven't had to reload them again, since I wrote my initial post to this list, but I have had to restart slapd on the non-primary nodes twice, after watching contextCSNs go out of sync.
All of this is a new issue for me, we've run openldap in a variety of configurations, and this particular configuration has been very reliable for 3 or 4 years now, so I'm puzzled.
Ben
On Thu, Jul 2, 2026 at 11:55 PM Windl, Ulrich u.windl@ukr.de wrote:
Hi!
Maybe explain how you did “reload” your DB exactly (just in case).
Kind regards,
Ulrich Windl
*From:* Ben Poliakoff benp@reed.edu *Sent:* Wednesday, July 1, 2026 11:40 PM *To:* openldap-technical@openldap.org *Subject:* [EXT] Issues with syncrepl replication potentially since update from 2.6.10 to 2.6.13
We've started experiencing syncrepl replication issues recently that look like this in the logs:
syncrepl_entry: rid=011 be_search (0) syncrepl_entry: rid=011 uid=example,ou=People,dc=foo,dc=bar syncrepl_null_callback : error code 0x7a syncrepl_entry: rid=011 be_modify uid=example,ou=People,dc=foo,dc=bar (122)
We have a 3 node multi-master/provider cluster that we recently upgraded from 2.6.10 to 2.6.13.
We've been using the Symas packages, on Debian 12, FWIW.
Our config uses simple syncrepl (not delta-syncrepl). We replicate both cn=config as well as our main base dn.
Upon upgrading to 2.6.13, we faced an initial hiccup because the symas packages transitioned to use the OS provided cyrus sasl libraries, which meant the sasl config file changed locations. Once we sorted that out things seemed to be functioning properly. I should have checked the changelog on the package.
As a practice we make all ldap entry updates/additions/modifications via one designated "primary" cluster member. My initial attempt to resolve the issue involved reloading the db on the two "non-primary" nodes from a slapcat from the "primary" node. This seemed to fix the replication issue (I could make changes via our primary node and they'd replicate properly to the other nodes), however a day later I'm noticing that replication has stalled out again.
Does this sort of issue ring any bells for anyone? At first glance I don't see anything in the CHANGES file between v2.6.10 and v2.6.13 that would trigger this issue, but I'm certainly not an expert. Our cluster has been performing flawlessly with the 2.6 branch for several years now, so this is a bit disconcerting.
Ben
--
Ben Poliakoff benp@reed.edu - Technology Infrastructure Services - Reed College
Hi Ben,
Do you run slapd as the root user or as I would typically advise, as a mere mortal "ldap" or "openldap" user? If you do not run as root, do not forget to chown the files after the slapadd. Also, I don't think it matters, but I typically use "-l /path/to/slapcat.ldif" and not a redirect.
Chris
On 7/6/26 2:04 PM, Ben Poliakoff wrote:
- systemctl stop symas-openldap-server (stops slapd)
- within the db directory (in my case /var/symas/openldap-data): mv
data.mdb data.mdb.prev; mv lock.mdb lock.mdb.prev 3. $SLAPADD -b $BASE -F $SLAPD_D -v < /path/to/slapcat-dump-file 4. systemctl start symas-openldap-server
Ben
On Mon, Jul 6, 2026 at 1:13 PM Erik de Waard erikdewaard@gmail.com wrote:
Full commands. On Mon, Jul 6, 2026, 18:24 Ben Poliakoff <benp@reed.edu> wrote: The "reload" process that I've followed is the same sort of thing that I'd do if I were standing up a new cluster node: 1. grab a backup from the primary node (slapcat output of the base DN for the directory) 2. shut down slapd on the node that's to be "reloaded" 3. move aside the data.mdb and lock.mdb files on the node that's being "reloaded" 4. repopulate the db on the node that's being "reloaded" via slapadd, using the backup slapcat dump mentioned above 5. start slapd After the above process, the "reloaded" node generally catches up pretty quickly with the primary node (observed by watching contextCSNs sync up across the cluster). I haven't had to reload them again, since I wrote my initial post to this list, but I have had to restart slapd on the non-primary nodes twice, after watching contextCSNs go out of sync. All of this is a new issue for me, we've run openldap in a variety of configurations, and this particular configuration has been very reliable for 3 or 4 years now, so I'm puzzled. Ben On Thu, Jul 2, 2026 at 11:55 PM Windl, Ulrich <u.windl@ukr.de> wrote: Hi! Maybe explain how you did “reload” your DB exactly (just in case). Kind regards, Ulrich Windl *From:*Ben Poliakoff <benp@reed.edu> *Sent:* Wednesday, July 1, 2026 11:40 PM *To:* openldap-technical@openldap.org *Subject:* [EXT] Issues with syncrepl replication potentially since update from 2.6.10 to 2.6.13 We've started experiencing syncrepl replication issues recently that look like this in the logs: syncrepl_entry: rid=011 be_search (0) syncrepl_entry: rid=011 uid=example,ou=People,dc=foo,dc=bar syncrepl_null_callback : error code 0x7a syncrepl_entry: rid=011 be_modify uid=example,ou=People,dc=foo,dc=bar (122) We have a 3 node multi-master/provider cluster that we recently upgraded from 2.6.10 to 2.6.13. We've been using the Symas packages, on Debian 12, FWIW. Our config uses simple syncrepl (not delta-syncrepl). We replicate both cn=config as well as our main base dn. Upon upgrading to 2.6.13, we faced an initial hiccup because the symas packages transitioned to use the OS provided cyrus sasl libraries, which meant the sasl config file changed locations. Once we sorted that out things seemed to be functioning properly. I should have checked the changelog on the package. As a practice we make all ldap entry updates/additions/modifications via one designated "primary" cluster member. My initial attempt to resolve the issue involved reloading the db on the two "non-primary" nodes from a slapcat from the "primary" node. This seemed to fix the replication issue (I could make changes via our primary node and they'd replicate properly to the other nodes), however a day later I'm noticing that replication has stalled out again. Does this sort of issue ring any bells for anyone? At first glance I don't see anything in the CHANGES file between v2.6.10 and v2.6.13 that would trigger this issue, but I'm certainly not an expert. Our cluster has been performing flawlessly with the 2.6 branch for several years now, so this is a bit disconcerting. Ben -- Ben Poliakoff <benp@reed.edu> - Technology Infrastructure Services - Reed College
Hi Chris,
slapd is definitely running as a non-privileged user, so no worries there. And the scripts that I use to run slapadd all run as that user as well. Definitely not a file permissions issue.
Ben
On Tue, Jul 7, 2026 at 7:37 AM Chris Paul chris.paul@rexconsulting.net wrote:
Hi Ben,
Do you run slapd as the root user or as I would typically advise, as a mere mortal "ldap" or "openldap" user? If you do not run as root, do not forget to chown the files after the slapadd. Also, I don't think it matters, but I typically use "-l /path/to/slapcat.ldif" and not a redirect.
Chris On 7/6/26 2:04 PM, Ben Poliakoff wrote:
- systemctl stop symas-openldap-server (stops slapd)
- within the db directory (in my case /var/symas/openldap-data): mv
data.mdb data.mdb.prev; mv lock.mdb lock.mdb.prev 3. $SLAPADD -b $BASE -F $SLAPD_D -v < /path/to/slapcat-dump-file 4. systemctl start symas-openldap-server
Ben
On Mon, Jul 6, 2026 at 1:13 PM Erik de Waard erikdewaard@gmail.com wrote:
Full commands.
On Mon, Jul 6, 2026, 18:24 Ben Poliakoff benp@reed.edu wrote:
The "reload" process that I've followed is the same sort of thing that I'd do if I were standing up a new cluster node:
- grab a backup from the primary node (slapcat output of the base DN
for the directory) 2. shut down slapd on the node that's to be "reloaded" 3. move aside the data.mdb and lock.mdb files on the node that's being "reloaded" 4. repopulate the db on the node that's being "reloaded" via slapadd, using the backup slapcat dump mentioned above 5. start slapd
After the above process, the "reloaded" node generally catches up pretty quickly with the primary node (observed by watching contextCSNs sync up across the cluster). I haven't had to reload them again, since I wrote my initial post to this list, but I have had to restart slapd on the non-primary nodes twice, after watching contextCSNs go out of sync.
All of this is a new issue for me, we've run openldap in a variety of configurations, and this particular configuration has been very reliable for 3 or 4 years now, so I'm puzzled.
Ben
On Thu, Jul 2, 2026 at 11:55 PM Windl, Ulrich u.windl@ukr.de wrote:
Hi!
Maybe explain how you did “reload” your DB exactly (just in case).
Kind regards,
Ulrich Windl
*From:* Ben Poliakoff benp@reed.edu *Sent:* Wednesday, July 1, 2026 11:40 PM *To:* openldap-technical@openldap.org *Subject:* [EXT] Issues with syncrepl replication potentially since update from 2.6.10 to 2.6.13
We've started experiencing syncrepl replication issues recently that look like this in the logs:
syncrepl_entry: rid=011 be_search (0) syncrepl_entry: rid=011 uid=example,ou=People,dc=foo,dc=bar syncrepl_null_callback : error code 0x7a syncrepl_entry: rid=011 be_modify uid=example,ou=People,dc=foo,dc=bar (122)
We have a 3 node multi-master/provider cluster that we recently upgraded from 2.6.10 to 2.6.13.
We've been using the Symas packages, on Debian 12, FWIW.
Our config uses simple syncrepl (not delta-syncrepl). We replicate both cn=config as well as our main base dn.
Upon upgrading to 2.6.13, we faced an initial hiccup because the symas packages transitioned to use the OS provided cyrus sasl libraries, which meant the sasl config file changed locations. Once we sorted that out things seemed to be functioning properly. I should have checked the changelog on the package.
As a practice we make all ldap entry updates/additions/modifications via one designated "primary" cluster member. My initial attempt to resolve the issue involved reloading the db on the two "non-primary" nodes from a slapcat from the "primary" node. This seemed to fix the replication issue (I could make changes via our primary node and they'd replicate properly to the other nodes), however a day later I'm noticing that replication has stalled out again.
Does this sort of issue ring any bells for anyone? At first glance I don't see anything in the CHANGES file between v2.6.10 and v2.6.13 that would trigger this issue, but I'm certainly not an expert. Our cluster has been performing flawlessly with the 2.6 branch for several years now, so this is a bit disconcerting.
Ben
--
Ben Poliakoff benp@reed.edu - Technology Infrastructure Services - Reed College
--
Chris Paul | Rex Consulting, Inc. | https://www.rexconsulting.net
On Wed, Jul 01, 2026 at 02:39:50PM -0700, Ben Poliakoff wrote:
We've started experiencing syncrepl replication issues recently that look like this in the logs:
syncrepl_entry: rid=011 be_search (0) syncrepl_entry: rid=011 uid=example,ou=People,dc=foo,dc=bar syncrepl_null_callback : error code 0x7a syncrepl_entry: rid=011 be_modify uid=example,ou=People,dc=foo,dc=bar (122)
Hi Ben, is there anything more from that session? This can happen if a local change happens between the time syncrepl reads the local copy and tries to update it[0] and as such, it should only need to try once more and succeed, not loop forever.
If it keeps happening for the same change, something might be interfering, so please post your sanitised config and logs from both sides (provider+consumer) of the affected session.
[0]. ITS#10358 https://bugs.openldap.org/show_bug.cgi?id=10358
Regards,
Hi Ondřej,
Which parts of the config would be good to see (sending over the contents of all the ldif files in slapd.d seems like overkill to send to the list)?
Replication appears to have been stable for the past several days, but I do continue to see repeated instances of logs (on the consumer) like this, even when replication seems to be working (verified both by watching contextCSN values on producer and consumer, as well as by making changes to entries on the primary provider and seeing those changes replicated in the consumer):
Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d06e5be 0x7f6d411fc6c0 conn=-1 op=0 syncprov_matchops: recording uuid for dn=uid=example-user,ou=People,dc=foo,dc=bar on opc=0x7f6d2800c550 Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d06f638 0x7f6d411fc6c0 conn=1869 op=4 syncprov_matchops: skipping original sid 001 Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d082b1f 0x7f6d411fc6c0 syncrepl_null_callback : error code 0x7a Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d08671e 0x7f6d411fc6c0 syncrepl_entry: rid=011 be_modify uid=example-user,ou=People,dc=foo,dc=bar (122) Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d0896d0 0x7f6d411fc6c0 syncrepl_entry: rid=011 be_search (0) Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d08a8b7 0x7f6d411fc6c0 syncrepl_entry: rid=011 uid=example-user,ou=People,dc=foo,dc=bar Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d08c461 0x7f6d411fc6c0 conn=-1 op=0 syncprov_matchops: recording uuid for dn=uid=example-user,ou=People,dc=foo,dc=bar on opc=0x7f6d2800c550
Ben
On Tue, Jul 7, 2026 at 3:33 AM Ondřej Kuzník ondra@mistotebe.net wrote:
On Wed, Jul 01, 2026 at 02:39:50PM -0700, Ben Poliakoff wrote:
We've started experiencing syncrepl replication issues recently that look like this in the logs:
syncrepl_entry: rid=011 be_search (0) syncrepl_entry: rid=011 uid=example,ou=People,dc=foo,dc=bar syncrepl_null_callback : error code 0x7a syncrepl_entry: rid=011 be_modify uid=example,ou=People,dc=foo,dc=bar
(122)
Hi Ben, is there anything more from that session? This can happen if a local change happens between the time syncrepl reads the local copy and tries to update it[0] and as such, it should only need to try once more and succeed, not loop forever.
If it keeps happening for the same change, something might be interfering, so please post your sanitised config and logs from both sides (provider+consumer) of the affected session.
[0]. ITS#10358 https://bugs.openldap.org/show_bug.cgi?id=10358
Regards,
-- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP
--On Wednesday, July 8, 2026 12:33 PM -0700 Ben Poliakoff benp@reed.edu wrote:
Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d06e5be 0x7f6d411fc6c0 conn=-1 op=0 syncprov_matchops: recording uuid for dn=uid=example-user,ou=People,dc=foo,dc=bar on opc=0x7f6d2800c550 Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d06f638 0x7f6d411fc6c0 conn=1869 op=4 syncprov_matchops: skipping original sid 001 Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d082b1f 0x7f6d411fc6c0 syncrepl_null_callback : error code 0x7a Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d08671e 0x7f6d411fc6c0 syncrepl_entry: rid=011 be_modify uid=example-user,ou=People,dc=foo,dc=bar (122)
From: https://ldap.com/ldap-result-code-reference-other-server-side-result-codes/
------------------------------------------------------------------------------------- assertionFailed (122)
Defined in Specification: RFC 4528
Applicable operation types: add, compare, delete, extended, modify, modify DN, search
The assertionFailed result code is used to indicate that the associated request included an LDAP assertion request control with a filter that did not match the target entry. The assertion request control may be used to indicate that the associated operation should only be processed if the target entry matches a specified filter. This is particularly useful for write operations, as it allows for a kind of atomic "check and set" operation, but you can also use it for compare and search operations (and for a search operation, the target entry is the one indicated by the search base DN). Some directory servers may also support it for certain extended operations.
If the filter included in the assertion request control does not match the target entry, then the operation will fail, and the response will include the assertionFailed result code. If the filter does match the target entry, then the operation will be processed as normal, and may succeed, or may fail for some reason unrelated to the inclusion of the assertion request control.
-------------------------------------------------------------------------------------
so looks like a non-fatal issue. I am curious what control is being asserted, and if that indicates a missing overlay in your stack.
Regards, Quanah
On Wed, Jul 08, 2026 at 11:33:05AM -0700, Ben Poliakoff wrote:
Hi Ondřej,
Which parts of the config would be good to see (sending over the contents of all the ldif files in slapd.d seems like overkill to send to the list)?
Hi Ben, a list of overlays (the dns of the database's cn=config subtree) plus the sanitised contents of syncprov and syncrepl config should be enough.
Replication appears to have been stable for the past several days, but I do continue to see repeated instances of logs (on the consumer) like this, even when replication seems to be working (verified both by watching contextCSN values on producer and consumer, as well as by making changes to entries on the primary provider and seeing those changes replicated in the consumer):
Could you provide a little more context from the log? In particular, is the entry being modified locally at the same time? When syncrepl tries to update an entry, it now attaches an assert control ensuring the entry still carries the entryCSN it did as it was trying to figure out what changes were needed. If that failed, it's because **something** updated the entry before syncrepl's change managed to land there and the assert control didn't match anymore.
And (but there isn't sufficient information in the logs) the retried modification (rereading the entry and checking what to do) probably succeeded. If you temporarily enable accesslog on the database and record both successful and failed operations, we might be able to see what happened in detail.
Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d06e5be 0x7f6d411fc6c0 conn=-1 op=0 syncprov_matchops: recording uuid for dn=uid=example-user,ou=People,dc=foo,dc=bar on opc=0x7f6d2800c550 Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d06f638 0x7f6d411fc6c0 conn=1869 op=4 syncprov_matchops: skipping original sid 001 Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d082b1f 0x7f6d411fc6c0 syncrepl_null_callback : error code 0x7a Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d08671e 0x7f6d411fc6c0 syncrepl_entry: rid=011 be_modify uid=example-user,ou=People,dc=foo,dc=bar (122) Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d0896d0 0x7f6d411fc6c0 syncrepl_entry: rid=011 be_search (0) Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d08a8b7 0x7f6d411fc6c0 syncrepl_entry: rid=011 uid=example-user,ou=People,dc=foo,dc=bar Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d08c461 0x7f6d411fc6c0 conn=-1 op=0 syncprov_matchops: recording uuid for dn=uid=example-user,ou=People,dc=foo,dc=bar on opc=0x7f6d2800c550
Regards,
On Thu, Jul 9, 2026 at 2:47 AM Ondřej Kuzník ondra@mistotebe.net wrote:
On Wed, Jul 08, 2026 at 11:33:05AM -0700, Ben Poliakoff wrote:
Hi Ondřej,
Which parts of the config would be good to see (sending over the contents of all the ldif files in slapd.d seems like overkill to send to the
list)?
Hi Ben, a list of overlays (the dns of the database's cn=config subtree) plus the sanitised contents of syncprov and syncrepl config should be enough.
The active overlays on the db in question are: dynlist, syncprov, and unique
Here are the sanitized syncrepl configs, please let me know if there are other bits of config I can provide:
dn: olcDatabase={2}mdb objectClass: olcDatabaseConfig objectClass: olcMdbConfig olcDatabase: {2}mdb olcDbDirectory: /var/symas/openldap-data olcSuffix: dc=foo,dc=bar olcAddContentAcl: FALSE olcLastMod: TRUE olcLastBind: FALSE olcLastBindPrecision: 0 olcMaxDerefDepth: 15 olcRootDN: cn=admin,dc=foo,dc=bar olcSecurity: simple_bind=1 olcSyncUseSubentry: FALSE olcSyncrepl: {0}rid=011 provider=ldap://slapd-a.ldap.foo.bar/ bin dmethod=sasl saslmech=gssapi searchbase="dc=foo,dc=bar" type=refreshAnd Persist retry="5 5 300 5" timeout=1 starttls=critical tls_reqcert=dem and olcSyncrepl: {1}rid=012 provider=ldap://slapd-b.ldap.foo.bar/ bin dmethod=sasl saslmech=gssapi searchbase="dc=foo,dc=bar" type=refreshAnd Persist retry="5 5 300 5" timeout=1 starttls=critical tls_reqcert=dem and olcSyncrepl: {2}rid=013 provider=ldap://slapd-c.ldap.foo.bar/ bin dmethod=sasl saslmech=gssapi searchbase="dc=foo,dc=bar" type=refreshAnd Persist retry="5 5 300 5" timeout=1 starttls=critical tls_reqcert=dem and olcMultiProvider: TRUE
Replication appears to have been stable for the past several days, but I
do
continue to see repeated instances of logs (on the consumer) like this, even when replication seems to be working (verified both by watching contextCSN values on producer and consumer, as well as by making changes
to
entries on the primary provider and seeing those changes replicated in
the
consumer):
Could you provide a little more context from the log? In particular, is the entry being modified locally at the same time? When syncrepl tries to update an entry, it now attaches an assert control ensuring the entry still carries the entryCSN it did as it was trying to figure out what changes were needed. If that failed, it's because **something** updated the entry before syncrepl's change managed to land there and the assert control didn't match anymore.
We do, as a practice, force almost all write operations through a single "primary" slapd instance (slapd-a). So, in general, only syncrepl should be making any changes/updates to entries on the two non-primary slapd instances (slapd-b and slapd-c).
And (but there isn't sufficient information in the logs) the retried modification (rereading the entry and checking what to do) probably succeeded. If you temporarily enable accesslog on the database and record both successful and failed operations, we might be able to see what happened in detail.
Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d06e5be 0x7f6d411fc6c0 conn=-1 op=0 syncprov_matchops: recording uuid for dn=uid=example-user,ou=People,dc=foo,dc=bar on opc=0x7f6d2800c550 Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d06f638 0x7f6d411fc6c0 conn=1869 op=4 syncprov_matchops: skipping original sid
001
Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d082b1f 0x7f6d411fc6c0 syncrepl_null_callback : error code 0x7a Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d08671e 0x7f6d411fc6c0 syncrepl_entry: rid=011 be_modify uid=example-user,ou=People,dc=foo,dc=bar (122) Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d0896d0 0x7f6d411fc6c0 syncrepl_entry: rid=011 be_search (0) Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d08a8b7 0x7f6d411fc6c0 syncrepl_entry: rid=011 uid=example-user,ou=People,dc=foo,dc=bar Jul 8 10:39:28 slapd-c daemon.info slapd[129831]: 6a3f4436.0d08c461 0x7f6d411fc6c0 conn=-1 op=0 syncprov_matchops: recording uuid for dn=uid=example-user,ou=People,dc=foo,dc=bar on opc=0x7f6d2800c550
The syncrepl process seems to get hung up "every so often" (might run for several days without issue). Today I found that it had gotten "stuck" again (newer change to an entry on slapd-a that wasn't being propagated to slap-b and slapd-c). Sometimes in these cases, simply restarting slapd on the non-primary instances gets things moving again. Today, though, that didn't work, and I was only able to get replication moving again, by deleting the entry (on the primary slapd instance) that was being referenced in the logs, and then recreating it (also on the primary slapd instance).
I'll try enabling accesslog on our test cluster (to make sure I know how to do that cleanly) and then try enabling it on our test cluster.
Ben
--On Thursday, July 9, 2026 4:38 PM -0700 Ben Poliakoff benp@reed.edu wrote:
The syncrepl process seems to get hung up "every so often" (might run for several days without issue). Today I found that it had gotten "stuck" again (newer change to an entry on slapd-a that wasn't being propagated to slap-b and slapd-c). Sometimes in these cases, simply restarting slapd on the non-primary instances gets things moving again. Today, though, that didn't work, and I was only able to get replication moving again, by deleting the entry (on the primary slapd instance) that was being referenced in the logs, and then recreating it (also on the primary slapd instance).
Hi Ben,
I'm curious if rather than deleting/adding the entry, if you modify an attribute on the entry back to its same value if that fixes it. That's worked for me in the past.
like:
dn: uid=blah,dc... changetype: modify replace: description description: account for blah
Regards, Quanah
On Thu, Jul 09, 2026 at 03:38:28PM -0700, Ben Poliakoff wrote:
The active overlays on the db in question are: dynlist, syncprov, and unique
Here are the sanitized syncrepl configs, please let me know if there are other bits of config I can provide:
dn: olcDatabase={2}mdb [...]
Right, those should be ok with all that.
Could you provide a little more context from the log? In particular, is the entry being modified locally at the same time? When syncrepl tries to update an entry, it now attaches an assert control ensuring the entry still carries the entryCSN it did as it was trying to figure out what changes were needed. If that failed, it's because **something** updated the entry before syncrepl's change managed to land there and the assert control didn't match anymore.
We do, as a practice, force almost all write operations through a single "primary" slapd instance (slapd-a). So, in general, only syncrepl should be making any changes/updates to entries on the two non-primary slapd instances (slapd-b and slapd-c).
Sure, but a longer excerpt of the log for rid=011 might show what else happened during the session as far as the consumer was concerned.
And (but there isn't sufficient information in the logs) the retried modification (rereading the entry and checking what to do) probably succeeded. If you temporarily enable accesslog on the database and record both successful and failed operations, we might be able to see what happened in detail.
The syncrepl process seems to get hung up "every so often" (might run for several days without issue). Today I found that it had gotten "stuck" again (newer change to an entry on slapd-a that wasn't being propagated to slap-b and slapd-c). Sometimes in these cases, simply restarting slapd on the non-primary instances gets things moving again. Today, though, that didn't work, and I was only able to get replication moving again, by deleting the entry (on the primary slapd instance) that was being referenced in the logs, and then recreating it (also on the primary slapd instance).
I'll try enabling accesslog on our test cluster (to make sure I know how to do that cleanly) and then try enabling it on our test cluster.
Great, that will also help. Thanks.
On Mon, Jul 13, 2026 at 2:34 AM Ondřej Kuzník ondra@mistotebe.net wrote:
Could you provide a little more context from the log? In particular, is the entry being modified locally at the same time? When syncrepl tries to update an entry, it now attaches an assert control ensuring the entry still carries the entryCSN it did as it was trying to figure out what changes were needed. If that failed, it's because **something** updated the entry before syncrepl's change managed to land there and the assert control didn't match anymore.
We do, as a practice, force almost all write operations through a single "primary" slapd instance (slapd-a). So, in general, only syncrepl should
be
making any changes/updates to entries on the two non-primary slapd instances (slapd-b and slapd-c).
Sure, but a longer excerpt of the log for rid=011 might show what else happened during the session as far as the consumer was concerned.
Here's an excerpt from this morning, from one of the consumers, when this issue came up. As near as I can tell, the changes for "user-a" and "user-b" were replicated properly, but in "user-c's" case replication stopped working, and the consumer fell into a loop, which continued until I intervened (by deleting the entry for user-c on the provider, recreating it via ldapadd, and then restarting slapd on both of the consumers):
Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11cb285b 0x7fe93dbfe6c0 do_syncrep2: rid=011 cookie=rid=011,sid=001,csn=20260713112535.267821Z#000000#001#000000 Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11cc71cf 0x7fe93dbfe6c0 syncrepl_message_to_entry: rid=011 DN: uid=user-a,ou=People,dc=foo,dc=bar, UUID: 1c96ac22-62f9-1027-8440-e9a9b4c99043 Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11cff8c3 0x7fe93dbfe6c0 syncrepl_entry: rid=011 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20260713112535.267821Z#000000#001#000000 tid 0x7fe93dbfe6c0 Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11d1566a 0x7fe93dbfe6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11d1d463 0x7fe93dbfe6c0 syncrepl_entry: rid=011 uid=user-a,ou=People,dc=foo,dc=bar Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.150bdf03 0x7fe93dbfe6c0 syncrepl_entry: rid=011 be_modify uid=user-a,ou=People,dc=foo,dc=bar (0) Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e1472f 0x7fe8f13fc6c0 do_syncrep2: rid=011 cookie=rid=011,sid=001,csn=20260713141706.387664Z#000000#001#000000 Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e1e882 0x7fe8f13fc6c0 syncrepl_message_to_entry: rid=011 DN: uid=user-b,ou=People,dc=foo,dc=bar, UUID: 3a002ff4-912d-103d-9e5f-4fce8756df8d Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e3f06b 0x7fe8f13fc6c0 syncrepl_entry: rid=011 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20260713141706.387664Z#000000#001#000000 tid 0x7fe8f13fc6c0 Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e49d6c 0x7fe8f13fc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e4b9c6 0x7fe8f13fc6c0 syncrepl_entry: rid=011 uid=user-b,ou=People,dc=foo,dc=bar Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.1b702746 0x7fe8f13fc6c0 syncrepl_entry: rid=011 be_modify uid=user-b,ou=People,dc=foo,dc=bar (0) Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.3670134f 0x7fe92ddfd6c0 do_syncrep2: rid=011 cookie=rid=011,sid=001,csn=20260713141741.874176Z#000000#001#000000 Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.3670baf2 0x7fe92ddfd6c0 syncrepl_message_to_entry: rid=011 DN: uid=user-a,ou=People,dc=foo,dc=bar, UUID: 1c96ac22-62f9-1027-8440-e9a9b4c99043 Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.36727fb0 0x7fe92ddfd6c0 syncrepl_entry: rid=011 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20260713141741.874176Z#000000#001#000000 tid 0x7fe92ddfd6c0 Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.36732e80 0x7fe92ddfd6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.36734c82 0x7fe92ddfd6c0 syncrepl_entry: rid=011 uid=user-a,ou=People,dc=foo,dc=bar Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.38289301 0x7fe92ddfd6c0 syncrepl_entry: rid=011 be_modify uid=user-a,ou=People,dc=foo,dc=bar (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.19271e69 0x7fe93cbfc6c0 do_syncrep2: rid=011 cookie=rid=011,sid=001,csn=20260713162914.373110Z#000000#001#000000 Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.192880c7 0x7fe93cbfc6c0 syncrepl_message_to_entry: rid=011 DN: uid=user-c,ou=People,dc=foo,dc=bar, UUID: 1b6a45a2-f7c6-1033-8b7c-2b6875adc332 Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.192d38b8 0x7fe93cbfc6c0 syncrepl_entry: rid=011 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20260713162914.373110Z#000000#001#000000 tid 0x7fe93cbfc6c0 Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.192ea929 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.192ef1a2 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1936d2f0 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1937a1db 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1937e217 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.193df998 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.193ea7e9 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.193ee30b 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1944b039 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.19456ca6 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1945b723 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.194c31ad 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.194cdb08 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.194d15b1 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1952a424 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122)
...etc
And (but there isn't sufficient information in the logs) the retried modification (rereading the entry and checking what to do) probably succeeded. If you temporarily enable accesslog on the database and record both successful and failed operations, we might be able to see what happened in detail.
The syncrepl process seems to get hung up "every so often" (might run
for
several days without issue). Today I found that it had gotten "stuck"
again
(newer change to an entry on slapd-a that wasn't being propagated to
slap-b
and slapd-c). Sometimes in these cases, simply restarting slapd on the non-primary instances gets things moving again. Today, though, that
didn't
work, and I was only able to get replication moving again, by deleting
the
entry (on the primary slapd instance) that was being referenced in the logs, and then recreating it (also on the primary slapd instance).
I'll try enabling accesslog on our test cluster (to make sure I know how
to
do that cleanly) and then try enabling it on our test cluster.
Great, that will also help. Thanks.
-- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP
On Mon, Jul 13, 2026 at 11:29 AM Ben Poliakoff benp@reed.edu wrote:
On Mon, Jul 13, 2026 at 2:34 AM Ondřej Kuzník ondra@mistotebe.net wrote:
Could you provide a little more context from the log? In particular, is the entry being modified locally at the same time? When syncrepl tries to update an entry, it now attaches an assert control ensuring the
entry
still carries the entryCSN it did as it was trying to figure out what changes were needed. If that failed, it's because **something** updated the entry before syncrepl's change managed to land there and the assert control didn't match anymore.
We do, as a practice, force almost all write operations through a single "primary" slapd instance (slapd-a). So, in general, only syncrepl
should be
making any changes/updates to entries on the two non-primary slapd instances (slapd-b and slapd-c).
Sure, but a longer excerpt of the log for rid=011 might show what else happened during the session as far as the consumer was concerned.
Here's an excerpt from this morning, from one of the consumers, when this issue came up. As near as I can tell, the changes for "user-a" and "user-b" were replicated properly, but in "user-c's" case replication stopped working, and the consumer fell into a loop, which continued until I intervened (by deleting the entry for user-c on the provider, recreating it via ldapadd, and then restarting slapd on both of the consumers):
Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11cb285b 0x7fe93dbfe6c0 do_syncrep2: rid=011 cookie=rid=011,sid=001,csn=20260713112535.267821Z#000000#001#000000 Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11cc71cf 0x7fe93dbfe6c0 syncrepl_message_to_entry: rid=011 DN: uid=user-a,ou=People,dc=foo,dc=bar, UUID: 1c96ac22-62f9-1027-8440-e9a9b4c99043 Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11cff8c3 0x7fe93dbfe6c0 syncrepl_entry: rid=011 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20260713112535.267821Z#000000#001#000000 tid 0x7fe93dbfe6c0 Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11d1566a 0x7fe93dbfe6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11d1d463 0x7fe93dbfe6c0 syncrepl_entry: rid=011 uid=user-a,ou=People,dc=foo,dc=bar Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.150bdf03 0x7fe93dbfe6c0 syncrepl_entry: rid=011 be_modify uid=user-a,ou=People,dc=foo,dc=bar (0) Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e1472f 0x7fe8f13fc6c0 do_syncrep2: rid=011 cookie=rid=011,sid=001,csn=20260713141706.387664Z#000000#001#000000 Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e1e882 0x7fe8f13fc6c0 syncrepl_message_to_entry: rid=011 DN: uid=user-b,ou=People,dc=foo,dc=bar, UUID: 3a002ff4-912d-103d-9e5f-4fce8756df8d Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e3f06b 0x7fe8f13fc6c0 syncrepl_entry: rid=011 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20260713141706.387664Z#000000#001#000000 tid 0x7fe8f13fc6c0 Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e49d6c 0x7fe8f13fc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e4b9c6 0x7fe8f13fc6c0 syncrepl_entry: rid=011 uid=user-b,ou=People,dc=foo,dc=bar Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.1b702746 0x7fe8f13fc6c0 syncrepl_entry: rid=011 be_modify uid=user-b,ou=People,dc=foo,dc=bar (0) Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.3670134f 0x7fe92ddfd6c0 do_syncrep2: rid=011 cookie=rid=011,sid=001,csn=20260713141741.874176Z#000000#001#000000 Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.3670baf2 0x7fe92ddfd6c0 syncrepl_message_to_entry: rid=011 DN: uid=user-a,ou=People,dc=foo,dc=bar, UUID: 1c96ac22-62f9-1027-8440-e9a9b4c99043 Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.36727fb0 0x7fe92ddfd6c0 syncrepl_entry: rid=011 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20260713141741.874176Z#000000#001#000000 tid 0x7fe92ddfd6c0 Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.36732e80 0x7fe92ddfd6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.36734c82 0x7fe92ddfd6c0 syncrepl_entry: rid=011 uid=user-a,ou=People,dc=foo,dc=bar Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.38289301 0x7fe92ddfd6c0 syncrepl_entry: rid=011 be_modify uid=user-a,ou=People,dc=foo,dc=bar (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.19271e69 0x7fe93cbfc6c0 do_syncrep2: rid=011 cookie=rid=011,sid=001,csn=20260713162914.373110Z#000000#001#000000 Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.192880c7 0x7fe93cbfc6c0 syncrepl_message_to_entry: rid=011 DN: uid=user-c,ou=People,dc=foo,dc=bar, UUID: 1b6a45a2-f7c6-1033-8b7c-2b6875adc332 Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.192d38b8 0x7fe93cbfc6c0 syncrepl_entry: rid=011 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20260713162914.373110Z#000000#001#000000 tid 0x7fe93cbfc6c0 Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.192ea929 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.192ef1a2 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1936d2f0 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1937a1db 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1937e217 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.193df998 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.193ea7e9 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.193ee30b 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1944b039 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.19456ca6 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1945b723 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.194c31ad 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.194cdb08 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.194d15b1 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1952a424 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122)
...etc
Oh and here's a corresponding excerpt from the primary provider slapd (following "conn=181661"):
Jul 13 04:25:35 slapd-a daemon.info slapd[573262]: 6a54cb2f.11af231d 0x7fd0db7fe6c0 conn=181661 op=4 syncprov_qresp: set up a new syncres mode=2 csn=20260713112535.267821Z#000000#001#000000 Jul 13 04:25:35 slapd-a daemon.info slapd[573262]: 6a54cb2f.11b11f8e 0x7fd0d9ffb6c0 conn=181661 op=4 syncprov_sendresp: to=002, cookie=rid=011,sid=001,csn=20260713112535.267821Z#000000#001#000000 Jul 13 04:25:35 slapd-a daemon.info slapd[573262]: 6a54cb2f.11b159ae 0x7fd0d9ffb6c0 conn=181661 op=4 syncprov_sendresp: sending LDAP_SYNC_MODIFY, dn=uid=user-a,ou=people,dc=foo,dc=bar Jul 13 04:25:35 slapd-a daemon.info slapd[573262]: 6a54cb2f.11ce2997 0x7fd0d9ffb6c0 conn=181661 op=4 ENTRY dn="uid=user-a,ou=people,dc=foo,dc=bar" Jul 13 07:17:06 slapd-a daemon.info slapd[573262]: 6a54f362.175ba39d 0x7fd0d9ffb6c0 conn=181661 op=4 syncprov_qresp: set up a new syncres mode=2 csn=20260713141706.387664Z#000000#001#000000 Jul 13 07:17:06 slapd-a daemon.info slapd[573262]: 6a54f362.175e6cc6 0x7fd11b1fe6c0 conn=181661 op=4 syncprov_sendresp: to=002, cookie=rid=011,sid=001,csn=20260713141706.387664Z#000000#001#000000 Jul 13 07:17:06 slapd-a daemon.info slapd[573262]: 6a54f362.175fcc2d 0x7fd11b1fe6c0 conn=181661 op=4 syncprov_sendresp: sending LDAP_SYNC_MODIFY, dn=uid=user-b,ou=people,dc=foo,dc=bar Jul 13 07:17:06 slapd-a daemon.info slapd[573262]: 6a54f362.17de88ac 0x7fd11b1fe6c0 conn=181661 op=4 ENTRY dn="uid=user-b,ou=people,dc=foo,dc=bar" Jul 13 07:17:41 slapd-a daemon.info slapd[573262]: 6a54f385.363ca03d 0x7fd11a1fc6c0 conn=181661 op=4 syncprov_qresp: set up a new syncres mode=2 csn=20260713141741.874176Z#000000#001#000000 Jul 13 07:17:41 slapd-a daemon.info slapd[573262]: 6a54f385.363e3ee1 0x7fd0db7fe6c0 conn=181661 op=4 syncprov_sendresp: to=002, cookie=rid=011,sid=001,csn=20260713141741.874176Z#000000#001#000000 Jul 13 07:17:41 slapd-a daemon.info slapd[573262]: 6a54f385.363e7d81 0x7fd0db7fe6c0 conn=181661 op=4 syncprov_sendresp: sending LDAP_SYNC_MODIFY, dn=uid=user-a,ou=people,dc=foo,dc=bar Jul 13 07:17:41 slapd-a daemon.info slapd[573262]: 6a54f385.365e0720 0x7fd0db7fe6c0 conn=181661 op=4 ENTRY dn="uid=user-a,ou=people,dc=foo,dc=bar" Jul 13 09:29:14 slapd-a daemon.info slapd[573262]: 6a55125a.18c8e928 0x7fd0dbfff6c0 conn=181661 op=4 syncprov_qresp: set up a new syncres mode=2 csn=20260713162914.373110Z#000000#001#000000 Jul 13 09:29:14 slapd-a daemon.info slapd[573262]: 6a55125a.18ca72db 0x7fd0daffd6c0 conn=181661 op=4 syncprov_sendresp: to=002, cookie=rid=011,sid=001,csn=20260713162914.373110Z#000000#001#000000 Jul 13 09:29:14 slapd-a daemon.info slapd[573262]: 6a55125a.18cabbd0 0x7fd0daffd6c0 conn=181661 op=4 syncprov_sendresp: sending LDAP_SYNC_MODIFY, dn=uid=user-c,ou=people,dc=foo,dc=bar Jul 13 09:29:14 slapd-a daemon.info slapd[573262]: 6a55125a.1954f880 0x7fd0daffd6c0 conn=181661 op=4 ENTRY dn="uid=user-c,ou=people,dc=foo,dc=bar" Jul 13 09:30:36 slapd-a daemon.info slapd[573262]: 6a5512ac.32067cfe 0x7fd110ffc6c0 conn=181661 op=4 syncprov_qresp: set up a new syncres mode=2 csn=20260713163036.837036Z#000000#001#000000 Jul 13 09:30:36 slapd-a daemon.info slapd[573262]: 6a5512ac.3207a99f 0x7fd11a9fd6c0 conn=181661 op=4 syncprov_sendresp: to=002, cookie=rid=011,sid=001,csn=20260713163036.837036Z#000000#001#000000 Jul 13 09:30:36 slapd-a daemon.info slapd[573262]: 6a5512ac.32081ec9 0x7fd11a9fd6c0 conn=181661 op=4 syncprov_sendresp: sending LDAP_SYNC_MODIFY, dn=uid=user-c,ou=people,dc=foo,dc=bar Jul 13 09:30:36 slapd-a daemon.info slapd[573262]: 6a5512ac.32449ca4 0x7fd11a9fd6c0 conn=181661 op=4 ENTRY dn="uid=user-c,ou=people,dc=foo,dc=bar" Jul 13 09:43:32 slapd-a daemon.info slapd[573262]: 6a5515b4.31258dd1 0x7fd110ffc6c0 conn=181661 op=5 UNBIND Jul 13 09:43:32 slapd-a daemon.info slapd[573262]: 6a5515b4.31288701 0x7fd110ffc6c0 conn=181661 fd=45 closed
On Mon, Jul 13, 2026 at 11:38 AM Ben Poliakoff benp@reed.edu wrote:
On Mon, Jul 13, 2026 at 11:29 AM Ben Poliakoff benp@reed.edu wrote:
On Mon, Jul 13, 2026 at 2:34 AM Ondřej Kuzník ondra@mistotebe.net wrote:
Could you provide a little more context from the log? In particular,
is
the entry being modified locally at the same time? When syncrepl tries to update an entry, it now attaches an assert control ensuring the
entry
still carries the entryCSN it did as it was trying to figure out what changes were needed. If that failed, it's because **something**
updated
the entry before syncrepl's change managed to land there and the
assert
control didn't match anymore.
We do, as a practice, force almost all write operations through a
single
"primary" slapd instance (slapd-a). So, in general, only syncrepl
should be
making any changes/updates to entries on the two non-primary slapd instances (slapd-b and slapd-c).
Sure, but a longer excerpt of the log for rid=011 might show what else happened during the session as far as the consumer was concerned.
Here's an excerpt from this morning, from one of the consumers, when this issue came up. As near as I can tell, the changes for "user-a" and "user-b" were replicated properly, but in "user-c's" case replication stopped working, and the consumer fell into a loop, which continued until I intervened (by deleting the entry for user-c on the provider, recreating it via ldapadd, and then restarting slapd on both of the consumers):
Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11cb285b 0x7fe93dbfe6c0 do_syncrep2: rid=011 cookie=rid=011,sid=001,csn=20260713112535.267821Z#000000#001#000000 Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11cc71cf 0x7fe93dbfe6c0 syncrepl_message_to_entry: rid=011 DN: uid=user-a,ou=People,dc=foo,dc=bar, UUID: 1c96ac22-62f9-1027-8440-e9a9b4c99043 Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11cff8c3 0x7fe93dbfe6c0 syncrepl_entry: rid=011 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20260713112535.267821Z#000000#001#000000 tid 0x7fe93dbfe6c0 Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11d1566a 0x7fe93dbfe6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.11d1d463 0x7fe93dbfe6c0 syncrepl_entry: rid=011 uid=user-a,ou=People,dc=foo,dc=bar Jul 13 04:25:35 slapd-c slapd[363788]: 6a54cb2f.150bdf03 0x7fe93dbfe6c0 syncrepl_entry: rid=011 be_modify uid=user-a,ou=People,dc=foo,dc=bar (0) Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e1472f 0x7fe8f13fc6c0 do_syncrep2: rid=011 cookie=rid=011,sid=001,csn=20260713141706.387664Z#000000#001#000000 Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e1e882 0x7fe8f13fc6c0 syncrepl_message_to_entry: rid=011 DN: uid=user-b,ou=People,dc=foo,dc=bar, UUID: 3a002ff4-912d-103d-9e5f-4fce8756df8d Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e3f06b 0x7fe8f13fc6c0 syncrepl_entry: rid=011 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20260713141706.387664Z#000000#001#000000 tid 0x7fe8f13fc6c0 Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e49d6c 0x7fe8f13fc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.17e4b9c6 0x7fe8f13fc6c0 syncrepl_entry: rid=011 uid=user-b,ou=People,dc=foo,dc=bar Jul 13 07:17:06 slapd-c slapd[363788]: 6a54f362.1b702746 0x7fe8f13fc6c0 syncrepl_entry: rid=011 be_modify uid=user-b,ou=People,dc=foo,dc=bar (0) Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.3670134f 0x7fe92ddfd6c0 do_syncrep2: rid=011 cookie=rid=011,sid=001,csn=20260713141741.874176Z#000000#001#000000 Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.3670baf2 0x7fe92ddfd6c0 syncrepl_message_to_entry: rid=011 DN: uid=user-a,ou=People,dc=foo,dc=bar, UUID: 1c96ac22-62f9-1027-8440-e9a9b4c99043 Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.36727fb0 0x7fe92ddfd6c0 syncrepl_entry: rid=011 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20260713141741.874176Z#000000#001#000000 tid 0x7fe92ddfd6c0 Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.36732e80 0x7fe92ddfd6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.36734c82 0x7fe92ddfd6c0 syncrepl_entry: rid=011 uid=user-a,ou=People,dc=foo,dc=bar Jul 13 07:17:41 slapd-c slapd[363788]: 6a54f385.38289301 0x7fe92ddfd6c0 syncrepl_entry: rid=011 be_modify uid=user-a,ou=People,dc=foo,dc=bar (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.19271e69 0x7fe93cbfc6c0 do_syncrep2: rid=011 cookie=rid=011,sid=001,csn=20260713162914.373110Z#000000#001#000000 Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.192880c7 0x7fe93cbfc6c0 syncrepl_message_to_entry: rid=011 DN: uid=user-c,ou=People,dc=foo,dc=bar, UUID: 1b6a45a2-f7c6-1033-8b7c-2b6875adc332 Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.192d38b8 0x7fe93cbfc6c0 syncrepl_entry: rid=011 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20260713162914.373110Z#000000#001#000000 tid 0x7fe93cbfc6c0 Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.192ea929 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.192ef1a2 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1936d2f0 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1937a1db 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1937e217 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.193df998 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.193ea7e9 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.193ee30b 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1944b039 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.19456ca6 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1945b723 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.194c31ad 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.194cdb08 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_search (0) Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.194d15b1 0x7fe93cbfc6c0 syncrepl_entry: rid=011 uid=user-c,ou=People,dc=foo,dc=bar Jul 13 09:29:14 slapd-c slapd[363788]: 6a55125a.1952a424 0x7fe93cbfc6c0 syncrepl_entry: rid=011 be_modify uid=user-c,ou=People,dc=foo,dc=bar (122)
...etc
Oh and here's a corresponding excerpt from the primary provider slapd (following "conn=181661"):
Jul 13 04:25:35 slapd-a daemon.info slapd[573262]: 6a54cb2f.11af231d 0x7fd0db7fe6c0 conn=181661 op=4 syncprov_qresp: set up a new syncres mode=2 csn=20260713112535.267821Z#000000#001#000000 Jul 13 04:25:35 slapd-a daemon.info slapd[573262]: 6a54cb2f.11b11f8e 0x7fd0d9ffb6c0 conn=181661 op=4 syncprov_sendresp: to=002, cookie=rid=011,sid=001,csn=20260713112535.267821Z#000000#001#000000 Jul 13 04:25:35 slapd-a daemon.info slapd[573262]: 6a54cb2f.11b159ae 0x7fd0d9ffb6c0 conn=181661 op=4 syncprov_sendresp: sending LDAP_SYNC_MODIFY, dn=uid=user-a,ou=people,dc=foo,dc=bar Jul 13 04:25:35 slapd-a daemon.info slapd[573262]: 6a54cb2f.11ce2997 0x7fd0d9ffb6c0 conn=181661 op=4 ENTRY dn="uid=user-a,ou=people,dc=foo,dc=bar" Jul 13 07:17:06 slapd-a daemon.info slapd[573262]: 6a54f362.175ba39d 0x7fd0d9ffb6c0 conn=181661 op=4 syncprov_qresp: set up a new syncres mode=2 csn=20260713141706.387664Z#000000#001#000000 Jul 13 07:17:06 slapd-a daemon.info slapd[573262]: 6a54f362.175e6cc6 0x7fd11b1fe6c0 conn=181661 op=4 syncprov_sendresp: to=002, cookie=rid=011,sid=001,csn=20260713141706.387664Z#000000#001#000000 Jul 13 07:17:06 slapd-a daemon.info slapd[573262]: 6a54f362.175fcc2d 0x7fd11b1fe6c0 conn=181661 op=4 syncprov_sendresp: sending LDAP_SYNC_MODIFY, dn=uid=user-b,ou=people,dc=foo,dc=bar Jul 13 07:17:06 slapd-a daemon.info slapd[573262]: 6a54f362.17de88ac 0x7fd11b1fe6c0 conn=181661 op=4 ENTRY dn="uid=user-b,ou=people,dc=foo,dc=bar" Jul 13 07:17:41 slapd-a daemon.info slapd[573262]: 6a54f385.363ca03d 0x7fd11a1fc6c0 conn=181661 op=4 syncprov_qresp: set up a new syncres mode=2 csn=20260713141741.874176Z#000000#001#000000 Jul 13 07:17:41 slapd-a daemon.info slapd[573262]: 6a54f385.363e3ee1 0x7fd0db7fe6c0 conn=181661 op=4 syncprov_sendresp: to=002, cookie=rid=011,sid=001,csn=20260713141741.874176Z#000000#001#000000 Jul 13 07:17:41 slapd-a daemon.info slapd[573262]: 6a54f385.363e7d81 0x7fd0db7fe6c0 conn=181661 op=4 syncprov_sendresp: sending LDAP_SYNC_MODIFY, dn=uid=user-a,ou=people,dc=foo,dc=bar Jul 13 07:17:41 slapd-a daemon.info slapd[573262]: 6a54f385.365e0720 0x7fd0db7fe6c0 conn=181661 op=4 ENTRY dn="uid=user-a,ou=people,dc=foo,dc=bar" Jul 13 09:29:14 slapd-a daemon.info slapd[573262]: 6a55125a.18c8e928 0x7fd0dbfff6c0 conn=181661 op=4 syncprov_qresp: set up a new syncres mode=2 csn=20260713162914.373110Z#000000#001#000000 Jul 13 09:29:14 slapd-a daemon.info slapd[573262]: 6a55125a.18ca72db 0x7fd0daffd6c0 conn=181661 op=4 syncprov_sendresp: to=002, cookie=rid=011,sid=001,csn=20260713162914.373110Z#000000#001#000000 Jul 13 09:29:14 slapd-a daemon.info slapd[573262]: 6a55125a.18cabbd0 0x7fd0daffd6c0 conn=181661 op=4 syncprov_sendresp: sending LDAP_SYNC_MODIFY, dn=uid=user-c,ou=people,dc=foo,dc=bar Jul 13 09:29:14 slapd-a daemon.info slapd[573262]: 6a55125a.1954f880 0x7fd0daffd6c0 conn=181661 op=4 ENTRY dn="uid=user-c,ou=people,dc=foo,dc=bar" Jul 13 09:30:36 slapd-a daemon.info slapd[573262]: 6a5512ac.32067cfe 0x7fd110ffc6c0 conn=181661 op=4 syncprov_qresp: set up a new syncres mode=2 csn=20260713163036.837036Z#000000#001#000000 Jul 13 09:30:36 slapd-a daemon.info slapd[573262]: 6a5512ac.3207a99f 0x7fd11a9fd6c0 conn=181661 op=4 syncprov_sendresp: to=002, cookie=rid=011,sid=001,csn=20260713163036.837036Z#000000#001#000000 Jul 13 09:30:36 slapd-a daemon.info slapd[573262]: 6a5512ac.32081ec9 0x7fd11a9fd6c0 conn=181661 op=4 syncprov_sendresp: sending LDAP_SYNC_MODIFY, dn=uid=user-c,ou=people,dc=foo,dc=bar Jul 13 09:30:36 slapd-a daemon.info slapd[573262]: 6a5512ac.32449ca4 0x7fd11a9fd6c0 conn=181661 op=4 ENTRY dn="uid=user-c,ou=people,dc=foo,dc=bar" Jul 13 09:43:32 slapd-a daemon.info slapd[573262]: 6a5515b4.31258dd1 0x7fd110ffc6c0 conn=181661 op=5 UNBIND Jul 13 09:43:32 slapd-a daemon.info slapd[573262]: 6a5515b4.31288701 0x7fd110ffc6c0 conn=181661 fd=45 closed
Just noticed another thing when this issue comes up.... the entryCSN for an entry that's not replicating from the primary server to the two consumers has a slightly different format:
Primary/provider:
entryCSN: 20260715174316.844052Z#000000#001#000000
Non-primary/consumer:
entryCSN: 20210115230536Z#000001#00#000000 entryCSN: 20210115230536Z#000001#00#000000
Not sure if that is meaningful at all. All three are running OpenLDAP 2.6.13 (Symas packages).
Ben
Ben Poliakoff wrote:
Just noticed another thing when this issue comes up.... the entryCSN for an entry that's not replicating from the primary server to the two consumers has a slightly different format:
Primary/provider: entryCSN: 20260715174316.844052Z#000000#001#000000 Non-primary/consumer: entryCSN: 20210115230536Z#000001#00#000000 entryCSN: 20210115230536Z#000001#00#000000
These are in an obsolete format, last used in OpenLDAP 2.3. No OpenLDAP software has produced them in the last 20 years.
Not sure if that is meaningful at all. All three are running OpenLDAP 2.6.13 (Symas packages).
Ben
On Thu, Jul 16, 2026 at 6:39 AM Howard Chu hyc@symas.com wrote:
Ben Poliakoff wrote:
Just noticed another thing when this issue comes up.... the entryCSN for
an entry that's not replicating from the primary server to the two consumers has a
slightly different format:
Primary/provider: entryCSN: 20260715174316.844052Z#000000#001#000000 Non-primary/consumer: entryCSN: 20210115230536Z#000001#00#000000 entryCSN: 20210115230536Z#000001#00#000000These are in an obsolete format, last used in OpenLDAP 2.3. No OpenLDAP software has produced them in the last 20 years.
That's interesting. In our environment we have a mix of these two formats (I assume the shorter format being the obsolete one). About a fifth of our ~22k entries (4291) have the shorter format of entryCSN.
I see now that I can reliably trigger this issue: by modifying an entry with the shorter (obsolete?) form of entryCSN. When I do that the entryCSN is updated on the primary/provider slapd instance (the *updated* entryCSN is in the longer format), and the syncrepl replication process fails to sync those changes on the non-primary/consumer slapd instances. Doing an ldapdelete of the entry in question (on the primary/provider) followed by an ldapadd of the entry on the primary/provider results in the newly created entry being replicated to the two consumer slapd instances.
So it appears that perhaps I need to "fix" all of the entries with the older format of entryCSN. My quick "fix" that I've been doing when this issue comes up has been to:
1. create an ldif of the entry in question 2. delete the entry in question on the primary/provider slapd instance 3. re-create the entry on the primary/provider, using the ldif from step one 4. at this point subsequent changes to the entry are replicated promptly and without issue
I don't love the idea of having to do this ~4k times, but can certainly do that if that's the best way forward. Is there any better/more efficient way to fix these entries with entryCSNs that are in the older format?
Ben
Ben Poliakoff wrote:
On Thu, Jul 16, 2026 at 6:39 AM Howard Chu <hyc@symas.com mailto:hyc@symas.com> wrote:
Ben Poliakoff wrote: > Just noticed another thing when this issue comes up.... the entryCSN for an entry that's not replicating from the primary server to the two consumers has a > slightly different format: > > Primary/provider: > > entryCSN: 20260715174316.844052Z#000000#001#000000 > > Non-primary/consumer: > > entryCSN: 20210115230536Z#000001#00#000000 > entryCSN: 20210115230536Z#000001#00#000000 These are in an obsolete format, last used in OpenLDAP 2.3. No OpenLDAP software has produced them in the last 20 years.That's interesting. In our environment we have a mix of these two formats (I assume the shorter format being the obsolete one). About a fifth of our ~22k entries (4291) have the shorter format of entryCSN.
I see now that I can reliably trigger this issue: by modifying an entry with the shorter (obsolete?) form of entryCSN. When I do that the entryCSN is updated on the primary/provider slapd instance (the *updated* entryCSN is in the longer format), and the syncrepl replication process fails to sync those changes on the non-primary/consumer slapd instances. Doing an ldapdelete of the entry in question (on the primary/provider) followed by an ldapadd of the entry on the primary/provider results in the newly created entry being replicated to the two consumer slapd instances.
So it appears that perhaps I need to "fix" all of the entries with the older format of entryCSN. My quick "fix" that I've been doing when this issue comes up has been to:
1. create an ldif of the entry in question 2. delete the entry in question on the primary/provider slapd instance 3. re-create the entry on the primary/provider, using the ldif from step one 4. at this point subsequent changes to the entry are replicated promptly and without issueI don't love the idea of having to do this ~4k times, but can certainly do that if that's the best way forward. Is there any better/more efficient way to fix these entries with entryCSNs that are in the older format?
Sounds like a bug in the consumer, it should be perfectly able to process the older CSN formats.
If you could supply a small sample DB and config that demonstrates this problem, submit it to the ITS.
On Thu, Jul 16, 2026 at 11:38:41AM -0700, Ben Poliakoff wrote:
So it appears that perhaps I need to "fix" all of the entries with the older format of entryCSN. My quick "fix" that I've been doing when this issue comes up has been to:
- create an ldif of the entry in question
- delete the entry in question on the primary/provider slapd instance
- re-create the entry on the primary/provider, using the ldif from step one
- at this point subsequent changes to the entry are replicated promptly
and without issue
I don't love the idea of having to do this ~4k times, but can certainly do that if that's the best way forward. Is there any better/more efficient way to fix these entries with entryCSNs that are in the older format?
Hi Ben, in order of (personal) preference: - slapcat DB, filter out entryCSNs with that format, slapadd and then use that DB as the seed for the rest of the cluster - do the above but transform the entryCSNs to the current format instead of removing them - your suggestion - downgrade to 2.6.10 and touch (an empty modify is ok) each of those entries, let it replicate then upgrade again
Regards,
Thank you Ondřej,
I did end up filing an ITS about this: https://bugs.openldap.org/show_bug.cgi?id=10541
I see the target milestone is set for the 2.6.14 release, so perhaps simply upgrading to that might resolve my issue with these older format versions of entryCSN....
But just to clarify, if I manually remove the older entryCSNs from the slapcat output, then slapd will generate new ones when I do a slapadd?
Ben
On Mon, Jul 20, 2026 at 5:12 AM Ondřej Kuzník ondra@mistotebe.net wrote:
On Thu, Jul 16, 2026 at 11:38:41AM -0700, Ben Poliakoff wrote:
So it appears that perhaps I need to "fix" all of the entries with the older format of entryCSN. My quick "fix" that I've been doing when this issue comes up has been to:
- create an ldif of the entry in question
- delete the entry in question on the primary/provider slapd instance
- re-create the entry on the primary/provider, using the ldif from step
one
- at this point subsequent changes to the entry are replicated promptly
and without issue
I don't love the idea of having to do this ~4k times, but can certainly
do
that if that's the best way forward. Is there any better/more efficient
way
to fix these entries with entryCSNs that are in the older format?
Hi Ben, in order of (personal) preference:
- slapcat DB, filter out entryCSNs with that format, slapadd and then use that DB as the seed for the rest of the cluster
- do the above but transform the entryCSNs to the current format instead of removing them
- your suggestion
- downgrade to 2.6.10 and touch (an empty modify is ok) each of those entries, let it replicate then upgrade again
Regards,
-- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP
--On Monday, July 20, 2026 2:18 PM -0700 Ben Poliakoff benp@reed.edu wrote:
Thank you Ondřej,
I did end up filing an ITS about this: https://bugs.openldap.org/show_bug.cgi?id=10541
I see the target milestone is set for the 2.6.14 release, so perhaps simply upgrading to that might resolve my issue with these older format versions of entryCSN....
But just to clarify, if I manually remove the older entryCSNs from the slapcat output, then slapd will generate new ones when I do a slapadd?
Yes. I suggest reading the slapadd man page, but you'll likely want to use the flags (-w) and (-S <SID>) while doing so, since you are doing multi-provider replication.
Regards, Quanah
openldap-technical@openldap.org
-
Ben Poliakoff -
Chris Paul -
Erik de Waard -
Howard Chu -
Ondřej Kuzník -
Quanah Gibson-Mount -
Windl, Ulrich