https://bugs.openldap.org/show_bug.cgi?id=10026
Issue ID: 10026 Summary: Refresh handling can skip entries (si_dirty not managed properly) Product: OpenLDAP Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Keywords: needs_review Severity: normal Priority: --- Component: overlays Assignee: bugs@openldap.org Reporter: ondra@mistotebe.net Target Milestone: ---
Take MPR plain syncrepl with 3+ providers.
When a provider's own syncrepl session transitions to persist and a it starts a new parallel session towards another host, that session always has to start as a refresh. If that refresh serves entries to us, our handling of si_dirty is not consistent: - if the existing persist session serves some of these entries to us, we can "forget" to pass the others to a newly connected consumer - same if the refresh is abandoned and we start refreshing from a different provider that might be behind what we were being served (again our consumers could suffer) - if we restart, si_dirty is forgotten and our consumers suffer even worse
We might need to be told (at the beginning of the refresh?) what the end state we're going for is, so we can keep si_dirty on until then. And somehow persist that knowledge in the DB...
https://bugs.openldap.org/show_bug.cgi?id=10026
--- Comment #1 from Ondřej Kuzník ondra@mistotebe.net --- Created attachment 954 --> https://bugs.openldap.org/attachment.cgi?id=954&action=edit Reproducer
A Python (3.10+) reproducer, requires OpenLDAP master (or 2.7) source with mod-harness compiled in (--enable-harness).
Extract the archive, create a virtual environment, install requirements, then run: TOP_SRCDIR=<path to openldap source> ./tester.py
The sequence to get started quickly (adjust "../openldap" accordingly): ``` tar xf test050.tgz cd test050 virtualenv ./venv source ./venv/bin/activate pip install -r ./requirements.txt TOP_SRCDIR=../openldap ./tester.py ```
https://bugs.openldap.org/show_bug.cgi?id=10026
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Keywords|needs_review | Target Milestone|--- |2.5.15
https://bugs.openldap.org/show_bug.cgi?id=10026
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@openldap.org |hyc@openldap.org
https://bugs.openldap.org/show_bug.cgi?id=10026
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|2.5.15 |2.5.16
https://bugs.openldap.org/show_bug.cgi?id=10026
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|2.5.16 |2.5.17
https://bugs.openldap.org/show_bug.cgi?id=10026
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|2.5.17 |2.5.18
https://bugs.openldap.org/show_bug.cgi?id=10026
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|2.5.18 |2.5.19
https://bugs.openldap.org/show_bug.cgi?id=10026
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|2.5.19 |2.5.20
https://bugs.openldap.org/show_bug.cgi?id=10026
Ondřej Kuzník ondra@mistotebe.net changed:
What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|2.5.20 |2.6.11
https://bugs.openldap.org/show_bug.cgi?id=10026
--- Comment #2 from Howard Chu hyc@openldap.org --- requirements.txt is incomplete.
I had to pip install ldif and pytest, but was still unable to run this:
$ pip install pytest Collecting pytest Downloading pytest-9.0.1-py3-none-any.whl.metadata (7.6 kB) Collecting iniconfig>=1.0.1 (from pytest) Downloading iniconfig-2.3.0-py3-none-any.whl.metadata (2.5 kB) Collecting packaging>=22 (from pytest) Downloading packaging-25.0-py3-none-any.whl.metadata (3.3 kB) Collecting pluggy<2,>=1.5 (from pytest) Downloading pluggy-1.6.0-py3-none-any.whl.metadata (4.8 kB) Collecting pygments>=2.7.2 (from pytest) Downloading pygments-2.19.2-py3-none-any.whl.metadata (2.5 kB) Downloading pytest-9.0.1-py3-none-any.whl (373 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 373.7/373.7 kB 6.8 MB/s eta 0:00:00 Downloading iniconfig-2.3.0-py3-none-any.whl (7.5 kB) Downloading packaging-25.0-py3-none-any.whl (66 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.5/66.5 kB 14.9 MB/s eta 0:00:00 Downloading pluggy-1.6.0-py3-none-any.whl (20 kB) Downloading pygments-2.19.2-py3-none-any.whl (1.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 33.8 MB/s eta 0:00:00 Installing collected packages: pygments, pluggy, packaging, iniconfig, pytest Successfully installed iniconfig-2.3.0 packaging-25.0 pluggy-1.6.0 pygments-2.19.2 pytest-9.0.1 (venv) hyc@viola:~/OD/hobj/tests/test050$ TOP_SRCDIR=../../../head ./tester.py Traceback (most recent call last): File "/home/hyc/OD/hobj/tests/test050/./tester.py", line 649, in <module> asyncio.run(main()) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/home/hyc/OD/hobj/tests/test050/./tester.py", line 507, in main server = Server(test_ldif) ^^^^^^^^^^^^^^^^^ File "/home/hyc/OD/hobj/tests/test050/./tester.py", line 147, in __init__ self.populate_db() File "/home/hyc/OD/hobj/tests/test050/./tester.py", line 151, in populate_db db = ldif.LDIFRecordList(open(self.dbname)) ^^^^^^^^^^^^^^^^^^^ AttributeError: module 'ldif' has no attribute 'LDIFRecordList'
https://bugs.openldap.org/show_bug.cgi?id=10026
Ondřej Kuzník ondra@mistotebe.net changed:
What |Removed |Added ---------------------------------------------------------------------------- Attachment #954 is|0 |1 obsolete| |
https://bugs.openldap.org/show_bug.cgi?id=10026
--- Comment #3 from Ondřej Kuzník ondra@mistotebe.net --- Created attachment 1093 --> https://bugs.openldap.org/attachment.cgi?id=1093&action=edit Reproducer
Updated reproducer
https://bugs.openldap.org/show_bug.cgi?id=10026
--- Comment #4 from Ondřej Kuzník ondra@mistotebe.net --- Looks like I forgot to mention some libraries that naturally exist on my system. so updated it now. The way to run it is still the same: - have OpenLDAP compiled somewhere (with `--enable-mdb=yes --enable-syncprov=yes --enable-harness`), below assumes `../openldap` as the path, adjust to taste - download the archive, then: ``` tar xf test050.tgz cd test050 virtualenv ./venv source ./venv/bin/activate pip install -r ./requirements.txt TOP_SRCDIR=../openldap ./tester.py ```
If the issue is fixed it will pass, if the issue still exists it probably won't (there are multiple ways to reproduce it, it only tries to trigger one of them).
https://bugs.openldap.org/show_bug.cgi?id=10026
--- Comment #5 from Howard Chu hyc@openldap.org --- During `pip install...`
building '_ldap' extension creating build/temp.linux-x86_64-cpython-312/Modules x86_64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -fPIC -DHAVE_SASL -DHAVE_TLS -DLDAPMODULE_VERSION=3.4.5 "-DLDAPMODULE_AUTHOR=python-ldap project" "-DLDAPMODULE_LICENSE=Python style" -IModules -I/home/hyc/OD/hobj/tests/test050/venv/include -I/usr/include/python3.12 -c Modules/LDAPObject.c -o build/temp.linux-x86_64-cpython-312/Modules/LDAPObject.o In file included from Modules/LDAPObject.c:3: Modules/common.h:20:2: error: #error Current python-ldap requires OpenLDAP 2.4.x 20 | #error Current python-ldap requires OpenLDAP 2.4.x | ^~~~~ error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1 [end of output]
note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for python-ldap Successfully built syncmonitor pldap Failed to build python-ldap ERROR: Could not build wheels for python-ldap, which is required to install pyproject.toml-based projects
https://bugs.openldap.org/show_bug.cgi?id=10026
--- Comment #6 from Ondřej Kuzník ondra@mistotebe.net --- On Fri, Nov 21, 2025 at 12:33:57PM +0000, openldap-its@openldap.org wrote:
During `pip install...`
building '_ldap' extension creating build/temp.linux-x86_64-cpython-312/Modules x86_64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2-Wall -fPIC -DHAVE_SASL -DHAVE_TLS -DLDAPMODULE_VERSION=3.4.5 "-DLDAPMODULE_AUTHOR=python-ldap project" "-DLDAPMODULE_LICENSE=Python style" -IModules -I/home/hyc/OD/hobj/tests/test050/venv/include -I/usr/include/python3.12 -c Modules/LDAPObject.c -o build/temp.linux-x86_64-cpython-312/Modules/LDAPObject.o In file included from Modules/LDAPObject.c:3: Modules/common.h:20:2: error: #error Current python-ldap requires OpenLDAP 2.4.x 20 | #error Current python-ldap requires OpenLDAP 2.4.x | ^~~~~ error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1 [end of output]
note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for python-ldap Successfully built syncmonitor pldap Failed to build python-ldap ERROR: Could not build wheels for python-ldap, which is required to install pyproject.toml-based projects
If you `make install`-ed libldap headers from master, they will report a bogus version. Just let the module use libldap development headers from your distro and you'll be fine (or patch your OpenLDAP tree to report something useful). This bug isn't related to libldap anyway.
https://bugs.openldap.org/show_bug.cgi?id=10026
--- Comment #7 from Howard Chu hyc@openldap.org --- Now there's nothing happening, no output anywhere.
### (venv) viola:~/OD/o10026/tests/test050> ./tester.py ###
2421890 pts/8 Sl+ 0:00 python3 ./tester.py 2421907 pts/8 Sl+ 0:00 /home/hyc/OD/o10026/servers/slapd/.libs/slapd -d -1 -f /tmp/tmpxvsn9rm6/port2389/slapd.conf 2421938 pts/8 Sl+ 0:00 /home/hyc/OD/o10026/servers/slapd/.libs/slapd -d -1 -f /tmp/tmpxvsn9rm6/port3389/slapd.conf 2422044 pts/13 R+ 0:00 ps g
https://bugs.openldap.org/show_bug.cgi?id=10026
--- Comment #8 from Howard Chu hyc@openldap.org --- Also your original description says this problem occurs with 3+ providers and the test case only uses 2 slapd instances. Is that intentional?
https://bugs.openldap.org/show_bug.cgi?id=10026
--- Comment #9 from Ondřej Kuzník ondra@mistotebe.net --- On Tue, Nov 25, 2025 at 04:24:17PM +0000, openldap-its@openldap.org wrote:
Also your original description says this problem occurs with 3+ providers and the test case only uses 2 slapd instances. Is that intentional?
There is another stub provider that is embedded in the Python script that makes sure messages are received/routed in the correct order to massage the victim syncprov into a state when it starts to lie.
-
openldap-its＠openldap.org