openldap-bugs December 2020

openldap-bugs@openldap.org

3 participants
90 discussions

[Issue 9316] New: performance issue when writing a high number of large objects
by openldap-its＠openldap.org 15 Apr '26

15 Apr '26

https://bugs.openldap.org/show_bug.cgi?id=9316 Issue ID: 9316 Summary: performance issue when writing a high number of large objects Product: LMDB Version: 0.9.24 Hardware: x86_64 OS: Linux Status: UNCONFIRMED Severity: normal Priority: --- Component: liblmdb Assignee: bugs(a)openldap.org Reporter: JGabler(a)univa.com Target Milestone: --- Created attachment 755 --> https://bugs.openldap.org/attachment.cgi?id=755&action=edit lmdb performance test reproducing the issue When writing a high number of big objects we see an extreme variation in performance from very fast to extremely slow. In the test scenario we write 10 chunks of 10.000 "jobs" (some 10kB) and their corresponding "job script" (some 40kB), 200.000 objects in total. Then delete all objects. We do 10 iterations of this scenario. When running this scenario as part of Univa Grid Engine with LMDB as database backend we get the following performance values (rows are the iteration, columns the chunk of jobs): Iteration 0 1 2 3 4 5 6 7 8 9 0 21.525 21.250 21.574 21.722 22.693 21.992 22.438 22.650 21.972 22.017 1 22.262 21.656 22.339 22.914 21.549 24.906 23.862 1531.189 1695.041 1491.255 2 36.071 21.619 22.074 22.927 23.455 27.239 22.640 22.802 633.956 1882.008 3 52.163 21.651 21.571 22.686 22.727 22.024 40.980 22.156 22.429 595.362 4 64.977 21.511 22.519 22.148 22.354 23.292 57.740 20.835 37.680 250.594 5 54.724 21.074 21.200 23.744 22.109 21.351 62.225 21.447 91.292 375.260 6 49.065 21.573 22.309 26.084 21.226 21.248 68.580 22.531 59.338 249.936 7 44.666 21.830 21.009 28.760 21.533 21.611 72.291 23.144 86.281 118.326 8 35.486 21.720 21.840 24.729 22.045 20.877 76.473 21.193 120.387 136.836 9 41.159 23.365 21.721 23.024 21.835 20.972 77.409 21.784 193.885 306.158 So usually writing of 10.000 "jobs"+"job_script" takes some 22 seconds but after some time performance massively breaks in. With other database backends we do not see this behaviour, see the following performance data of the same test done with PostgreSQL backend which is slower (as expected going over the network) but provides constant throughput: Iteration 0 1 2 3 4 5 6 7 8 9 0 36.937 37.110 36.952 37.279 37.580 37.364 37.950 37.390 37.682 37.439 1 37.464 38.110 37.679 38.366 37.576 37.624 37.476 37.412 37.265 37.727 2 36.394 37.635 37.347 37.603 37.402 37.515 37.802 37.898 37.355 37.939 3 37.213 37.539 36.771 37.706 37.055 37.780 37.283 37.488 36.955 37.460 4 36.554 37.557 37.368 37.960 37.070 37.892 37.459 37.857 37.228 37.833 5 37.047 38.164 37.167 37.885 37.268 37.676 37.355 37.572 37.347 37.569 6 37.118 37.735 36.857 37.602 36.717 37.716 37.444 37.685 37.085 38.151 7 36.787 37.647 36.844 37.601 36.934 37.440 37.632 37.291 37.174 37.926 8 36.884 37.560 37.117 37.239 37.034 37.748 37.289 37.635 36.822 37.693 9 37.178 37.496 36.849 37.799 37.289 37.644 37.461 37.622 37.022 37.670 We can reproduce the issue with a small C program (see attachment) which does essentially the same database operations as our database layer in Univa Grid Engine but depends only on liblmdb. It simulates the scenario described above and gives us the following performance data showing the extreme performance variation: Iteration 0 1 2 3 4 5 6 7 8 9 0 0.686 0.625 0.660 0.637 0.631 0.741 0.757 0.658 0.651 0.614 1 0.705 0.838 0.690 0.772 0.663 3.248 0.605 542.762 1114.374 898.477 2 13.336 1.299 0.659 0.637 0.626 0.712 11.172 0.663 29.833 1161.884 3 26.774 0.647 0.607 0.586 0.583 0.639 24.893 0.629 3.837 423.248 4 32.802 0.629 0.616 0.560 0.550 0.605 31.133 0.625 6.606 195.150 5 34.819 0.623 0.628 0.582 0.564 0.609 32.275 0.607 7.599 134.106 6 26.319 0.622 0.582 0.548 0.551 0.590 28.536 0.611 36.429 160.781 7 21.878 0.814 0.668 0.736 0.614 0.543 24.355 0.626 36.583 148.337 8 4.129 0.654 5.674 0.596 0.566 0.554 7.158 0.633 0.599 48.799 9 30.278 0.608 0.608 0.560 0.549 0.587 29.253 0.606 9.593 128.339 It can be compiled on Linux 64bit with gcc -I <path to lmdb>/include -L <path to lmdb>/lib -o test_lmdb_perf test_lmdb_perf.c -llmdb To run the given scenario call it with the following parameters: ./test_lmdb_perf <path to database directory> 10 10 10000 We built and ran it on - CentOS Linux release 7.7.1908 (Core) - Linux biber 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux - it was built with gcc (GCC) 7.2.1 20170829 (Red Hat 7.2.1-1) from devtoolset-7 -- You are receiving this mail because: You are on the CC list for the issue.

1 2

[Issue 9278] New: liblmdb: robust mutexes should not be unmapped
by openldap-its＠openldap.org 15 Apr '26

15 Apr '26

https://bugs.openldap.org/show_bug.cgi?id=9278 Issue ID: 9278 Summary: liblmdb: robust mutexes should not be unmapped Product: LMDB Version: unspecified Hardware: All OS: FreeBSD Status: UNCONFIRMED Severity: normal Priority: --- Component: liblmdb Assignee: bugs(a)openldap.org Reporter: delphij(a)freebsd.org Target Milestone: --- Created attachment 736 --> https://bugs.openldap.org/attachment.cgi?id=736&action=edit A possible workaround We recently noticed that lmdb would have the memory region containing the robust mutex unmapped on mdb_env_close0(): munmap((void *)env->me_txns, (env->me_maxreaders-1)*sizeof(MDB_reader)+sizeof(MDB_txninfo)); Note that if this is the last unmap for a robust mutex, the FreeBSD implementation would garbage-collect the mutex, making it no longer visible to other processes. As the result, a second instance of the attached test.c (from https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244493 with minor changes) would trigger the assertion at mdb_txn_begin() because the acquisition of the mutex would return 22 (EINVAL), because the mutex appeared to be a robust mutex, but was invalid. The attached lmdb.diff is a possible workaround for this (it would skip unmapping when setting up the robust mutex for the first time). -- You are receiving this mail because: You are on the CC list for the issue.

1 21

[Issue 9378] New: Crash in mdb_put() / mdb_page_dirty()
by openldap-its＠openldap.org 15 Apr '26

15 Apr '26

https://bugs.openldap.org/show_bug.cgi?id=9378 Issue ID: 9378 Summary: Crash in mdb_put() / mdb_page_dirty() Product: LMDB Version: 0.9.26 Hardware: All OS: Linux Status: UNCONFIRMED Severity: normal Priority: --- Component: liblmdb Assignee: bugs(a)openldap.org Reporter: nate(a)kde.org Target Milestone: --- The KDE Baloo file indexer uses lmdb as its database (source code available at https://invent.kde.org/frameworks/baloo). Our most common crash, with over 100 duplicate bug reports, is in lmdb. Here's the bug report tracking it: https://bugs.kde.org/show_bug.cgi?id=389848. The version of lmdb does not seem to matter much. We have bug reports from Arch users with lmdb 0.9.26 as well as bug reports from people using many earlier versions. Here's an example backtrace, taken from https://bugs.kde.org/show_bug.cgi?id=426195: #6 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #7 0x00007f3c0bbb9859 in __GI_abort () at abort.c:79 #8 0x00007f3c0b23ba83 in mdb_assert_fail (env=0x55e2ad710600, expr_txt=expr_txt@entry=0x7f3c0b23e02f "rc == 0", func=func@entry=0x7f3c0b23e978 <__func__.7221> "mdb_page_dirty", line=line@entry=2127, file=0x7f3c0b23e010 "mdb.c") at mdb.c:1542 #9 0x00007f3c0b2306d5 in mdb_page_dirty (mp=<optimized out>, txn=0x55e2ad7109f0) at mdb.c:2114 #10 mdb_page_dirty (txn=0x55e2ad7109f0, mp=<optimized out>) at mdb.c:2114 #11 0x00007f3c0b231966 in mdb_page_alloc (num=num@entry=1, mp=mp@entry=0x7f3c0727aee8, mc=<optimized out>) at mdb.c:2308 #12 0x00007f3c0b231ba3 in mdb_page_touch (mc=mc@entry=0x7f3c0727b420) at mdb.c:2495 #13 0x00007f3c0b2337c7 in mdb_cursor_touch (mc=mc@entry=0x7f3c0727b420) at mdb.c:6523 #14 0x00007f3c0b2368f9 in mdb_cursor_put (mc=mc@entry=0x7f3c0727b420, key=key@entry=0x7f3c0727b810, data=data@entry=0x7f3c0727b820, flags=flags@entry=0) at mdb.c:6657 #15 0x00007f3c0b23976b in mdb_put (txn=0x55e2ad7109f0, dbi=5, key=key@entry=0x7f3c0727b810, data=data@entry=0x7f3c0727b820, flags=flags@entry=0) at mdb.c:9022 #16 0x00007f3c0c7124c5 in Baloo::DocumentDB::put (this=this@entry=0x7f3c0727b960, docId=<optimized out>, docId@entry=27041423333263366, list=...) at ./src/engine/documentdb.cpp:79 #17 0x00007f3c0c743da7 in Baloo::WriteTransaction::replaceDocument (this=0x55e2ad7ea340, doc=..., operations=operations@entry=...) at ./src/engine/writetransaction.cpp:232 #18 0x00007f3c0c736b16 in Baloo::Transaction::replaceDocument (this=this@entry=0x7f3c0727bc10, doc=..., operations=operations@entry=...) at ./src/engine/transaction.cpp:295 #19 0x000055e2ac5d6cbc in Baloo::UnindexedFileIndexer::run (this=0x55e2ad79ca20) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qrefcount.h:60 #20 0x00007f3c0c177f82 in QThreadPoolThread::run (this=0x55e2ad717f20) at thread/qthreadpool.cpp:99 #21 0x00007f3c0c1749d2 in QThreadPrivate::start (arg=0x55e2ad717f20) at thread/qthread_unix.cpp:361 #22 0x00007f3c0b29d609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #23 0x00007f3c0bcb6103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 -- You are receiving this mail because: You are on the CC list for the issue.

1 31

[Issue 9431] New: back-mdb: Always have an equality index for objectClass
by openldap-its＠openldap.org 10 Apr '26

10 Apr '26

https://bugs.openldap.org/show_bug.cgi?id=9431 Issue ID: 9431 Summary: back-mdb: Always have an equality index for objectClass Product: OpenLDAP Version: 2.5 Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: backends Assignee: bugs(a)openldap.org Reporter: quanah(a)openldap.org Target Milestone: --- Data storage backends require an equality index on objectClass to function correctly. As this is a hard requirement it should be automatic with back-mdb. Why this wasn't done in the past with other backends isn't exactly clear, it may have been due to their requirements to have additional cache layers. That however is not necessary with back-mdb. -- You are receiving this mail because: You are on the CC list for the issue.

1 9

[Bug 9204] New: slapo-constraint allows anyone to apply Relax control
by openldap-its＠openldap.org 09 Apr '26

09 Apr '26

https://bugs.openldap.org/show_bug.cgi?id=9204 Bug ID: 9204 Summary: slapo-constraint allows anyone to apply Relax control Product: OpenLDAP Version: 2.4.49 Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: overlays Assignee: bugs(a)openldap.org Reporter: ryan(a)openldap.org Target Milestone: --- slapo-constraint doesn't limit who can use the Relax control, beyond the global limits applied by slapd. In practice, for many modifications this means any configured constraints are advisory only. In my opinion this should be considered a bug, in design if not implementation. I expect many admins would not read the man page closely enough to realize the behaviour does technically adhere to the letter of what's written there. Either slapd should require manage privileges for the Relax control globally, or slapo-constraint should perform a check for manage privilege itself, like slapo-unique does. Quoting ando in https://bugs.openldap.org/show_bug.cgi?id=5705#c4: > Well, a user with "manage" privileges on related data could bypass > constraints enforced by slapo-constraint(5) by using the "relax" > control. The rationale is that a user with manage privileges could be > able to repair an entry that needs to violate a constraint for good > reasons. Note that the user: > > - must have enough privileges to do it (manage) > > - must inform the DSA that intends to violate the constraint (by using > the control) but such privileges are currently not being required. -- You are receiving this mail because: You are on the CC list for the bug.

1 14

[Bug 9225] New: back-mdb: Add support for PREPARE/2-phase commit
by openldap-its＠openldap.org 08 Apr '26

08 Apr '26

https://bugs.openldap.org/show_bug.cgi?id=9225 Bug ID: 9225 Summary: back-mdb: Add support for PREPARE/2-phase commit Product: OpenLDAP Version: 2.4.50 Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: backends Assignee: bugs(a)openldap.org Reporter: quanah(a)openldap.org Target Milestone: --- Add support for PREPARE/2-phase commit in back-mdb -- You are receiving this mail because: You are on the CC list for the bug.

1 12

[Issue 9343] New: Expand ppolicy policy configuration to allow URL filter
by openldap-its＠openldap.org 07 Apr '26

07 Apr '26

https://bugs.openldap.org/show_bug.cgi?id=9343 Issue ID: 9343 Summary: Expand ppolicy policy configuration to allow URL filter Product: OpenLDAP Version: 2.5 Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: overlays Assignee: bugs(a)openldap.org Reporter: quanah(a)openldap.org Target Milestone: --- Currently, ppolicy only supports a single global default policy, and past that any policies must be manually added to a given user entry if they are supposed to have something other than the default policy. Also, some sites want no default policy, and only a specific subset to have a policy applied to them. For both of these cases, it would be helpful if it were possible to configure a policy to apply to a set of users via a URL similar to the way we handle creating groups of users in dynlist -- You are receiving this mail because: You are on the CC list for the issue.

1 18

[Issue 9367] New: back-mdb: encryption support
by openldap-its＠openldap.org 07 Apr '26

07 Apr '26

https://bugs.openldap.org/show_bug.cgi?id=9367 Issue ID: 9367 Summary: back-mdb: encryption support Product: OpenLDAP Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: backends Assignee: bugs(a)openldap.org Reporter: quanah(a)openldap.org Target Milestone: --- Need to add encryption support to the back-mdb backend, depends on issue#9364 -- You are receiving this mail because: You are on the CC list for the issue.

1 9

[Issue 9398] New: Stale accesslog cookie due to unclean shutdown
by openldap-its＠openldap.org 07 Apr '26

07 Apr '26

https://bugs.openldap.org/show_bug.cgi?id=9398 Issue ID: 9398 Summary: Stale accesslog cookie due to unclean shutdown Product: OpenLDAP Version: 2.4.56 Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: slapd Assignee: bugs(a)openldap.org Reporter: quanah(a)openldap.org Target Milestone: --- If slapd terminates uncleanly, a checkpoint will be lost on the accesslog db. Depending on the syncprov overlay checkpoint settings (usually no checkpointing is enabled on the accesslog db) this can cause the system to refuse engage in replication at startup. -- You are receiving this mail because: You are on the CC list for the issue.

1 10

[Bug 9244] New: API calls blocking after async connect
by openldap-its＠openldap.org 02 Mar '26

02 Mar '26

https://bugs.openldap.org/show_bug.cgi?id=9244 Bug ID: 9244 Summary: API calls blocking after async connect Product: OpenLDAP Version: 2.4.49 Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: libraries Assignee: bugs(a)openldap.org Reporter: ryan(a)openldap.org Target Milestone: --- Created attachment 721 --> https://bugs.openldap.org/attachment.cgi?id=721&action=edit async connect test without TLS My understanding of LDAP_OPT_CONNECT_ASYNC is that the attached program should not block. If the connection does not establish fast enough, the bind call is supposed to return LDAP_X_CONNECTING. (At least that's how I understand it, based on the original behaviour (circa 2.4.23 up to 2.4.40) as well as the bind loop in back-meta. On the other hand, the man page does "Subsequent calls to library routines will poll for completion of the connect before performing further operations" which might be interpreted as meaning they would block...) In current releases it does block, as demonstrated by strace on Linux (latency added using 'tc qdisc'): [...] connect(3, {sa_family=AF_INET, sin_port=htons(389), sin_addr=inet_addr("192.168.1.204")}, 16) = -1 EINPROGRESS (Operation now in progress) write(3, "0\f\2\1\1`\7\2\1\3\4\0\200\0", 14) = -1 EAGAIN (Resource temporarily unavailable) poll([{fd=3, events=POLLOUT|POLLERR|POLLHUP}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}]) write(3, "0\f\2\1\1`\7\2\1\3\4\0\200\0", 14) = 14 poll([{fd=3, events=POLLIN|POLLPRI}], 1, -1) = 1 ([{fd=3, revents=POLLIN}]) read(3, "0\f\2\1\1a\7\n", 8) = 8 read(3, "\1\0\4\0\4\0", 6) = 6 write(2, "OK: ldap_simple_bind_returned 0 "..., 42OK: ldap_simple_bind_returned 0 (Success) ) = 42 [...] As discussed in IRC, I believe I bisected this down to commit ae6347bac, from bug 8022. The reasoning is sound, but ldap_int_open_connection does not actually return -2, only -1 or 0. The patch is simple enough, but I'm also looking at some later commits that were probably done to work around this, and might not be needed now (bug 8957, bug 8968, bug 8980). Also need to test all setups thoroughly (ldap, ldaps, STARTTLS, not to mention back-meta/asyncmeta). I also notice that LDAP_OPT_CONNECT_ASYNC is not effective unless LDAP_OPT_NETWORK_TIMEOUT is also set. It might be intentional, but the man page doesn't mention this specifically, and I don't see why it would be necessary... -- You are receiving this mail because: You are on the CC list for the bug.

1 17

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs December 2020