openldap-bugs January 2019

openldap-bugs@openldap.org

21 participants
77 discussions

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 24 Jan '19

24 Jan '19

On 22.01.2019 19:55, Ondřej Kuzník wrote: > The problem is not that it's still scheduled when the pause is > requested, scheduled tasks don't prevent a pause (and I think some > of them need to be a bit more aware of that fact), but that it's been > picked up by a worker thread and is running while cn=config waits for > them to pause. _And_ that a pause request is all-or-nothing. This demo code of graduated wait-for-pause works for me. For RE24, it doesn't combine easily with Git master's multiple tpool queues:-( ftp://ftp.openldap.org/incoming/Hallvard-Furuseth-190124.patch.gz But I'm not volunteering to write one for master anytime soon, unless someone has less complicated ideas for that than I do. A real version looks like more work than I expected anyway. > What I was suggesting was that it could check every so often whether it > might be holding up a pause and join it just like some other tasks do. > But that's only a good idea if it's able to handle the changes that > happen during the pause (indexes reconfigured again, the database going > away, ...) as well as being able to release any locks temporarily. Yes. I should have answered that, but I still cannot decide what I think of it. I see no locks to worry about in that code, but the rest does make me nervous. Possibly we can find some times where we can pause safely, but not a general case. For example, in the case of back-mdb we could register scheduled config operations the database itself, in a "(TODO)" LMDB DB. Doesn't help with schema changes, but does help renumbering slapd DBs. -- Hallvard

1 0

Re: (ITS#8959) duplicate symbols
by hyc＠symas.com 24 Jan '19

24 Jan '19

ruga(a)protonmail.com wrote: > Full_Name: Rupert > Version: 2.4.47 > OS: Darwin alice 17.7.0 Darwin Kernel Version 17.7.0: Fri Nov 2 20:43:16 PDT 2018; root:xnu-4570.71.17~1/RELEASE_X86_64 x86_64 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (5.170.120.126) > > > NOTE > >> make > > [...] > duplicate symbol _ber_int_options in: > /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) > ../../libraries/liblber/.libs/liblber.a(options.o) > duplicate symbol _ber_set_option in: > /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) > ../../libraries/liblber/.libs/liblber.a(options.o) > duplicate symbol _ber_get_option in: > /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) > ../../libraries/liblber/.libs/liblber.a(options.o) > ld: 3 duplicate symbols for architecture x86_64 > clang-7: error: linker command failed with exit code 1 (use -v to see > invocation) > make[2]: *** [apitest] Error 1 > make[2]: *** Waiting for unfinished jobs.... > duplicate symbol _ber_int_options in: > /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) > ../../libraries/liblber/.libs/liblber.a(options.o) > duplicate symbol _ber_set_option in: > /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) > ../../libraries/liblber/.libs/liblber.a(options.o) > duplicate symbol _ber_get_option in: > /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) > ../../libraries/liblber/.libs/liblber.a(options.o) > ld: 3 duplicate symbols for architecture x86_64 > clang-7: error: linker command failed with exit code 1 (use -v to see > invocation) > make[2]: *** [dntest] Error 1 > make[1]: *** [all-common] Error 1 > make: *** [all-common] Error 1 > > > COMMENT > > Silly bug, tripping the compiler. Sounds like a linker bug or some other problem in your build environment. There are no duplicate symbols anywhere in this source tree. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8959) duplicate symbols
by quanah＠symas.com 24 Jan '19

24 Jan '19

--On Thursday, January 24, 2019 6:26 PM +0000 ruga(a)protonmail.com wrote: > Full_Name: Rupert > Version: 2.4.47 > OS: Darwin alice 17.7.0 Darwin Kernel Version 17.7.0: Fri Nov 2 20:43:16 > PDT 2018; root:xnu-4570.71.17~1/RELEASE_X86_64 x86_64 URL: > ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (5.170.120.126) Builds fine for me using clang compiler on FreeBSD. Also appears to work fine for Apple, since it ships the OpenLDAP libraries. You'll need to provide more details as to exactly what you're doing to encounter this issue. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#8960) code quality
by quanah＠symas.com 24 Jan '19

24 Jan '19

--On Thursday, January 24, 2019 6:47 PM +0000 ruga(a)protonmail.com wrote: > Full_Name: Rupert > Version: - > OS: - > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (5.170.120.126) > > > The source code of openldap went through lumnify's meat grinder, > receiving a score of 1 (fail) out of 5. Score 1 means "A lot of patch Sounds like lumnify is a poor quality product. You may wish to use something better written? In any case, the ITS system is for bug reports, this is clearly not a bug report. It will be closed. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#8960) code quality
by ruga＠protonmail.com 24 Jan '19

24 Jan '19

Full_Name: Rupert Version: - OS: - URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (5.170.120.126) The source code of openldap went through lumnify's meat grinder, receiving a score of 1 (fail) out of 5. Score 1 means "A lot of patch work and bandages, project is most likely in initial stage of development. Most of source code is highly unmaintainable and feels a bit chaotic. It is good starting point, but plenty of work ahead. Don't give up yet!". I see old bugs pending. Is the project active? [1] https://sysadmin.libhunt.com/openldap-alternatives [2] https://lumnify.com/grades/?rel=libhunt

1 0

(ITS#8959) duplicate symbols
by ruga＠protonmail.com 24 Jan '19

24 Jan '19

Full_Name: Rupert Version: 2.4.47 OS: Darwin alice 17.7.0 Darwin Kernel Version 17.7.0: Fri Nov 2 20:43:16 PDT 2018; root:xnu-4570.71.17~1/RELEASE_X86_64 x86_64 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (5.170.120.126) NOTE >make [...] duplicate symbol _ber_int_options in: /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) ../../libraries/liblber/.libs/liblber.a(options.o) duplicate symbol _ber_set_option in: /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) ../../libraries/liblber/.libs/liblber.a(options.o) duplicate symbol _ber_get_option in: /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) ../../libraries/liblber/.libs/liblber.a(options.o) ld: 3 duplicate symbols for architecture x86_64 clang-7: error: linker command failed with exit code 1 (use -v to see invocation) make[2]: *** [apitest] Error 1 make[2]: *** Waiting for unfinished jobs.... duplicate symbol _ber_int_options in: /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) ../../libraries/liblber/.libs/liblber.a(options.o) duplicate symbol _ber_set_option in: /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) ../../libraries/liblber/.libs/liblber.a(options.o) duplicate symbol _ber_get_option in: /opt/src/openldap-2.4.47/libraries/liblber/.libs/liblber.a(options.o) ../../libraries/liblber/.libs/liblber.a(options.o) ld: 3 duplicate symbols for architecture x86_64 clang-7: error: linker command failed with exit code 1 (use -v to see invocation) make[2]: *** [dntest] Error 1 make[1]: *** [all-common] Error 1 make: *** [all-common] Error 1 COMMENT Silly bug, tripping the compiler.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by ondra＠mistotebe.net 22 Jan '19

22 Jan '19

On Tue, Jan 22, 2019 at 05:49:12PM +0000, h.b.furuseth(a)usit.uio.no wrote: > On 1/21/19 7:49 PM, Ondřej Kuzník wrote: >> Except there are no locks as you know being the author of parts of >> code that deals with what I'm about to outline anyway: >> >> Whenever a cn=config op is about to be processed, a pause is requested. >> That tells worker threads to stop picking up new work and waits until >> they're all quiet. The indexing task is run by one of these worker >> threads. > > Duuh, right. I got stuck looking for what's special about the > indexing task and couldn't find it:-( I need to make it special. > > So, let tasks declare their expected speed until finish or > between pausechecks. At FAST=1 (default) and SLOW=0. > A pause only stops tasks with speed < ltp_pause. > In thread_pool_pause(), replace the WANT_PAUSE stage with > > while (++ltp_pause <= max speed) { > wait until no more tasks with speed < ltp_pause; > } > > Then fast tasks should breeze past slow ones when preparing > to pause. Until all threads have slow tasks, anyway. > > To mitigate that, we'd need to predeclare the speed when > submitting a task, and limit the number of parallel slow > tasks. pool_submit() could stash the rest in a "slow queue" > instead of submitting. But I don't want to go there yet. The problem is not that it's still scheduled when the pause is requested, scheduled tasks don't prevent a pause (and I think some of them need to be a bit more aware of that fact), but that it's been picked up by a worker thread and is running while cn=config waits for them to pause. What I was suggesting was that it could check every so often whether it might be holding up a pause and join it just like some other tasks do. But that's only a good idea if it's able to handle the changes that happen during the pause (indexes reconfigured again, the database going away, ...) as well as being able to release any locks temporarily. -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 22 Jan '19

22 Jan '19

On 1/21/19 7:49 PM, Ondřej Kuzník wrote: > Except there are no locks as you know being the author of parts of > code that deals with what I'm about to outline anyway: > > Whenever a cn=config op is about to be processed, a pause is requested. > That tells worker threads to stop picking up new work and waits until > they're all quiet. The indexing task is run by one of these worker > threads. Duuh, right. I got stuck looking for what's special about the indexing task and couldn't find it:-( I need to make it special. So, let tasks declare their expected speed until finish or between pausechecks. At FAST=1 (default) and SLOW=0. A pause only stops tasks with speed < ltp_pause. In thread_pool_pause(), replace the WANT_PAUSE stage with while (++ltp_pause <= max speed) { wait until no more tasks with speed < ltp_pause; } Then fast tasks should breeze past slow ones when preparing to pause. Until all threads have slow tasks, anyway. To mitigate that, we'd need to predeclare the speed when submitting a task, and limit the number of parallel slow tasks. pool_submit() could stash the rest in a "slow queue" instead of submitting. But I don't want to go there yet.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by ondra＠mistotebe.net 21 Jan '19

21 Jan '19

On Mon, Jan 21, 2019 at 03:23:55PM +0000, h.b.furuseth(a)usit.uio.no wrote: > On 20.01.2019 19:52, Hallvard Breien Furuseth wrote: > > Can cn=config have 2 lock levels? That fixes this ITS (my > > ldapwhoami hang) without introducing cn=config failures. > > > > The outer lock serializes cn=config ops including the indexer. > > Other ops do not lock it. The inner lock blocks slapd like > > cn=config does now, and should only be held briefly. Always > > hold the outer lock when taking the inner lock. > > Correction again: while taking and holding the inner lock. > Except non-config ops. Except there are no locks as you know being the author of parts of code that deals with what I'm about to outline anyway: Whenever a cn=config op is about to be processed, a pause is requested. That tells worker threads to stop picking up new work and waits until they're all quiet. The indexing task is run by one of these worker threads. Some long-running processes (syncrepl) will regularly check to see if they should pause as well: https://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=servers/s… Handling things in a similar way might be an option if we could survive anything changing in cn=config while we yield. That might be a big "if" to implement, though. -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 21 Jan '19

21 Jan '19

On 20.01.2019 19:52, Hallvard Breien Furuseth wrote: > Can cn=config have 2 lock levels? That fixes this ITS (my > ldapwhoami hang) without introducing cn=config failures. > > The outer lock serializes cn=config ops including the indexer. > Other ops do not lock it. The inner lock blocks slapd like > cn=config does now, and should only be held briefly. Always > hold the outer lock when taking the inner lock. Correction again: while taking and holding the inner lock. Except non-config ops. > Also, the inner lock could be taken and released several > times during a config op when feasible. That might improve > non-config op latency, for the price of slower config ops.

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs January 2019