openldap-bugs

openldap-bugs@openldap.org

1 participants
21001 discussions

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by ondra＠mistotebe.net 22 Jan '19

22 Jan '19

On Tue, Jan 22, 2019 at 05:49:12PM +0000, h.b.furuseth(a)usit.uio.no wrote: > On 1/21/19 7:49 PM, Ondřej Kuzník wrote: >> Except there are no locks as you know being the author of parts of >> code that deals with what I'm about to outline anyway: >> >> Whenever a cn=config op is about to be processed, a pause is requested. >> That tells worker threads to stop picking up new work and waits until >> they're all quiet. The indexing task is run by one of these worker >> threads. > > Duuh, right. I got stuck looking for what's special about the > indexing task and couldn't find it:-( I need to make it special. > > So, let tasks declare their expected speed until finish or > between pausechecks. At FAST=1 (default) and SLOW=0. > A pause only stops tasks with speed < ltp_pause. > In thread_pool_pause(), replace the WANT_PAUSE stage with > > while (++ltp_pause <= max speed) { > wait until no more tasks with speed < ltp_pause; > } > > Then fast tasks should breeze past slow ones when preparing > to pause. Until all threads have slow tasks, anyway. > > To mitigate that, we'd need to predeclare the speed when > submitting a task, and limit the number of parallel slow > tasks. pool_submit() could stash the rest in a "slow queue" > instead of submitting. But I don't want to go there yet. The problem is not that it's still scheduled when the pause is requested, scheduled tasks don't prevent a pause (and I think some of them need to be a bit more aware of that fact), but that it's been picked up by a worker thread and is running while cn=config waits for them to pause. What I was suggesting was that it could check every so often whether it might be holding up a pause and join it just like some other tasks do. But that's only a good idea if it's able to handle the changes that happen during the pause (indexes reconfigured again, the database going away, ...) as well as being able to release any locks temporarily. -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 22 Jan '19

22 Jan '19

On 1/21/19 7:49 PM, Ondřej Kuzník wrote: > Except there are no locks as you know being the author of parts of > code that deals with what I'm about to outline anyway: > > Whenever a cn=config op is about to be processed, a pause is requested. > That tells worker threads to stop picking up new work and waits until > they're all quiet. The indexing task is run by one of these worker > threads. Duuh, right. I got stuck looking for what's special about the indexing task and couldn't find it:-( I need to make it special. So, let tasks declare their expected speed until finish or between pausechecks. At FAST=1 (default) and SLOW=0. A pause only stops tasks with speed < ltp_pause. In thread_pool_pause(), replace the WANT_PAUSE stage with while (++ltp_pause <= max speed) { wait until no more tasks with speed < ltp_pause; } Then fast tasks should breeze past slow ones when preparing to pause. Until all threads have slow tasks, anyway. To mitigate that, we'd need to predeclare the speed when submitting a task, and limit the number of parallel slow tasks. pool_submit() could stash the rest in a "slow queue" instead of submitting. But I don't want to go there yet.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by ondra＠mistotebe.net 21 Jan '19

21 Jan '19

On Mon, Jan 21, 2019 at 03:23:55PM +0000, h.b.furuseth(a)usit.uio.no wrote: > On 20.01.2019 19:52, Hallvard Breien Furuseth wrote: > > Can cn=config have 2 lock levels? That fixes this ITS (my > > ldapwhoami hang) without introducing cn=config failures. > > > > The outer lock serializes cn=config ops including the indexer. > > Other ops do not lock it. The inner lock blocks slapd like > > cn=config does now, and should only be held briefly. Always > > hold the outer lock when taking the inner lock. > > Correction again: while taking and holding the inner lock. > Except non-config ops. Except there are no locks as you know being the author of parts of code that deals with what I'm about to outline anyway: Whenever a cn=config op is about to be processed, a pause is requested. That tells worker threads to stop picking up new work and waits until they're all quiet. The indexing task is run by one of these worker threads. Some long-running processes (syncrepl) will regularly check to see if they should pause as well: https://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=servers/s… Handling things in a similar way might be an option if we could survive anything changing in cn=config while we yield. That might be a big "if" to implement, though. -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 21 Jan '19

21 Jan '19

On 20.01.2019 19:52, Hallvard Breien Furuseth wrote: > Can cn=config have 2 lock levels? That fixes this ITS (my > ldapwhoami hang) without introducing cn=config failures. > > The outer lock serializes cn=config ops including the indexer. > Other ops do not lock it. The inner lock blocks slapd like > cn=config does now, and should only be held briefly. Always > hold the outer lock when taking the inner lock. Correction again: while taking and holding the inner lock. Except non-config ops. > Also, the inner lock could be taken and released several > times during a config op when feasible. That might improve > non-config op latency, for the price of slower config ops.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 20 Jan '19

20 Jan '19

I wrote: > Limiting the problems beyond that is a bonus. Or maybe not. > A cn=config which fails in some obscure cases is flaky. (...) Correction: Obscure _situations_, not obscure cases. As in, operation X normally succeds, but can fail if you just did the somewhat-related operation Y.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 20 Jan '19

20 Jan '19

Can cn=config have 2 lock levels? That fixes this ITS (my ldapwhoami hang) without introducing cn=config failures. The outer lock serializes cn=config ops including the indexer. Other ops do not lock it. The inner lock blocks slapd like cn=config does now, and should only be held briefly. Always hold the outer lock when taking the inner lock. Also, the inner lock could be taken and released several times during a config op when feasible. That might improve non-config op latency, for the price of slower config ops. On 1/20/19 4:42 PM, Howard Chu wrote: > Hallvard Breien Furuseth wrote: >> On 1/19/19 8:04 PM, Howard Chu wrote: >>> modifications to cn=config require no other threads to be active. Once the background indexer >>> starts running, its activity will of course block further cn=config modifications until it >>> completes. >> >> Nothing "of course" about it. I don't know why running >> the background indexer is considered a change to cn=config. > > It is not. But the indexing rules can be changed by further config mods, and we > don't want the config to change out from under it. OK. I was just parsing your worlds too literally. Also, I'm trying to think like an admin, not an OpenLDAP insider: >> The ldapmodify did the change to cn=config and completed. >> >> I can guess that it might block cn=config simply because that's >> the safe way to have both an indexer and dynamic config. Or >> I can read the code. But "read the code" != "of course". > > This was all discussed on the -devel mailing list when the feature was first implemented. > http://www.openldap.org/lists/openldap-devel/200504/msg00090.html Thanks. >> So... back to start, I guess I was suggesting that the >> indexing task should stop if another config mod comes in, >> if it can resume afterwards. And the config change need >> only fail if it might prevent the indexer from resuming. >> I suppose that gets complicated in a hurry though. > > Stopping and resuming is problematic, especially if other config changes to the DB occur > that would invalidate the in-progress indexing. OK. > We could arrange to return LDAP_BUSY to config mods if any background indexers are running. > Or as I already mentioned, we could try to only do this for relevant config mods, e.g. to > the active backend, or to global schema. I'm mostly interested in the worst expected case during normal cn=config use - whether hang or error. Limiting the problems beyond that is a bonus. Or maybe not. A cn=config which fails in some obscure cases is flaky. A cn=config which fails more frequently but in predictable ways can be annoying, but is easy to figure out how to operate. I suppose a simple test is how simple the documentation can be: If I want non-config operations to not hang, what do I do and what do I avoid? And same thing if want to write a config script which won't need to cope with expected failures.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by hyc＠symas.com 20 Jan '19

20 Jan '19

Hallvard Breien Furuseth wrote: > On 1/19/19 8:04 PM, Howard Chu wrote: >> modifications to cn=3Dconfig require no other threads to be active. On= ce the background indexer >> starts running, its activity will of course block further cn=3Dconfig = modifications until it >> completes. >=20 > Nothing "of course" about it.=C2=A0 I don't know why running > the background indexer is considered a change to cn=3Dconfig. It is not. But the indexing rules can be changed by further config mods, = and we don't want the config to change out from under it. > The ldapmodify did the change to cn=3Dconfig and completed. >=20 > I can guess that it might block cn=3Dconfig simply because that's > the safe way to have both an indexer and dynamic config.=C2=A0 Or > I can read the code.=C2=A0 But "read the code" !=3D "of course". This was all discussed on the -devel mailing list when the feature was fi= rst implemented. http://www.openldap.org/lists/openldap-devel/200504/msg00090.html > So... back to start, I guess I was suggesting that the > indexing task should stop if another config mod comes in, > if it can resume afterwards.=C2=A0 And the config change need > only fail if it might prevent the indexer from resuming. > I suppose that gets complicated in a hurry though. Stopping and resuming is problematic, especially if other config changes = to the DB occur that would invalidate the in-progress indexing. We could arrange to return LDAP_BUSY to config mods if any background ind= exers are running. Or as I already mentioned, we could try to only do this for relevant conf= ig mods, e.g. to the active backend, or to global schema. --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 19 Jan '19

19 Jan '19

On 1/19/19 8:04 PM, Howard Chu wrote: > modifications to cn=config require no other threads to be active. Once the background indexer > starts running, its activity will of course block further cn=config modifications until it > completes. Nothing "of course" about it. I don't know why running the background indexer is considered a change to cn=config. The ldapmodify did the change to cn=config and completed. I can guess that it might block cn=config simply because that's the safe way to have both an indexer and dynamic config. Or I can read the code. But "read the code" != "of course". So... back to start, I guess I was suggesting that the indexing task should stop if another config mod comes in, if it can resume afterwards. And the config change need only fail if it might prevent the indexer from resuming. I suppose that gets complicated in a hurry though.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 19 Jan '19

19 Jan '19

On 1/20/19 5:26 AM, Howard Chu wrote: > Hallvard Breien Furuseth wrote: >> On 1/20/19 4:48 AM, Hallvard Breien Furuseth wrote: >>> On 1/20/19 2:14 AM, Howard Chu wrote: >>>> This was the safest approach, considering that a config change could alter/remove context out >>>> from under running threads otherwise. >>> >>> No, the safest approach is to fail the 2nd cn=config mod >>> unless the user says it's OK to block the server. > > How would the user say so? An LDAP control. Or the modify operation could modify an operational attribute which which doesn't exist in LDAP, it just changes the operation's behavior. Or only accept cn=config mods after the connection has added an ephemeral control entry which says how cn=config mods should behave in that connection. Maybe that's the least annoying way, since there won't be any special cases for the use to cope with. Either way, it sounds like there's no right answer for RE24's default behavior, and the only fix there would involve a config option which says which behavior to pick. (It'd be nice to be able to say things like "it's OK to hang if ldapi:// is the only listener", but I suppose then people have a hundred other conditions which would be nice too.) I don't like any of my own answers to this though. I just dislike server hangs more. >> Or fail the 1st cn=config mod too, if that's easier. > > Huh? That op has already completed. The indexer is started as a background task after > the mod op completes. I meant fail when the operation is issued, but never mind. That was silly, and cumbersome for the user to find out when he can safely issue the next operation. Either way, diagnosticMessage "RTFM: back-mdb(5) about olcDbIndex and slapd-config(5) about possible server hangs". And we should document when cn=config mods can be dangerous and how to use it safely without the users noticing. There should in any case be a method to ask the server "is cn=config very busy now, or is another cn=config mod safe?". Like failing the operation if it is not, like I said first.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by hyc＠symas.com 19 Jan '19

19 Jan '19

Hallvard Breien Furuseth wrote: > On 1/20/19 4:48 AM, Hallvard Breien Furuseth wrote: >> On 1/20/19 2:14 AM, Howard Chu wrote: >>> This was the safest approach, considering that a config change could alter/remove context out >>> from under running threads otherwise. >> >> No, the safest approach is to fail the 2nd cn=config mod >> unless the user says it's OK to block the server. How would the user say so? > Or fail the 1st cn=config mod too, if that's easier. Huh? That op has already completed. The indexer is started as a background task after the mod op completes. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs