openldap-bugs January 2019

openldap-bugs@openldap.org

21 participants
77 discussions

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 20 Jan '19

20 Jan '19

I wrote: > Limiting the problems beyond that is a bonus. Or maybe not. > A cn=config which fails in some obscure cases is flaky. (...) Correction: Obscure _situations_, not obscure cases. As in, operation X normally succeds, but can fail if you just did the somewhat-related operation Y.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 20 Jan '19

20 Jan '19

Can cn=config have 2 lock levels? That fixes this ITS (my ldapwhoami hang) without introducing cn=config failures. The outer lock serializes cn=config ops including the indexer. Other ops do not lock it. The inner lock blocks slapd like cn=config does now, and should only be held briefly. Always hold the outer lock when taking the inner lock. Also, the inner lock could be taken and released several times during a config op when feasible. That might improve non-config op latency, for the price of slower config ops. On 1/20/19 4:42 PM, Howard Chu wrote: > Hallvard Breien Furuseth wrote: >> On 1/19/19 8:04 PM, Howard Chu wrote: >>> modifications to cn=config require no other threads to be active. Once the background indexer >>> starts running, its activity will of course block further cn=config modifications until it >>> completes. >> >> Nothing "of course" about it. I don't know why running >> the background indexer is considered a change to cn=config. > > It is not. But the indexing rules can be changed by further config mods, and we > don't want the config to change out from under it. OK. I was just parsing your worlds too literally. Also, I'm trying to think like an admin, not an OpenLDAP insider: >> The ldapmodify did the change to cn=config and completed. >> >> I can guess that it might block cn=config simply because that's >> the safe way to have both an indexer and dynamic config. Or >> I can read the code. But "read the code" != "of course". > > This was all discussed on the -devel mailing list when the feature was first implemented. > http://www.openldap.org/lists/openldap-devel/200504/msg00090.html Thanks. >> So... back to start, I guess I was suggesting that the >> indexing task should stop if another config mod comes in, >> if it can resume afterwards. And the config change need >> only fail if it might prevent the indexer from resuming. >> I suppose that gets complicated in a hurry though. > > Stopping and resuming is problematic, especially if other config changes to the DB occur > that would invalidate the in-progress indexing. OK. > We could arrange to return LDAP_BUSY to config mods if any background indexers are running. > Or as I already mentioned, we could try to only do this for relevant config mods, e.g. to > the active backend, or to global schema. I'm mostly interested in the worst expected case during normal cn=config use - whether hang or error. Limiting the problems beyond that is a bonus. Or maybe not. A cn=config which fails in some obscure cases is flaky. A cn=config which fails more frequently but in predictable ways can be annoying, but is easy to figure out how to operate. I suppose a simple test is how simple the documentation can be: If I want non-config operations to not hang, what do I do and what do I avoid? And same thing if want to write a config script which won't need to cope with expected failures.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by hyc＠symas.com 20 Jan '19

20 Jan '19

Hallvard Breien Furuseth wrote: > On 1/19/19 8:04 PM, Howard Chu wrote: >> modifications to cn=3Dconfig require no other threads to be active. On= ce the background indexer >> starts running, its activity will of course block further cn=3Dconfig = modifications until it >> completes. >=20 > Nothing "of course" about it.=C2=A0 I don't know why running > the background indexer is considered a change to cn=3Dconfig. It is not. But the indexing rules can be changed by further config mods, = and we don't want the config to change out from under it. > The ldapmodify did the change to cn=3Dconfig and completed. >=20 > I can guess that it might block cn=3Dconfig simply because that's > the safe way to have both an indexer and dynamic config.=C2=A0 Or > I can read the code.=C2=A0 But "read the code" !=3D "of course". This was all discussed on the -devel mailing list when the feature was fi= rst implemented. http://www.openldap.org/lists/openldap-devel/200504/msg00090.html > So... back to start, I guess I was suggesting that the > indexing task should stop if another config mod comes in, > if it can resume afterwards.=C2=A0 And the config change need > only fail if it might prevent the indexer from resuming. > I suppose that gets complicated in a hurry though. Stopping and resuming is problematic, especially if other config changes = to the DB occur that would invalidate the in-progress indexing. We could arrange to return LDAP_BUSY to config mods if any background ind= exers are running. Or as I already mentioned, we could try to only do this for relevant conf= ig mods, e.g. to the active backend, or to global schema. --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 19 Jan '19

19 Jan '19

On 1/19/19 8:04 PM, Howard Chu wrote: > modifications to cn=config require no other threads to be active. Once the background indexer > starts running, its activity will of course block further cn=config modifications until it > completes. Nothing "of course" about it. I don't know why running the background indexer is considered a change to cn=config. The ldapmodify did the change to cn=config and completed. I can guess that it might block cn=config simply because that's the safe way to have both an indexer and dynamic config. Or I can read the code. But "read the code" != "of course". So... back to start, I guess I was suggesting that the indexing task should stop if another config mod comes in, if it can resume afterwards. And the config change need only fail if it might prevent the indexer from resuming. I suppose that gets complicated in a hurry though.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 19 Jan '19

19 Jan '19

On 1/20/19 5:26 AM, Howard Chu wrote: > Hallvard Breien Furuseth wrote: >> On 1/20/19 4:48 AM, Hallvard Breien Furuseth wrote: >>> On 1/20/19 2:14 AM, Howard Chu wrote: >>>> This was the safest approach, considering that a config change could alter/remove context out >>>> from under running threads otherwise. >>> >>> No, the safest approach is to fail the 2nd cn=config mod >>> unless the user says it's OK to block the server. > > How would the user say so? An LDAP control. Or the modify operation could modify an operational attribute which which doesn't exist in LDAP, it just changes the operation's behavior. Or only accept cn=config mods after the connection has added an ephemeral control entry which says how cn=config mods should behave in that connection. Maybe that's the least annoying way, since there won't be any special cases for the use to cope with. Either way, it sounds like there's no right answer for RE24's default behavior, and the only fix there would involve a config option which says which behavior to pick. (It'd be nice to be able to say things like "it's OK to hang if ldapi:// is the only listener", but I suppose then people have a hundred other conditions which would be nice too.) I don't like any of my own answers to this though. I just dislike server hangs more. >> Or fail the 1st cn=config mod too, if that's easier. > > Huh? That op has already completed. The indexer is started as a background task after > the mod op completes. I meant fail when the operation is issued, but never mind. That was silly, and cumbersome for the user to find out when he can safely issue the next operation. Either way, diagnosticMessage "RTFM: back-mdb(5) about olcDbIndex and slapd-config(5) about possible server hangs". And we should document when cn=config mods can be dangerous and how to use it safely without the users noticing. There should in any case be a method to ask the server "is cn=config very busy now, or is another cn=config mod safe?". Like failing the operation if it is not, like I said first.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by hyc＠symas.com 19 Jan '19

19 Jan '19

Hallvard Breien Furuseth wrote: > On 1/20/19 4:48 AM, Hallvard Breien Furuseth wrote: >> On 1/20/19 2:14 AM, Howard Chu wrote: >>> This was the safest approach, considering that a config change could alter/remove context out >>> from under running threads otherwise. >> >> No, the safest approach is to fail the 2nd cn=config mod >> unless the user says it's OK to block the server. How would the user say so? > Or fail the 1st cn=config mod too, if that's easier. Huh? That op has already completed. The indexer is started as a background task after the mod op completes. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 19 Jan '19

19 Jan '19

On 1/20/19 4:48 AM, Hallvard Breien Furuseth wrote: > On 1/20/19 2:14 AM, Howard Chu wrote: >> This was the safest approach, considering that a config change could >> alter/remove context out >> from under running threads otherwise. > > No, the safest approach is to fail the 2nd cn=config mod > unless the user says it's OK to block the server. Or fail the 1st cn=config mod too, if that's easier.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 19 Jan '19

19 Jan '19

On 1/20/19 2:14 AM, Howard Chu wrote: > Ah, the 2nd cn=config mod has requested a pause, which can't succeed until all > background threads complete. And no new threads will be run while a pause is being requested. Exactly. > This was the safest approach, considering that a config change could alter/remove context out > from under running threads otherwise. No, the safest approach is to fail the 2nd cn=config mod unless the user says it's OK to block the server.

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by hyc＠symas.com 19 Jan '19

19 Jan '19

Hallvard Breien Furuseth wrote: > On 1/19/19 9:55 PM, h.b.furuseth(a)usit.uio.no wrote: >> On 1/19/19 8:04 PM, Howard Chu wrote: >>> The *server* is not hung. Only modifications to the cn=config DB are blocked. >> >> Yes it is. > > That is, the 2nd mod to cn=config blocks, and that's fine by me, except > _then_ the server is hung. Ah, the 2nd cn=config mod has requested a pause, which can't succeed until all background threads complete. And no new threads will be run while a pause is being requested. This was the safest approach, considering that a config change could alter/remove context out from under running threads otherwise. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8958) 2nd cn=config update blocks slapd while adding subordinate index
by h.b.furuseth＠usit.uio.no 19 Jan '19

19 Jan '19

On 1/19/19 9:55 PM, h.b.furuseth(a)usit.uio.no wrote: > On 1/19/19 8:04 PM, Howard Chu wrote: >> The *server* is not hung. Only modifications to the cn=config DB are blocked. > > Yes it is. That is, the 2nd mod to cn=config blocks, and that's fine by me, except _then_ the server is hung. > See the ldapwhoami in my example. And the 'time ldapwhoami' result in output.txt. Sorry, unreadable scripting on my part. Notice the single '&' in ldapmodify & { sleep; ldapwhoami; } That is, ldapmodify in the background and then ldapwhoami. I was pasting to/from the command line, and didn't want the ldapmodify to start until I pressed Return for the ldapwhoami. Thus the weird concatenation.

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs January 2019