11:40 a.m.
If I modify olcDbDirectory of a BDB database to a broken directory, the
Modify returns success but BDB says "failed to reopen database" and
slapd shuts down. Maybe other modifications can shut down slapd too, I
don't know.
It would be nice if slapd tried to restore the old config value, reopen
the old directory, and return unwillingToPerform or something to the
Modify. However I don't know how hard that would be, or how obscure a
case it is.
For that matter, maybe if the Modify broke something it could return
some diagnosticMessage. Not sure if it should return non-success since
that supposedly means the modify did not happen. Though clients may not
display the diagnosticMessage if it comes with 'success'.
--
Hallvard
3:52 p.m.
Hallvard B Furuseth wrote:
If I modify olcDbDirectory of a BDB database to a broken directory,
the
Modify returns success but BDB says "failed to reopen database" and
slapd shuts down. Maybe other modifications can shut down slapd too, I
don't know.
It would be nice if slapd tried to restore the old config value, reopen
the old directory, and return unwillingToPerform or something to the
Modify. However I don't know how hard that would be, or how obscure a
case it is.
The Modify operation has already completed though, there's nothing to
"return"
an error message to by then.
We do as much pre-checking as possible to validate the syntax of changes
before committing them. But here you've got an input with perfectly valid
syntax. The only way to know that it's bad is to commit the change.
We already undo bad changes when we can detect them...
For this particular case, ITS#4829 needs to be considered as well. What is the
definition of a "broken directory"? If you specify a directory that doesn't
exist, the backend ought to just create it and use it. In that case, you'll
end up with an empty context in the server. That might be what was intended,
and also might not be; we have no way to tell. Therefore there is no sensible
pre-check that we can perform.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
12:24 a.m.
Howard Chu wrote:
Hallvard B Furuseth wrote:
> If I modify olcDbDirectory of a BDB database to a broken directory, the
> Modify returns success but BDB says "failed to reopen database" and
> slapd shuts down. Maybe other modifications can shut down slapd too, I
> don't know.
>
> It would be nice if slapd tried to restore the old config value, reopen
> the old directory, and return unwillingToPerform or something to the
> Modify. However I don't know how hard that would be, or how obscure a
> case it is.
The Modify operation has already completed though, there's nothing to
"return" an error message to by then.
We do as much pre-checking as possible to validate the syntax of changes
before committing them. But here you've got an input with perfectly
valid syntax. The only way to know that it's bad is to commit the change.
We already undo bad changes when we can detect them...
For this particular case, ITS#4829 needs to be considered as well. What
is the definition of a "broken directory"? If you specify a directory
that doesn't exist, the backend ought to just create it and use it. In
that case, you'll end up with an empty context in the server. That might
be what was intended, and also might not be; we have no way to tell.
Therefore there is no sensible pre-check that we can perform.
According to Howard's and Hallvard's analysis, slapd should do its best
to determine if the change is inconsistent. As soon as it's
self-consistent (although possibly not what the user intended; but then
it's the user's fault), and no matter what, it shouldn't result in
shutting down slapd. In this case, I think it is the responsibility of
back-config to try and open a database, but gracefully handle failure.
p.
Ing. Pierangelo Masarati
OpenLDAP Core Team
SysNet s.r.l.
via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
---------------------------------------
Office: +39 02 23998309
Mobile: +39 333 4963172
Email: pierangelo.masarati(a)sys-net.it
---------------------------------------
6:01 a.m.
Howard Chu writes:
The Modify operation has already completed though, there's
nothing to
"return" an error message to by then.
I see.
We do as much pre-checking as possible to validate the syntax of
changes
before committing them. But here you've got an input with perfectly valid
syntax. The only way to know that it's bad is to commit the change.
We already undo bad changes when we can detect them...
That's good, but that also means that a success response doesn't
indicate that the change was made to cn=config.
So let me step back a bit: I'd like slapd to send the ModifyResponse
_after_ the change has taken effect (or failed to take effect).
How can a program currently wait for a cn=config change to take effect?
Just loop and read the directory, waiting for either an expected visible
change or for the cn=config change to disappear (if it gets undone)?
For this particular case, ITS#4829 needs to be considered as
well. What is the definition of a "broken directory"? If you specify a
directory that doesn't exist, the backend ought to just create it and
use it. In that case, you'll end up with an empty context in the
server. That might be what was intended, and also might not be; we
have no way to tell. Therefore there is no sensible pre-check that we
can perform.
"Slapd does not want to try to guess what you want to happen" seems a
sensible enough error to check for.
Beyond that, I guess we need some way to tell slapd what we want to
happen. A control, or an attribute sent as part of the entry over the
protocol but which is not written to cn=config. (I expect this is
getting into an old discussion, but I'm not going to look that up now.)
BTW, unless I've missed something the doc says very little about how
back-config itself works. Mostly the slapd-config(5) and the config
parts of the admin guide describe the _contents_ of cn=config.
--
Hallvard
1:32 a.m.
Hallvard B Furuseth wrote:
Howard Chu writes:
> The Modify operation has already completed though, there's nothing to
> "return" an error message to by then.
I see.
> We do as much pre-checking as possible to validate the syntax of changes
> before committing them. But here you've got an input with perfectly valid
> syntax. The only way to know that it's bad is to commit the change.
>
> We already undo bad changes when we can detect them...
That's good, but that also means that a success response doesn't
indicate that the change was made to cn=config.
No, the changes that we can detect are done before the response is sent. As
such, the failure is sent back to the client (as it should be).
So let me step back a bit: I'd like slapd to send the
ModifyResponse
_after_ the change has taken effect (or failed to take effect).
Right, it's just that some changes need to be done at a later point than
others. It's all about keeping a logical sequencing.
How can a program currently wait for a cn=config change to take
effect?
Just loop and read the directory, waiting for either an expected visible
change or for the cn=config change to disappear (if it gets undone)?
The only change that has a delayed effect is adding a new index, which starts
a background task. Everything else actually takes place before the response
goes back to the client. The problem right now is that the return code from
the cleanup handler isn't being checked. That's a simple fix. The other
problem is that we need to keep the old set of attributes around to undo the
modify if the cleanup fails.
BTW, unless I've missed something the doc says very little about
how
back-config itself works. Mostly the slapd-config(5) and the config
parts of the admin guide describe the _contents_ of cn=config.
For the most part, internal implementation details never get documented. Only
developers need to know, and they can just read the code.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
1:31 a.m.
Howard Chu wrote:
Hallvard B Furuseth wrote:
> Howard Chu writes:
>> The Modify operation has already completed though, there's nothing to
>> "return" an error message to by then.
> I see.
>
>> We do as much pre-checking as possible to validate the syntax of changes
>> before committing them. But here you've got an input with perfectly valid
>> syntax. The only way to know that it's bad is to commit the change.
>>
>> We already undo bad changes when we can detect them...
> That's good, but that also means that a success response doesn't
> indicate that the change was made to cn=config.
No, the changes that we can detect are done before the response is sent. As
such, the failure is sent back to the client (as it should be).
> So let me step back a bit: I'd like slapd to send the ModifyResponse
> _after_ the change has taken effect (or failed to take effect).
That's actually the way it has been implemented. Just that in this case, we
weren't validating the olcDbDirectory argument before trying to use it. This
is now being done in HEAD.
As for bad changes shutting down the server - that was also a conscious
decision; the kinds of errors that can cause this failure are presumably the
type that we cannot rollback and recover from. If you find any other trivially
recoverable errors that cause this to occur, feel free to patch them, but
leave the overall shutdown trigger in place.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
2:57 p.m.
Howard Chu wrote:
>Hallvard B Furuseth wrote:
>> So let me step back a bit: I'd like slapd to send the ModifyResponse
>> _after_ the change has taken effect (or failed to take effect).
That's actually the way it has been implemented. Just that in this
case, we weren't validating the olcDbDirectory argument before trying
to use it. This is now being done in HEAD.
Great.
As for bad changes shutting down the server - that was also a
conscious decision; the kinds of errors that can cause this failure
are presumably the type that we cannot rollback and recover from.
I presume you mean badness that is discovered too late to be able to
reject or undo during the cn=config update request? If so, fair enough.
Though that's one thing which needs some documentation - how careful
should one be with cn=config changes? Expanding on that a bit:
[Copying from an earlier message]
Howard Chu wrote:
Hallvard B Furuseth wrote:
> BTW, unless I've missed something the doc says very little about how
> back-config itself works. Mostly the slapd-config(5) and the config
> parts of the admin guide describe the _contents_ of cn=config.
For the most part, internal implementation details never get
documented. Only developers need to know, and they can just read the
code.
No, we are talking about user-visible effects here, even if the
explanation dives into the internals.
Here are some things it would be useful for the manpage to answer.
(Note, I'm listing questions, not asking them. I'm in no hurry at
the moment, I know some answers wonder about otheres.)
Are there any config options specific to back-config? Does it honor
access controls? (No .SH CONFIGURATION nor ACCESS CONTROL section.)
Are changes atomic as seen by clients, i.e. they won't see the affected
database or whatever in a temporary state when the config change is in
progress? (Like I wondered with moving olcDbDirectory, for example.)
Do changes that succeed take effect immediately (unless otherwise
stated)? How are operations that are already progress affected?
In a number of cases, it'll break the database to edit slapd.conf and
restart slapd. To what degree are the same changes safe to try with
cn=config (either because it'll handle it or the modify operation will
return failure)? E.g. adding an index works. How about changing
olcIndexSubstrAnyLen? Removing olcAttributeOptions or object classes
that are is in use in the database? Removing an unused attribute
allowed by some objectClass? (I know these could be documented under
the individual options' docs, but a general guideline would be good.
Might fit best in the EXAMPLES section, depends on how firm principles
are involved.)
Related, could use an indication of how "magical" cn=config changes are.
Like my open/close database question, or modify olcLogFile to contain
its current value after rotating the file to make it open and close
that, etc. (Haven't tried. Point is, there are a _lot_ of things like
that to try out, it'd be useful to have a general feel for how much to
expect.)
If I've broken my configuration so slapd won't start, and want to fix it
instead of starting from scratch, what to do? Just edit the cn=config
LDIFs, or is there some extra magic to look out for? The manpage
doesn't even mention that the config is stored in an LDIF database,
which might be going a bit far in the "don't document internals"
direction.
What are those {x} "indexes" in DNs and attributes?
Security issues: Remind users that since an attacker can make slapd
perform various OS and filesystem actions as well as access the LDAP
databases. So remember to restrict cn=config access tightly, and it
gets more important not to run slapd as root.
Some but all attributes get defaulted when created from slapd.conf, so
presumably changing the default later in the frontend won't affect them.
Yet at least in the case of olcReadOnly, I could delete the attribute
with ldapmodify. So why does it get defaulted, what's going on?
slapd-config(5) says:
GLOBAL DATABASE OPTIONS
Options in this section may be set in the special "frontend"
database and inherited in all the other databases. These options
may be altered by further settings in each specific database. (...)
That doesn't look quite true, maybe some get copied from the frontend
upon database creation and others inherit at runtime, I don't know.
--
Hallvard
4:14 p.m.
Hallvard B Furuseth wrote:
Howard Chu wrote:
>> Hallvard B Furuseth wrote:
>>> So let me step back a bit: I'd like slapd to send the ModifyResponse
>>> _after_ the change has taken effect (or failed to take effect).
> That's actually the way it has been implemented. Just that in this
> case, we weren't validating the olcDbDirectory argument before trying
> to use it. This is now being done in HEAD.
Great.
> As for bad changes shutting down the server - that was also a
> conscious decision; the kinds of errors that can cause this failure
> are presumably the type that we cannot rollback and recover from.
I presume you mean badness that is discovered too late to be able to
reject or undo during the cn=config update request? If so, fair enough.
Though that's one thing which needs some documentation - how careful
should one be with cn=config changes?
That seems like a pretty silly question. cn=config is so potentially dangerous
that in OpenLDAP 2.3 we restricted it to only be usable by its rootDN. How
careful should one be - you can never afford to be careless.
> For the most part, internal implementation details never get
> documented. Only developers need to know, and they can just read the
> code.
No, we are talking about user-visible effects here, even if the
explanation dives into the internals.
Here are some things it would be useful for the manpage to answer.
(Note, I'm listing questions, not asking them. I'm in no hurry at
the moment, I know some answers wonder about otheres.)
OK, so the following is stuff to send in Gavin's direction... ;)
Are there any config options specific to back-config? Does it honor
access controls? (No .SH CONFIGURATION nor ACCESS CONTROL section.)
No, yes. back-config would be unusable if it had any backend-specific options.
The chicken'n'egg problem would be unresolvable.
Are changes atomic as seen by clients, i.e. they won't see the
affected
database or whatever in a temporary state when the config change is in
progress? (Like I wondered with moving olcDbDirectory, for example.)
That's implicit to LDAP, doesn't bear further mention.
Do changes that succeed take effect immediately (unless otherwise
stated)? How are operations that are already progress affected?
The former doesn't bear mention, it is already advertised as such. (E.g. the
Admin Guide:
The LDAP configuration engine allows all of slapd's configuration options
to be changed on the fly, generally without requiring a server restart for the
changes to take effect.
There are never any other operations in progress when changes take effect in
cn=config.
In a number of cases, it'll break the database to edit slapd.conf
and
restart slapd. To what degree are the same changes safe to try with
cn=config (either because it'll handle it or the modify operation will
return failure)? E.g. adding an index works. How about changing
olcIndexSubstrAnyLen? Removing olcAttributeOptions or object classes
that are is in use in the database? Removing an unused attribute
allowed by some objectClass? (I know these could be documented under
the individual options' docs, but a general guideline would be good.
Might fit best in the EXAMPLES section, depends on how firm principles
are involved.)
Changing the olcIndexSubstr parameters will require a full re-run of
slapindex. Removing attributes and objectclasses will not impair the currently
running server, although it may cause problems on a subsequent restart.
(Deleting schema elements at runtime only makes them inaccessible to users;
their definitions remain in memory because there may still be active
references to them at runtime. Any attempt to create new references to them
will of course fail.)
Related, could use an indication of how "magical" cn=config
changes are.
Like my open/close database question, or modify olcLogFile to contain
its current value after rotating the file to make it open and close
that, etc. (Haven't tried. Point is, there are a _lot_ of things like
that to try out, it'd be useful to have a general feel for how much to
expect.)
I guess someone will have to walk thru them all and try them out. The intent
for cn=config was for everything to Just Work, so we should only need to
document where limitations exist. (E.g., olcIndexSubstr*)
If I've broken my configuration so slapd won't start, and
want to fix it
instead of starting from scratch, what to do? Just edit the cn=config
LDIFs, or is there some extra magic to look out for? The manpage
doesn't even mention that the config is stored in an LDIF database,
which might be going a bit far in the "don't document internals"
direction.
OK. The cn=config design actually has very few dependencies on the underlying
backend, but it currently uses back-ldif for two main reasons - back-ldif has
only one relevant config directive (olcDbDirectory) and its storage format is
human readable. Is it safe to assume that no other backend offers this
simplicity and we will therefore never change it? I believe so, so it's fair
to at least mention that back-config uses back-ldif.
What are those {x} "indexes" in DNs and attributes?
Already documented in the Admin Guide.
Security issues: Remind users that since an attacker can make slapd
perform various OS and filesystem actions as well as access the LDAP
databases. So remember to restrict cn=config access tightly, and it
gets more important not to run slapd as root.
Right.
Some but all attributes get defaulted when created from slapd.conf,
so
presumably changing the default later in the frontend won't affect them.
Yet at least in the case of olcReadOnly, I could delete the attribute
with ldapmodify. So why does it get defaulted, what's going on?
slapd-config(5) says:
GLOBAL DATABASE OPTIONS
Options in this section may be set in the special "frontend"
database and inherited in all the other databases. These options
may be altered by further settings in each specific database. (...)
That doesn't look quite true, maybe some get copied from the frontend
upon database creation and others inherit at runtime, I don't know.
I guess that description was only true for slapd.conf and initial startup.
Changes to the frontend made at runtime don't get propagated to all other
databases. I wonder if they should. Probably one would expect changes to the
frontend/global ACLs to take effect for all other backends.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
3:30 a.m.
Howard Chu writes:
Hallvard B Furuseth wrote:
> (...) Though that's one thing which needs some documentation - how
> careful should one be with cn=config changes?
That seems like a pretty silly question. cn=config is so potentially
dangerous that in OpenLDAP 2.3 we restricted it to only be usable by
its rootDN. How careful should one be - you can never afford to be
careless.
Well, yes, but it's not currently well enough documented to say that,
unless the recommendation for now is to not try anything creative
which isn't documented.
[Skipping the "OK, good" parts]
(...)
OK, so the following is stuff to send in Gavin's direction... ;)
(...)
> Are changes atomic as seen by clients, i.e. they won't see the affected
> database or whatever in a temporary state when the config change is in
> progress? (Like I wondered with moving olcDbDirectory, for example.)
That's implicit to LDAP, doesn't bear further mention.
Yes, for two reasons. First, it would conform to the LDAP spec if a
change to the cn=config LDAP data is atomic but the effect on slapd
happens gradually. E.g. existing connections might be immune from a
tightened olcIdleTimeout, olcTLSCipherSuite or BDB olcDbSearchStack.
Second, OpenLDAP does break the LDAP spec on occation. Sometimes
blatantly like with "userPassword: {hash method}...". Someitmes when
interaction between various modules gets too hairy. And cn=config is
just about the hairiest of all. It'd be unsurprising to see slapd need
time to adjust to some changes it handles at run-time. (Changes due to
restart are of course seen as atomic:-)
> Do changes that succeed take effect immediately (unless
otherwise
> stated)? (...)
The former doesn't bear mention, it is already advertised as such. (E.g. the
Admin Guide:
The LDAP configuration engine allows all of slapd's configuration
options to be changed on the fly, generally without requiring a server
restart for the changes to take effect.
That doesn't say "immediately", in particular as in "by the time the
operation has returned success to the client". Like the change which
started this thread, and my olcIdleTimeout example above.
> Related, could use an indication of how "magical"
cn=config changes are.
> Like my open/close database question, or modify olcLogFile to contain
> its current value after rotating the file to make it open and close
> that, etc. (Haven't tried. Point is, there are a _lot_ of things like
> that to try out, it'd be useful to have a general feel for how much to
> expect.)
I guess someone will have to walk thru them all and try them out.
And check the code, in case something works just accidentally or only in
cases when it's easy for slapd to do (again like the change which
started this thread).
The intent for cn=config was for everything to Just Work, so we
should
only need to document where limitations exist. (E.g., olcIndexSubstr*)
That's great. And that's the statement I was asking for this time,
instead of the details of each individual attribute. Follow it with the
advice (above) not to try anything unless documented, and together
they'll give a good feel for how to use back-config.
Perhaps the doc should mark[*] all attrs for which it Just Works, or for
which it doesn't always Just Work.
--
Regards,
Hallvard
5710
days inactive
5714
days old
8 comments
3 participants
participants (3)
-
Hallvard B Furuseth -
Howard Chu -
Pierangelo Masarati