Hi,
as back-config does currently not have support for the delete operation (config_back_delete() just returns LDAP_UNWILLING_TO_PERFORM currently) I am trying to figure out what is needed to get at least delete support for simple overlays (e.g. ppolicy or memberof) running.
What I am currently doing is: - renumber the indexes of the siblings of the overlay that's going to be deleted - remove the CfEntryInfo of the overlay from the internal config tree - remove the Entry from the underlying LDIF database
Additionally I need to remove the overlay's slap_overinst structure from the overlay list of the backend and in case it was the last overlay, restore the original BackendInfo structure and delete the SLAP_DBFLAG_GLOBAL_OVERLAY flag from the BackendDB structure of the underlying database (for global overlays). Currently I use the overlay_destroy_one() function for this task. This also calls the db_destroy() hook of the overlay which should take are of free resources held by the overlay.
I am a bit unsure about the db_destroy() part. Is that enough or is it needed to call the db_close() hooks (when provided by the overlay) before calling db_destroy()? Did I miss something else? Is anybody else currently working on this?
Up to now I am testing this only with some simpler overlays (memberof, pcache), I assume that deleting overlays that instantiate their own databases (like pcache, translucent(?)) might get a little more hairy.
You also need to walk through the database and delete any attributes object classes defined by the overlay. That'll be at least any operational attributes since they must be hardcoded into the C source.
Only the overlay can know how to do this, so the overlay needs undo_me() code. Then that code must figure out how to interact with backends and other overlays - like translucent. Or how to figure out whether it is able to figure it out, and return LDAP_UNWILLING_TO_PERFORM if not.
I don't quite remember how ppolicy works but I assume it can "deactivate" an existing userPassword attribute. If so it also needs a policy about what to do with deactivated userPasswords which will be reactivated when ppolicy is deleted. I.e. should these userPasswords be deleted? And if an overlay does that - deletes data which might not be managed by itself - then things could get _really_ hairy to get right...
I wrote:
You also need to walk through the database and delete any attributes object classes defined by the overlay. That'll be at least any operational attributes since they must be hardcoded into the C source.
Er - might be easer for the manager to do this (maybe with help from the overlay) so the overlay merely needs to check if the database is in a state where the overlay can be deleted. Might not be possible with all overlays though, if an entry with no attributes from the overlay would be invalid as far as the overlay is concerned.
On Donnerstag, 5. Juni 2008, Hallvard B Furuseth wrote:
You also need to walk through the database and delete any attributes object classes defined by the overlay. That'll be at least any operational attributes since they must be hardcoded into the C source.
Only the overlay can know how to do this, so the overlay needs undo_me() code. Then that code must figure out how to interact with backends and other overlays - like translucent. Or how to figure out whether it is able to figure it out, and return LDAP_UNWILLING_TO_PERFORM if not.
I don't quite remember how ppolicy works but I assume it can "deactivate" an existing userPassword attribute. If so it also needs a policy about what to do with deactivated userPasswords which will be reactivated when ppolicy is deleted. I.e. should these userPasswords be deleted?
Hm, if the Admin decides to delete the password policy overlay from a database IMO it is correct to assume that he does not want any policies for his passwords anymore. That also means that accounts which have been locked through ppolicy we be unlocked after the removal of ppolicy. I don't think this is a problem, it should probably be documented, though.
And if an overlay does that - deletes data which might not be managed by itself - then things could get _really_ hairy to get right...
On Donnerstag, 5. Juni 2008, Hallvard B Furuseth wrote:
You also need to walk through the database and delete any attributes object classes defined by the overlay. That'll be at least any operational attributes since they must be hardcoded into the C source.
Another problem with this is that it can take a long time, which is something that we probably not want inside back-config, when the thread-pool is paused. ;)
[..]
Hallvard B Furuseth wrote:
You also need to walk through the database and delete any attributes object classes defined by the overlay. That'll be at least any operational attributes since they must be hardcoded into the C source.
I wouldn't bother with that, since the schema definitions are still present. Those attributes would simply stop being updated/generated...
On the other hand, if the overlay is a dynamically loaded module, and you want to unload it (by deleting the moduleLoad) then all of its schema needs to be unregistered, etc. I already mentioned this at some point in the past, that we need to write fini() entry points for all of our dynamically loadable modules...
Only the overlay can know how to do this, so the overlay needs undo_me() code. Then that code must figure out how to interact with backends and other overlays - like translucent. Or how to figure out whether it is able to figure it out, and return LDAP_UNWILLING_TO_PERFORM if not.
I don't quite remember how ppolicy works but I assume it can "deactivate" an existing userPassword attribute. If so it also needs a policy about what to do with deactivated userPasswords which will be reactivated when ppolicy is deleted. I.e. should these userPasswords be deleted? And if an overlay does that - deletes data which might not be managed by itself - then things could get _really_ hairy to get right...
I think that's wrong. When you remove an overlay, you should expect to lose all the functionality that the overlay provided. If the overlay had locked an account, the lock will simply disappear when the overlay is removed.
Howard Chu wrote:
Hallvard B Furuseth wrote:
You also need to walk through the database and delete any attributes object classes defined by the overlay. That'll be at least any operational attributes since they must be hardcoded into the C source.
I wouldn't bother with that, since the schema definitions are still present. Those attributes would simply stop being updated/generated...
Hmm, I'd consider this to be seriously flawed. It strikes me that there might still exist operational attributes in some entries which are no longer maintained by the DSA and there's no schema information available anymore.
Think of backup/restore situations where slapadd checks the schema...
I think that's wrong. When you remove an overlay, you should expect to lose all the functionality that the overlay provided.
Agreed.
If the overlay had locked an account, the lock will simply disappear when the overlay is removed.
Which IMO means that also the attribute containing the lock (and all other attributes related to the removed overlay) should be removed.
Ciao, Michael.
Michael Ströder wrote:
Howard Chu wrote:
Hallvard B Furuseth wrote:
You also need to walk through the database and delete any attributes object classes defined by the overlay. That'll be at least any operational attributes since they must be hardcoded into the C source.
I wouldn't bother with that, since the schema definitions are still present. Those attributes would simply stop being updated/generated...
Hmm, I'd consider this to be seriously flawed. It strikes me that there might still exist operational attributes in some entries which are no longer maintained by the DSA and there's no schema information available anymore.
Think of backup/restore situations where slapadd checks the schema...
That's a broken administrative process then. E.g., I backup my DB using slapcat, then delete the overlay. Now the backup is essentially invalid, unless I also backed up cn=config and restored everything together.
We can trigger a background thread to try to cleanup the DB contents, but if the slapd is shutdown while the thread is running, then the job will only be half done. There's no remedy for that situation.
No matter what, you have to be prepared to clean up the LDIF as a separate step, if the slapadd runs into unknown schema.
On 6/5/08 11:50 AM, Michael Ströder wrote:
Howard Chu wrote:
Hallvard B Furuseth wrote:
You also need to walk through the database and delete any attributes object classes defined by the overlay. That'll be at least any operational attributes since they must be hardcoded into the C source.
I wouldn't bother with that, since the schema definitions are still present. Those attributes would simply stop being updated/generated...
Hmm, I'd consider this to be seriously flawed. It strikes me that there might still exist operational attributes in some entries which are no longer maintained by the DSA and there's no schema information available anymore.
Think of backup/restore situations where slapadd checks the schema...
How is that different from pre-cn=config setups where the administrator shut down slapd and removed the schema from the slapd.conf and restarted slapd?
It's the same administrative responsibility to remove the use of the schema (and the attributes in the database) before removing the schema definition.
Francis Swasey wrote:
On 6/5/08 11:50 AM, Michael Ströder wrote:
Howard Chu wrote:
Hallvard B Furuseth wrote:
You also need to walk through the database and delete any attributes object classes defined by the overlay. That'll be at least any operational attributes since they must be hardcoded into the C source.
I wouldn't bother with that, since the schema definitions are still present. Those attributes would simply stop being updated/generated...
Hmm, I'd consider this to be seriously flawed. It strikes me that there might still exist operational attributes in some entries which are no longer maintained by the DSA and there's no schema information available anymore.
Think of backup/restore situations where slapadd checks the schema...
How is that different from pre-cn=config setups where the administrator shut down slapd and removed the schema from the slapd.conf and restarted slapd?
It's the same administrative responsibility to remove the use of the schema (and the attributes in the database) before removing the schema definition.
Yes, it has the same administrative consequences. But using cn=config makes the admins think that everything happens automagically. I foresee question on the mailing lists concerning this.
Ciao, Michael.
On Friday 06 June 2008 18:58:11 Michael Ströder wrote:
Francis Swasey wrote:
On 6/5/08 11:50 AM, Michael Ströder wrote:
Howard Chu wrote:
Hallvard B Furuseth wrote:
You also need to walk through the database and delete any attributes object classes defined by the overlay.
IMHO, configuration changes should not result in changes to data.
That'll be at least any operational attributes since they must be hardcoded into the C source.
I wouldn't bother with that, since the schema definitions are still present. Those attributes would simply stop being updated/generated...
Hmm, I'd consider this to be seriously flawed. It strikes me that there might still exist operational attributes in some entries which are no longer maintained by the DSA and there's no schema information available anymore.
Think of backup/restore situations where slapadd checks the schema...
How is that different from pre-cn=config setups where the administrator shut down slapd and removed the schema from the slapd.conf and restarted slapd?
It's the same administrative responsibility to remove the use of the schema (and the attributes in the database) before removing the schema definition.
Yes, it has the same administrative consequences. But using cn=config makes the admins think that everything happens automagically. I foresee question on the mailing lists concerning this.
IMHO, the cleanest would be for any configuration change which would remove schema definitions should do a search for the use of those schema definitions, and fail with a suitable message if any of the schema definitions are in use.
Then the administrator: 1)Is aware of what changes need to occur before the configuration change can be made 2)Will not unknowingly remove data
An intelligent DUA should be able to prompt the administrator to delete the relevant data first, and effect the required changes to the data, before applying the configuration change.
The problem though is that at present some data (whose attribute definitions are marked as "NO-USER-MODIFICATION") cannot presently be removed. ppolicy attributes (pwdAccountLockedTime etc.) come to mind.
Regards, Buchan
Buchan Milne wrote:
On Friday 06 June 2008 18:58:11 Michael Ströder wrote:
Francis Swasey wrote:
On 6/5/08 11:50 AM, Michael Ströder wrote:
Howard Chu wrote:
Hallvard B Furuseth wrote:
You also need to walk through the database and delete any attributes object classes defined by the overlay.
IMHO, configuration changes should not result in changes to data.
SHOULD NOT or MUST NOT (in spirit of RFC 2119)?
IMHO, the cleanest would be for any configuration change which would remove schema definitions should do a search for the use of those schema definitions, and fail with a suitable message if any of the schema definitions are in use.
Would be a possible option too.
The problem though is that at present some data (whose attribute definitions are marked as "NO-USER-MODIFICATION") cannot presently be removed. ppolicy attributes (pwdAccountLockedTime etc.) come to mind.
The question is whether a change in the dynamic configuration justifies requiring the use of slapcat/manual edit/slapadd. The main aim for deploying back-config is to minimize down time, isn't it?
Some operational attributes could be removed with the help of the Relax Rules extended control. Unfortunately one would also have to specify the behaviour of this control concerning ppolicy attributes.
Ciao, Michael.
Michael Ströder wrote:
Buchan Milne wrote:
On Friday 06 June 2008 18:58:11 Michael Ströder wrote:
Francis Swasey wrote:
On 6/5/08 11:50 AM, Michael Ströder wrote:
Howard Chu wrote:
Hallvard B Furuseth wrote: > You also need to walk through the database and delete any attributes > object classes defined by the overlay.
IMHO, configuration changes should not result in changes to data.
SHOULD NOT or MUST NOT (in spirit of RFC 2119)?
IMHO, the cleanest would be for any configuration change which would remove schema definitions should do a search for the use of those schema definitions, and fail with a suitable message if any of the schema definitions are in use.
Would be a possible option too.
The problem though is that at present some data (whose attribute definitions are marked as "NO-USER-MODIFICATION") cannot presently be removed. ppolicy attributes (pwdAccountLockedTime etc.) come to mind.
The question is whether a change in the dynamic configuration justifies requiring the use of slapcat/manual edit/slapadd. The main aim for deploying back-config is to minimize down time, isn't it?
Yes, I agree 100%. Any changes to cn=config MUST NOT result in any manual intervention (except at the moment creating a new driectory i.e. it must exist on the file system first).
The will always be a manual step needed in backups like Howard discussed further up the thread.
Howard Chu writes:
Hallvard B Furuseth wrote:
You also need to walk through the database and delete any attributes object classes defined by the overlay. That'll be at least any operational attributes since they must be hardcoded into the C source.
I wouldn't bother with that, since the schema definitions are still present. Those attributes would simply stop being updated/generated...
OK, but their presence is still misleading.
(...) I.e. should these userPasswords be deleted? (...)
I think that's wrong.
Yes, me too. I was thinking cross-eyed. Cross-brained?
When you remove an overlay, you should expect to lose all the functionality that the overlay provided. If the overlay had locked an account, the lock will simply disappear when the overlay is removed.
Not necessarily, but it's the admin's problem to decide what to do. Though he might want the overlay to refuse to go away until he has cleaned up. Maybe some ppolicy attribute can get a presence index, and the overlay would refuse to be deleted if a search for that attr found something.
Ralf Haferkamp wrote:
Hi,
as back-config does currently not have support for the delete operation (config_back_delete() just returns LDAP_UNWILLING_TO_PERFORM currently) I am trying to figure out what is needed to get at least delete support for simple overlays (e.g. ppolicy or memberof) running.
Very brave of you. ;)
What I am currently doing is:
- renumber the indexes of the siblings of the overlay that's going to be
deleted
- remove the CfEntryInfo of the overlay from the internal config tree
- remove the Entry from the underlying LDIF database
Additionally I need to remove the overlay's slap_overinst structure from the overlay list of the backend and in case it was the last overlay, restore the original BackendInfo structure and delete the SLAP_DBFLAG_GLOBAL_OVERLAY flag from the BackendDB structure of the underlying database (for global overlays). Currently I use the overlay_destroy_one() function for this task. This also calls the db_destroy() hook of the overlay which should take are of free resources held by the overlay.
I am a bit unsure about the db_destroy() part. Is that enough or is it needed to call the db_close() hooks (when provided by the overlay) before calling db_destroy()? Did I miss something else? Is anybody else currently working on this?
You should call the db_close() hook first.
Up to now I am testing this only with some simpler overlays (memberof, pcache), I assume that deleting overlays that instantiate their own databases (like pcache, translucent(?)) might get a little more hairy.
Right. We need to distinguish between tearing down due to a normal shutdown, which should leave underlying data intact, and tearing down due to removal...
On Donnerstag, 5. Juni 2008, Howard Chu wrote:
Ralf Haferkamp wrote:
Hi,
as back-config does currently not have support for the delete operation (config_back_delete() just returns LDAP_UNWILLING_TO_PERFORM currently) I am trying to figure out what is needed to get at least delete support for simple overlays (e.g. ppolicy or memberof) running.
Very brave of you. ;)
:) Let's see how far I can get.
[..]
I am a bit unsure about the db_destroy() part. Is that enough or is it needed to call the db_close() hooks (when provided by the overlay) before calling db_destroy()? Did I miss something else? Is anybody else currently working on this?
You should call the db_close() hook first.
Up to now I am testing this only with some simpler overlays (memberof, pcache), I assume that deleting overlays that instantiate their own databases (like pcache, translucent(?)) might get a little more hairy.
Right. We need to distinguish between tearing down due to a normal shutdown, which should leave underlying data intact, and tearing down due to removal...
Yes, we probably need to add a flag to the db_destroy hooks to indicate the difference. Or add an additional hook for the remove-case.
On Donnerstag, 5. Juni 2008, Howard Chu wrote:
Ralf Haferkamp wrote:
Hi,
as back-config does currently not have support for the delete operation (config_back_delete() just returns LDAP_UNWILLING_TO_PERFORM currently) I am trying to figure out what is needed to get at least delete support for simple overlays (e.g. ppolicy or memberof) running.
Very brave of you. ;)
What I am currently doing is:
- renumber the indexes of the siblings of the overlay that's going
to be deleted
- remove the CfEntryInfo of the overlay from the internal config
tree - remove the Entry from the underlying LDIF database
Additionally I need to remove the overlay's slap_overinst structure from the overlay list of the backend and in case it was the last overlay, restore the original BackendInfo structure and delete the SLAP_DBFLAG_GLOBAL_OVERLAY flag from the BackendDB structure of the underlying database (for global overlays). Currently I use the overlay_destroy_one() function for this task. This also calls the db_destroy() hook of the overlay which should take are of free resources held by the overlay.
I am a bit unsure about the db_destroy() part. Is that enough or is it needed to call the db_close() hooks (when provided by the overlay) before calling db_destroy()? Did I miss something else? Is anybody else currently working on this?
You should call the db_close() hook first.
Ok. I implemented that in a separate function (overlay_remove()) now. Basically that calls the db_close and db_destroy hooks, removes the overlay from the oi_list and cleans up the BackendDB structure when the last overlay is being removed.
I have some preliminary code for back-config delete support ready I think. It currently only supports overlays which do not instantiate child objects (pcache, translucent) and does not touch any data the overlay might have created in the underlying database.
I hope to be able commit that code during the next days. Should it be hidden behind #ifdefs for now? (Probably be enabled with "LDAP_DEVEL")
Up to now I am testing this only with some simpler overlays (memberof, pcache), I assume that deleting overlays that instantiate their own databases (like pcache, translucent(?)) might get a little more hairy.
Right. We need to distinguish between tearing down due to a normal shutdown, which should leave underlying data intact, and tearing down due to removal...
Thinking about this again, if we agree that removing config entries should not touch any data, as previously suggested in this thread, we probably don't need to make this distinction. Though we would need to decide whether back-config should make in implicit subtree delete for overlays that instantiate their own database or if the admin should take care of cleaning up the config tree in the correct order.
An additional hook that could be used to do some config-related cleanup in overlays/databases and that allows to check whether the overlay can be deleted at all, as suggested by Hallvard, might still be useful though.
On Dienstag, 10. Juni 2008, Ralf Haferkamp wrote:
On Donnerstag, 5. Juni 2008, Howard Chu wrote:
Ralf Haferkamp wrote:
Hi,
as back-config does currently not have support for the delete operation (config_back_delete() just returns LDAP_UNWILLING_TO_PERFORM currently) I am trying to figure out what is needed to get at least delete support for simple overlays (e.g. ppolicy or memberof) running.
Very brave of you. ;)
What I am currently doing is:
- renumber the indexes of the siblings of the overlay that's
going to be deleted
- remove the CfEntryInfo of the overlay from the internal config
tree - remove the Entry from the underlying LDIF database
Additionally I need to remove the overlay's slap_overinst structure from the overlay list of the backend and in case it was the last overlay, restore the original BackendInfo structure and delete the SLAP_DBFLAG_GLOBAL_OVERLAY flag from the BackendDB structure of the underlying database (for global overlays). Currently I use the overlay_destroy_one() function for this task. This also calls the db_destroy() hook of the overlay which should take are of free resources held by the overlay.
I am a bit unsure about the db_destroy() part. Is that enough or is it needed to call the db_close() hooks (when provided by the overlay) before calling db_destroy()? Did I miss something else? Is anybody else currently working on this?
You should call the db_close() hook first.
Ok. I implemented that in a separate function (overlay_remove()) now. Basically that calls the db_close and db_destroy hooks, removes the overlay from the oi_list and cleans up the BackendDB structure when the last overlay is being removed.
I have some preliminary code for back-config delete support ready I think. It currently only supports overlays which do not instantiate child objects (pcache, translucent) and does not touch any data the overlay might have created in the underlying database.
I hope to be able commit that code during the next days. Should it be hidden behind #ifdefs for now? (Probably be enabled with "LDAP_DEVEL")
Done, please test and comment. The feature is currently hidden behind "#ifdef SLAP_CONFIG_DELETE" which will be enabled with the LDAP_DEVEL macro.
-
Buchan Milne -
Francis Swasey -
Gavin Henry -
Hallvard B Furuseth -
Howard Chu -
Michael Ströder -
Ralf Haferkamp