On Thu, Dec 11, 2008 at 10:37:26AM -0500, Andrzej Jan Taramina wrote:
Howard replied:
This was just discussed on -technical as well. You can delete individual schema elements using ldapmodify. You cannot delete entire cn=config entries (using ldapdelete). There are no plans to change this behavior for schema in the future.
Using ldapmodify for individual entries is painful in the extreme, since each entry only has a single attribute...basically a monster string with all the schema constraints in it.
What are you talking about? A schema entry under cn=config has individual attribute values per schema element. Editing individual schema elements is far more useful than deleting the entire schema and reloading it.
Howard:
What are you talking about? A schema entry under cn=config has individual attribute values per schema element.
There is one attribute per schema element, that is, each objectClass or attribute defined in a schema is expressed by a single olcAttributeTypes or olcObjectClasses attribute, which contains a long, complex string which effectively defines other "attributes" of the schema object.
Would have been nicer if the hierarchy looked like this instead:
cn=config cn=schema cn={4}myshema cn={1}myattribute1 objectClass: olcAttributeTypes oid: 1.3.6.1.4... name: myAttr1 desc: This is a sample schema attribute equality: caseIgnoreMatch syntax: 1.3.6.1.4... cn={2}myattribute2 objectClass: olcAttributeTypes oid: 1.3.6.1.4... name: myAttr2 desc: This is a sample schema attribute 2 equality: caseIgnoreMatch syntax: 1.3.6.1.4... cn={3}myclass objectClass: olcObjectClass name: myAttr2 sup: top type: auxiliary must: cn={1}myattribute1 $ cn={2}myattribute2
rather than embedding all the attributes of a schema object in a monolithic string.
Then I would have agreed with you, Howard, that it would be easier to edit individual elements.
Editing individual schema elements is far more useful than deleting the entire schema and reloading it.
I find it a PITfA, since you have to respecify the whole schema object definition string each time...you can't just change the SUP or the description alone with the current structure, and searches are a pain too. But that is just IMO. When developing a schema iteratively, I find it's easier to have replace the whole thing when you make significant changes than to futz around trying to replace individual definitions, crafing intricate ldif modify directives.
As it turns out, cn=config is NOT stored in the database (though the documentation misleads about this)...it is stored in a special subdirectory (/etc/ldap/slapd.d/cn=config on my Ubuntu boxes). Maybe it's loaded into the db at startup...I don't know. Not sure this is much better than the old slapd.conf approach.
Then Ralf said:
But is very time consuming and resource intensive. E.g. checking for the presence of an AttributeType requires a full scan of the database if that AttributeType is not indexed, I think. Additionally this check would have to be done with the thread pool paused (i.e. no other operation would be processed) or at least with every database set into read only mode, to avoid races. I am not even sure if it would be possilbe with all backends types that we currently support (think of back-ldap or back-meta).
Hmmm...sounds like a classic referential integrity issue. Triggers can be your friend! ;-)
But the safest option we can currently offer, IMO.
Sounds like it...though that makes me shudder, having some dbms background.
And yes of course things can always be improved. Help with that is always welcome, BTW.
'Fraid I have limited time to devote to open source (read: unpaid) projects, and all my spare time is already committed to another open source project (eXist...XML database).
So the best I can do is to ask questions and offer suggestions at this time, wrt OpenLDAP.
Then again, I did recently write and donate an eXist extension module that lets you easily query an LDAP directory using high-level functions in an XQuery. Does that count? ;-)
--On Friday, December 12, 2008 3:18 PM -0500 Andrzej Jan Taramina andrzej@chaeron.com wrote:
As it turns out, cn=config is NOT stored in the database (though the documentation misleads about this)...it is stored in a special subdirectory (/etc/ldap/slapd.d/cn=config on my Ubuntu boxes). Maybe it's loaded into the db at startup...I don't know. Not sure this is much better than the old slapd.conf approach.
Incorrect. It is stored in a database of type "ldif". This may not be "bdb" or "hdb" that you are used to, but it is a type of OpenLDAP database backend.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah:
Incorrect. It is stored in a database of type "ldif". This may not be "bdb" or "hdb" that you are used to, but it is a type of OpenLDAP database backend.
You know, I tend to call databases, databases and flat files, flat files (or sometimes text files). There does tend to be a bit of a difference between the two.
And the *.ldif files found in /etc/ldap/slapd.d/cn=config/cn=schema are pretty much simple flat files, in a slightly modified ldif format, basically with some extra attributes defined at the end of the file.
And if they are "databases" (which I find hard to believe having looked at their internal structure), then one must question the advisability of using a .ldif filename suffix, which has very specific meaning.
Either way, sorry but I'm not buying it.
Andrzej Jan Taramina wrote:
Quanah:
Incorrect. It is stored in a database of type "ldif". This may not be "bdb" or "hdb" that you are used to, but it is a type of OpenLDAP database backend.
You know, I tend to call databases, databases and flat files, flat files (or sometimes text files). There does tend to be a bit of a difference between the two.
Well, database records are most times stored in files at OS level with a certain file format. The point is that access to cn=config works via LDAP like with any other OpenLDAP database backend.
And the *.ldif files found in /etc/ldap/slapd.d/cn=config/cn=schema are pretty much simple flat files, in a slightly modified ldif format, basically with some extra attributes defined at the end of the file.
Not sure what "flat files" means for you especially compared to "database files". LDIF is just a file format. And it was a good idea to use it since if anything goes wrong in cn=config which would prevent slapd from start up you can edit LDIF with a simple text editor to fix the issue.
And if they are "databases" (which I find hard to believe having looked at their internal structure), then one must question the advisability of using a .ldif filename suffix, which has very specific meaning.
No problem with the filename suffix .ldif because it is LDIF which could be parsed by 3rd-party LDIF parsers.
Either way, sorry but I'm not buying it.
...and I think you're plain wrong. ;-)
Ciao, Michael.
Andrzej Jan Taramina writes:
Howard: (...)
Editing individual schema elements is far more useful than deleting the entire schema and reloading it.
Depends on the schema, if I understand the distinction correctly. Once in a while I've needed to upgrade to a new version of a published schema. And to retire a schema. But that's rare enough that it'd be OK to stop slapd and update some cn=config files tree by hand.
Still, I do find it easier to edit slapd.conf than to modify cn=config. Among other things because there are fewer cases which don't work... We are moving to cn=config here anyway so we can update it dynamically, but we'll likely keep a slapd.conf around as "documentation" and maybe to regenerate the config from if we want to do something cn=config doesn't want to do.
I find it a PITfA, since you have to respecify the whole schema object definition string each time...you can't just change the SUP or the description alone with the current structure, and searches are a pain too. But that is just IMO.
If nobody has mentioned it yet - I hope you are aware of various LDAP browser/editors which can make this a bit easier. phpLDAPadmin, ldapvi, etc. Some ar listed here: http://www.openldap.org/faq/data/cache/271.html
Regarding searching - why does olcAttributeTypes have syntax Directory String instead of using attributeTypes' syntax Attribute Type Description? That would enable a search for (olcAttributeTypes=title), as well as ldapsearch -E'!mv=(olcAttributeTypes=title)' to extract just the definition of 'title'.
It needs to be a string type, a SUBSTR matching rule would help. Then we could search for "(olcAttributeTypes=*'title'*)".
There are LDAP tools that make editing LDAP entries easier -
As it turns out, cn=config is NOT stored in the database (though the documentation misleads about this)...
It's what slapd considers a "database" and which you configure as a "database" in slapd.conf. A slapd "database" is an instance of a backend, and a backend is a module which can store/provide LDAP data for slapd.
Only a few backends in slapd use what you'd normally call a "database". Do you have a suggestion for where it could have made clearer, and a brief wording for it?
it is stored in a special subdirectory (/etc/ldap/slapd.d/cn=config on my Ubuntu boxes). Maybe it's loaded into the db at startup...I don't know. Not sure this is much better than the old slapd.conf approach.
The point of cn=config is that you can update it over the LDAP protocol without stopping slapd. Therefore the configuration must be LDAP entries. I imagine rewriting LDAP config entries back to slapd.conf format would add an additional set of hassles.
[About checking if a schema element to be deleted is in use]
Hmmm...sounds like a classic referential integrity issue. Triggers can be your friend! ;-)
Sure it can be done. It's just that the code was not written with dynamic configuration in mind, and it's quite a job to update it all to support it. (I don't know how much, but I wouldn't be surprised if it's yet another issue which in the end leads to "maybe we should just start over"...)
--On Saturday, December 13, 2008 12:07 AM +0100 Hallvard B Furuseth h.b.furuseth@usit.uio.no wrote:
Sure it can be done. It's just that the code was not written with dynamic configuration in mind, and it's quite a job to update it all to support it. (I don't know how much, but I wouldn't be surprised if it's yet another issue which in the end leads to "maybe we should just start over"...)
AKA OpenLDAP 3.0? ;)
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount writes:
(I don't know how much, but I wouldn't be surprised if it's yet another issue which in the end leads to "maybe we should just start over"...)
AKA OpenLDAP 3.0? ;)
Or 4.0... The thought is getting more attractive, but OTOH several projects have died or almost died by trying to start over. Either because It Takes Longer Than You Think or because their grand new design had its own flaws which were only discovered after years of development.
Hallvard B Furuseth wrote:
Quanah Gibson-Mount writes:
(I don't know how much, but I wouldn't be surprised if it's yet another issue which in the end leads to "maybe we should just start over"...)
AKA OpenLDAP 3.0? ;)
Or 4.0... The thought is getting more attractive, but OTOH several projects have died or almost died by trying to start over. Either because It Takes Longer Than You Think or because their grand new design had its own flaws which were only discovered after years of development.
Agreed. We talk fondly of throwing out all of libldap and starting over, but getting everybody to adopt any new API would be extremely difficult. I'd like to replace liblber with a streaming implementation, but for the most part the current one works well enough now and there's very few real-world users who would see any noticeable difference.
At this point I think we would do well enough with refactoring along the lines from 2.0 -> 2.1; things that improve the code without changing the structure so much as to make it unrecognizable.
Hallvard:
If nobody has mentioned it yet - I hope you are aware of various LDAP browser/editors which can make this a bit easier. phpLDAPadmin, ldapvi, etc. Some ar listed here: http://www.openldap.org/faq/data/cache/271.html
Yup....I use the command line utilities (ldapsearch/add/modify) a lot with some routines that generate appropriate ldif files from raw XML. I also have found luma to be quite handy as a GUI-based LDAP viewer/editor. Even managed to get it set up so that I can walk and modify the cn=config entries, including schemas.
There are LDAP tools that make editing LDAP entries easier -
Yup...like XSLT and XQuery! ;-)
It's what slapd considers a "database" and which you configure as a "database" in slapd.conf. A slapd "database" is an instance of a backend, and a backend is a module which can store/provide LDAP data for slapd.
Then the term "database" is being misused rather badly, IMO. They are just ldif-formatted flat files.
Only a few backends in slapd use what you'd normally call a "database". Do you have a suggestion for where it could have made clearer, and a brief wording for it?
Backend storage mechanism.
The point of cn=config is that you can update it over the LDAP protocol without stopping slapd. Therefore the configuration must be LDAP entries.
Which can be handy at times...just not for doing iterative custom schema development. ;-)
I imagine rewriting LDAP config entries back to slapd.conf format would add an additional set of hassles.
Yup...not really interested in going that route.
Sure it can be done. It's just that the code was not written with dynamic configuration in mind, and it's quite a job to update it all to support it. (I don't know how much, but I wouldn't be surprised if it's yet another issue which in the end leads to "maybe we should just start over"...)
"Starting over" is always a very, very bad idea.
The dynamic cn=config stuff is a bit young and raw around the edges, but the concept is a good one.
Andrzej Jan Taramina wrote:
Hallvard:
If nobody has mentioned it yet - I hope you are aware of various LDAP browser/editors which can make this a bit easier. phpLDAPadmin, ldapvi, etc. Some ar listed here: http://www.openldap.org/faq/data/cache/271.html
Yup....I use the command line utilities (ldapsearch/add/modify) a lot with some routines that generate appropriate ldif files from raw XML. I also have found luma to be quite handy as a GUI-based LDAP viewer/editor. Even managed to get it set up so that I can walk and modify the cn=config entries, including schemas.
There are LDAP tools that make editing LDAP entries easier -
Yup...like XSLT and XQuery! ;-)
It's what slapd considers a "database" and which you configure as a "database" in slapd.conf. A slapd "database" is an instance of a backend, and a backend is a module which can store/provide LDAP data for slapd.
Then the term "database" is being misused rather badly, IMO. They are just ldif-formatted flat files.
At the risk of sparking a semantic war, a database is simply a collection of records. How they are formatted or stored is an implementation detail.
Russell Jackson wrote:
Andrzej Jan Taramina wrote:
Then the term "database" is being misused rather badly, IMO. They are just ldif-formatted flat files.
At the risk of sparking a semantic war, a database is simply a collection of records. How they are formatted or stored is an implementation detail.
Exactly. "database" != "RDBMS", no matter how much the RDBMS folks like to claim otherwise.
Andrzej Jan Taramina wrote:
Hallvard:
Only a few backends in slapd use what you'd normally call a "database". Do you have a suggestion for where it could have made clearer, and a brief wording for it?
Backend storage mechanism.
This does not say much. In the way I understand your wording this definition is plain wrong since it lacks the functional model of a database. And that's exactly what's missing in your understanding.
The point of cn=config is that you can update it over the LDAP protocol without stopping slapd. Therefore the configuration must be LDAP entries.
Which can be handy at times...just not for doing iterative custom schema development. ;-)
I usually do schema development in a dedicated test slapd instance by simply editing schema text files included with slapd.conf. I'd use cn=config only if there's a real need for dynamic re-configuration of a long-running slapd instance for minimizing downtime.
Ciao, Michael.
Andrzej Jan Taramina writes:
Hallvard:
Only a few backends in slapd use what you'd normally call a "database". Do you have a suggestion for where it could have made clearer, and a brief wording for it?
Backend storage mechanism.
Acutally what I meant to ask was, where could the doc describe what it means with "database" so that you would have caught on?
The terminology comes from the Umich LDAP implementation (which had 3 backends, I think) and is over a decade old now. It's a bit late to change it on everyone now.
Andrzej Jan Taramina wrote:
Howard:
What are you talking about? A schema entry under cn=config has individual attribute values per schema element.
There is one attribute per schema element, that is, each objectClass or attribute defined in a schema is expressed by a single olcAttributeTypes or olcObjectClasses attribute, which contains a long, complex string which effectively defines other "attributes" of the schema object.
Would have been nicer if the hierarchy looked like this instead:
cn=config cn=schema cn={4}myshema cn={1}myattribute1 objectClass: olcAttributeTypes oid: 1.3.6.1.4... name: myAttr1 desc: This is a sample schema attribute equality: caseIgnoreMatch syntax: 1.3.6.1.4... cn={2}myattribute2 objectClass: olcAttributeTypes oid: 1.3.6.1.4... name: myAttr2 desc: This is a sample schema attribute 2 equality: caseIgnoreMatch syntax: 1.3.6.1.4... cn={3}myclass objectClass: olcObjectClass name: myAttr2 sup: top type: auxiliary must: cn={1}myattribute1 $ cn={2}myattribute2
rather than embedding all the attributes of a schema object in a monolithic string.
Then I would have agreed with you, Howard, that it would be easier to edit individual elements.
What you've described is similar to the original design, 3 years ago, but it proved unworkable. You can read through the openldap-devel archives from that time period for more background on the decisions. As I recall, we ran into trouble with schema extensions and a few other elements that didn't lend themselves well to being treated as distinct attributes.
You should learn a bit more about why and how things work, before suggesting they be changed.
openldap-software@openldap.org
-
Andrzej Jan Taramina -
Hallvard B Furuseth -
Howard Chu -
hyc＠symas.com
-
Michael Ströder -
Quanah Gibson-Mount -
Russell Jackson