Here's a project for someone fairly ambitious:
The goal is to be able to dynamically add support for new syntaxes to slapd simply by feeding in their specifications. Typically this would mean feeding in an ABNF description from an IETF document, though it may also mean feeding in ASN.1 documents. E.g., we want to be able to insert (using ldapmodify) the ABNF into an LDAP entry somewhere and have it automatically converted into an executable parser.
There are a couple of tools out there for working with ABNF. The IETF actually uses this one http://www.coasttocoastresearch.com/Apg5_0/Apg5_0_description.htm for validating IETF drafts. There's also this one http://www.2p.cz/en/abnf_gen
I don't think either of these are suitable as they exist, because they don't generate output that is embeddable and directly executable. I guess you could feed the generated C code into an embedded C interpreter, but that seems like a ridiculously wasteful solution. Most of these existing tools will analyze a grammar and spit out source code to recognize it, but we don't want the source code, we just want the recognition action.
I think a more practical route will involve embedding a Forth engine into the OpenLDAP code. There's already been a fair amount of work done with BNF in Forth, so it shouldn't be too hard to get started: http://www.zetetics.com/bj/papers/bnfparse.htm http://www-personal.umich.edu/~williams/archive/forth/strings/index.html http://www-personal.umich.edu/~williams/archive/forth/strings/expr.html
The nice thing about Forth is that the core language is tiny, it's extremely portable, it's extremely extensible, and any implementation is inherently both an interpreter and a compiler. As interpreted languages goes, it beats pretty much every other common scripting language for memory and CPU efficiency.
It ought to be straightforward to convert a grammar into a set of Forth words to recognize the grammar, for a slapd syntax validator. I'm not sure yet what we would do for Prettifiers or Normalizers, nor what to do for Matching rules. We could simply accept them as Forth source code. Personally I haven't written Forth in over 20 years but it seems to me that a TIL is the most suitable type of language for the job.
On the whole it doesn't seem like new syntaxes pop up in LDAP specs very often. Some may say this is because they're not really needed. I believe they've been sorely needed, and people have just avoided them because they required too much work to properly support in existing implementations. We can fix that.
Howard Chu wrote:
Here's a project for someone fairly ambitious:
The goal is to be able to dynamically add support for new syntaxes to slapd simply by feeding in their specifications. Typically this would mean feeding in an ABNF description from an IETF document, though it may also mean feeding in ASN.1 documents. E.g., we want to be able to insert (using ldapmodify) the ABNF into an LDAP entry somewhere and have it automatically converted into an executable parser.
I think a more practical route will involve embedding a Forth engine into the OpenLDAP code. There's already been a fair amount of work done with BNF in Forth, so it shouldn't be too hard to get started: http://www.zetetics.com/bj/papers/bnfparse.htm http://www-personal.umich.edu/~williams/archive/forth/strings/index.html http://www-personal.umich.edu/~williams/archive/forth/strings/expr.html
The nice thing about Forth is that the core language is tiny, it's extremely portable, it's extremely extensible, and any implementation is inherently both an interpreter and a compiler. As interpreted languages goes, it beats pretty much every other common scripting language for memory and CPU efficiency.
A parser that spits out Forth code would also make a good mechanism for the component-based matching functions. Even though we have dynamically loadable modules, relying on ASN.1 compilers like eSNACC is pretty clumsy. (And of course, according to the ITS, eSNACC is currently broken anyway.) This sort of thing ought to Just Work, without end-users needing any other development tools. I.e., LDAPadd an entry with ASN.1 structure definitions and immediately be able to perform component matching on the structures.
It ought to be straightforward to convert a grammar into a set of Forth words to recognize the grammar, for a slapd syntax validator. I'm not sure yet what we would do for Prettifiers or Normalizers, nor what to do for Matching rules. We could simply accept them as Forth source code. Personally I haven't written Forth in over 20 years but it seems to me that a TIL is the most suitable type of language for the job.
On the whole it doesn't seem like new syntaxes pop up in LDAP specs very often. Some may say this is because they're not really needed. I believe they've been sorely needed, and people have just avoided them because they required too much work to properly support in existing implementations. We can fix that.
Howard Chu wrote:
On the whole it doesn't seem like new syntaxes pop up in LDAP specs very often. Some may say this is because they're not really needed. I believe they've been sorely needed, and people have just avoided them because they required too much work to properly support in existing implementations. We can fix that.
This topic is somehwat interesting. I've banged my head against the wall several times when implementing schema support in web2ldap. The big question was where to draw the line between handling syntaxes and special attribute types, especially when having to override incorrect schema information retrieved from the server.
I have no strong opinion just some thoughts regarding how to dynamically make use of it at the client side:
Something like (an improved) slapo-constraint will cover most cases for which one might also consider a syntax, e.g. limited set of option values for implementing the typical select lists. Or a certain string format etc. AFAIK slapo-constraint shipped in RE24 only supports regex checking defined in slapd.conf. Dynamic option value/description lists could be easily defined by defining an LDAP URL which searches the option values. IIRC someone is already working in this direction to improve slapo-constraint. (A base class for this is in web2ldap's plugin API now). But currently the client cannot query the constraints and adjust the UI accordingly in a vendor independent way. The client could read cn=config if back-config is deployed though.
A separate syntaxes is only needed if there's a strong need for distinct matching rules or normalization etc. Also a client has to implement the syntax. IMO this is the reason for spec authors to avoid syntaxes.
So I'm thinking about whether it would be worth to extend the subschema declaration of attribute types to define additional constraints. A client could interpret this and act accordingly. Well, a long way to go through standardization off course...
Any thoughts?
Ciao, Michael.
On Tue, Nov 27, 2007 at 11:46:45AM +0100, Michael Ströder wrote:
Dynamic option value/description lists could be easily defined by defining an LDAP URL which searches the option values. IIRC someone is already working in this direction to improve slapo-constraint.
I contributed something that sounds like this, but it's burried in ITS: http://www.openldap.org/its/index.cgi/Contrib?id=4987;expression=slapo-const...
Emmanuel Dreyfus wrote:
On Tue, Nov 27, 2007 at 11:46:45AM +0100, Michael Ströder wrote:
Dynamic option value/description lists could be easily defined by defining an LDAP URL which searches the option values. IIRC someone is already working in this direction to improve slapo-constraint.
I contributed something that sounds like this, but it's burried in ITS: http://www.openldap.org/its/index.cgi/Contrib?id=4987;expression=slapo-const...
Yes. I expect to get this reviewed and into CVS later this week. There are several contrib items that need reviewing/testing still. But at the moment, there are several open bug reports that still need to be closed; i.e., it looks like their fixes have been tested and released but the bug reports still haven't been updated. I'd like to get 2.4.7 out this week, but we need to make sure everything closed in 2.3.39/2.4.6 is cleared off first.
Michael Ströder wrote:
Howard Chu wrote:
On the whole it doesn't seem like new syntaxes pop up in LDAP specs very often. Some may say this is because they're not really needed. I believe they've been sorely needed, and people have just avoided them because they required too much work to properly support in existing implementations. We can fix that.
This topic is somehwat interesting. I've banged my head against the wall several times when implementing schema support in web2ldap. The big question was where to draw the line between handling syntaxes and special attribute types, especially when having to override incorrect schema information retrieved from the server.
Overriding the server? That doesn't sound good...
I have no strong opinion just some thoughts regarding how to dynamically make use of it at the client side:
Something like (an improved) slapo-constraint will cover most cases for which one might also consider a syntax, e.g. limited set of option values for implementing the typical select lists. Or a certain string format etc. AFAIK slapo-constraint shipped in RE24 only supports regex checking defined in slapd.conf. Dynamic option value/description lists could be easily defined by defining an LDAP URL which searches the option values. IIRC someone is already working in this direction to improve slapo-constraint. (A base class for this is in web2ldap's plugin API now). But currently the client cannot query the constraints and adjust the UI accordingly in a vendor independent way. The client could read cn=config if back-config is deployed though.
Right, slapo-constraint is fine if you just need a constrained form of a string or text-based value. It's no help for more complex data types, like ASN.1 structures or multimedia objects (images, audio clips, etc.).
I suppose it's debatable whether it's a good idea to store a lot of these things in a directory. But in a lot of ways, it's better than storing somewhere else. E.g., an embedded address book in a cellphone stores contact info along with an image of the person and a custom ringtone. In the past we've recommended that directories only store references (URLs) to these other blobs, but that makes deployment and management much harder. Use LDAP for everything and you only need one protocol for syncing the cellphone to a main directory. (Of course, OpenLDAP has been embedded in cellphones before, but with open architectures like Google is pushing out with Android now, it becomes even more interesting.) On the flip side, it may be OK to just declare things to be blobs (octetString) in LDAP and let the application worry about figuring out what it really is. But that requires a lot of specialized parsing intelligence to be present redundantly in every application, instead of just a single generalized parser. It works, most apps today get along fine, but it's wasteful.
A separate syntaxes is only needed if there's a strong need for distinct matching rules or normalization etc. Also a client has to implement the syntax. IMO this is the reason for spec authors to avoid syntaxes.
Yes, this was always one of the hassles of setting up X.500 clients. But I think it's unavoidable if you want everything to integrate properly. One of the reasons for exploring the route of implementing dynamic syntaxes in a language like Forth is that you can have a complete Forth implementation in only 30K or less of object code. We can bundle it under libldap and use it uniformly from both slapd and from clients.
So I'm thinking about whether it would be worth to extend the subschema declaration of attribute types to define additional constraints. A client could interpret this and act accordingly. Well, a long way to go through standardization off course...
Any thoughts?
Constraints are only a partial solution. It's not worth it (to me) to go through all of that standardization effort for less than a complete solution.
-
Emmanuel Dreyfus -
Howard Chu -
Michael Ströder