Hello Hallvard,
Am 19.06.2007 um 17:22 schrieb Hallvard B Furuseth:
Dagobert Michelsen writes:
- each Perl backend database has a separate Perl interpreter
- each connection gets an interpreter cloned from the specific backend. Interpreters are recycled in a pool for performance
- concurrent requests to different interpreters can run concurrently in separate threads
Is there a max number of Perl interpreters per backend?
I suppose you mean "per database". In my understanding there are different backends coded in slapd/back-* where the back-perl I am working on is one backend which can be instantiated in slapd.conf. There may be more databases configured for the Perl backend.
There is exactly one interpreter in use for each active connection and each configured back-perl database because allocation is done on connection_init.
The maximum number of interpreters is the maximum count of connections (dtblsize) multiplied with the number of back-perl databases.
What happens if there are more long-lived connections than that?
As this is the maximum this can never be the case.
Does a connection release its Perl interpreter to the pool if a few requests on the connection use the Perl database and then the rest of the requests use a non-Perl database?
Perl interpreters are allocated when a connection to OpenLDAP is made and released to the pool when the connection is terminated.
This method has some advantages: - As access to the Perl interpreter must be exclusive there is no contention because no two operations can be active for one connection at the same time - Data from a connection is automatically available in all method invocations for this connection as the calls are all directed to the same Perl interpreter
The disadvange is that there is no data sharing between connections. Implementing data sharing with this model is quite complex and involves attaching (Perl-)magic to shared variables which implements mutex'ed access between interpreters.
Currently Perl interpreters are never freed once they are released. Releasing of more than, say, 5 idle interpreters may be implemented as an optimization. I look into that if I have a fully working prototype.
Are you using threads created by Perl, or ldap_pvt_thread_create(), or are you somehow giving the interpreters short-lived tasks via ldap_pvt_thread_pool_submit()? There are some slapd features which can only be used if you do the latter, since that creates a "thread context" which holds thread-specific variables.
I don't do any explicit thread management. All threads have been set up by OpenLDAP when the backend entrypoints are called, so I assume that this is done right :-)
- argument passing is done in a style similar to Net::LDAP, parsing is no longer needed as attributes and values are separately stored
You might not even need to do that. Could wrap an Entry in a class which translates slapd attributes to Perl lists on demand.
This is possible, however not trivial: The C data structures would be implemented as blessed scalars containing the C pointer. Access method could then convert the data on demand. However, the C pointer must be some kind of "weak" pointer which is only valid in the context of this method invocation. Otherwise the scalar with the C reference could be stored in the interpreter and deferenced later in another method entry where the C data has already been freed. This all is a bit complex and can be done later without API change if proven useful.
As the method usually does something with the parameters the data is needed anyway, so savings here should be minimal if not negative. Another thing is access to seldom used data like connection parameters or operation details where transformation to your proposed structure would indeed be useful.
- switching to old-backend-compatible-mode is done when no connection_init method is found
That sounds like a hack, one might not be interested in defining a connection_init method anyway.
In my current implementation it is mandatory to implement connection_init. The returned value must be a reference to an object on which the methods for the operations are called (like search, compare, etc.)
If you have a usecase for an application without connection_init I would like to hear it.
The new backend is not 100% compatible to the old backend due to the lack of data sharing. This is a complex issue but maybe solvable with the help of the mod_perl code. Alternatively the code could fall back to a single interpreter when compatibility mode is on.
Unless the incompatibilities are quite small:
I think it would be a bad idea to automatically invoke an "almost- compatible" mode. Then a number of old Perl programs can break mysteriously, and maybe munge whatever data they are maintaining before it is discovered.
I suggest you make the new database fail to configure if provided an old Perl program and vice versa.
Ok. I see the importance now and make the changes accordingly. To summarize: - Exit with fault if old Perl-module is used with new backend - Provide compatibility-module which must be enabled explicitly - Exit with fault if new Perl-module is used with old backend
I also suggest you do not let compatibility get in the way of good design decisions for the new code. If good design leads to bad compatibility, we could always put the old back-perl in contrib/back-oldperl or something.
Including the old backend is something I really would like to avoid as this means adapting autoconf-stuff and be confusing.
It is still not possible to do all things from Perl which can be done in C. Exposing more datastructures and functions to Perl may even make it possible to write overlays in Perl in some distant future.
Both are possible, in particular if you provide Perl types as wrappers around the basic C types instead of translating them to arrays or hashes before delivering them to Perl. (If you've translated them it's a bit of a problem to call a number of C functions because you've lost the original data structure which you would pass _to_ the C function.
This is true but otherwise you need "weak" references or induce the possibility of subtle bugs with dangling pointers. If this feature is provided it must be as safe as possible in my opinion.
Or the user has made updates in Perl structures which are not reflected in the C structures they came from. I looked into doing this with Python once, but punted since the Python C API #includes a header with Python autoconf variables which conflict with slapd's autoconf variables.
Do you have some design notes of this left which you can send me?
When I have something which is usable for testing I'll post.
Best regards
-- Dagobert
BTW: For my personal use I set up OpenGrok for browsing the OpenLDAP sources which I really enjoy. Feel free to have a look at http://openldap.baltic-online.de