<quote who="hallvard@OpenLDAP.org">
Update of /repo/OpenLDAP/pkg/ldap/servers/slapd/back-perl
Modified Files: SampleLDAP.pm 1.12 -> 1.13
I've run Perl::Critic over this and fixed/updated to latest "Perl Best Practices". We might as well have an up to date sample.
Do you mind if I commit my changes?
cpan Perl::Critic perlcritic -1 SampleLDAP.pm
Then fixed accordingly.
Thanks,
Gavin.
Gavin Henry writes:
I've run Perl::Critic over this and fixed/updated to latest "Perl Best Practices". We might as well have an up to date sample.
Do you mind if I commit my changes?
My impression is that back-perl hovers on the verge of code rot, and any change is likely for the better.
As for RE23, I don't know what Perl::Critic is, but as close we are to a release I suggest a fix to SampleLDAP.pm can wait for 2.3.37.
<quote who="Hallvard B Furuseth">
Gavin Henry writes:
I've run Perl::Critic over this and fixed/updated to latest "Perl Best Practices". We might as well have an up to date sample.
Do you mind if I commit my changes?
My impression is that back-perl hovers on the verge of code rot, and any change is likely for the better.
;-)
As for RE23, I don't know what Perl::Critic is, but as close we are to a release I suggest a fix to SampleLDAP.pm can wait for 2.3.37.
I agree.
-- Regards, Hallvard
Hello,
Von Hallvard B Furuseth (Fri, 15 Jun 2007 16:18:34 +0200):
My impression is that back-perl hovers on the verge of code rot, and any change is likely for the better.
I have a reworked Perl backend almost finished. Please don't kick the backend out now, it should be ready in a few weeks.
Best regards
-- Dagobert
<quote who="Dagobert Michelsen">
Hello,
Von Hallvard B Furuseth (Fri, 15 Jun 2007 16:18:34 +0200):
My impression is that back-perl hovers on the verge of code rot, and any change is likely for the better.
I have a reworked Perl backend almost finished. Please don't kick the backend out now, it should be ready in a few weeks.
Best regards
-- Dagobert
I'm pretty sure that's not the plan. What sort of things are you doing?
Gavin.
Hello,
Am 15.06.2007 um 16:52 schrieb Gavin Henry:
<quote who="Dagobert Michelsen"> > I have a reworked Perl backend almost finished. Please don't > kick the backend out now, it should be ready in a few weeks.
I'm pretty sure that's not the plan. What sort of things are you doing?
The current Perl backend is practically unusable because of the following reasons:
- every call goes through a single mutex so the backend is essentially singlethreaded - the backend has no idea of connections so requests can not be associated to a specific bind - binary attributes can not be passed as only zero-terminated strings are passed to perl - argument passing in general is cumbersome as the detailed structure is lost via entry2str - important entry points like 'extended' is not implemented which is needed for ldappasswd - there is no support for multiple databases with Perl backend
The reworked backend works as follows:
- each Perl backend database has a separate Perl interpreter - each connection gets an interpreter cloned from the specific backend. Interpreters are recycled in a pool for performance - concurrent requests to different interpreters can run concurrently in separate threads - connections are correctly tracked with individual objects - binary attributed are correctly passed in Perl scalars - argument passing is done in a style similar to Net::LDAP, parsing is no longer needed as attributes and values are separately stored - named arguments make future extensions possible - additional information about the connection is optionally passed as named parameter. It may be advisable later to used magic scalars allowing access to all internal data without the overhead of processing the arguments ahead of time. - entry points for connection_init, connection_destroy, extended ops and unbind are implemented. - multiple databases are supported. - unimplemented entrypoints automatically return 'unwilling to perform' - switching to old-backend-compatible-mode is done when no connection_init method is found
What is missing:
The new backend is not 100% compatible to the old backend due to the lack of data sharing. This is a complex issue but maybe solvable with the help of the mod_perl code. Alternatively the code could fall back to a single interpreter when compatibility mode is on. I am trying to avoid legacy compatibility code in the C part and move that to a Perl module interposed between back-perl and the oldstyle module. To make this as easy both for legacy- and new users is a thing I am still working on. Personally I dislike the idea of breaking compatibility, however I don't know how much people actually use the Perl backend broken as it is. The new API is designed to be as extendable as possible to avoid incompatible changes in the future.
The 'abandon' entry point is not implemented. That would imply signaling between threads I haven't done yet.
I see the use of the Perl backend twofold:
(1) rapid prototyping for functionality later implemented in C (2) implement complex tasks which would be too expensive (in terms of time) to do in C
It is still not possible to do all things from Perl which can be done in C. Exposing more datastructures and functions to Perl may even make it possible to write overlays in Perl in some distant future.
Best regards
-- Dagobert
--On June 15, 2007 5:28:36 PM +0200 Dagobert Michelsen dam@baltic-online.de wrote:
What is missing:
The new backend is not 100% compatible to the old backend due to the lack of data sharing. This is a complex issue but maybe solvable with the help of the mod_perl code. Alternatively the code could fall back to a single interpreter when compatibility mode is on. I am trying to avoid legacy compatibility code in the C part and move that to a Perl module interposed between back-perl and the oldstyle module. To make this as easy both for legacy- and new users is a thing I am still working on. Personally I dislike the idea of breaking compatibility, however I don't know how much people actually use the Perl backend broken as it is. The new API is designed to be as extendable as possible to avoid incompatible changes in the future.
I think very few people use the current back-perl. I would be fine if the new one dropped any attempt at compatibility going backwards if it was included in the 2.4 release. As you noted, as it exists now, it is pretty much unusable, and anyone wanting to use it in 2.4 should update. Just my 2c. ;)
--Quanah
-- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
The new backend is not 100% compatible to the old backend due to the lack of data sharing. This is a complex issue but maybe solvable with the help of the mod_perl code.
Everyone I'd seen using back-perl was using private patches that were completely incompatible anyway. ...And most were abandoned some time during the 2.1.x era.
The single-threaded nature of the existing back-perl makes it mostly useless in the modern slapds with overlays, so anything that fixes this is a huge improvement.
But if we're changing APIs... please consider a more callback oriented system, at least for the search operation. Last I looked at it, lots of time was wasted allocating and deallocating memory for the search op return value. Returning one object at a time with a separate call from perl back into slapd was a huge performance win.
Matthew Backes Symas Corporation mbackes@symas.com
Hello,
Am 17.06.2007 um 06:53 schrieb Matthew Backes:
The new backend is not 100% compatible to the old backend due to the lack of data sharing. This is a complex issue but maybe solvable with the help of the mod_perl code.
But if we're changing APIs... please consider a more callback oriented system, at least for the search operation.
Taking 'send_search_entry' out wouldn't be that hard. I already changed the return format from string to a format more suitable for conversion to an entry and which avoids base64 for binary values. I don't see any more uses for callbacks. If you have suggestions on more uses let me know.
Last I looked at it, lots of time was wasted allocating and deallocating memory for the search op return value. Returning one object at a time with a separate call from perl back into slapd was a huge performance win.
Good to know. I'll fix that.
Thanks
-- Dagobert
Dagobert Michelsen writes:
I have a reworked Perl backend almost finished. Please don't kick the backend out now, (...)
Relax, I was not suggesting that. Something needed to be done. Glad to hear you are doing it.
- each Perl backend database has a separate Perl interpreter
- each connection gets an interpreter cloned from the specific backend. Interpreters are recycled in a pool for performance
- concurrent requests to different interpreters can run concurrently in separate threads
Is there a max number of Perl interpreters per backend? What happens if there are more long-lived connections than that?
Does a connection release its Perl interpreter to the pool if a few requests on the connection use the Perl database and then the rest of the requests use a non-Perl database?
Are you using threads created by Perl, or ldap_pvt_thread_create(), or are you somehow giving the interpreters short-lived tasks via ldap_pvt_thread_pool_submit()? There are some slapd features which can only be used if you do the latter, since that creates a "thread context" which holds thread-specific variables.
- argument passing is done in a style similar to Net::LDAP, parsing is no longer needed as attributes and values are separately stored
You might not even need to do that. Could wrap an Entry in a class which translates slapd attributes to Perl lists on demand.
- switching to old-backend-compatible-mode is done when no connection_init method is found
That sounds like a hack, one might not be interested in defining a connection_init method anyway.
The new backend is not 100% compatible to the old backend due to the lack of data sharing. This is a complex issue but maybe solvable with the help of the mod_perl code. Alternatively the code could fall back to a single interpreter when compatibility mode is on.
Unless the incompatibilities are quite small:
I think it would be a bad idea to automatically invoke an "almost- compatible" mode. Then a number of old Perl programs can break mysteriously, and maybe munge whatever data they are maintaining before it is discovered.
I suggest you make the new database fail to configure if provided an old Perl program and vice versa. And provide a trivial way to update a Perl program work both with the old back-perl and with your compat mode. One could insert a line &slapd::compat if defined &slapd::compat; or something.
I also suggest you do not let compatibility get in the way of good design decisions for the new code. If good design leads to bad compatibility, we could always put the old back-perl in contrib/back-oldperl or something.
It is still not possible to do all things from Perl which can be done in C. Exposing more datastructures and functions to Perl may even make it possible to write overlays in Perl in some distant future.
Both are possible, in particular if you provide Perl types as wrappers around the basic C types instead of translating them to arrays or hashes before delivering them to Perl. (If you've translated them it's a bit of a problem to call a number of C functions because you've lost the original data structure which you would pass _to_ the C function. Or the user has made updates in Perl structures which are not reflected in the C structures they came from. I looked into doing this with Python once, but punted since the Python C API #includes a header with Python autoconf variables which conflict with slapd's autoconf variables.
Hello Hallvard,
Am 19.06.2007 um 17:22 schrieb Hallvard B Furuseth:
Dagobert Michelsen writes:
- each Perl backend database has a separate Perl interpreter
- each connection gets an interpreter cloned from the specific backend. Interpreters are recycled in a pool for performance
- concurrent requests to different interpreters can run concurrently in separate threads
Is there a max number of Perl interpreters per backend?
I suppose you mean "per database". In my understanding there are different backends coded in slapd/back-* where the back-perl I am working on is one backend which can be instantiated in slapd.conf. There may be more databases configured for the Perl backend.
There is exactly one interpreter in use for each active connection and each configured back-perl database because allocation is done on connection_init.
The maximum number of interpreters is the maximum count of connections (dtblsize) multiplied with the number of back-perl databases.
What happens if there are more long-lived connections than that?
As this is the maximum this can never be the case.
Does a connection release its Perl interpreter to the pool if a few requests on the connection use the Perl database and then the rest of the requests use a non-Perl database?
Perl interpreters are allocated when a connection to OpenLDAP is made and released to the pool when the connection is terminated.
This method has some advantages: - As access to the Perl interpreter must be exclusive there is no contention because no two operations can be active for one connection at the same time - Data from a connection is automatically available in all method invocations for this connection as the calls are all directed to the same Perl interpreter
The disadvange is that there is no data sharing between connections. Implementing data sharing with this model is quite complex and involves attaching (Perl-)magic to shared variables which implements mutex'ed access between interpreters.
Currently Perl interpreters are never freed once they are released. Releasing of more than, say, 5 idle interpreters may be implemented as an optimization. I look into that if I have a fully working prototype.
Are you using threads created by Perl, or ldap_pvt_thread_create(), or are you somehow giving the interpreters short-lived tasks via ldap_pvt_thread_pool_submit()? There are some slapd features which can only be used if you do the latter, since that creates a "thread context" which holds thread-specific variables.
I don't do any explicit thread management. All threads have been set up by OpenLDAP when the backend entrypoints are called, so I assume that this is done right :-)
- argument passing is done in a style similar to Net::LDAP, parsing is no longer needed as attributes and values are separately stored
You might not even need to do that. Could wrap an Entry in a class which translates slapd attributes to Perl lists on demand.
This is possible, however not trivial: The C data structures would be implemented as blessed scalars containing the C pointer. Access method could then convert the data on demand. However, the C pointer must be some kind of "weak" pointer which is only valid in the context of this method invocation. Otherwise the scalar with the C reference could be stored in the interpreter and deferenced later in another method entry where the C data has already been freed. This all is a bit complex and can be done later without API change if proven useful.
As the method usually does something with the parameters the data is needed anyway, so savings here should be minimal if not negative. Another thing is access to seldom used data like connection parameters or operation details where transformation to your proposed structure would indeed be useful.
- switching to old-backend-compatible-mode is done when no connection_init method is found
That sounds like a hack, one might not be interested in defining a connection_init method anyway.
In my current implementation it is mandatory to implement connection_init. The returned value must be a reference to an object on which the methods for the operations are called (like search, compare, etc.)
If you have a usecase for an application without connection_init I would like to hear it.
The new backend is not 100% compatible to the old backend due to the lack of data sharing. This is a complex issue but maybe solvable with the help of the mod_perl code. Alternatively the code could fall back to a single interpreter when compatibility mode is on.
Unless the incompatibilities are quite small:
I think it would be a bad idea to automatically invoke an "almost- compatible" mode. Then a number of old Perl programs can break mysteriously, and maybe munge whatever data they are maintaining before it is discovered.
I suggest you make the new database fail to configure if provided an old Perl program and vice versa.
Ok. I see the importance now and make the changes accordingly. To summarize: - Exit with fault if old Perl-module is used with new backend - Provide compatibility-module which must be enabled explicitly - Exit with fault if new Perl-module is used with old backend
I also suggest you do not let compatibility get in the way of good design decisions for the new code. If good design leads to bad compatibility, we could always put the old back-perl in contrib/back-oldperl or something.
Including the old backend is something I really would like to avoid as this means adapting autoconf-stuff and be confusing.
It is still not possible to do all things from Perl which can be done in C. Exposing more datastructures and functions to Perl may even make it possible to write overlays in Perl in some distant future.
Both are possible, in particular if you provide Perl types as wrappers around the basic C types instead of translating them to arrays or hashes before delivering them to Perl. (If you've translated them it's a bit of a problem to call a number of C functions because you've lost the original data structure which you would pass _to_ the C function.
This is true but otherwise you need "weak" references or induce the possibility of subtle bugs with dangling pointers. If this feature is provided it must be as safe as possible in my opinion.
Or the user has made updates in Perl structures which are not reflected in the C structures they came from. I looked into doing this with Python once, but punted since the Python C API #includes a header with Python autoconf variables which conflict with slapd's autoconf variables.
Do you have some design notes of this left which you can send me?
When I have something which is usable for testing I'll post.
Best regards
-- Dagobert
BTW: For my personal use I set up OpenGrok for browsing the OpenLDAP sources which I really enjoy. Feel free to have a look at http://openldap.baltic-online.de
Dagobert Michelsen writes:
Am 19.06.2007 um 17:22 schrieb Hallvard B Furuseth:
Is there a max number of Perl interpreters per backend?
I suppose you mean "per database".
Right.
There is exactly one interpreter in use for each active connection and each configured back-perl database because allocation is done on connection_init.
Hmm. That sounds like adding a perl database can have a noticeable impact on performance (and memory use?) even if it is rarely queried. In particular if it calls a user-defined connection_init Perl method. Ought to test and see, I guess...
(...) The disadvange is that there is no data sharing between connections. Implementing data sharing with this model is quite complex and involves attaching (Perl-)magic to shared variables which implements mutex'ed access between interpreters.
Perl has "use threads::shared", but maybe that only works for threads created by Perl.
- argument passing is done in a style similar to Net::LDAP, parsing is no longer needed as attributes and values are separately stored
You might not even need to do that. Could wrap an Entry in a class which translates slapd attributes to Perl lists on demand.
This is possible, however not trivial: The C data structures would be implemented as blessed scalars containing the C pointer. Access method could then convert the data on demand.
Yes...
However, the C pointer must be some kind of "weak" pointer which is only valid in the context of this method invocation. Otherwise the scalar with the C reference could be stored in the interpreter and deferenced later in another method entry where the C data has already been freed.
Yes. A number of slapd stuctures have a .<something>_private member which can be used for this, and there are also some callback methods to clean up. Overlays can make this rather hairy though, I still haven't gotten around to learn how that works. Together, I think these can be used to implement the "weak" pointers. I have not tried that myself though. And if it can be done later there is no hurry to do it now.
As the method usually does something with the parameters the data is needed anyway, so savings here should be minimal if not negative.
Depends on the API and which data we are talking about, I guess. As you say:
Another thing is access to seldom used data like connection parameters or operation details where transformation to your proposed structure would indeed be useful.
- switching to old-backend-compatible-mode is done when no connection_init method is found
That sounds like a hack, one might not be interested in defining a connection_init method anyway.
In my current implementation it is mandatory to implement connection_init. The returned value must be a reference to an object on which the methods for the operations are called (like search, compare, etc.)
If you have a usecase for an application without connection_init I would like to hear it.
Well, only a few backends and overlays set it, so there are many use cases for not having the C variant. Obviously there are none when it's used for setup like you describe.
Possibly you could eliminate the need for it by using a blessed connection type instead, and assign a perl interpreter to a connection only at the first operation which has need for one. Easy to say when I haven't seen the code, of course...
I also suggest you do not let compatibility get in the way of good design decisions for the new code. If good design leads to bad compatibility, we could always put the old back-perl in contrib/back-oldperl or something.
Including the old backend is something I really would like to avoid as this means adapting autoconf-stuff and be confusing.
OpenLDAP's configure doesn't touch contrib - so it becomes the user's problem instead... But yes, it would indeed be nice to avoid that.
Or the user has made updates in Perl structures which are not reflected in the C structures they came from. I looked into doing this with Python once, but punted since the Python C API #includes a header with Python autoconf variables which conflict with slapd's autoconf variables.
Do you have some design notes of this left which you can send me?
Not really, but I'll see if I can dig something up. Too large a portion of the "design" part consisted of getting around the conflicting autoconf.
Am 22.06.2007 um 16:21 schrieb Hallvard B Furuseth:
Dagobert Michelsen writes:
There is exactly one interpreter in use for each active connection and each configured back-perl database because allocation is done on connection_init.
Hmm. That sounds like adding a perl database can have a noticeable impact on performance (and memory use?) even if it is rarely queried.
Yes, currently this is the case. For my applications this was no problem as the Perl backend was the only one active.
In particular if it calls a user-defined connection_init Perl method. Ought to test and see, I guess...
It is possible to defer Perl interpreter allocation and calling connection_init to the first time an operation is issued. This should be fairly easy to implement. Thanks for the inspiration!
(...) The disadvange is that there is no data sharing between connections. Implementing data sharing with this model is quite complex and involves attaching (Perl-)magic to shared variables which implements mutex'ed access between interpreters.
Perl has "use threads::shared", but maybe that only works for threads created by Perl.
I haven't checked that in detail. If done improperly this can lead to hard-to-detect memory leaks of scalars between interpreters. The code from threads::shared was the origin of mod_perl which more resembles the structure of back_perl. Again: This is not an easy task and can be added later if needed.
As the method usually does something with the parameters the data is needed anyway, so savings here should be minimal if not negative.
Depends on the API and which data we are talking about, I guess.
I'll prepare some documentation for discussion. BTW: Should the documentation for back-perl go to doc/guide/admin/backends.sdf ?
If you have a usecase for an application without connection_init I would like to hear it.
Possibly you could eliminate the need for it by using a blessed connection type instead, and assign a perl interpreter to a connection only at the first operation which has need for one. Easy to say when I haven't seen the code, of course...
Good idea for connection_init! See above...
Not really, but I'll see if I can dig something up.
I'm looking forward to that.
Best regards
-- Dagobert
I wrote:
Gavin Henry writes:
I've run Perl::Critic over this and fixed/updated to latest "Perl Best Practices". We might as well have an up to date sample.
Do you mind if I commit my changes?
My impression is that back-perl hovers on the verge of code rot, and any change is likely for the better.
Er... not this one, though:-( Our Perl version (5.8.8) interprets print <STDERR>, "Here in new\n"; as "read data from STDERR, then print that and the argument string to STDOUT". Is this some bleeding edge change to Perl?
Also I think the initial 'package' and 'use' statements should go below the comment at the top, and I'm not sure how new the "our" keyword is. I suspect there are some pretty old Perl versions out there, in particular on Windows.
<quote who="Hallvard B Furuseth">
I wrote:
Gavin Henry writes:
I've run Perl::Critic over this and fixed/updated to latest "Perl Best Practices". We might as well have an up to date sample.
Do you mind if I commit my changes?
My impression is that back-perl hovers on the verge of code rot, and any change is likely for the better.
Er... not this one, though:-( Our Perl version (5.8.8) interprets print <STDERR>, "Here in new\n"; as "read data from STDERR, then print that and the argument string to STDOUT". Is this some bleeding edge change to Perl?
Nope, you are right. My mistake. It does work, but should be:
print {*STDERR} "Here in new\n";
Example:
1 #!/usr/bin/perl 2 use strict; 3 use warnings; 4 5 print <STDERR>, "Here in new\n"; 6 print {*STDERR} "Here in new\n"; 7 print STDERR "Here in new\n";
Output:
Filehandle STDERR opened only for output at ./test.pl line 5. Here in new Here in new Here in new
See:
http://search.cpan.org/~thaljef/Perl-Critic-1.06/lib/Perl/Critic/Policy/Inpu...
Also I think the initial 'package' and 'use' statements should go below the comment at the top,
Sure, done.
and I'm not sure how new the "our" keyword is.
http://perldoc.perl.org/perl56delta.html#%22our%22-declarations
5.6
Also, see why it's used:
http://search.cpan.org/~thaljef/Perl-Critic-1.06/lib/Perl/Critic/Policy/Modu...
I suspect there are some pretty old Perl versions out there, in particular on Windows.
We should change it to:
$SampleLDAP::VERSION = '1.00';
That will work on any 5+ version of Perl.
I've made the changes ready for commit.
Gavin.
-- Hallvard
Gavin Henry writes:
Nope, you are right. My mistake. It does work, but should be: print {*STDERR} "Here in new\n"; (...) See: http://search.cpan.org/~thaljef/Perl-Critic-1.06/lib/Perl/Critic/Policy/Inpu...
Heh. Capital-letter STDERR stands out enough for me, but i guess code looks different to different people.
(...)
I suspect there are some pretty old Perl versions out there, in particular on Windows.
We should change it to:
$SampleLDAP::VERSION = '1.00';
That will work on any 5+ version of Perl.
Great. I guess it's 1.01 by now though.
<quote who="Hallvard B Furuseth">
Gavin Henry writes:
Nope, you are right. My mistake. It does work, but should be: print {*STDERR} "Here in new\n"; (...) See: http://search.cpan.org/~thaljef/Perl-Critic-1.06/lib/Perl/Critic/Policy/Inpu...
Heh. Capital-letter STDERR stands out enough for me, but i guess code looks different to different people.
It's means slightly different than that ;-)
(...)
I suspect there are some pretty old Perl versions out there, in particular on Windows.
We should change it to:
$SampleLDAP::VERSION = '1.00';
That will work on any 5+ version of Perl.
Great. I guess it's 1.01 by now though.
Yup.
-- Regards, Hallvard
-
Dagobert Michelsen -
Gavin Henry -
Hallvard B Furuseth -
Matthew Backes -
Quanah Gibson-Mount