This is a follow-up to my email to -technical, this time more into the future of OpenLDAP's test suite and tools we could build to make managing OpenLDAP deployments much easier for everyone.
Historically, there has been a decent coverage in the test suite and that's what's being run before anyone pushes, but it's not been enough to capture some of the issues. It also takes ages to finish, and is very limited in what can be tested, mostly because we can only rely on POSIX shell and command line tools.
That means a lot of boilerplate (and since shell functions are not available, that boilerplate has to be copy-pasted everywhere) and we can never actively do two things at the same time, let alone submit several operations over the same connection. There is no way we can get notified when things happen (servers have started up, all members think they're in sync, ...) so we have to rely on polling and manual delays which slows it down significantly, or introduces false positives on hosts where these delays are too short. The tests can't be run in parallel either, not from the same build environment.
None of the above can be resolved without a major overhaul. I've been thinking about all of that for a while now and want to get started improving the situation.
If we switched to an actual programming language to write the tests in, something like Python 3, Golang or Rust, we could build a support library including a harness to manage slapd through, tools to configure and interact with the server in a high level way (continue as soon as the server has started up, or once an MMR environment settles, ...). All of this should then make it much easier to write new tests and maintain and understand existing ones.
I have already started the part that people without intimate experience with OpenLDAP internals would struggle with - a loadable module that knows how to connect back to the program that started it and informs it of the port it's listening on and when it has become ready to process traffic. That code is here: https://github.com/mistotebe/openldap/tree/mod_harness
To see it in action, just run `nc -l -p <port>`, then start slapd with `-h ldap://localhost:0` (yes, an explicit zero) and make sure this is in slapd's configuration:
``` moduleload path/to/mod_harness
backend harness host localhost port <your port above> ```
I would appreciate if people joined to help define the testing infrastructure. My preference is Python 3 as the host language, but anything that can express concurrency well and has a good LDAP client implementation should be acceptable if there are enough volunteers. We will also have to look into defining the protocol the test suite would use to communicate with the slapd module as the existing plain text protocol is just a placeholder for easy experimentation.
This would do well for testing in the 2.5 lifecycle. In the future, I want to extend this to support fine grained testing where the inner workings of slapd are much more transparent to the test code. Eventually going as far as making it possible to unit-test any OpenLDAP code. The first step in that direction is probably going to be learning how to host a C compiler within the loadable module so we can compile, link and inject code and callbacks into slapd as and when needed to meld the test suite with the server much better. Clang (the LLVM C/C++ compiler) seems to be capable of this.
All of the above will introduce new helper APIs, so in the end it will overlap with efforts to develop new user-facing tools to configure, monitor and manage OpenLDAP deployments: - user friendly tools to manage cn=config - high-level tools to set up and manage common replication topologies - tools to monitor replication progress - many more as we progress with making OpenLDAP user friendly...
(Please don't turn this thread into another flame war on slapd.conf vs. cn=config, start another one if you really can't resist.)
Again, thanks for being brave and getting this far, let us know what you think. This is mostly vaporware yet, giving everyone a great opportunity to shape the project and leave their mark - remember that without your help, it will take a long while before it's ready!
Regards,
Ondřej Kuzník wrote:
This is a follow-up to my email to -technical, this time more into the future of OpenLDAP's test suite and tools we could build to make managing OpenLDAP deployments much easier for everyone.
Historically, there has been a decent coverage in the test suite and that's what's being run before anyone pushes, but it's not been enough to capture some of the issues. It also takes ages to finish, and is very limited in what can be tested, mostly because we can only rely on POSIX shell and command line tools.
That means a lot of boilerplate (and since shell functions are not available, that boilerplate has to be copy-pasted everywhere)
We could always push common stuff into a script that gets sourced from the individual test scripts.
and we can never actively do two things at the same time
Like what?
let alone submit several
operations over the same connection.
We can always write specific slapd-tester programs for whatever purpose arises.
There is no way we can get notified
when things happen (servers have started up, all members think they're in sync, ...) so we have to rely on polling and manual delays which slows it down significantly, or introduces false positives on hosts where these delays are too short.
No matter what we use, polling is still needed.
The tests can't be run in parallel
either, not from the same build environment.
All of the environment params are in overridable shell variables. We can have as many instances running as we want, but nobody has ever expressed a desire to run things in parallel before.
None of the above can be resolved without a major overhaul.
I think you're overestimating the problem. AFAICS it wasn't even a "problem" until just now, nobody has ever mentioned this before.
I've been
thinking about all of that for a while now and want to get started improving the situation.
If we switched to an actual programming language to write the tests in, something like Python 3, Golang or Rust,
Not golang, not rust. They don't have nearly enough platform coverage, nor actual maturity.
we could build a support
library including a harness to manage slapd through,
Sounds like the library should be in C, which is then binary compatible with any other language you'd care to use.
tools to configure
and interact with the server in a high level way (continue as soon as the server has started up, or once an MMR environment settles, ...). All of this should then make it much easier to write new tests and maintain and understand existing ones.
I have already started the part that people without intimate experience with OpenLDAP internals would struggle with - a loadable module that knows how to connect back to the program that started it and informs it of the port it's listening on and when it has become ready to process traffic. That code is here: https://github.com/mistotebe/openldap/tree/mod_harness
To see it in action, just run `nc -l -p <port>`, then start slapd with `-h ldap://localhost:0` (yes, an explicit zero) and make sure this is in slapd's configuration:
moduleload path/to/mod_harness backend harness host localhost port <your port above>I would appreciate if people joined to help define the testing infrastructure. My preference is Python 3 as the host language, but anything that can express concurrency well and has a good LDAP client implementation should be acceptable if there are enough volunteers. We will also have to look into defining the protocol the test suite would use to communicate with the slapd module as the existing plain text protocol is just a placeholder for easy experimentation.
This would do well for testing in the 2.5 lifecycle. In the future, I want to extend this to support fine grained testing where the inner workings of slapd are much more transparent to the test code. Eventually going as far as making it possible to unit-test any OpenLDAP code. The first step in that direction is probably going to be learning how to host a C compiler within the loadable module so we can compile, link and inject code and callbacks into slapd as and when needed to meld the test suite with the server much better. Clang (the LLVM C/C++ compiler) seems to be capable of this.
All of the above will introduce new helper APIs, so in the end it will overlap with efforts to develop new user-facing tools to configure, monitor and manage OpenLDAP deployments:
- user friendly tools to manage cn=config
- high-level tools to set up and manage common replication topologies
- tools to monitor replication progress
- many more as we progress with making OpenLDAP user friendly...
(Please don't turn this thread into another flame war on slapd.conf vs. cn=config, start another one if you really can't resist.)
Again, thanks for being brave and getting this far, let us know what you think. This is mostly vaporware yet, giving everyone a great opportunity to shape the project and leave their mark - remember that without your help, it will take a long while before it's ready!
Regards,
On Thu, Jul 25, 2019 at 12:34:13AM +0100, Howard Chu wrote:
Ondřej Kuzník wrote:
Historically, there has been a decent coverage in the test suite and that's what's being run before anyone pushes, but it's not been enough to capture some of the issues. It also takes ages to finish, and is very limited in what can be tested, mostly because we can only rely on POSIX shell and command line tools.
That means a lot of boilerplate (and since shell functions are not available, that boilerplate has to be copy-pasted everywhere)
We could always push common stuff into a script that gets sourced from the individual test scripts.
That would work some way to replace functions, fair point. But array variables are not available either AFAIK so even this is limited.
And it still doesn't give us a good way to test things like response controls.
and we can never actively do two things at the same time
Like what?
There have been times I could opt for an easier/more efficient/more readable test if I could use several connections simultaneously.
But these are currently impossible: - slapd-sock testing - refreshAndPersist (and as a result, more intelligent MMR convergence monitoring) - transaction interaction - one client starts it and another one comes in while it's not committed yet
let alone submit several operations over the same connection.
We can always write specific slapd-tester programs for whatever purpose arises.
That doesn't really scale. Testing SASL bind handling, e.g. transition between mechanisms, is one example where you just need to be able to send a few requests that you can craft.
There is no way we can get notified when things happen (servers have started up, all members think they're in sync, ...) so we have to rely on polling and manual delays which slows it down significantly, or introduces false positives on hosts where these delays are too short.
No matter what we use, polling is still needed.
There's polling and polling. If you can do an equivalent of select() on a file descriptor or a few, you can do orders of magnitude better than a sleep-ldapsearch cycle ever could.
And once it's possible to react to things as soon as they happen, we can have a global timeout in order of minutes, not seconds because we know it's only there to recover from a lockup.
The tests can't be run in parallel either, not from the same build environment.
All of the environment params are in overridable shell variables. We can have as many instances running as we want, but nobody has ever expressed a desire to run things in parallel before.
Not all of them, but that's not the point. It's ok if you want to test two different codebases, you just set SLAPD_BASEPORT and you're done.
For parallel tests you need a mktemp-like testrun directory and better (but unpredictable) listen port allocation. And I never bothered to complain/improve on this because it's so hard to write a decent test already.
An overhaul solves many other issues, so might as well add this as a requirement since it's cheap if you design for it.
None of the above can be resolved without a major overhaul.
I think you're overestimating the problem. AFAICS it wasn't even a "problem" until just now, nobody has ever mentioned this before.
All of the above have been a persistent problem for me whenever a new test needed to be written. But no point for me to ask for things that can't be done if we're explicitly targeting POSIX shell and nothing else.
If you want an example where this is a major issue, imagine testing the load balancer if the only tool you have is slapd. If only you could put slapd-sock into the mix and observe/interact through it you would get quite far but not even that's available.
I've been thinking about all of that for a while now and want to get started improving the situation.
If we switched to an actual programming language to write the tests in, something like Python 3, Golang or Rust,
Not golang, not rust. They don't have nearly enough platform coverage, nor actual maturity.
Sure, I prefer Python anyway, but if we had noone willing to do that, but a healthy number from another camp, I was willing to at least consider it.
Not many people understand how to do asyncio in Python yet that stuff is easy in Golang.
we could build a support library including a harness to manage slapd through,
Sounds like the library should be in C, which is then binary compatible with any other language you'd care to use.
I was thinking more along the lines of: - set up a new empty slapd - restart it - set up an n-way MMR and give me a list of servers I can talk to (this is popular among our regression tests already) - let me resume my test once a given replica/all servers are in sync
-
Howard Chu -
Ondřej Kuzník