Hi, I am at an early stage of deploying OpenLDAP running on Monstavista Carrier Grade linux, with Linux-2.6 kernel but modified scheduler. http://www.mvista.com For this I compiled and installed OpenLDAP-2.3.27 on my TP T43 notebook and on a x86 Carrier Grade hardware, I'm not sure about the processor manufacturer and type of this hardware. On both machines identical DB_CONFIG files, slapd.conf and data have been set up. I created only 10,000 entries as this would the expected average database size be. A ldapsearch on localhost with SuSE-9.3 Linux, after some warming up the results were levelling at
real 0m0.010s user 0m0.002s sys 0m0.003s
with Montavista
real 0m0.827s user 0m0.014s sys 0m0.015s
Due to a real time scheduler I was expecting the results to be vice versa. Can a RT scheduler or other modifications to the operating system have a negative influence on performance of OpenLDAP?
-Dieter
Dieter Kluenter wrote:
Hi, I am at an early stage of deploying OpenLDAP running on Monstavista Carrier Grade linux, with Linux-2.6 kernel but modified scheduler. http://www.mvista.com For this I compiled and installed OpenLDAP-2.3.27 on my TP T43 notebook and on a x86 Carrier Grade hardware, I'm not sure about the processor manufacturer and type of this hardware. On both machines identical DB_CONFIG files, slapd.conf and data have been set up. I created only 10,000 entries as this would the expected average database size be. A ldapsearch on localhost with SuSE-9.3 Linux, after some warming up the results were levelling at
real 0m0.010s user 0m0.002s sys 0m0.003s
with Montavista
real 0m0.827s user 0m0.014s sys 0m0.015s
Due to a real time scheduler I was expecting the results to be vice versa.
Can a RT scheduler or other modifications to the operating system have a negative influence on performance of OpenLDAP?
You can't expect results to be any better than under non-RT os, since a RT-scheduler will always add overhead to non-RT processes since it will preempt much more than a normal OS. I have no experience with Monta-Vista, however my institution (Politecnico di Milano) develops RTAI, the Real-Time Application Interface for Linux (http://www.rtai.org/); I'll check what happens when running OpenLDAP with that.
In the meanwhile, can you tell something more about Monta Vista's handling of RT? Does it include the PREEMPT_RT patch, which will likely become part of Linux in a year or so, or does it rather implement some custom support to handle preemption?
About your test: are your results an average over a realistic number of trials? Is the database entirely in cache or does it require any access to disk? Mallocs under a real-time scheduler might also represent a significant bottleneck; in this case, slapd should really handle memory internally, to minimize the amount of system calls.
p.
Ing. Pierangelo Masarati OpenLDAP Core Team
SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati@sys-net.it ------------------------------------------
Pierangelo Masarati ando@sys-net.it writes:
Dieter Kluenter wrote:
Hi, I am at an early stage of deploying OpenLDAP running on Monstavista Carrier Grade linux, with Linux-2.6 kernel but modified scheduler. http://www.mvista.com For this I compiled and installed OpenLDAP-2.3.27 on my TP T43 notebook and on a x86 Carrier Grade hardware, I'm not sure about the processor manufacturer and type of this hardware. On both machines identical DB_CONFIG files, slapd.conf and data have been set up. I created only 10,000 entries as this would the expected average database size be. A ldapsearch on localhost with SuSE-9.3 Linux, after some warming up the results were levelling at
[...]
Due to a real time scheduler I was expecting the results to be vice versa. Can a RT scheduler or other modifications to the operating system have a negative influence on performance of OpenLDAP?
You can't expect results to be any better than under non-RT os, since a RT-scheduler will always add overhead to non-RT processes since it will preempt much more than a normal OS. I have no experience with Monta-Vista, however my institution (Politecnico di Milano) develops RTAI, the Real-Time Application Interface for Linux (http://www.rtai.org/); I'll check what happens when running OpenLDAP with that.
In the meanwhile, can you tell something more about Monta Vista's handling of RT? Does it include the PREEMPT_RT patch, which will likely become part of Linux in a year or so, or does it rather implement some custom support to handle preemption?
I don't know much about Montavista yet, as they are more than restrictive about their information policy. But they claim to have introduced a preemtive kernel in 1999 and their preemtive patch is now a standard kernel option for 2.6 kernels. http://www.mvista.com/products/realtime.html
About your test: are your results an average over a realistic number of trials? Is the database entirely in cache or does it require any access to disk? Mallocs under a real-time scheduler might also represent a significant bottleneck; in this case, slapd should really handle memory internally, to minimize the amount of system calls.
My results are actually the last result of about 20 test runs, but after a warm up time of 5 runs, they didn't vary much. Both machines have 2GB ram and this are the settings for DB_CONFIG
set_lg_dir /opt/openldap/ldapDB set_cachesize 0 20000000 0 set_lg_regionmax 262144 set_lg_bsize 2097152
I didn't notice hardly any disk access during the initial tests.
-Dieter
Dieter Kluenter wrote:
Due to a real time scheduler I was expecting the results to be vice versa. Can a RT scheduler or other modifications to the operating system have a negative influence on performance of OpenLDAP?
You can't expect results to be any better than under non-RT os, since a RT-scheduler will always add overhead to non-RT processes since it will preempt much more than a normal OS. I have no experience with Monta-Vista, however my institution (Politecnico di Milano) develops RTAI, the Real-Time Application Interface for Linux (http://www.rtai.org/); I'll check what happens when running OpenLDAP with that.
In the meanwhile, can you tell something more about Monta Vista's handling of RT? Does it include the PREEMPT_RT patch, which will likely become part of Linux in a year or so, or does it rather implement some custom support to handle preemption?
I don't know much about Montavista yet, as they are more than restrictive about their information policy. But they claim to have introduced a preemtive kernel in 1999 and their preemtive patch is now a standard kernel option for 2.6 kernels. http://www.mvista.com/products/realtime.html
About your test: are your results an average over a realistic number of trials? Is the database entirely in cache or does it require any access to disk? Mallocs under a real-time scheduler might also represent a significant bottleneck; in this case, slapd should really handle memory internally, to minimize the amount of system calls.
My results are actually the last result of about 20 test runs, but after a warm up time of 5 runs, they didn't vary much. Both machines have 2GB ram and this are the settings for DB_CONFIG
set_lg_dir /opt/openldap/ldapDB set_cachesize 0 20000000 0 set_lg_regionmax 262144 set_lg_bsize 2097152
I didn't notice hardly any disk access during the initial tests.
Dieter,
I don't know how much this could be of any help, but today I tried to run a simple test using OpenLDAP 2.3.28 running on Linux 2.6.17 with RTAI, compared to the stock 2.3.6.15 kernel f the distribution used on that PC. I created a database with ~10000 users, fully indexed and fully cacheable (cachesize 10000). What I found (strange enough) is that the execution time of ldapsearch would differ a lot on the two systems, but sort of the reverse of what you experienced. I checked few cases:
- entry not yet in cache vs. entry already in cache - system unloaded vs. system heavily loaded
in the case of RTAI, I also considered some hard real-time load, which gets the highest priority and accurate scheduling.
- searching for an entry not in cache, w/ RTAI, requires a little bit longer than w/o RTAI; this requires access to disk; but - searching for the same entry, after it's in cache, requires ~0.008s w/ RTAI, and ~0.080s w/o RTAI (ten times longer! It's a slightly different kernel, but running on exactly the same machine, so, without too much investigation, I have no clue about the exact reason)
- under load (repeated disk and network access), the above entry in cache times are occasionally delayed, but not that much, on both systems. w/ RTAI the average goes to ~0.010s, with peaks to ~0.020s; w/o RTAI no significant extra delay is added to the ~0.080s. The latter case makes sense, since the system already appears significantly delayed.
- under hard RT load (30% of one CPU, and 15% of the other CPU on a dual CPU system), performances drop significantly; occasionally, the response time for not-in-cache entry, which requires access to disk, may raise to ~0.500s, with an average above ~0.100s. The response time when the entry is in cache is not significantly worse than in the case with non-RT load, ~0.020s.
This seems to indicate that with a (possibly) different hard RT scheduling there is no significant impact on basic OpenLDAP's slapd performances, so its design seems to have no impact on its execution in conjunction with RT schedulers. I have no clue, right now, about how the two RT systems compare.
I would like to acknowledge the support of Paolo Mantegazza, the main developer of RTAI, in performing those tests with a devel snapshot (called "magma") of that system; it should not differ from recent releases (3.4) in terms of scheduling policies and related issues.
Cheers, p.
Ing. Pierangelo Masarati OpenLDAP Core Team
SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati@sys-net.it ------------------------------------------
Hello Ando,
Pierangelo Masarati ando@sys-net.it writes:
Dieter Kluenter wrote:
Due to a real time scheduler I was expecting the results to be vice
Dieter,
I don't know how much this could be of any help, but today I tried to run a simple test using OpenLDAP 2.3.28 running on Linux 2.6.17 with RTAI, compared to the stock 2.3.6.15 kernel f the distribution used on that PC. I created a database with ~10000 users, fully indexed and fully cacheable (cachesize 10000). What I found (strange enough) is that the execution time of ldapsearch would differ a lot on the two systems, but sort of the reverse of what you experienced. I checked few cases:
- entry not yet in cache vs. entry already in cache
- system unloaded vs. system heavily loaded
in the case of RTAI, I also considered some hard real-time load, which gets the highest priority and accurate scheduling.
- searching for an entry not in cache, w/ RTAI, requires a little bit
longer than w/o RTAI; this requires access to disk; but
- searching for the same entry, after it's in cache, requires ~0.008s
w/ RTAI, and ~0.080s w/o RTAI (ten times longer! It's a slightly different kernel, but running on exactly the same machine, so, without too much investigation, I have no clue about the exact reason)
- under load (repeated disk and network access), the above entry in
cache times are occasionally delayed, but not that much, on both systems. w/ RTAI the average goes to ~0.010s, with peaks to ~0.020s; w/o RTAI no significant extra delay is added to the ~0.080s. The latter case makes sense, since the system already appears significantly delayed.
- under hard RT load (30% of one CPU, and 15% of the other CPU on a
dual CPU system), performances drop significantly; occasionally, the response time for not-in-cache entry, which requires access to disk, may raise to ~0.500s, with an average above ~0.100s. The response time when the entry is in cache is not significantly worse than in the case with non-RT load, ~0.020s.
This seems to indicate that with a (possibly) different hard RT scheduling there is no significant impact on basic OpenLDAP's slapd performances, so its design seems to have no impact on its execution in conjunction with RT schedulers. I have no clue, right now, about how the two RT systems compare.
I would like to acknowledge the support of Paolo Mantegazza, the main developer of RTAI, in performing those tests with a devel snapshot (called "magma") of that system; it should not differ from recent releases (3.4) in terms of scheduling policies and related issues.
Thank you for your test and report and thanks to Paolo Mantegazza. I think I have to dive a bit more into Montavista scheduler and have a chat with their support.
-Dieter
Pierangelo Masarati wrote:
run a simple test using OpenLDAP 2.3.28 running on Linux 2.6.17 with RTAI, compared to the stock 2.3.6.15 kernel f the distribution
^^^^^^^^ Do you mean stock 2.6.15?
- searching for the same entry, after it's in cache, requires ~0.008s w/
RTAI, and ~0.080s w/o RTAI (ten times longer! It's a slightly different kernel, but running on exactly the same machine, [..]
Are you sure there haven't been any significant changes to the scheduler between these two kernel versions anyway? Also distributions most times ship with a patched kernel. So exact comparison should be done with a vanilla kernel and the very same kernel with RTAI.
Ciao, Michael.
Michael Ströder wrote:
Pierangelo Masarati wrote:
run a simple test using OpenLDAP 2.3.28 running on Linux 2.6.17 with RTAI, compared to the stock 2.3.6.15 kernel f the distribution
^^^^^^^^Do you mean stock 2.6.15?
No, I have no idea.
- searching for the same entry, after it's in cache, requires ~0.008s w/
RTAI, and ~0.080s w/o RTAI (ten times longer! It's a slightly different kernel, but running on exactly the same machine, [..]
Are you sure there haven't been any significant changes to the scheduler between these two kernel versions anyway? Also distributions most times ship with a patched kernel. So exact comparison should be done with a vanilla kernel and the very same kernel with RTAI.
I concur. The point is that to apply RTAI patch we need an official kernel release (usually the latest, or occasionally the latest-but-one), but usually we only rebuild it with the RTAI patch, and not without (there's no need to). So, just to have an idea of how OpenLDAP behaved with the RT scheduling, I simply ran the same tests with the distribution's kernel. The distribution was Ubuntu, but I don't remember the version or any other detail. Only in case of dramatically slower response I would have checked more carefully those details.
p.
openldap-software@openldap.org
-
Dieter Kluenter -
Michael Ströder -
Pierangelo Masarati