Hi,
we had some performance issues on our ldap servers running Solaris 10 sparc.
I did some tests using slamd http://www.slamd.com/ and got disturbing results:
ldap-service: OpenLDAP 2.4.23, setup identical on both boxes, threads=64, identical content.
box1: hardware: Sun Microsystems sun4v SPARC Enterprise T5120 memory:32 GB RAM os: Solaris 10 s10s_u9wos_14a searches (avg/second): 1521
box2: hardware: AMD 64 phenom 9850 memory: 8 GB RAM os: Linux dsv00000 2.6.36.2 #1 SMP Mon Dec 27 10:09:29 CET 2010 x86_64 AMD Phenom(tm) 9850 Quad-Core Processor AuthenticAMD GNU/Linux searches (avg/second): 22968
I am looking for some hints to improve performance on box1 to a similar level as on box2 or an explanation why on box1 performance is so bad.
As slapd.conf, ldap.conf and DB_CONFIG are identical there should not be such a big gap in performance.
Juergen Sprenger
Am Thu, 24 Feb 2011 15:53:43 +0100 schrieb Juergen.Sprenger@swisscom.com:
Hi,
we had some performance issues on our ldap servers running Solaris 10 sparc.
I did some tests using slamd http://www.slamd.com/ and got disturbing results:
ldap-service: OpenLDAP 2.4.23, setup identical on both boxes, threads=64, identical content.
box1: hardware: Sun Microsystems sun4v SPARC Enterprise T5120 memory:32 GB RAM os: Solaris 10 s10s_u9wos_14a searches (avg/second): 1521
box2: hardware: AMD 64 phenom 9850 memory: 8 GB RAM os: Linux dsv00000 2.6.36.2 #1 SMP Mon Dec 27 10:09:29 CET 2010 x86_64 AMD Phenom(tm) 9850 Quad-Core Processor AuthenticAMD GNU/Linux searches (avg/second): 22968
I am looking for some hints to improve performance on box1 to a similar level as on box2 or an explanation why on box1 performance is so bad.
As slapd.conf, ldap.conf and DB_CONFIG are identical there should not be such a big gap in performance.
It is hard to guess, as you didn't provide any build information, but /dev/poll vs. libpthread and no. of IO's might be a hint.
-Dieter
--On Thursday, February 24, 2011 3:53 PM +0100 Juergen.Sprenger@swisscom.com wrote:
Hi,
we had some performance issues on our ldap servers running Solaris 10 sparc.
I did some tests using slamd http://www.slamd.com/ and got disturbing results:
ldap-service: OpenLDAP 2.4.23, setup identical on both boxes, threads=64, identical content.
box1: hardware: Sun Microsystems sun4v SPARC Enterprise T5120 memory:32 GB RAM os: Solaris 10 s10s_u9wos_14a searches (avg/second): 1521
Slowaris is always tedious with OpenLDAP. At a previous job, I was able to replace 9 slowaris boxes with 4 linux boxes, and even then, just one of the Linux boxes could handle the complete load it took the 9 slowaris boxes to run.
I will note that if you are going to use slowaris, I highly advise you set a memory key rather than using on disk cache for BDB if your DB is any size over about 4 GB. Other than that, you'll generally just have to deal with the fact it will be significantly slower than Linux.
Some time ago, I did benchmarks of Linux 2.6 vs Slowaris-x86 on identical hardware, and it still was noticeably slower, although not as bad as when using the sparc architecture.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Keep in mind that gcc on slowaris is 32 bit, and not very well optimised by default. Linux on a 64 bit linux platform usually compiles as 64 bit, slowaris by default will be 32 bit if you use the onboard gcc compiler.
Using the compiler from sun studio (since v11 now free) helps quite alot but openldap/bdb configure wont use the correct compiler options by default either.
Cheers Brett
On 25/02/2011, at 12:53 AM, Juergen.Sprenger@swisscom.com wrote:
Hi,
we had some performance issues on our ldap servers running Solaris 10 sparc.
I did some tests using slamd http://www.slamd.com/ and got disturbing results:
ldap-service: OpenLDAP 2.4.23, setup identical on both boxes, threads=64, identical content.
box1: hardware: Sun Microsystems sun4v SPARC Enterprise T5120 memory:32 GB RAM os: Solaris 10 s10s_u9wos_14a searches (avg/second): 1521
box2: hardware: AMD 64 phenom 9850 memory: 8 GB RAM os: Linux dsv00000 2.6.36.2 #1 SMP Mon Dec 27 10:09:29 CET 2010 x86_64 AMD Phenom(tm) 9850 Quad-Core Processor AuthenticAMD GNU/Linux searches (avg/second): 22968
I am looking for some hints to improve performance on box1 to a similar level as on box2 or an explanation why on box1 performance is so bad.
As slapd.conf, ldap.conf and DB_CONFIG are identical there should not be such a big gap in performance.
Juergen Sprenger
--On Friday, February 25, 2011 10:02 AM +1000 Brett Maxfield brett.maxfield@gmail.com wrote:
Keep in mind that gcc on slowaris is 32 bit, and not very well optimised by default. Linux on a 64 bit linux platform usually compiles as 64 bit, slowaris by default will be 32 bit if you use the onboard gcc compiler.
Using the compiler from sun studio (since v11 now free) helps quite alot but openldap/bdb configure wont use the correct compiler options by default either.
OpenLDAP on Solaris is slow, regardless of whether or not you use Sun Studio to build it 64-bit or gcc to build it 64 bit. I only ever used 64-bit binaries on my 64-bit systems, same with the benchmarks I did. Symas may even still have them posted somewhere.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
--On Thursday, February 24, 2011 3:53 PM +0100 Juergen.Sprenger@swisscom.com wrote:
Hi,
we had some performance issues on our ldap servers running Solaris 10 sparc.
I did some tests using slamd http://www.slamd.com/ and got disturbing results:
ldap-service: OpenLDAP 2.4.23, setup identical on both boxes, threads=64, identical content.
box1: hardware: Sun Microsystems sun4v SPARC Enterprise T5120 memory:32 GB RAM os: Solaris 10 s10s_u9wos_14a searches (avg/second): 1521
Slowaris is always tedious with OpenLDAP. At a previous job, I was able to replace 9 slowaris boxes with 4 linux boxes, and even then, just one of the Linux boxes could handle the complete load it took the 9 slowaris boxes to run.
I will note that if you are going to use slowaris, I highly advise you set a memory key rather than using on disk cache for BDB if your DB is any size over about 4 GB. Other than that, you'll generally just have to deal with the fact it will be significantly slower than Linux.
Actually, you should always use a shared memory key on Solaris. Using mmap'd files is just too slow on that OS.
Some time ago, I did benchmarks of Linux 2.6 vs Slowaris-x86 on identical hardware, and it still was noticeably slower, although not as bad as when using the sparc architecture.
On Fri, Feb 25, 2011 at 10:25 AM, Howard Chu hyc@symas.com wrote:
I will note that if you are going to use slowaris, I highly advise you set
a memory key rather than using on disk cache for BDB if your DB is any size over about 4 GB. Other than that, you'll generally just have to deal with the fact it will be significantly slower than Linux.
Actually, you should always use a shared memory key on Solaris. Using mmap'd files is just too slow on that OS.
Incidentlly, What is the uniqueness of a shared memory key, is it unique amoung all instances on a single server ?
I have personally only ever needed a shared memory key on one of a group of ldap servers.
But i noticed that the bdb logs are not written to disk when shared memory is configured, is this a good or bad thing ?
If the power is lost, without bdb logs, could this leave bdb in an unrecoverable state ?
Cheers Brett
Brett @Google wrote:
On Fri, Feb 25, 2011 at 10:25 AM, Howard Chu <hyc@symas.com mailto:hyc@symas.com> wrote:
I will note that if you are going to use slowaris, I highly advise you set a memory key rather than using on disk cache for BDB if your DB is any size over about 4 GB. Other than that, you'll generally just have to deal with the fact it will be significantly slower than Linux. Actually, you should always use a shared memory key on Solaris. Using mmap'd files is just too slow on that OS.
Incidentlly, What is the uniqueness of a shared memory key, is it unique amoung all instances on a single server ?
The significance of a shared memory key depends entirely on your operating system. But by its nature, shared memory is global to a machine, so you can assume that all keys must be unique on a particular machine.
I have personally only ever needed a shared memory key on one of a group of ldap servers.
But i noticed that the bdb logs are not written to disk when shared memory is configured, is this a good or bad thing ?
The use of shared memory has no bearing on how the BDB library writes its transaction logs.
Howard Chu wrote:
Actually, you should always use a shared memory key on Solaris. Using mmap'd files is just too slow on that OS.
SunOS dns05.unix 5.10 Generic_141445-09 i86pc i386 i86pc
Is this still the case? I did a quick (naive?) test to see if it would improve things for our LDAP server. I had an old/replace DNS server to test on, using production values.
# wc -l p 142149 p # time /usr/local/bin/ldapsearch -h 0 -x -D 'cn=admin,dc=company,dc=com' -w secret -f p -b ou=dns,dc=company,dc=com "(%s)" dn > /dev/null
pass 1 real 0m20.660s user 0m3.134s sys 0m1.283s
pass 2 real 0m20.853s user 0m3.132s sys 0m1.309s
I added only:
+ shm_key 1000 + dbconfig set_shm_key 1000
to slapd.conf, deleted DB_CONFIG, stopped slapd, ran db_recover, started slapd:
# ipcs IPC status from <running system> as of Tuesday, March 1, 2011 11:32:05 AM JST T ID KEY MODE OWNER GROUP Message Queues: Shared Memory: m 23 0x3ed --rw------- root root m 22 0x3ec --rw------- root root m 21 0x3eb --rw------- root root m 20 0x3ea --rw------- root root m 19 0x3e9 --rw------- root root m 18 0x3e8 --rw------- root root
pass 1 real 0m19.772s user 0m3.109s sys 0m1.275s
pass 2 real 0m19.858s user 0m3.102s sys 0m1.275s
I was hoping for a larger difference, but maybe my tests are a little too simple.
Why delete DB_CONFIG, surely that would make performance worse, apart from the fact you now compare apples and oranges.. ?
On 01/03/2011, at 12:41 PM, Jorgen Lundman lundman@lundman.net wrote:
Howard Chu wrote:
Actually, you should always use a shared memory key on Solaris. Using mmap'd files is just too slow on that OS.
SunOS dns05.unix 5.10 Generic_141445-09 i86pc i386 i86pc
Is this still the case? I did a quick (naive?) test to see if it would improve things for our LDAP server. I had an old/replace DNS server to test on, using production values.
# wc -l p 142149 p # time /usr/local/bin/ldapsearch -h 0 -x -D 'cn=admin,dc=company,dc=com' -w secret -f p -b ou=dns,dc=company,dc=com "(%s)" dn > /dev/null
pass 1 real 0m20.660s user 0m3.134s sys 0m1.283s
pass 2 real 0m20.853s user 0m3.132s sys 0m1.309s
I added only:
- shm_key 1000
- dbconfig set_shm_key 1000
to slapd.conf, deleted DB_CONFIG, stopped slapd, ran db_recover, started slapd:
# ipcs IPC status from <running system> as of Tuesday, March 1, 2011 11:32:05 AM JST T ID KEY MODE OWNER GROUP Message Queues: Shared Memory: m 23 0x3ed --rw------- root root m 22 0x3ec --rw------- root root m 21 0x3eb --rw------- root root m 20 0x3ea --rw------- root root m 19 0x3e9 --rw------- root root m 18 0x3e8 --rw------- root root
pass 1 real 0m19.772s user 0m3.109s sys 0m1.275s
pass 2 real 0m19.858s user 0m3.102s sys 0m1.275s
I was hoping for a larger difference, but maybe my tests are a little too simple.
-- Jorgen Lundman | lundman@lundman.net Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work) Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell) Japan | +81 (0)3 -3375-1767 (home)
The DB_CONFIG values are in slapd.conf, and to make slapd re-write/update them, you delete the file before starting slapd.
Brett Maxfield wrote:
Why delete DB_CONFIG, surely that would make performance worse, apart from the fact you now compare apples and oranges.. ?
On 01/03/2011, at 12:41 PM, Jorgen Lundmanlundman@lundman.net wrote:
Howard Chu wrote:
Actually, you should always use a shared memory key on Solaris. Using mmap'd files is just too slow on that OS.
SunOS dns05.unix 5.10 Generic_141445-09 i86pc i386 i86pc
Is this still the case? I did a quick (naive?) test to see if it would improve things for our LDAP server. I had an old/replace DNS server to test on, using production values.
# wc -l p 142149 p # time /usr/local/bin/ldapsearch -h 0 -x -D 'cn=admin,dc=company,dc=com' -w secret -f p -b ou=dns,dc=company,dc=com "(%s)" dn> /dev/null
pass 1 real 0m20.660s user 0m3.134s sys 0m1.283s
pass 2 real 0m20.853s user 0m3.132s sys 0m1.309s
I added only:
- shm_key 1000
- dbconfig set_shm_key 1000
to slapd.conf, deleted DB_CONFIG, stopped slapd, ran db_recover, started slapd:
# ipcs IPC status from<running system> as of Tuesday, March 1, 2011 11:32:05 AM JST T ID KEY MODE OWNER GROUP Message Queues: Shared Memory: m 23 0x3ed --rw------- root root m 22 0x3ec --rw------- root root m 21 0x3eb --rw------- root root m 20 0x3ea --rw------- root root m 19 0x3e9 --rw------- root root m 18 0x3e8 --rw------- root root
pass 1 real 0m19.772s user 0m3.109s sys 0m1.275s
pass 2 real 0m19.858s user 0m3.102s sys 0m1.275s
I was hoping for a larger difference, but maybe my tests are a little too simple.
-- Jorgen Lundman |lundman@lundman.net Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work) Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell) Japan | +81 (0)3 -3375-1767 (home)
openldap-technical@openldap.org