Hi,

Thanks a lot for the quick reply.

I Tried out further tests with a set of 50 clients with 1 thread. I used 2 slamd client manager from 2 different DL585 machines, Further, Started 50 clients on both the servers making in all 100 SLAMD clients.

The results obtained with the tests with 60, 80 and 100 clients resulted in CPU Utilization of 100% in the initial run of ten minutes. CPU reported an average of 100% utilized while after 10 minutes, it started oscillating between 0% utilized and 100% utilized, resulting in slow down of average.

Can you please suggest any other tool/ way to achieved a sustained CPU utilization of about 70% by generating read load on openLDAP.

Currently the CPU graph shows spikes with shoot up in utilization and then a sudden drop. It would be great, if you could suggest some possible reasons for this behavior.

Also what is the average response time of a read request in a heavy load scenario for openLDAP ?

Thanks a ton in advance.

Regards Shilpa Sethi

A R I C E N T Plot 6, Electronic City, Sector - 18, Gurgaon 122015 Haryana, India

-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Monday, November 10, 2008 9:51 PM To: Shilpa Sethi; openldap-technical@openldap.org Subject: Re: Issues in Performance Tesing of OpenLDAP using SLAMD

--On Monday, November 10, 2008 5:28 PM +0530 Shilpa Sethi shilpa2.sethi@aricent.com wrote:

Please suggest a way by which Predictable/ similar read rates could be achieved on both the servers as both being exactly same.

I always found slamd to be massively inefficient. The only way to get reliable results from it in a read scenario was to have numerous client systems querying the LDAP server(s), so that the clients could reliably max out the server.

--Quanah

--

Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error,please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."

Hi,

I changed the approach for Performance testing. We developed an in-house Java based tool that uses JLDAP library to query openLDAP database.

4 injector machines were used to inject the Traffic to two openLDAP server running in Mirror-mode replication.

However result pattern obtained is very similar to those reported by SLAMD. PFA, the result sheet that shows query made to Openldap by 4 machines and the respective response time/interval.

Please look into the sheet attached and suggest a possible cause for such a behavior by openLDAP. In some intervals the responses returned are very high while in some others it is 0.

Looking forward for a positive reply from the members of forum.

Please note - Cache was primed before the queries were made.

Regards Shilpa Sethi

A R I C E N T Plot 6, Electronic City, Sector - 18, Gurgaon 122015 Haryana, India

-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Saturday, November 15, 2008 1:03 AM To: Shilpa Sethi; openldap-technical@openldap.org Subject: RE: Issues in Performance Tesing of OpenLDAP using SLAMD

--On Friday, November 14, 2008 5:49 PM +0530 Shilpa Sethi shilpa2.sethi@aricent.com wrote:

Hi,

Thanks a lot for the quick reply.

I Tried out further tests with a set of 50 clients with 1 thread. I used 2 slamd client manager from 2 different DL585 machines, Further, Started 50 clients on both the servers making in all 100 SLAMD clients.

The results obtained with the tests with 60, 80 and 100 clients resulted in CPU Utilization of 100% in the initial run of ten minutes. CPU reported an average of 100% utilized while after 10 minutes, it started oscillating between 0% utilized and 100% utilized, resulting in slow down of average.

(a) Did you prime the cache before you ran your read test? (b) You may have to adjust interval periods between reads on the slamd clients if you're shooting for a particular CPU usage rate. That was never the objective of any of my tests, so I don't know.

--Quanah

--

Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error,please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."

Shilpa Sethi wrote:

Hi,

I changed the approach for Performance testing. We developed an in-house Java based tool that uses JLDAP library to query openLDAP database.

4 injector machines were used to inject the Traffic to two openLDAP server running in Mirror-mode replication.

However result pattern obtained is very similar to those reported by SLAMD. PFA, the result sheet that shows query made to Openldap by 4 machines and the respective response time/interval.

Please look into the sheet attached and suggest a possible cause for such a behavior by openLDAP. In some intervals the responses returned are very high while in some others it is 0.

Looking forward for a positive reply from the members of forum.

You haven't provided any useful information to which anyone can respond. Benchmarking is a lot more demanding than just tuning, but you haven't even provided any indication that you've done the basic tuning.

You haven't provided the basic information about server configurations or platform or software versions. You haven't provided any information on your data characteristics or query types. You haven't provided any information on the network topology, client load generator characteristics, etc.

Without such basic information, it's impossible to give any meaningful advice.

Please note - Cache was primed before the queries were made.

Regards Shilpa Sethi

A R I C E N T Plot 6, Electronic City, Sector - 18, Gurgaon 122015 Haryana, India

-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Saturday, November 15, 2008 1:03 AM To: Shilpa Sethi; openldap-technical@openldap.org Subject: RE: Issues in Performance Tesing of OpenLDAP using SLAMD

--On Friday, November 14, 2008 5:49 PM +0530 Shilpa Sethi shilpa2.sethi@aricent.com wrote:

...

Hi,

Thanks a lot for the quick reply.

I Tried out further tests with a set of 50 clients with 1 thread. I used 2 slamd client manager from 2 different DL585 machines, Further, Started 50 clients on both the servers making in all 100 SLAMD clients.

The results obtained with the tests with 60, 80 and 100 clients resulted in CPU Utilization of 100% in the initial run of ten minutes. CPU reported an average of 100% utilized while after 10 minutes, it started oscillating between 0% utilized and 100% utilized, resulting in slow down of average.

(a) Did you prime the cache before you ran your read test? (b) You may have to adjust interval periods between reads on the slamd clients if you're shooting for a particular CPU usage rate. That was never the objective of any of my tests, so I don't know.

--Quanah

--

Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc

---

Zimbra :: the leader in open source messaging and collaboration

"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error,please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."

Hi,

Information that you asked for was provided in the first mail to the forum about the issue. http://markmail.org/message/bqkx6tk5gmc3fuw4

However, describing the setup again here for quick reference:

Aim - Benchmark openLDAP Server. Hardware Used ************** Test Servers - 1. Two identical servers (Servers 'NodeA' and 'NodeB') with the following configuration: * Sun Fire X4200 M2 x64 Server * 2x AMD Opteron Model 2220 (2.8GHz/1MB) processor * 1x 4GB PC2-5300 DDR2-667 memory * 2x 146GB HDD 15K RPM (Hardware RAID-1, so available disk space is 146GB).

Load Generator Server Specs - 1. One Server with specifications : * Sun Fire X4200 M2 x64 Server * 2x AMD Opteron Model 2218 * 4x 2GB PC2-5300 DDR2-667 memory * 4x 73GB HDD 10K RPM. 2. Two DL585 Machines with specifications: * 2x AMD Opteron(tm) Processor 8220 * 8Gb memory

Software installation common to all servers: ******************************************** 1. JDK 1.6 2. OpenLDAP version 2.3.43.0 (CDS 3.11.0)

For traffic injection, Java based in-house tool that is built over JLDAP library is being used.

The database is having 1 Million subscribers and both the cache sizes are set as follows in slapd.conf : cachesize 250000 idlcachesize 250000 i.e 25% of the subscriber base.

bdb cache settings in DB_CONFIG set_cachesize 1 0 1

Node A and Node B are setup in Mirror-Mode replication.

Read Traffic - Read Query to the LDAP aims to retrieve a non-indexed 10 digit numbered attribute. This number is generated at random using Java based tool mentioned above.

Servers are connected via 1 Gbps Switch in a star topology. Cache was primed prior to running test.

Hope the information provided would suffice or let me know if further info is required to resolve the matter. Also please suggest what all should be included in the system tuning. Thanks in advance.

Regards Shilpa Sethi

A R I C E N T Plot 6, Electronic City, Sector - 18, Gurgaon 122015 Haryana, India

-----Original Message----- From: Howard Chu [mailto:hyc@symas.com] Sent: Thursday, November 20, 2008 3:54 AM To: Shilpa Sethi Cc: Quanah Gibson-Mount; openldap-technical@openldap.org Subject: Re: Issues in Performance Tesing of OpenLDAP using SLAMD

Shilpa Sethi wrote:

Hi,

I changed the approach for Performance testing. We developed an in-house Java based tool that uses JLDAP library to query openLDAP database.

4 injector machines were used to inject the Traffic to two openLDAP server running in Mirror-mode replication.

However result pattern obtained is very similar to those reported by SLAMD. PFA, the result sheet that shows query made to Openldap by 4 machines and the respective response time/interval.

Please look into the sheet attached and suggest a possible cause for such a behavior by openLDAP. In some intervals the responses returned are very high while in some others it is 0.

Looking forward for a positive reply from the members of forum.

You haven't provided any useful information to which anyone can respond. Benchmarking is a lot more demanding than just tuning, but you haven't even provided any indication that you've done the basic tuning.

You haven't provided the basic information about server configurations or platform or software versions. You haven't provided any information on your data characteristics or query types. You haven't provided any information on the network topology, client load generator characteristics, etc.

Without such basic information, it's impossible to give any meaningful advice.

Please note - Cache was primed before the queries were made.

Regards Shilpa Sethi

A R I C E N T Plot 6, Electronic City, Sector - 18, Gurgaon 122015 Haryana, India

-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Saturday, November 15, 2008 1:03 AM To: Shilpa Sethi; openldap-technical@openldap.org Subject: RE: Issues in Performance Tesing of OpenLDAP using SLAMD

--On Friday, November 14, 2008 5:49 PM +0530 Shilpa Sethi shilpa2.sethi@aricent.com wrote:

...

Hi,

Thanks a lot for the quick reply.

I Tried out further tests with a set of 50 clients with 1 thread. I used 2 slamd client manager from 2 different DL585 machines, Further, Started 50 clients on both the servers making in all 100 SLAMD clients.

The results obtained with the tests with 60, 80 and 100 clients resulted in CPU Utilization of 100% in the initial run of ten minutes. CPU reported an average of 100% utilized while after 10 minutes, it started oscillating between 0% utilized and 100% utilized, resulting in slow down of average.

(a) Did you prime the cache before you ran your read test? (b) You may have to adjust interval periods between reads on the slamd clients if you're shooting for a particular CPU usage rate. That was never the objective of any of my tests, so I don't know.

--Quanah

--

Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc

---

Zimbra :: the leader in open source messaging and collaboration

"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error,please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."

-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error,please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."

Both the benchmarking servers have Solaris 10 - x86. I am not sure about the Memory Management S/w. Is it the one bundled with OpenLDAP or is it the one , the server uses.

My understanding is that, it may help but currently, we are not facing any memory related issues in running the tests.

The values for set_cachesize was set to 1 0 1 as that is the value we have in the production system. Though it should be 2 0 1 as suggested in DB_CONFIG documentation. Can this be a cause of the problems that we are facing.

The following is the list of problems that we have been facing in our Benchmarking activity -

Regards Shilpa Sethi

A R I C E N T Plot 6, Electronic City, Sector - 18, Gurgaon 122015 Haryana, India

-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Friday, November 21, 2008 12:03 AM To: Shilpa Sethi; Howard Chu Cc: openldap-technical@openldap.org Subject: RE: Issues in Performance Tesing of OpenLDAP using SLAMD

--On Thursday, November 20, 2008 7:59 PM +0530 Shilpa Sethi shilpa2.sethi@aricent.com wrote:

Hi,

Information that you asked for was provided in the first mail to the forum about the issue. http://markmail.org/message/bqkx6tk5gmc3fuw4

What OS? What memory manager? Have you verified that DB_CONFIG is properly tuned (i.e., how did you come up with the 1 0 1 setting?) etc.

--Quanah

--

Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error,please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."