Hi Dieter,

I need to tune any parameter in DB_CONFIG file for this or not? Because I am using default DB_CONFIG file.

Thanks & Regards, Devender Singh Senior Unix Administrator, (SOA Support Team) ________________________________________________________________________________________________________________________ SDG Software India Pvt. Ltd A-10, Sector 2, Noida 201301, U.P, India M: +91-9910024231 O: +91.120.401.4000 F: +91.120.401.4020 website : www.sdgc.com ________________________________________________________________________________________________________________________ Please Note: The e-mail content is intended for the sole use of the intended recipient/s and may contain material that is CONFIDENTIAL AND PRIVATE COMPANY INFORMATION. Any review or reliance by others or copying or distribution or forwarding of any or all of the contents in this message is STRICTLY PROHIBITED. If you have erroneously received this message, please delete it immediately and notify the sender. Before opening any attachments please check them for viruses and defects.

-----Original Message----- From: openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] On Behalf Of Dieter Kluenter Sent: Tuesday, August 17, 2010 1:12 PM To: openldap-technical@openldap.org Subject: Re: Openldap2.4.16 performance issue

"Singh, Devender (GE Capital, consultant)" Devender.Singh2@ge.com writes:

Hi All,

I need help for openldap slapd 200% cpu utilization issue.

I have configured three openldap servers(2 Masters and 1 Slave). I have

configure PEN load balancer for failover setup on 2 master openldap servers.It

means application server first of all hit the loadbalancer port and than PEN LB

will forwad that request to Master 1 or 2 openldap server.

Hardware configuration on all boxes:

OS: RHEL5(x86_64)

RAM: 2GB

Number of CPU: 2(Intel(R) Xeon(R)2.00GHz)

Number of BDB databases: 2

Databse1:

Number of users : 830000

Number of dns: 830000

Database2:

Number of users: 2000

Number of dns: 800000

My Application1 using Databse1 and Application2 using Databse2.

Application2 just authenticating the users and store last 10 password history

only, It means Application1 is not using openldap too much.

In Application2(90% dependent on openldap) every user have 1000+ sub leafs

entries. when I want to do major changes(number of leafs write) into this, it.s

got hanged and got socket closed error in application logs and CPU utilization

goes 200% and RAM 52%. My DB_CONFIG file is below:

Most likely to many concurrent write operations. You should move the db logfiles onto a different disk.

-Dieter

Off the top of my head:

1. What indexes have been created? Do they match the attributes that your applications use most often?

2. In this age of cheap RAM, 2GB RAM for a server seems puny. Latest Dell R710s come packed with 32-64GB RAM. Consider a hardware upgrade.

3. Configure LB for active-active instead of active-passive?

4. Turn on logging (any), run a small sample from the application and see what transactions eat up most cycles on the LDAP server.

5. Upgrade from 2.4.16 to 2.4.xx?

From: openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] On Behalf Of Singh, Devender (GE Capital, consultant) Sent: Wednesday, August 18, 2010 3:15 PM To: hyc@symas.com Cc: openldap-technical@openldap.org Subject: RE: Openldap2.4.16 performance issue

Hi Chu,

Please help me on my below issue. It's very urgent.

Thanks & Regards, Devender Singh Senior Unix Administrator,

From: openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] On Behalf Of Singh, Devender (GE Capital, consultant) Sent: Tuesday, August 17, 2010 5:12 AM To: openldap-technical@openldap.org Subject: Openldap2.4.16 performance issue

Hi All,

I need help for openldap slapd 200% cpu utilization issue.

I have configured three openldap servers(2 Masters and 1 Slave). I have

configure PEN load balancer for failover setup on 2 master openldap servers.It

means application server first of all hit the loadbalancer port and than PEN LB

will forwad that request to Master 1 or 2 openldap server.

Hardware configuration on all boxes:

OS: RHEL5(x86_64)

RAM: 2GB

Number of CPU: 2(Intel(R) Xeon(R)2.00GHz)

Number of BDB databases: 2

Databse1:

Number of users : 830000

Number of dns: 830000

Database2:

Number of users: 2000

Number of dns: 800000

My Application1 using Databse1 and Application2 using Databse2.

Application2 just authenticating the users and store last 10 password history

only, It means Application1 is not using openldap too much.

In Application2(90% dependent on openldap) every user have 1000+ sub leafs

entries. when I want to do major changes(number of leafs write) into this, it.s

got hanged and got socket closed error in application logs and CPU utilization

goes 200% and RAM 52%. My DB_CONFIG file is below:

# $OpenLDAP: pkg/ldap/servers/slapd/DB_CONFIG,v 1.3.2.4 2007/12/18 11:53:27

ghenry Exp $

# Example DB_CONFIG file for use with slapd(8) BDB/HDB databases.

#

# See the Oracle Berkeley DB documentation

# http://www.oracle.com/technology/documentation/berkeley-db/db/ref/env/db_config.html

# for detail description of DB_CONFIG syntax and semantics.

#

# Hints can also be found in the OpenLDAP Software FAQ

# http://www.openldap.org/faq/index.cgi?file=2

# in particular:

# http://www.openldap.org/faq/index.cgi?file=1075

# Note: most DB_CONFIG settings will take effect only upon rebuilding

# the DB environment.

# one 0.25 GB cache

set_cachesize 0 268435456 1

# Data Directory

#set_data_dir db

# Transaction Log settings

set_lg_regionmax 262144

set_lg_bsize 2097152

#set_lg_dir logs

# Note: special DB_CONFIG flags are no longer needed for "quick"

# slapadd(8) or slapindex(8) access (see their -q option).

Please help me here that what I need to do for better performance. Thanks in

advance.

My contact number is +919650477441

Thanks & Regards, Devender Singh Senior Unix Administrator,

--On August 19, 2010 4:41:14 AM +0530 "Singh, Devender (GE Capital, consultant)" Devender.Singh2@ge.com wrote:

5.   Upgrade from 2.4.16 to 2.4.xx?---I don’t think that up
   

gradation will solve this issue.

I disagree, I think it could have a significant result in your issue.

Here my concern is:

I think I need to set below parameter in DB_CONFIG file:

set_cachesize (but what should be the tuned value according to my data?)

Based on what you wrote before, I'd set it to use 500MB

--Quanah

Hi, are you sure that just "equality index" is sufficient? Most applications do substring searches, e.g. "sn=sin*".

I'd recommend to add "substr" to the often used attributes.

Regards, Jochen.

Am 19.08.2010 01:11, schrieb Singh, Devender (GE Capital, consultant):

Please find the answers:

1. What indexes have been created? Do they match the attributes that

your applications use most often? --- All required attributes related application indexed(equality)

2. In this age of cheap RAM, 2GB RAM for a server seems puny. Latest

Dell R710s come packed with 32-64GB RAM. Consider a hardware upgrade.—We can increase it by 3-4 GB only

3. Configure LB for active-active instead of active-passive?—I have

configured PEN LB as a active-active(both master are active-active)

4. Turn on logging (any), run a small sample from the application and

see what transactions eat up most cycles on the LDAP server.(I have configured loglevel 256, if I set loglevel -1, it will slow the read write speed from/to openldap server and also eating cpu. )

5. Upgrade from 2.4.16 to 2.4.xx?---I don’t think that up gradation will

solve this issue.

Here my concern is:

I think I need to set below parameter in DB_CONFIG file:

*/set_cachesize (/*but what should be the tuned value according to my data?)

Thanks & Regards,//

*Devender Singh*

Senior Unix Administrator,//

(SOA Support Team)//

/________________________________________________________________________________________________________________________///

*SDG Software India Pvt. Ltd*//

A-10, Sector 2, Noida 201301, U.P, India//

M: +91-9910024231 O: +91.120.401.4000 F: +91.120.401.4020//

website : www.sdgc.com http://www.sdgc.com///

/________________________________________________________________________________________________________________________///

*Please Note:* The e-mail content is intended for the sole use of the intended recipient/s and may contain material that is CONFIDENTIAL AND PRIVATE COMPANY INFORMATION. Any review or reliance by others or copying or distribution or forwarding of any or all of the contents in this message is STRICTLY PROHIBITED. If you have erroneously received this message, please delete it immediately and notify the sender. Before opening any attachments please check them for viruses and defects.

*From:* openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] *On Behalf Of *Siddhartha Jain *Sent:* Thursday, August 19, 2010 4:17 AM *To:* openldap-technical@openldap.org *Subject:* RE: Openldap2.4.16 performance issue

Off the top of my head:

6. What indexes have been created? Do they match the attributes that

your applications use most often?

7. In this age of cheap RAM, 2GB RAM for a server seems puny. Latest

Dell R710s come packed with 32-64GB RAM. Consider a hardware upgrade.

8. Configure LB for active-active instead of active-passive?

9. Turn on logging (any), run a small sample from the application and

see what transactions eat up most cycles on the LDAP server.

10. Upgrade from 2.4.16 to 2.4.xx?

*From:* openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] *On Behalf Of *Singh, Devender (GE Capital, consultant) *Sent:* Wednesday, August 18, 2010 3:15 PM *To:* hyc@symas.com *Cc:* openldap-technical@openldap.org *Subject:* RE: Openldap2.4.16 performance issue

Hi Chu,

Please help me on my below issue. It’s very urgent.

Thanks & Regards,//

*Devender Singh*

Senior Unix Administrator,//

*From:* openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] *On Behalf Of *Singh, Devender (GE Capital, consultant) *Sent:* Tuesday, August 17, 2010 5:12 AM *To:* openldap-technical@openldap.org *Subject:* Openldap2.4.16 performance issue

Hi All,

I need help for openldap slapd 200% cpu utilization issue.

I have configured three openldap servers(2 Masters and 1 Slave). I have

configure PEN load balancer for failover setup on 2 master openldap servers.It

means application server first of all hit the loadbalancer port and than PEN LB

will forwad that request to Master 1 or 2 openldap server.

Hardware configuration on all boxes:

OS: RHEL5(x86_64)

RAM: 2GB

Number of CPU: 2(Intel(R) Xeon(R)2.00GHz)

Number of BDB databases: 2

Databse1:

Number of users : 830000

Number of dns: 830000

Database2:

Number of users: 2000

Number of dns: 800000

My Application1 using Databse1 and Application2 using Databse2.

Application2 just authenticating the users and store last 10 password history

only, It means Application1 is not using openldap too much.

In Application2(90% dependent on openldap) every user have 1000+ sub leafs

entries. when I want to do major changes(number of leafs write) into this, it.s

got hanged and got socket closed error in application logs and CPU utilization

goes 200% and RAM 52%. My DB_CONFIG file is below:

*/# $OpenLDAP: pkg/ldap/servers/slapd/DB_CONFIG,v 1.3.2.4 2007/12/18 11:53:27/*

*/ghenry Exp $/*

*/# Example DB_CONFIG file for use with slapd(8) BDB/HDB databases./*

*/#/*

*/# See the Oracle Berkeley DB documentation/*

*/# http://www.oracle.com/technology/documentation/berkeley-db/db/ref/env/db_config.html/*

*/# for detail description of DB_CONFIG syntax and semantics./*

*/#/*

*/# Hints can also be found in the OpenLDAP Software FAQ/*

*/# http://www.openldap.org/faq/index.cgi?file=2/*

*/# in particular:/*

*/# http://www.openldap.org/faq/index.cgi?file=1075/*

*/ /*

*/# Note: most DB_CONFIG settings will take effect only upon rebuilding/*

*/# the DB environment./*

*/ /*

*/# one 0.25 GB cache/*

*/set_cachesize 0 268435456 1/*

*/ /*

*/# Data Directory/*

*/#set_data_dir db/*

*/ /*

*/# Transaction Log settings/*

*/set_lg_regionmax 262144/*

*/set_lg_bsize 2097152/*

*/#set_lg_dir logs/*

*/ /*

*/# Note: special DB_CONFIG flags are no longer needed for"quick"/*

*/# slapadd(8) or slapindex(8) access (see their -q option)./*

Please help me here that what I need to do for better performance. Thanks in

advance.

My contact number is +919650477441

Thanks & Regards,//

*Devender Singh*

Senior Unix Administrator,//

In that case, you should ask the *client* to give you a solution. Seriously, if you do not have complete control over LDAP configuration or if *client* dictates certain config parameters then it is best to report this as a bug to your application team.

Do you have a test instance to do test config changes on?

-----Original Message----- From: openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] On Behalf Of Singh, Devender (GE Capital, consultant) Sent: Wednesday, August 18, 2010 5:02 PM To: Keutel, Jochen; openldap-technical@openldap.org Subject: RE: Openldap2.4.16 performance issue

As per the client requirement there is no need of substring indexing

-----Original Message----- From: openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] On Behalf Of Keutel, Jochen Sent: Thursday, August 19, 2010 5:26 AM To: openldap-technical@openldap.org Subject: Re: Openldap2.4.16 performance issue

Hi, are you sure that just "equality index" is sufficient? Most applications do substring searches, e.g. "sn=sin*".

I'd recommend to add "substr" to the often used attributes.

Regards, Jochen.

Am 19.08.2010 01:11, schrieb Singh, Devender (GE Capital, consultant):

Please find the answers:

1. What indexes have been created? Do they match the attributes that

your applications use most often? --- All required attributes related application indexed(equality)

2. In this age of cheap RAM, 2GB RAM for a server seems puny. Latest

Dell R710s come packed with 32-64GB RAM. Consider a hardware

upgrade.-We

can increase it by 3-4 GB only

3. Configure LB for active-active instead of active-passive?-I have

configured PEN LB as a active-active(both master are active-active)

4. Turn on logging (any), run a small sample from the application and

see what transactions eat up most cycles on the LDAP server.(I have configured loglevel 256, if I set loglevel -1, it will slow the read write speed from/to openldap server and also eating cpu. )

5. Upgrade from 2.4.16 to 2.4.xx?---I don't think that up gradation

will

solve this issue.

Here my concern is:

I think I need to set below parameter in DB_CONFIG file:

*/set_cachesize (/*but what should be the tuned value according to my

data?)

Thanks & Regards,//

*Devender Singh*

Senior Unix Administrator,//

(SOA Support Team)//

/_______________________________________________________________________ _________________________________________________///

*SDG Software India Pvt. Ltd*//

A-10, Sector 2, Noida 201301, U.P, India//

M: +91-9910024231 O: +91.120.401.4000 F: +91.120.401.4020//

website : www.sdgc.com http://www.sdgc.com///

/_______________________________________________________________________ _________________________________________________///

*Please Note:* The e-mail content is intended for the sole use of the intended recipient/s and may contain material that is CONFIDENTIAL AND PRIVATE COMPANY INFORMATION. Any review or reliance by others or

copying

or distribution or forwarding of any or all of the contents in this message is STRICTLY PROHIBITED. If you have erroneously received this message, please delete it immediately and notify the sender. Before opening any attachments please check them for viruses and defects.

*From:* openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] *On Behalf Of *Siddhartha Jain *Sent:* Thursday, August 19, 2010 4:17 AM *To:* openldap-technical@openldap.org *Subject:* RE: Openldap2.4.16 performance issue

Off the top of my head:

6. What indexes have been created? Do they match the attributes that

your applications use most often?

7. In this age of cheap RAM, 2GB RAM for a server seems puny. Latest

Dell R710s come packed with 32-64GB RAM. Consider a hardware upgrade.

8. Configure LB for active-active instead of active-passive?

9. Turn on logging (any), run a small sample from the application and

see what transactions eat up most cycles on the LDAP server.

10. Upgrade from 2.4.16 to 2.4.xx?

*From:* openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] *On Behalf Of *Singh, Devender (GE Capital, consultant) *Sent:* Wednesday, August 18, 2010 3:15 PM *To:* hyc@symas.com *Cc:* openldap-technical@openldap.org *Subject:* RE: Openldap2.4.16 performance issue

Hi Chu,

Please help me on my below issue. It's very urgent.

Thanks & Regards,//

*Devender Singh*

Senior Unix Administrator,//

*From:* openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] *On Behalf Of *Singh, Devender (GE Capital, consultant) *Sent:* Tuesday, August 17, 2010 5:12 AM *To:* openldap-technical@openldap.org *Subject:* Openldap2.4.16 performance issue

Hi All,

I need help for openldap slapd 200% cpu utilization issue.

I have configured three openldap servers(2 Masters and 1 Slave). I

have

configure PEN load balancer for failover setup on 2 master openldap

servers.It

means application server first of all hit the loadbalancer port and

than PEN LB

will forwad that request to Master 1 or 2 openldap server.

Hardware configuration on all boxes:

OS: RHEL5(x86_64)

RAM: 2GB

Number of CPU: 2(Intel(R) Xeon(R)2.00GHz)

Number of BDB databases: 2

Databse1:

Number of users : 830000

Number of dns: 830000

Database2:

Number of users: 2000

Number of dns: 800000

My Application1 using Databse1 and Application2 using Databse2.

Application2 just authenticating the users and store last 10 password

history

only, It means Application1 is not using openldap too much.

In Application2(90% dependent on openldap) every user have 1000+ sub

leafs

entries. when I want to do major changes(number of leafs write) into

this, it.s

got hanged and got socket closed error in application logs and CPU

utilization

goes 200% and RAM 52%. My DB_CONFIG file is below:

*/# $OpenLDAP: pkg/ldap/servers/slapd/DB_CONFIG,v 1.3.2.4 2007/12/18

11:53:27/*

*/ghenry Exp $/*

*/# Example DB_CONFIG file for use with slapd(8) BDB/HDB databases./*

*/#/*

*/# See the Oracle Berkeley DB documentation/*

*/#

http://www.oracle.com/technology/documentation/berkeley-db/db/ref/env/d b_config.html/*

*/# for detail description of DB_CONFIG syntax and semantics./*

*/#/*

*/# Hints can also be found in the OpenLDAP Software FAQ/*

*/# http://www.openldap.org/faq/index.cgi?file=2/*

*/# in particular:/*

*/# http://www.openldap.org/faq/index.cgi?file=1075/*

*/ /*

*/# Note: most DB_CONFIG settings will take effect only upon

rebuilding/*

*/# the DB environment./*

*/ /*

*/# one 0.25 GB cache/*

*/set_cachesize 0 268435456 1/*

*/ /*

*/# Data Directory/*

*/#set_data_dir db/*

*/ /*

*/# Transaction Log settings/*

*/set_lg_regionmax 262144/*

*/set_lg_bsize 2097152/*

*/#set_lg_dir logs/*

*/ /*

*/# Note: special DB_CONFIG flags are no longer needed for"quick"/*

*/# slapadd(8) or slapindex(8) access (see their -q option)./*

Please help me here that what I need to do for better performance.

Thanks in

advance.

My contact number is +919650477441

Thanks & Regards,//

*Devender Singh*

Senior Unix Administrator,//

1) You seriously need to use OpenLDAP 2.4.23. I don't care if you don't think that'll solve the issue or not. ;)

2) You need to state which backend you are using (back-hdb, back-bdb, etc)

3) You need to state your updated DB_CONFIG, based on the cachesize info I suggested earlier

4) You need to state what version of BDB back-hdb/back-bdb is using. This is critical to know

5) You need to state if you have modified the "threads" setting for slapd.conf or related for cn=config, depending which you are using.

--Quanah

--On August 19, 2010 6:05:19 AM +0530 "Singh, Devender (GE Capital, consultant)" Devender.Singh2@ge.com wrote:

Before migration, the application was running fine without any issue on IBM Tivoli directory server 5.2.

Yes we have a test server. Please suggest me configuration level changes.

Thanks & Regards, Devender Singh Senior Unix Administrator, (SOA Support Team) ________________________________________________________________________ ________________________________________________ SDG Software India Pvt. Ltd A-10, Sector 2, Noida 201301, U.P, India M: +91-9910024231 O: +91.120.401.4000 F: +91.120.401.4020 website : www.sdgc.com ________________________________________________________________________ ________________________________________________ Please Note: The e-mail content is intended for the sole use of the intended recipient/s and may contain material that is CONFIDENTIAL AND PRIVATE COMPANY INFORMATION. Any review or reliance by others or copying or distribution or forwarding of any or all of the contents in this message is STRICTLY PROHIBITED. If you have erroneously received this message, please delete it immediately and notify the sender. Before opening any attachments please check them for viruses and defects.

-----Original Message----- From: openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] On Behalf Of Siddhartha Jain Sent: Thursday, August 19, 2010 5:58 AM To: openldap-technical@openldap.org Subject: RE: Openldap2.4.16 performance issue

In that case, you should ask the *client* to give you a solution. Seriously, if you do not have complete control over LDAP configuration or if *client* dictates certain config parameters then it is best to report this as a bug to your application team.

Do you have a test instance to do test config changes on?

-----Original Message----- From: openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] On Behalf Of Singh, Devender (GE Capital, consultant) Sent: Wednesday, August 18, 2010 5:02 PM To: Keutel, Jochen; openldap-technical@openldap.org Subject: RE: Openldap2.4.16 performance issue

As per the client requirement there is no need of substring indexing

-----Original Message----- From: openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] On Behalf Of Keutel, Jochen Sent: Thursday, August 19, 2010 5:26 AM To: openldap-technical@openldap.org Subject: Re: Openldap2.4.16 performance issue

Hi, are you sure that just "equality index" is sufficient? Most applications do substring searches, e.g. "sn=sin*".

I'd recommend to add "substr" to the often used attributes.

Regards, Jochen.

Am 19.08.2010 01:11, schrieb Singh, Devender (GE Capital, consultant):

...

Please find the answers:

1. What indexes have been created? Do they match the attributes that

your applications use most often? --- All required attributes related application indexed(equality)

2. In this age of cheap RAM, 2GB RAM for a server seems puny. Latest

Dell R710s come packed with 32-64GB RAM. Consider a hardware

upgrade.-We

...

can increase it by 3-4 GB only

3. Configure LB for active-active instead of active-passive?-I have

configured PEN LB as a active-active(both master are active-active)

4. Turn on logging (any), run a small sample from the application and

see what transactions eat up most cycles on the LDAP server.(I have configured loglevel 256, if I set loglevel -1, it will slow the read write speed from/to openldap server and also eating cpu. )

5. Upgrade from 2.4.16 to 2.4.xx?---I don't think that up gradation

will

...

solve this issue.

Here my concern is:

I think I need to set below parameter in DB_CONFIG file:

*/set_cachesize (/*but what should be the tuned value according to my

data?)

...

Thanks & Regards,//

*Devender Singh*

Senior Unix Administrator,//

(SOA Support Team)//

/_______________________________________________________________________ _________________________________________________///

...

*SDG Software India Pvt. Ltd*//

A-10, Sector 2, Noida 201301, U.P, India//

M: +91-9910024231 O: +91.120.401.4000 F: +91.120.401.4020//

website : www.sdgc.com http://www.sdgc.com///

/_______________________________________________________________________ _________________________________________________///

...

*Please Note:* The e-mail content is intended for the sole use of the intended recipient/s and may contain material that is CONFIDENTIAL AND PRIVATE COMPANY INFORMATION. Any review or reliance by others or

copying

...

or distribution or forwarding of any or all of the contents in this message is STRICTLY PROHIBITED. If you have erroneously received this message, please delete it immediately and notify the sender. Before opening any attachments please check them for viruses and defects.

*From:* openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] *On Behalf Of *Siddhartha Jain *Sent:* Thursday, August 19, 2010 4:17 AM *To:* openldap-technical@openldap.org *Subject:* RE: Openldap2.4.16 performance issue

Off the top of my head:

6. What indexes have been created? Do they match the attributes that

your applications use most often?

7. In this age of cheap RAM, 2GB RAM for a server seems puny. Latest

Dell R710s come packed with 32-64GB RAM. Consider a hardware upgrade.

8. Configure LB for active-active instead of active-passive?

9. Turn on logging (any), run a small sample from the application and

see what transactions eat up most cycles on the LDAP server.

10. Upgrade from 2.4.16 to 2.4.xx?

*From:* openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] *On Behalf Of *Singh, Devender (GE Capital, consultant) *Sent:* Wednesday, August 18, 2010 3:15 PM *To:* hyc@symas.com *Cc:* openldap-technical@openldap.org *Subject:* RE: Openldap2.4.16 performance issue

Hi Chu,

Please help me on my below issue. It's very urgent.

Thanks & Regards,//

*Devender Singh*

Senior Unix Administrator,//

*From:* openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] *On Behalf Of *Singh, Devender (GE Capital, consultant) *Sent:* Tuesday, August 17, 2010 5:12 AM *To:* openldap-technical@openldap.org *Subject:* Openldap2.4.16 performance issue

Hi All,

I need help for openldap slapd 200% cpu utilization issue.

I have configured three openldap servers(2 Masters and 1 Slave). I

have

...

configure PEN load balancer for failover setup on 2 master openldap

servers.It

...

means application server first of all hit the loadbalancer port and

than PEN LB

...

will forwad that request to Master 1 or 2 openldap server.

Hardware configuration on all boxes:

OS: RHEL5(x86_64)

RAM: 2GB

Number of CPU: 2(Intel(R) Xeon(R)2.00GHz)

Number of BDB databases: 2

Databse1:

Number of users : 830000

Number of dns: 830000

Database2:

Number of users: 2000

Number of dns: 800000

My Application1 using Databse1 and Application2 using Databse2.

Application2 just authenticating the users and store last 10 password

history

...

only, It means Application1 is not using openldap too much.

In Application2(90% dependent on openldap) every user have 1000+ sub

leafs

...

entries. when I want to do major changes(number of leafs write) into

this, it.s

...

got hanged and got socket closed error in application logs and CPU

utilization

...

goes 200% and RAM 52%. My DB_CONFIG file is below:

*/# $OpenLDAP: pkg/ldap/servers/slapd/DB_CONFIG,v 1.3.2.4 2007/12/18

11:53:27/*

...

*/ghenry Exp $/*

*/# Example DB_CONFIG file for use with slapd(8) BDB/HDB databases./*

*/#/*

*/# See the Oracle Berkeley DB documentation/*

*/#

http://www.oracle.com/technology/documentation/berkeley-db/db/ref/env/d b_config.html/*

...

*/# for detail description of DB_CONFIG syntax and semantics./*

*/#/*

*/# Hints can also be found in the OpenLDAP Software FAQ/*

*/# http://www.openldap.org/faq/index.cgi?file=2/*

*/# in particular:/*

*/# http://www.openldap.org/faq/index.cgi?file=1075/*

*/ /*

*/# Note: most DB_CONFIG settings will take effect only upon

rebuilding/*

...

*/# the DB environment./*

*/ /*

*/# one 0.25 GB cache/*

*/set_cachesize 0 268435456 1/*

*/ /*

*/# Data Directory/*

*/#set_data_dir db/*

*/ /*

*/# Transaction Log settings/*

*/set_lg_regionmax 262144/*

*/set_lg_bsize 2097152/*

*/#set_lg_dir logs/*

*/ /*

*/# Note: special DB_CONFIG flags are no longer needed for"quick"/*

*/# slapadd(8) or slapindex(8) access (see their -q option)./*

Please help me here that what I need to do for better performance.

Thanks in

...

advance.

My contact number is +919650477441

Thanks & Regards,//

*Devender Singh*

Senior Unix Administrator,//

Yes I did it, but not getting good performance. I restart slapd every time when cpu goes 200%.

Thanks & Regards,

Devender Singh

Senior Unix Administrator,

From: Chris Jacobs [mailto:Chris.Jacobs@apollogrp.edu] Sent: Thursday, August 19, 2010 4:17 AM To: Singh, Devender (GE Capital, consultant); hyc@symas.com Cc: openldap-technical@openldap.org Subject: RE: Openldap2.4.16 performance issue

Devender,

You did see this email reply, right:

"Singh, Devender (GE Capital, consultant)" Devender.Singh2@ge.com writes:

Hi Dieter,

 I need to tune any parameter in DB_CONFIG file for this or not? Because I am using default DB_CONFIG file.

Just edit DB_CONFIG

set_log_dir /mountpoint/path/to/

-Dieter

Please find the below answers:

[root@abc openldap-data-ge_cw]# du -sh *.bdb 3.6M br.bdb 72K cn.bdb 32K displayName.bdb 234M dn2id.bdb 104K gr.bdb 419M id2entry.bdb 56K mail.bdb 1.4M objectClass.bdb 2.9M pf.bdb 212K pr.bdb 72K sn.bdb 72K uid.bdb

[root@abc openldap-data-ge_cw]# getconf LONG_BIT 32

Thanks & Regards, Devender Singh Senior Unix Administrator, (SOA Support Team) ________________________________________________________________________ ________________________________________________ SDG Software India Pvt. Ltd A-10, Sector 2, Noida 201301, U.P, India M: +91-9910024231 O: +91.120.401.4000 F: +91.120.401.4020 website : www.sdgc.com ________________________________________________________________________ ________________________________________________ Please Note: The e-mail content is intended for the sole use of the intended recipient/s and may contain material that is CONFIDENTIAL AND PRIVATE COMPANY INFORMATION. Any review or reliance by others or copying or distribution or forwarding of any or all of the contents in this message is STRICTLY PROHIBITED. If you have erroneously received this message, please delete it immediately and notify the sender. Before opening any attachments please check them for viruses and defects.

-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Thursday, August 19, 2010 4:34 AM To: Singh, Devender (GE Capital, consultant); Chris Jacobs Cc: openldap-technical@openldap.org Subject: RE: Openldap2.4.16 performance issue

--On August 19, 2010 4:23:31 AM +0530 "Singh, Devender (GE Capital, consultant)" Devender.Singh2@ge.com wrote:

Yes I did it, but not getting good performance. I restart slapd every time when cpu goes 200%.

What is the size of your *.bdb files? Are you on a 32-bit or 64-bit server?

--Quanah

I agree with you. Please suggest me what to do for resolution of this issue.

Thanks & Regards, Devender Singh Senior Unix Administrator, (SOA Support Team) ________________________________________________________________________________________________________________________ SDG Software India Pvt. Ltd A-10, Sector 2, Noida 201301, U.P, India M: +91-9910024231 O: +91.120.401.4000 F: +91.120.401.4020 website : www.sdgc.com ________________________________________________________________________________________________________________________ Please Note: The e-mail content is intended for the sole use of the intended recipient/s and may contain material that is CONFIDENTIAL AND PRIVATE COMPANY INFORMATION. Any review or reliance by others or copying or distribution or forwarding of any or all of the contents in this message is STRICTLY PROHIBITED. If you have erroneously received this message, please delete it immediately and notify the sender. Before opening any attachments please check them for viruses and defects.

-----Original Message----- From: openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] On Behalf Of Dieter Kluenter Sent: Thursday, August 19, 2010 2:06 PM To: openldap-technical@openldap.org Subject: Re: Openldap2.4.16 performance issue

"Singh, Devender (GE Capital, consultant)" Devender.Singh2@ge.com writes:

Please find the below answers:

[root@abc openldap-data-ge_cw]# du -sh *.bdb 3.6M br.bdb 72K cn.bdb 32K displayName.bdb 234M dn2id.bdb 104K gr.bdb 419M id2entry.bdb 56K mail.bdb 1.4M objectClass.bdb 2.9M pf.bdb 212K pr.bdb 72K sn.bdb 72K uid.bdb

I have seen the problems you describe before. Although a configured cache size of 250M and a database size of some 660M is not sufficient, it still is not such a bottleneck. To my experience a heavy cpu load is most likely based on heavy disk operations. If moving the transaction logs onto a separate disk didn't solve it, look for other concurrent read/write operations. Check whether the logs report constantly deadlocks. In some cases a journaling file system reduced performance. I experienced rather bad results with xfs.

-Dieter

Quanah Gibson-Mount quanah@zimbra.com writes:

--On August 19, 2010 9:39:58 PM +0530 "Singh, Devender (GE Capital, consultant)" Devender.Singh2@ge.com wrote:

...

I agree with you. Please suggest me what to do for resolution of this issue.

Unfortunately, Dieter is ignoring potential issues with known problems with BDB, and the fact that a number of serious issues were fixed since OpenLDAP 2.4.16. Please go answer my list of questions I mailed you yesterday.

Quanah, I quite agree with you, there have been problems with early 2.4 versions of OpenLDAP and BDB, and I am quite aware of the fixed issues since 2.4.13. But let's solve hardware problems first, if there are any. If it is assured that hardware and filesystem as the culprid can be excluded, the focus can put on actual OpenLDAP versions.

-Dieter

Quanah Gibson-Mount quanah@zimbra.com writes:

--On August 19, 2010 8:21:01 PM +0200 Dieter Kluenter dieter@dkluenter.de wrote:

...

Quanah Gibson-Mount quanah@zimbra.com writes:

...

--On August 19, 2010 9:39:58 PM +0530 "Singh, Devender (GE Capital, consultant)" Devender.Singh2@ge.com wrote:

...

I agree with you. Please suggest me what to do for resolution of this issue.

Unfortunately, Dieter is ignoring potential issues with known problems with BDB, and the fact that a number of serious issues were fixed since OpenLDAP 2.4.16. Please go answer my list of questions I mailed you yesterday.

Quanah, I quite agree with you, there have been problems with early 2.4 versions of OpenLDAP and BDB, and I am quite aware of the fixed issues since 2.4.13. But let's solve hardware problems first, if there are any. If it is assured that hardware and filesystem as the culprid can be excluded, the focus can put on actual OpenLDAP versions.

Except for that fact that I know of people who saw this exact issue with earlier OpenLDAP versions, and we know for a fact there are issues with unpatched BDB 4.7, and there are issues with BDB 4.8 prior to 4.8.30. So it is *extremely* relevant to (a) ensure he is on a current release and (b) that he's using a valid BDB version.

I still quite agree, on the other hand, some of the systems I have installed with OpenLDAP-2.4.12/13 and db-4.2.52 and a few M entries are still running without any problem, even a system with db.4.3.x and a few 10k entries is still running. There are issues with some db versions, and there are issues with OpenLDAP versions, but to my experience, filesystem and hardware related issues should not be neglected.

-Dieter

and I saw this issue with Linux KVM, RAID5 and xfs a few times.

Well, when it comes to LDAP, there are three things wrong with that sentence.

Hi Dieter,

Please find the below details:

1. hardware related

- type of storage - Simply SATA had disk attached.

- raid level, if any- No RAID

- file system of disk(s)- ext3 on LVm

- type of network, 100MB, 1G, 10G

2. is this host running on a virtual machine or on bare metal.

- if virtual machine, -Yes, OS installed on VM

-- what type ---Don’t know

Thanks & Regards,

Devender Singh

Senior Unix Administrator,

(SOA Support Team)

-----Original Message----- From: openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] On Behalf Of Dieter Kluenter Sent: Thursday, August 19, 2010 11:40 PM To: openldap-technical@openldap.org Subject: Re: Openldap2.4.16 performance issue

"Singh, Devender (GE Capital, consultant)" Devender.Singh2@ge.com writes:

I agree with you. Please suggest me what to do for resolution of this issue.

Frankly, this is simple unix system administration.

A few questions:

1. hardware related

- type of storage

- raid level, if any

- file system of disk(s)

- type of network, 100MB, 1G, 10G

2. is this host running on a virtual machine or on bare metal.

- if virtual machine,

-- what type

-Dieter

"Singh, Devender (GE Capital, consultant)" Devender.Singh2@ge.com writes:

...

Please find the below answers:

...

...

[root@abc openldap-data-ge_cw]# du -sh *.bdb

...

3.6M br.bdb

...

72K cn.bdb

...

32K displayName.bdb

...

234M dn2id.bdb

...

104K gr.bdb

...

419M id2entry.bdb

...

56K mail.bdb

...

1.4M objectClass.bdb

...

2.9M pf.bdb

...

212K pr.bdb

...

72K sn.bdb

...

72K uid.bdb

I have seen the problems you describe before. Although a configured

cache size of 250M and a database size of some 660M is not sufficient,

it still is not such a bottleneck. To my experience a heavy cpu load

is most likely based on heavy disk operations.

If moving the transaction logs onto a separate disk didn't solve it,

look for other concurrent read/write operations. Check whether the

logs report constantly deadlocks.

In some cases a journaling file system reduced performance. I

experienced rather bad results with xfs.

-Dieter

I think I see the problem here: echo "$sender" | grep -i consultant; echo $?

Seriously though, I agree with you Benjamin.

Devender: Please read ALL emails sent to you much more carefully as you have missed/ignored several of the "free" attempts to help you. If you want "free" help from a voluntary community, treat them with more respect, acknowledge all advice given (definitely do not ignore it), and try to help everyone by simply responding with extra information as and when it is asked of you!

Just out of interest, what kind of "Senior Unix Administrator" are you if you can not see a problem with this:

--snip-- - type of storage - Simply SATA had disk attached.

- raid level, if any- No RAID --snip-- if virtual machine, -Yes --snip--

and finally:

--snip-- Databse1:

Number of users : 830000 Number of dns: 830000

Database2:

Number of users: 2000 Number of dns: 800000 --snip-- In Application2(90% dependent on openldap) every user have 1000+ sub leafs entries --snip--

!!!!!!!!!!

To top it off, you're running a potentially very active LDAP database in a VM!! How would you ever expect that level of <quotation_fingers>hardware</quotation_fingers> to cope with such potential volumes of network traffic anyway??

My advice before you post back: 1. Go back through the email trail and read ALL advice and act on it appropriately.

2. Upgrade OpenLDAP to a more (if not the most) recent version.

3. Get it on bare metal (preferably something with higher spec for improved IO (and maybe an off-loaded network card).

My 2pence....

Russell Knighton

On Fri, 2010-08-20 at 13:15 +0100, Benjamin Griese wrote:

Hello,

this message exchange is really funny, since someone who calls himself an senior unix administrator is not able to answer simple questions about his own systems and wants uberfast results from an application point of view he doesn't pay any support for.

Dear Devender, please review your questions and the answers you gave on this mailinglist and how you wrote and may think of the impression other people get from you.

Thank for reading. bye. :)

On Fri, Aug 20, 2010 at 12:18, Singh, Devender (GE Capital, consultant) Devender.Singh2@ge.com wrote:

...

Hi Dieter,

Please find the below details:

1. hardware related

   • type of storage - Simply SATA had disk attached.

   • raid level, if any- No RAID

   • file system of disk(s)- ext3 on LVm

   • type of network, 100MB, 1G, 10G

2. is this host running on a virtual machine or on bare metal.

   • if virtual machine, -Yes, OS installed on VM

   -- what type ---Don’t know

Thanks & Regards,

Devender Singh

Senior Unix Administrator,

(SOA Support Team)

-----Original Message-----

From: openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] On Behalf Of Dieter Kluenter Sent: Thursday, August 19, 2010 11:40 PM To: openldap-technical@openldap.org Subject: Re: Openldap2.4.16 performance issue

"Singh, Devender (GE Capital, consultant)" Devender.Singh2@ge.com

writes:

...

...

I agree with you. Please suggest me what to do for resolution of

this

...

...

issue.

Frankly, this is simple unix system administration.

A few questions:

1. hardware related

   • type of storage

   • raid level, if any

   • file system of disk(s)

   • type of network, 100MB, 1G, 10G

2. is this host running on a virtual machine or on bare metal.

   • if virtual machine,

   -- what type

-Dieter

...

"Singh, Devender (GE Capital, consultant)" Devender.Singh2@ge.com writes:

...

...

...

Please find the below answers:

...

...

...

...

[root@abc openldap-data-ge_cw]# du -sh *.bdb

...

...

3.6M br.bdb

...

...

72K cn.bdb

...

...

32K displayName.bdb

...

...

234M dn2id.bdb

...

...

104K gr.bdb

...

...

419M id2entry.bdb

...

...

56K mail.bdb

...

...

1.4M objectClass.bdb

...

...

2.9M pf.bdb

...

...

212K pr.bdb

...

...

72K sn.bdb

...

...

72K uid.bdb

...

...

I have seen the problems you describe before. Although a configured

...

cache size of 250M and a database size of some 660M is not

sufficient,

...

...

it still is not such a bottleneck. To my experience a heavy cpu

load

...

...

is most likely based on heavy disk operations.

...

If moving the transaction logs onto a separate disk didn't solve

it,

...

...

look for other concurrent read/write operations. Check whether the

...

logs report constantly deadlocks.

...

In some cases a journaling file system reduced performance. I

...

experienced rather bad results with xfs.

...

...

-Dieter

--

Dieter Klünter | Systemberatung

sip: 7770535@sipgate.de

http://www.dpunkt.de/buecher/2104.html

GPG Key ID:8EF7B6C6

-- To be or not to be -- Shakespeare | To do is to be -- Nietzsche | To be is to do -- Sartre | Do be do be do -- Sinatra

Hi,

On Fri, 20 Aug 2010, Singh, Devender (GE Capital, consultant) wrote:

Hi Dieter,

Please find the below details:

1. hardware related

• type of storage - Simply SATA had disk attached.

• raid level, if any- No RAID

• file system of disk(s)- ext3 on LVm

Apart from what all others have already said ext3 with a heavy write load is going to be a Problem with recent linux kernels.

How is the rest of you guys coping with ext3 and the fsync issue ?

Greetings Christian