FWIW, our entire ldap infrastructure is on VMs - with one exception: on an overloaded VM cluster, we had tons of random auth failures - put in a pizza box machine to replace one of the VMs (and changed load balancer config to use the VM ldap box in that location only as a fail over backup).

Our mirrored masters are VMs as well.

It's all about the load - there's nothing inherently bad about OpenLDAP on VMs - many people simply overload them.

- chris

Chris Jacobs, Systems Administrator Apollo Group | Apollo Marketing | Aptimus 2001 6th Ave Ste 3200 | Seattle, WA 98121 phone: 206.839-8245 | cell: 206.601.3256 | Fax: 208.441.9661 email: chris.jacobs@apollogrp.edu

----- Original Message ----- From: openldap-technical-bounces@OpenLDAP.org openldap-technical-bounces@OpenLDAP.org To: Dieter Kluenter dieter@dkluenter.de Cc: openldap-technical@openldap.org openldap-technical@openldap.org Sent: Mon Aug 23 11:13:43 2010 Subject: Re: Openldap2.4.16 performance issue

----- Original Message -----

From: "Dieter Kluenter" dieter@dkluenter.de To: openldap-technical@openldap.org Sent: Saturday, August 21, 2010 3:07:04 AM Subject: Re: Openldap2.4.16 performance issue

"Singh, Devender (GE Capital, consultant)" Devender.Singh2@ge.com writes:

...

Hi Dieter,

Please find the below details:

1. hardware related

   • type of storage - Simply SATA had disk attached.

   • raid level, if any- No RAID

   • file system of disk(s)- ext3 on LVm

   • type of network, 100MB, 1G, 10G

2. is this host running on a virtual machine or on bare metal.

   • if virtual machine, -Yes, OS installed on VM

   -- what type ---Don’t know

There is nothing suspicious, except for the virtual machine. Your really should carefully check layout and configuration of this VM, do not use virtual disks.

Hi Dieter,

Could you kindly explain what this means? I've been all over the inter-webs and I'm not finding anything concrete about OpenLDAP and VM's or Databases and VM's in general. The closest I came was about some database latency studies and some VMware propaganda.

We are about to launch a master and two replicas (utilizing delta-syncrepl) running in Ubuntu 10.04 on a VMware VSphere ESXi 4.0 cluster with four IBM x3650-M2's (2 Quad Nahalem and 64GB memory each) with virtual disks carved from NFS mounts to all of the VSphere servers in the cluster to facilitate HA and FT - by the VMware book.

(BTW, I took one of Howard's old posts to heart and we are following the Ubuntu playbook and we have purchased Canonical 24x7 support/maintenance. :-) )

We have had no problems with this environment in development and get better results than on bare metal with or without RAID. I know that it is recommended that the logs live on another "disk" from the database and RAID is frowned upon, but I have difficulty with a few points:

1) Separate, unprotected disks seems illogical. The last log and the BDB files are necessary to start BDB in slapd, correct? So, if you lose either disk, you're in trouble. Backups are ok, but daily seems too long a time. Seeing as we process new user accounts every 15 minutes, this would not be ideal for us.

2) RAID-1 I can understand having an issue on writes. But what about LUNs in FC from a NetApp 3140? Virtual disks on NFS in VMware? Both of these are in the best practices of both VMware and NetApp documentation.

3) 13-disk 7200RPM SATA RAID-DP (NetApp) is far faster than a single disk, dual diak or RAID-1, so why wouldn't you use SAN/NAS storage?

I seriously want to understand the VM concern as it pertains to OpenLDAP. I think more and more people are doing this very thing and will benefit from this discussion.

Our database is 86K+ DN's averaging about 40 attributes each. We've tuned the HDB cache to 768MB in a shared memory segment and the pertinent master slapd.conf file shows:

shm_key 100 cachesize 200000 idlcachesize 600000 dncachesize 400000 checkpoint 1024 15 . . . # main database . . . index objectClass eq index cn eq,sub index sn eq,sub index gn eq,sub index mail eq,sub index uid eq,sub index displayname sub,eq index memberUid eq,sub index uidNumber eq index gidNumber eq index sambaSID eq index sambaSIDlist eq index sambaDomainName eq index sambaPrimaryGroupSID eq index sambaGroupType eq index entryCSN eq index entryUUID eq index default sub,eq

Replicas have identical indexes and shared memory usage. Basically, just running database population tests with full checking turned on, I get the following results:

Ubuntu 10.04.1 on all with OpenLDAP 2.4.21/BDB 4.7.25 (all generate 200-10MB log files):

IBM x3550 2-quad 5450 16GB, RAID-1 15K 73GB 3.0Gb/s SAS, 86K DN's - 30 minutes IBM x3550 2-quad 5450 16GB, Single 15K 73GB 3.0Gb/s SAS, 86K DN's - 28 minutes IBM x3550 2-quad 5450 16GB, 2-single 15K 73GB (db and logs on separate disks) 3.0Gb/s SAS, 86K DN's - 28 minutes VMware guest 2-vCPU, 3GB memory, 100GB virtual disk on VMware NFS mount of 13-1TB 7200RPM SATA disks in NetApp 3140 - 4 minutes.

We can replicate to another VM in 9 minutes and two VMs simultaneously to the same 13-disk aggregate in 13 minutes. Aside from VM clock skew problems, I don't see the benefit of Bare Metal and I'm feeling pretty dumb at the moment.

Any insight from you, Quanah and/or Howard is humbly accepted and appreciated - I am here to learn. :-)

Thank you, Bill

-Dieter

-- Dieter Klünter | Systemberatung sip: 7770535@sipgate.de http://www.dpunkt.de/buecher/2104.html GPG Key ID:8EF7B6C6

This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.

On 23/08/10 14:13 -0400, William E Jojo wrote:

Hi Dieter,

Could you kindly explain what this means? I've been all over the inter-webs and I'm not finding anything concrete about OpenLDAP and VM's or Databases and VM's in general. The closest I came was about some database latency studies and some VMware propaganda.

We are about to launch a master and two replicas (utilizing delta-syncrepl) running in Ubuntu 10.04 on a VMware VSphere ESXi 4.0 cluster with four IBM x3650-M2's (2 Quad Nahalem and 64GB memory each) with virtual disks carved from NFS mounts to all of the VSphere servers in the cluster to facilitate HA and FT - by the VMware book.

I think NFS is a red flag here. To what extent to you use NFS in your setup? Do you store any berkeley files on an NFS mount?

If so, you should carefully research the mixing of the two. The Berkeley documentation discusses it:

http://tinyurl.com/28ehu3s

William E Jojo w.jojo@hvcc.edu writes: [...]

...

"Singh, Devender (GE Capital, consultant)" Devender.Singh2@ge.com writes:

...

Hi Dieter,  Please find the below details:

1. hardware related

- type of storage - Simply SATA had disk attached.    - raid level, if any- No RAID    - file system of disk(s)- ext3 on LVm    - type of network, 100MB, 1G, 10G

2. is this host running on a virtual machine or on bare metal.

- if virtual machine, -Yes, OS installed on VM    -- what type ---Don’t know

There is nothing suspicious, except for the virtual machine. Your really should carefully check layout and configuration of this VM, do not use virtual disks.

Hi Dieter,

Could you kindly explain what this means? I've been all over the inter-webs and I'm not finding anything concrete about OpenLDAP and VM's or Databases and VM's in general. The closest I came was about some database latency studies and some VMware propaganda.

My comments are based on my own experience. I am not against virutal machines in principle, but only against full virtualisation, that is virtualised disks and cpus, in a production environment. A para virtualisation might be acceptable, if the hypervisor provides sufficient resources and exclusive read/write access to a disk and client connections are moderate. But in all cases you have to expect performance loss.

We are about to launch a master and two replicas (utilizing delta-syncrepl) running in Ubuntu 10.04 on a VMware VSphere ESXi 4.0 cluster with four IBM x3650-M2's (2 Quad Nahalem and 64GB memory each) with virtual disks carved from NFS mounts to all of the VSphere servers in the cluster to facilitate HA and FT - by the VMware book.

(BTW, I took one of Howard's old posts to heart and we are following the Ubuntu playbook and we have purchased Canonical 24x7 support/maintenance. :-) )

We have had no problems with this environment in development and get better results than on bare metal with or without RAID. I know that it is recommended that the logs live on another "disk" from the database and RAID is frowned upon, but I have difficulty with a few points:

1. Separate, unprotected disks seems illogical. The last log and the

BDB files are necessary to start BDB in slapd, correct? So, if you lose either disk, you're in trouble. Backups are ok, but daily seems too long a time. Seeing as we process new user accounts every 15 minutes, this would not be ideal for us.

Yes, slapd requires the last transaction logs in order to recover a corrupted database, but the risks can be minimized by setting appropriate checkpoints. Writing new entries every 15 minutes wouldn't cause much trouble if you checkpoint every 4K and every 3 minutes.

2. RAID-1 I can understand having an issue on writes. But what about

LUNs in FC from a NetApp 3140? Virtual disks on NFS in VMware? Both of these are in the best practices of both VMware and NetApp documentation.

Just a scenario based on my own experience: a organisation wide addressbook with some 100,000 user entries, each user may have arbitrary private addressbooks underneath its entry node. Mail clients use this addressbook for address completion, mail clients are badly configured to use the root entry as search base. Can you imagine the number of parallel read operations? In addition to this read operations consecutive write operations of adding new entries, writing to index databases writing transaction logs. If you do not allow the transaction logs to reside on a different disk, you sure will experience db deadlocks in search opeerations.

3. 13-disk 7200RPM SATA RAID-DP (NetApp) is far faster than a single

disk, dual diak or RAID-1, so why wouldn't you use SAN/NAS storage?

I seriously want to understand the VM concern as it pertains to OpenLDAP. I think more and more people are doing this very thing and will benefit from this discussion.

The main concern is performance. If you only expect a few connections per second, performance is not a questions, but if you expect a few thousend connections per second, performance is an issue.

[...]

-Dieter