Hi,

Maybe this is not the best way to ask but I would like to get some performance expectations or maybe suggestions how to improve performance. I do have relatively long experience with OpenLDAP as a precompiled package and with much less users so performance was not an issue for those installations.

Now I need to put few million users (now one million for test), custom tailored schema and search performance is crucial but also modify performance is big issue. Before MDB I have used HDB as backend.

What to do to improve write part of performance or performance in general (when adding data – from time to time – OpenLDAP stalls in a way)? Just around 5 add/mod operations during that time then it continue with much higher speed.

I have used many different sources to find out other peoples experience and I didn’t choose to write to list lightly but I really need some help/hints.

As a hardware I am using two ProLiant DL360p Gen8 servers with:

48GB RAM,

HP Smart Array P420i Controller,

2 x EG0600FBDSR 558 GB hard disk in RAID 1.

2 Ethernet ports are bonded to achieve redundancy and performance and connected to Cisco 3750 switch

OS is Ubuntu 12.04.2 LTS server with no unnecessary daemons installed. Disk is partitioned this way.

Filesystem Size Used Avail Use% Mounted on

/dev/sda1 46G 1.4G 43G 4% /

udev 24G 4.0K 24G 1% /dev

tmpfs 9.5G 240K 9.5G 1% /run

none 5.0M 0 5.0M 0% /run/lock

none 24G 0 24G 0% /run/shm

/dev/sda3 232G 32G 189G 15% /var

/dev/sda4 230G 2.4G 216G 2% /opt

OpenLDAP is latest one – 2.4.35 compiled on Ubuntu server with addition of Berkley db 5.3.21 (I am not using it but …) and no special switches are used during configuration except the on which puts installation to /opt/openldap directory.

I am using also libhoard.so as a memory manager (latest one downloaded from HOARD web site).

OpenLDAP is configured as MultiMaster N Way and I am using MDB as backend database.

Config is in conf.d stile.

root@spr1:~# more /opt/openldap/etc/openldap/slapd.d/cn=config.ldif

# AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify.

# CRC32 47664f80

dn: cn=config

objectClass: olcGlobal

cn: config

olcArgsFile: /var/run/slapd/slapd.args

olcLogLevel: none

olcPidFile: /var/run/slapd/slapd.pid

structuralObjectClass: olcGlobal

creatorsName: cn=config

entryUUID: a8df00ce-80fb-1031-8652-fff9cf1d6a3e

createTimestamp: 20120822232009Z

olcServerID: 1 ldap://spr1.lab.os

olcServerID: 2 ldap://spr2.lab.os

olcIdleTimeout: 5

olcThreads: 6

entryCSN: 20130404141239.680528Z#000000#001#000000

modifiersName: cn=admin,cn=config

modifyTimestamp: 20130404141239Z

contextCSN: 20130404141404.368953Z#000000#001#000000

contextCSN: 20130329141804.133907Z#000000#002#000000

And database:

root@spr1:~# more /opt/openldap/etc/openldap/slapd.d/cn=config/olcDatabase={1}mdb.ldif

# AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify.

# CRC32 405c8aaf

dn: olcDatabase={1}mdb

objectClass: olcDatabaseConfig

objectClass: olcMdbConfig

olcDatabase: {1}mdb

olcDbDirectory: /var/lib/ldap

olcSuffix: dc=spr

olcAccess: {0}to attrs=userPassword,shadowLastChange by self write by anonymou

s auth by dn="cn=admin,dc=spr" write by * none

olcAccess: {1}to attrs=shadowLastChange by self write by * read

olcAccess: {2}to dn.base="" by * read

olcAccess: {3}to * by self write by dn="cn=admin,dc=spr" write by * read

olcLastMod: TRUE

olcRootDN: cn=admin,dc=spr

olcRootPW:: e1NTSEF9eDRVRTBiV2Z4YnFSNnZDVDdKRWEwSWRhWFRhMDN2M3I=

olcDbNoSync: TRUE

olcDbMaxSize: 107374182400

structuralObjectClass: olcMdbConfig

entryUUID: 804a8ede-2cc3-1032-9b7a-c7b2eb845e03

creatorsName: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth

createTimestamp: 20130329135129Z

olcDbIndex: objectClass eq

olcDbIndex: uid eq

olcDbIndex: MSISDN eq

olcDbIndex: IMSI eq

olcDbIndex: pfUsername eq

olcDbIndex: entryCSN eq

olcDbIndex: entryUUID eq

olcDbIndex: contextCSN eq

olcMirrorMode: TRUE

olcSyncrepl: {0}rid=003 provider=ldap://spr1.lab.os binddn="cn=admin,dc=spr" b

indmethod=simple credentials=siemens searchbase="dc=spr" type=refreshOnly int

erval=00:00:00:10 retry="5 5 300 5" timeout=1

olcSyncrepl: {1}rid=004 provider=ldap://spr2.lab.os binddn="cn=admin,dc=spr" b

indmethod=simple credentials=siemens searchbase="dc=spr" type=refreshOnly int

erval=00:00:00:10 retry="5 5 300 5" timeout=1

olcDbCheckpoint: 8192 15

entryCSN: 20130404141404.368953Z#000000#001#000000

modifiersName: cn=admin,cn=config

modifyTimestamp: 20130404141404Z

This is output from iotop during ldapmodify test (1 mil user, two clients modifying each user in sequence , 1 DN / 1 attribute per user):

Total DISK READ: 0.00 B/s | Total DISK WRITE: 15.30 M/s

TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND

4403 be/4 root 0.00 B/s 0.00 B/s 0.00 % 98.88 % [flush-8:0]

4838 be/4 openldap 0.00 B/s 3.41 M/s 0.00 % 21.68 % slapd -h ldap:/// -F /opt/openldap/etc/~ldap/slapd.d -g openldap -u openldap -4

4837 be/4 openldap 0.00 B/s 3.19 M/s 0.00 % 19.13 % slapd -h ldap:/// -F /opt/openldap/etc/~ldap/slapd.d -g openldap -u openldap -4

4836 be/4 openldap 0.00 B/s 2.83 M/s 0.00 % 17.48 % slapd -h ldap:/// -F /opt/openldap/etc/~ldap/slapd.d -g openldap -u openldap -4

10889 be/4 openldap 0.00 B/s 2.94 M/s 0.00 % 16.30 % slapd -h ldap:/// -F /opt/openldap/etc/~ldap/slapd.d -g openldap -u openldap -4

10962 be/4 openldap 0.00 B/s 2.92 M/s 0.00 % 15.25 % slapd -h ldap:/// -F /opt/openldap/etc/~ldap/slapd.d -g openldap -u openldap -4

1816 be/3 root 0.00 B/s 0.00 B/s 0.00 % 3.23 % [jbd2/sda3-8]

And top:

top - 11:56:27 up 4 days, 1:43, 1 user, load average: 2.71, 2.88, 3.19

Tasks: 224 total, 1 running, 223 sleeping, 0 stopped, 0 zombie

Cpu(s): 2.6%us, 0.1%sy, 0.0%ni, 91.3%id, 6.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 49424228k total, 35362892k used, 14061336k free, 127844k buffers

Swap: 46874876k total, 0k used, 46874876k free, 32631568k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

4833 openldap 20 0 101g 30g 29g S 91 64.1 196:30.20 slapd

4403 root 20 0 0 0 0 D 2 0.0 2:41.89 flush-8:0

1 root 20 0 24472 2444 1360 S 0 0.0 0:04.36 init

2 root 20 0 0 0 0 S 0 0.0 0:00.03 kthreadd

3 root 20 0 0 0 0 S 0 0.0 0:00.37 ksoftirqd/0