Hi,
Maybe this is not the best way to ask but I would like to get some performance expectations or maybe suggestions how to improve performance. I do have relatively long experience with OpenLDAP as a precompiled package and with much less users so performance was not an issue for those installations.
Now I need to put few million users (now one million for test), custom tailored schema and search performance is crucial but also modify performance is big issue. Before MDB I have used HDB as backend.
What to do to improve write part of performance or performance in general (when adding data – from time to time – OpenLDAP stalls in a way)? Just around 5 add/mod operations during that time then it continue with much higher speed.
I have used many different sources to find out other peoples experience and I didn’t choose to write to list lightly but I really need some help/hints.
As a hardware I am using two ProLiant DL360p Gen8 servers with:
48GB RAM,
HP Smart Array P420i Controller,
2 x EG0600FBDSR 558 GB hard disk in RAID 1.
2 Ethernet ports are bonded to achieve redundancy and performance and connected to Cisco 3750 switch
OS is Ubuntu 12.04.2 LTS server with no unnecessary daemons installed. Disk is partitioned this way.
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 46G 1.4G 43G 4% /
udev 24G 4.0K 24G 1% /dev
tmpfs 9.5G 240K 9.5G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 24G 0 24G 0% /run/shm
/dev/sda3 232G 32G 189G 15% /var
/dev/sda4 230G 2.4G 216G 2% /opt
OpenLDAP is latest one – 2.4.35 compiled on Ubuntu server with addition of Berkley db 5.3.21 (I am not using it but …) and no special switches are used during configuration except the on which puts installation to /opt/openldap directory.
I am using also libhoard.so as a memory manager (latest one downloaded from HOARD web site).
OpenLDAP is configured as MultiMaster N Way and I am using MDB as backend database.
Config is in conf.d stile.
root@spr1:~# more /opt/openldap/etc/openldap/slapd.d/cn=config.ldif
# AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify.
# CRC32 47664f80
dn: cn=config
objectClass: olcGlobal
cn: config
olcArgsFile: /var/run/slapd/slapd.args
olcLogLevel: none
olcPidFile: /var/run/slapd/slapd.pid
structuralObjectClass: olcGlobal
creatorsName: cn=config
entryUUID: a8df00ce-80fb-1031-8652-fff9cf1d6a3e
createTimestamp: 20120822232009Z
olcServerID: 1 ldap://spr1.lab.os
olcServerID: 2 ldap://spr2.lab.os
olcIdleTimeout: 5
olcThreads: 6
entryCSN: 20130404141239.680528Z#000000#001#000000
modifiersName: cn=admin,cn=config
modifyTimestamp: 20130404141239Z
contextCSN: 20130404141404.368953Z#000000#001#000000
contextCSN: 20130329141804.133907Z#000000#002#000000
And database:
root@spr1:~# more /opt/openldap/etc/openldap/slapd.d/cn=config/olcDatabase={1}mdb.ldif
# AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify.
# CRC32 405c8aaf
dn: olcDatabase={1}mdb
objectClass: olcDatabaseConfig
objectClass: olcMdbConfig
olcDatabase: {1}mdb
olcDbDirectory: /var/lib/ldap
olcSuffix: dc=spr
olcAccess: {0}to attrs=userPassword,shadowLastChange by self write by anonymou
s auth by dn="cn=admin,dc=spr" write by * none
olcAccess: {1}to attrs=shadowLastChange by self write by * read
olcAccess: {2}to dn.base="" by * read
olcAccess: {3}to * by self write by dn="cn=admin,dc=spr" write by * read
olcLastMod: TRUE
olcRootDN: cn=admin,dc=spr
olcRootPW:: e1NTSEF9eDRVRTBiV2Z4YnFSNnZDVDdKRWEwSWRhWFRhMDN2M3I=
olcDbNoSync: TRUE
olcDbMaxSize: 107374182400
structuralObjectClass: olcMdbConfig
entryUUID: 804a8ede-2cc3-1032-9b7a-c7b2eb845e03
creatorsName: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
createTimestamp: 20130329135129Z
olcDbIndex: objectClass eq
olcDbIndex: uid eq
olcDbIndex: MSISDN eq
olcDbIndex: IMSI eq
olcDbIndex: pfUsername eq
olcDbIndex: entryCSN eq
olcDbIndex: entryUUID eq
olcDbIndex: contextCSN eq
olcMirrorMode: TRUE
olcSyncrepl: {0}rid=003 provider=ldap://spr1.lab.os binddn="cn=admin,dc=spr" b
indmethod=simple credentials=siemens searchbase="dc=spr" type=refreshOnly int
erval=00:00:00:10 retry="5 5 300 5" timeout=1
olcSyncrepl: {1}rid=004 provider=ldap://spr2.lab.os binddn="cn=admin,dc=spr" b
indmethod=simple credentials=siemens searchbase="dc=spr" type=refreshOnly int
erval=00:00:00:10 retry="5 5 300 5" timeout=1
olcDbCheckpoint: 8192 15
entryCSN: 20130404141404.368953Z#000000#001#000000
modifiersName: cn=admin,cn=config
modifyTimestamp: 20130404141404Z
This is output from iotop during ldapmodify test (1 mil user, two clients modifying each user in sequence , 1 DN / 1 attribute per user):
Total DISK READ: 0.00 B/s | Total DISK WRITE: 15.30 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
4403 be/4 root 0.00 B/s 0.00 B/s 0.00 % 98.88 % [flush-8:0]
4838 be/4 openldap 0.00 B/s 3.41 M/s 0.00 % 21.68 % slapd -h ldap:/// -F /opt/openldap/etc/~ldap/slapd.d -g openldap -u openldap -4
4837 be/4 openldap 0.00 B/s 3.19 M/s 0.00 % 19.13 % slapd -h ldap:/// -F /opt/openldap/etc/~ldap/slapd.d -g openldap -u openldap -4
4836 be/4 openldap 0.00 B/s 2.83 M/s 0.00 % 17.48 % slapd -h ldap:/// -F /opt/openldap/etc/~ldap/slapd.d -g openldap -u openldap -4
10889 be/4 openldap 0.00 B/s 2.94 M/s 0.00 % 16.30 % slapd -h ldap:/// -F /opt/openldap/etc/~ldap/slapd.d -g openldap -u openldap -4
10962 be/4 openldap 0.00 B/s 2.92 M/s 0.00 % 15.25 % slapd -h ldap:/// -F /opt/openldap/etc/~ldap/slapd.d -g openldap -u openldap -4
1816 be/3 root 0.00 B/s 0.00 B/s 0.00 % 3.23 % [jbd2/sda3-8]
And top:
top - 11:56:27 up 4 days, 1:43, 1 user, load average: 2.71, 2.88, 3.19
Tasks: 224 total, 1 running, 223 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.6%us, 0.1%sy, 0.0%ni, 91.3%id, 6.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 49424228k total, 35362892k used, 14061336k free, 127844k buffers
Swap: 46874876k total, 0k used, 46874876k free, 32631568k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4833 openldap 20 0 101g 30g 29g S 91 64.1 196:30.20 slapd
4403 root 20 0 0 0 0 D 2 0.0 2:41.89 flush-8:0
1 root 20 0 24472 2444 1360 S 0 0.0 0:04.36 init
2 root 20 0 0 0 0 S 0 0.0 0:00.03 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:00.37 ksoftirqd/0
Saša-Stjepan Bakša wrote:
Hi,
Maybe this is not the best way to ask but I would like to get some performance expectations or maybe suggestions how to improve performance. I do have relatively long experience with OpenLDAP as a precompiled package and with much less users so performance was not an issue for those installations.
Now I need to put few million users (now one million for test), custom tailored schema and search performance is crucial but also modify performance is big issue. Before MDB I have used HDB as backend.
What to do to improve write part of performance or performance in general (when adding data – from time to time – OpenLDAP stalls in a way)? Just around 5 add/mod operations during that time then it continue with much higher speed.
I have used many different sources to find out other peoples experience and I didn’t choose to write to list lightly but I really need some help/hints.
There's nothing particular to LMDB to tune. But if you're seeing pauses due to disk I/O, as your iotop output seems to indicate, you might want to look into using a different I/O scheduler.
Here's an old discussion on that topic:
http://stackoverflow.com/questions/1009577/selecting-a-linux-i-o-scheduler
As a hardware I am using two ProLiant DL360p Gen8 servers with:
48GB RAM,
HP Smart Array P420i Controller,
2 x EG0600FBDSR 558 GB hard disk in RAID 1.
2 Ethernet ports are bonded to achieve redundancy and performance and connected to Cisco 3750 switch
OS is Ubuntu 12.04.2 LTS server with no unnecessary daemons installed. Disk is partitioned this way.
FilesystemSizeUsed Avail Use% Mounted on
/dev/sda146G1.4G43G4% /
udev24G4.0K24G1% /dev
tmpfs9.5G240K9.5G1% /run
none5.0M05.0M0% /run/lock
none24G024G0% /run/shm
/dev/sda3232G32G189G15% /var
/dev/sda4230G2.4G216G2% /opt
OpenLDAP is latest one – 2.4.35 compiled on Ubuntu server with addition of Berkley db 5.3.21 (I am not using it but …) and no special switches are used during configuration except the on which puts installation to /opt/openldap directory.
I am using also libhoard.so as a memory manager (latest one downloaded from HOARD web site).
OpenLDAP is configured as MultiMaster N Way and I am using MDB as backend database.
Config is in conf.d stile.
root@spr1:~# more /opt/openldap/etc/openldap/slapd.d/cn=config.ldif
Never muck with the files inside slapd.d. Use "slapcat -n0" to look at the configuration. The slapd.d is a slapd database and its internal format is subject to change without notice. The only guarantees we make are that the slap* tools (and ldap* tools thru slapd) will work on them correctly.
In the near future we may migrate the slapd.d format to a pure binary format. (E.g., using LMDB to leverage its crash resistance.) Anyone foolish enough to have operated on these files directly, despite all our warnings, will be unable to continue to do so. Everyone who uses only the supported commands will just continue without any problem.
# AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify.
And top:
top - 11:56:27 up 4 days,1:43,1 user,load average: 2.71, 2.88, 3.19
Tasks: 224 total,1 running, 223 sleeping,0 stopped,0 zombie
Cpu(s):2.6%us,0.1%sy,0.0%ni, 91.3%id,6.0%wa,0.0%hi,0.0%si,0.0%st
Mem:49424228k total, 35362892k used, 14061336k free,127844k buffers
Swap: 46874876k total,0k used, 46874876k free, 32631568k cached
PID USERPRNIVIRTRESSHR S %CPU %MEMTIME+COMMAND
4833 openldap200101g30g29g S91 64.1 196:30.20 slapd
4403 root200000 D20.02:41.89 flush-8:0
1 root200 24472 2444 1360 S00.00:04.36 init
2 root200000 S00.00:00.03 kthreadd
3 root2000 00 S00.00:00.37 ksoftirqd/0
--On Monday, April 08, 2013 4:49 AM -0700 Howard Chu hyc@symas.com wrote:
Saša-Stjepan Bakša wrote:
Hi,
Maybe this is not the best way to ask but I would like to get some performance expectations or maybe suggestions how to improve performance. I do have relatively long experience with OpenLDAP as a precompiled package and with much less users so performance was not an issue for those installations.
Now I need to put few million users (now one million for test), custom tailored schema and search performance is crucial but also modify performance is big issue. Before MDB I have used HDB as backend.
What to do to improve write part of performance or performance in general (when adding data – from time to time – OpenLDAP stalls in a way)? Just around 5 add/mod operations during that time then it continue with much higher speed.
I have used many different sources to find out other peoples experience and I didn't choose to write to list lightly but I really need some help/hints.
There's nothing particular to LMDB to tune. But if you're seeing pauses due to disk I/O, as your iotop output seems to indicate, you might want to look into using a different I/O scheduler.
Not quite true. On Linux, I suggest setting the olcDbFlags as follows:
olcDbEnvFlags: writemap olcDbEnvFlags: nometasync
With those flags set, writes are 65x faster for me with back-mdb than they are with back-hdb/back-bdb.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
openldap-technical@openldap.org
-
Howard Chu -
Quanah Gibson-Mount -
Saša-Stjepan Bakša