Dear Folks,
I'm upgrading a cluster of OpenLDAP servers from 2.3.43-25.el5 to 2.4.32 with BDB 4.8.30 on CentOS 5, x86_64, on HP BL460cG6 blades with two 4-core CPUs, and 12GB RAM. These are slaves, and have eleven trees on them. I have dumped and restored six of the LDAP databases in reasonable time, but the seventh is taking a long time. Here are the sizes of the slapcatted LDIF files:
320M 252M 80M 225K 246K 833K 2.6G 1.2G 24M 947M 522K
It's the 2.6G LDIF file that's taking the time to slapadd: -############ 63.34% eta 02h09m elapsed 03h43m30s spd 2.7 k/s
As you can see, it has slowed to a crawl.
# cat DB_CONFIG set_flags DB_LOG_AUTOREMOVE set_cachesize 0 286162472 0
# egrep 'tool-threads|cachesize' /etc/openldap/slapd.conf tool-threads 8 cachesize 100000 idlcachesize 100000 cachesize 100000 idlcachesize 100000 cachesize 100000 idlcachesize 100000 cachesize 100000 idlcachesize 100000 cachesize 100000 idlcachesize 100000 cachesize 100000 idlcachesize 100000 cachesize 100000 idlcachesize 100000 cachesize 100000 idlcachesize 100000 cachesize 100000 idlcachesize 100000 cachesize 100000 idlcachesize 100000 cachesize 100000 idlcachesize 100000
On another member of the cluster, not upgraded from 2.3.43 yet, we have, for this big tree: # sudo -u ldap slapd_db_stat -d id2entry.bdb Tue Aug 21 15:15:38 2012 Local time 53162 Btree magic number 9 Btree version number Little-endian Byte order Flags 2 Minimum keys per-page 16384 Underlying database page size 3 Number of levels in the tree 4611665 Number of unique keys in the tree 4611665 Number of data items in the tree 418 Number of tree internal pages 36096 Number of bytes free in tree internal pages (99% ff) 308745 Number of tree leaf pages 871M Number of bytes free in tree leaf pages (82% ff) 0 Number of tree duplicate pages 0 Number of bytes free in tree duplicate pages (0% ff) 0 Number of tree overflow pages 0 Number of bytes free in tree overflow pages (0% ff) 0 Number of empty pages 0 Number of pages on the free list # sudo -u ldap slapd_db_stat -d dn2id.bdb Tue Aug 21 15:34:30 2012 Local time 53162 Btree magic number 9 Btree version number Little-endian Byte order duplicates, sorted duplicates Flags 2 Minimum keys per-page 4096 Underlying database page size 4 Number of levels in the tree 9227811 Number of unique keys in the tree 9263210 Number of data items in the tree 2185 Number of tree internal pages 2781648 Number of bytes free in tree internal pages (68% ff) 219185 Number of tree leaf pages 287M Number of bytes free in tree leaf pages (68% ff) 37 Number of tree duplicate pages 40214 Number of bytes free in tree duplicate pages (73% ff) 0 Number of tree overflow pages 0 Number of bytes free in tree overflow pages (0% ff) 0 Number of empty pages 0 Number of pages on the free list
QUESTIONS: ==========
Any suggestions on how to optimise this a little more towards slapadd?
Would mdb possibly be faster than bdb?
--On Tuesday, August 21, 2012 4:03 PM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote:
Dear Folks,
I'm upgrading a cluster of OpenLDAP servers from 2.3.43-25.el5 to 2.4.32 with BDB 4.8.30 on CentOS 5, x86_64, on HP BL460cG6 blades with two 4-core CPUs, and 12GB RAM. These are slaves, and have eleven trees on them. I have dumped and restored six of the LDAP databases in reasonable time, but the seventh is taking a long time. Here are the sizes of the slapcatted LDIF files:
It's the 2.6G LDIF file that's taking the time to slapadd: -############ 63.34% eta 02h09m elapsed 03h43m30s spd 2.7 k/s
As you can see, it has slowed to a crawl.
# cat DB_CONFIG set_flags DB_LOG_AUTOREMOVE set_cachesize 0 286162472 0
# egrep 'tool-threads|cachesize' /etc/openldap/slapd.conf tool-threads 8
cachesize and idlcachesize have no bearing on slapadd. Only toolthreads does.
You do not mention if you are using slapadd -q or not.
Any suggestions on how to optimise this a little more towards slapadd?
If the LDIF file is 2.6GB, I fully assume the BDB database will be significantly larger than 2.6 GB. Probably closer to 6 or 8 GB, depending on your indexing. What you really need to look at from your 2.3 installation is the total size of that database (du -c -h *.bdb).
Right now, you only have a 272MB BDB cache set... which is definitely going to be too small. I would change DB_CONFIG For that database to be something like:
set_cachesize 6 0 0
I would strongly advise examining using a shared memory key for that DB as well.
Finally, look at using MDB, which has zero tuning bits required, and only requires a tool-threads of 2.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Dear Quanah,
Thank you for your generous help.
On 21/08/12 07:32 -0700, Quanah Gibson-Mount wrote:
--On Tuesday, August 21, 2012 4:03 PM +1000 Nick Urbanik nick.urbanik@optusnet.com.au wrote:
Dear Folks,
I'm upgrading a cluster of OpenLDAP servers from 2.3.43-25.el5 to 2.4.32 with BDB 4.8.30 on CentOS 5, x86_64, on HP BL460cG6 blades with two 4-core CPUs, and 12GB RAM. These are slaves, and have eleven trees on them. I have dumped and restored six of the LDAP databases in reasonable time, but the seventh is taking a long time. Here are the sizes of the slapcatted LDIF files:
It's the 2.6G LDIF file that's taking the time to slapadd: -############ 63.34% eta 02h09m elapsed 03h43m30s spd 2.7 k/s
As you can see, it has slowed to a crawl.
# cat DB_CONFIG set_flags DB_LOG_AUTOREMOVE set_cachesize 0 286162472 0
# egrep 'tool-threads|cachesize' /etc/openldap/slapd.conf tool-threads 8
cachesize and idlcachesize have no bearing on slapadd. Only toolthreads does.
Thank you.
You do not mention if you are using slapadd -q or not.
Yes, this is with the options -q -w.
Any suggestions on how to optimise this a little more towards slapadd?
If the LDIF file is 2.6GB, I fully assume the BDB database will be significantly larger than 2.6 GB. Probably closer to 6 or 8 GB, depending on your indexing. What you really need to look at from your 2.3 installation is the total size of that database (du -c -h *.bdb).
In the biggest LDAP directory on the (now finally converted) machine: 6.1G
Over all eleven LDAP directories on that same machine: 13G
Right now, you only have a 272MB BDB cache set... which is definitely going to be too small. I would change DB_CONFIG For that database to be something like:
set_cachesize 6 0 0
Okay; with 12G total RAM, and 6.9G of other directories, am I likely to overcommit memory?
I would strongly advise examining using a shared memory key for that DB as well.
Okay: I could create that with a command like ipcmk -M 16777216
But what size would be appropriate with 12G of RAM, eleven LDAP databases, and 13G of bdb files?
Finally, look at using MDB, which has zero tuning bits required, and only requires a tool-threads of 2.
So with eight cores (and 16 hyperthreaded siblings) I am better off specifying tool-threads 2?
Okay, my next one I'll convert to mdb and see how that behaves.
slapd-mdb(5) seems to indicate that mdb needs no shared memory configuration. Is that correct?
--On Wednesday, August 22, 2012 7:15 AM +1000 Nick Urbanik nicku@nicku.org wrote:
Dear Quanah,
Thank you for your generous help.
In the biggest LDAP directory on the (now finally converted) machine: 6.1G
So for optimal *slapadd*, this should be your DB_CONFIG setting. You can reduce it down to just the size of id2entry.bdb + dn2id.bdb + some room for growth (10-20% of that dn2id.bdb +id2entry.bdb) after the slapadd completes.
So if id2entry.bdb is 1 GB and dn2id is 1 GB, I would probably use a 2.5 GB cachesize after slapadd completes.
Over all eleven LDAP directories on that same machine: 13G
For running slapd, the DB_CONFIG sizes should match the above formula.
Right now, you only have a 272MB BDB cache set... which is definitely going to be too small. I would change DB_CONFIG For that database to be something like:
set_cachesize 6 0 0
Okay; with 12G total RAM, and 6.9G of other directories, am I likely to overcommit memory?
Not for the slapadd. See above.
I would strongly advise examining using a shared memory key for that DB as well.
Okay: I could create that with a command like ipcmk -M 16777216
*No*. Please see the shm_key parameter in the manual pages for slapd-bdb.
Finally, look at using MDB, which has zero tuning bits required, and only requires a tool-threads of 2.
So with eight cores (and 16 hyperthreaded siblings) I am better off specifying tool-threads 2?
No. tool-threads is only used by slapadd. Since no other databases are being accessed while you are performing the load, you would want all CPUs dedicated to this slapadd process.
Okay, my next one I'll convert to mdb and see how that behaves.
slapd-mdb(5) seems to indicate that mdb needs no shared memory configuration. Is that correct?
Correct. There is no tuning for MDB at all, other than:
a) Ensuring either noatime or relatime is set on the filesystem hosting the DB
b) Ensuring you have a maxsize value that is large enough. This is for the maximum size you ever want the DB to grow, not a performance tuning value. It needs to be large enough to contain your DB and have room for growth. It cannot be larger than the size of the disk it is located on.
c) toolthreads for slapadd with mdb should be set to 2. No more, no less.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
On 21/08/12 14:27 -0700, Quanah Gibson-Mount wrote:
--On Wednesday, August 22, 2012 7:15 AM +1000 Nick Urbanik nicku@nicku.org wrote:
I would strongly advise examining using a shared memory key for that
DB as well.
Okay: I could create that with a command like ipcmk -M 16777216
*No*. Please see the shm_key parameter in the manual pages for slapd-bdb.
So I can just choose any number for this, as long as it is different from the any shmid in the output of ipcs -m, and slapd will create the appropriate shared memory segment?
--On Wednesday, August 22, 2012 7:37 AM +1000 Nick Urbanik nicku@nicku.org wrote:
On 21/08/12 14:27 -0700, Quanah Gibson-Mount wrote:
--On Wednesday, August 22, 2012 7:15 AM +1000 Nick Urbanik nicku@nicku.org wrote:
I would strongly advise examining using a shared memory key for that
DB as well.
Okay: I could create that with a command like ipcmk -M 16777216
*No*. Please see the shm_key parameter in the manual pages for slapd-bdb.
So I can just choose any number for this, as long as it is different from the any shmid in the output of ipcs -m, and slapd will create the appropriate shared memory segment?
Right. And if you use shm segments for more than one DB, they need to be unique across them.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Dear Quanah,
Thank you again. You are most generous!
On 21/08/12 14:40 -0700, Quanah Gibson-Mount wrote:
--On Wednesday, August 22, 2012 7:37 AM +1000 Nick Urbanik nicku@nicku.org wrote:
On 21/08/12 14:27 -0700, Quanah Gibson-Mount wrote:
--On Wednesday, August 22, 2012 7:15 AM +1000 Nick Urbanik nicku@nicku.org wrote:
I would strongly advise examining using a shared memory key for that
DB as well.
Okay: I could create that with a command like ipcmk -M 16777216
*No*. Please see the shm_key parameter in the manual pages for slapd-bdb.
So I can just choose any number for this, as long as it is different from the any shmid in the output of ipcs -m, and slapd will create the appropriate shared memory segment?
Right. And if you use shm segments for more than one DB, they need to be unique across them.
openldap-technical@openldap.org
-
Nick Urbanik -
Nick Urbanik -
Quanah Gibson-Mount