Ok, I have been running 2.4.32 for some time with no issues. Yesterday, 2 different servers (both part of a 4-way MMR) produced an "index add failure" and an "index delete failure". I went back over the bdb DB_CONFIG Settings (listed below) and everything looks nominal to me. Would it just make more sense to switch from bdb to mdb instead of troubleshooting these "random" errors too much? I also noticed that the number of "deadlocks" corresponds to the number of errors that were produced. Is there correlation there?
Thanks!
578 Last allocated locker ID 0x7fffffff Current maximum unused locker ID 9 Number of lock modes 3000 Maximum number of locks possible 1500 Maximum number of lockers possible 1500 Maximum number of lock objects possible 1 Number of lock object partitions 15 Number of current locks 1029 Maximum number of locks at any one time 17 Maximum number of locks in any one bucket 0 Maximum number of locks stolen by for an empty partition 0 Maximum number of locks stolen for any one partition 123 Number of current lockers 224 Maximum number of lockers at any one time 15 Number of current lock objects 526 Maximum number of lock objects at any one time 5 Maximum number of lock objects in any one bucket 0 Maximum number of objects stolen by for an empty partition 0 Maximum number of objects stolen for any one partition 3581M Total number of locks requested (3581768929) 3581M Total number of locks released (3581768869) 0 Total number of locks upgraded 77 Total number of locks downgraded 7041 Lock requests not available due to conflicts, for which we waited 43 Lock requests not available due to conflicts, for which we did not wait 2 Number of deadlocks 0 Lock timeout value 0 Number of locks that have timed out 0 Transaction timeout value 0 Number of transactions that have timed out 1MB 392KB The size of the lock region 0 The number of partition locks that required waiting (0%) 0 The maximum number of times any partition lock was waited for (0%) 0 The number of object queue operations that required waiting (0%) 577 The number of locker allocations that required waiting (0%) 32148 The number of region locks that required waiting (0%) 5 Maximum hash bucket length
On Wed, Aug 29, 2012 at 12:04 PM, Quanah Gibson-Mount quanah@zimbra.comwrote:
--On Wednesday, August 29, 2012 11:32 AM -0400 Kyle Smith < alacer.cogitatus@gmail.com> wrote:
Quanah, Thanks for the info, I have confirmed I'm hitting the lock maxes
of 1000. And I will be upgrading to 2.4.32. I was wondering, what steps should be done to have the changes in DB_CONFIG take effect?
stop slapd make changes to DB_CONFIG db_recover start slapd
Will this also auto remove the log.* files? ( I plan on setting this: "set_flags DB_LOG_AUTOREMOVE" in DB_CONFIG)
If you have checkpointing set in slapd.conf/cn=config, it should, yes.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc.
Zimbra :: the leader in open source messaging and collaboration
--On Thursday, November 01, 2012 12:36 PM -0400 Kyle Smith alacer.cogitatus@gmail.com wrote:
Ok, I have been running 2.4.32 for some time with no issues. Yesterday, 2 different servers (both part of a 4-way MMR) produced an "index add failure" and an "index delete failure". I went back over the bdb DB_CONFIG Settings (listed below) and everything looks nominal to me. Would it just make more sense to switch from bdb to mdb instead of troubleshooting these "random" errors too much? I also noticed that the number of "deadlocks" corresponds to the number of errors that were produced. Is there correlation there?
I would suggest back-mdb in OpenLDAP 2.4.33 over back-hdb, yes. It will remove these deadlock issues entirely. When you configure the back-mdb backend in cn=config, be sure to have:
olcDbEnvFlags: writemap olcDbEnvFlags: nometasync
set as well. This makes back-mdb writes significantly faster than writes to back-bdb/hdb. (Some 2x faster in my testing so far).
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Kyle Smith wrote:
Ok, I have been running 2.4.32 for some time with no issues. Yesterday, 2 different servers (both part of a 4-way MMR) produced an "index add failure" and an "index delete failure". I went back over the bdb DB_CONFIG Settings (listed below) and everything looks nominal to me. Would it just make more sense to switch from bdb to mdb instead of troubleshooting these "random" errors too much? I also noticed that the number of "deadlocks" corresponds to the number of errors that were produced. Is there correlation there?
Probably, but that's not an indication of any actual failure. Deadlocks are normal occurrences in BerkeleyDB and the backends automatically retry when they occur. You can basically ignore any error that accompanies a deadlock.
And yes, if you switch to MDB, all of these issues go away.
Given that reads in MDB are 5-20x faster than BDB, and writes are 2-5x faster, and MDB uses 1/4 as much RAM as BDB, there's hardly any reason to use BDB any more*. No tuning, no maintenance. MDB just works, quickly and efficiently.
*If you're still using a 32 bit machine, you may be better off using BDB, especially if you have databases 1GB or larger. But seriously, why are you still using a 32 bit machine?
Thanks!
578 Last allocated locker ID 0x7fffffff Current maximum unused locker ID 9 Number of lock modes 3000 Maximum number of locks possible 1500 Maximum number of lockers possible 1500 Maximum number of lock objects possible 1 Number of lock object partitions 15 Number of current locks 1029 Maximum number of locks at any one time 17 Maximum number of locks in any one bucket 0 Maximum number of locks stolen by for an empty partition 0 Maximum number of locks stolen for any one partition 123 Number of current lockers 224 Maximum number of lockers at any one time 15 Number of current lock objects 526 Maximum number of lock objects at any one time 5 Maximum number of lock objects in any one bucket 0 Maximum number of objects stolen by for an empty partition 0 Maximum number of objects stolen for any one partition 3581M Total number of locks requested (3581768929) 3581M Total number of locks released (3581768869) 0 Total number of locks upgraded 77 Total number of locks downgraded 7041 Lock requests not available due to conflicts, for which we waited 43 Lock requests not available due to conflicts, for which we did not wait 2 Number of deadlocks 0 Lock timeout value 0 Number of locks that have timed out 0 Transaction timeout value 0 Number of transactions that have timed out 1MB 392KB The size of the lock region 0 The number of partition locks that required waiting (0%) 0 The maximum number of times any partition lock was waited for (0%) 0 The number of object queue operations that required waiting (0%) 577 The number of locker allocations that required waiting (0%) 32148 The number of region locks that required waiting (0%) 5 Maximum hash bucket length
On Wed, Aug 29, 2012 at 12:04 PM, Quanah Gibson-Mount <quanah@zimbra.com mailto:quanah@zimbra.com> wrote:
--On Wednesday, August 29, 2012 11:32 AM -0400 Kyle Smith <alacer.cogitatus@gmail.com <mailto:alacer.cogitatus@gmail.com>> wrote: Quanah, Thanks for the info, I have confirmed I'm hitting the lock maxes of 1000. And I will be upgrading to 2.4.32. I was wondering, what steps should be done to have the changes in DB_CONFIG take effect? stop slapd make changes to DB_CONFIG db_recover start slapd Will this also auto remove the log.* files? ( I plan on setting this: "set_flags DB_LOG_AUTOREMOVE" in DB_CONFIG) If you have checkpointing set in slapd.conf/cn=config, it should, yes. --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
One last question. For now anyways. I am testing the procedure to convert from hdb to mdb using slapcat/add. Should a slapindex be done as well? When I tried it, I got this error. Thanks in advance!
50a17983 => mdb_idl_insert_keys: c_put id failed: MDB_MAP_FULL: Environment mapsize limit reached (-30792) 50a17983 => mdb_tool_entry_reindex: txn_aborted! err=80
--On Monday, November 12, 2012 5:38 PM -0500 Kyle Smith alacer.cogitatus@gmail.com wrote:
One last question. For now anyways. I am testing the procedure to convert from hdb to mdb using slapcat/add. Should a slapindex be done as well? When I tried it, I got this error. Thanks in advance!
50a17983 => mdb_idl_insert_keys: c_put id failed: MDB_MAP_FULL: Environment mapsize limit reached (-30792) 50a17983 => mdb_tool_entry_reindex: txn_aborted! err=80
a) Slapadd does a slapindex. Why would you slapadd and then slapindex?
b) What version of OpenLDAP? slapindex still shouldn't cause a failure.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Hi Kyle,
--On Monday, November 12, 2012 5:38 PM -0500 Kyle Smith alacer.cogitatus@gmail.com wrote:
One last question. For now anyways. I am testing the procedure to convert from hdb to mdb using slapcat/add. Should a slapindex be done as well? When I tried it, I got this error. Thanks in advance!
50a17983 => mdb_idl_insert_keys: c_put id failed: MDB_MAP_FULL: Environment mapsize limit reached (-30792) 50a17983 => mdb_tool_entry_reindex: txn_aborted! err=80
MDB_MAP_FULL means your database ran out of room. You need to specify a larger maxsize for the MDB database. Personally, I set mine to 80GB (larger than any database I've worked with) so I don't really have to worry about it. ;)
I would also suggest applying http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=patch;h=bb36bdcd1c22d1fbc6575452ef5c9112715ab083
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
"quanah" == Quanah Gibson-Mount quanah@zimbra.com writes:
quanah> MDB_MAP_FULL means your database ran out of room. You need to quanah> specify a larger maxsize for the MDB database. Personally, I quanah> set mine to 80GB (larger than any database I've worked with) quanah> so I don't really have to worry about it. ;)
quanah> I would also suggest applying quanah> http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=patch;h=bb36bdcd1c22d1fbc6575452ef5c9112715ab083
Is there a measure of how "full" a MDB database is? Maybe a "Monitor" value of some sort?
--On Tuesday, November 13, 2012 6:38 PM -0500 "Allan E. Johannesen" aej@WPI.EDU wrote:
Is there a measure of how "full" a MDB database is? Maybe a "Monitor" value of some sort?
du <file.mdb>
I.e., the filesize of mdb grows as it gets data, at least on Linux. This is why I set my maxsize to 80GB. This allows it to grow up to 80GB before running out of room. Even with BDB, which takes substantially more space, I've never had an 80GB DB.
One of these days, I plan on writing a monitoring script that compares the max size of the DB with the actual size on disk.
I found that OSX acts different -- It actually allocates the entire size of the database on disk, regardless of how much is used. That may be common to all the BSDs.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
--On Tuesday, November 13, 2012 6:38 PM -0500 "Allan E. Johannesen" aej@WPI.EDU wrote:
Is there a measure of how "full" a MDB database is? Maybe a "Monitor" value of some sort?
du <file.mdb>
I.e., the filesize of mdb grows as it gets data, at least on Linux. This is why I set my maxsize to 80GB. This allows it to grow up to 80GB before running out of room. Even with BDB, which takes substantially more space, I've never had an 80GB DB.
One of these days, I plan on writing a monitoring script that compares the max size of the DB with the actual size on disk.
I found that OSX acts different -- It actually allocates the entire size of the database on disk, regardless of how much is used. That may be common to all the BSDs.
Sounds like whatever filesystem is default on BSDish systems doesn't support sparse files.
Anyway, you can use mdb_stat to see what's in the MDB environment.
--On Tuesday, November 13, 2012 4:25 PM -0800 Howard Chu hyc@symas.com wrote:
I found that OSX acts different -- It actually allocates the entire size of the database on disk, regardless of how much is used. That may be common to all the BSDs.
Sounds like whatever filesystem is default on BSDish systems doesn't support sparse files.
Anyway, you can use mdb_stat to see what's in the MDB environment.
Oh, one additional note about the BSDs... I didn't see that behavior until I enabled writemap. Disabling writemap reverted it to the same behavior as linux, but then of course the advantage of writemap is lost. :P
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
--On Tuesday, November 13, 2012 4:25 PM -0800 Howard Chu hyc@symas.com wrote:
I found that OSX acts different -- It actually allocates the entire size of the database on disk, regardless of how much is used. That may be common to all the BSDs.
Sounds like whatever filesystem is default on BSDish systems doesn't support sparse files.
Anyway, you can use mdb_stat to see what's in the MDB environment.
Oh, one additional note about the BSDs... I didn't see that behavior until I enabled writemap. Disabling writemap reverted it to the same behavior as linux, but then of course the advantage of writemap is lost. :P
That's to be expected. When writemap is enabled, we ask the OS to preallocate the entire map space. Otherwise, writing to a new page would get a SIGBUS.
--On Tuesday, November 13, 2012 4:54 PM -0800 Howard Chu hyc@symas.com wrote:
Quanah Gibson-Mount wrote:
--On Tuesday, November 13, 2012 4:25 PM -0800 Howard Chu hyc@symas.com wrote:
I found that OSX acts different -- It actually allocates the entire size of the database on disk, regardless of how much is used. That may be common to all the BSDs.
Sounds like whatever filesystem is default on BSDish systems doesn't support sparse files.
Anyway, you can use mdb_stat to see what's in the MDB environment.
Oh, one additional note about the BSDs... I didn't see that behavior until I enabled writemap. Disabling writemap reverted it to the same behavior as linux, but then of course the advantage of writemap is lost. :P
That's to be expected. When writemap is enabled, we ask the OS to preallocate the entire map space. Otherwise, writing to a new page would get a SIGBUS.
That's not quite what I see.
On Linux, you get:
[zimbra@ldap01-zcs db]$ ls -l total 250588 -rw------- 1 zimbra zimbra 85899345920 Nov 13 17:23 data.mdb -rw------- 1 zimbra zimbra 8192 Nov 13 17:23 lock.mdb [zimbra@ldap01-zcs db]$ du -c -h data.mdb 245M data.mdb 245M total
But on OSX, the du -c -h shows 80GB as well.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
"quanah" == Quanah Gibson-Mount quanah@zimbra.com writes:
quanah> I found that OSX acts different -- It actually allocates the quanah> entire size of the database on disk, regardless of how much is quanah> used. That may be common to all the BSDs.
Redhat, too. The DB is the maxsize size. That's why I asked.
Thanks anyway.
--On Tuesday, November 13, 2012 8:10 PM -0500 "Allan E. Johannesen" aej@WPI.EDU wrote:
"quanah" == Quanah Gibson-Mount quanah@zimbra.com writes:
quanah> I found that OSX acts different -- It actually allocates the quanah> entire size of the database on disk, regardless of how much is quanah> used. That may be common to all the BSDs.
Redhat, too. The DB is the maxsize size. That's why I asked.
Thanks anyway.
Not sure what you are doing then. This is not the behavior I see on RedHat:
[zimbra@ldap01-zcs db]$ ls -l total 250588 -rw------- 1 zimbra zimbra 85899345920 Nov 13 17:23 data.mdb -rw------- 1 zimbra zimbra 8192 Nov 13 17:23 lock.mdb [zimbra@ldap01-zcs db]$ du -c -h data.mdb 245M data.mdb 245M total
[zimbra@ldap01-zcs db]$ cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.2 (Santiago)
I.e., the reported file size vs the actual file size differ. So my DB is 245MB in size, not 80GB.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
"quanah" == Quanah Gibson-Mount quanah@zimbra.com writes:
quanah> --On Tuesday, November 13, 2012 8:10 PM -0500 "Allan quanah> E. Johannesen" aej@WPI.EDU wrote:
> "quanah" == Quanah Gibson-Mount quanah@zimbra.com writes:
quanah> I found that OSX acts different -- It actually allocates the quanah> entire size of the database on disk, regardless of how much is quanah> used. That may be common to all the BSDs.
Redhat, too. The DB is the maxsize size. That's why I asked.
Thanks anyway.
quanah> Not sure what you are doing then. This is not the behavior I quanah> see on RedHat:
quanah> du -c -h data.mdb 245M data.mdb 245M total
Yes, I see now. Yes, that's right. I see now where I stand with the allocation. Thanks.
Allan E. Johannesen wrote:
"quanah" == Quanah Gibson-Mount quanah@zimbra.com writes:
quanah> MDB_MAP_FULL means your database ran out of room. You need to quanah> specify a larger maxsize for the MDB database. Personally, I quanah> set mine to 80GB (larger than any database I've worked with) quanah> so I don't really have to worry about it. ;)
quanah> I would also suggest applying quanah> http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=patch;h=bb36bdcd1c22d1fbc6575452ef5c9112715ab083
Is there a measure of how "full" a MDB database is? Maybe a "Monitor" value of some sort?
There's nothing in cn=monitor at the moment, but that sounds like a good idea. You should submit an enhancement request to the ITS.
openldap-technical@openldap.org
-
Allan E. Johannesen -
Howard Chu -
Kyle Smith -
Quanah Gibson-Mount