Attached to this message you'll find a quick and dirty bash script that should determine the cachesize for an in-production bdb/hdb backend by taking dn2id and id2entry as well as all existing indexes into account.
Internally the script calls the db_stat tool in combination with the formulas found in the 2.4 admin guide under: http://www.openldap.org/doc/admin24/tuning.html#Caching
Based on our local production database files (run by slapd 2.4.21, bdb 4.5) which contain round about 460.000 (small) entries...
# du -h dn2id.bdb id2entry.bdb 48M dn2id.bdb 755M id2entry.bdb
# du -h *.bdb | grep -v dn2id | grep -v id2entry 12M cn.bdb 20M entryCSN.bdb 14M entryUUID.bdb 1.1M loginDisabled.bdb 8.6M mail.bdb 5.4M objectClass.bdb
...the attached script produces the following results:
DB Cachesize Results (dn2id.bdb and id2entry.bdb): Overall DB Cachesize: 1871872 bytes
Index Cachesize Results: Overall Index Cachesize: 245760 bytes (50% Index HitRatio) Overall Index Cachesize: 491520 bytes
Resulting Overall Cachesize (DB and Indexes): Overall Cache Size: 2117632 bytes (50% Index HitRatio) Overall Cache Size: 2363392 bytes
Including +15% bytes for growth: Overall Cache Size: 2435276.80 bytes (50% Index HitRatio) Overall Cache Size: 2717900.80 bytes
Do the above results (2.4 - 2.7 MBytes?!) sound reasonable in comparison to the above bdb file sizes?
As I'm currently unsure whether the result is correct or not I would appreciate your feedback for example whether the script's result makes sense within your environment or not.
Thanks a lot!
--On Monday, July 12, 2010 5:01 PM +0200 openldap-ml@stresst.net wrote:
Attached to this message you'll find a quick and dirty bash script that should determine the cachesize for an in-production bdb/hdb backend by taking dn2id and id2entry as well as all existing indexes into account.
Internally the script calls the db_stat tool in combination with the formulas found in the 2.4 admin guide under: http://www.openldap.org/doc/admin24/tuning.html#Caching
Are you trying to determine the size of the cache in use, or what the size of the cache should be? If it is the latter, you just use the du -c -h output of *.bdb. If it is the former, it sounds like your script isn't calculating the right data.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On 07/12/2010 07:40 PM, Quanah Gibson-Mount wrote:
--On Monday, July 12, 2010 5:01 PM +0200 openldap-ml@stresst.net wrote:
Attached to this message you'll find a quick and dirty bash script that should determine the cachesize for an in-production bdb/hdb backend by taking dn2id and id2entry as well as all existing indexes into account.
Internally the script calls the db_stat tool in combination with the formulas found in the 2.4 admin guide under: http://www.openldap.org/doc/admin24/tuning.html#Caching
Are you trying to determine the size of the cache in use, or what the size of the cache should be? If it is the latter, you just use the du -c -h output of *.bdb. If it is the former, it sounds like your script isn't calculating the right data.
Ahh I see - thanks! It seems that we do not need to calculate the cachesize (see below). Nevertheless, as I'm using cn=config I'm a little bit confused regarding the available different cache settings: The related cn=config configuration directives are all prefixed by "olcDb..." where olcDbCachesize for example seems to store the related DB_CONFIG setting but olcDbDnCachesize seems to be focused on slapd internal use.
BTW: slapd.conf's "dbconfig" directive ("convenience settings" mentioned in man slapd-bdb.5) are not transformed into cn=config format when using slaptest -f -F (I'm using slapd 2.4.21). slapd.conf's dbconfig directive seems to be ignored during slaptest -f -F.
Last but not least please let me summarize my current understanding after reading AdminGuide2.4 once again:
Section 21.4.1. ==> Berkeley DB Cache
21.4.1.(a) ==> slapadd import cachesize Cachesize is the overall filesize of *.bdb
21.4.1.(b) ==> operation cachesize Cachesize is the filesize of id2entry.bdb + 10%
BDB Cachesize is configureable various ways: - slapd.conf: (dbconfig ...) - cn=config: (olcDbCacheSize: x) - DB_CONFIG: (set_cachesize 0 x 1) DB_CONFIG (if exists) takes precedence
Section 21.4.1.1. ==> Calculating Cachesize
only needed in case the above file sizes do not completely fit into available free memory. As we've currently enough free RAM we can skip these calculations.
Section 21.4.2. ==> slapd(8) Entry Cache (cachesize)
cn=config (olcDbDNcacheSize: y)
Section 21.4.3. ==> IDL Cache (idlcachesize)
cn=config (olcDbIDLcacheSize: z) recommendation: z=y (for bdb) recommendation: z=3x y (for hdb)
On Tue, 13 Jul 2010 11:19:06 +0200, openldap-ml@stresst.net wrote:
On 07/12/2010 07:40 PM, Quanah Gibson-Mount wrote:
--On Monday, July 12, 2010 5:01 PM +0200 openldap-ml@stresst.net wrote:
Attached to this message you'll find a quick and dirty bash script that should determine the cachesize for an in-production bdb/hdb backend by taking dn2id and id2entry as well as all existing indexes into account.
Internally the script calls the db_stat tool in combination with the formulas found in the 2.4 admin guide under: http://www.openldap.org/doc/admin24/tuning.html#Caching
Are you trying to determine the size of the cache in use, or what the size of the cache should be? If it is the latter, you just use the du -c -h output of *.bdb. If it is the former, it sounds like your script isn't calculating the right data.
Ahh I see - thanks! It seems that we do not need to calculate the cachesize (see below). Nevertheless, as I'm using cn=config I'm a little bit confused regarding the available different cache settings: The related cn=config configuration directives are all prefixed by "olcDb..." where olcDbCachesize for example seems to store the related DB_CONFIG setting but olcDbDnCachesize seems to be focused on slapd internal use.
Ah, these are two different settings: - olcDbCachesize is the size (in number of entries) of OpenLDAP's internal cache. This is not related to BerkeleyDB. It is recommended to set this to a value bigger than the total number of entries, if possible. - set_cachesize from DB_CONFIG is the size (in bytes) of BerkeleyDB's cache. It is separate form the former, and does not store entries, but indexes (basically). This can be set for convenience via the attribute olcDbConfig in cn=config.
BTW: slapd.conf's "dbconfig" directive ("convenience settings" mentioned in man slapd-bdb.5) are not transformed into cn=config format when using slaptest -f -F (I'm using slapd 2.4.21). slapd.conf's dbconfig directive seems to be ignored during slaptest -f -F.
I've not tested this, but the "dbconfig" should be transformed into "olcDbConfig" attribute. If this is not the case in the latest release, please file a bug in the ITS.
Last but not least please let me summarize my current understanding after reading AdminGuide2.4 once again:
Section 21.4.1. ==> Berkeley DB Cache
21.4.1.(a) ==> slapadd import cachesize Cachesize is the overall filesize of *.bdb
21.4.1.(b) ==> operation cachesize Cachesize is the filesize of id2entry.bdb + 10%
BDB Cachesize is configureable various ways:
- slapd.conf: (dbconfig ...)
- cn=config: (olcDbCacheSize: x)
No: as explained above, this is a different setting: olcDbConfig is what you're looking for.
- DB_CONFIG: (set_cachesize 0 x 1)
DB_CONFIG (if exists) takes precedence
Yes.
Section 21.4.1.1. ==> Calculating Cachesize
only needed in case the above file sizes do not completely fit into available free memory. As we've currently enough free RAM we can skip these calculations.
Section 21.4.2. ==> slapd(8) Entry Cache (cachesize)
cn=config (olcDbDNcacheSize: y)
Entry cache is olcDbCachesize. olcDbDnCacheSize is the DN cache setting. See man slapd-bdb(5) for details.
Jonathan
--On Tuesday, July 13, 2010 11:44 AM +0200 Jonathan Clarke jonathan@phillipoux.net wrote:
Cachesize is the filesize of id2entry.bdb + 10%
It is dn2id.bdb + id2entry.bdb + 10% for the BDB set_cachesize value for operational mode.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
openldap-technical@openldap.org
-
Jonathan Clarke -
openldap-ml＠stresst.net -
Quanah Gibson-Mount