Hi,
I'm wondering, how I could restrict the memory usage of the slapd process.
We got a quite large amount of data (23 million entries) in our login ldap. Since we are still using OpenLDAP 2.0 and of course want to migrate now, I added our data into an OpenLDAP 2.3.37 (this version will be updated soon). The problem I stumble over is that while inserting the data with ldapadd (mainly to check the performance we have to expect in the future), the amount of memory the slapd uses endlessly grows. The slapd process uses 11GB of memory after the insert has finished.
We are using the BDB backend which is currently configured to use 4GB of shared memory.
Machine: Linux masterldap 2.6.21.5 #1 SMP Mon Jul 2 13:54:33 CEST 2007 x86_64 GNU/Linux, with 2 Dual-Core AMD Opteron(tm) Processor 2218 and 16GB of RAM.
slapd.conf (converted to cn=config format):
include /usr/local/our/ldap/etc/openldap/schema/core.schema include /usr/local/our/ldap/etc/openldap/schema/our.schema
pidfile /usr/local/our/ldap/var/ldap/run/slapd.pid argsfile /usr/local/our/ldap/var/ldap/run/slapd.args
# Load dynamic backend modules:
modulepath /usr/local/our/ldap/lib moduleload back_bdb.la
access to * by * write
loglevel 0 sizelimit 10000 timelimit 3600
cachesize 1000000 tool-threads 2
# backend definition backend bdb
database config rootdn "cn=root,cn=config" rootpw {SSHA}***
####################################################################### # BDB database definitions #######################################################################
# first database definition & config directives database bdb
directory /usr/local/our/ldap/var/ldap/openldap-data/ replogfile /usr/local/our/ldap/log/replica.log
suffix "o=our"
rootdn "cn=root,o=our" rootpw {SSHA}***
index cid eq index cn eq,sub index objectClass eq index folderName eq index locked eq
DB_CONFIG: set_flags DB_LOG_AUTOREMOVE set_cachesize 4 0 2 set_lg_max 524288000 set_lg_regionmax 512000 set_lg_bsize 268435456
On Monday 31 March 2008 11:21:15 Ralf Narozny wrote:
Hi,
I'm wondering, how I could restrict the memory usage of the slapd process.
We got a quite large amount of data (23 million entries) in our login ldap. Since we are still using OpenLDAP 2.0 and of course want to migrate now, I added our data into an OpenLDAP 2.3.37 (this version will be updated soon). The problem I stumble over is that while inserting the data with ldapadd (mainly to check the performance we have to expect in the future), the amount of memory the slapd uses endlessly grows. The slapd process uses 11GB of memory after the insert has finished.
We are using the BDB backend which is currently configured to use 4GB of shared memory.
Machine: Linux masterldap 2.6.21.5 #1 SMP Mon Jul 2 13:54:33 CEST 2007 x86_64 GNU/Linux, with 2 Dual-Core AMD Opteron(tm) Processor 2218 and 16GB of RAM.
slapd.conf (converted to cn=config format):
[...]
cachesize 1000000
This belongs in the database section, I am not sure how it operates when set globally, but I doubt that after your 4GB of memory for BDB you will have sufficient memory to cache 1 million entries (although you would want to check what your entry size is when tuning this value).
However, since there is no way that you will cache your entire database in the entry cache, you should probably reduce it to a more reasonable working set, and ensure that you have sufficient BD cachesize.
/me notes that it would be nice to have more detail on the entry cache available via back-monitor, such as the number of entries in the cache, and the amount of entry cache that is used ...
Regards, Buchan
Buchan Milne wrote:
/me notes that it would be nice to have more detail on the entry cache available via back-monitor, such as the number of entries in the cache, and the amount of entry cache that is used ...
Something like
bash-3.1$ ldapsearch -x -H ldap://:9011 -b 'cn=Databases,cn=Monitor' \ '(objectclass=olmBDBDatabase)' @olmBDBDatabase # extended LDIF # # LDAPv3 # base <cn=Databases,cn=Monitor> with scope subtree # filter: (objectclass=olmBDBDatabase) # requesting: @olmBDBDatabase #
# Database 1, Databases, Monitor dn: cn=Database 1,cn=Databases,cn=Monitor objectClass: monitoredObject objectClass: olmBDBDatabase olmBDBEntryCache: 0 olmBDBDNCache: 0 olmBDBIDLCache: 0 olmDbDirectory: /home/masarati/Ldap/tests/testrun/db.1.a/
# search result search: 2 result: 0 Success
# numResponses: 2 # numEntries: 1
p.
Ing. Pierangelo Masarati OpenLDAP Core Team
SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati@sys-net.it ---------------------------------------
Hi,
Pierangelo Masarati schrieb:
Buchan Milne wrote:
/me notes that it would be nice to have more detail on the entry cache available via back-monitor, such as the number of entries in the cache, and the amount of entry cache that is used ...
Something like
bash-3.1$ ldapsearch -x -H ldap://:9011 -b 'cn=Databases,cn=Monitor' \ '(objectclass=olmBDBDatabase)' @olmBDBDatabase
I configured the slapd to create a monitor, but the information you want is not present.
Maybe I missed something to configure, but the manual is not too thouroughly written by now ;-)
database monitor rootdn "cn=root,cn=monitor" rootpw {SSHA}...
ldapsearch -D 'cn=root,cn=monitor' -W -b 'cn=Databases,cn=Monitor' 'objectclass=*' '*' '+'
(as far as I understood, this should show all data for the entries below cn=Databases,cn=Monitor')
# extended LDIF # # LDAPv3 # base <cn=Databases,cn=Monitor> with scope subtree # filter: objectclass=* # requesting: * + #
# Databases, Monitor dn: cn=Databases,cn=Monitor objectClass: monitorContainer structuralObjectClass: monitorContainer cn: Databases creatorsName: cn=root,cn=monitor modifiersName: cn=root,cn=monitor createTimestamp: 20080401132535Z modifyTimestamp: 20080401132535Z description: This subsystem contains information about configured databases. readOnly: FALSE namingContexts: cn=config namingContexts: o=our monitorContext: cn=Monitor entryDN: cn=Databases,cn=Monitor subschemaSubentry: cn=Subschema hasSubordinates: TRUE
# Database 0, Databases, Monitor dn: cn=Database 0,cn=Databases,cn=Monitor objectClass: monitoredObject structuralObjectClass: monitoredObject cn: Database 0 monitoredInfo: config monitorIsShadow: FALSE creatorsName: cn=root,cn=monitor modifiersName: cn=root,cn=monitor createTimestamp: 20080401132535Z modifyTimestamp: 20080401132535Z namingContexts: cn=config readOnly: FALSE seeAlso: cn=Backend 0,cn=Backends,cn=Monitor entryDN: cn=Database 0,cn=Databases,cn=Monitor subschemaSubentry: cn=Subschema hasSubordinates: FALSE
# Database 1, Databases, Monitor dn: cn=Database 1,cn=Databases,cn=Monitor objectClass: monitoredObject structuralObjectClass: monitoredObject cn: Database 1 monitoredInfo: bdb monitorIsShadow: FALSE creatorsName: cn=root,cn=monitor modifiersName: cn=root,cn=monitor createTimestamp: 20080401132535Z modifyTimestamp: 20080401132535Z namingContexts: o=our readOnly: FALSE labeledURI:: ZmlsZTovLy91c3IvbG9jYWwvZnJlZW5ldC9sZGFwL3Zhci9sZGFwL29wZW5sZGFwL WRhdGEvAC8= seeAlso: cn=Backend 2,cn=Backends,cn=Monitor entryDN: cn=Database 1,cn=Databases,cn=Monitor subschemaSubentry: cn=Subschema hasSubordinates: FALSE
# Database 2, Databases, Monitor dn: cn=Database 2,cn=Databases,cn=Monitor objectClass: monitoredObject structuralObjectClass: monitoredObject cn: Database 2 monitoredInfo: monitor monitorIsShadow: FALSE creatorsName: cn=root,cn=monitor modifiersName: cn=root,cn=monitor createTimestamp: 20080401132535Z modifyTimestamp: 20080401132535Z monitorContext: cn=Monitor readOnly: FALSE seeAlso: cn=Backend 5,cn=Backends,cn=Monitor entryDN: cn=Database 2,cn=Databases,cn=Monitor subschemaSubentry: cn=Subschema hasSubordinates: FALSE
# search result search: 2 result: 0 Success
# numResponses: 5 # numEntries: 4
Ralf Narozny wrote:
Hi,
Pierangelo Masarati schrieb:
Buchan Milne wrote:
/me notes that it would be nice to have more detail on the entry cache available via back-monitor, such as the number of entries in the cache, and the amount of entry cache that is used ...
Something like
bash-3.1$ ldapsearch -x -H ldap://:9011 -b 'cn=Databases,cn=Monitor' \ '(objectclass=olmBDBDatabase)' @olmBDBDatabase
I configured the slapd to create a monitor, but the information you want is not present.
Maybe I missed something to configure, but the manual is not too thouroughly written by now ;-)
database monitor rootdn "cn=root,cn=monitor" rootpw {SSHA}...
ldapsearch -D 'cn=root,cn=monitor' -W -b 'cn=Databases,cn=Monitor' 'objectclass=*' '*' '+'
(as far as I understood, this should show all data for the entries below cn=Databases,cn=Monitor')
Well, that information is only available since OpenLDAP 2.4; I infer you're using an earlier distribution. In any case, the monitor has nothing to do with the entry cache configuration, it only shows the current usage. Refer to slapd.conf or back-config for what is configured for your system.
p.
Ing. Pierangelo Masarati OpenLDAP Core Team
SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati@sys-net.it ---------------------------------------
Pierangelo Masarati schrieb:
Ralf Narozny wrote:
Hi,
Pierangelo Masarati schrieb:
Buchan Milne wrote:
/me notes that it would be nice to have more detail on the entry cache available via back-monitor, such as the number of entries in the cache, and the amount of entry cache that is used ...
Something like
bash-3.1$ ldapsearch -x -H ldap://:9011 -b 'cn=Databases,cn=Monitor' \ '(objectclass=olmBDBDatabase)' @olmBDBDatabase
I configured the slapd to create a monitor, but the information you want is not present.
Maybe I missed something to configure, but the manual is not too thouroughly written by now ;-)
database monitor rootdn "cn=root,cn=monitor" rootpw {SSHA}...
ldapsearch -D 'cn=root,cn=monitor' -W -b 'cn=Databases,cn=Monitor' 'objectclass=*' '*' '+'
(as far as I understood, this should show all data for the entries below cn=Databases,cn=Monitor')
Well, that information is only available since OpenLDAP 2.4; I infer you're using an earlier distribution. In any case, the monitor has nothing to do with the entry cache configuration, it only shows the current usage. Refer to slapd.conf or back-config for what is configured for your system.
Yep, as I wrote in my initial mail, we are using 2.3.32 (for testing so far).
And I wrote that we are using BDB which is configured to use 4GB of shared mem. The only problem I have is that with 1000000 entries configured as entry cache, slapd uses 11GB out of 16 GB of RAM after the insert with ldapadd. Which makes it use 7GB for entry cache (and whatever else).
Our entries have (in LDIF of course) an average size of below 200 Bytes. So taking 6GB out of the 7GB used as size of the entry cache, it would mean that each entry consumes about 6K of RAM. Is that correct?
If so, is there any documentation on how to configure the slapd for a larger amount of entries like ours?
p.
Ing. Pierangelo Masarati OpenLDAP Core Team
SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it
Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati@sys-net.it
--On Wednesday, April 02, 2008 10:41 AM +0200 Ralf Narozny rnarozny@web.de wrote:
Yep, as I wrote in my initial mail, we are using 2.3.32 (for testing so far).
The current release is 2.3.41. Why are you using such an old release? There've been many known problems fixed since that release, including security issues, and possibly some memory leaks.
What version of BDB are you using? Have you patched it with all the relevant patches?
And I wrote that we are using BDB which is configured to use 4GB of shared mem. The only problem I have is that with 1000000 entries configured as entry cache, slapd uses 11GB out of 16 GB of RAM after the insert with ldapadd. Which makes it use 7GB for entry cache (and whatever else).
What's the contents of your DB_CONFIG file? What's the size du -c -h *.bdb in your database directory? What is your idlcachesize setting? How many file descriptors do you allocate to slapd?
Since you are using ldapadd to populate the database (rather than slapadd), the entry cache is getting immediately allocated. As I described in a previous email writeup I did on performant tuning of slapd, the DB_CONFIG setting is the most important (to hold the size of your entire DB), followed by the entry cache size.
The entry cache really only needs to be the size of your active set of entries, rather than all entries. For example, at a previous position, even though we had about 500,000 entries in our database, our entry cache was only 20,000. And it was very performant.
Our entries have (in LDIF of course) an average size of below 200 Bytes. So taking 6GB out of the 7GB used as size of the entry cache, it would mean that each entry consumes about 6K of RAM. Is that correct?
If so, is there any documentation on how to configure the slapd for a larger amount of entries like ours?
How many entries do you have?
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount schrieb:
--On Wednesday, April 02, 2008 10:41 AM +0200 Ralf Narozny rnarozny@web.de wrote:
Yep, as I wrote in my initial mail, we are using 2.3.32 (for testing so far).
The current release is 2.3.41. Why are you using such an old release? There've been many known problems fixed since that release, including security issues, and possibly some memory leaks.
We are using that version, because it is a project that is not highly priorized and the package was compiled long a ago. But I just initialized an upgrade. Memory leaks are for sure a show stopper.
What version of BDB are you using? Have you patched it with all the relevant patches?
We are using BDB 4.4.20. With as far as I know no patches. Any recommended version or neccessary patches?
And I wrote that we are using BDB which is configured to use 4GB of shared mem. The only problem I have is that with 1000000 entries configured as entry cache, slapd uses 11GB out of 16 GB of RAM after the insert with ldapadd. Which makes it use 7GB for entry cache (and whatever else).
What's the contents of your DB_CONFIG file? What's the size du -c -h *.bdb in your database directory? What is your idlcachesize setting? How many file descriptors do you allocate to slapd?
DB_CONFIG: set_flags DB_LOG_AUTOREMOVE set_cachesize 4 0 2 set_lg_max 524288000 set_lg_regionmax 512000 set_lg_bsize 268435456
du -c -h *.bdb: 792M cid.bdb 5,7G cn.bdb 4,1G dn2id.bdb 36M folderName.bdb 15G id2entry.bdb 948K locked.bdb 1,9M objectClass.bdb
idlcachesize: 0, because we only use it for customer logins, so there are no searches besides the ones for the customer id
If you mean the number of file descriptors ready to be used, it is 4096 currently, but might be raised, if neccessary.
Since you are using ldapadd to populate the database (rather than slapadd), the entry cache is getting immediately allocated. As I described in a previous email writeup I did on performant tuning of slapd, the DB_CONFIG setting is the most important (to hold the size of your entire DB), followed by the entry cache size.
Yep, that is my problem, we will surely need a lot of entries in cache, because, we got more than 2 million customers logging in per day.
Can you tell me the subject of the mail writeup so I can find it in the list archive?
The entry cache really only needs to be the size of your active set of entries, rather than all entries. For example, at a previous position, even though we had about 500,000 entries in our database, our entry cache was only 20,000. And it was very performant.
We got 23 million entries of which more than 2 million are active entries...we can not really reduce the size of our database. Maybe a few thousands, but not millions ;-)
Our entries have (in LDIF of course) an average size of below 200 Bytes. So taking 6GB out of the 7GB used as size of the entry cache, it would mean that each entry consumes about 6K of RAM. Is that correct?
If so, is there any documentation on how to configure the slapd for a larger amount of entries like ours?
How many entries do you have?
Ralf Narozny wrote:
We got 23 million entries of which more than 2 million are active entries...we can not really reduce the size of our database. Maybe a few thousands, but not millions ;-)
This most likely will not work in OpenLDAP 2.3. You will need to use OpenLDAP 2.4 and configure back-bdb's DN cache size. (The DN cache is not configurable in 2.3, it simply grows without limit.)
Quanah Gibson-Mount schrieb:
--On Wednesday, April 02, 2008 10:41 AM +0200 Ralf Narozny rnarozny@web.de wrote:
Yep, as I wrote in my initial mail, we are using 2.3.32 (for testing so far).
The current release is 2.3.41. Why are you using such an old release? There've been many known problems fixed since that release, including security issues, and possibly some memory leaks.
We are using that version, because it is a project that is not highly priorized and the package was compiled long a ago. But I just initialized an upgrade. Memory leaks are for sure a show stopper.
What version of BDB are you using? Have you patched it with all the relevant patches?
We are using BDB 4.4.20. With as far as I know no patches. Any recommended version or neccessary patches?
And I wrote that we are using BDB which is configured to use 4GB of shared mem. The only problem I have is that with 1000000 entries configured as entry cache, slapd uses 11GB out of 16 GB of RAM after the insert with ldapadd. Which makes it use 7GB for entry cache (and whatever else).
What's the contents of your DB_CONFIG file? What's the size du -c -h *.bdb in your database directory? What is your idlcachesize setting? How many file descriptors do you allocate to slapd?
DB_CONFIG: set_flags DB_LOG_AUTOREMOVE set_cachesize 4 0 2 set_lg_max 524288000 set_lg_regionmax 512000 set_lg_bsize 268435456
du -c -h *.bdb: 792M cid.bdb 5,7G cn.bdb 4,1G dn2id.bdb 36M folderName.bdb 15G id2entry.bdb 948K locked.bdb 1,9M objectClass.bdb
idlcachesize: 0, because we only use it for customer logins, so there are no searches besides the ones for the customer id
If you mean the number of file descriptors ready to be used, it is 4096 currently, but might be raised, if neccessary.
Since you are using ldapadd to populate the database (rather than slapadd), the entry cache is getting immediately allocated. As I described in a previous email writeup I did on performant tuning of slapd, the DB_CONFIG setting is the most important (to hold the size of your entire DB), followed by the entry cache size.
Yep, that is my problem, we will surely need a lot of entries in cache, because, we got more than 2 million customers logging in per day.
Can you tell me the subject of the mail writeup so I can find it in the list archive?
The entry cache really only needs to be the size of your active set of entries, rather than all entries. For example, at a previous position, even though we had about 500,000 entries in our database, our entry cache was only 20,000. And it was very performant.
We got 23 million entries of which more than 2 million are active entries...we can not really reduce the size of our database. Maybe a few thousands, but not millions ;-)
Our entries have (in LDIF of course) an average size of below 200 Bytes. So taking 6GB out of the 7GB used as size of the entry cache, it would mean that each entry consumes about 6K of RAM. Is that correct?
If so, is there any documentation on how to configure the slapd for a larger amount of entries like ours?
How many entries do you have?
--On April 14, 2008 5:04:07 PM +0200 Ralf Narozny rnarozny@web.de wrote:
Quanah Gibson-Mount schrieb:
What version of BDB are you using? Have you patched it with all the relevant patches?
We are using BDB 4.4.20. With as far as I know no patches. Any recommended version or neccessary patches?
I guess you didn't take a look at the Oracle download site for BDB. It clearly lists that there are 4 patches available for BDB 4.4.20.
http://www.oracle.com/technology/software/products/berkeley-db/db/index.html
4.4.20 Requires log file format upgrade. change log - patches (4)
What's the contents of your DB_CONFIG file? What's the size du -c -h *.bdb in your database directory? What is your idlcachesize setting? How many file descriptors do you allocate to slapd?
DB_CONFIG: set_flags DB_LOG_AUTOREMOVE set_cachesize 4 0 2 set_lg_max 524288000 set_lg_regionmax 512000 set_lg_bsize 268435456
In past tests I did, I found that partitioning BDB into multiple segments had a negative impact on performance. I don't advise doing so unless it is absolutely necessary.
Also, as Howard noted, your dn cache is going to grow until it matches the full size of the DB. The only way you can resolve that is to upgrade to OpenLDAP 2.4.
I think the long term solutions to your memory issues will be to upgrade to 2.4 (I'm working on getting 2.4.9 release ready this week, but no promises).
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On Wed, Apr 2, 2008 at 10:41 AM, Ralf Narozny rnarozny@web.de wrote:
Pierangelo Masarati schrieb:
Ralf Narozny wrote:
Hi,
Pierangelo Masarati schrieb:
Buchan Milne wrote:
/me notes that it would be nice to have more detail on the entry
cache available via back-monitor, such as the number of entries in the cache, and the amount of entry cache that is used ...
Something like
bash-3.1$ ldapsearch -x -H ldap://:9011 -b 'cn=Databases,cn=Monitor' \ '(objectclass=olmBDBDatabase)' @olmBDBDatabase
I configured the slapd to create a monitor, but the information you want
is not present.
Maybe I missed something to configure, but the manual is not too
thouroughly written by now ;-)
database monitor rootdn "cn=root,cn=monitor" rootpw {SSHA}...
ldapsearch -D 'cn=root,cn=monitor' -W -b 'cn=Databases,cn=Monitor'
'objectclass=*' '*' '+'
(as far as I understood, this should show all data for the entries below
cn=Databases,cn=Monitor')
Well, that information is only available since OpenLDAP 2.4; I infer
you're using an earlier distribution. In any case, the monitor has nothing to do with the entry cache configuration, it only shows the current usage. Refer to slapd.conf or back-config for what is configured for your system.
Yep, as I wrote in my initial mail, we are using 2.3.32 (for testing so far).
And I wrote that we are using BDB which is configured to use 4GB of shared mem. The only problem I have is that with 1000000 entries configured as entry cache, slapd uses 11GB out of 16 GB of RAM after the insert with ldapadd.
Firstly, what is the problem with slapd using 11 of 16GB ? My production LDAP servers typically run consuming at least 4GB of the available 6GB, and that's the way I want it (or, maybe using a tad more, but leaving enough free to run a slapcat without causing the server to swap). Unused ram is wasted ram (at least on Unix) ...
Which makes it use 7GB for entry cache (and whatever else).
Plus the overhead of approx 10MB per-thread, a few kB per file descriptor etc.
Our entries have (in LDIF of course) an average size of below 200 Bytes. So taking 6GB out of the 7GB used as size of the entry cache, it would mean that each entry consumes about 6K of RAM. Is that correct?
Roughly ... assuming that you are using a decent memory allocator, and that you have applied the memory leak patches for Berkeley DB (I don't see that you provide your Berkeley DB version). The glibc memory allocator is probably going to do quite badly in this specific scenario (bulk add over the wire), using one of the better allocators (e.g. hoard, tcalloc), would probably reduce this value considerably. Howard has published extensive benchmarks on this ...
If so, is there any documentation on how to configure the slapd for a larger amount of entries like ours?
Yes, which ones have you read so far?
Regards, Buchan
Buchan Milne schrieb:
On Wed, Apr 2, 2008 at 10:41 AM, Ralf Narozny rnarozny@web.de wrote:
Pierangelo Masarati schrieb:
Ralf Narozny wrote:
Hi,
Pierangelo Masarati schrieb:
Buchan Milne wrote:
/me notes that it would be nice to have more detail on the entry
cache available via back-monitor, such as the number of entries in the cache, and the amount of entry cache that is used ...
Something like
bash-3.1$ ldapsearch -x -H ldap://:9011 -b 'cn=Databases,cn=Monitor' \ '(objectclass=olmBDBDatabase)' @olmBDBDatabase
I configured the slapd to create a monitor, but the information you want
is not present.
Maybe I missed something to configure, but the manual is not too
thouroughly written by now ;-)
database monitor rootdn "cn=root,cn=monitor" rootpw {SSHA}...
ldapsearch -D 'cn=root,cn=monitor' -W -b 'cn=Databases,cn=Monitor'
'objectclass=*' '*' '+'
(as far as I understood, this should show all data for the entries below
cn=Databases,cn=Monitor')
Well, that information is only available since OpenLDAP 2.4; I infer
you're using an earlier distribution. In any case, the monitor has nothing to do with the entry cache configuration, it only shows the current usage. Refer to slapd.conf or back-config for what is configured for your system.
Yep, as I wrote in my initial mail, we are using 2.3.32 (for testing so far).
And I wrote that we are using BDB which is configured to use 4GB of shared mem. The only problem I have is that with 1000000 entries configured as entry cache, slapd uses 11GB out of 16 GB of RAM after the insert with ldapadd.
Firstly, what is the problem with slapd using 11 of 16GB ? My production LDAP servers typically run consuming at least 4GB of the available 6GB, and that's the way I want it (or, maybe using a tad more, but leaving enough free to run a slapcat without causing the server to swap). Unused ram is wasted ram (at least on Unix) ...
No problem, I want to have slapd use about 14GB of the memory, but I'm not able to predict the RAM slapd uses, that's why I ask :-) On the other side I read about the importance of the BDB cache and that it should be using most of the available resources. But I cannot raise it above 4GB, because my machine will start to swap after inserting a few million entries. And that is really worst case.
Which makes it use 7GB for entry cache (and whatever else).
Plus the overhead of approx 10MB per-thread, a few kB per file descriptor etc.
Our entries have (in LDIF of course) an average size of below 200 Bytes. So taking 6GB out of the 7GB used as size of the entry cache, it would mean that each entry consumes about 6K of RAM. Is that correct?
Roughly ... assuming that you are using a decent memory allocator, and that you have applied the memory leak patches for Berkeley DB (I don't see that you provide your Berkeley DB version). The glibc memory allocator is probably going to do quite badly in this specific scenario (bulk add over the wire), using one of the better allocators (e.g. hoard, tcalloc), would probably reduce this value considerably. Howard has published extensive benchmarks on this ...
It is hard to know what info to provide for you to be able to help:
BDB 4.4.20 with no patches. Linux Kernel 2.6.21.5 SMP glibc-2.7-1 libc6 2.7-6
I'm not the one compiling the packet so I mainly have no idea of how to change anything like that.
If so, is there any documentation on how to configure the slapd for a larger amount of entries like ours?
Yes, which ones have you read so far?
I searched the OpenLDAP docs and a few pages everywhere in the net, but since they all think about 500k entries are a lot, it does not really help me with my growing 23 million entries. :-(
The size of the bdb files: 792M cid.bdb 5,7G cn.bdb 4,1G dn2id.bdb 36M folderName.bdb 15G id2entry.bdb 948K locked.bdb 1,9M objectClass.bdb
Which will never fit into my 16 GB :-)
Anything else you would need?
Regards, Buchan
Buchan Milne schrieb:
On Wed, Apr 2, 2008 at 10:41 AM, Ralf Narozny rnarozny@web.de wrote:
Pierangelo Masarati schrieb:
Ralf Narozny wrote:
Hi,
Pierangelo Masarati schrieb:
Buchan Milne wrote:
/me notes that it would be nice to have more detail on the entry
cache available via back-monitor, such as the number of entries in the cache, and the amount of entry cache that is used ...
Something like
bash-3.1$ ldapsearch -x -H ldap://:9011 -b 'cn=Databases,cn=Monitor' \ '(objectclass=olmBDBDatabase)' @olmBDBDatabase
I configured the slapd to create a monitor, but the information you want
is not present.
Maybe I missed something to configure, but the manual is not too
thouroughly written by now ;-)
database monitor rootdn "cn=root,cn=monitor" rootpw {SSHA}...
ldapsearch -D 'cn=root,cn=monitor' -W -b 'cn=Databases,cn=Monitor'
'objectclass=*' '*' '+'
(as far as I understood, this should show all data for the entries below
cn=Databases,cn=Monitor')
Well, that information is only available since OpenLDAP 2.4; I infer
you're using an earlier distribution. In any case, the monitor has nothing to do with the entry cache configuration, it only shows the current usage. Refer to slapd.conf or back-config for what is configured for your system.
Yep, as I wrote in my initial mail, we are using 2.3.32 (for testing so far).
And I wrote that we are using BDB which is configured to use 4GB of shared mem. The only problem I have is that with 1000000 entries configured as entry cache, slapd uses 11GB out of 16 GB of RAM after the insert with ldapadd.
Firstly, what is the problem with slapd using 11 of 16GB ? My production LDAP servers typically run consuming at least 4GB of the available 6GB, and that's the way I want it (or, maybe using a tad more, but leaving enough free to run a slapcat without causing the server to swap). Unused ram is wasted ram (at least on Unix) ...
No problem, I want to have slapd use about 14GB of the memory, but I'm not able to predict the RAM slapd uses, that's why I ask :-) On the other side I read about the importance of the BDB cache and that it should be using most of the available resources. But I cannot raise it above 4GB, because my machine will start to swap after inserting a few million entries. And that is really worst case.
Which makes it use 7GB for entry cache (and whatever else).
Plus the overhead of approx 10MB per-thread, a few kB per file descriptor etc.
Our entries have (in LDIF of course) an average size of below 200 Bytes. So taking 6GB out of the 7GB used as size of the entry cache, it would mean that each entry consumes about 6K of RAM. Is that correct?
Roughly ... assuming that you are using a decent memory allocator, and that you have applied the memory leak patches for Berkeley DB (I don't see that you provide your Berkeley DB version). The glibc memory allocator is probably going to do quite badly in this specific scenario (bulk add over the wire), using one of the better allocators (e.g. hoard, tcalloc), would probably reduce this value considerably. Howard has published extensive benchmarks on this ...
It is hard to know what info to provide for you to be able to help:
BDB 4.4.20 with no patches. Linux Kernel 2.6.21.5 SMP glibc-2.7-1 libc6 2.7-6
I'm not the one compiling the packet so I mainly have no idea of how to change anything like that.
If so, is there any documentation on how to configure the slapd for a larger amount of entries like ours?
Yes, which ones have you read so far?
I searched the OpenLDAP docs and a few pages everywhere in the net, but since they all think about 500k entries are a lot, it does not really help me with my growing 23 million entries. :-(
The size of the bdb files: 792M cid.bdb 5,7G cn.bdb 4,1G dn2id.bdb 36M folderName.bdb 15G id2entry.bdb 948K locked.bdb 1,9M objectClass.bdb
Which will never fit into my 16 GB :-)
Anything else you would need?
Regards, Buchan
--On April 14, 2008 5:04:24 PM +0200 Ralf Narozny rnarozny@web.de wrote:
Roughly ... assuming that you are using a decent memory allocator, and that you have applied the memory leak patches for Berkeley DB (I don't see that you provide your Berkeley DB version). The glibc memory allocator is probably going to do quite badly in this specific scenario (bulk add over the wire), using one of the better allocators (e.g. hoard, tcalloc), would probably reduce this value considerably. Howard has published extensive benchmarks on this ...
It is hard to know what info to provide for you to be able to help:
BDB 4.4.20 with no patches. Linux Kernel 2.6.21.5 SMP glibc-2.7-1 libc6 2.7-6
I'm not the one compiling the packet so I mainly have no idea of how to change anything like that.
One problem is that glibc is a terrible memory allocator. You really should use an alternative like tcmalloc or hoard.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount schrieb:
--On April 14, 2008 5:04:24 PM +0200 Ralf Narozny rnarozny@web.de wrote:
Roughly ... assuming that you are using a decent memory allocator, and that you have applied the memory leak patches for Berkeley DB (I don't see that you provide your Berkeley DB version). The glibc memory allocator is probably going to do quite badly in this specific scenario (bulk add over the wire), using one of the better allocators (e.g. hoard, tcalloc), would probably reduce this value considerably. Howard has published extensive benchmarks on this ...
It is hard to know what info to provide for you to be able to help:
BDB 4.4.20 with no patches. Linux Kernel 2.6.21.5 SMP glibc-2.7-1 libc6 2.7-6
I'm not the one compiling the packet so I mainly have no idea of how to change anything like that.
One problem is that glibc is a terrible memory allocator. You really should use an alternative like tcmalloc or hoard.
Any advise on the BDB version? Is the newer one better (performing and still free)?
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
--On April 14, 2008 6:31:53 PM +0200 Ralf Narozny rnarozny@web.de wrote:
One problem is that glibc is a terrible memory allocator. You really should use an alternative like tcmalloc or hoard.
Any advise on the BDB version? Is the newer one better (performing and still free)?
BDB 4.6 is not supported with OpenLDAP 2.3. You'd have to upgrade to OpenLDAP 2.4 first.
I use BDB 4.2.52, although I plan on moving to BDB 4.6 once I move to OpenLDAP 2.4. I never saw any improvements for OpenLDAP with 4.4 or 4.5 (which actually benched slightly slower in my tests than 4.2).
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
openldap-software@openldap.org
-
Aaron Richton -
Buchan Milne -
Buchan Milne -
Howard Chu -
Pierangelo Masarati -
Quanah Gibson-Mount -
Ralf Narozny