On 8/11/07, Howard Chu hyc@symas.com wrote:

Sumith Narayanan wrote:

...

Hi Group,

We have a coorporate openldap database in production which has more than 4 million entries. The slapd process serves three different physical dabases of sizes 4 GB , 12 GB and 24 GB respecievely. Earlier it was runnning in ldab backend and was crashing once in 3 -4 days. We upgraded to OpenLDAP 2.3.27 and BDB backend 4.4.20. Mac OS Tiger with 4GB RAM in the master and 8 GB ram in the slave. Now it crashes often may be couple of times in a day or sometimes once in 2 days. There is nothing much written in the log files but at times it shows out of memory exception. However , the search performance is still good.

• What is the maximum size of the database that OpenLDAP can support ?
• What is the maxmum number of dns it can hold ?
• Is there any solution for the above problem ?

For a machine with 4GB of RAM I think it will max out at around 1-2 million DNs. Unfortunately there is no way to control the size of the DN cache in OpenLDAP 2.3, it simply grows without bound. A config keyword for the DN cache will be in OpenLDAP 2.4. --

This is news to me. Is there a discussion somewhere of the different caches in openldap/bdb, which ones are tunable and which ones are not?

Could we get an idea with the bdb cache (DB_CONFIG) were set to the lowest values calculated from db_stat and (FAQ) "How do I determine the proper BDB/HDB database cache size?" and then working out a slapd.conf cachesize/idlcachesize?

Hi All,

Thanks for the suggestions :

I will look forward to upgrade the systems to 64 bit processors and increase the memory to 16 GB, since it will take time , I am doing some more tweaking to see whether it will help.

I reduced the set_cachesize in the DB_CONFIG by 10 times and also in the slapd.conf file. Restarted the process and it was consuming less memory now. But the master from whether the updates flow to slave is still slowing using up the memory and crashing , but the slave seems to be stable with a usage of around 200 MB of memory. I don't think I will be able to avoid that with the current situation.

I have few questions now :

- Any idea on how big the openldap DB can be ? - I see some DB transaction files in the db directory , which are of same size and more than one is created in a day :

-rw------- 1 ldapp ldapp 20485760 Aug 13 21:40 log.0000001070 -rw------- 1 ldapp ldapp 20485760 Aug 14 05:52 log.0000001071 -rw------- 1 ldapp ldapp 20485760 Aug 14 16:08 log.0000001072 -rw------- 1 ldapp ldapp 20485760 Aug 14 23:06 log.0000001073

I took a backup of a lot of this inorder for the slapd process to start faster. It keeps on getting accumulated. I guess these are DB transaction log files. Is it supposed to get deleted automatically ? Why is it accumulating ?

Thanks, Sumith.

On 8/11/07, Howard Chu hyc@symas.com wrote:

matthew sporleder wrote:

...

On 8/11/07, Howard Chu hyc@symas.com wrote:

...

Sumith Narayanan wrote:

...

Hi Group,

We have a coorporate openldap database in production which has more than 4 million entries. The slapd process serves three different physical dabases of sizes 4 GB , 12 GB and 24 GB respecievely. Earlier it was runnning in ldab backend and was crashing once in 3 -4 days. We upgraded to OpenLDAP 2.3.27 and BDB backend 4.4.20. Mac OS Tiger with 4GB RAM in the master and 8 GB ram in the slave. Now it crashes often may be couple of times in a day or sometimes once in 2 days. There is nothing much written in the log files but at times it shows out of memory exception. However , the search performance is still good.

• What is the maximum size of the database that OpenLDAP can support ?
• What is the maxmum number of dns it can hold ?
• Is there any solution for the above problem ?

For a machine with 4GB of RAM I think it will max out at around 1-2 million DNs. Unfortunately there is no way to control the size of the DN cache in OpenLDAP 2.3, it simply grows without bound. A config keyword for the DN cache will be in OpenLDAP 2.4. --

This is news to me. Is there a discussion somewhere of the different caches in openldap/bdb, which ones are tunable and which ones are not?

That 1-2 million estimate was wrong. This post was closer: http://www.openldap.org/lists/openldap-software/200611/msg00051.html

The limit isn't really the 4GB total RAM, but rather the available address space in a 32 bit process (usually only 2GB, sometimes as much as 3GB depending on the OS and any tweaks in the OS).

As already noted above, the DN cache is not tunable in 2.3. I frankly didn't see any wisdom in accommodating extremely large databases on 32-bit servers.

...

Could we get an idea with the bdb cache (DB_CONFIG) were set to the lowest values calculated from db_stat and (FAQ) "How do I determine the proper BDB/HDB database cache size?" and then working out a slapd.conf cachesize/idlcachesize?

I think Gavin Henry has written up a tuning section in the 2.4 Admin Guide. Balancing the slapd caches vs the BDB caches is already addressed in the FAQ article. Bottom of the page http://www.openldap.org/faq/index.cgi?file=1075 though of course it doesn't talk about the DN cache. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/