Hello all,
I have a set of 11 servers setup with syncrepl, all running openldap 2.4.6: 1 syncrepl provider, and 10 consumers.
The provider is having a hard time staying up for more than 18-24 hours at a time - it will either stop responding, or just die. I'm working on getting the debug log output right before it dies, but for now, about 60% of my provider's log is filled with messages such as this:
Dec 28 15:04:15 ldapm01 slapd[10335]: slap_sl_malloc of 32 bytes failed, using ch_malloc Dec 28 15:04:15 ldapm01 slapd[10335]: slap_sl_malloc of 56 bytes failed, using ch_malloc Dec 28 15:04:15 ldapm01 slapd[10335]: slap_sl_malloc of 56 bytes failed, using ch_malloc
The logs are quite copious, but I haven't seen anything else indicative of an error. I've been researching for over a week, and attempting to tweak my settings, but I can't find any data on this.
Does this have anything to do with the 'searchstack' setting? I looked in the source code, and noticed that it didn't seem like a terrible error, but this is the only output I have for the time being.
Any help would be greatly appreciated, as these are production machines. If the information I provided doesn't help, please tell me what you would need to understand the problem better.
########Provider: database bdb suffix "dc=testbox,dc=net" directory "/var/lib/ldap/master/" mode 0600 cachesize 20000 idlcachesize 60000 index objectClass eq index ou eq index entryCSN,entryUUID eq index uid,uidNumber,gidNumber eq idletimeout 30 overlay syncprov syncprov-checkpoint 100 10 syncprov-sessionlog 1000 syncprov-reloadhint TRUE lastmod on
############Consumer: database bdb suffix "dc=testbox,dc=net" directory "/var/lib/ldap/slave/" mode 0600 cachesize 10000 idlcachesize 30000
#### Consumer checkpoint 512 5 index uid,uidNumber,gidNumber,member pres,eq index ou,objectClass,entryCSN,entryUUID,homeDirectory,dc,contextCSN,associatedDomain,cn eq idletimeout 30 lastmod on
syncrepl rid=233 provider=ldaps://ldapm01.testbox.net type=refreshOnly retry="10 100 30 +" interval="00:00:05:00" searchbase="dc=eigbox,dc=net" filter="(!(entryDN:dnSubtreeMatch:=ou=unwanted,ou=pub,dc=testbox,dc=net))" scope=sub attrs="*,+" schemachecking=off bindmethod=simple binddn="cn=sync-user,dc=testbox,dc=net" credentials=secret #############
Thanks in advance! ------------------------------------------ Marcel Chastain Senior Systems Administrator Endurance International Group Email: Marcel.Chastain@Gmail.com ------------------------------------------
Marcel Chastain skrev, on 30-12-2007 06:17:
I have a set of 11 servers setup with syncrepl, all running openldap 2.4.6: 1 syncrepl provider, and 10 consumers.
The provider is having a hard time staying up for more than 18-24 hours at a time - it will either stop responding, or just die. I'm working on getting the debug log output right before it dies, but for now, about 60% of my provider's log is filled with messages such as this:
Dec 28 15:04:15 ldapm01 slapd[10335]: slap_sl_malloc of 32 bytes failed, using ch_malloc Dec 28 15:04:15 ldapm01 slapd[10335]: slap_sl_malloc of 56 bytes failed, using ch_malloc Dec 28 15:04:15 ldapm01 slapd[10335]: slap_sl_malloc of 56 bytes failed, using ch_malloc
1: Did you know that 2.4.7 has been available for quite some time now? It might be as well to install this on the provider and at least one consumer to ascertain that the fault still occurs with this version;
2: It might be as well to report your OS distro and version when reporting such errors.
Best,
--Tonni
On Dec 29, 2007 11:44 PM, Tony Earnshaw tonni@hetnet.nl wrote:
1: Did you know that 2.4.7 has been available for quite some time now? It might be as well to install this on the provider and at least one consumer to ascertain that the fault still occurs with this version;
I recall reading somewhere that upgrading to 2.4.7 requires a full re-import or re-index, and I'm running with about a million records.
However, looking through the changelog, I see that the fixes in 2.4.7 are pretty significant, and I'm working on upgrading now.
In the meantime, I finally got a bit more log data about the crash:
<...normal operations...> Dec 30 04:41:02 ldapm01 slapd[29500]: bdb(dc=testbox,dc=net): malloc: Cannot allocate memory: 3145764 Dec 30 04:41:02 ldapm01 slapd[29500]: bdb(dc=testbox,dc=net): txn_checkpoint: failed to flush the buffer cache Cannot allocate memory Dec 30 04:41:02 ldapm01 slapd[29500]: conn=3877 fd=21 closed (connection lost) Dec 30 04:41:02 ldapm01 slapd[29500]: conn=3876 op=2 RESULT tag=105 err=0 text= Dec 30 04:41:02 ldapm01 slapd[29500]: bdb(dc=testbox,dc=net): malloc: Cannot allocate memory: 3145764 Dec 30 04:41:02 ldapm01 slapd[29500]: bdb(dc=testbox,dc=net): txn_checkpoint: failed to flush the buffer cache Cannot allocate memory Dec 30 04:41:02 ldapm01 slapd[29500]: conn=3876 fd=12 closed (connection lost) Dec 30 04:41:03 ldapm01 slapd[29500]: ch_realloc of 8676 bytes failed <daemon dies>
I saw a few search results for this, including http://www.openldap.org/lists/openldap-software/200504/msg00088.html , in which the error text ("Cannot allocate memory: 3145764") is the exact same as mine.
2: It might be as well to report your OS distro and version when reporting such errors.
Running Debian 3.1, Kernel 2.6.18.x, openldap 2.4.6
I appreciate the feedback, Thanks!
--On Sunday, December 30, 2007 2:33 PM -0800 Marcel Chastain marcel.chastain@gmail.com wrote:
On Dec 29, 2007 11:44 PM, Tony Earnshaw tonni@hetnet.nl wrote:
1: Did you know that 2.4.7 has been available for quite some time now? It might be as well to install this on the provider and at least one consumer to ascertain that the fault still occurs with this version;
I recall reading somewhere that upgrading to 2.4.7 requires a full re-import or re-index, and I'm running with about a million records.
However, looking through the changelog, I see that the fixes in 2.4.7 are pretty significant, and I'm working on upgrading now.
No, it requires reindexing any attribute that uses the integer matching rules. Note that in OpenLDAP 2.4, you can selectively re-index only the attributes you are interested in updating the indices for. So you can do something like:
slapindex -q -t <attr1> <attr2>
etc.
See the examples section of:
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
--On Sunday, December 30, 2007 2:33 PM -0800 Marcel Chastain marcel.chastain@gmail.com wrote:
I recall reading somewhere that upgrading to 2.4.7 requires a full re-import or re-index, and I'm running with about a million records.
However, looking through the changelog, I see that the fixes in 2.4.7 are pretty significant, and I'm working on upgrading now.
No, it requires reindexing any attribute that uses the integer matching rules. Note that in OpenLDAP 2.4, you can selectively re-index only the attributes you are interested in updating the indices for. So you can do something like:
slapindex -q -t <attr1> <attr2>
etc.
See the examples section of:
Actually, as I noted here http://www.openldap.org/lists/openldap-software/200712/msg00193.html you also need to reindex any attribute that uses presence indexing.
--On Sunday, December 30, 2007 7:36 PM -0800 Howard Chu hyc@symas.com wrote:
Quanah Gibson-Mount wrote:
--On Sunday, December 30, 2007 2:33 PM -0800 Marcel Chastain marcel.chastain@gmail.com wrote:
I recall reading somewhere that upgrading to 2.4.7 requires a full re-import or re-index, and I'm running with about a million records.
However, looking through the changelog, I see that the fixes in 2.4.7 are pretty significant, and I'm working on upgrading now.
No, it requires reindexing any attribute that uses the integer matching rules. Note that in OpenLDAP 2.4, you can selectively re-index only the attributes you are interested in updating the indices for. So you can do something like:
slapindex -q -t <attr1> <attr2>
etc.
See the examples section of:
Actually, as I noted here http://www.openldap.org/lists/openldap-software/200712/msg00193.html you also need to reindex any attribute that uses presence indexing.
Okay, so all presence indexes, and all attributes that are indexed and use the integer matching rules. Which still (hopefully) is less than everything. ;)
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
openldap-software@openldap.org
-
Howard Chu -
Marcel Chastain -
Quanah Gibson-Mount -
Tony Earnshaw