Hi,
I just ran into this yesterday on CentOS 6 / OpenLdap 2.4.36, my own build against bdb 5.1.29 and OpenSSL.
Dec 2 16:55:40 animal slapd[13735]: bdb(dc=clarku,dc=edu): /var/lib/ldap/log.0000032796: log file unreadable: Too many open files Dec 2 16:55:40 animal slapd[13735]: bdb(dc=clarku,dc=edu): PANIC: Too many open files Dec 2 16:55:40 animal slapd[13735]: bdb(dc=clarku,dc=edu): PANIC: fatal region error detected; run recovery Dec 2 16:55:40 animal slapd[13735]: bdb(dc=clarku,dc=edu): PANIC: fatal region error detected; run recovery Dec 2 16:55:40 animal slapd[13735]: null_callback : error code 0x50
I never did anything to change the default process limit, looks like it's set to 1024 in /proc/$(pidof slapd)/limits.
Is there anything I can do short of an automatic restart to prevent that from cropping up again? I have an MMR via Syncrepl setup with nothing too complicated; can't think of any reason offhand that slapd would need to have more than 1024 files open.
Best,
Aaron
--- Aaron Bennett Manager of Systems Administration Clark University ITS W:508.793.7315
On Dec 3, 2013, at 3:57 PM, Aaron Bennett wrote:
Dec 2 16:55:40 animal slapd[13735]: bdb(dc=clarku,dc=edu): /var/lib/ldap/log.0000032796: log file unreadable: Too many open files
Try setting fs.file-max (with sysctl or similar).
celia(SSD1):~# cat /proc/sys/fs/file-max 65536 celia(SSD1):~# grep file-max /etc/sysctl.conf fs.file-max = 65536
On 03/12/2013 14:57, Aaron Bennett wrote:
can't think of any reason offhand that slapd would need to have more than 1024 files open.
Each client connection to slapd uses a file handle. I've hit this problem in the past when a cluster of 300 systems came online for the first time, all of which had five threads configured in nslcd.conf. That was 1500 file handles straight away.
Liam Gretton liam.gretton@leicester.ac.uk schrieb am 03.12.2013 um 17:20 in
Nachricht 529E04BD.5000503@leicester.ac.uk:
On 03/12/2013 14:57, Aaron Bennett wrote:
can't think of any reason offhand that slapd would need to have more than
1024 files open.
Each client connection to slapd uses a file handle. I've hit this problem in the past when a cluster of 300 systems came online for the first time, all of which had five threads configured in nslcd.conf. That was 1500 file handles straight away.
With recent help from this list, I managed to get these values from my (older) server (a small landscape): # Max File Descriptors, Connections, Monitor dn: cn=Max File Descriptors,cn=Connections,cn=Monitor monitorCounter: 1024
# Total, Connections, Monitor dn: cn=Total,cn=Connections,cn=Monitor monitorCounter: 26972
# Current, Connections, Monitor dn: cn=Current,cn=Connections,cn=Monitor monitorCounter: 119
Maybe compare with your values. I was using a query like this (you'll get more output): "ldapsearch -Y EXTERNAL -H ldapi:/// -b 'cn=monitor' -s sub '(objectClass=*)' description monitoredInfo monitorCounter monitorOpInitiated monitorOpCompleted monitorConnectionPeerAddress monitorConnectionOpsReceived monitorConnectionActivityTime monitorTimestamp"
Regards, Ulrich
-- Liam Gretton liam.gretton@le.ac.uk Systems Specialist http://www.le.ac.uk/its IT Services Tel: +44 (0)116 2522254 University of Leicester, University Road Leicestershire LE1 7RH, United Kingdom
On Tue, 3 Dec 2013, Aaron Bennett wrote:
Dec 2 16:55:40 animal slapd[13735]: bdb(dc=clarku,dc=edu): /var/lib/ldap/log.0000032796: log file unreadable: Too many open files
[...]
Is there anything I can do short of an automatic restart to prevent that from cropping up again? I have an MMR via Syncrepl setup with nothing too complicated; can?t think of any reason offhand that slapd would need to have more than 1024 files open.
If you're not expecting this load, perhaps you should verify it with lsof(8) / pfiles(1) / similar tools ... then you can modify the limit and/or rectify the behavior as appropriate. If the behavior involves "unnecessary" long-lived connections, please read slapd.conf(5) paying particular attention to the idletimeout and writetimeout directives. I don't see any reason why restarts should be necessary.
Aaron Bennett abennett@clarku.edu schrieb am 03.12.2013 um 15:57 in Nachricht
A3F98FF07EAAD840ADBD429BEA17559026DA34E6@PERCH.ad.clarku.edu:
Hi,
I just ran into this yesterday on CentOS 6 / OpenLdap 2.4.36, my own build against bdb 5.1.29 and OpenSSL.
did you do an "lsof -p 13735"? I wonder what files are open...
Dec 2 16:55:40 animal slapd[13735]: bdb(dc=clarku,dc=edu): /var/lib/ldap/log.0000032796: log file unreadable: Too many open files Dec 2 16:55:40 animal slapd[13735]: bdb(dc=clarku,dc=edu): PANIC: Too many open files Dec 2 16:55:40 animal slapd[13735]: bdb(dc=clarku,dc=edu): PANIC: fatal region error detected; run recovery Dec 2 16:55:40 animal slapd[13735]: bdb(dc=clarku,dc=edu): PANIC: fatal region error detected; run recovery Dec 2 16:55:40 animal slapd[13735]: null_callback : error code 0x50
I never did anything to change the default process limit, looks like it's set to 1024 in /proc/$(pidof slapd)/limits.
Is there anything I can do short of an automatic restart to prevent that from cropping up again? I have an MMR via Syncrepl setup with nothing too complicated; can't think of any reason offhand that slapd would need to have more than 1024 files open.
Best,
Aaron
Aaron Bennett Manager of Systems Administration Clark University ITS W:508.793.7315
openldap-technical@openldap.org
-
Aaron Bennett -
Aaron Richton -
Liam Gretton -
Turbo Fredriksson -
Ulrich Windl