Hello,
the problem is, that many clients generate more than 1024 concurrent sockets to the LDAP service which block the remaining incoming sockets.
I already do all the possible changes to the server (ulimit, sysctl, etc) without a solution. All the incoming connections stop at the 1024 concurrent sockets. The only solution is restarting slapd.
The version of openldap is 2.4.11.
In /var/log/syslog I found the following entry:
Mar 1 14:45:15 ldap1 slapd[25320]: warning: /etc/hosts.allow, line 19: cannot open /etc/hosts.allow: Too many open files
cat /proc/sys/fs/file-max: 203609
cat /proc/<slapd pid>/limits: Max open files 4096 4096 files
Regards, Klemens
Klemens Kittan kittan@cs.uni-potsdam.de writes:
Hello,
the problem is, that many clients generate more than 1024 concurrent sockets to the LDAP service which block the remaining incoming sockets.
I already do all the possible changes to the server (ulimit, sysctl, etc) without a solution. All the incoming connections stop at the 1024 concurrent sockets. The only solution is restarting slapd.
The version of openldap is 2.4.11.
In /var/log/syslog I found the following entry:
Mar 1 14:45:15 ldap1 slapd[25320]: warning: /etc/hosts.allow, line 19: cannot open /etc/hosts.allow: Too many open files
Do you really need tcp wrapper support?
-Dieter
I had similar issue a while back with like the 2.2 or 2.3 series, and part of the database was corrupt. The database couldn't respond to xyz search and I assume would leave the sockets hanging thus use them all up. I ended up rebuilding the database.
Quoting Dieter Kluenter dieter@dkluenter.de:
Klemens Kittan kittan@cs.uni-potsdam.de writes:
Hello,
the problem is, that many clients generate more than 1024 concurrent sockets to the LDAP service which block the remaining incoming sockets.
I already do all the possible changes to the server (ulimit, sysctl, etc) without a solution. All the incoming connections stop at the 1024 concurrent sockets. The only solution is restarting slapd.
The version of openldap is 2.4.11.
In /var/log/syslog I found the following entry:
Mar 1 14:45:15 ldap1 slapd[25320]: warning: /etc/hosts.allow, line 19: cannot open /etc/hosts.allow: Too many open files
Do you really need tcp wrapper support?
-Dieter
-- Dieter Klünter | Systemberatung http://dkluenter.de GPG Key ID:8EF7B6C6 53°37'09,95"N 10°08'02,42"E
On Wed, 10 Mar 2010, Klemens Kittan wrote:
Mar 1 14:45:15 ldap1 slapd[25320]: warning: /etc/hosts.allow, line 19: cannot open /etc/hosts.allow: Too many open files
cat /proc/sys/fs/file-max: 203609
cat /proc/<slapd pid>/limits: Max open files 4096 4096 files
Sounds like you're mostly on the right track, but I didn't hear mention of compiling with a suitable OPENLDAP_FD_SETSIZE. Are your CPPFLAGS set accordingly?
On Fri, Mar 12, 2010 at 7:10 AM, Aaron Richton richton@nbcs.rutgers.eduwrote:
On Wed, 10 Mar 2010, Klemens Kittan wrote:
Mar 1 14:45:15 ldap1 slapd[25320]: warning: /etc/hosts.allow, line 19:
cannot open /etc/hosts.allow: Too many open files
cat /proc/sys/fs/file-max: 203609
cat /proc/<slapd pid>/limits: Max open files 4096 4096 files
Sounds like you're mostly on the right track, but I didn't hear mention of compiling with a suitable OPENLDAP_FD_SETSIZE. Are your CPPFLAGS set accordingly?
Klemens,
Few weeks ago I had a similar issue, I found this thread very useful: http://www.sunmanagers.org/pipermail/summaries/2005-March/006226.html but at the end the issue seemed to come from avahi daemon. I'm not familiar with avahi but the config had:
[rlimits] #rlimit-as= rlimit-core=0 rlimit-data=4194304 rlimit-fsize=0 rlimit-nofile=30 rlimit-stack=4194304 rlimit-nproc=3
The machine restarted and started avahi(it was stopped before but not disabled) and when ldap started to get some connections I received the same output: Feb 18 00:49:05 ldap01 slapd[3704]: warning: cannot open /etc/hosts.deny: Too many open files
Hope that helps.
On Fri, Mar 12, 2010 at 7:10 AM, Aaron Richton richton@nbcs.rutgers.edu wrote: On Wed, 10 Mar 2010, Klemens Kittan wrote:
Mar 1 14:45:15 ldap1 slapd[25320]: warning: /etc/hosts.allow, line 19: cannot open /etc/hosts.allow: Too many open files cat /proc/sys/fs/file-max: 203609 cat /proc/<slapd pid>/limits: Max open files 4096 4096 files Sounds like you're mostly on the right track, but I didn't hear mention of compiling with a suitable OPENLDAP_FD_SETSIZE. Are your CPPFLAGS set accordingly?Klemens,
Few weeks ago I had a similar issue, I found this thread very useful: http://www.sunmanagers.org/pipermail/summaries/2005-March/006226.html but at the end the issue seemed to come from avahi daemon. I'm not familiar with avahi but the config had:
[rlimits] #rlimit-as= rlimit-core=0 rlimit-data=4194304 rlimit-fsize=0 rlimit-nofile=30 rlimit-stack=4194304 rlimit-nproc=3
The machine restarted and started avahi(it was stopped before but not disabled) and when ldap started to get some connections I received the same output: Feb 18 00:49:05 ldap01 slapd[3704]: warning: cannot open /etc/hosts.deny: Too many open files
I looked at that thread. They recommend exactly the things I tried already, e.g. setting ulimit in the startup script. I checked that with cat /proc/<slapd pid>/limits. Nevertheless the LDAP stopped responding after 1024 open connections. I didn't change "idletimeout" in slapd.conf for I found the follwing in the LDAP documentation (and we use syncrepl):
"... Caution: This is a server wide value so that all bind connections are affected by it. If this server is either a replication consumer (using the syncrepl directive with a type value of refreshAndPersist) or a provider (using the overlay syncprov directive with a one of more consumers with a type of refreshAndPersist) then it is highly likely that these links will remain idle for prolonged periods of time. Extreme caution should be used when defining the idletimeout directive in either of these conditions because the net effect may be to change such replication connections into type refreshOnly which may not be a welcome side effect..."
On my systems the avahi daemon is not installed.
On Wed, 10 Mar 2010, Klemens Kittan wrote:
Mar 1 14:45:15 ldap1 slapd[25320]: warning: /etc/hosts.allow, line 19: cannot open /etc/hosts.allow: Too many open files
cat /proc/sys/fs/file-max: 203609
cat /proc/<slapd pid>/limits: Max open files 4096 4096 files
Sounds like you're mostly on the right track, but I didn't hear mention of compiling with a suitable OPENLDAP_FD_SETSIZE. Are your CPPFLAGS set accordingly?
I use the OpenLDAP version from the distribution (Debian 5.0.4), so I can more easily install updates. If there is no other solution, I will try it.
--On Tuesday, March 16, 2010 8:34 AM +0100 Klemens Kittan kittan@cs.uni-potsdam.de wrote:
Sounds like you're mostly on the right track, but I didn't hear mention of compiling with a suitable OPENLDAP_FD_SETSIZE. Are your CPPFLAGS set accordingly?
I use the OpenLDAP version from the distribution (Debian 5.0.4),
There's your first mistake.
http://www.openldap.org/faq/data/cache/1456.html
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On Friday, 12 March 2010 16:10:01 Aaron Richton wrote:
On Wed, 10 Mar 2010, Klemens Kittan wrote:
Mar 1 14:45:15 ldap1 slapd[25320]: warning: /etc/hosts.allow, line 19: cannot open /etc/hosts.allow: Too many open files
cat /proc/sys/fs/file-max: 203609
cat /proc/<slapd pid>/limits: Max open files 4096 4096 files
Sounds like you're mostly on the right track, but I didn't hear mention of compiling with a suitable OPENLDAP_FD_SETSIZE. Are your CPPFLAGS set accordingly?
I ran into this at some stage, and built some binaries (for RHEL3 I think) with a higher OPENLDAP_FD_SETSIZE, however I am sure the discussion about this concluded that on modern linux distros, this is no longer necessary, due to the use of epoll. While it might still apply on other non-Linux-2.6 platforms (doesn't Solaris have it's own API for similar purposes?), this should not be a concern for anyone one a non-ancient Linux installation.
This bug report on Debian which relates to the same issue, seems to have come to the same conclusion:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=378261
On my laptop, running Mandriva 2010.0 x86_64 with the Mandriva OpenLDAP package I ran the perl script in that bug report after increasing the file size limit for OpenLDAP:
[root@tiger ~]# grep ^MAX /etc/sysconfig/ldap MAXFILES=4096
which resulted in the correct limit being applied to the slapd pid: [root@tiger ~]# grep files /proc/`pidof slapd`/limits Max open files 4096 4096 files
and the limit for the shell from which I ran the script:
[root@tiger ~]# ulimit -n 4096
[root@tiger ~]# perl /home/bgmilne/bin/ldapportcheck.pl [...] 4091 4092 Couldn't create socket 4093: Too many open files at /home/bgmilne/bin/ldapportcheck.pl line 11.
slapd had been happy to have more than 4000 files open:
Mar 19 11:05:13 tiger slapd[22716]: conn=6086 fd=4083 ACCEPT from IP=127.0.0.1:35432 (IP=0.0.0.0:389)
So, my conclusion is: 1)You're doing something wrong in your testing 2)For some reason Ubuntu is not biulding with epoll support 3)There is some obscure bug in OpenLDAP that results in file limit not being adhered to even when built with epoll support
Regards, Buchan
openldap-software@openldap.org
-
Aaron Richton -
Buchan Milne -
Dieter Kluenter -
Klemens Kittan -
omalleys＠msu.edu -
Quanah Gibson-Mount -
Tony G.