Hello list,
Although the subject suggests a fairly know issue, i cannot seem to understand the cause.
Log file presents
Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open /etc/hosts.allow: Too many open files Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open /etc/hosts.deny: Too many open files
, but
proxyldap1:~# cat /proc/sys/fs/file-max 100000 proxyldap1:~# cat /proc/sys/fs/file-nr 1088 0 100000 proxyldap1:~# lsof -p `pidof slapd`| wc -l 460 proxyldap1:~# su -c "ulimit -n" openldap 8192
slapd is version 2.4.17-2 (Debian Unstable) and is running on a xen vm, using 64bit kernel 2.6.26-2-xen-amd64 in a Debian Lenny system.
Have i overlooked any aspect?
Thank you in advance,
Hugo Monteiro.
--On Thursday, April 22, 2010 1:10 PM +0100 Hugo Monteiro hugo.monteiro@fct.unl.pt wrote:
Hello list,
Although the subject suggests a fairly know issue, i cannot seem to understand the cause.
Log file presents
Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open /etc/hosts.allow: Too many open files Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open /etc/hosts.deny: Too many open files
slapd is version 2.4.17-2 (Debian Unstable) and is running on a xen vm, using 64bit kernel 2.6.26-2-xen-amd64 in a Debian Lenny system.
http://www.openldap.org/lists/openldap-technical/200806/msg00084.html
Generally caused by linking against tcp wrappers, although I thought 64-bit OS shouldn't hit that issue.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On 04/22/2010 06:04 PM, Quanah Gibson-Mount wrote:
--On Thursday, April 22, 2010 1:10 PM +0100 Hugo Monteiro hugo.monteiro@fct.unl.pt wrote:
Hello list,
Although the subject suggests a fairly know issue, i cannot seem to understand the cause.
Log file presents
Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open /etc/hosts.allow: Too many open files Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open /etc/hosts.deny: Too many open files
slapd is version 2.4.17-2 (Debian Unstable) and is running on a xen vm, using 64bit kernel 2.6.26-2-xen-amd64 in a Debian Lenny system.
http://www.openldap.org/lists/openldap-technical/200806/msg00084.html
Generally caused by linking against tcp wrappers, although I thought 64-bit OS shouldn't hit that issue.
--Quanah
Hello Quanah,
I also saw that thread, and in fact the debian packages are built with --enable-wrappers, but i also assumed it wouldn't be an issue on a 64bit machine. I'll recompile without tcp wrapper support and see how it goes.
Regards,
Hugo Monteiro.
On Thursday, 22 April 2010 13:10:03 Hugo Monteiro wrote:
Hello list,
Although the subject suggests a fairly know issue, i cannot seem to understand the cause.
Log file presents
Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open /etc/hosts.allow: Too many open files Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open /etc/hosts.deny: Too many open files
, but
proxyldap1:~# cat /proc/sys/fs/file-max 100000 proxyldap1:~# cat /proc/sys/fs/file-nr 1088 0 100000 proxyldap1:~# lsof -p `pidof slapd`| wc -l 460
And 'grep files /proc/`pidof slapd`/limits' ?
proxyldap1:~# su -c "ulimit -n" openldap 8192
slapd is version 2.4.17-2 (Debian Unstable) and is running on a xen vm, using 64bit kernel 2.6.26-2-xen-amd64 in a Debian Lenny system.
Have i overlooked any aspect?
The preceding lines, showing the ACCEPT lines with the fd number, prior to the "Too many open files" error would have been useful.
I recently tested (in the "max open files" thread from March), and found that with tcp_wrappers, I could get more than 4000 connections (with ulimit -n set to 4096).
You may want to review some Debian bug reports, specifically this one, which also has a script you can use to test with:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=378261
Regards, Buchan
On 04/23/2010 11:55 AM, Buchan Milne wrote:
And 'grep files /proc/`pidof slapd`/limits' ?
Bingo.
proxyldap1:~# grep files /proc/`pidof slapd`/limits Max open files 1024 1024 files
Now this i don't undestand. Howcome is this value set to 1024 while i defined 8192 for nofile in /etc/security/limits ?
So now i added ulimit -n 4096 to /etc/default/slapd and i restarted the server. I now get a proper value in /proc/`pidof slapd`/limits.
Now it's wait and see, but i'm feeling confident.
Thank you for your help.
Hugo Monteiro.
Hello,
don't want to discourage you, but my slapd (2.4.11 on Debian Lenny) still has problems with more than 1024 files.
I have -- ulimit -n 4096 in /etc/default/slapd -- checked the output of /proc/`pidof slapd`/limits -- downloaded the source package from Debian (version 2.4.11), did a grep for FD_SETSIZE and found that it is set to 4096 -- built a new package from these sources with configure options: --disable-tcpwrapper
In a loop from a python script, I then connect, bind, search the ldap server and keep all the connections open, like so:
... def ldap_connection_func(): global n l = ldap.initialize("ldaps://ldap...") l.simple_bind_s(username, password) l.search_s('dc=haiti,...', ldap.SCOPE_SUBTREE, 'objectclass=*') n.append(l)
for x in range(0,1100): ldap_connection_func() ...
I see the number of open files of the slapd process rise to about 1080, then all connections opened from the script are closed while slapd keeps running.
We now will try to set the idletimeout parameter to some sensible value
- Hopefully, this will not lead to trouble with syncrepl.
Klemens
On Fri, 2010-04-23 at 16:59 +0100, Hugo Monteiro wrote:
On 04/23/2010 11:55 AM, Buchan Milne wrote:
And 'grep files /proc/`pidof slapd`/limits' ?
Bingo.
proxyldap1:~# grep files /proc/`pidof slapd`/limits Max open files 1024 1024 files
Now this i don't undestand. Howcome is this value set to 1024 while i defined 8192 for nofile in /etc/security/limits ?
So now i added ulimit -n 4096 to /etc/default/slapd and i restarted the server. I now get a proper value in /proc/`pidof slapd`/limits.
Now it's wait and see, but i'm feeling confident.
Thank you for your help.
Hugo Monteiro.
On Friday, 30 April 2010 11:16:09 Klemens Kittan wrote:
Hello,
don't want to discourage you, but my slapd (2.4.11 on Debian Lenny) still has problems with more than 1024 files.
I have -- ulimit -n 4096 in /etc/default/slapd -- checked the output of /proc/`pidof slapd`/limits -- downloaded the source package from Debian (version 2.4.11), did a grep for FD_SETSIZE and found that it is set to 4096 -- built a new package from these sources with configure options: --disable-tcpwrapper
In a loop from a python script, I then connect, bind, search the ldap server and keep all the connections open, like so:
... def ldap_connection_func(): global n l = ldap.initialize("ldaps://ldap...") l.simple_bind_s(username, password) l.search_s('dc=haiti,...', ldap.SCOPE_SUBTREE, 'objectclass=*') n.append(l)
for x in range(0,1100): ldap_connection_func() ...
What was the ulimit -n in the shell from which you ran this python script? If you are using a single process to generate the load to the server, this process obviously also needs to be able to open sufficient files.
I see the number of open files of the slapd process rise to about 1080, then all connections opened from the script are closed while slapd keeps running.
It is funny that slapd opened more than 1024 files then, as it should either have stopped at less than 1024 fd's (as it has other files open), or the script should have reached 1100 connections, or your script is running with a file descriptor limit.
We now will try to set the idletimeout parameter to some sensible value
It looks like your slapd file descriptor limit is now taking effect, you just need to test better, and/or interpret the results better ....
Regards, Buchan
Hello,
Hello,
don't want to discourage you, but my slapd (2.4.11 on Debian Lenny) still has problems with more than 1024 files.
I have -- ulimit -n 4096 in /etc/default/slapd -- checked the output of /proc/`pidof slapd`/limits -- downloaded the source package from Debian (version 2.4.11), did a grep for FD_SETSIZE and found that it is set to 4096 -- built a new package from these sources with configure options: --disable-tcpwrapper
In a loop from a python script, I then connect, bind, search the ldap server and keep all the connections open, like so:
... def ldap_connection_func(): global n l = ldap.initialize("ldaps://ldap...") l.simple_bind_s(username, password) l.search_s('dc=haiti,...', ldap.SCOPE_SUBTREE, 'objectclass=*') n.append(l)
for x in range(0,1100): ldap_connection_func() ...
What was the ulimit -n in the shell from which you ran this python script? If you are using a single process to generate the load to the server, this process obviously also needs to be able to open sufficient files.
Ah, I see. I just have adjusted the ulimit size on the server side (to 4096). I never thought about the client's side, where I started the script. Here, the "ulimit -n" is 1024. Indeed, if the client tries to open less than 1024 files, everything is fine. Even if I start the script 3 times (each in a separate shell) everything runs as expected. The clients and the server do not have a problem, in detail, the server opened more than 1024 files at the same time successfully.
I see the number of open files of the slapd process rise to about 1080, then all connections opened from the script are closed while slapd keeps running.
It is funny that slapd opened more than 1024 files then, as it should either have stopped at less than 1024 fd's (as it has other files open), or the script should have reached 1100 connections, or your script is running with a file descriptor limit.
We now will try to set the idletimeout parameter to some sensible value
It looks like your slapd file descriptor limit is now taking effect, you just need to test better, and/or interpret the results better ....
Yes, you are right. I should set ulimit in the /etc/default/slapd and it will work.
Regards, Buchan
Regards, Klemens
openldap-software@openldap.org
-
Buchan Milne -
Hugo Monteiro -
Klemens Kittan -
Quanah Gibson-Mount