I'm using xinetd forwarding to allow a number of compute nodes that don't have a direct path to our ldap server to get forward on by a host that does. When running a highly parallel job that starts over 1K instances at the same time, I see all these xinetd instances also start up on my forwarding server and in fact they don't seem to go away, at not any time soon. Meanwhile back on my ldap server I see number of 'too many open files' errors in /var/log/messages and if I try to "su user" on one of the nodes I'll see it hang for awhile. I have bumped the number of open files very high on the ldap server and in fact:
[root@aicgateway ~]# cat /proc/sys/fs/file-nr 5610 0 201116
So out of a pool of 200K we're only using 5K.
Anyhow, I'm wondering if there are any special tricks to configuring the environment to handling this type of load on ldap OR should it be able to handle it the way I'm currently configured? Any special tuning hints? Any more info I can supply?
-mark
"Seger, Mark" mark.seger@hp.com writes:
I’m using xinetd forwarding to allow a number of compute nodes that don’t have a direct path to our ldap server to get forward on by a host that does. When running a highly parallel job that starts over 1K instances at the same time, I see all these xinetd instances also start up on my forwarding server and in fact they don’t seem to go away, at not any time soon. Meanwhile back on my ldap server I see number of ‘too many open files’ errors in /var/log/messages and if I try to “su user” on one of the nodes I’ll see it hang for awhile. I have bumped the number of open files very high on the ldap server and in fact:
[root@aicgateway ~]# cat /proc/sys/fs/file-nr
5610 0 201116
So out of a pool of 200K we’re only using 5K.
Anyhow, I’m wondering if there are any special tricks to configuring the environment to handling this type of load on ldap OR should it be able to handle it the way I’m currently configured? Any special tuning hints? Any more info I can supply?
ulimit -n, ulimit -Sn, ulimit -Hn
-Dieter
-----Original Message----- From: openldap-technical-bounces+mark.seger=hp.com@OpenLDAP.org [mailto:openldap-technical-bounces+mark.seger=hp.com@OpenLDAP.org] On Behalf Of Dieter Kluenter Sent: Saturday, April 03, 2010 2:18 AM To: openldap-technical@openldap.org Subject: Re: too many open files and over 1K xinetd running
"Seger, Mark" mark.seger@hp.com writes:
I’m using xinetd forwarding to allow a number of compute nodes that
don’t have
a direct path to our ldap server to get forward on by a host that
does. When
running a highly parallel job that starts over 1K instances at the
same time,
I see all these xinetd instances also start up on my forwarding server
and in
fact they don’t seem to go away, at not any time soon. Meanwhile back
on my
ldap server I see number of ‘too many open files’ errors in
/var/log/messages
and if I try to “su user” on one of the nodes I’ll see it hang for
awhile. I
have bumped the number of open files very high on the ldap server and
in fact:
[root@aicgateway ~]# cat /proc/sys/fs/file-nr
5610 0 201116
So out of a pool of 200K we’re only using 5K.
Anyhow, I’m wondering if there are any special tricks to configuring
the
environment to handling this type of load on ldap OR should it be able
to
handle it the way I’m currently configured? Any special tuning hints?
Any
more info I can supply?
ulimit -n, ulimit -Sn, ulimit -Hn
[root@aicgateway ~]# ulimit -n 16535 [root@aicgateway ~]# ulimit -Sn 16535 [root@aicgateway ~]# ulimit -Hn 32768 [root@aicgateway ~]#
Does this help? -mark
-Dieter
-- Dieter Klünter | Systemberatung sip: +49.40.20932173 http://www.dpunkt.de/buecher/2104.html GPG Key ID:8EF7B6C6
"Seger, Mark" mark.seger@hp.com writes:
"Seger, Mark" mark.seger@hp.com writes:
more info I can supply?
ulimit -n, ulimit -Sn, ulimit -Hn
[root@aicgateway ~]# ulimit -n 16535 [root@aicgateway ~]# ulimit -Sn 16535 [root@aicgateway ~]# ulimit -Hn 32768 [root@aicgateway ~]#
Does this help?
This should be more than sufficient, the default usually is 1024
-Dieter
On Fri, Apr 02, 2010 at 04:31:10PM +0000, Seger, Mark wrote:
I'm using xinetd forwarding to allow a number of compute nodes that don't have a direct path to our ldap server to get forward on by a host that does. When running a highly parallel job that starts over 1K instances at the same time, I see all these xinetd instances also start up on my forwarding server and in fact they don't seem to go away, at not any time soon. Meanwhile back on my ldap server I see number of 'too many open files' errors in /var/log/messages and if I try to "su user" on one of the nodes I'll see it hang for awhile. I have bumped the number of open files very high on the ldap server and in fact:
[root@aicgateway ~]# cat /proc/sys/fs/file-nr 5610 0 201116
So out of a pool of 200K we're only using 5K.
Which version of OpenLDAP, and did you build it yourself? What OS, and what bulid options?
What I am wondering here is whether slapd is built in such a way that it is artificially limited to (say) 1024 usable file descriptors.
In any case, I think there is a better solution: use slapd instead of xinetd on your proxy host. This is likely to be much more efficient, and in some circumstances it can do connection-pooling. Here is an extract from 'man slapd-ldap':
The LDAP backend to slapd(8) is not an actual database; instead it acts as a proxy to forward incoming requests to another LDAP server. While processing requests it will also chase referrals, so that referrals are fully processed instead of being returned to the slapd client.
Sessions that explicitly Bind to the back-ldap database always create their own private connection to the remote LDAP server. Anonymous sessions will share a single anonymous connection to the remote server. For sessions bound through other mechanisms, all sessions with the same DN will share the same connection. This connection pooling strategy can enhance the proxy's efficiency by reducing the overhead of repeatedly making/breaking multiple connections.
Andrew
openldap-technical@openldap.org
-
Andrew Findlay -
Dieter Kluenter -
Seger, Mark