openldap-software August 2009

openldap-software@openldap.org

27 participants
29 discussions

Possible linux tuning for high/constant load connections
by Rodrigo Costa 25 Aug '09

25 Aug '09

OpenLdap software community, I would like to request your experience for some issue I'm facing trying to use jmeter for testing openldap load test. I made a script that keeps around 10 simultaneous thread connections with a ramp time of 1 second. I also use a csv file with the records filters so the bind+search+unbind always pass by all records existent in the DB. The issue I'm facing is that after sometime I see some connections being refused(no bind) but after sometime the connections start to be accepted again. Analyzing the logs and everything else I could notice the following things happening in the system : 1) I see that all connections are accept by the openldap machine and then the TIME_WAIT connections start to increase(what make sense); [root@brtldp11 ~]# netstat -an --protocol=inet -t|wc -l 11636 (around this) 2) After normally the number of connections(ESTABLISHED, TIME_WAIT, etc) increase to values normally over 12,000 then it start to reduce; 3) When these connections start to reduce then I start to see after sometime in the output error for jmeter connections ; <sample t="177" lt="0" ts="1251214974190" s="false" lb="LDAP Bind" rc="800" rm="javax.naming.CommunicationException: 10.142.15.170:389 [Root exception is java.net.ConnectException: Cannot assign requested address]" tn="Thread Group1 1-2" dt="text" by="409"> <responseData class="java.lang.String"><ldapanswer><operation><opertype>bind</opertype> <baseobj>ou=CONTENT,o=domain,c=fr</baseobj> <binddn>cn=admin,ou=CONTENT,o=domain,c=fr</binddn> <connectionTO>3000</connectionTO> </operation> <responsecode>800</responsecode> <responsemessage>javax.naming.CommunicationException: 10.142.15.170:389 [Root exception is java.net.ConnectException: Cannot assign requested address]</responsemessage> </ldapanswer> </responseData> </sample> 4) Also start to see at syslog(LOG4) for openldap(loglevel 2) messages like below : Aug 25 12:42:59 brtldp11 slapd[2720]: connection_read(27): no connection! Aug 25 12:42:59 brtldp11 slapd[2720]: connection_read(43): no connection! Aug 25 12:42:59 brtldp11 slapd[2720]: connection_read(28): no connection! Aug 25 12:42:59 brtldp11 slapd[2720]: connection_read(40): no connection! Aug 25 12:43:00 brtldp11 slapd[2720]: connection_read(37): no connection! Aug 25 12:43:00 brtldp11 slapd[2720]: connection_read(30): no connection! Aug 25 12:43:01 brtldp11 slapd[2720]: connection_read(23): no connection! Aug 25 12:43:02 brtldp11 slapd[2720]: connection_read(36): no connection! Aug 25 12:43:02 brtldp11 slapd[2720]: connection_read(41): no connection! 5) The netstat results decrease and all start over again until behavior follows all steps again. [root@brtldp11 ~]# netstat -an --protocol=inet -t|wc -l 9217 I tried to check all tuning parameter I knew and I could not find explanation. Some parameters are : [root@brtldp11 ~]# cat /proc/sys/net/ipv4/ip_local_port_range 32768 61000 [root@brtldp11 ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 196607 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 65535 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 196607 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited I can be sure the test is trying to behave as expected since if I check the ESTABLISHED connections I can see the simultaneous connections I expect. Like : [root@brtldp11 ~]# grep ESTA* log1.txt |wc -l 9 Like I configured in the test for around 10 simultaneous connections with 1 second ramp. This was not supposed to be a so high load test but this strange behavior is happening. Even with this small number of connections I have this issue and then time by time there are some connections refused. Also when I see this situation I try to call ldapsearch and I saw : [root@brtldp12 jakarta-jmeter-2.3.4]# [root@brtldp12 jakarta-jmeter-2.3.4]# time ldapsearch -LLL -x -D "cn=admin,ou=CONTENT,o=domain,c=fr" -w secret -b "ou=CONTENT,o=domain,c=fr" -H ldap://10.142.15.170:389 'pnnumber=+554184096814' ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1) real 0m0.059s user 0m0.000s sys 0m0.023s Running after some seconds I can recover data from this query. Any ideas? Can this be some openldap configuration? Anyone already configured some heavy load test and OS tuning for openldap? Best Regards, Rodrigo.

1 0

Re: tlsverifyclient security implications
by Josh.Mullis＠cox.com 24 Aug '09

24 Aug '09

..."If the client does not send a certificate, it can still connect." Does that mean that traffic is still encrypted if a certificate is not used? ----- Original Message ----- From: Emmanuel Dreyfus <manu(a)netbsd.org> To: Mullis, Josh (CCI-Atlanta); openldap-software(a)openldap.org <openldap-software(a)openldap.org> Sent: Sun Aug 23 02:59:05 2009 Subject: Re: tlsverifyclient security implications Josh Mullis <josh.mullis(a)cox.com> wrote: > What are the security implications concerning the following setting in > slapd.conf: > tlsverifyclient allow As far as I understand, if the client sends a certificate, then slapd can use it to map client to a LDAP DN, like this: authz-regexp cn=foo uid=foo,dc=example,dc=net If the client does not send a certificate, it can still connect. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz manu(a)netbsd.org

7 8

tlsverifyclient security implications
by Josh Mullis 22 Aug '09

22 Aug '09

Hello all, Having a hard time finding the answer to this question... What are the security implications concerning the following setting in slapd.conf: tlsverifyclient allow I see a lot of openldap/tls related howto's stating to use this setting. Thanks for any help. -Josh

2 1

Re: OpenLDAP Overlay modules load problem
by Matthew Backes 20 Aug '09

20 Aug '09

> After that I configure slapd.conf to load denyop overlay module but > slapd can not load it .I have not seen any file name denyop or > similar one in file sysem.Where is a path of openldap overlay > moudules? Other issues aside, why are you trying to use denyop? It's rather obsolete and was only designed to be a tutorial for how to write a module. In that regard it is now useless as it predates the dynamic configuration system. See the modules in the servers/slapd/ overlays directory for more up to date examples. To accomplish the same deny-functionality, see the restrict option for slapd.conf, or olcRestrict for back-config. Matthew Backes Symas Corporation mbackes(a)symas.com

1 0

Re: Using replication on RedHat servers
by Pierre Laporte 19 Aug '09

19 Aug '09

On Tue, Aug 18, 2009 at 9:50 PM, Rick Stevens <rps2(a)socal.rr.com> wrote: > > Did you install openldap-servers-overlays? I think syncprov is in that > RPM. > > Well, sice i installed the symas' rpm, everything related to openldap goes in /opt/symas/ and is built for openldap 2.4.16. I think the rpm you mention (openldap-servers-overlay) is the RedHat one, so i'm not sure it'd be compatible. On Tue, Aug 18, 2009 at 9:53 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote: > > Or it is already built into slapd, and no reason to module load it. > > I tought that at the beggining, I just can't include the syncprov module. When I add the line "overlay syncprov" in my slapd.conf file (without loading syncprov.la), slapd fails to start and i get this error message : Aug 6 17:29:21 localhost slapd[25880]: overlay "syncprov" not found

2 1

Re: syncrepl replication taking too long(not sync)
by Rodrigo Costa 19 Aug '09

19 Aug '09

Buchan, I have a 32bit system so I can only allocate 3GB for slapd. The machines have 12GB each but I can only allocate 3GB for a single process in a 32bit system with CentOS5.3. The tuning is done based on memory constraints and I think it should be more than enough since the traffic I have is low; only DB is a little large(4 million entrances). In the end of the day I think that I cannot have dncache with a smaller number than records in your DB. This means I cannot have a DB that cannot have all dncache allocated in memory. I was wondering if this is the case so I will about to use search or replication. The DB can run only in single system with these restrictions. Thanks, Rodrigo. Buchan Milne wrote: > On Tuesday, 18 August 2009 21:30:31 Rodrigo Costa wrote: > >> openldap software community, >> >> I'm facing some difficulties to have database synchronized with >> syncrepl. I'm running the latest openldap 2.4.17 version which after >> these issues I compiled with gdb. >> >> I have a DB(divided really in 2 DBs) where each one has around 4 million >> entrances. Based in memory limitations I have a dncachesize configured >> with around 3000000, or smaller than the maximum number of entrances in >> DBs. >> >> I loaded both server with all indexes and the same data. Starting both >> there isn't any need for syncrepl(thread from slapd) to make any search >> and then both mirrors are in sync and consuming each other. If a new >> entrance is create the other consumes since both are listening right on >> when it happens. >> >> If I stop one mirror and create even small number of entrances in the >> other, like 10, when I try to start the other provider the syncrepl >> enters in conventional syncrepl replication which search the DB for >> synchronization. >> >> This never ends causing mirrors not in synchronization. What I can see is : >> >> 1) Stop the Second mirror, like for slapcat(calling second and first as >> reference); >> 2) Add a few entrances in First mirror(kept on-line); >> 3) Second mirror start again after First mirror had some new entrances >> added by normal operation; >> 4) Syncrepl in second mirror enters in the conventional syncrepl >> replication since it detects that something is different between mirrors; >> 5) Until dncache is not filled the First mirror slapd cpu consumption is >> below 100%(around 50%) and search happens in a good manner since monitor >> shows it; >> 6) After dncache is filled(oscillates above 3mi) the First mirror cpu >> consumption enter in 100% consumption, oscillating between 98% to 102%; >> 7) The search never ends and then systems are never in sync. Cpu is >> permanently in high consumption, almost always in 100%. >> >> I let days this process running and I could see only a one or two >> entrances in sync. By the CPU looks like something is hanging the search >> where some loop is keeping the thread consuming one full cpu processing. >> >> I could collect some GDB information which I'm sending attached. Not >> sure how to interpret this overlay_walk. >> >> The idea is to stop one mirror for backup releasing this task from the >> primary server. For this replication would need to happen. >> >> Your comments are very welcome. >> > > You have provided absolutely no configuration information. There may well be > other explanations for this behaviour than the dncachesize. I can think of at > least two. > > You also haven't provided information on the systems you are using. E.g., you > may be trying on systems with too little memory (e.g., <1GB), which might be > totally inadequate for the amount of data you have. > > Regards, > Buchan > > >

2 1

Using replication on RedHat servers
by Pierre Laporte 19 Aug '09

19 Aug '09

Hello, I am setting up several openldap servers, and would like them to be synchronized, multi-master style. Configuration : Version: 2.4.16 packaged by Symas OS: RedHat 5.3 I have tried following the Admin guide but for some reason, I just can't include the syncprov module. When I add the line "overlay syncprov" in my slapd.conf file, slapd fails to start and i get this error message : Aug 6 17:29:21 localhost slapd[25880]: overlay "syncprov" not found I found some information about this bug at https://bugzilla.redhat.com/show_bug.cgi?id=466937 , but i'm not sure whether it applies for symas'package : "syncprov is compiled statically into slapd daemon to keep backward compatibility" I tried to skip this part, but as expected, replication doesn't work. I keep getting this message in logs. Aug 6 17:33:10 localhost slapd[25901]: slap_global_control: unrecognized control: 1.3.6.1.4.1.4203.1.9.1.1 According to a thread in this mailing list, this error appears when syncprov is not loaded in slapd.conf ( http://www.openldap.org/lists/openldap-software/200605/msg00277.html) So here I am, stuck. I can't find syncprov.la in my module folder (/opt/symas/lib64/openldap), and I can't find any way to solve this problem. Did I miss something ? How can i get this to work ? Any advice would be very appreciated. -- Pierre

3 5

syncrepl replication taking too long(not sync)
by Rodrigo Costa 19 Aug '09

19 Aug '09

openldap software community, I'm facing some difficulties to have database synchronized with syncrepl. I'm running the latest openldap 2.4.17 version which after these issues I compiled with gdb. I have a DB(divided really in 2 DBs) where each one has around 4 million entrances. Based in memory limitations I have a dncachesize configured with around 3000000, or smaller than the maximum number of entrances in DBs. I loaded both server with all indexes and the same data. Starting both there isn't any need for syncrepl(thread from slapd) to make any search and then both mirrors are in sync and consuming each other. If a new entrance is create the other consumes since both are listening right on when it happens. If I stop one mirror and create even small number of entrances in the other, like 10, when I try to start the other provider the syncrepl enters in conventional syncrepl replication which search the DB for synchronization. This never ends causing mirrors not in synchronization. What I can see is : 1) Stop the Second mirror, like for slapcat(calling second and first as reference); 2) Add a few entrances in First mirror(kept on-line); 3) Second mirror start again after First mirror had some new entrances added by normal operation; 4) Syncrepl in second mirror enters in the conventional syncrepl replication since it detects that something is different between mirrors; 5) Until dncache is not filled the First mirror slapd cpu consumption is below 100%(around 50%) and search happens in a good manner since monitor shows it; 6) After dncache is filled(oscillates above 3mi) the First mirror cpu consumption enter in 100% consumption, oscillating between 98% to 102%; 7) The search never ends and then systems are never in sync. Cpu is permanently in high consumption, almost always in 100%. I let days this process running and I could see only a one or two entrances in sync. By the CPU looks like something is hanging the search where some loop is keeping the thread consuming one full cpu processing. I could collect some GDB information which I'm sending attached. Not sure how to interpret this overlay_walk. The idea is to stop one mirror for backup releasing this task from the primary server. For this replication would need to happen. Your comments are very welcome. Regards, Rodrigo.

3 2

Re: syncrepl replication taking too long(not sync)
by Rodrigo Costa 18 Aug '09

18 Aug '09

Quanah, I forget to mention that even if I stop the second mirror is stopped again the first mirror cpu still being consumed. I was expecting that after the search from the second mirror is stopped the cpu consumption would be finished. Not sure if with dncache unlimited, even all other caches equal to 0, if the memory will be enough. Looks like memory needs to be enough to handle all dncache for DBs in system. Thanks, Rodrigo. Quanah Gibson-Mount wrote: > --On Tuesday, August 18, 2009 2:13 PM -0700 Rodrigo Costa > <rlvcosta(a)yahoo.com> wrote: > >> Quanah, >> >> The issue is that if I do not limit dncache the slapd crashes since >> after sometime it cannot allocate more system memory(malloc). Only >> dncache in two DBs can consume all memory possible to allocate. > > See below. You obviously need to better tune other things. > >>> Limit your memory consumption in other ways if you need >>> to (Smaller BDB cache, smaller entry cache, smaller idlcache). >>> Limiting the dncachesize is pretty much suicidal in any large db. > > --Quanah > > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration > >

1 0

Re: syncrepl replication taking too long(not sync)
by Rodrigo Costa 18 Aug '09

18 Aug '09

Quanah, The issue is that if I do not limit dncache the slapd crashes since after sometime it cannot allocate more system memory(malloc). Only dncache in two DBs can consume all memory possible to allocate. Thanks, Rodrigo. Quanah Gibson-Mount wrote: > --On Tuesday, August 18, 2009 1:30 PM -0700 Rodrigo Costa > <rlvcosta(a)yahoo.com> wrote: > >> openldap software community, >> >> I'm facing some difficulties to have database synchronized with >> syncrepl. I'm running the latest openldap 2.4.17 version which after >> these issues I compiled with gdb. >> >> I have a DB(divided really in 2 DBs) where each one has around 4 million >> entrances. Based in memory limitations I have a dncachesize configured >> with around 3000000, or smaller than the maximum number of entrances in >> DBs. > > Don't use a limited dncachesize. With 2.4.18, you'll be able to set > it to zero (unlimited), and that will be the default, to match the old > 2.3 behavior. Limit your memory consumption in other ways if you need > to (Smaller BDB cache, smaller entry cache, smaller idlcache). > Limiting the dncachesize is pretty much suicidal in any large db. > > --Quanah > > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration > >

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-software August 2009