openldap-bugs January 2009

openldap-bugs@openldap.org

50 participants
201 discussions

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by rlvcosta＠yahoo.com 26 Jan '09

26 Jan '09

Quanah, In this specific test I reported in my previous e-mail it was used glibc. Now I tested with tcmalloc and the memory again was much more consumed and never released. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10652 ldap 18 0 2006m 1.8g 68m S 100 15.8 8:36.75 slapd Once it get this much memory it then stabilize but it is much more than before that was 385m. The scenario was : 1) Start a full ldapsearch for one specific dn(the DB has 4 dn's); 2) After sometime start a new full ldapsearch to behave as queries to DB when replication for LDIF backup is being done; 3) Export one of the queries to a file so speed verification can be done with grep and wc. After sometime the memory starts to be consumed in large chunks until it stopped in around 2G. Since one of the queries is being redirect to a file I can tail to have a visual rate of the entrances being returned by slapd. As the issue before after sometime the rate appears to get stuck and come in small 16 entrances chunks but only after some minutes. This behavior above appears to happen after the second ldapsearch fills its own dn cache even monitor always shows only one information as I could see. [root@brtldp11 ~]# date; grep -e '^pnnum*' /backup/temp.ldif |wc -l Mon Jan 26 21:43:31 BRST 2009 482432 [root@brtldp11 ~]# date; grep -e '^pnnum*' /backup/temp.ldif |wc -l Mon Jan 26 21:44:56 BRST 2009 482448 This behavior is similar as before with the initial issue with a single process. I just notice when slapd starts there are these threads : [root@brtldp12 ~]# ps -mu ldap PID TTY TIME CMD 10638 pts/2 00:00:00 slapd - - 00:00:00 - - - 00:00:00 - Then after the 2 queries are started I have : [root@brtldp12 ~]# ps -mu ldap PID TTY TIME CMD 10195 pts/2 00:44:22 slapd - - 00:00:00 - - - 00:00:00 - - - 00:26:09 - - - 00:18:13 - - - 00:00:00 - So 3 new threads. Looks like for some reason with multiples queries the issue with response getting stuck repeats. Even one ldapsearch is stopped the number of threads continues the same, memory continues allocated and the other search get with this slow rate. These outputs are using now tcmalloc but similar as with glibc. Regards, Rodrigo. --- On Mon, 1/26/09, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > From: Quanah Gibson-Mount <quanah(a)zimbra.com> > Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 > To: rlvcosta(a)yahoo.com, openldap-its(a)openldap.org > Date: Monday, January 26, 2009, 9:25 PM > --On Monday, January 26, 2009 3:18 PM -0800 Rodrigo Costa > <rlvcosta(a)yahoo.com> wrote: > > > Quanah and Howard, > > > > Sorry to come back. > > > > I tried to make 2 ldapsearch to the same DB > simultaneously, doing some > > stress test like dump the LDIF and queries coming. > > > > The slapd after sometime start to get more memory. By > what I could > > observe looks like new thread was started consuming > again the same memory > > as before so the slad now is consuming around the > double of memory as > > before. > > Are you also using an alternate malloc() instead of glibc? > I know you were > investigating this, just want to confirm you are using > tcmalloc or hoard > and not glibc. > > --Quanah > > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and > collaboration

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by quanah＠zimbra.com 26 Jan '09

26 Jan '09

--On Monday, January 26, 2009 3:18 PM -0800 Rodrigo Costa <rlvcosta(a)yahoo.com> wrote: > Quanah and Howard, > > Sorry to come back. > > I tried to make 2 ldapsearch to the same DB simultaneously, doing some > stress test like dump the LDIF and queries coming. > > The slapd after sometime start to get more memory. By what I could > observe looks like new thread was started consuming again the same memory > as before so the slad now is consuming around the double of memory as > before. Are you also using an alternate malloc() instead of glibc? I know you were investigating this, just want to confirm you are using tcmalloc or hoard and not glibc. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by rlvcosta＠yahoo.com 26 Jan '09

26 Jan '09

Quanah and Howard, Sorry to come back. I tried to make 2 ldapsearch to the same DB simultaneously, doing some stress test like dump the LDIF and queries coming. The slapd after sometime start to get more memory. By what I could observe looks like new thread was started consuming again the same memory as before so the slad now is consuming around the double of memory as before. I believe the cache would not grow beyond the boundary as like per thread since once cache is allocated it is never released. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10195 ldap 18 0 1913m 1.7g 68m S 100 15.0 31:25.71 slapd Then performance start to degrade again. Like before looks like the dump get stuck and comes in chunks. See example below : Mon Jan 26 21:05:16 BRST 2009 378752 Mon Jan 26 21:09:21 BRST 2009 378800 See that after around 4 minutes I have only 48 adds to the output LDIF from my first ldapsearch. Looks like if multiples queries are happening something still getting lost. Both queries are to the same DB and the monitor has : readOnly: FALSE olmBDBEntryCache: 1000 olmBDBDNCache: 1000953 olmBDBIDLCache: 1 Sorry to come back with this issue but I could detect this now. Thanks, Rodrigo. --- On Mon, 1/26/09, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > From: Quanah Gibson-Mount <quanah(a)zimbra.com> > Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 > To: rlvcosta(a)yahoo.com, openldap-its(a)openldap.org > Date: Monday, January 26, 2009, 8:53 PM > --On Monday, January 26, 2009 10:44 PM +0000 > rlvcosta(a)yahoo.com wrote: > > > Just some last questions. Is there any expectation for > a new release, for > > example 2.4.15, with this ITS included? Just to use a > formal release. > > 2.4.14 will have the fix for this ITS. There's no set > release date at this time, but hopefully soon. > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and > collaboration

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by quanah＠zimbra.com 26 Jan '09

26 Jan '09

--On Monday, January 26, 2009 10:44 PM +0000 rlvcosta(a)yahoo.com wrote: > Just some last questions. Is there any expectation for a new release, for > example 2.4.15, with this ITS included? Just to use a formal release. 2.4.14 will have the fix for this ITS. There's no set release date at this time, but hopefully soon. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by rlvcosta＠yahoo.com 26 Jan '09

26 Jan '09

Howard, Now it fits under my expectation. Under the initial load without the fix I had slapd not respecting any boundary and then all DN information being cached. In this way the first query would take longer since the information wasn't yet cached. At that time I had, without cache, a search for around 1million entrances in around 7 minutes and after cached this time reduce to around half. So now with the new load I was expecting something a little longer as the initial not cached since now slapd will not be able to cache all information and so the cache will be often overwritten. Now in my tests I obtained : First time: real 6m50.017s user 0m17.346s sys 0m9.159s Second time: real 6m47.790s user 0m17.032s sys 0m9.137s Third time: real 7m17.459s user 0m17.412s sys 0m9.320s This is more than perfect. Amazing how you guys could solve this problem so quickly. I saw sometimes the DN cache allocating a little more memory since it sometimes pass a little the boundary, like seen below : olmBDBEntryCache: 1000 olmBDBDNCache: 1000247 olmBDBIDLCache: 1 The maximum I saw was 1000268 but nothing considerable. In this way now slapd is keeping memory allocation stable as it is supposed to behave. The only comment is if dn cache is passed this memory allocated appears to never be released. The maximum size slapd obtained was : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10195 ldap 18 0 385m 266m 68m S 0 2.2 20:54.16 slapd This is more than perfect ! Thanks for all your quick solution of this problem. I believe many users of openLDAP will see their system more stable with this fix. Just some last questions. Is there any expectation for a new release, for example 2.4.15, with this ITS included? Just to use a formal release. By the tests I was expecting the cachefree as the cache size bringing some improvements, at least in a full query, since system would "cycle" pointer memory and overwritten all cache at once. What is the purpose of cachefree? Just to understand better and then tune later with some better idea about cache purposes. Best Regards, Rodrigo. --- On Mon, 1/26/09, Howard Chu <hyc(a)symas.com> wrote: > From: Howard Chu <hyc(a)symas.com> > Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 > To: rlvcosta(a)yahoo.com > Cc: openldap-its(a)OpenLDAP.org > Date: Monday, January 26, 2009, 6:00 PM > Rodrigo Costa wrote: > > Howard, > > > > I do not think the ITS is completely solved. Let me > try to explain better. > > > > I have the system respecting now the DN cachesize > boundary. At the same time I wasn't sure about the > cachefree idea and now I changed to 1, or equal the default > value. > > > > In any case what happens is : > > > > 1) I have the system doing sequential search in a > reasonable speed until the dncachesize is reached; > > 2) After dncchesize is reached the sequential search > hangs, or the output from the search get stuck for a long > time(I'm forwarding to a file so I do not have screen > actualization delays); > > Ex: > > [root@brtldp11 backup]# date; cat temp.ldif |grep -e > '^pnnum*' |wc -l > > Mon Jan 26 17:22:21 BRST 2009 > > 250016 > > [root@brtldp11 backup]# date; cat temp.ldif |grep -e > '^pnnum*' |wc -l > > Mon Jan 26 17:23:00 BRST 2009 > > 250016 > > See above that even almost after 1 minute passed not > new LDIF entrance was included. > > 3) Then query get stuck and looks like deterministic > it time by time only dumps 16 entrances and get stuck. This > behavior repeats and with these stuck that sometime gets > minutes the query never ends. > > > > olmBDBEntryCache: 884 > > olmBDBDNCache: 1000261 > > olmBDBIDLCache: 1 > > > > olmBDBEntryCache: 611 > > olmBDBDNCache: 1000261 > > olmBDBIDLCache: 1 > > > >Even with cachesize as 1000 and cachefree as 1, the > olmBDBEntryCache continues > to decrease, just slow now. > > > > I was expecting that a cachefree as 1000 would purge > all entrances and then > > cache again all 1000 new with in sequence always > answering the search. So > > the search would never hangs like it is happening now. > > I see. OK, this is now fixed in HEAD. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP > http://www.openldap.org/project/

1 0

Re: (ITS#5336)
by hyc＠symas.com 26 Jan '09

26 Jan '09

quanah(a)zimbra.com wrote: > A similar bug has been reported in the debian bug tracking system, with > some debugging information and reproducibility steps. > > <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=465915> Noting that this was closed in Debian as of 2.4.9. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by hyc＠symas.com 26 Jan '09

26 Jan '09

Rodrigo Costa wrote: > Howard, > > I do not think the ITS is completely solved. Let me try to explain better. > > I have the system respecting now the DN cachesize boundary. At the same time I wasn't sure about the cachefree idea and now I changed to 1, or equal the default value. > > In any case what happens is : > > 1) I have the system doing sequential search in a reasonable speed until the dncachesize is reached; > 2) After dncchesize is reached the sequential search hangs, or the output from the search get stuck for a long time(I'm forwarding to a file so I do not have screen actualization delays); > Ex: > [root@brtldp11 backup]# date; cat temp.ldif |grep -e '^pnnum*' |wc -l > Mon Jan 26 17:22:21 BRST 2009 > 250016 > [root@brtldp11 backup]# date; cat temp.ldif |grep -e '^pnnum*' |wc -l > Mon Jan 26 17:23:00 BRST 2009 > 250016 > See above that even almost after 1 minute passed not new LDIF entrance was included. > 3) Then query get stuck and looks like deterministic it time by time only dumps 16 entrances and get stuck. This behavior repeats and with these stuck that sometime gets minutes the query never ends. > > olmBDBEntryCache: 884 > olmBDBDNCache: 1000261 > olmBDBIDLCache: 1 > > olmBDBEntryCache: 611 > olmBDBDNCache: 1000261 > olmBDBIDLCache: 1 > >Even with cachesize as 1000 and cachefree as 1, the olmBDBEntryCache continues to decrease, just slow now. > > I was expecting that a cachefree as 1000 would purge all entrances and then > cache again all 1000 new with in sequence always answering the search. So > the search would never hangs like it is happening now. I see. OK, this is now fixed in HEAD. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by rlvcosta＠yahoo.com 26 Jan '09

26 Jan '09

Howard, I do not think the ITS is completely solved. Let me try to explain better. I have the system respecting now the DN cachesize boundary. At the same time I wasn't sure about the cachefree idea and now I changed to 1, or equal the default value. In any case what happens is : 1) I have the system doing sequential search in a reasonable speed until the dncachesize is reached; 2) After dncchesize is reached the sequential search hangs, or the output from the search get stuck for a long time(I'm forwarding to a file so I do not have screen actualization delays); Ex: [root@brtldp11 backup]# date; cat temp.ldif |grep -e '^pnnum*' |wc -l Mon Jan 26 17:22:21 BRST 2009 250016 [root@brtldp11 backup]# date; cat temp.ldif |grep -e '^pnnum*' |wc -l Mon Jan 26 17:23:00 BRST 2009 250016 See above that even almost after 1 minute passed not new LDIF entrance was included. 3) Then query get stuck and looks like deterministic it time by time only dumps 16 entrances and get stuck. This behavior repeats and with these stuck that sometime gets minutes the query never ends. olmBDBEntryCache: 884 olmBDBDNCache: 1000261 olmBDBIDLCache: 1 olmBDBEntryCache: 611 olmBDBDNCache: 1000261 olmBDBIDLCache: 1 Even with cachesize as 1000 and cachefree as 1, the olmBDBEntryCache continues to decrease, just slow now. I was expecting that a cachefree as 1000 would purge all entrances and then cache again all 1000 new with in sequence always answering the search. So the search would never hangs like it is happening now. It get stuck and will never ends since it responds only 16 entrances in order of minutes and in bursts. The previous load was more reasonable than now since even taking much longer it would end the search. [root@brtldp11 backup]# date; cat temp.ldif |grep -e '^pnnum*' |wc -l Mon Jan 26 17:31:36 BRST 2009 250128 Please let me know if you need more information, Rodrigo. --- On Mon, 1/26/09, Howard Chu <hyc(a)symas.com> wrote: > From: Howard Chu <hyc(a)symas.com> > Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 > To: rlvcosta(a)yahoo.com > Cc: openldap-its(a)OpenLDAP.org > Date: Monday, January 26, 2009, 5:18 PM > Rodrigo Costa wrote: > > Howard, > > > > I download the new HEAD version and made some testing. > Now the > olmBDBEntryCache is following the cache configuration. > > Good, then this ITS is resolved. Usage questions should be > directed to the > -software mailing list. > > > I also made some small change in slapd.conf including > cachefree > configuration. Please see below in the end how slapd is > configured related to > bdb cache : > > > line 123 (cachesize 1000) > > line 124 (cachefree 1000) > > line 125 (idlcachesize 1000) > > line 126 (dncachesize 1000000) > > Setting cachefree equal to cachesize will effectively cause > the entire entry > cache to be dumped each time it reaches its maximum size. > That's clearly not a > good idea. > > > I increase the dncachesize since with a small value > the search takes > considerable more time. > > > > First I tested with a value greater than the value > that was consuming > > before > without the memory boundary. In this way I was expecting > some order of > magnitude as before, what it achieved (line 126 > (dncachesize 5000000)): > > > BEFORE CACHED: > > 1000000 > > > > real 5m23.084s > > user 0m28.026s > > sys 0m6.686s > > > > Then Cache has : > > > > olmBDBEntryCache: 1001 > > olmBDBDNCache: 4000264 > > olmBDBIDLCache: 1 > > > > AFTER CACHED: > > > > 1000000 > > > > real 2m31.623s > > user 0m28.145s > > sys 0m8.637s > > > > Just to let clear these tests above where with the new > logic and where the > DN Cache Size is bigger than the final number I had with > the old logic where > all information is cached into memory. > > > > For each entrance I have 4 dn's that compose the > entrance. Since my filter > is for only one of the dn I was expecting this cache to be > only related with > the filter and then in 1,000,000. > > > Then I change the dncachesize to 1000000 (line 126 > (dncachesize 1000000) : > > > > The total time for a search using a filter for one > index dn became : > > > > ldap_result: Can't contact LDAP server (-1) > > 255801 > > > > real 469m26.619s > > user 0m7.027s > > sys 0m1.756s > > > > I needed to kill the server so I would have an idea > about how many > > entrances > it searched. The time is too long and I'm not sure if > it would even end. I let > all night but the process did not end. This was the same > ldapsearch as when > all entrances would be allocated in a DN cache into memory. > > > > But the cache process was something I wasn't > expecting. Before the > dncachesize was reached I had at monitor : > > > > olmBDBEntryCache: 1001 > > olmBDBDNCache: 921296 > > olmBDBIDLCache: 1 > > > > Then after the boundary was reached it became : > > > > olmBDBEntryCache: 1 > > olmBDBDNCache: 1000262 > > olmBDBIDLCache: 1 > > > > Where these numbers didn't change anymore. Not > sure why after the > dncachesize is reached the cachesize(or olmBDBEntryCache) > became 1. > > Because you set cachefree to 1000. 1001 - 1000 = 1. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP > http://www.openldap.org/project/

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by hyc＠symas.com 26 Jan '09

26 Jan '09

Rodrigo Costa wrote: > Howard, > > I download the new HEAD version and made some testing. Now the olmBDBEntryCache is following the cache configuration. Good, then this ITS is resolved. Usage questions should be directed to the -software mailing list. > I also made some small change in slapd.conf including cachefree configuration. Please see below in the end how slapd is configured related to bdb cache : > line 123 (cachesize 1000) > line 124 (cachefree 1000) > line 125 (idlcachesize 1000) > line 126 (dncachesize 1000000) Setting cachefree equal to cachesize will effectively cause the entire entry cache to be dumped each time it reaches its maximum size. That's clearly not a good idea. > I increase the dncachesize since with a small value the search takes considerable more time. > > First I tested with a value greater than the value that was consuming > before without the memory boundary. In this way I was expecting some order of magnitude as before, what it achieved (line 126 (dncachesize 5000000)): > BEFORE CACHED: > 1000000 > > real 5m23.084s > user 0m28.026s > sys 0m6.686s > > Then Cache has : > > olmBDBEntryCache: 1001 > olmBDBDNCache: 4000264 > olmBDBIDLCache: 1 > > AFTER CACHED: > > 1000000 > > real 2m31.623s > user 0m28.145s > sys 0m8.637s > > Just to let clear these tests above where with the new logic and where the DN Cache Size is bigger than the final number I had with the old logic where all information is cached into memory. > > For each entrance I have 4 dn's that compose the entrance. Since my filter is for only one of the dn I was expecting this cache to be only related with the filter and then in 1,000,000. > Then I change the dncachesize to 1000000 (line 126 (dncachesize 1000000) : > > The total time for a search using a filter for one index dn became : > > ldap_result: Can't contact LDAP server (-1) > 255801 > > real 469m26.619s > user 0m7.027s > sys 0m1.756s > > I needed to kill the server so I would have an idea about how many > entrances it searched. The time is too long and I'm not sure if it would even end. I let all night but the process did not end. This was the same ldapsearch as when all entrances would be allocated in a DN cache into memory. > > But the cache process was something I wasn't expecting. Before the dncachesize was reached I had at monitor : > > olmBDBEntryCache: 1001 > olmBDBDNCache: 921296 > olmBDBIDLCache: 1 > > Then after the boundary was reached it became : > > olmBDBEntryCache: 1 > olmBDBDNCache: 1000262 > olmBDBIDLCache: 1 > > Where these numbers didn't change anymore. Not sure why after the dncachesize is reached the cachesize(or olmBDBEntryCache) became 1. Because you set cachefree to 1000. 1001 - 1000 = 1. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5906) slapo-rwm and cn=config
by ando＠sys-net.it 26 Jan '09

26 Jan '09

ldap(a)htam.net wrote: > Full_Name: Mathieu MILLET > Version: 2.4.13 > OS: Linux Fedora 10 > URL: > Submission from: (NULL) (81.80.3.6) > > > When configuring the slapo-rwm overlay inside a relay database, using cn=config > style, it is impossible to load (online or offline) the attribute > olcRwmRewrite. Should be fixed in HEAD. Please test and report. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ----------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Fax: +39 0382 476497 Email: ando(a)sys-net.it -----------------------------------

1 0

← Newer
1
2
3
4
5
6
7
8
9
...
21
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs January 2009