openldap-bugs

openldap-bugs@openldap.org

1 participants
21065 discussions

Re: (ITS#5540) Normalization assertion in attr.c
by hyc＠symas.com 27 Jan '09

27 Jan '09

unix.gurus(a)gmail.com wrote: > Hi, > > On Tue, Jun 24, 2008 at 12:50 PM, Sean Burford<unix.gurus(a)gmail.com> wrote: >> namingContexts was the trickiest, since RFC 4512 defines it without an >> equality matching rule but monitor_back_register_database() compares >> it with backend suffixes. It has to deal with capitalised values >> properly otherwise we end up with duplicate namingContexts entries, so >> normalization is required. > > I have uploaded > ftp://ftp.openldap.org/incoming/sean-burford-monitor-test-080706.patch > which adds a test for various things under cn=monitor. If duplicate > namingContexts creep in (as mentioned above) it detects that. > > The duplicate naming contexts thing was fixed by comparing a_vals with > be_suffix, I just wrote the test since there was no existing > cn=monitor test. Sorry for dropping the ball on this. These last two patches are now committed in HEAD. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by hyc＠symas.com 27 Jan '09

27 Jan '09

rlvcosta(a)yahoo.com wrote: > Quanah, > > In this specific test I reported in my previous e-mail it was used glibc. > > Now I tested with tcmalloc and the memory again was much more consumed and never released. > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 10652 ldap 18 0 2006m 1.8g 68m S 100 15.8 8:36.75 slapd > > Once it get this much memory it then stabilize but it is much more than before that was 385m. We may need to have sample data from you to reproduce this situation. I haven't seen anything like it on my tests. Running two searches simultaneously still behaves normally for me. > > The scenario was : Likewise, the exact commands and arguments will be needed... > > 1) Start a full ldapsearch for one specific dn(the DB has 4 dn's); > 2) After sometime start a new full ldapsearch to behave as queries to DB when replication for LDIF backup is being done; > 3) Export one of the queries to a file so speed verification can be done with grep and wc. > > After sometime the memory starts to be consumed in large chunks until it stopped in around 2G. > > Since one of the queries is being redirect to a file I can tail to have a visual rate of the entrances being returned by slapd. As the issue before after sometime the rate appears to get stuck and come in small 16 entrances chunks but only after some minutes. > > This behavior above appears to happen after the second ldapsearch fills its own dn cache even monitor always shows only one information as I could see. > > [root@brtldp11 ~]# date; grep -e '^pnnum*' /backup/temp.ldif |wc -l > Mon Jan 26 21:43:31 BRST 2009 > 482432 > [root@brtldp11 ~]# date; grep -e '^pnnum*' /backup/temp.ldif |wc -l > Mon Jan 26 21:44:56 BRST 2009 > 482448 > > This behavior is similar as before with the initial issue with a single process. > > I just notice when slapd starts there are these threads : > > [root@brtldp12 ~]# ps -mu ldap > PID TTY TIME CMD > 10638 pts/2 00:00:00 slapd > - - 00:00:00 - > - - 00:00:00 - > > Then after the 2 queries are started I have : > > [root@brtldp12 ~]# ps -mu ldap > PID TTY TIME CMD > 10195 pts/2 00:44:22 slapd > - - 00:00:00 - > - - 00:00:00 - > - - 00:26:09 - > - - 00:18:13 - > - - 00:00:00 - > > So 3 new threads. Looks like for some reason with multiples queries the issue with response getting stuck repeats. > > Even one ldapsearch is stopped the number of threads continues the same, memory continues allocated and the other search get with this slow rate. > > These outputs are using now tcmalloc but similar as with glibc. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by hyc＠symas.com 27 Jan '09

27 Jan '09

Rodrigo Costa wrote: > Howard, > I saw sometimes the DN cache allocating a little more memory since it sometimes pass a little the boundary, like seen below : > olmBDBEntryCache: 1000 > olmBDBDNCache: 1000247 > olmBDBIDLCache: 1 > > The maximum I saw was 1000268 but nothing considerable. In this way now slapd is keeping memory allocation stable as it is supposed to behave. The DN cache is hierarchical; it will only free leaf nodes (entries that have no children). So depending on the structure of your tree, it's normal for the Monitor to report more elements in the DN cache than you configured. > > The only comment is if dn cache is passed this memory allocated appears to never be released. The maximum size slapd obtained was : > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 10195 ldap 18 0 385m 266m 68m S 0 2.2 20:54.16 slapd > > This is more than perfect ! > > Thanks for all your quick solution of this problem. I believe many users > of openLDAP will see their system more stable with this fix. > > Just some last questions. Is there any expectation for a new release, for example 2.4.15, with this ITS included? Just to use a formal release. > > By the tests I was expecting the cachefree as the cache size bringing some improvements, at least in a full query, since system would "cycle" pointer memory and overwritten all cache at once. What is the purpose of cachefree? cachefree applies only to the entry cache. The idea is to minimize the number of times it must cycle through the cache trying to free entries. Typically when we configure large servers with caches of 1 million entries or more, we configure this to 1000 or so (0.1% of cache size). -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by rlvcosta＠yahoo.com 26 Jan '09

26 Jan '09

Quanah, In this specific test I reported in my previous e-mail it was used glibc. Now I tested with tcmalloc and the memory again was much more consumed and never released. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10652 ldap 18 0 2006m 1.8g 68m S 100 15.8 8:36.75 slapd Once it get this much memory it then stabilize but it is much more than before that was 385m. The scenario was : 1) Start a full ldapsearch for one specific dn(the DB has 4 dn's); 2) After sometime start a new full ldapsearch to behave as queries to DB when replication for LDIF backup is being done; 3) Export one of the queries to a file so speed verification can be done with grep and wc. After sometime the memory starts to be consumed in large chunks until it stopped in around 2G. Since one of the queries is being redirect to a file I can tail to have a visual rate of the entrances being returned by slapd. As the issue before after sometime the rate appears to get stuck and come in small 16 entrances chunks but only after some minutes. This behavior above appears to happen after the second ldapsearch fills its own dn cache even monitor always shows only one information as I could see. [root@brtldp11 ~]# date; grep -e '^pnnum*' /backup/temp.ldif |wc -l Mon Jan 26 21:43:31 BRST 2009 482432 [root@brtldp11 ~]# date; grep -e '^pnnum*' /backup/temp.ldif |wc -l Mon Jan 26 21:44:56 BRST 2009 482448 This behavior is similar as before with the initial issue with a single process. I just notice when slapd starts there are these threads : [root@brtldp12 ~]# ps -mu ldap PID TTY TIME CMD 10638 pts/2 00:00:00 slapd - - 00:00:00 - - - 00:00:00 - Then after the 2 queries are started I have : [root@brtldp12 ~]# ps -mu ldap PID TTY TIME CMD 10195 pts/2 00:44:22 slapd - - 00:00:00 - - - 00:00:00 - - - 00:26:09 - - - 00:18:13 - - - 00:00:00 - So 3 new threads. Looks like for some reason with multiples queries the issue with response getting stuck repeats. Even one ldapsearch is stopped the number of threads continues the same, memory continues allocated and the other search get with this slow rate. These outputs are using now tcmalloc but similar as with glibc. Regards, Rodrigo. --- On Mon, 1/26/09, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > From: Quanah Gibson-Mount <quanah(a)zimbra.com> > Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 > To: rlvcosta(a)yahoo.com, openldap-its(a)openldap.org > Date: Monday, January 26, 2009, 9:25 PM > --On Monday, January 26, 2009 3:18 PM -0800 Rodrigo Costa > <rlvcosta(a)yahoo.com> wrote: > > > Quanah and Howard, > > > > Sorry to come back. > > > > I tried to make 2 ldapsearch to the same DB > simultaneously, doing some > > stress test like dump the LDIF and queries coming. > > > > The slapd after sometime start to get more memory. By > what I could > > observe looks like new thread was started consuming > again the same memory > > as before so the slad now is consuming around the > double of memory as > > before. > > Are you also using an alternate malloc() instead of glibc? > I know you were > investigating this, just want to confirm you are using > tcmalloc or hoard > and not glibc. > > --Quanah > > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and > collaboration

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by quanah＠zimbra.com 26 Jan '09

26 Jan '09

--On Monday, January 26, 2009 3:18 PM -0800 Rodrigo Costa <rlvcosta(a)yahoo.com> wrote: > Quanah and Howard, > > Sorry to come back. > > I tried to make 2 ldapsearch to the same DB simultaneously, doing some > stress test like dump the LDIF and queries coming. > > The slapd after sometime start to get more memory. By what I could > observe looks like new thread was started consuming again the same memory > as before so the slad now is consuming around the double of memory as > before. Are you also using an alternate malloc() instead of glibc? I know you were investigating this, just want to confirm you are using tcmalloc or hoard and not glibc. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by rlvcosta＠yahoo.com 26 Jan '09

26 Jan '09

Quanah and Howard, Sorry to come back. I tried to make 2 ldapsearch to the same DB simultaneously, doing some stress test like dump the LDIF and queries coming. The slapd after sometime start to get more memory. By what I could observe looks like new thread was started consuming again the same memory as before so the slad now is consuming around the double of memory as before. I believe the cache would not grow beyond the boundary as like per thread since once cache is allocated it is never released. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10195 ldap 18 0 1913m 1.7g 68m S 100 15.0 31:25.71 slapd Then performance start to degrade again. Like before looks like the dump get stuck and comes in chunks. See example below : Mon Jan 26 21:05:16 BRST 2009 378752 Mon Jan 26 21:09:21 BRST 2009 378800 See that after around 4 minutes I have only 48 adds to the output LDIF from my first ldapsearch. Looks like if multiples queries are happening something still getting lost. Both queries are to the same DB and the monitor has : readOnly: FALSE olmBDBEntryCache: 1000 olmBDBDNCache: 1000953 olmBDBIDLCache: 1 Sorry to come back with this issue but I could detect this now. Thanks, Rodrigo. --- On Mon, 1/26/09, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > From: Quanah Gibson-Mount <quanah(a)zimbra.com> > Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 > To: rlvcosta(a)yahoo.com, openldap-its(a)openldap.org > Date: Monday, January 26, 2009, 8:53 PM > --On Monday, January 26, 2009 10:44 PM +0000 > rlvcosta(a)yahoo.com wrote: > > > Just some last questions. Is there any expectation for > a new release, for > > example 2.4.15, with this ITS included? Just to use a > formal release. > > 2.4.14 will have the fix for this ITS. There's no set > release date at this time, but hopefully soon. > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and > collaboration

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by quanah＠zimbra.com 26 Jan '09

26 Jan '09

--On Monday, January 26, 2009 10:44 PM +0000 rlvcosta(a)yahoo.com wrote: > Just some last questions. Is there any expectation for a new release, for > example 2.4.15, with this ITS included? Just to use a formal release. 2.4.14 will have the fix for this ITS. There's no set release date at this time, but hopefully soon. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by rlvcosta＠yahoo.com 26 Jan '09

26 Jan '09

Howard, Now it fits under my expectation. Under the initial load without the fix I had slapd not respecting any boundary and then all DN information being cached. In this way the first query would take longer since the information wasn't yet cached. At that time I had, without cache, a search for around 1million entrances in around 7 minutes and after cached this time reduce to around half. So now with the new load I was expecting something a little longer as the initial not cached since now slapd will not be able to cache all information and so the cache will be often overwritten. Now in my tests I obtained : First time: real 6m50.017s user 0m17.346s sys 0m9.159s Second time: real 6m47.790s user 0m17.032s sys 0m9.137s Third time: real 7m17.459s user 0m17.412s sys 0m9.320s This is more than perfect. Amazing how you guys could solve this problem so quickly. I saw sometimes the DN cache allocating a little more memory since it sometimes pass a little the boundary, like seen below : olmBDBEntryCache: 1000 olmBDBDNCache: 1000247 olmBDBIDLCache: 1 The maximum I saw was 1000268 but nothing considerable. In this way now slapd is keeping memory allocation stable as it is supposed to behave. The only comment is if dn cache is passed this memory allocated appears to never be released. The maximum size slapd obtained was : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10195 ldap 18 0 385m 266m 68m S 0 2.2 20:54.16 slapd This is more than perfect ! Thanks for all your quick solution of this problem. I believe many users of openLDAP will see their system more stable with this fix. Just some last questions. Is there any expectation for a new release, for example 2.4.15, with this ITS included? Just to use a formal release. By the tests I was expecting the cachefree as the cache size bringing some improvements, at least in a full query, since system would "cycle" pointer memory and overwritten all cache at once. What is the purpose of cachefree? Just to understand better and then tune later with some better idea about cache purposes. Best Regards, Rodrigo. --- On Mon, 1/26/09, Howard Chu <hyc(a)symas.com> wrote: > From: Howard Chu <hyc(a)symas.com> > Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 > To: rlvcosta(a)yahoo.com > Cc: openldap-its(a)OpenLDAP.org > Date: Monday, January 26, 2009, 6:00 PM > Rodrigo Costa wrote: > > Howard, > > > > I do not think the ITS is completely solved. Let me > try to explain better. > > > > I have the system respecting now the DN cachesize > boundary. At the same time I wasn't sure about the > cachefree idea and now I changed to 1, or equal the default > value. > > > > In any case what happens is : > > > > 1) I have the system doing sequential search in a > reasonable speed until the dncachesize is reached; > > 2) After dncchesize is reached the sequential search > hangs, or the output from the search get stuck for a long > time(I'm forwarding to a file so I do not have screen > actualization delays); > > Ex: > > [root@brtldp11 backup]# date; cat temp.ldif |grep -e > '^pnnum*' |wc -l > > Mon Jan 26 17:22:21 BRST 2009 > > 250016 > > [root@brtldp11 backup]# date; cat temp.ldif |grep -e > '^pnnum*' |wc -l > > Mon Jan 26 17:23:00 BRST 2009 > > 250016 > > See above that even almost after 1 minute passed not > new LDIF entrance was included. > > 3) Then query get stuck and looks like deterministic > it time by time only dumps 16 entrances and get stuck. This > behavior repeats and with these stuck that sometime gets > minutes the query never ends. > > > > olmBDBEntryCache: 884 > > olmBDBDNCache: 1000261 > > olmBDBIDLCache: 1 > > > > olmBDBEntryCache: 611 > > olmBDBDNCache: 1000261 > > olmBDBIDLCache: 1 > > > >Even with cachesize as 1000 and cachefree as 1, the > olmBDBEntryCache continues > to decrease, just slow now. > > > > I was expecting that a cachefree as 1000 would purge > all entrances and then > > cache again all 1000 new with in sequence always > answering the search. So > > the search would never hangs like it is happening now. > > I see. OK, this is now fixed in HEAD. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP > http://www.openldap.org/project/

1 0

Re: (ITS#5336)
by hyc＠symas.com 26 Jan '09

26 Jan '09

quanah(a)zimbra.com wrote: > A similar bug has been reported in the debian bug tracking system, with > some debugging information and reproducibility steps. > > <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=465915> Noting that this was closed in Debian as of 2.4.9. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by hyc＠symas.com 26 Jan '09

26 Jan '09

Rodrigo Costa wrote: > Howard, > > I do not think the ITS is completely solved. Let me try to explain better. > > I have the system respecting now the DN cachesize boundary. At the same time I wasn't sure about the cachefree idea and now I changed to 1, or equal the default value. > > In any case what happens is : > > 1) I have the system doing sequential search in a reasonable speed until the dncachesize is reached; > 2) After dncchesize is reached the sequential search hangs, or the output from the search get stuck for a long time(I'm forwarding to a file so I do not have screen actualization delays); > Ex: > [root@brtldp11 backup]# date; cat temp.ldif |grep -e '^pnnum*' |wc -l > Mon Jan 26 17:22:21 BRST 2009 > 250016 > [root@brtldp11 backup]# date; cat temp.ldif |grep -e '^pnnum*' |wc -l > Mon Jan 26 17:23:00 BRST 2009 > 250016 > See above that even almost after 1 minute passed not new LDIF entrance was included. > 3) Then query get stuck and looks like deterministic it time by time only dumps 16 entrances and get stuck. This behavior repeats and with these stuck that sometime gets minutes the query never ends. > > olmBDBEntryCache: 884 > olmBDBDNCache: 1000261 > olmBDBIDLCache: 1 > > olmBDBEntryCache: 611 > olmBDBDNCache: 1000261 > olmBDBIDLCache: 1 > >Even with cachesize as 1000 and cachefree as 1, the olmBDBEntryCache continues to decrease, just slow now. > > I was expecting that a cachefree as 1000 would purge all entrances and then > cache again all 1000 new with in sequence always answering the search. So > the search would never hangs like it is happening now. I see. OK, this is now fixed in HEAD. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs