openldap-bugs

openldap-bugs@openldap.org

1 participants
20961 discussions

Re: (ITS#5903) memberof stacked instance member-attr delete failure
by hyc＠symas.com 27 Jan '09

27 Jan '09

mbackes(a)symas.com wrote: > Full_Name: Matthew Backes > Version: 2.4, head > OS: any > URL: > Submission from: (NULL) (76.88.99.93) > > > If slapo-memberof is instanced more than once, deleting the > member-attr values from a group or deleting the group object will not > remove the memberof-attr values from the members. > > Adds are not affected. > > Internally, the operation fails because it tries using the > memberof-attribute name from the last memberof instance in the stack. The memberOf code is using a single static variable to hold its state. This is obviously broken. It's also using thread keys for its temporary storage, which is totally unnecessary; it should simply use its own callback and the cb_private field of that callback. As such, much of this code needs to be gutted and rewritten. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#5907) accesslog should use overlay BackendInfo to read old entry
by rein＠OpenLDAP.org 27 Jan '09

27 Jan '09

Full_Name: Rein Tollevik Version: CVS HEAD OS: Irrelevant URL: Submission from: (NULL) (81.93.160.250) Submitted by: rein I use accesslog stacked above the glue overlay to have it log access to all of the glued databases. When accesslog reads the old entry it goes directly to the original BackendInfo, which causes it to not find entries in the subordinate DBs. A fix is coming. Rein Tollevik Basefarm AS

1 0

Re: (ITS#5661) contextCSN gets corrupted on the stand by mirror
by carl.johnstone＠gmgrd.co.uk 27 Jan '09

27 Jan '09

I'm seeing the same think on a 3-way multi-master setup here. Two servers (#001 & #002) are currently sat next to each other. The third (#003) is at a remote location. We're currently doing all amends through #001, although in the long term we'll be doing amends through all the servers. When I checked them yesterday I spotted that the remote server had a corrupt contextCSN for server #001. I dropped the DB and synced both the config and data from the main server again overnight. On checking again today the contextCSN is once again corrupt. In my case it's the first 8 bytes rather than the first 4. I'm running 2.4.13 with bdb 4.7.25 (first 3 patches applied). build: ./configure --enable-dynamic --enable-crypt --enable-modules=yes --enable-backends=mod --enable-overlays=mod --enable-sql=no --enable-ndb=no Carl

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by rlvcosta＠yahoo.com 27 Jan '09

27 Jan '09

Howard, One more comment. If you take a longer time between the ldapsearches the issue doesn't appear sometimes. So you need to start the first ldapsearh and like 15 seconds later start the second one. With this scenario the problem will for sure appear. In this way we will see large chunks of memory being allocated and after it stabilizes the query get stuck returning as the issue before 16 records per minute, or very slow and getting stuck all the time. Regards, Rodrigo. --- On Mon, 1/26/09, Howard Chu <hyc(a)symas.com> wrote: > From: Howard Chu <hyc(a)symas.com> > Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 > To: rlvcosta(a)yahoo.com > Cc: openldap-its(a)openldap.org > Date: Monday, January 26, 2009, 11:16 PM > rlvcosta(a)yahoo.com wrote: > > Quanah, > > > > In this specific test I reported in my previous e-mail > it was used glibc. > > > > Now I tested with tcmalloc and the memory again was > much more consumed and never released. > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM > TIME+ COMMAND > > 10652 ldap 18 0 2006m 1.8g 68m S 100 15.8 > 8:36.75 slapd > > > > Once it get this much memory it then stabilize but it > is much more than before that was 385m. > > We may need to have sample data from you to reproduce this > situation. I > haven't seen anything like it on my tests. Running two > searches simultaneously > still behaves normally for me. > > > > The scenario was : > > Likewise, the exact commands and arguments will be > needed... > > > > 1) Start a full ldapsearch for one specific dn(the DB > has 4 dn's); > > 2) After sometime start a new full ldapsearch to > behave as queries to DB when replication for LDIF backup is > being done; > > 3) Export one of the queries to a file so speed > verification can be done with grep and wc. > > > > After sometime the memory starts to be consumed in > large chunks until it stopped in around 2G. > > > > Since one of the queries is being redirect to a file I > can tail to have a visual rate of the entrances being > returned by slapd. As the issue before after sometime the > rate appears to get stuck and come in small 16 entrances > chunks but only after some minutes. > > > > This behavior above appears to happen after the second > ldapsearch fills its own dn cache even monitor always shows > only one information as I could see. > > > > [root@brtldp11 ~]# date; grep -e '^pnnum*' > /backup/temp.ldif |wc -l > > Mon Jan 26 21:43:31 BRST 2009 > > 482432 > > [root@brtldp11 ~]# date; grep -e '^pnnum*' > /backup/temp.ldif |wc -l > > Mon Jan 26 21:44:56 BRST 2009 > > 482448 > > > > This behavior is similar as before with the initial > issue with a single process. > > > > I just notice when slapd starts there are these > threads : > > > > [root@brtldp12 ~]# ps -mu ldap > > PID TTY TIME CMD > > 10638 pts/2 00:00:00 slapd > > - - 00:00:00 - > > - - 00:00:00 - > > > > Then after the 2 queries are started I have : > > > > [root@brtldp12 ~]# ps -mu ldap > > PID TTY TIME CMD > > 10195 pts/2 00:44:22 slapd > > - - 00:00:00 - > > - - 00:00:00 - > > - - 00:26:09 - > > - - 00:18:13 - > > - - 00:00:00 - > > > > So 3 new threads. Looks like for some reason with > multiples queries the issue with response getting stuck > repeats. > > > > Even one ldapsearch is stopped the number of threads > continues the same, memory continues allocated and the other > search get with this slow rate. > > > > These outputs are using now tcmalloc but similar as > with glibc. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP > http://www.openldap.org/project/

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by rlvcosta＠yahoo.com 27 Jan '09

27 Jan '09

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by rlvcosta＠yahoo.com 27 Jan '09

27 Jan '09

--0-1644073040-1233052012=:88983 Content-Type: text/plain; charset=us-ascii Howard, Initially the search behaves normally but after some time something changes and memory starts to be consumed again until it reaches another stable condition, even very large(around 2G) and the search get stuck as before. Once search get stuck even I stop both searches and start a new one it get stuck. So in my test I stopped my 2 searches and started a new one. Then I saw the new search with only 240 returns and then get stuck entering in the same model as before with 16 entrances in bursts in order of minutes to enter in these bursts. As I tried to explain this appears to happen when the seconds ldapsearch, started at a later time than the first ldapsearch, reaches the dn cachesize. This is what appears to happen. I beleive for you to reproduce you need to have a DB large enough and a slapd.conf where dn cachesize will overpass the memory boundary imposed. Then after the second ldapsearch memory boundary is reached the issue should appear. The commands I runned were : time ldapsearch -LLL -x -D "cn=admin,ou=INDEXES,o=alcatel,c=fr" -w secret -b "ou=INDEXES,o=alcatel,c=fr" -H ldap://10.142.15.42:389 'pnnumber=*' > /backup/temp.ldif and time ldapsearch -LLL -x -D "cn=admin,ou=INDEXES,o=alcatel,c=fr" -w secret -b "ou=INDEXES,o=alcatel,c=fr" -H ldap://10.142.15.42:389 'pnnumber=*'| egrep -e '^pnnumber*' |wc -l With the first command, redirecting to a file, I can tail or wc and see it's progress. After the second ldapsearch process starts to make slapd consume memory in large chunks until the new 2G boundary is reached. Once this happen the search get stuck as before with the single process returning 16 records per minute(s). Once this condition is reached even stopping the old searches and starting new ones the rate will be very slow, like 16 reports per minute(s). It get stuck as before. Please see attached a single entrance in my DB where you can change DB keys and load it as needed. Thanks, Rodrigo. --- On Mon, 1/26/09, Howard Chu <hyc(a)symas.com> wrote: > From: Howard Chu <hyc(a)symas.com> > Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 > To: rlvcosta(a)yahoo.com > Cc: openldap-its(a)openldap.org > Date: Monday, January 26, 2009, 11:16 PM > rlvcosta(a)yahoo.com wrote: > > Quanah, > > > > In this specific test I reported in my previous e-mail > it was used glibc. > > > > Now I tested with tcmalloc and the memory again was > much more consumed and never released. > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM > TIME+ COMMAND > > 10652 ldap 18 0 2006m 1.8g 68m S 100 15.8 > 8:36.75 slapd > > > > Once it get this much memory it then stabilize but it > is much more than before that was 385m. > > We may need to have sample data from you to reproduce this > situation. I > haven't seen anything like it on my tests. Running two > searches simultaneously > still behaves normally for me. > > > > The scenario was : > > Likewise, the exact commands and arguments will be > needed... > > > > 1) Start a full ldapsearch for one specific dn(the DB > has 4 dn's); > > 2) After sometime start a new full ldapsearch to > behave as queries to DB when replication for LDIF backup is > being done; > > 3) Export one of the queries to a file so speed > verification can be done with grep and wc. > > > > After sometime the memory starts to be consumed in > large chunks until it stopped in around 2G. > > > > Since one of the queries is being redirect to a file I > can tail to have a visual rate of the entrances being > returned by slapd. As the issue before after sometime the > rate appears to get stuck and come in small 16 entrances > chunks but only after some minutes. > > > > This behavior above appears to happen after the second > ldapsearch fills its own dn cache even monitor always shows > only one information as I could see. > > > > [root@brtldp11 ~]# date; grep -e '^pnnum*' > /backup/temp.ldif |wc -l > > Mon Jan 26 21:43:31 BRST 2009 > > 482432 > > [root@brtldp11 ~]# date; grep -e '^pnnum*' > /backup/temp.ldif |wc -l > > Mon Jan 26 21:44:56 BRST 2009 > > 482448 > > > > This behavior is similar as before with the initial > issue with a single process. > > > > I just notice when slapd starts there are these > threads : > > > > [root@brtldp12 ~]# ps -mu ldap > > PID TTY TIME CMD > > 10638 pts/2 00:00:00 slapd > > - - 00:00:00 - > > - - 00:00:00 - > > > > Then after the 2 queries are started I have : > > > > [root@brtldp12 ~]# ps -mu ldap > > PID TTY TIME CMD > > 10195 pts/2 00:44:22 slapd > > - - 00:00:00 - > > - - 00:00:00 - > > - - 00:26:09 - > > - - 00:18:13 - > > - - 00:00:00 - > > > > So 3 new threads. Looks like for some reason with > multiples queries the issue with response getting stuck > repeats. > > > > Even one ldapsearch is stopped the number of threads > continues the same, memory continues allocated and the other > search get with this slow rate. > > > > These outputs are using now tcmalloc but similar as > with glibc. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP > http://www.openldap.org/project/ --0-1644073040-1233052012=:88983 Content-Type: text/plain; name="one_INDEXES_entrance.txt" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="one_INDEXES_entrance.txt" ZG46IHBubnVtYmVyPVwyQjU1NDE4NDAxMTQ5NixvdT1yYW5nZTk2LG91PW1z aXNkbixvdT1JTkRFWEVTLG89YWxjYXRlbCxjPWZyDQpzdWJwbmlkOiAwDQpo b3N0OiBicnRsZHAxMmUNCm9iamVjdENsYXNzOiBwaG9uZWluZm8NCm9iamVj dENsYXNzOiBpbmRleG5vZGUNCnJlZmRuOiB1aWQ9MTIxOTg0Mzc5NjY1N1wy QjU1NDE4NDAxMTQ5NixvdT1SRVBPU0lUT1JZLG91PUNPTlRFTlQsbz1hbGNh dGVsLGM9Zg0KIHINCnBubnVtYmVyOiArNTU0MTg0MDExNDk2DQoNCmRuOiB3 ZWJsb2dpbj1cMkI1NTQxODQwMTE0OTYsb3U9cmFuZ2U2NyxvdT13ZWJsb2dp bixvdT1JTkRFWEVTLG89YWxjYXRlbCxjPWZyDQp3ZWJsb2dpbjogKzU1NDE4 NDAxMTQ5Ng0KaG9zdDogYnJ0bGRwMTJlDQphY2NvdW50U3RhdHVzOiAxDQpv YmplY3RDbGFzczogY29tbW9udXNlcg0Kb2JqZWN0Q2xhc3M6IGluZGV4bm9k ZQ0KcmVmZG46IHVpZD0xMjE5ODQzNzk2NjU3XDJCNTU0MTg0MDExNDk2LG91 PVJFUE9TSVRPUlksb3U9Q09OVEVOVCxvPWFsY2F0ZWwsYz1mDQogcg0KY246 ICs1NTQxODQwMTE0OTYNCg0KZG46IG1haWw9XDJCNTU0MTg0MDExNDk2L3R5 cGVcM0RQTE1OQG1tcy5icmFzaWx0ZWxlY29tLmNvbS5icixvdT1yYW5nZTg0 LG91PW1hDQogaWwsb3U9SU5ERVhFUyxvPWFsY2F0ZWwsYz1mcg0KbWFpbDog KzU1NDE4NDAxMTQ5Ni90eXBlPVBMTU5AbW1zLmJyYXNpbHRlbGVjb20uY29t LmJyDQpob3N0OiBicnRsZHAxMmUNCmFjY291bnRTdGF0dXM6IDENCm9iamVj dENsYXNzOiBjb21tb251c2VyDQpvYmplY3RDbGFzczogaW5kZXhub2RlDQpy ZWZkbjogdWlkPTEyMTk4NDM3OTY2NTdcMkI1NTQxODQwMTE0OTYsb3U9UkVQ T1NJVE9SWSxvdT1DT05URU5ULG89YWxjYXRlbCxjPWYNCiByDQpjbjogKzU1 NDE4NDAxMTQ5Ng0KDQpkbjogbWFpbGxvZ2luPVwyQjU1NDE4NDAxMTQ5Ni90 eXBlXDNEUExNTkBtbXMuYnJhc2lsdGVsZWNvbS5jb20uYnIsb3U9cmFuZ2U4 NCwNCiBvdT1pbWFwbG9naW4sb3U9SU5ERVhFUyxvPWFsY2F0ZWwsYz1mcg0K c3VibXhpZDogMA0KbWFpbGxvZ2luOiArNTU0MTg0MDExNDk2L3R5cGU9UExN TkBtbXMuYnJhc2lsdGVsZWNvbS5jb20uYnINCmhvc3Q6IGJydGxkcDEyZQ0K b2JqZWN0Q2xhc3M6IGNvbW1vbm1haWxib3gNCm9iamVjdENsYXNzOiBpbmRl eG5vZGUNCnJlZmRuOiB1aWQ9MTIxOTg0Mzc5NjY1N1wyQjU1NDE4NDAxMTQ5 NixvdT1SRVBPU0lUT1JZLG91PUNPTlRFTlQsbz1hbGNhdGVsLGM9Zg0KIHIN Cg== --0-1644073040-1233052012=:88983 Content-Type: application/x-zip; name="index_simplex.zip" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="index_simplex.zip" UEsDBBQAAgAIAC9DOzrp93S+DBEAADUEAQASABUAaW5kZXhfc2ltcGxleC5s ZGlmVVQJAAMa4X5JGuF+SVV4BADuAwEC7Z1tb+JIFoW/8yv4PlEPVS6/pRVp e9IZKVJ3Opukd2dWI43AdtKeBZsF0y+z2f++NraBlE+RgA+ku4eWZkNwcc71 fVy36jrAhslxN52dnF+8Pvvl7PooPekPg34WDY+Ck9tJJx38EQVZMOxPp/mw yV0/if/sZ3Ga9IfvkzjrpLPjbvXSTicspd68fnV5+u7i56MWsqvHh2F/PJ0N psEkHhdjptXh0/LwNM6iOLlN57HU3p1R//OD1ySD467TK/51Gs/nT+UaSX8U HXeTNPs9jG7jJArLEwrjrDhy8vr8RhxtcXb56+fRVTrH3UKoU5zTh3RaPTWY ZMNwLGQ0f74fhpOoeKnovRBKvhD2CyXnR8bpJDvuWp5fqGVfxlGV/E4/yOKP eQDhcffm6v1Z/nuQKwRpkk3S4XH3n1fnN2f5ed/exp/X0Z6kaVacc5Cc9MNR nBytHzr+lGc/CiZRlmvPg4vC0nmUhlF+AqNpPA2To0/RYJje5XKjfjw8ikf9 8fzXRn4lK7+Sl988lJuzi5tGhvOznl+w6TiazB88MfOV3lMyv2boSubrWXd+ 3XrSFXO5VClVo4LXNJp8jCZxeNI72txlRaHzQK2YdCtPlJhGX7pBOpvE0aTb fA2mNh1lFTVpd4orq/xFKKsznqQf42l+enFyV8HJJv0kZxP276J/R1+Ouz8W jE8u37y9WKTx8urdz+dvztqnckVoIX79/qfr06vzy5vzdxfV8esjpikyKN3z bNzGw6jmSIxktZxWLp2FW8G5+qViPJoW9bU/G2adLAo+JHHQH6bjcgaJl72X wcuX9+KleBm+fNkcIPMDxQCZD4QDrGqAZRqgKguVD/ww/ymrn1b1U1U/7eqn U/10C8H8p1f83hS25zELcMSpLJ15bKJ4IPOhFhjq5kOzfKhrSoBvOiB6xiOL rFZpLR7IwgaMlfVYaZSz6iGWcYgy5kLYJZh7YdfZyB/J+SMw2qlHl7mTaIxX x+OZXcu0oZfLXvVyaUygrJMijUmRdVKkMSlSGY/Y9Ytt4xC3HmK8MqRf5Ur6 5RwAk6NnnDbmCWWcSVYxZQpDq5gz2NA2vrgmazn1jHXml8H8gWWQc+sXuaZT VLWuqnQ7wzQ/HA3jab4sRMlRXr3y7cg4r1yf0klRnoodYZjm60yyKFAvBpP+ NB7m5S4K0tGL4r9BvhZ9HsflUp/MRoNokk/5fOM4X4TzJSWqpIoFa6lerdHF k/+ZpVm/mKfKs11HL8piv0VZNIvyXToMt6rI1YVpGlDPDOPEOFTkQ0U+VGRS RVZ1Ra6KqqqLqjIWVXqNrsa6xi2YU131qpqH30iNlvut0fJhjR5Gd/3gy672 zIfKfKjMi8pc6fhGnUOJblGipeHIYa/81L1yj3YL4/Rpu+WedhNj9thNDONW 2bhF/poKcFYV4LAuwNl2BdgznddXUpnbF2RzGTYUX3zEr+N9ZFcc1iU3MJbc sC65gbHkmutpWNfTwFgsg7pYblj4VF3e9ldjTvN/5hrz94c1Jk6y4jZ0eTt1 PInm9+t/sKz5HebVu9bFTebV3xvFSey7OIlmcbobpoP+0Fib5pvFYE2Nujde QdVSfV+t3c9Qs+7rlb/cCz1Sw7KWm8hq0t57poQcdpmmQlfuMosxxSOj4lOL 3yP7zTqnUhhP9akF8pE96RN3nmG98zTsoeu9uPTKLWHxoEXlrdcHyzi3rfpq s4Rp8tb1+96yjIGv2cI+3vtb9VlbXkW7eGS6NlTNXZXcfR8Nqk9LGU+rXnzu 1eIaV4tr/LtfkOS+FyTZXJCKAOMgQiuS2Hi3vOZ2Rbjpbtd0pJ7B7nwN+eY2 t+Z1wLQHpextlyW/xY2Fp+1yt63dot7l9tbU2sPm1lBLrH3XEqtZSz7Gk2y2 ZnfL6rwNtcRwYKPG+TtrmJ+9pjxfv7zdPcVDJempfVcS1awk+RM5gSwKD50y rVM+dMiHDvnQIX+rHfL3v+7Y+153bHB71vBeq6rUGVuTunkxXpx/uSVHVDdn e9/xzdmgXnrCx5eeqmIIx5QR/S0AX9fS0/4NA5u/LeDbWnqCeukJDzdnv4fl yNn3cuSAm7Px+LAaHVajw2r0F1mN/spLjSyvW1WTWSw63/9S4+57qXG1T5kE yXfxtjl5eNvc1/K2ua/53XJr6vk2f2KsL5f9lalXr56hTEnb3vPHLOzGDZrg 93rcDj7uRvnsA3taf9OfNNjbJwEe/eAVfO++ZdwkoTuuvNlstX3HfT6/Xr1+ e37x/vrsivs1BUvZ0mn+dSDzbwFZ/EWu/B6NI0YM8y/UmOU15+HT84oVRHMc nUUE81q0GkJ5aJ60PGX/Pb369fLmf3FydWtnP/05ev+P9G5ZTUqIxecTircB F++8Kt4xUfyts7jvXDT7xaao+LKZcuTye0kKBoP0890knY3LmhRPVwPsVt9g sXiuLLH50563fHLJu1vMe0utjq9i7JZ/C9Xynl91QVDkSJBTvpLYpQfMaf+P oPfhlx8v78L0X28fz+mm+UQnPJ2No0kREeWkF2qNk9aOrD1xdDZa7PPsfpqO dxd17fAw3kpEDyevSfnQHV44lQEMRuhXSn6dHEkcotx1iBKGKDcI0dp1iBYM 0dogRLXrEBUMUW0Qor3rEG0Yor1BiM6uQ3RgiM4GIbq7DtGFIbobhOjtOkQP huhtEKK/6xB9GKK/JsTcs/pmtla7uFJjIbn4lrdWorXKQnbxdXHtdLVvnStS UHwZXbsE5AoLuUk/uYvKT6Yyktut9JA+J9PrHEhJX2fRPv0GdUFGIJA+FQF0 4CKAFjQEurokI5BIn4oAOnARQAsaAl3dIiOwkD4VAXTgIoAWNAS6uiIjUEif igA6cBFACxoCXd0mI7CRPhUBdOAigBY0BLq6Q0bgIH0qAujARQAtaAh0dZeM wEX6VATQgYsAWtAQ6OoeGYGH9KkIoAMXAbSgIdDVfTICH+lTEUAHLgJoQUOg qQtydyx6SJ+JADtQEWALFoKGOrk7FgLpUxFABy4CaEFDoKuTu2MhkT4VAXTg IoAWNAS6Ork7FhbSpyKADlwE0IKGQFcnd8dCIX0qAujARQAtaAh0dXJ3LGyk T0UAHbgIoAUNga5O7o6Fg/SpCKADFwG0oCHQ1cndsXCRPhUBdOAigBY0BLo6 uTsWHtKnIoAOXATQgoZAVyd3x8JH+lQE0IGLAFrQEGjqktwdyx7SZyLADlQE 2IKFoKFO7o6lQPpUBNCBiwBa0BDo6uTuWEqkT0UAHbgIoAUNga5O7o6lhfSp CKADFwG0oCHQ1cndsVRIn4oAOnARQAsaAl2d3B1LG+lTEUAHLgJoQUOgq5O7 Y+kgfSoC6MBFAC1oCHR1cncsXaRPRQAduAigBQ2Brk7ujqWH9KkIoAMXAbSg IdDVyd2x9JE+FQF04CKAFjQEmrpF7o6tHtJnIsAOVATYgoWgoU7uji2B9KkI oAMXAbSgIdDVyd2xJZE+FQF04CKAFjQEujq5O7YspE9FAB24CKAFDYGuTu6O LYX0qQigAxcBtKAh0NXJ3bFlI30qAujARQAtaAh0dXJ3bDlIn4oAOnARQAsa Al2d3B1bLtKnIoAOXATQgoZAVyd3x5aH9KkIoAMXAbSgIdDVyd2x5SN9KgLo wEUALWgINHVF7o5VD+kzEWAHKgJswULQUCd3x0ogfSoC6MBFAC1oCHR1cnes JNKnIoAOXATQgoZAVyd3x8pC+lQE0IGLAFrQEOjq5O5YKaRPRQAduAigBQ2B rk7ujpWN9KkIoAMXAbSgIdDVyd2xcpA+FQF04CKAFjQEujq5O1Yu0qcigA5c BNCChkBXJ3fHykP6VATQgYsAWtAQ6Ork7lj5SJ+KADpwEUALGgJN3SZ3x3YP 6TMRYAcqAmzBQtBQJ3fHtkD6VATQgYsAWtAQ6Ork7tiWSJ+KADpwEUALGgJd ndwd2xbSpyKADlwE0IKGQFcnd8e2QvpUBNCBiwBa0BDo6uTu2LaRPhUBdOAi gBY0BLo6uTu2HaRPRQAduAigBQ2Brk7ujm0X6VMRQAcuAmhBQ6Crk7tj20P6 VATQgYsAWtAQ6Ork7tj2kT4VAXTgIoAWNASaukPujp0e0mciwA5UBNiChaCh Tu6OHYH0qQigAxcBtKAh0NXJ3bEjkT4VAXTgIoAWNAS6Ork7diykT0UAHbgI oAUNga5O7o4dhfSpCKADFwG0oCHQ1cndsWMjfSoC6MBFAC1oCHR1cnfsOEif igA6cBFACxoCXZ3cHTsu0qcigA5cBNCChkBXJ3fHjof0qQigAxcBtKAh0NXJ 3bHjI30qAujARQAtaAg0dZfcHbs9pM9EgB2oCLAFC0FDndwduwLpUxFABy4C aEFDoKuTu2NXIn0qAujARQAtaAh0dXJ37FpIn4oAOnARQAsaAl2d3B27CulT EUAHLgJoQUOgq5O7Y9dG+lQE0IGLAFrQEOjq5O7YdZA+FQF04CKAFjQEujq5 O3ZdpE9FAB24CKAFDYGuTu6OXQ/pUxFABy4CaEFDoKuTu2PXR/pUBNCBiwBa 0BBo6h65O/Z6SJ+JADtQEWALFoKGOrk79gTSpyKADlwE0IKGQFcnd8eeRPpU BNCBiwBa0BDo6uTu2LOQPhUBdOAigBY0BLo6uTv2FNKnIoAOXATQgoZAVyd3 x56N9KkIoAMXAbSgIdDVyd2x5yB9KgLowEUALWgIdHVyd+y5SJ+KADpwEUAL GgJdndwdex7SpyKADlwE0IKGQFcnd8eej/SpCKADFwG0oCHQ1H1yd+z3kD4T AXagIsAWLAQNdXJ37AukT0UAHbgIoAUNga5O7o59ifSpCKADFwG0oCHQ1cnd sW8hfSoC6MBFAC1oCHR1cnfsK6RPRQAduAigBQ2Brk7ujn0b6VMRQAcuAmhB Q6Crk7tj30H6VATQgYsAWtAQ6Ork7th3kT4VAXTgIoAWNAS6Ork79j2kT0UA HbgIoAUNga5O7o59H+lTEUAHLgJoQUNQqY+TZDYaRJOT337oLf4d1Z69LXrm 8Yc0ieLkNn34dJyE0eckDaNObXncXbHsTGeDcRKHx91e50M6zY67g0k2DMdC Rp1JdFuEOovDk3k8V2eX767Pb95d/Vr8dvru4ubs4kaPaX52RbZORqNp8Lf8 f14MJv1pPMxHREE6elH8N5gcbf934VxglCazaTQxnWmhlgexzr8T5GFOv0yz aNTpB0E6S7Jp1s9muY4gJqK+9jHmzf/vUx8/99rxIeX9nG1xJuXpbsL+6bP3 dDUJhdkg/fzw0DIP+XU9+lxe14uwHrsmOJnQ5rf8qaYgtp3hp3iGo3Ou5zKY 7Ln9ujOUvZ4npHBkT0mh8uHNyB87/04XXfk4ARu+2f3UcP2fPuH6F9UEePBE NQ2unzINyKmZV8eHEj9mX8bRbyeXb95ePGHagBWp022buLJo/qAHtT6mry6v 6JrbIrmLmtTpUqvSSinaNNMrBW0/KV1Tx8SyjolnqGMbXVSyGXn7OrZMwIaf GGlXx4Q+30Sr+dYyNc06JjaaauiPHjuoY2KLOvbseUXX3BbJ3W8dE7urYy1S uqaOyWUdk89Qx+QmGbCakbevY8sEbPixq3Z1TOrzTbaaby1T06xjcrOpJvZS x+QWdezZ84quuS2Su986JndXx1qkdE0ds5Z1zHqGOmZtkgHVjLx9HVsmYMPP LrarY5Y+36xW861lapp1zNpsqsm91DFrizr27HlF19wWyd1vHbN2V8dapHRN HVPLOqaeoY6pTTJgNyNvX8eWCdjwA8Dt6pjS55tqNd9apqZZx9RmU83aSx1T W9SxZ88ruua2SO5+65jaXR1rkdLO/wFQSwECFwMUAAIACAAvQzs66fd0vgwR AAA1BAEAEgANAAAAAAABAAAApIEAAAAAaW5kZXhfc2ltcGxleC5sZGlmVVQF AAMa4X5JVXgAAFBLBQYAAAAAAQABAE0AAABREQAAAAA= --0-1644073040-1233052012=:88983--

1 0

Re: (ITS#5540) Normalization assertion in attr.c
by hyc＠symas.com 27 Jan '09

27 Jan '09

unix.gurus(a)gmail.com wrote: > Hi, > > On Tue, Jun 24, 2008 at 12:50 PM, Sean Burford<unix.gurus(a)gmail.com> wrote: >> namingContexts was the trickiest, since RFC 4512 defines it without an >> equality matching rule but monitor_back_register_database() compares >> it with backend suffixes. It has to deal with capitalised values >> properly otherwise we end up with duplicate namingContexts entries, so >> normalization is required. > > I have uploaded > ftp://ftp.openldap.org/incoming/sean-burford-monitor-test-080706.patch > which adds a test for various things under cn=monitor. If duplicate > namingContexts creep in (as mentioned above) it detects that. > > The duplicate naming contexts thing was fixed by comparing a_vals with > be_suffix, I just wrote the test since there was no existing > cn=monitor test. Sorry for dropping the ball on this. These last two patches are now committed in HEAD. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by hyc＠symas.com 27 Jan '09

27 Jan '09

rlvcosta(a)yahoo.com wrote: > Quanah, > > In this specific test I reported in my previous e-mail it was used glibc. > > Now I tested with tcmalloc and the memory again was much more consumed and never released. > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 10652 ldap 18 0 2006m 1.8g 68m S 100 15.8 8:36.75 slapd > > Once it get this much memory it then stabilize but it is much more than before that was 385m. We may need to have sample data from you to reproduce this situation. I haven't seen anything like it on my tests. Running two searches simultaneously still behaves normally for me. > > The scenario was : Likewise, the exact commands and arguments will be needed... > > 1) Start a full ldapsearch for one specific dn(the DB has 4 dn's); > 2) After sometime start a new full ldapsearch to behave as queries to DB when replication for LDIF backup is being done; > 3) Export one of the queries to a file so speed verification can be done with grep and wc. > > After sometime the memory starts to be consumed in large chunks until it stopped in around 2G. > > Since one of the queries is being redirect to a file I can tail to have a visual rate of the entrances being returned by slapd. As the issue before after sometime the rate appears to get stuck and come in small 16 entrances chunks but only after some minutes. > > This behavior above appears to happen after the second ldapsearch fills its own dn cache even monitor always shows only one information as I could see. > > [root@brtldp11 ~]# date; grep -e '^pnnum*' /backup/temp.ldif |wc -l > Mon Jan 26 21:43:31 BRST 2009 > 482432 > [root@brtldp11 ~]# date; grep -e '^pnnum*' /backup/temp.ldif |wc -l > Mon Jan 26 21:44:56 BRST 2009 > 482448 > > This behavior is similar as before with the initial issue with a single process. > > I just notice when slapd starts there are these threads : > > [root@brtldp12 ~]# ps -mu ldap > PID TTY TIME CMD > 10638 pts/2 00:00:00 slapd > - - 00:00:00 - > - - 00:00:00 - > > Then after the 2 queries are started I have : > > [root@brtldp12 ~]# ps -mu ldap > PID TTY TIME CMD > 10195 pts/2 00:44:22 slapd > - - 00:00:00 - > - - 00:00:00 - > - - 00:26:09 - > - - 00:18:13 - > - - 00:00:00 - > > So 3 new threads. Looks like for some reason with multiples queries the issue with response getting stuck repeats. > > Even one ldapsearch is stopped the number of threads continues the same, memory continues allocated and the other search get with this slow rate. > > These outputs are using now tcmalloc but similar as with glibc. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by hyc＠symas.com 27 Jan '09

27 Jan '09

Rodrigo Costa wrote: > Howard, > I saw sometimes the DN cache allocating a little more memory since it sometimes pass a little the boundary, like seen below : > olmBDBEntryCache: 1000 > olmBDBDNCache: 1000247 > olmBDBIDLCache: 1 > > The maximum I saw was 1000268 but nothing considerable. In this way now slapd is keeping memory allocation stable as it is supposed to behave. The DN cache is hierarchical; it will only free leaf nodes (entries that have no children). So depending on the structure of your tree, it's normal for the Monitor to report more elements in the DN cache than you configured. > > The only comment is if dn cache is passed this memory allocated appears to never be released. The maximum size slapd obtained was : > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 10195 ldap 18 0 385m 266m 68m S 0 2.2 20:54.16 slapd > > This is more than perfect ! > > Thanks for all your quick solution of this problem. I believe many users > of openLDAP will see their system more stable with this fix. > > Just some last questions. Is there any expectation for a new release, for example 2.4.15, with this ITS included? Just to use a formal release. > > By the tests I was expecting the cachefree as the cache size bringing some improvements, at least in a full query, since system would "cycle" pointer memory and overwritten all cache at once. What is the purpose of cachefree? cachefree applies only to the entry cache. The idea is to minimize the number of times it must cycle through the cache trying to free entries. Typically when we configure large servers with caches of 1 million entries or more, we configure this to 1000 or so (0.1% of cache size). -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by rlvcosta＠yahoo.com 27 Jan '09

27 Jan '09

Quanah, In this specific test I reported in my previous e-mail it was used glibc. Now I tested with tcmalloc and the memory again was much more consumed and never released. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10652 ldap 18 0 2006m 1.8g 68m S 100 15.8 8:36.75 slapd Once it get this much memory it then stabilize but it is much more than before that was 385m. The scenario was : 1) Start a full ldapsearch for one specific dn(the DB has 4 dn's); 2) After sometime start a new full ldapsearch to behave as queries to DB when replication for LDIF backup is being done; 3) Export one of the queries to a file so speed verification can be done with grep and wc. After sometime the memory starts to be consumed in large chunks until it stopped in around 2G. Since one of the queries is being redirect to a file I can tail to have a visual rate of the entrances being returned by slapd. As the issue before after sometime the rate appears to get stuck and come in small 16 entrances chunks but only after some minutes. This behavior above appears to happen after the second ldapsearch fills its own dn cache even monitor always shows only one information as I could see. [root@brtldp11 ~]# date; grep -e '^pnnum*' /backup/temp.ldif |wc -l Mon Jan 26 21:43:31 BRST 2009 482432 [root@brtldp11 ~]# date; grep -e '^pnnum*' /backup/temp.ldif |wc -l Mon Jan 26 21:44:56 BRST 2009 482448 This behavior is similar as before with the initial issue with a single process. I just notice when slapd starts there are these threads : [root@brtldp12 ~]# ps -mu ldap PID TTY TIME CMD 10638 pts/2 00:00:00 slapd - - 00:00:00 - - - 00:00:00 - Then after the 2 queries are started I have : [root@brtldp12 ~]# ps -mu ldap PID TTY TIME CMD 10195 pts/2 00:44:22 slapd - - 00:00:00 - - - 00:00:00 - - - 00:26:09 - - - 00:18:13 - - - 00:00:00 - So 3 new threads. Looks like for some reason with multiples queries the issue with response getting stuck repeats. Even one ldapsearch is stopped the number of threads continues the same, memory continues allocated and the other search get with this slow rate. These outputs are using now tcmalloc but similar as with glibc. Regards, Rodrigo. --- On Mon, 1/26/09, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > From: Quanah Gibson-Mount <quanah(a)zimbra.com> > Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 > To: rlvcosta(a)yahoo.com, openldap-its(a)openldap.org > Date: Monday, January 26, 2009, 9:25 PM > --On Monday, January 26, 2009 3:18 PM -0800 Rodrigo Costa > <rlvcosta(a)yahoo.com> wrote: > > > Quanah and Howard, > > > > Sorry to come back. > > > > I tried to make 2 ldapsearch to the same DB > simultaneously, doing some > > stress test like dump the LDIF and queries coming. > > > > The slapd after sometime start to get more memory. By > what I could > > observe looks like new thread was started consuming > again the same memory > > as before so the slad now is consuming around the > double of memory as > > before. > > Are you also using an alternate malloc() instead of glibc? > I know you were > investigating this, just want to confirm you are using > tcmalloc or hoard > and not glibc. > > --Quanah > > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and > collaboration

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs