--0-1644073040-1233052012=:88983 Content-Type: text/plain; charset=us-ascii
Howard,
Initially the search behaves normally but after some time something changes and memory starts to be consumed again until it reaches another stable condition, even very large(around 2G) and the search get stuck as before.
Once search get stuck even I stop both searches and start a new one it get stuck. So in my test I stopped my 2 searches and started a new one. Then I saw the new search with only 240 returns and then get stuck entering in the same model as before with 16 entrances in bursts in order of minutes to enter in these bursts.
As I tried to explain this appears to happen when the seconds ldapsearch, started at a later time than the first ldapsearch, reaches the dn cachesize. This is what appears to happen.
I beleive for you to reproduce you need to have a DB large enough and a slapd.conf where dn cachesize will overpass the memory boundary imposed. Then after the second ldapsearch memory boundary is reached the issue should appear.
The commands I runned were :
time ldapsearch -LLL -x -D "cn=admin,ou=INDEXES,o=alcatel,c=fr" -w secret -b "ou=INDEXES,o=alcatel,c=fr" -H ldap://10.142.15.42:389 'pnnumber=*' > /backup/temp.ldif
and
time ldapsearch -LLL -x -D "cn=admin,ou=INDEXES,o=alcatel,c=fr" -w secret -b "ou=INDEXES,o=alcatel,c=fr" -H ldap://10.142.15.42:389 'pnnumber=*'| egrep -e '^pnnumber*' |wc -l
With the first command, redirecting to a file, I can tail or wc and see it's progress. After the second ldapsearch process starts to make slapd consume memory in large chunks until the new 2G boundary is reached. Once this happen the search get stuck as before with the single process returning 16 records per minute(s).
Once this condition is reached even stopping the old searches and starting new ones the rate will be very slow, like 16 reports per minute(s). It get stuck as before.
Please see attached a single entrance in my DB where you can change DB keys and load it as needed.
Thanks,
Rodrigo.
--- On Mon, 1/26/09, Howard Chu hyc@symas.com wrote:
From: Howard Chu hyc@symas.com Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 To: rlvcosta@yahoo.com Cc: openldap-its@openldap.org Date: Monday, January 26, 2009, 11:16 PM rlvcosta@yahoo.com wrote:
Quanah,
In this specific test I reported in my previous e-mail
it was used glibc.
Now I tested with tcmalloc and the memory again was
much more consumed and never released.
PID USER PR NI VIRT RES SHR S %CPU %MEM
TIME+ COMMAND
10652 ldap 18 0 2006m 1.8g 68m S 100 15.8
8:36.75 slapd
Once it get this much memory it then stabilize but it
is much more than before that was 385m.
We may need to have sample data from you to reproduce this situation. I haven't seen anything like it on my tests. Running two searches simultaneously still behaves normally for me.
The scenario was :
Likewise, the exact commands and arguments will be needed...
- Start a full ldapsearch for one specific dn(the DB
has 4 dn's);
- After sometime start a new full ldapsearch to
behave as queries to DB when replication for LDIF backup is being done;
- Export one of the queries to a file so speed
verification can be done with grep and wc.
After sometime the memory starts to be consumed in
large chunks until it stopped in around 2G.
Since one of the queries is being redirect to a file I
can tail to have a visual rate of the entrances being returned by slapd. As the issue before after sometime the rate appears to get stuck and come in small 16 entrances chunks but only after some minutes.
This behavior above appears to happen after the second
ldapsearch fills its own dn cache even monitor always shows only one information as I could see.
[root@brtldp11 ~]# date; grep -e '^pnnum*'
/backup/temp.ldif |wc -l
Mon Jan 26 21:43:31 BRST 2009 482432 [root@brtldp11 ~]# date; grep -e '^pnnum*'
/backup/temp.ldif |wc -l
Mon Jan 26 21:44:56 BRST 2009 482448
This behavior is similar as before with the initial
issue with a single process.
I just notice when slapd starts there are these
threads :
[root@brtldp12 ~]# ps -mu ldap PID TTY TIME CMD 10638 pts/2 00:00:00 slapd - - 00:00:00 - - - 00:00:00 -
Then after the 2 queries are started I have :
[root@brtldp12 ~]# ps -mu ldap PID TTY TIME CMD 10195 pts/2 00:44:22 slapd - - 00:00:00 - - - 00:00:00 - - - 00:26:09 - - - 00:18:13 - - - 00:00:00 -
So 3 new threads. Looks like for some reason with
multiples queries the issue with response getting stuck repeats.
Even one ldapsearch is stopped the number of threads
continues the same, memory continues allocated and the other search get with this slow rate.
These outputs are using now tcmalloc but similar as
with glibc.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
--0-1644073040-1233052012=:88983 Content-Type: text/plain; name="one_INDEXES_entrance.txt" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="one_INDEXES_entrance.txt"
ZG46IHBubnVtYmVyPVwyQjU1NDE4NDAxMTQ5NixvdT1yYW5nZTk2LG91PW1z aXNkbixvdT1JTkRFWEVTLG89YWxjYXRlbCxjPWZyDQpzdWJwbmlkOiAwDQpo b3N0OiBicnRsZHAxMmUNCm9iamVjdENsYXNzOiBwaG9uZWluZm8NCm9iamVj dENsYXNzOiBpbmRleG5vZGUNCnJlZmRuOiB1aWQ9MTIxOTg0Mzc5NjY1N1wy QjU1NDE4NDAxMTQ5NixvdT1SRVBPU0lUT1JZLG91PUNPTlRFTlQsbz1hbGNh dGVsLGM9Zg0KIHINCnBubnVtYmVyOiArNTU0MTg0MDExNDk2DQoNCmRuOiB3 ZWJsb2dpbj1cMkI1NTQxODQwMTE0OTYsb3U9cmFuZ2U2NyxvdT13ZWJsb2dp bixvdT1JTkRFWEVTLG89YWxjYXRlbCxjPWZyDQp3ZWJsb2dpbjogKzU1NDE4 NDAxMTQ5Ng0KaG9zdDogYnJ0bGRwMTJlDQphY2NvdW50U3RhdHVzOiAxDQpv YmplY3RDbGFzczogY29tbW9udXNlcg0Kb2JqZWN0Q2xhc3M6IGluZGV4bm9k ZQ0KcmVmZG46IHVpZD0xMjE5ODQzNzk2NjU3XDJCNTU0MTg0MDExNDk2LG91 PVJFUE9TSVRPUlksb3U9Q09OVEVOVCxvPWFsY2F0ZWwsYz1mDQogcg0KY246 ICs1NTQxODQwMTE0OTYNCg0KZG46IG1haWw9XDJCNTU0MTg0MDExNDk2L3R5 cGVcM0RQTE1OQG1tcy5icmFzaWx0ZWxlY29tLmNvbS5icixvdT1yYW5nZTg0 LG91PW1hDQogaWwsb3U9SU5ERVhFUyxvPWFsY2F0ZWwsYz1mcg0KbWFpbDog KzU1NDE4NDAxMTQ5Ni90eXBlPVBMTU5AbW1zLmJyYXNpbHRlbGVjb20uY29t LmJyDQpob3N0OiBicnRsZHAxMmUNCmFjY291bnRTdGF0dXM6IDENCm9iamVj dENsYXNzOiBjb21tb251c2VyDQpvYmplY3RDbGFzczogaW5kZXhub2RlDQpy ZWZkbjogdWlkPTEyMTk4NDM3OTY2NTdcMkI1NTQxODQwMTE0OTYsb3U9UkVQ T1NJVE9SWSxvdT1DT05URU5ULG89YWxjYXRlbCxjPWYNCiByDQpjbjogKzU1 NDE4NDAxMTQ5Ng0KDQpkbjogbWFpbGxvZ2luPVwyQjU1NDE4NDAxMTQ5Ni90 eXBlXDNEUExNTkBtbXMuYnJhc2lsdGVsZWNvbS5jb20uYnIsb3U9cmFuZ2U4 NCwNCiBvdT1pbWFwbG9naW4sb3U9SU5ERVhFUyxvPWFsY2F0ZWwsYz1mcg0K c3VibXhpZDogMA0KbWFpbGxvZ2luOiArNTU0MTg0MDExNDk2L3R5cGU9UExN TkBtbXMuYnJhc2lsdGVsZWNvbS5jb20uYnINCmhvc3Q6IGJydGxkcDEyZQ0K b2JqZWN0Q2xhc3M6IGNvbW1vbm1haWxib3gNCm9iamVjdENsYXNzOiBpbmRl eG5vZGUNCnJlZmRuOiB1aWQ9MTIxOTg0Mzc5NjY1N1wyQjU1NDE4NDAxMTQ5 NixvdT1SRVBPU0lUT1JZLG91PUNPTlRFTlQsbz1hbGNhdGVsLGM9Zg0KIHIN Cg==
--0-1644073040-1233052012=:88983 Content-Type: application/x-zip; name="index_simplex.zip" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="index_simplex.zip"
UEsDBBQAAgAIAC9DOzrp93S+DBEAADUEAQASABUAaW5kZXhfc2ltcGxleC5s ZGlmVVQJAAMa4X5JGuF+SVV4BADuAwEC7Z1tb+JIFoW/8yv4PlEPVS6/pRVp e9IZKVJ3Opukd2dWI43AdtKeBZsF0y+z2f++NraBlE+RgA+ku4eWZkNwcc71 fVy36jrAhslxN52dnF+8Pvvl7PooPekPg34WDY+Ck9tJJx38EQVZMOxPp/mw yV0/if/sZ3Ga9IfvkzjrpLPjbvXSTicspd68fnV5+u7i56MWsqvHh2F/PJ0N psEkHhdjptXh0/LwNM6iOLlN57HU3p1R//OD1ySD467TK/51Gs/nT+UaSX8U HXeTNPs9jG7jJArLEwrjrDhy8vr8RhxtcXb56+fRVTrH3UKoU5zTh3RaPTWY ZMNwLGQ0f74fhpOoeKnovRBKvhD2CyXnR8bpJDvuWp5fqGVfxlGV/E4/yOKP eQDhcffm6v1Z/nuQKwRpkk3S4XH3n1fnN2f5ed/exp/X0Z6kaVacc5Cc9MNR nBytHzr+lGc/CiZRlmvPg4vC0nmUhlF+AqNpPA2To0/RYJje5XKjfjw8ikf9 8fzXRn4lK7+Sl988lJuzi5tGhvOznl+w6TiazB88MfOV3lMyv2boSubrWXd+ 3XrSFXO5VClVo4LXNJp8jCZxeNI72txlRaHzQK2YdCtPlJhGX7pBOpvE0aTb fA2mNh1lFTVpd4orq/xFKKsznqQf42l+enFyV8HJJv0kZxP276J/R1+Ouz8W jE8u37y9WKTx8urdz+dvztqnckVoIX79/qfr06vzy5vzdxfV8esjpikyKN3z bNzGw6jmSIxktZxWLp2FW8G5+qViPJoW9bU/G2adLAo+JHHQH6bjcgaJl72X wcuX9+KleBm+fNkcIPMDxQCZD4QDrGqAZRqgKguVD/ww/ymrn1b1U1U/7eqn U/10C8H8p1f83hS25zELcMSpLJ15bKJ4IPOhFhjq5kOzfKhrSoBvOiB6xiOL rFZpLR7IwgaMlfVYaZSz6iGWcYgy5kLYJZh7YdfZyB/J+SMw2qlHl7mTaIxX x+OZXcu0oZfLXvVyaUygrJMijUmRdVKkMSlSGY/Y9Ytt4xC3HmK8MqRf5Ur6 5RwAk6NnnDbmCWWcSVYxZQpDq5gz2NA2vrgmazn1jHXml8H8gWWQc+sXuaZT VLWuqnQ7wzQ/HA3jab4sRMlRXr3y7cg4r1yf0klRnoodYZjm60yyKFAvBpP+ NB7m5S4K0tGL4r9BvhZ9HsflUp/MRoNokk/5fOM4X4TzJSWqpIoFa6lerdHF k/+ZpVm/mKfKs11HL8piv0VZNIvyXToMt6rI1YVpGlDPDOPEOFTkQ0U+VGRS RVZ1Ra6KqqqLqjIWVXqNrsa6xi2YU131qpqH30iNlvut0fJhjR5Gd/3gy672 zIfKfKjMi8pc6fhGnUOJblGipeHIYa/81L1yj3YL4/Rpu+WedhNj9thNDONW 2bhF/poKcFYV4LAuwNl2BdgznddXUpnbF2RzGTYUX3zEr+N9ZFcc1iU3MJbc sC65gbHkmutpWNfTwFgsg7pYblj4VF3e9ldjTvN/5hrz94c1Jk6y4jZ0eTt1 PInm9+t/sKz5HebVu9bFTebV3xvFSey7OIlmcbobpoP+0Fib5pvFYE2Nujde QdVSfV+t3c9Qs+7rlb/cCz1Sw7KWm8hq0t57poQcdpmmQlfuMosxxSOj4lOL 3yP7zTqnUhhP9akF8pE96RN3nmG98zTsoeu9uPTKLWHxoEXlrdcHyzi3rfpq s4Rp8tb1+96yjIGv2cI+3vtb9VlbXkW7eGS6NlTNXZXcfR8Nqk9LGU+rXnzu 1eIaV4tr/LtfkOS+FyTZXJCKAOMgQiuS2Hi3vOZ2Rbjpbtd0pJ7B7nwN+eY2 t+Z1wLQHpextlyW/xY2Fp+1yt63dot7l9tbU2sPm1lBLrH3XEqtZSz7Gk2y2 ZnfL6rwNtcRwYKPG+TtrmJ+9pjxfv7zdPcVDJempfVcS1awk+RM5gSwKD50y rVM+dMiHDvnQIX+rHfL3v+7Y+153bHB71vBeq6rUGVuTunkxXpx/uSVHVDdn e9/xzdmgXnrCx5eeqmIIx5QR/S0AX9fS0/4NA5u/LeDbWnqCeukJDzdnv4fl yNn3cuSAm7Px+LAaHVajw2r0F1mN/spLjSyvW1WTWSw63/9S4+57qXG1T5kE yXfxtjl5eNvc1/K2ua/53XJr6vk2f2KsL5f9lalXr56hTEnb3vPHLOzGDZrg 93rcDj7uRvnsA3taf9OfNNjbJwEe/eAVfO++ZdwkoTuuvNlstX3HfT6/Xr1+ e37x/vrsivs1BUvZ0mn+dSDzbwFZ/EWu/B6NI0YM8y/UmOU15+HT84oVRHMc nUUE81q0GkJ5aJ60PGX/Pb369fLmf3FydWtnP/05ev+P9G5ZTUqIxecTircB F++8Kt4xUfyts7jvXDT7xaao+LKZcuTye0kKBoP0890knY3LmhRPVwPsVt9g sXiuLLH50563fHLJu1vMe0utjq9i7JZ/C9Xynl91QVDkSJBTvpLYpQfMaf+P oPfhlx8v78L0X28fz+mm+UQnPJ2No0kREeWkF2qNk9aOrD1xdDZa7PPsfpqO dxd17fAw3kpEDyevSfnQHV44lQEMRuhXSn6dHEkcotx1iBKGKDcI0dp1iBYM 0dogRLXrEBUMUW0Qor3rEG0Yor1BiM6uQ3RgiM4GIbq7DtGFIbobhOjtOkQP huhtEKK/6xB9GKK/JsTcs/pmtla7uFJjIbn4lrdWorXKQnbxdXHtdLVvnStS UHwZXbsE5AoLuUk/uYvKT6Yyktut9JA+J9PrHEhJX2fRPv0GdUFGIJA+FQF0 4CKAFjQEurokI5BIn4oAOnARQAsaAl3dIiOwkD4VAXTgIoAWNAS6uiIjUEif igA6cBFACxoCXd0mI7CRPhUBdOAigBY0BLq6Q0bgIH0qAujARQAtaAh0dZeM wEX6VATQgYsAWtAQ6OoeGYGH9KkIoAMXAbSgIdDVfTICH+lTEUAHLgJoQUOg qQtydyx6SJ+JADtQEWALFoKGOrk7FgLpUxFABy4CaEFDoKuTu2MhkT4VAXTg IoAWNAS6Ork7FhbSpyKADlwE0IKGQFcnd8dCIX0qAujARQAtaAh0dXJ3LGyk T0UAHbgIoAUNga5O7o6Fg/SpCKADFwG0oCHQ1cndsXCRPhUBdOAigBY0BLo6 uTsWHtKnIoAOXATQgoZAVyd3x8JH+lQE0IGLAFrQEGjqktwdyx7SZyLADlQE 2IKFoKFO7o6lQPpUBNCBiwBa0BDo6uTuWEqkT0UAHbgIoAUNga5O7o6lhfSp CKADFwG0oCHQ1cndsVRIn4oAOnARQAsaAl2d3B1LG+lTEUAHLgJoQUOgq5O7 Y+kgfSoC6MBFAC1oCHR1cncsXaRPRQAduAigBQ2Brk7ujqWH9KkIoAMXAbSg IdDVyd2x9JE+FQF04CKAFjQEmrpF7o6tHtJnIsAOVATYgoWgoU7uji2B9KkI oAMXAbSgIdDVyd2xJZE+FQF04CKAFjQEujq5O7YspE9FAB24CKAFDYGuTu6O LYX0qQigAxcBtKAh0NXJ3bFlI30qAujARQAtaAh0dXJ3bDlIn4oAOnARQAsa Al2d3B1bLtKnIoAOXATQgoZAVyd3x5aH9KkIoAMXAbSgIdDVyd2x5SN9KgLo wEUALWgINHVF7o5VD+kzEWAHKgJswULQUCd3x0ogfSoC6MBFAC1oCHR1cnes JNKnIoAOXATQgoZAVyd3x8pC+lQE0IGLAFrQEOjq5O5YKaRPRQAduAigBQ2B rk7ujpWN9KkIoAMXAbSgIdDVyd2xcpA+FQF04CKAFjQEujq5O1Yu0qcigA5c BNCChkBXJ3fHykP6VATQgYsAWtAQ6Ork7lj5SJ+KADpwEUALGgJN3SZ3x3YP 6TMRYAcqAmzBQtBQJ3fHtkD6VATQgYsAWtAQ6Ork7tiWSJ+KADpwEUALGgJd ndwd2xbSpyKADlwE0IKGQFcnd8e2QvpUBNCBiwBa0BDo6uTu2LaRPhUBdOAi gBY0BLo6uTu2HaRPRQAduAigBQ2Brk7ujm0X6VMRQAcuAmhBQ6Crk7tj20P6 VATQgYsAWtAQ6Ork7tj2kT4VAXTgIoAWNASaukPujp0e0mciwA5UBNiChaCh Tu6OHYH0qQigAxcBtKAh0NXJ3bEjkT4VAXTgIoAWNAS6Ork7diykT0UAHbgI oAUNga5O7o4dhfSpCKADFwG0oCHQ1cndsWMjfSoC6MBFAC1oCHR1cnfsOEif igA6cBFACxoCXZ3cHTsu0qcigA5cBNCChkBXJ3fHjof0qQigAxcBtKAh0NXJ 3bHjI30qAujARQAtaAg0dZfcHbs9pM9EgB2oCLAFC0FDndwduwLpUxFABy4C aEFDoKuTu2NXIn0qAujARQAtaAh0dXJ37FpIn4oAOnARQAsaAl2d3B27CulT EUAHLgJoQUOgq5O7Y9dG+lQE0IGLAFrQEOjq5O7YdZA+FQF04CKAFjQEujq5 O3ZdpE9FAB24CKAFDYGuTu6OXQ/pUxFABy4CaEFDoKuTu2PXR/pUBNCBiwBa 0BBo6h65O/Z6SJ+JADtQEWALFoKGOrk79gTSpyKADlwE0IKGQFcnd8eeRPpU BNCBiwBa0BDo6uTu2LOQPhUBdOAigBY0BLo6uTv2FNKnIoAOXATQgoZAVyd3 x56N9KkIoAMXAbSgIdDVyd2x5yB9KgLowEUALWgIdHVyd+y5SJ+KADpwEUAL GgJdndwdex7SpyKADlwE0IKGQFcnd8eej/SpCKADFwG0oCHQ1H1yd+z3kD4T AXagIsAWLAQNdXJ37AukT0UAHbgIoAUNga5O7o59ifSpCKADFwG0oCHQ1cnd sW8hfSoC6MBFAC1oCHR1cnfsK6RPRQAduAigBQ2Brk7ujn0b6VMRQAcuAmhB Q6Crk7tj30H6VATQgYsAWtAQ6Ork7th3kT4VAXTgIoAWNAS6Ork79j2kT0UA HbgIoAUNga5O7o59H+lTEUAHLgJoQUNQqY+TZDYaRJOT337oLf4d1Z69LXrm 8Yc0ieLkNn34dJyE0eckDaNObXncXbHsTGeDcRKHx91e50M6zY67g0k2DMdC Rp1JdFuEOovDk3k8V2eX767Pb95d/Vr8dvru4ubs4kaPaX52RbZORqNp8Lf8 f14MJv1pPMxHREE6elH8N5gcbf934VxglCazaTQxnWmhlgexzr8T5GFOv0yz aNTpB0E6S7Jp1s9muY4gJqK+9jHmzf/vUx8/99rxIeX9nG1xJuXpbsL+6bP3 dDUJhdkg/fzw0DIP+XU9+lxe14uwHrsmOJnQ5rf8qaYgtp3hp3iGo3Ou5zKY 7Ln9ujOUvZ4npHBkT0mh8uHNyB87/04XXfk4ARu+2f3UcP2fPuH6F9UEePBE NQ2unzINyKmZV8eHEj9mX8bRbyeXb95ePGHagBWp022buLJo/qAHtT6mry6v 6JrbIrmLmtTpUqvSSinaNNMrBW0/KV1Tx8SyjolnqGMbXVSyGXn7OrZMwIaf GGlXx4Q+30Sr+dYyNc06JjaaauiPHjuoY2KLOvbseUXX3BbJ3W8dE7urYy1S uqaOyWUdk89Qx+QmGbCakbevY8sEbPixq3Z1TOrzTbaaby1T06xjcrOpJvZS x+QWdezZ84quuS2Su986JndXx1qkdE0ds5Z1zHqGOmZtkgHVjLx9HVsmYMPP LrarY5Y+36xW861lapp1zNpsqsm91DFrizr27HlF19wWyd1vHbN2V8dapHRN HVPLOqaeoY6pTTJgNyNvX8eWCdjwA8Dt6pjS55tqNd9apqZZx9RmU83aSx1T W9SxZ88ruua2SO5+65jaXR1rkdLO/wFQSwECFwMUAAIACAAvQzs66fd0vgwR AAA1BAEAEgANAAAAAAABAAAApIEAAAAAaW5kZXhfc2ltcGxleC5sZGlmVVQF AAMa4X5JVXgAAFBLBQYAAAAAAQABAE0AAABREQAAAAA=
--0-1644073040-1233052012=:88983--
-
rlvcosta＠yahoo.com