On 06/18/2012 01:55 PM, hyc@symas.com wrote:

Jan Synacek wrote:

...

On 06/15/2012 03:16 PM, hyc@symas.com wrote:

...

jsynacek@redhat.com wrote:

...

Full_Name: Jan Synacek Version: git (c73ec15) OS: linux-fedora17 URL: http://jsynacek.fedorapeople.org/openldap/leak/openldap-mmr-leak.tar.gz Submission from: (NULL) (209.132.186.34)

I'm using a 2-node mmr setup on my local machine - configuration files and 'uploader' scripts are provided in the archive.

1. I have the two nodes running.
2. Execute run.sh (only a wrapper for ldapusradm.sh) and start monitoring

slapd's memory usage. 3) After some time (at about 2k users on my system), slapd consumes a large amount of memory which is still growing

Note that not using ldapmodify to add members to 'cn=users,dc=yes,dc=my', but using it e.g. for modifying each user's email, does NOT result in any memory leakage.

I have also created a massif output using valgrind's massif tool: http://jsynacek.fedorapeople.org/openldap/leak/massif.out.17906

I found a very similar bug (#7292), but I'm not sure if it's related.

Running RE24 with valgrind --leak-check=full I see no leak when running your test. That should be the same as git c73ec15. No idea what leak you're seeing.

The memory consumption still grows and slapd eventually gets killed by oom-killer.

Perhaps you're seeing fragmentation in the glibc malloc library. Please test RE24, and try with some other malloc (e.g. tcmalloc). I've run your entire test with both valgrind and with tcmalloc's heap checker and neither one shows any leak.

I've done more tests and have some additional observations to add:

* tcmalloc doesn't make any difference

* if syncing finishes, memory usage drops considerably

* putting sleep between ldapadd and ldapmodify in the script seems to stop the memory consumption

* also, generating one big ldif, that is used modify all the users in groupOfNames just once (calling ldapmodify only once that is, not calling it once for every user), seems to help as well; the ldif is formatted like so:

dn: cn=users,dc=yes,dc=my changetype:modify add: member member: uid=test-1@yes1.my,dc=yes,dc=my - add: member member: uid=test-2@yes1.my,dc=yes,dc=my - ...

I'm also including my configure setup:

export CFLAGS="$(rpm --eval %{optflags}) -O2 -fPIC -D_REENTRANT -DHAVE_TLS -DHAVE_MOZNSS" export LIBS="-lpthread -lssl3 -lsmime3 -lnss3 -lnssutil3 -lplds4 -lplc4 -lnspr4"

CFLAGS="${CFLAGS:--g -pipe -Wall }"; export CFLAGS; CXXFLAGS="${CXXFLAGS:--g -pipe -Wall }"; export CXXFLAGS; FFLAGS="${FFLAGS:--g -pipe -Wall }"; export FFLAGS;

./configure \ --with-threads=posix --disable-static --with-tls=no \ --with-pic --with-cyrus-sasl \ --enable-bdb \ --enable-overlays --enable-slapd --enable-spasswd --enable-modules --enable-slapi \ --enable-accesslog \ --enable-wrappers