Hello,
thanks for the reply.
The script works fine with the use of a 2nd connection for the modify request but ... during the execution of my script, the file data.mdb grows up considerably.
First time, I configured maxsize value with a 3GB in slapd.conf. Then I modified the script to use 2 connections. After that, I ran the script. Everything went right until and I was confronted to a MDB_MAP_FULL error in ldap.log. Indeed, data.mdb reaches the max size. So, I stopped slapd, I modified the maxsize to 5Gb value in slapd.conf and then, I restarted slapd. I ran the script again and I checked the size of data.mdb. Before runnig the script, the real size of data.mdb is about 700M. Once the script finished, the real size of data.mdb is about 4.5G. During execution, memory consumption of slapd increases up progressively to reach 83% at the end of execution.
Of course, if I make a slapcat then a slapadd, the size of data.mdb return to 696M.
I run mdb_stat -f before the use of script : mdb_stat -f /var/lib/ldap Freelist Status Tree depth: 1 Branch pages: 0 Leaf pages: 1 Overflow pages: 35 Entries: 14 Free pages: 17300 Status of Main DB Tree depth: 1 Branch pages: 0 Leaf pages: 1 Overflow pages: 0 Entries: 25
And then after the execution of script : mdb_stat -f /var/lib/ldap Freelist Status Tree depth: 3 Branch pages: 12 Leaf pages: 2418 Overflow pages: 7 Entries: 55122 Free pages: 986942 Status of Main DB Tree depth: 1 Branch pages: 0 Leaf pages: 1 Overflow pages: 0 Entries: 25
After some search, I found the message "(ITS#7565) unacceptable growth on delete with back-mdb" which looks like my problem.
Have you got any suggestions ?
Thanks,
Grégory Trucy
2014/1/6 Howard Chu hyc@symas.com
Grégory T wrote:
Hello,
before starting, we wish a happy new year to the community !
Now, we start with presentation. We have compiled the Openldap build version 2.4.38 on a ubuntu 12.04. This openldap server is a dual processor Intel Xeon with 4 Gb Memory. This machine is virtualized (vmware). Openldap is configured with back-mdb, overlays memberof and syncprov, and envflags writemap and nosync.
We encounter a problem with a callback procedure in a perl script. This script have to remove a couple of attributes for each entry (~120 000). In order to save memory consumption, we use a callback function to treat each entry one after one.
Everything works with back-bdb but ... there are mistakes with back-mdb.
In ldap.log, this kind of message appears many times : ldapserver slapd[7264]: connection_input: conn=1014 deferring operation: awaiting write ldapserver last message repeated 55 times
This is an interesting problem. The above message indicates that a slapd thread on that connection has been blocked writing a result to the client. I.e., back-mdb is so much faster than back-bdb that it's returning results faster than your client can read them. Once it gets blocked like this, it defers any new requests on that connection until it gets unblocked. (I.e., it waits until the client catches up on reading.)
And suddenly, ldapserver slapd[7264]: conn=1014 fd=14 closed (connection lost)
connection lost actually means the client closed the connection without performing an Unbind.
Our script result in "connection reset by peer" message. The script stop around the 1040 entry.
Although the connection was closed by server, it still continue to run. However, no explicit message in ldap.log.
If we run the script once again, the next ~1040 entries will be treated and the script stop.
We have try with envflags writemap and nosync deactivated ... but unsuccessful.
This is the ldapsearch request : my $result=$ldapMaster->search(base=>$baseDN,scope=>'sub', filter=>$filter,typesonly=>'1',callback=>&callback);
And this is the callback function : sub callback{ my ($search,$ent)=@_; return unless defined $ent; my $entry = $search->shift_entry(); my @tabAttribToDel=(); if (defined $entry){ my $dn=$entry->dn(); $countDN++; foreach my $attrib($entry->attributes){ if ($attrib ~~ @tabOldAttrib){ $countAttrib++; push (@tabAttribToDel,$attrib); } } if (($SUPPR==1) && (@tabAttribToDel)) { $countTrace++; my $result = $ldapMaster->modify($dn,delete => @tabAttribToDel); warn "(nbEntry:$countTrace, nbAttr:$countAttrib) Error with $dn: ".$result->error."\n" if $result->code; } } }
Why is this script working well with back-bdb but not with back-mdb ?
Working with back-bdb because it's slow enough that your client never falls behind. With back-mdb your perl script is too slow to keep up.
One workaround, the avoid the deferring - use a 2nd connection for your modify requests.
Thanks,
-- Grégory Trucy Ingénieur système et réseau Aix-Marseille Université
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/