Grégory T wrote:

Hello,

before starting, we wish a happy new year to the community !

Now, we start with presentation.
We have compiled the Openldap build version 2.4.38 on a ubuntu 12.04.
This openldap server is a dual processor Intel Xeon with 4 Gb Memory. This
machine is virtualized (vmware).
Openldap is configured with back-mdb, overlays memberof and syncprov, and
envflags writemap and nosync.


We encounter a problem with a callback procedure in a perl script.
This script have to remove a couple of attributes for each entry (~120 000).
In order to save memory consumption, we use a callback function to treat each
entry one after one.

Everything works with back-bdb but ... there are mistakes with back-mdb.

In ldap.log, this kind of message appears many times :
ldapserver slapd[7264]: connection_input: conn=1014 deferring operation:
awaiting write
ldapserver last message repeated 55 times

This is an interesting problem. The above message indicates that a slapd thread on that connection has been blocked writing a result to the client. I.e., back-mdb is so much faster than back-bdb that it's returning results faster than your client can read them. Once it gets blocked like this, it defers any new requests on that connection until it gets unblocked. (I.e., it waits until the client catches up on reading.)

And suddenly,
ldapserver slapd[7264]: conn=1014 fd=14 closed (connection lost)

connection lost actually means the client closed the connection without performing an Unbind.

Our script result in "connection reset by peer" message.
The script stop around the 1040 entry.

Although the connection was closed by server, it still continue to run.
However, no explicit message in ldap.log.

If we run the script once again, the next ~1040 entries will be treated and
the script stop.

We have try with envflags writemap and nosync deactivated ... but unsuccessful.

This is the ldapsearch request :
my
$result=$ldapMaster->search(base=>$baseDN,scope=>'sub',filter=>$filter,typesonly=>'1',callback=>\&callback);

And this is the callback function :
sub callback{
    my ($search,$ent)=@_;
    return unless defined $ent;
    my $entry = $search->shift_entry();
    my @tabAttribToDel=();
    if (defined $entry){
       my $dn=$entry->dn();
       $countDN++;
       foreach my $attrib($entry->attributes){
          if ($attrib ~~ @tabOldAttrib){
             $countAttrib++;
             push (@tabAttribToDel,$attrib);
          }
       }
       if (($SUPPR==1) && (@tabAttribToDel)) {
          $countTrace++;
          my $result = $ldapMaster->modify($dn,delete => \@tabAttribToDel);
          warn "(nbEntry:$countTrace, nbAttr:$countAttrib) Error with $dn:
".$result->error."\n" if $result->code;
       }
    }
}


Why is this script working well with back-bdb but not with back-mdb ?

Working with back-bdb because it's slow enough that your client never falls behind. With back-mdb your perl script is too slow to keep up.

One workaround, the avoid the deferring - use a 2nd connection for your modify requests.

Thanks,

--
Grégory Trucy
Ingénieur système et réseau
Aix-Marseille Université

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/