Hi,
Thank you for your reply.
My knowledge about the bdb internals is limited. I have reproduced this issue and reduced it to a small amount of data (see below). I think I have done everything right with regards to setting up and preparing the database and indexes. I've used both ldapmodify and slapadd/slapindex to prepare the db.
If you think my indexes are corrupt then can you please give me pointers how to verify?
I don't think there exist test cases for operators LE "<=3D" or GE = ">=3D" in the source codes.
Thanks, Petteri
-----Original Message----- From: Howard Chu [mailto:hyc@symas.com]=20 Sent: Tuesday, November 09, 2010 10:45 AM To: Petteri Stenius Cc: openldap-its@openldap.org Subject: Re: (ITS#6683) DDS fails with expired branches
Petteri.Stenius@ubisecure.com wrote:
Hello,
Further investigation shows this issue is caused by operator LE search failing with indexed attributes. Also this indexed search issue is NOT limited to DDS.=3D20
I have reproduced the issue with integerOrderingMatch and generalizedTimeOrderingMatch.=3D20
The piece of code I find suspicious is in servers/back-bdb/idl.c, somewhere in the middle it reads
/* skip presence key on range inequality lookups */ while (rc =3D3D=3D3D 0&& kptr->size !=3D3D len) { rc =3D3D cursor->c_get( cursor, kptr,&data, flags | DB_NEXT_NODUP ); }
If I remove this block then LE search works as expected with indexed attributes. The key here seems to be the DB_NEXT_NODUP flag. This flag causes the iterator block a few lines below to return partial matches.
That implies that there's something else corrupt in the index, because the=20 presence key will never be the same size as an equality key.
Thanks, Petteri
-----Original Message----- From: openldap-bugs-bounces@OpenLDAP.org [mailto:openldap-bugs-bounces@OpenLDAP.org] On Behalf Of petteri.stenius@ubisecure.com Sent: Monday, October 25, 2010 1:35 PM To: openldap-its@openldap.org Subject: (ITS#6683) DDS fails with expired branches
Full_Name:=3D20 Version: 2.4.23 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (195.197.205.34)
Hello,
I have a directory with branches of dynamicObject entries. It looks
like
if the entryExpireTimestamp value is the same on objects within a branch then DDS search for expired objects will only find the top-most object. This results in remove failing with message
DDS dn=3D3D"cn=3D3Dtop,cn=3D3Droot,dc=3D3Dtest" is non-leaf; =
deferring.
To reproduce
OpenLDAP 2.4.23, Berkeley DB 4.6.21
Use slapadd to prepare directory with following
dn: cn=3D3DRoot,dc=3D3Dtest objectClass: top objectClass: applicationProcess cn: Root
dn: cn=3D3Dtop,cn=3D3DRoot,dc=3D3Dtest objectClass: top objectClass: device objectClass: dynamicObject entryTTL: 60 entryExpireTimestamp: 20101024113626Z cn: top
dn: cn=3D3Dleaf1,cn=3D3Dtop,cn=3D3DRoot,dc=3D3Dtest objectClass: top objectClass: device objectClass: dynamicObject entryTTL: 60 entryExpireTimestamp: 20101024113626Z cn: leaf1
dn: cn=3D3Dleaf2,cn=3D3Dtop,cn=3D3DRoot,dc=3D3Dtest objectClass: top objectClass: device objectClass: dynamicObject entryTTL: 60 entryExpireTimestamp: 20101024113626Z cn: leaf2
dn: cn=3D3Dleaf3,cn=3D3Dtop,cn=3D3DRoot,dc=3D3Dtest objectClass: top objectClass: device objectClass: dynamicObject entryTTL: 60 entryExpireTimestamp: 20101024113626Z cn: leaf3
Relevant slapd.conf entries
database bdb suffix "cn=3D3DRoot,dc=3D3Dtest" rootdn "cn=3D3DRoot,dc=3D3Dtest" rootpw "password"
overlay dds dds-default-ttl 3600 dds-min-ttl 60 dds-interval 60 dds-state true index entryExpireTimestamp eq,pres
access to dn.subtree=3D3D"cn=3D3DRoot,dc=3D3Dtest" by users write by * read
Running "slapd -d 1 -d 256" produces following
put_filter:
"(&(objectClass=3D3DdynamicObject)(entryExpireTimestamp<=3D3D201010250824= 46Z )=3D
)" put_filter: AND put_filter_list
"(objectClass=3D3DdynamicObject)(entryExpireTimestamp<=3D3D20101025082446= Z)"
put_filter: "(objectClass=3D3DdynamicObject)" put_filter: simple put_simple_filter: "objectClass=3D3DdynamicObject" put_filter: "(entryExpireTimestamp<=3D3D20101025082446Z)" put_filter: simple put_simple_filter: "entryExpireTimestamp<=3D3D20101025082446Z" ber_scanf fmt ({mm}) ber: ber_scanf fmt ({mm}) ber: =3D3D> bdb_search bdb_dn2entry("cn=3D3Droot,dc=3D3Dtest") =3D3D> bdb_dn2id("cn=3D3Droot,dc=3D3Dtest") <=3D3D bdb_dn2id: got id=3D3D0x1 entry_decode: "cn=3D3DRoot,dc=3D3Dtest" <=3D3D entry_decode(cn=3D3DRoot,dc=3D3Dtest) search_candidates: base=3D3D"cn=3D3Droot,dc=3D3Dtest" (0x00000001) =
scope=3D3D2
=3D3D> bdb_dn2idl("cn=3D3Droot,dc=3D3Dtest") =3D3D> bdb_equality_candidates (objectClass) =3D3D> key_read <=3D3D bdb_index_read: failed (-30989) <=3D3D bdb_equality_candidates: id=3D3D0, first=3D3D0, last=3D3D0 =3D3D> bdb_equality_candidates (objectClass) =3D3D> key_read <=3D3D bdb_index_read 4 candidates <=3D3D bdb_equality_candidates: id=3D3D4, first=3D3D2, last=3D3D5 =3D3D> bdb_inequality_candidates (entryExpireTimestamp) =3D3D> key_read <=3D3D bdb_index_read 1 candidates =3D3D> key_read <=3D3D bdb_index_read: failed (-30989) <=3D3D bdb_inequality_candidates: id=3D3D1, first=3D3D2, last=3D3D2 bdb_search_candidates: id=3D3D1 first=3D3D2 last=3D3D2 entry_decode: "cn=3D3Dtop,cn=3D3DRoot,dc=3D3Dtest" <=3D3D entry_decode(cn=3D3Dtop,cn=3D3DRoot,dc=3D3Dtest) =3D3D> bdb_dn2id("cn=3D3Dtop,cn=3D3Droot,dc=3D3Dtest") <=3D3D bdb_dn2id: got id=3D3D0x2 send_ldap_result: conn=3D3D-1 op=3D3D0 p=3D3D0 bdb_dn2entry("cn=3D3Dtop,cn=3D3Droot,dc=3D3Dtest") =3D3D> bdb_dn2id_children("cn=3D3Dtop,cn=3D3Droot,dc=3D3Dtest") <=3D3D bdb_dn2id_children("cn=3D3Dtop,cn=3D3Droot,dc=3D3Dtest"): (0) send_ldap_result: conn=3D3D-1 op=3D3D0 p=3D3D0 DDS dn=3D3D"cn=3D3Dtop,cn=3D3Droot,dc=3D3Dtest" is non-leaf; =
deferring.
DDS expired=3D3D0
ldapsearch "(entryExpireTimestamp=3D3D*)" produces
dn: cn=3D3Dtop,cn=3D3DRoot,dc=3D3Dtest entryExpireTimestamp: 20101024113626Z
dn: cn=3D3Dleaf1,cn=3D3Dtop,cn=3D3DRoot,dc=3D3Dtest entryExpireTimestamp: 20101024113626Z
dn: cn=3D3Dleaf2,cn=3D3Dtop,cn=3D3DRoot,dc=3D3Dtest entryExpireTimestamp: 20101024113626Z
dn: cn=3D3Dleaf3,cn=3D3Dtop,cn=3D3DRoot,dc=3D3Dtest entryExpireTimestamp: 20101024113626Z
where ldapsearch "(entryExpireTimestamp<=3D3D20101024113626Z)" only
finds
dn: cn=3D3Dtop,cn=3D3DRoot,dc=3D3Dtest entryExpireTimestamp: 20101024113626Z
If I change all timestamps to distinct values then expiration of complete branches works as expected.
Thanks, Petteri
--=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/