[Issue 9349] New: Performance problem when trying to remove a value from an indexed attribute
https://bugs.openldap.org/show_bug.cgi?id=9349
Issue ID: 9349 Summary: Performance problem when trying to remove a value from an indexed attribute Product: OpenLDAP Version: 2.4.45 Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: backends Assignee: bugs@openldap.org Reporter: gbuades@soffid.com Target Milestone: ---
Created attachment 762 --> https://bugs.openldap.org/attachment.cgi?id=762&action=edit Patch to improve attribute removal
I have an LDAP with MDB backend, and a groupOfUniqueNames with 50.000 members. The uniqueMember attribute is indexed.
When I try to remove a single user, the process takes more or less one minute to complete.
After debugging, I've just found out the process to identify the old values to remove from the index is not optimum, as the time required to compare the old and the new values depends on number of values ^ 2.
Taking advantage of the fact, that the values in the old and new attributes are ordered in the same way, I've build a patch to opmtimize this search process.
Now, the same operation takes 0.2 seconds, when it used to take 50s.
Thanks a lot for your effort, and I hope this contribution helps you and other users.
The attached file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Soffid. Soffid has not assigned rights and/or interest in this work to any party. I, Gabriel Buades am authorized by Soffid, my employer, to release this work under the following terms.
Soffid hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.
The attached modifications to OpenLDAP Software are subject to the following notice: Copyright 2020 Soffid Redistribution and use in source and binary forms, with or without modification, are permitted only as authorized by the OpenLDAP Public License.
https://bugs.openldap.org/show_bug.cgi?id=9349
--- Comment #1 from Howard Chu hyc@openldap.org --- (In reply to Gabriel Buades from comment #0)
Taking advantage of the fact, that the values in the old and new attributes are ordered in the same way, I've build a patch to opmtimize this search process.
Now, the same operation takes 0.2 seconds, when it used to take 50s.
This can't be right:
+ for ( i=k=0; i < ap->a_numvals; i++ ) { + char found = 0; + BerValue* current = &ap->a_nvals[i]; + int k2 = k; + for (k2 = k ; k2 < a2->a_numvals; k2 ++) { + int match = -1, rc; + const char *text; + + rc = ordered_value_match( &match, a2, + ap->a_desc->ad_type->sat_equality, 0, + &ap->a_nvals[i], current, &text );
current is the same as &ap->a_nvals[i].
https://bugs.openldap.org/show_bug.cgi?id=9349
--- Comment #2 from Gabriel Buades gbuades@soffid.com --- You are right Howard.
I did something wrong while cleaning up the code to submit the patch. As you have pointed out, the comparison should be: + rc = ordered_value_match( &match, a2, + ap->a_desc->ad_type->sat_equality, 0, + &a2->a_nvals[k2], current, &text );
Anyway, let me test and check it. I will submit the patch again once it has been fixed.
Kind regards.
Gabriel Buades *Soffid* technical team leader ✉ gbuades@soffid.com ✆ +34-871962912 📱 +34-639660155 Skype: gbuadesr
*Please consider the environment before printing this email*
This email and any attachments are property of Soffid IAM SL. It may contain confidential or privileged information, so it is absolutely forbidden to copy, reproduce, publish and distribute their contents and the use of this material for any other purpose that is not strictly authorized, as stated at Intellectual Property and Data Protection laws.
El mié., 16 sept. 2020 a las 14:52, openldap-its@openldap.org escribió:
https://bugs.openldap.org/show_bug.cgi?id=9349
--- Comment #1 from Howard Chu hyc@openldap.org --- (In reply to Gabriel Buades from comment #0)
Taking advantage of the fact, that the values in the old and new
attributes
are ordered in the same way, I've build a patch to opmtimize this search process.
Now, the same operation takes 0.2 seconds, when it used to take 50s.
This can't be right:
for ( i=k=0; i < ap->a_numvals; i++ ) {char found = 0;BerValue* current = &ap->a_nvals[i];int k2 = k;for (k2 = k ; k2 < a2->a_numvals; k2 ++) {int match = -1, rc;const char *text;rc = ordered_value_match( &match, a2,ap->a_desc->ad_type->sat_equality, 0,&ap->a_nvals[i], current, &text );current is the same as &ap->a_nvals[i].
-- You are receiving this mail because: You reported the issue.
https://bugs.openldap.org/show_bug.cgi?id=9349
--- Comment #3 from Gabriel Buades gbuades@soffid.com --- Hello all.
This is the right patch. Apologies for any inconvenience.
Kind regards.
El mié., 16 sept. 2020 a las 15:01, Gabriel Buades (gbuades@soffid.com) escribió:
You are right Howard.
I did something wrong while cleaning up the code to submit the patch. As you have pointed out, the comparison should be:
rc = ordered_value_match( &match, a2,ap->a_desc->ad_type->sat_equality, 0,&a2->a_nvals[k2], current, &text );Anyway, let me test and check it. I will submit the patch again once it has been fixed.
Kind regards.
Gabriel Buades *Soffid* technical team leader ✉ gbuades@soffid.com ✆ +34-871962912 📱 +34-639660155 Skype: gbuadesr
*Please consider the environment before printing this email*
This email and any attachments are property of Soffid IAM SL. It may contain confidential or privileged information, so it is absolutely forbidden to copy, reproduce, publish and distribute their contents and the use of this material for any other purpose that is not strictly authorized, as stated at Intellectual Property and Data Protection laws.
El mié., 16 sept. 2020 a las 14:52, openldap-its@openldap.org escribió:
https://bugs.openldap.org/show_bug.cgi?id=9349
--- Comment #1 from Howard Chu hyc@openldap.org --- (In reply to Gabriel Buades from comment #0)
Taking advantage of the fact, that the values in the old and new
attributes
are ordered in the same way, I've build a patch to opmtimize this search process.
Now, the same operation takes 0.2 seconds, when it used to take 50s.
This can't be right:
for ( i=k=0; i < ap->a_numvals; i++ ) {char found = 0;BerValue* current = &ap->a_nvals[i];int k2 = k;for (k2 = k ; k2 < a2->a_numvals; k2 ++) {int match = -1, rc;const char *text;rc = ordered_value_match( &match, a2,ap->a_desc->ad_type->sat_equality, 0,&ap->a_nvals[i], current, &text );current is the same as &ap->a_nvals[i].
-- You are receiving this mail because: You reported the issue.
https://bugs.openldap.org/show_bug.cgi?id=9349
--- Comment #4 from Howard Chu hyc@openldap.org --- (In reply to Gabriel Buades from comment #3)
Created attachment 764 [details] 0001-Performance-improvement-for-indexed-attributes-remov.patch
Hello all.
This is the right patch. Apologies for any inconvenience.
This still should not compile
+ rc = ordered_value_match( &match, a2, + ap->a_desc->ad_type->sat_equality, 0,
The 2nd argument should be an AttributeDescription*, not an Attribute*.
https://bugs.openldap.org/show_bug.cgi?id=9349
--- Comment #5 from Gabriel Buades gbuades@soffid.com --- You are right. The compiler should not compile, but it simply issued a warning I didn't notice.
I hope this one will work.
Thanks in advance.
https://bugs.openldap.org/show_bug.cgi?id=9349
--- Comment #6 from Howard Chu hyc@openldap.org --- (In reply to Gabriel Buades from comment #5)
Created attachment 766 [details] 0001-Performance-improvement-for-indexed-attributes-remov.patch
You are right. The compiler should not compile, but it simply issued a warning I didn't notice.
I hope this one will work.
Thanks in advance.
Thanks, that looks better, will test it out shortly.
But for your use case you should probably have used sortvals (see slapd.conf(5) or slapd-config(5)) as that would also improve performance of filter matchng, new value insertion, etc. etc.
Improving delete performance is nice, but using sortvals improves everything at once.
https://bugs.openldap.org/show_bug.cgi?id=9349
Howard Chu hyc@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |TEST Status|UNCONFIRMED |RESOLVED
https://bugs.openldap.org/show_bug.cgi?id=9349
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|--- |2.5.0
https://bugs.openldap.org/show_bug.cgi?id=9349
--- Comment #7 from Quanah Gibson-Mount quanah@openldap.org --- trunk:
Commits: • 984ecd11 by Gabriel Buades at 2020-09-17T18:21:53+01:00 ITS#9349 slapd-mdb: optimize index delete
Performance improvement for indexed attributes removal
https://bugs.openldap.org/show_bug.cgi?id=9349
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Resolution|TEST |FIXED Status|RESOLVED |VERIFIED
-
openldap-its＠openldap.org