I have a server with 20,000 account entries and one group that has 10,000 members. Deleting members from that group is very slow on OpenLDAP 2.4.10, though adding goes at a reasonable speed.
Here is the test operation:
----------------------------------------------------------------------- dn: uniqueIdentifier=tenK,ou=groups,uniqueIdentifier=o_000000,dc=orgs,dc=dir,dc=example,dc=com changetype: modify delete: member member: uniqueIdentifier=a_004996,ou=accounts,uniqueIdentifier=o_000006,dc=orgs,dc=dir,dc=example,dc=com - add: member member: uniqueIdentifier=a_999999,ou=accounts,uniqueIdentifier=o_000005,dc=orgs,dc=dir,dc=example,dc=com -----------------------------------------------------------------------
On 2.3.42 that takes about 1s with the caches warm, but on 2.4.10 it takes almost 9s, most of which is CPU time.
The test machine has 2GB of memory and the DB_CONFIG looks like this for BDB:
----------------------------------------------------------------------- # memory cache size # gigabytes bytes number-of-regions # set_cachesize 0 150000000 1
# Logbuffer size # bytes # set_lg_bsize 2097152
# Remove the roll-forward transaction logs automatically # This reduces the space used by the database but it also reduces # the disaster-recovery options.
set_flags DB_LOG_AUTOREMOVE -----------------------------------------------------------------------
There is no disk activity during most of the 9s, with a large burst of writes at the end.
There have been various discussions about *adding* members to large groups in the past, but the problems appeared to have been solved in recent versions.
The problem is certainly related to indexing, as if I remove indexing from the 'member' attribute I get times of about 0.3s on 2.4.10. (Note: overall times are measured by timing an ldapmodify script, so not very accurate below 0.5s)
Does anyone know why this operation should have got slower from 2.3.x to 2.4.x ?
Andrew
Andrew Findlay wrote:
I have a server with 20,000 account entries and one group that has 10,000 members. Deleting members from that group is very slow on OpenLDAP 2.4.10, though adding goes at a reasonable speed.
Try configuring sortvals - see slapd.conf(5).
On Wed, Jun 25, 2008 at 09:11:49AM -0700, Howard Chu wrote:
I have a server with 20,000 account entries and one group that has 10,000 members. Deleting members from that group is very slow on OpenLDAP 2.4.10, though adding goes at a reasonable speed.
Try configuring sortvals - see slapd.conf(5).
That helps a lot, thanks. It reduces the delete time to under 0.5s.
Unfortunately it seems to introduce a bug. If I *just* add or delete a value all is OK, but if I delete one value and add another in the same LDAP operation the new value does not get indexed. The value is in the entry, but it cannot be searched for. Further, it cannot be deleted (as it is 'not there') yet it can be added again, leading to multiple identical values in the entry!
This only happens on *large* groups where the value being added would sort to the end of the list of values. I don't know how large the group has to be: 3 is not enough, and 10,000 is. My test case has a group that starts with 10,000 members inserted in numerical order:
dn: uniqueIdentifier=tenK,ou=groups,uniqueIdentifier=o_000000,dc=orgs,dc=dir,dc=example,dc=org objectclass: myGroup uniqueIdentifier: tenK displayName: Managers group tenK description: Sample group of managers member: uniqueIdentifier=a_000000,ou=accounts,uniqueIdentifier=o_000003,dc=orgs,dc=dir,dc=example,dc=org member: uniqueIdentifier=a_000001,ou=accounts,uniqueIdentifier=o_000000,dc=orgs,dc=dir,dc=example,dc=org member: uniqueIdentifier=a_000002,ou=accounts,uniqueIdentifier=o_000004,dc=orgs,dc=dir,dc=example,dc=org member: uniqueIdentifier=a_000003,ou=accounts,uniqueIdentifier=o_000003,dc=orgs,dc=dir,dc=example,dc=org
Deleting an entry from the middle of the list and adding one of the form: uniqueIdentifier=a_999999,ou=accounts,uniqueIdentifier=o_000005,... causes the problem, but if I add one like this it does not: uniqueIdentifier=a_00500Z,ou=accounts,uniqueIdentifier=o_000005,... (that would go near the middle of the list)
Andrew
--On June 25, 2008 6:11:24 PM +0100 Andrew Findlay andrew.findlay@skills-1st.co.uk wrote:
Unfortunately it seems to introduce a bug. If I *just* add or delete a value all is OK, but if I delete one value and add another in the same LDAP operation the new value does not get indexed. The value is in the entry, but it cannot be searched for. Further, it cannot be deleted (as it is 'not there') yet it can be added again, leading to multiple identical values in the entry!
This only happens on *large* groups where the value being added would sort to the end of the list of values. I don't know how large the group has to be: 3 is not enough, and 10,000 is. My test case has a group that starts with 10,000 members inserted in numerical order:
Please file an ITS. :)
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On Wed, Jun 25, 2008 at 10:21:36AM -0700, Quanah Gibson-Mount wrote:
This only happens on *large* groups where the value being added would sort to the end of the list of values. I don't know how large the group has to be: 3 is not enough, and 10,000 is. My test case has a group that starts with 10,000 members inserted in numerical order:
Please file an ITS. :)
Done: ITS#5578
Fortunately I managed to cut the size of the test down by several orders of magnitude... It looks as if the key to the bug is adding a value that sorts to the end of the list.
Andrew
openldap-software@openldap.org
-
Andrew Findlay -
Howard Chu -
Quanah Gibson-Mount