Thanks for responding, Howard.  That does sound similar, but I don't think it's what I'm hitting.  I'm using back-hdb.  I thought that an indexing issue might be relevant so I did some further testing:

Ubuntu 18's slapd config comes with "olcDbIndex: cn,uid eq” so that’s what I’ve been using so far.

With that index in place:
-b dc=example,dc=com 5487.20/sec
-b ou=People,dc=example,dc=com 194.30/sec


With that index removed:
-b dc=example,dc=com 1.00/sec
-b ou=People,dc=example,dc=com 0.95/sec


I added a "uid eq" index and waited a few minutes until the CPU load from slapd returned to negligible - that's the only way I know to tell when an index is done rebuilding.


With the new index:
-b dc=example,dc=com 5550.20/sec.
-b ou=People,dc=example,dc=com 198.75/sec.


This is with the following config:

# {1}hdb, config
dn: olcDatabase={1}hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {1}hdb
olcDbDirectory: /var/lib/ldap
olcSuffix: dc=example,dc=com
olcAccess: {0}to attrs=userPassword by self write by anonymous auth by * none
olcAccess: {1}to attrs=shadowLastChange by self write by * read
olcAccess: {2}to * by * read
olcLastMod: TRUE
olcRootDN: cn=admin,dc=carnegielearning,dc=com
olcRootPW: {SSHA}HXcLFZzpcjFGgjV89OeR7rbfjBVGZHx0
olcDbCheckpoint: 512 30
olcDbConfig: {0}set_cachesize 0 2097152 0
olcDbConfig: {1}set_lk_max_objects 1500
olcDbConfig: {2}set_lk_max_locks 1500
olcDbConfig: {3}set_lk_max_lockers 1500
olcDbIndex: objectClass eq
olcDbIndex: uidNumber,gidNumber eq
olcDbIndex: member,memberUid eq
olcDbIndex: uid eq

I can provide slapcats of both the config and the dataset, or just an LDIF of the 100K inetOrgPerson objects if anybody wants to see if they can replicate this.


From: Howard Chu <hyc@symas.com>
Date: September 19, 2018 at 10:29:00 AM
To: Glover, Matthew <mglover@carnegielearning.com>, openldap-technical@openldap.org <openldap-technical@openldap.org>
Subject:  [Possible Spam] Re: Worse search performance on a branch than at the tree base

Glover, Matthew wrote:
> I'm having a performance problem with searches on a branch and I'm hoping someone can explain it.
>
> A search for "(uid=12345)" on basedn "dc=example,dc=com" with a subtree scope returns much more quickly than the same filter and scope on basedn
> "ou=People,dc=example,dc=com".

Sounds vaguely like ITS#7473 but that was fixed 5 years ago. Maybe your index is not configured?
>
> To confirm this wasn't just a problem with my schema or configuration, I loaded Ubuntu 18 (slapd 2.4.45+dfsg-1ubuntu1), used ldctl to generate 100,000
> inetOrgPerson records with uids from 00000 to 99999, loaded them in with ldapadd, then used ldctl to test performance of searches:
>
> Both tests were scope subtree, 20 seconds, hitting the test server as fast as possible with randomly generated filters of "(uid=<random uid from 00000 to 99999>)".
> basedn: dc=example,dc=com - 5455.30/sec - total: 109106 successful searches
> basedn: ou=People,dc=example,dc=com - 198.80/sec - total: 3976 successful searches
>
> Also, while testing against ou=People, the CPU load from slaps went through the roof.  It feels rather like the index isn’t being used when the base isn’t the
> root.  
>
> I can provide the ldif of my test users and the parameters I used for ldctl tests if desired.
>
>
> Thanks for any help you can offer,
> Matthew


--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/