On Sep 16, 2015, at 1:47 PM, Karol Chrapek kchrapek@fgtsa.com wrote:

I have a problem with Slapd and Alias dereferencing. In the ldap we have created a special subtree that allocate logical structure for our application. For this purpose we use the aliases. They allow us in one subtree have normal structure used by our internal services and for second subtree has a dedicate logical structure for special apps. Currently we tested different version of ldap:

• 2.4.28 with HDB
• 2.4.31 with HDB and MBD
• 2.4.41 with mdb
• 2.4.42 with MDB <--- this version we are currently using

On all version the dereferencing aliases is works very fast when we have about 2000 aliases and about 200000 entries in the Database. When we add additional 2000 aliases each search with aliases dereferencing hangs for 3 second. When we add additional 2000 (so 6000 alieases in DB) the search time increase for next 3 seconds. I.E. search time for filter objectclass=user with -a always for: -2000 aliases in DB is about 0,031s -4000 aliases in DB is about 3,031s -6000 aliases in DB is about 6,123s And this search time was increase even if we add additional 2000 aliases outside the search base dn. We observed that during this 3, 6 second hangs one CPU core is about 100%, system does not wait for resources, memory is on the same level. In the log I saw that slapd very fast dereferencing all aliases in subtree, hang for 3,6,9 seconds depends of aliases count and after that I saw:

mdb_dn2entry("cn=.....") Sep 4 14:53:50 ds1 slapd[4280]: => mdb_dn2id("cn=....") Sep 4 14:53:50 ds1 slapd[4280]: <= mdb_dn2id: got id=0x231 Sep 4 14:53:50 ds1 slapd[4280]: => mdb_entry_decode: Sep 4 14:53:50 ds1 slapd[4280]: <= mdb_entry_decode Sep 4 14:53:50 ds1 slapd[4280]: => mdb_filter_candidates Sep 4 14:53:50 ds1 slapd[4280]: #011OR Sep 4 14:53:50 ds1 slapd[4280]: => mdb_list_candidates 0xa1 Sep 4 14:53:50 ds1 slapd[4280]: => mdb_filter_candidates Sep 4 14:53:50 ds1 slapd[4280]: #011EQUALITY Sep 4 14:53:50 ds1 slapd[4280]: => mdb_equality_candidates (objectClass) Sep 4 14:53:50 ds1 slapd[4280]: => key_read Sep 4 14:53:50 ds1 slapd[4280]: <= mdb_index_read: failed (-30798) Sep 4 14:53:50 ds1 slapd[4280]: <= mdb_equality_candidates: id=0, first=0, last=0 Sep 4 14:53:50 ds1 slapd[4280]: <= mdb_filter_candidates: id=0 first=0 last=0 Sep 4 14:53:50 ds1 slapd[4280]: => mdb_filter_candidates Sep 4 14:53:50 ds1 slapd[4280]: #011EQUALITY Sep 4 14:53:50 ds1 slapd[4280]: => mdb_equality_candidates (objectClass) Sep 4 14:53:50 ds1 slapd[4280]: => key_read Sep 4 14:53:50 ds1 slapd[4280]: <= mdb_index_read 3585 candidates Sep 4 14:53:50 ds1 slapd[4280]: <= mdb_equality_candidates: id=3585, first=18960, last=239706 Sep 4 14:53:50 ds1 slapd[4280]: <= mdb_filter_candidates: id=3585 first=18960 last=239706 Sep 4 14:53:50 ds1 slapd[4280]: <= mdb_list_candidates: id=3585 first=18960 last=239706 Sep 4 14:53:50 ds1 slapd[4280]: <= mdb_filter_candidates: id=3585 first=18960 last=239706 Sep 4 14:53:50 ds1 slapd[4280]: mdb_search_candidates: id=3585 first=18960 last=239706 Sep 4 14:53:50 ds1 slapd[4280]: => mdb_entry_decode: Sep 4 14:53:50 ds1 slapd[4280]: <= mdb_entry_decode Sep 4 14:53:50 ds1 slapd[4280]: => test_filter Sep 4 14:53:50 ds1 slapd[4280]: EQUALITY Sep 4 14:53:50 ds1 slapd[4280]: => access_allowed: search access to "cn......." "objectClass" requested Sep 4 14:53:50 ds1 slapd[4280]: <= root access granted

After that ldap start return object that is also very fast. When the query was finished I saw in log this info:

Sep 4 14:53:50 ds1 slapd[4280]: mdb_search: 18985 scope not okay Sep 4 14:53:50 ds1 slapd[4280]: mdb_search: 18986 scope not okay

All other query that not derf. aliases are processed very fast. Search time about 32k entries in subtree without aliases is about 0,526s.

Our server DB and indexing settings:

maxsize 10737418240 checkpoint 1024 10 sizelimit 100000

maxderefdepth 2 searchstack 10

index accountid eq index objectClass eq index cn eq index id eq index name eq index entryCSN eq index entryUUID eq

Do you have any idea how we can tune search with aliases? Regards Karol

My guess would be you are doing "always" for dereferencing. Try "search" or "find" and see if either of those meets your requirements and performs better. Using "always" is known to have this effect.

--Quanah

+1, it would be nice to get this issue fixed as it means I'm still stuck using HDB because one of my user groups makes extensive use of aliases to present subsets of the complete user list.

FWIW MDB is superior in just about every other way and has been rock-solid on our Test infrastructure.

On 17/09/15 06:43, Karol Chrapek wrote:

Unfortunately -a search for the query take the same amount of time and find return different result to always and search option. I know that our structure is problematic and understand that ldap need to dereferencing aliases. I try to understand why when you have up to 2000 aliases in DB queries are very fast, almost instant but when you have more aliases in DB slapd take 100% of one core (probably one thread) and count something.

*Karol Chrapek* | Infrastructure Architect

Infrastructure Division

mobile: +48 660 623 422

2015-09-16 23:43 GMT+02:00 Quanah Gibson-Mount <quanah@zimbra.com mailto:quanah@zimbra.com>:

On Sep 16, 2015, at 1:47 PM, Karol Chrapek <kchrapek@fgtsa.com
<mailto:kchrapek@fgtsa.com>> wrote:

...

I have a problem with Slapd and Alias dereferencing. In the ldap
we have created a special subtree that allocate logical structure
for our application. For this purpose we use the aliases. They
allow us in one subtree have normal structure used by our internal
services and for second subtree has a dedicate logical structure
for special apps.
Currently we tested different version of ldap: 
- 2.4.28 with HDB
- 2.4.31 with HDB and MBD 
- 2.4.41 with mdb
- 2.4.42 with MDB <--- this version we are currently using

On all version the dereferencing aliases is works very fast when
we have about 2000 aliases and about 200000 entries in the
Database. When we add additional 2000 aliases each search with
aliases dereferencing hangs for 3 second. When we add additional
2000 (so 6000 alieases in DB) the search time increase for next 3
seconds.
I.E. search time for filter objectclass=user with -a always for:
-2000 aliases in DB is about  0,031s
-4000 aliases in DB is about 3,031s
-6000 aliases in DB is about 6,123s 
And this search time was increase even if we add additional 2000
aliases outside the search base dn. We observed that during this
3, 6 second hangs one CPU core is about 100%, system does not wait
for resources, memory is on the same level. 
In the log I saw that slapd very fast dereferencing all aliases in
subtree, hang for 3,6,9 seconds depends of aliases count and after
that I saw: 

 mdb_dn2entry("cn=.....")
Sep  4 14:53:50 ds1 slapd[4280]: => mdb_dn2id("cn=....")
Sep  4 14:53:50 ds1 slapd[4280]: <= mdb_dn2id: got id=0x231
Sep  4 14:53:50 ds1 slapd[4280]: => mdb_entry_decode:
Sep  4 14:53:50 ds1 slapd[4280]: <= mdb_entry_decode
Sep  4 14:53:50 ds1 slapd[4280]: => mdb_filter_candidates
Sep  4 14:53:50 ds1 slapd[4280]: #011OR
Sep  4 14:53:50 ds1 slapd[4280]: => mdb_list_candidates 0xa1
Sep  4 14:53:50 ds1 slapd[4280]: => mdb_filter_candidates
Sep  4 14:53:50 ds1 slapd[4280]: #011EQUALITY
Sep  4 14:53:50 ds1 slapd[4280]: => mdb_equality_candidates
(objectClass)
Sep  4 14:53:50 ds1 slapd[4280]: => key_read
Sep  4 14:53:50 ds1 slapd[4280]: <= mdb_index_read: failed (-30798)
Sep  4 14:53:50 ds1 slapd[4280]: <= mdb_equality_candidates: id=0,
first=0, last=0
Sep  4 14:53:50 ds1 slapd[4280]: <= mdb_filter_candidates: id=0
first=0 last=0
Sep  4 14:53:50 ds1 slapd[4280]: => mdb_filter_candidates
Sep  4 14:53:50 ds1 slapd[4280]: #011EQUALITY
Sep  4 14:53:50 ds1 slapd[4280]: => mdb_equality_candidates
(objectClass)
Sep  4 14:53:50 ds1 slapd[4280]: => key_read
Sep  4 14:53:50 ds1 slapd[4280]: <= mdb_index_read 3585 candidates
Sep  4 14:53:50 ds1 slapd[4280]: <= mdb_equality_candidates:
id=3585, first=18960, last=239706
Sep  4 14:53:50 ds1 slapd[4280]: <= mdb_filter_candidates: id=3585
first=18960 last=239706
Sep  4 14:53:50 ds1 slapd[4280]: <= mdb_list_candidates: id=3585
first=18960 last=239706
Sep  4 14:53:50 ds1 slapd[4280]: <= mdb_filter_candidates: id=3585
first=18960 last=239706
Sep  4 14:53:50 ds1 slapd[4280]: mdb_search_candidates: id=3585
first=18960 last=239706
Sep  4 14:53:50 ds1 slapd[4280]: => mdb_entry_decode:
Sep  4 14:53:50 ds1 slapd[4280]: <= mdb_entry_decode
Sep  4 14:53:50 ds1 slapd[4280]: => test_filter
Sep  4 14:53:50 ds1 slapd[4280]:     EQUALITY
Sep  4 14:53:50 ds1 slapd[4280]: => access_allowed: search access
to "cn......." "objectClass" requested
Sep  4 14:53:50 ds1 slapd[4280]: <= root access granted

After that ldap start return object that is also very fast. When
the query was finished I saw in log this info:


Sep  4 14:53:50 ds1 slapd[4280]: mdb_search: 18985 scope not okay

Sep  4 14:53:50 ds1 slapd[4280]: mdb_search: 18986 scope not okay


All other query that not derf. aliases are processed very fast.
Search time about 32k entries in subtree without aliases is about
0,526s.

Our server DB and indexing settings: 

maxsize 10737418240
checkpoint 1024 10
sizelimit 100000

maxderefdepth 2
searchstack 10

index accountid eq
index objectClass eq
index cn eq
index id eq
index name eq
index entryCSN eq
index entryUUID eq

Do you have any idea how we can tune search with aliases? 
Regards 
Karol

My guess would be you are doing "always" for dereferencing. Try
"search" or "find" and see if either of those meets your
requirements and performs better. Using "always" is known to have
this effect. 

--Quanah