Hello,
I tried a little bit more time to investigate the problem.
First, I installed a 64bit test machine, with 16GByte RAM and 2 CPUs under VMware ESX with own SAS-Storage (RAID10) for only this Machine. I configured slapd.conf as following:
####################################################################### include /etc/openldap/schema/core.schema include /etc/openldap/schema/cosine.schema include /etc/openldap/schema/inetorgperson.schema include /etc/openldap/schema/rfc2307bis.schema include /etc/openldap/schema/own.schema
pidfile /var/run/slapd/slapd.pid argsfile /var/run/slapd/slapd.args
modulepath /usr/lib/ldap moduleload back_hdb
sizelimit -1 timelimit 300 disallow bind_anon
gentlehup on tool-threads 2
# hdb database definitions
database hdb suffix "ou=root" rootdn "uid=admin,ou=root" checkpoint 4096 15 # loglevel only for test, not during time measuring loglevel 33 rootpw password directory /var/lib/ldap_hdb logfile /var/log/openldap.log cachesize 1000000 dncachesize 1000000 idlcachesize 3000000 dbnosync
index objectClass,entryUUID,entryCSN eq index subEngine eq index cn eq,sub #######################################################################
The backend for the database uses hdb.
In DB_CONF i set 2 GB BDB page cache (set_cachesize 2 0 1)
The entire directory holds 470812 entires.
(=> ldapsearch -x -h localhost -wpassword -D"uid=admin,ou=root" -b"ou=root" "(objectClass=*)" dn | grep "^dn:" | wc -l)
Task:
I search Objects with a special objectClass (subEngine) only in a dedicated oontainer (set via Base DN).
The objectClass "subEngine" exists 104384 times in the entire directory: (=> ldapsearch -x -h localhost -D"uid=admin,ou=root" -b"cn=ou=root>" "(ObjectClass=subEngine)" dn | grep "^dn:" | wc -l)
But the objectClass "subEngine" exist only one time in the dedicated Container.
When I do the search: ldapsearch -x -h localhost -wpassword -D"uid=admin,ou=root" -b"cn=ownPath,ou=root" "(ObjectClass=subEngine)"
in the logfile I can see:
=> bdb_equality_candidates (objectClass) => key_read <= bdb_index_read 470601 candidates <= bdb_equality_candidates: id=-1, first=228, last=470828 <= bdb_filter_candidates: id=-1 first=228 last=470828 <= bdb_list_candidates: id=-1 first=228 last=470828 <= bdb_filter_candidates: id=-1 first=228 last=470828 <= bdb_list_candidates: id=-1 first=40595 last=470828 <= bdb_filter_candidates: id=-1 first=40595 last=470828 bdb_search_candidates: id=-1 first=40595 last=470828
What does does these messages mean?
I can't see, how they are related with the directory.
If I search then in the logfile, I see 430233 messages like: "hdb_search: <candidate> <message: does not match filter | scope not okay>"
So the 430233 comes from 470828-40595=430233. Why so much searches?
Should the index for the objectClass=subEngine not hold only 104384 entires?
What are this for values, and how is the search done?
I guess, the first is the index lookup. But the index holds only the IDs and does nothing know about the DN from the entry. So in the next step, all from Index ID's will used to query the id2Entry.bdb and check the DN?
This search takes about 40seconds (with logging turned off!) for the first time. During this time, I can see a heavy write (about 25M/s) load from the slapd (seen with iotop)
After the cache is filled, the lookup takes about 2 seconds (with logging turned off!)...
The only difference in the logs between the first an the second search is, that in the log for the first search for each hdb_search a
entry_decode: "" <= entry_decode()
is seen.
But in the log from the second search the hdb_search is done also 430233 times!
Is this correct?
Thanks in advance
Steeg Carson
2010/12/17 Bjørn Ruberg bjorn@ruberg.no:
Steeg Carson:
2010/12/17 Bjørn Rubergbjorn@ruberg.no:
Steeg Carson: [...]
I have a database, and my search is like shown above. The search takes long.
Did you run slapindex after adding the index? Is the index file owned by the proper user account?
You didn't answer the above question...
[...]
But what I see, is that the write IO from LDAP is enormously (seen with iotop). During the whole search, the write IO is higher than the read IO. Why?
What is slapd's current loglevel?
loglevel is 0 I know, I should better use 256, but for this reason, I did switch off logging :-( for testing.
I asked because you said there's much *write* activity. If there's no logging, something else must be writing and you should find out what it is. This is probably the reason why the search is slow.
[...]
But what about the first part of my question. in this Posting. How will be a ldapsearch processed? Does the slapd, search the whole database despite of indexes?
I'm no authority on this, but generally the main purpose of using indexes is -not- having to do a full scan. This will of course require that the index has been properly built (se above).
If your original statement is still correct - that is, you've built an "eq" index (equality, exact match) and you search for the exact value - the index should have made a difference.
However, if you've built an equality index and then search for a substring, the index will not speed it up.
As a side note, you should be aware that while most attributes can be indexed with "eq", some attributes won't allow substring indexing.
Hope this helps.
-- Bjørn