Dear all
I've configured accesslog overlay on our directory project because all modification history is very (very) important for us. The most frequent query (and currently the only query) to the accesslog database is to look for modification to a certain record. Typically like this:
(reqDN=uid=paul_smith,ou=contacts,ou=xxx,dc=xxx,dc=xxx)
Currently the database behave strangely:
(reqAuthzID=uid=paul_smith,ou=contacts,ou=xxx,dc=xxx,dc=xxx) -> return result in 4 seconds (reqDN=uid=paul_smith,ou=contacts,ou=xxx,dc=xxx,dc=xxx) -> return result in 4 minutes
The interesting thing: I have configured to index reqDN but not reqAuthzID:
I read the admin's manual (2.4), as in the manual the set up of index is as simple as configure it in slapd.conf and run slapindex, which I did, with no error at all.
[snip from slapd.conf] database bdb suffix "cn=accesslog" rootdn "cn=admin,cn=accesslog" rootpw [hidden] directory "/var/lib/ldap-log" index reqDN,reqType eq [/snip]
I also double checked /var/lib/ldap-log to confirm there is "reqDN.dbd" (2MB)
The total number of modification records is around 50k. Openldap 2.3.27 running on Debian/x86
My question is: The 4 minute search time is not reasonable (Excel can do much faster with same number of records), and not acceptable for our use (because checking modification history is daily routine). How can I improve it? What other information I didn't provider to have asked this question properly? What else (besides admin manual which only described indexing a few pages) should I read?
Thanks a lot in advance! Best regards
--On Wednesday, November 07, 2007 10:37 PM +0800 Zhang Weiwu zhangweiwu@realss.com wrote:
My question is: The 4 minute search time is not reasonable (Excel can do much faster with same number of records), and not acceptable for our use (because checking modification history is daily routine). How can I improve it? What other information I didn't provider to have asked this question properly? What else (besides admin manual which only described indexing a few pages) should I read?
Sounds like your index may be corrupt, or you haven't properly tuned the DB_CONFIG file for the accesslog DB. Alternately, when you ran slapindex, you failed to specify which database to index. See the -b and -n options to slapindex.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On Wed, 7 Nov 2007, Quanah Gibson-Mount wrote:
--On Wednesday, November 07, 2007 10:37 PM +0800 Zhang Weiwu zhangweiwu@realss.com wrote:
My question is: The 4 minute search time is not reasonable (Excel can do much faster with same number of records), and not acceptable for our use (because checking modification history is daily routine). How can I improve it? What other information I didn't provider to have asked this question properly? What else (besides admin manual which only described indexing a few pages) should I read?
Sounds like your index may be corrupt, or you haven't properly tuned the DB_CONFIG file for the accesslog DB. Alternately, when you ran slapindex, you failed to specify which database to index. See the -b and -n options to slapindex.
Thank you for the help (again). I can only be sure I have properly run slapindex with '-b' but I did not tune DB_CONFIG. Will do it asap after I read documents on how to do that. Thanks!
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
--On Wednesday, November 07, 2007 10:37 PM +0800 Zhang Weiwu zhangweiwu@realss.com wrote:
My question is: The 4 minute search time is not reasonable (Excel can do much faster with same number of records), and not acceptable for our use (because checking modification history is daily routine). How can I improve it? What other information I didn't provider to have asked this question properly? What else (besides admin manual which only described indexing a few pages) should I read?
Sounds like your index may be corrupt, or you haven't properly tuned the DB_CONFIG file for the accesslog DB. Alternately, when you ran slapindex, you failed to specify which database to index. See the -b and -n options to slapindex.
Thank you for the hint. As I explained my index is produced with -b option. When I try to tweak settings in DB_CONFIG I found tutorials about it http://k12linux.mesd.k12.or.us/ldap/performance.html which suggest me this:
The size of dbcachesize should be big enough to hold your largest index file (*.dbb).
Unfortunately I do not have any *.dbb file. Besides log files I only get several other files that looks like index files (they ends in .bdb)
-rw-rw---- 1 openldap openldap 126M 2007-11-09 10:35 dn2id.bdb -rw-rw---- 1 openldap openldap 1,7G 2007-11-09 10:35 id2entry.bdb -rw-rw---- 1 openldap openldap 2,1M 2007-11-09 10:35 reqDN.bdb -rw-rw---- 1 openldap openldap 13M 2007-06-03 13:28 reqStart.bdb -rw-rw---- 1 openldap openldap 660K 2007-11-09 10:35 reqType.bdb
Since the biggest file (id2entry.bdb) is at 1,7G, does that suggest I should have dbcachesize of that size? I cannot understand why the cache can be so large, after all there are only 50,000 records in it. After all the query result of 'ldapsearch reqStart=* reqstart' is only 6.8MB. But anyway I added these configuration:
dbcachesize 1740000000 cachesize 6000
Which is very close to my biggest .bdb file size / number of records. The query time is reduced from 4 minutes to 3 minutes, still far from user expectation (less than 10 seconds). Tested query is like this:
(reqDN=uid=paul_smith,ou=contacts,ou=xxx,dc=xxx,dc=xxx)
I am going to try a lot more experiments, but would also like to hear if there are something looking significantly wrong in my system? Thank!
Zhang Weiwu wrote:
Quanah Gibson-Mount wrote:
--On Wednesday, November 07, 2007 10:37 PM +0800 Zhang Weiwu zhangweiwu@realss.com wrote:
My question is: The 4 minute search time is not reasonable (Excel can do much faster with same number of records), and not acceptable for our use (because checking modification history is daily routine). How can I improve it? What other information I didn't provider to have asked this question properly? What else (besides admin manual which only described indexing a few pages) should I read?
Sounds like your index may be corrupt, or you haven't properly tuned the DB_CONFIG file for the accesslog DB. Alternately, when you ran slapindex, you failed to specify which database to index. See the -b and -n options to slapindex.
Thank you for the hint. As I explained my index is produced with -b option. When I try to tweak settings in DB_CONFIG I found tutorials about it http://k12linux.mesd.k12.or.us/ldap/performance.html which suggest me this:
The size of dbcachesize should be big enough to hold your largest index file (*.dbb).Unfortunately I do not have any *.dbb file. Besides log files I only get several other files that looks like index files (they ends in .bdb)
-rw-rw---- 1 openldap openldap 126M 2007-11-09 10:35 dn2id.bdb -rw-rw---- 1 openldap openldap 1,7G 2007-11-09 10:35 id2entry.bdb -rw-rw---- 1 openldap openldap 2,1M 2007-11-09 10:35 reqDN.bdb -rw-rw---- 1 openldap openldap 13M 2007-06-03 13:28 reqStart.bdb -rw-rw---- 1 openldap openldap 660K 2007-11-09 10:35 reqType.bdb
Since the biggest file (id2entry.bdb) is at 1,7G, does that suggest I should have dbcachesize of that size? I cannot understand why the cache can be so large, after all there are only 50,000 records in it. After all the query result of 'ldapsearch reqStart=* reqstart' is only 6.8MB. But anyway I added these configuration:
dbcachesize 1740000000 cachesize 6000
Which is very close to my biggest .bdb file size / number of records. The query time is reduced from 4 minutes to 3 minutes, still far from user expectation (less than 10 seconds). Tested query is like this:
(reqDN=uid=paul_smith,ou=contacts,ou=xxx,dc=xxx,dc=xxx)
I am going to try a lot more experiments, but would also like to hear if there are something looking significantly wrong in my system? Thank!
This time I think I am able to answer my own question.
1. id2entry.bdb is not only an index, it looks like the whole database; 2. the tutorial I refer to is about ldbm while I am using bdb. (should have noticed this when there is a difference in file extension name)
Sorry for your time because of my stupid mistake.
openldap-software@openldap.org
-
Quanah Gibson-Mount -
Zhang Weiwu