10:05 a.m.
Hi Quanah,
Thanks for replying, I have few more observation regrading my load runs,
real time of load run with MDB is 98% even at 10 TPS.
What version of OpenLDAP are you using?
---2.4.32
What type of disk?
--root@tspatca2103> fdisk -l /dev/mapper/vg00-root
Disk /dev/mapper/vg00-root: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/vg00-root doesn't contain a valid partition table
root@tspatca2103>
What type of file system?
--ext4 (/dev/mapper/vg00-root on / type ext4 (rw,nouser_xattr))
What is your *exact* slapadd command?
--/opt/openldap/yaju/sbin/slapadd -q -w -f
/opt/openldap/yaju/etc/openldap/slapd.conf -l /root/yaju/db.ldif
I am using jmeter to simulate the load.
--
Thanks and Regards
Yajuvendra
On Tue, Aug 28, 2012 at 8:19 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote:
--On Tuesday, August 28, 2012 5:29 PM +0530 Yajuvendra Singh <
yajuvendra.singh(a)gmail.com> wrote:
Dear Experts,
>
>
> Today we tried with to run the load with the below schema. We added about
> .6M entries in the DB.
> Still our performance is severely poor. (10 TPS)
>
>
> Can anybody review our slapd.conf file and point us where we are wrong,
> is there any other config we have missed out.
>
You don't provide any useful or relevant information, so it is impossible
to help you.
What version of OpenLDAP are you using?
What type of disk?
What type of file system?
What is your *exact* slapadd command?
etc.
There is virtually no tuning involved with MDB, although I strongly
recommend you read Howard's notes about the writeback bits for EXT4 etc he
made in a recent post to -technical about MDB.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
10:14 p.m.
New subject: Performance of MDB and BDB Please suggest?
Hi,
I did some profiling on the 2.4.32 build and the output of callgrind is
attached as graph.
below four functions are coming as bottleneck while searching
mdb_cursor_set
mdb_entry_decode
mdb_i2dl_insert
mdb_i2dl_search
Can anyone help why below functions are taking too much CPU. Why
mdb_entry_decode mdb_cursor_set and mdb_i2dl_search are taking all most all
CPU.
Regards,
Yajuvendra
On Tue, Aug 28, 2012 at 10:35 PM, Yajuvendra Singh <
yajuvendra.singh(a)gmail.com> wrote:
Hi Quanah,
Thanks for replying, I have few more observation regrading my load runs,
real time of load run with MDB is 98% even at 10 TPS.
What version of OpenLDAP are you using?
---2.4.32
What type of disk?
--root@tspatca2103> fdisk -l /dev/mapper/vg00-root
Disk /dev/mapper/vg00-root: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/vg00-root doesn't contain a valid partition table
root@tspatca2103>
What type of file system?
--ext4 (/dev/mapper/vg00-root on / type ext4 (rw,nouser_xattr))
What is your *exact* slapadd command?
--/opt/openldap/yaju/sbin/slapadd -q -w -f
/opt/openldap/yaju/etc/openldap/slapd.conf -l /root/yaju/db.ldif
I am using jmeter to simulate the load.
--
Thanks and Regards
Yajuvendra
On Tue, Aug 28, 2012 at 8:19 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote:
> --On Tuesday, August 28, 2012 5:29 PM +0530 Yajuvendra Singh <
> yajuvendra.singh(a)gmail.com> wrote:
>
> Dear Experts,
>>
>>
>> Today we tried with to run the load with the below schema. We added about
>> .6M entries in the DB.
>> Still our performance is severely poor. (10 TPS)
>>
>>
>> Can anybody review our slapd.conf file and point us where we are wrong,
>> is there any other config we have missed out.
>>
>
> You don't provide any useful or relevant information, so it is impossible
> to help you.
>
> What version of OpenLDAP are you using?
> What type of disk?
> What type of file system?
> What is your *exact* slapadd command?
> etc.
>
> There is virtually no tuning involved with MDB, although I strongly
> recommend you read Howard's notes about the writeback bits for EXT4 etc he
> made in a recent post to -technical about MDB.
>
>
> --Quanah
>
>
> --
>
> Quanah Gibson-Mount
> Sr. Member of Technical Staff
> Zimbra, Inc
> A Division of VMware, Inc.
> --------------------
> Zimbra :: the leader in open source messaging and collaboration
>
10:35 p.m.
New subject: Performance of MDB and BDB Please suggest?
Yajuvendra Singh wrote:
Hi,
I did some profiling on the 2.4.32 build and the output of callgrind is
attached as graph.
below four functions are coming as bottleneck while searching
mdb_cursor_set
mdb_entry_decode
mdb_i2dl_insert
mdb_i2dl_search
Can anyone help why below functions are taking too much CPU. Why
mdb_entry_decode mdb_cursor_set and mdb_i2dl_search are taking all most all CPU.
Looks like your search is not indexed.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
11:21 p.m.
New subject: Performance of MDB and BDB Please suggest?
Hi,
I feel search is indexed, last time in one of your replies you had
mentioned mdb_stat tool and i think the indexes are present.
Any way to debug the problem.
root@tspatca2103> ./mdb_stat
/root/haroon/openldap/new1/OnBoardSPR/database/ objectClass
Page size: 4096
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 44
root@tspatca2103> ./mdb_stat
/root/haroon/openldap/new1/OnBoardSPR/database/ entryCSN
Page size: 4096
Tree depth: 2
Branch pages: 1
Leaf pages: 4
Overflow pages: 0
Entries: 1000016
root@tspatca2103> ./mdb_stat
/root/haroon/openldap/new1/OnBoardSPR/database/ entryUUID
Page size: 4096
Tree depth: 3
Branch pages: 33
Leaf pages: 7775
Overflow pages: 0
Entries: 1000016
root@tspatca2103> ./mdb_stat
/root/haroon/openldap/new1/OnBoardSPR/database/ msisdn
Page size: 4096
Tree depth: 3
Branch pages: 20
Leaf pages: 4446
Overflow pages: 0
Entries: 400001
root@tspatca2103>
Regards,
Yajuvendra
On Fri, Aug 31, 2012 at 11:05 AM, Howard Chu <hyc(a)symas.com> wrote:
Yajuvendra Singh wrote:
> Hi,
>
> I did some profiling on the 2.4.32 build and the output of callgrind is
> attached as graph.
>
> below four functions are coming as bottleneck while searching
>
> mdb_cursor_set
> mdb_entry_decode
> mdb_i2dl_insert
> mdb_i2dl_search
>
> Can anyone help why below functions are taking too much CPU. Why
> mdb_entry_decode mdb_cursor_set and mdb_i2dl_search are taking all most
all CPU.
Looks like your search is not indexed.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
11:37 p.m.
New subject: Performance of MDB and BDB Please suggest?
Yajuvendra Singh wrote:
Hi,
I feel search is indexed, last time in one of your replies you had mentioned
mdb_stat tool and i think the indexes are present.
What you feel is irrelevant. Give us facts.
What is the actual search request? What filter is in the request? You post no
useful information in any of your emails.
In the slapd.conf you posted previously, some of your index directives were
commented out. It sounds like you actually have no idea what is or isn't indexed.
Regards,
Yajuvendra
On Fri, Aug 31, 2012 at 11:05 AM, Howard Chu <hyc(a)symas.com
<mailto:hyc@symas.com>> wrote:
Yajuvendra Singh wrote:
> Hi,
>
> I did some profiling on the 2.4.32 build and the output of callgrind is
> attached as graph.
>
> below four functions are coming as bottleneck while searching
>
> mdb_cursor_set
> mdb_entry_decode
> mdb_i2dl_insert
> mdb_i2dl_search
>
> Can anyone help why below functions are taking too much CPU. Why
> mdb_entry_decode mdb_cursor_set and mdb_i2dl_search are taking all most
all CPU.
Looks like your search is not indexed.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
12:02 a.m.
New subject: Performance of MDB and BDB Please suggest?
Hi Howard,
I am attaching the wireshark logs for the search request. Also attaching
the conf file which we are currently using for load and performance run.
*Startup logs.*
50405dd3 line 20 (pidfile
/root/haroon/openldap/new1/OnBoardSPR/var/run/slapd.pid)
50405dd3 line 21 (argsfile
/root/haroon/openldap/new1/OnBoardSPR/var/run/slapd.args)
50405dd3 line 23 (logfile /tmp/log/onboardspr.log)
50405dd3 line 26 (sizelimit 10000)
50405dd3 line 27 (timelimit 2)
50405dd3 line 29 (database mdb)
50405dd3 line 30 (suffix "dc=C-NTDB")
50405dd3 line 31 (rootdn "dc=C-NTDB")
50405dd3 line 32 (rootpw ***)
50405dd3 line 37 (access to dn.subtree="dc=MSISDN, dc=C-NTDB"
filter=(objectClass=*) by
group.exact="cn=pcsAccess,ou=groups,dc=C-NTDB" write by
group.exact="cn=sdfAccess,ou=groups,dc=C-NTDB" write by * read)
50405dd3 line 41 (access to dn.subtree="ds=DEVICE, dc=C-NTDB"
filter=(objectClass=*) by
group.exact="cn=sdfAccess,ou=groups,dc=C-NTDB" write by * read)
50405dd3 line 45 (access to dn.subtree="o=service, dc=C-NTDB"
filter=(objectClass=*) by
group.exact="cn=sdfAccess,ou=groups,dc=C-NTDB" write by * read)
50405dd3 line 50 (access to * by users read by dn="dc=C-NTDB"
write by * read)
50405dd3
/root/haroon/openldap/new1/OnBoardSPR/etc/openldap/onboardSPR.conf: line
50: rootdn is always granted unlimited privileges.
50405dd3 line 52 (index objectClass eq)
50405dd3 index objectClass 0x0004
50405dd3 line 54 (index msisdn eq)
50405dd3 index msisdn 0x0004
50405dd3 line 56 (index entryCSN,entryUUID eq)
50405dd3 index entryCSN 0x0004
50405dd3 index entryUUID 0x0004
50405dd3 line 57 (maxsize 16106127360)
50405dd3 line 60 (directory
/root/haroon/openldap/new1/OnBoardSPR/database)
50405dd3 config_back_db_open: No explicit ACL for back-config configured.
Using hardcoded default
50405dd3 mdb_monitor_db_open: monitoring disabled; configure monitor
database to enable
50405dd3 slapd starting
Regards,
Yajuvendra
On Fri, Aug 31, 2012 at 12:07 PM, Howard Chu <hyc(a)symas.com> wrote:
Yajuvendra Singh wrote:
> Hi,
>
> I feel search is indexed, last time in one of your replies you had
mentioned
> mdb_stat tool and i think the indexes are present.
What you feel is irrelevant. Give us facts.
What is the actual search request? What filter is in the request? You post
no
useful information in any of your emails.
In the slapd.conf you posted previously, some of your index directives were
commented out. It sounds like you actually have no idea what is or isn't
indexed.
>
> Regards,
> Yajuvendra
>
>
> On Fri, Aug 31, 2012 at 11:05 AM, Howard Chu <hyc(a)symas.com
> <mailto:hyc@symas.com>> wrote:
>
> Yajuvendra Singh wrote:
> > Hi,
> >
> > I did some profiling on the 2.4.32 build and the output of
callgrind is
> > attached as graph.
> >
> > below four functions are coming as bottleneck while searching
> >
> > mdb_cursor_set
> > mdb_entry_decode
> > mdb_i2dl_insert
> > mdb_i2dl_search
> >
> > Can anyone help why below functions are taking too much CPU. Why
> > mdb_entry_decode mdb_cursor_set and mdb_i2dl_search are taking all
most
> all CPU.
>
> Looks like your search is not indexed.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
1:53 a.m.
New subject: Performance of MDB and BDB Please suggest?
Hi Howard,
I was trying to find more in the indexing part. I see that there is no
"dn2id" and "id2entry" in the mdb database which i had created but in
one
of the responses you had mentioned about them (
http://www.openldap.org/lists/openldap-technical/201208/msg00175.html)
Is it the place where i am wrong?
I am executing below commands.
root@tspatca2103> ./mdb_stat
/root/haroon/openldap/new1/OnBoardSPR/database/ dn2id
mdb_open failed 1, error -30798
root@tspatca2103> ./mdb_stat
/root/haroon/openldap/new1/OnBoardSPR/database/ id2entry
mdb_open failed 1, error -30798
root@tspatca2103>
Regards,
Yajuvendra
On Fri, Aug 31, 2012 at 12:32 PM, Yajuvendra Singh <
yajuvendra.singh(a)gmail.com> wrote:
Hi Howard,
I am attaching the wireshark logs for the search request. Also attaching
the conf file which we are currently using for load and performance run.
*Startup logs.*
50405dd3 line 20 (pidfile
/root/haroon/openldap/new1/OnBoardSPR/var/run/slapd.pid)
50405dd3 line 21 (argsfile
/root/haroon/openldap/new1/OnBoardSPR/var/run/slapd.args)
50405dd3 line 23 (logfile /tmp/log/onboardspr.log)
50405dd3 line 26 (sizelimit 10000)
50405dd3 line 27 (timelimit 2)
50405dd3 line 29 (database mdb)
50405dd3 line 30 (suffix "dc=C-NTDB")
50405dd3 line 31 (rootdn "dc=C-NTDB")
50405dd3 line 32 (rootpw ***)
50405dd3 line 37 (access to dn.subtree="dc=MSISDN, dc=C-NTDB"
filter=(objectClass=*) by
group.exact="cn=pcsAccess,ou=groups,dc=C-NTDB" write by
group.exact="cn=sdfAccess,ou=groups,dc=C-NTDB" write by * read)
50405dd3 line 41 (access to dn.subtree="ds=DEVICE, dc=C-NTDB"
filter=(objectClass=*) by
group.exact="cn=sdfAccess,ou=groups,dc=C-NTDB" write by * read)
50405dd3 line 45 (access to dn.subtree="o=service, dc=C-NTDB"
filter=(objectClass=*) by
group.exact="cn=sdfAccess,ou=groups,dc=C-NTDB" write by * read)
50405dd3 line 50 (access to * by users read by dn="dc=C-NTDB"
write by * read)
50405dd3
/root/haroon/openldap/new1/OnBoardSPR/etc/openldap/onboardSPR.conf: line
50: rootdn is always granted unlimited privileges.
50405dd3 line 52 (index objectClass eq)
50405dd3 index objectClass 0x0004
50405dd3 line 54 (index msisdn eq)
50405dd3 index msisdn 0x0004
50405dd3 line 56 (index entryCSN,entryUUID eq)
50405dd3 index entryCSN 0x0004
50405dd3 index entryUUID 0x0004
50405dd3 line 57 (maxsize 16106127360)
50405dd3 line 60 (directory
/root/haroon/openldap/new1/OnBoardSPR/database)
50405dd3 config_back_db_open: No explicit ACL for back-config configured.
Using hardcoded default
50405dd3 mdb_monitor_db_open: monitoring disabled; configure monitor
database to enable
50405dd3 slapd starting
Regards,
Yajuvendra
On Fri, Aug 31, 2012 at 12:07 PM, Howard Chu <hyc(a)symas.com> wrote:
> Yajuvendra Singh wrote:
> > Hi,
> >
> > I feel search is indexed, last time in one of your replies you had
> mentioned
> > mdb_stat tool and i think the indexes are present.
>
> What you feel is irrelevant. Give us facts.
>
> What is the actual search request? What filter is in the request? You
> post no
> useful information in any of your emails.
>
> In the slapd.conf you posted previously, some of your index directives
> were
> commented out. It sounds like you actually have no idea what is or isn't
> indexed.
>
>
> >
> > Regards,
> > Yajuvendra
> >
> >
> > On Fri, Aug 31, 2012 at 11:05 AM, Howard Chu <hyc(a)symas.com
> > <mailto:hyc@symas.com>> wrote:
> >
> > Yajuvendra Singh wrote:
> > > Hi,
> > >
> > > I did some profiling on the 2.4.32 build and the output of
> callgrind is
> > > attached as graph.
> > >
> > > below four functions are coming as bottleneck while searching
> > >
> > > mdb_cursor_set
> > > mdb_entry_decode
> > > mdb_i2dl_insert
> > > mdb_i2dl_search
> > >
> > > Can anyone help why below functions are taking too much CPU. Why
> > > mdb_entry_decode mdb_cursor_set and mdb_i2dl_search are taking
> all most
> > all CPU.
> >
> > Looks like your search is not indexed.
>
> --
> -- Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
>
2:41 a.m.
New subject: Performance of MDB and BDB Please suggest?
Bonjour,
I'm no LDAP expert, so what follows may be stupid. I'm OK with this,
but I'd prefer receiving pointers to make my mind clearer.
Looking at your network capture, I see the search you're doing has its
baseObject set to "msisdn=9800002868;dc=MSISDN,dc=C-NTDB". Is the
semicolon intentional?
Looking at RFC4514, chapter 3 "Parsing a String Back to a
Distinguished Name", a COMMA separates the RDN elements, and a SEMI
(semicolon, U+003B) must be escaped.
My guess is this semicolon is an error and should be a comma instead.
2012/8/31 Yajuvendra Singh <yajuvendra.singh(a)gmail.com>:
Hi Howard,
I am attaching the wireshark logs for the search request. Also attaching the
conf file which we are currently using for load and performance run.
--
Erwann.
3:04 a.m.
New subject: Performance of MDB and BDB Please suggest?
Le 8/31/12 11:41 AM, Erwann Abalea a écrit :
Bonjour,
I'm no LDAP expert, so what follows may be stupid. I'm OK with this,
but I'd prefer receiving pointers to make my mind clearer.
Looking at your network capture, I see the search you're doing has its
baseObject set to "msisdn=9800002868;dc=MSISDN,dc=C-NTDB". Is the
semicolon intentional?
Looking at RFC4514, chapter 3 "Parsing a String Back to a
Distinguished Name", a COMMA separates the RDN elements, and a SEMI
(semicolon, U+003B) must be escaped.
My guess is this semicolon is an error and should be a comma instead.
Semi-colon are legal separators in DNs. It's just old school, and you'd
better avoid using them...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
3861
days inactive
3864
days old
openldap-technical@openldap.org
8 comments
4 participants
participants (4)
-
Emmanuel Lécharny -
Erwann Abalea -
Howard Chu -
Yajuvendra Singh