Hi,
Is there a way to log the time each operation took?
I have strange CPU load (~200%) with just ~15 operations per second. SRCH is >90% of all operations. All attributed involved in search a indexed (many single attribute indexes, ~30).
The point is to find which search operations a taking long time to develop a solution.
--On Thursday, September 05, 2013 9:05 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
Hi,
Is there a way to log the time each operation took?
I have strange CPU load (~200%) with just ~15 operations per second. SRCH is >90% of all operations. All attributed involved in search a indexed (many single attribute indexes, ~30).
The point is to find which search operations a taking long time to develop a solution.
OS? OpenLDAP version? OpenLDAP Backend? Do you see any issues in I/O wait? Size of database? Number of entries in database? etc. Provide some relevant information.
--Quanah
--
Quanah Gibson-Mount Lead Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
В Чтв, 05/09/2013 в 11:35 -0700, Quanah Gibson-Mount пишет:
--On Thursday, September 05, 2013 9:05 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
Hi,
Is there a way to log the time each operation took?
I have strange CPU load (~200%) with just ~15 operations per second. SRCH is >90% of all operations. All attributed involved in search a indexed (many single attribute indexes, ~30).
The point is to find which search operations a taking long time to develop a solution.
OS? OpenLDAP version? OpenLDAP Backend? Do you see any issues in I/O wait? Size of database? Number of entries in database? etc. Provide some relevant information.
OS: Ubuntu 12.04.2 LTS Slapd: 2.4.28-1.1ubuntu4.3 Backend: HDB FULL database ldif size: 2M /var/lib/ldap size: 220Mb (190Mb of which are 19 log.* files) Database entries: 1,7k
I/O wait as well as Disks Utilization is near zero.
The servers in question are slaves, so there is no visible disk activity, except for syslog.
The question is in high CPU usage considering small database and low rps.
--On Thursday, September 05, 2013 10:58 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
В Чтв, 05/09/2013 в 11:35 -0700, Quanah Gibson-Mount пишет:
--On Thursday, September 05, 2013 9:05 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
Hi,
Is there a way to log the time each operation took?
I have strange CPU load (~200%) with just ~15 operations per second. SRCH is >90% of all operations. All attributed involved in search a indexed (many single attribute indexes, ~30).
The point is to find which search operations a taking long time to develop a solution.
OS? OpenLDAP version? OpenLDAP Backend? Do you see any issues in I/O wait? Size of database? Number of entries in database? etc. Provide some relevant information.
OS: Ubuntu 12.04.2 LTS Slapd: 2.4.28-1.1ubuntu4.3
Ugh, ancient.
Backend: HDB
Yuck.
FULL database ldif size: 2M /var/lib/ldap size: 220Mb (190Mb of which are 19 log.* files) Database entries: 1,7k
Well, that is pretty tiny. So not likely to be a configuration issue.
Could be some of what was fixed in 2.4.32: Fixed slapd-bdb/hdb cache hang under high load (ITS#7222) or 2.4.31: Fixed slapd-bdb/hdb idlcache with only one element (ITS#7231)
personally, I would highly advise upgrading to a current release (2.4.36) and switching to back-mdb.
--Quanah
--
Quanah Gibson-Mount Lead Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount quanah@zimbra.com schrieb am 05.09.2013 um 22:58 in
Nachricht <0FCBC02976FFDC0CF5D9A489@[192.168.1.22]>:
--On Thursday, September 05, 2013 10:58 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
[...]
OS: Ubuntu 12.04.2 LTS Slapd: 2.4.28-1.1ubuntu4.3
Ugh, ancient.
Backend: HDB
Yuck.
[...]
Hi guys!
While I have nothing against bug-free software, I cannot read that "update to the latest version and database" any more: Is it really because the releases of the previous years that many people used were so terrible (which, by induction, means that the latest versions recommended by that time were terrible, so, as seen from tomorrow's perspective, the versions advertised today are also terribly full of bugs. In effect this means that there will never be a version that is not full of terrible bugs), or is it that no-one wants to care or take a look at about previous releases? Or are you just recruiting beta-testers for the current release?
Regards, Ulrich
Ulrich Windl wrote:
Quanah Gibson-Mount quanah@zimbra.com schrieb am 05.09.2013 um 22:58 in
Nachricht <0FCBC02976FFDC0CF5D9A489@[192.168.1.22]>:
--On Thursday, September 05, 2013 10:58 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
[...]
OS: Ubuntu 12.04.2 LTS Slapd: 2.4.28-1.1ubuntu4.3
Ugh, ancient.
Backend: HDB
Yuck.
[...]
Hi guys!
While I have nothing against bug-free software, I cannot read that "update to the latest version and database" any more: Is it really because the releases of the previous years that many people used were so terrible (which, by induction, means that the latest versions recommended by that time were terrible, so, as seen from tomorrow's perspective, the versions advertised today are also terribly full of bugs. In effect this means that there will never be a version that is not full of terrible bugs), or is it that no-one wants to care or take a look at about previous releases? Or are you just recruiting beta-testers for the current release?
It is Project policy to only investigate issues in the current release. There is no sense in tracing back thru old code whose bugs have already been fixed.
В Птн, 06/09/2013 в 04:42 -0700, Howard Chu пишет:
Ulrich Windl wrote:
Quanah Gibson-Mount quanah@zimbra.com schrieb am 05.09.2013 um 22:58 in
Nachricht <0FCBC02976FFDC0CF5D9A489@[192.168.1.22]>:
--On Thursday, September 05, 2013 10:58 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
[...]
OS: Ubuntu 12.04.2 LTS Slapd: 2.4.28-1.1ubuntu4.3
Ugh, ancient.
Backend: HDB
Yuck.
[...]
Hi guys!
While I have nothing against bug-free software, I cannot read that "update to the latest version and database" any more: Is it really because the releases of the previous years that many people used were so terrible (which, by induction, means that the latest versions recommended by that time were terrible, so, as seen from tomorrow's perspective, the versions advertised today are also terribly full of bugs. In effect this means that there will never be a version that is not full of terrible bugs), or is it that no-one wants to care or take a look at about previous releases? Or are you just recruiting beta-testers for the current release?
It is Project policy to only investigate issues in the current release. There is no sense in tracing back thru old code whose bugs have already been fixed.
This means old versions are not supported and makes problems with openldap distribution packages as distributions don't update upstream versions inside distribution version. :(
For Debian that means staying with bugs for >2 years. It's hard to call this policy "right".
??????????? ??????casper@meteor.dp.ua schrieb am 06.09.2013 um 14:05 in
Nachricht 1378469133.18073.55.camel@casper-hp.friendin.net:
В Птн, 06/09/2013 в 04:42 -0700, Howard Chu пишет:
Ulrich Windl wrote:
Quanah Gibson-Mount quanah@zimbra.com schrieb am 05.09.2013 um 22:58
in
Nachricht <0FCBC02976FFDC0CF5D9A489@[192.168.1.22]>:
--On Thursday, September 05, 2013 10:58 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
[...]
OS: Ubuntu 12.04.2 LTS Slapd: 2.4.28-1.1ubuntu4.3
Ugh, ancient.
Backend: HDB
Yuck.
[...]
Hi guys!
While I have nothing against bug-free software, I cannot read that
"update
to
the latest version and database" any more: Is it really because the
releases of
the previous years that many people used were so terrible (which, by
induction,
means that the latest versions recommended by that time were terrible,
so,
as
seen from tomorrow's perspective, the versions advertised today are also terribly full of bugs. In effect this means that there will never be a
version
that is not full of terrible bugs), or is it that no-one wants to care or
take
a look at about previous releases? Or are you just recruiting
beta-testers
for
the current release?
It is Project policy to only investigate issues in the current release.
There
is no sense in tracing back thru old code whose bugs have already been
fixed.
This means old versions are not supported and makes problems with openldap distribution packages as distributions don't update upstream versions inside distribution version. :(
For Debian that means staying with bugs for >2 years. It's hard to call this policy "right".
Hi!
Actually I don't know which distributors are "back-porting" fixes, but from my personal experience distributors don't trust the latest release either (and thus keep what they have) ;-)
Regards, Ulrich
--On Friday, September 06, 2013 2:16 PM +0200 Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de wrote:
Hi!
Actually I don't know which distributors are "back-porting" fixes, but from my personal experience distributors don't trust the latest release either (and thus keep what they have) ;-)
I would note that even the Debian openldap maintainers say not to use their packages for production:
http://www.openldap.org/faq/data/cache/1456.html
--Quanah
--
Quanah Gibson-Mount Lead Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
В Птн, 06/09/2013 в 06:55 -0700, Quanah Gibson-Mount пишет:
--On Friday, September 06, 2013 2:16 PM +0200 Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de wrote:
Hi!
Actually I don't know which distributors are "back-porting" fixes, but from my personal experience distributors don't trust the latest release either (and thus keep what they have) ;-)
I would note that even the Debian openldap maintainers say not to use their packages for production:
Look at my previous message. Personally, I think that switching development model to something that can support few old version would be a win for both sides. Like, adding 4th version component for backported/redone bug fixes.
As far as I know, many software following such model, linux kernel is an example.
Покотиленко Костик wrote:
В Птн, 06/09/2013 в 04:42 -0700, Howard Chu пишет:
Ulrich Windl wrote:
Quanah Gibson-Mount quanah@zimbra.com schrieb am 05.09.2013 um 22:58 in
Nachricht <0FCBC02976FFDC0CF5D9A489@[192.168.1.22]>:
--On Thursday, September 05, 2013 10:58 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
[...]
OS: Ubuntu 12.04.2 LTS Slapd: 2.4.28-1.1ubuntu4.3
Ugh, ancient.
Backend: HDB
Yuck.
[...]
Hi guys!
While I have nothing against bug-free software, I cannot read that "update to the latest version and database" any more: Is it really because the releases of the previous years that many people used were so terrible (which, by induction, means that the latest versions recommended by that time were terrible, so, as seen from tomorrow's perspective, the versions advertised today are also terribly full of bugs. In effect this means that there will never be a version that is not full of terrible bugs), or is it that no-one wants to care or take a look at about previous releases? Or are you just recruiting beta-testers for the current release?
It is Project policy to only investigate issues in the current release. There is no sense in tracing back thru old code whose bugs have already been fixed.
This means old versions are not supported and makes problems with openldap distribution packages as distributions don't update upstream versions inside distribution version. :(
For Debian that means staying with bugs for >2 years. It's hard to call this policy "right".
Distro packages are supported by their distros. We have no way to support them anyway since they tend to insert their own private patches and we have no visibility into what they changed. (Nor do we want it - there's doeznes of distros out there and it's not our responsibility to keep track of what they're all doing.) And in the specific case of Debian, given their history of introducing critical bugs into their builds http://www.schneier.com/blog/archives/2008/05/random_number_b.html there is no way any upstream project will ever take responsibility for supporting Debian packages.
You may not think this policy is "right" but it's the only practical approach when distros take liberties with what they release.
On Sep 6, 2013, at 14:05, Покотиленко Костик casper@meteor.dp.ua wrote:
В Птн, 06/09/2013 в 04:42 -0700, Howard Chu пишет:
It is Project policy to only investigate issues in the current release. There is no sense in tracing back thru old code whose bugs have already been fixed.
This means old versions are not supported and makes problems with openldap distribution packages as distributions don't update upstream versions inside distribution version. :(
For Debian that means staying with bugs for >2 years. It's hard to call this policy "right".
You're complaining in the wrong place. How about complaining about the <insert your favorite Linux distribution here> policy to not track the OpenLDAP releases closely?
You seem to think if a Linux distribution releases a certain old OpenLDAP version then it's the OpenLDAP maintainers' job to support that version as long as the distribution ships it. That's pretty unfair to the OpenLDAP maintainers. They had no say in the distribution's decision to ship the old version. Think about it.
jens
В Птн, 06/09/2013 в 15:24 +0200, Jens Vagelpohl пишет:
On Sep 6, 2013, at 14:05, Покотиленко Костик casper@meteor.dp.ua wrote:
В Птн, 06/09/2013 в 04:42 -0700, Howard Chu пишет:
It is Project policy to only investigate issues in the current release. There is no sense in tracing back thru old code whose bugs have already been fixed.
This means old versions are not supported and makes problems with openldap distribution packages as distributions don't update upstream versions inside distribution version. :(
For Debian that means staying with bugs for >2 years. It's hard to call this policy "right".
You're complaining in the wrong place. How about complaining about the <insert your favorite Linux distribution here> policy to not track the OpenLDAP releases closely?
You seem to think if a Linux distribution releases a certain old OpenLDAP version then it's the OpenLDAP maintainers' job to support that version as long as the distribution ships it. That's pretty unfair to the OpenLDAP maintainers. They had no say in the distribution's decision to ship the old version. Think about it.
I'm not complaining. I'm looking for a better way of upstream -> end-user.
What I was trying to tell was: if openldap team could backport fixes (without new features) to old versions - then distributors could update packages not breaking their policy.
The thing is that I see clear split to conservative anti-distro point - "compile yourself" and distro-oriented "stay with bugs".
If it's possible to find a compromise - everybody will benefit, isn't it?
--On Friday, September 06, 2013 7:05 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
I'm not complaining. I'm looking for a better way of upstream -> end-user.
What I was trying to tell was: if openldap team could backport fixes (without new features) to old versions - then distributors could update packages not breaking their policy.
The thing is that I see clear split to conservative anti-distro point - "compile yourself" and distro-oriented "stay with bugs".
I don't think you understand software versionsing:
MAJOR.MINOR.PATCH
There is no reason for distributions not to update to a later patch level. "backporting" fixes inside a release makes no sense.. it is still the 2.4 release.
--Quanah
--
Quanah Gibson-Mount Lead Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
В Птн, 06/09/2013 в 09:31 -0700, Quanah Gibson-Mount пишет:
--On Friday, September 06, 2013 7:05 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
I'm not complaining. I'm looking for a better way of upstream -> end-user.
What I was trying to tell was: if openldap team could backport fixes (without new features) to old versions - then distributors could update packages not breaking their policy.
The thing is that I see clear split to conservative anti-distro point - "compile yourself" and distro-oriented "stay with bugs".
I don't think you understand software versionsing:
MAJOR.MINOR.PATCH
There is no reason for distributions not to update to a later patch level. "backporting" fixes inside a release makes no sense.. it is still the 2.4 release.
The reason is that openldap's PATCH component includes new features (that by itself introduces new bugs) rather than only FIXES to existing features. This breaks disto's policy and this is the point.
Покотиленко Костик wrote:
The reason is that openldap's PATCH component includes new features (that by itself introduces new bugs) rather than only FIXES to existing features. This breaks disto's policy and this is the point.
Distribution policy does not matter here. What matters is continous improvement of OpenLDAP itself.
Ciao, Michael.
Michael: I cannot tell if you're being sarcastic or not, so, I'm running with your words:
Software isn't developed in a vacuum - when truly useful, it's intended use it to be used and it cannot be used sans distros (in any realistic production operation; sure you can compile everything from source and create StroderOS). While you may be blessed with using whatever software, from whatever source you desire, with any (or no) support available, many system administrators are under edicts and must work within the policies and instructions of their company.
SOX is a big deal at any organization that is publicly traded or works with government entities.
The support model of essentially "it's not the latest; go away until you update (compile it)" isn't helpful.
As for the request that started this thread; he was after information (on how to get performance related information), not 'please tell me what is broken in my environment': "Is there a way to log the time each operation took?" ... "The point is to find which search operations a taking long time to develop a solution." Which generated only one (useful) response: Quanah: "I would highly advise upgrading to a current release (2.4.36) and switching to back-mdb."
If the information Casper requested isn't available, say so. If it is, how would he get it?
Either way "upgrade to the latest" doesn’t address his actual question(s).
Continuous improvement is an excellent goal, but I don't see how answering questions on "not very recent" versions is a bad thing.
- chris
PS: Just my 2 cents.
-----Original Message----- From: openldap-technical-bounces@OpenLDAP.org [mailto:openldap-technical-bounces@OpenLDAP.org] On Behalf Of Michael Ströder Sent: Friday, September 06, 2013 11:37 AM To: casper@meteor.dp.ua Cc: openldap-technical@openldap.org Subject: Re: Antw: Re: Log service time?
Покотиленко Костик wrote:
The reason is that openldap's PATCH component includes new features (that by itself introduces new bugs) rather than only FIXES to existing features. This breaks disto's policy and this is the point.
Distribution policy does not matter here. What matters is continous improvement of OpenLDAP itself.
Ciao, Michael.
This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.
Chris Jacobs wrote:
Michael: I cannot tell if you're being sarcastic or not, so, I'm running with your words:
I'm completely serious.
Software isn't developed in a vacuum - when truly useful, it's intended use it to be used and it cannot be used sans distros (in any realistic production operation; sure you can compile everything from source and create StroderOS).
Are you sarcastic here?
Serious: Using packages from a Linux distribution is the normal case and works fine mostly for cases where you don't use very advanced features of a given software. If you experience certain bugs in a software packaged by your distro then build a newer distro package of that software. Yes, that's work but it's worth the effort if the component is really important for your infrastructure. This is about getting real control over important components.
While you may be blessed with using whatever software, from whatever source you desire, with any (or no) support available, many system administrators are under edicts and must work within the policies and instructions of their company.
Policies and instructions are always subject to controlled change process. If you don't have a change process then your policies are missing a very essential part.
SOX is a big deal at any organization that is publicly traded or works with government entities.
Be assured that I'm quite aware of what it means to run systems governed by lots of policies.
The support model of essentially "it's not the latest; go away until you update (compile it)" isn't helpful.
Do you have a support contract with the OpenLDAP community? Everything here is community effort. If you insist on getting commercial-grade support model then pay people providing support (e.g. buy Symas' build for your favourite OS platform).
Quanah: "I would highly advise upgrading to a current release (2.4.36) and switching to back-mdb."
While I personally agree that Quanah should not just write "yuck" because of someone is using back-hdb (given that back-mdb is really usable just since 2.4.36) he's right in pointing out that someone should try to reproduce issues with the latest release first before asking.
Ciao, Michael.
??????????? ??????casper@meteor.dp.ua schrieb am 06.09.2013 um 19:05 in
Nachricht 1378487147.18073.75.camel@casper-hp.friendin.net:
В Птн, 06/09/2013 в 09:31 -0700, Quanah Gibson-Mount пишет:
--On Friday, September 06, 2013 7:05 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
I'm not complaining. I'm looking for a better way of upstream -> end-user.
What I was trying to tell was: if openldap team could backport fixes (without new features) to old versions - then distributors could update packages not breaking their policy.
The thing is that I see clear split to conservative anti-distro point - "compile yourself" and distro-oriented "stay with bugs".
I don't think you understand software versionsing:
MAJOR.MINOR.PATCH
There is no reason for distributions not to update to a later patch level.
"backporting" fixes inside a release makes no sense.. it is still the 2.4 release.
The reason is that openldap's PATCH component includes new features (that by itself introduces new bugs) rather than only FIXES to existing features. This breaks disto's policy and this is the point.
So maybe what's missing is a definition what qualifies as MAJOR version, MINOR version, and PATCH (plus the policy enforcement to comply with the definitions).
(Non-native English, I hope I used the correct words)
Regards, Ulrich
В Птн, 06/09/2013 в 08:15 +0200, Ulrich Windl пишет:
Quanah Gibson-Mount quanah@zimbra.com schrieb am 05.09.2013 um 22:58 in
Nachricht <0FCBC02976FFDC0CF5D9A489@[192.168.1.22]>:
--On Thursday, September 05, 2013 10:58 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
[...]
OS: Ubuntu 12.04.2 LTS Slapd: 2.4.28-1.1ubuntu4.3
Ugh, ancient.
Backend: HDB
Yuck.
[...]
Hi guys!
While I have nothing against bug-free software, I cannot read that "update to the latest version and database" any more: Is it really because the releases of the previous years that many people used were so terrible (which, by induction, means that the latest versions recommended by that time were terrible, so, as seen from tomorrow's perspective, the versions advertised today are also terribly full of bugs. In effect this means that there will never be a version that is not full of terrible bugs), or is it that no-one wants to care or take a look at about previous releases? Or are you just recruiting beta-testers for the current release?
+1
В Чтв, 05/09/2013 в 13:58 -0700, Quanah Gibson-Mount пишет:
--On Thursday, September 05, 2013 10:58 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
В Чтв, 05/09/2013 в 11:35 -0700, Quanah Gibson-Mount пишет:
--On Thursday, September 05, 2013 9:05 PM +0300 Покотиленко Костик casper@meteor.dp.ua wrote:
Hi,
Is there a way to log the time each operation took?
I have strange CPU load (~200%) with just ~15 operations per second. SRCH is >90% of all operations. All attributed involved in search a indexed (many single attribute indexes, ~30).
The point is to find which search operations a taking long time to develop a solution.
OS? OpenLDAP version? OpenLDAP Backend? Do you see any issues in I/O wait? Size of database? Number of entries in database? etc. Provide some relevant information.
OS: Ubuntu 12.04.2 LTS Slapd: 2.4.28-1.1ubuntu4.3
Ugh, ancient.
Backend: HDB
Yuck.
FULL database ldif size: 2M /var/lib/ldap size: 220Mb (190Mb of which are 19 log.* files) Database entries: 1,7k
Well, that is pretty tiny. So not likely to be a configuration issue.
Could be some of what was fixed in 2.4.32: Fixed slapd-bdb/hdb cache hang under high load (ITS#7222) or 2.4.31: Fixed slapd-bdb/hdb idlcache with only one element (ITS#7231)
personally, I would highly advise upgrading to a current release (2.4.36) and switching to back-mdb.
Thanks for the pointers. I'll consider trying more recent versions.
Покотиленко Костик wrote:
Hi,
Is there a way to log the time each operation took?
Every message in syslog already has a timestamp.
Also, recent versions of OpenLDAP include a timestamp on debug output too. You didn't mention which version you're using so can't tell you much more.
I have strange CPU load (~200%) with just ~15 operations per second. SRCH is >90% of all operations. All attributed involved in search a indexed (many single attribute indexes, ~30).
The point is to find which search operations a taking long time to develop a solution.
В Чтв, 05/09/2013 в 11:56 -0700, Howard Chu пишет:
Покотиленко Костик wrote:
Hi,
Is there a way to log the time each operation took?
Every message in syslog already has a timestamp.
Every message in syslog has time the operation has finished at. What I need is the time it took each operation to complete (in milliseconds or microseconds).
It would be strange if the is no way to get this information.
Also, recent versions of OpenLDAP include a timestamp on debug output too. You didn't mention which version you're using so can't tell you much more.
I have strange CPU load (~200%) with just ~15 operations per second. SRCH is >90% of all operations. All attributed involved in search a indexed (many single attribute indexes, ~30).
The point is to find which search operations a taking long time to develop a solution.
openldap-technical@openldap.org
-
Chris Jacobs -
Howard Chu -
Jens Vagelpohl -
Michael Ströder -
Quanah Gibson-Mount -
Ulrich Windl -
Покотиленко Костик