6:27 p.m.
Does anybody know of any good tools that can rip through an openldap log
file and analyze it, creating a report of what queries are being made
and how long they are taking to process?
All of the information I'm interested in is included in the log:
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 fd=84 ACCEPT from
IP=134.71.247.28:46592 (IP=0.0.0.0:636)
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 fd=84 TLS established
tls_ssf=256 ssf=256 tls_proto=TLSv1.2 tls_cipher=ECDHE-RSA-AES256-GCM-SHA384
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=0 BIND
dn="cn=it_boomi,ou=user,ou=service,dc=cpp,dc=edu" method=128
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=0 BIND
dn="cn=it_boomi,ou=user,ou=service,dc=cpp,dc=edu" mech=SIMPLE bind_ssf=0
ssf=256
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=0 RESULT tag=97 err=0
qtime=0.000031 etime=0.000189 text=
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=1 SRCH
base="ou=user,dc=cpp,dc=edu" scope=2 deref=3
filter="(&(objectClass=person)(calstateEduPersonEmplID=014532336))"
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=1 SRCH attr=memberOf
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=2 UNBIND
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=1 SEARCH RESULT
tag=101 err=0 qtime=0.000016 etime=0.192994 nentries=1 text=
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 fd=84 closed
but split up into a number of different lines which need to be
correlated to summarize it. Before I try it myself I was hoping somebody
else had already scratched that itch :). The only things I can find
searching are either really old or commercial products.
Thanks much…
7:12 p.m.
https://www.ltb-project.org/documentation/ldap-stats.html
This is a nice script for that
On Feb 4, 2022, 9:27 PM -0500, Paul B. Henson <henson(a)acm.org>, wrote:
Does anybody know of any good tools that can rip through an openldap
log
file and analyze it, creating a report of what queries are being made
and how long they are taking to process?
All of the information I'm interested in is included in the log:
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 fd=84 ACCEPT from
IP=134.71.247.28:46592 (IP=0.0.0.0:636)
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 fd=84 TLS established
tls_ssf=256 ssf=256 tls_proto=TLSv1.2 tls_cipher=ECDHE-RSA-AES256-GCM-SHA384
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=0 BIND
dn="cn=it_boomi,ou=user,ou=service,dc=cpp,dc=edu" method=128
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=0 BIND
dn="cn=it_boomi,ou=user,ou=service,dc=cpp,dc=edu" mech=SIMPLE bind_ssf=0
ssf=256
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=0 RESULT tag=97 err=0
qtime=0.000031 etime=0.000189 text=
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=1 SRCH
base="ou=user,dc=cpp,dc=edu" scope=2 deref=3
filter="(&(objectClass=person)(calstateEduPersonEmplID=014532336))"
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=1 SRCH attr=memberOf
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=2 UNBIND
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=1 SEARCH RESULT
tag=101 err=0 qtime=0.000016 etime=0.192994 nentries=1 text=
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 fd=84 closed
but split up into a number of different lines which need to be
correlated to summarize it. Before I try it myself I was hoping somebody
else had already scratched that itch :). The only things I can find
searching are either really old or commercial products.
Thanks much…
11:57 a.m.
--On Friday, February 4, 2022 10:12 PM -0500 Dave Macias <davama(a)gmail.com>
wrote:
Is that the one I used to help maintain? I don't believe it's been updated
for 2.5 and later, unless it was forked and someone else has started
working on it.
--Quanah
6:14 p.m.
On Fri, Feb 04, 2022 at 10:12:40PM -0500, Dave Macias wrote:
Thanks for the pointer. There doesn't seem to be any way to download
just the script? You have to get their whole tool package? I don't
really want to add a repo just for this, I tried the manual download
option on their page for CentoOS 8:
https://www.ltb-project.org/download.html
https://www.ltb-project.org/archives/openldap-ltb-2.5.11-el8.tar.gz
But the second link just takes me to:
https://www.ltb-project.org/
Meh. I'm really looking for query times too, which it doesn't seem to
provide. I set up a test script which does a memberOf query every 5
minutes to alert me when they start taking 30+ seconds. Most of the time
they're subsecond, but every now and again the exact same query takes
5-20 seconds, just here and there. Odd. At least the occasional slow
query is better than continuous slow queries.
3:44 p.m.
Hello,
Le 06/02/2022 à 03:14, Paul B. Henson a écrit :
On Fri, Feb 04, 2022 at 10:12:40PM -0500, Dave Macias wrote:
> https://www.ltb-project.org/documentation/ldap-stats.html
Thanks for the pointer. There doesn't seem to be any way to download
just the script? You have to get their whole tool package? I don't
really want to add a repo just for this, I tried the manual download
option on their page for CentoOS 8:
https://www.ltb-project.org/download.html
https://www.ltb-project.org/archives/openldap-ltb-2.5.11-el8.tar.gz
But the second link just takes me to:
https://www.ltb-project.org/
For information, there was indeed a broken link on LTB website. It is
now fixed.
Anyway, the scripts are not provided in the global OpenLDAP package archive.
You can get them in the github repository here :
https://github.com/ltb-project/ldap-scripts
(and as mentionned by Quanah before, you can read the documentation
here: https://www.ltb-project.org/documentation/index.html)
Regards,
David
Meh. I'm really looking for query times too, which it doesn't seem to
provide. I set up a test script which does a memberOf query every 5
minutes to alert me when they start taking 30+ seconds. Most of the time
they're subsecond, but every now and again the exact same query takes
5-20 seconds, just here and there. Odd. At least the occasional slow
query is better than continuous slow queries.
8:55 a.m.
--On Monday, February 7, 2022 12:44 AM +0100 David Coutadeur
<david.coutadeur(a)gmail.com> wrote:
Hello,
Le 06/02/2022 à 03:14, Paul B. Henson a écrit :
> On Fri, Feb 04, 2022 at 10:12:40PM -0500, Dave Macias wrote:
>> https://www.ltb-project.org/documentation/ldap-stats.html
> Thanks for the pointer. There doesn't seem to be any way to download
> just the script? You have to get their whole tool package? I don't
> really want to add a repo just for this, I tried the manual download
> option on their page for CentoOS 8:
>
> https://www.ltb-project.org/download.html
>
> https://www.ltb-project.org/archives/openldap-ltb-2.5.11-el8.tar.gz
>
> But the second link just takes me to:
>
> https://www.ltb-project.org/
For information, there was indeed a broken link on LTB website. It is now
fixed.
Anyway, the scripts are not provided in the global OpenLDAP package
archive.
You can get them in the github repository here :
https://github.com/ltb-project/ldap-scripts
The official repository for it is at
<https://github.com/Matty9191/misc-shell-scripts>
Simply following the URL listed in the top comments of the script as to its
home takes you there, not sure why you were unable to find it previously.
I want to talk to Matt though about perhaps moving it into and the other
ldap specific scripts into their own separate repo for ease of pushing
updates. I have at least one queued.
Regards,
Quanah
1:54 a.m.
On 2/5/22 03:27, Paul B. Henson wrote:
Does anybody know of any good tools that can rip through an openldap
log
file and analyze it, creating a report of what queries are being made
and how long they are taking to process?
ldap-stats.pl tool mentioned by Dave, is indeed very useful for off-line
analysis.
You could also look into tools which extract metrics from logs and
provide them as a Prometheus-compatible exporter: mtail, promtail etc.
Personally I'm using mtail:
https://github.com/google/mtail
My mtail prog does not correlate request and response lines and does not
extract filters but will extract some metrics useful as indication to
search for problems.
Of course you can adapt it to your own needs:
https://code.stroeder.com/AE-DIR/ansible-ae-dir-server/src/branch/master/...
Note: The above is a Jinja2 template used by ansible, so you have to
replace the parts enclosed in double curly braces with your
site-specific values.
Ciao, Michael.
4:57 a.m.
openldap also has a monitor backend IIRC, have you looked into that?
You won't get query details, but other metrics that could be useful,
and without having to enable logging.
On Sat, Feb 5, 2022 at 6:54 AM Michael Ströder <michael(a)stroeder.com> wrote:
>
> On 2/5/22 03:27, Paul B. Henson wrote:
> > Does anybody know of any good tools that can rip through an openldap log
> > file and analyze it, creating a report of what queries are being made
> > and how long they are taking to process?
>
> ldap-stats.pl tool mentioned by Dave, is indeed very useful for off-line
> analysis.
>
> You could also look into tools which extract metrics from logs and
> provide them as a Prometheus-compatible exporter: mtail, promtail etc.
>
> Personally I'm using mtail:
>
> https://github.com/google/mtail
>
> My mtail prog does not correlate request and response lines and does not
> extract filters but will extract some metrics useful as indication to
> search for problems.
>
> Of course you can adapt it to your own needs:
>
>
https://code.stroeder.com/AE-DIR/ansible-ae-dir-server/src/branch/master/...
>
> Note: The above is a Jinja2 template used by ansible, so you have to
> replace the parts enclosed in double curly braces with your
> site-specific values.
>
> Ciao, Michael.
12:15 p.m.
On 2/5/22 13:57, Andreas Hasenack wrote:
openldap also has a monitor backend IIRC, have you looked into that?
Yes. ;-)
See also: https://code.stroeder.com/ldap/slapdcheck
You won't get query details, but other metrics that could be
useful,
and without having to enable logging.
But using mtail I can extract some more
metrics, e.g. count "deferred"
messages or result codes until ITS#9186 is implemented. ;-]
https://bugs.openldap.org/show_bug.cgi?id=9186
Ciao, Michael.
On Sat, Feb 5, 2022 at 6:54 AM Michael Ströder
<michael(a)stroeder.com> wrote:
>
> On 2/5/22 03:27, Paul B. Henson wrote:
>> Does anybody know of any good tools that can rip through an openldap log
>> file and analyze it, creating a report of what queries are being made
>> and how long they are taking to process?
>
> ldap-stats.pl tool mentioned by Dave, is indeed very useful for off-line
> analysis.
>
> You could also look into tools which extract metrics from logs and
> provide them as a Prometheus-compatible exporter: mtail, promtail etc.
>
> Personally I'm using mtail:
>
> https://github.com/google/mtail
>
> My mtail prog does not correlate request and response lines and does not
> extract filters but will extract some metrics useful as indication to
> search for problems.
>
> Of course you can adapt it to your own needs:
>
>
https://code.stroeder.com/AE-DIR/ansible-ae-dir-server/src/branch/master/...
>
> Note: The above is a Jinja2 template used by ansible, so you have to
> replace the parts enclosed in double curly braces with your
> site-specific values.
>
> Ciao, Michael.
Beste Grüße,
Michael Ströder
--
Michael Ströder (Dipl.-Inform.)
Klauprechtstr. 11
D-76137 Karlsruhe, Germany
Tel.: +49 721 8304316
Mobil: +49 170 2391920
E-Mail: michael(a)stroeder.com
Datenschutzhinweise:
https://stroeder.com/datenschutz.html
6:19 p.m.
On Sat, Feb 05, 2022 at 09:57:15AM -0300, Andreas Hasenack wrote:
openldap also has a monitor backend IIRC, have you looked into that?
Yes, historically we've used that with icinga and munin, although we're
looking to replace munin. That doesn't provide the per query timing
analysis I'm looking for to address a specific performance issue though.
Thanks...
4:26 a.m.
On 2/6/22 03:19, Paul B. Henson wrote:
On Sat, Feb 05, 2022 at 09:57:15AM -0300, Andreas Hasenack wrote:
> openldap also has a monitor backend IIRC, have you looked into that?
Yes, historically we've used that with icinga and munin, although we're
looking to replace munin. That doesn't provide the per query timing
analysis I'm looking for to address a specific performance issue though.
My mtail prog also extracts etime from SEARCH RESULT lines and sorts the
values into a histogram.
Not sure how to get that into Icinga or Munin though. mtail serves JSON,
Prometheus and graphite format.
Many of my customers are using Prometheus for scraping metrics together
with alertmanager to submit alarms based on these metrics.
Ciao, Michael.
1:41 a.m.
New subject: Antw: [EXT] Re: log analysis tools
>> "Paul B. Henson" <henson(a)acm.org> schrieb am
06.02.2022 um 03:19 in
Nachricht
<Yf8wTf5AtGrDOBl7(a)zaphod.pbhware.com>:
On Sat, Feb 05, 2022 at 09:57:15AM ‑0300, Andreas Hasenack wrote:
> openldap also has a monitor backend IIRC, have you looked into that?
Yes, historically we've used that with icinga and munin, although we're
looking to replace munin. That doesn't provide the per query timing
analysis I'm looking for to address a specific performance issue though.
Anyway: Any plans for such (enhanced LDAP monitoring)? Or something like an
"explain" (that SQL databases do to explain how a query will be processes and
what the estimated costs are)?
Regards,
Ulrich
Thanks...
11:55 p.m.
On Sat, 05 Feb 2022 11:27:19 +0900,
Paul B. Henson wrote:
Does anybody know of any good tools that can rip through an openldap
log file and analyze it, creating a report of what queries are being made and how long
they are taking to process?
https://github.com/fumiyas/ldap-hack/blob/master/openldap/bin/slapdstatsl...
converts raw OpenLDAP stats log to JSONL (JSON lines), but
currently does NOT support OpenLDAP 2.5+ stats log.
Just FYI.
All of the information I'm interested in is included in the log:
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 fd=84 ACCEPT from IP=134.71.247.28:46592
(IP=0.0.0.0:636)
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 fd=84 TLS established tls_ssf=256 ssf=256
tls_proto=TLSv1.2 tls_cipher=ECDHE-RSA-AES256-GCM-SHA384
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=0 BIND
dn="cn=it_boomi,ou=user,ou=service,dc=cpp,dc=edu" method=128
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=0 BIND
dn="cn=it_boomi,ou=user,ou=service,dc=cpp,dc=edu" mech=SIMPLE bind_ssf=0
ssf=256
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=0 RESULT tag=97 err=0 qtime=0.000031
etime=0.000189 text=
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=1 SRCH
base="ou=user,dc=cpp,dc=edu" scope=2 deref=3
filter="(&(objectClass=person)(calstateEduPersonEmplID=014532336))"
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=1 SRCH attr=memberOf
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=2 UNBIND
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 op=1 SEARCH RESULT tag=101 err=0
qtime=0.000016 etime=0.192994 nentries=1 text=
Feb 4 18:23:54 ldap-01 slapd[1207]: conn=46272 fd=84 closed
but split up into a number of different lines which need to be correlated to summarize
it. Before I try it myself I was hoping somebody else had already scratched that itch :).
The only things I can find searching are either really old or commercial products.
--
-- Name: SATOH Fumiyasu @ OSSTech Corp., Japan (fumiyas @ osstech co jp)
-- Business Home: https://www.OSSTech.co.jp/
-- GitHub Home: https://GitHub.com/fumiyas/
-- PGP Fingerprint: BBE1 A1C9 525A 292E 6729 CDEC ADC2 9DCA 5E1C CBCA
594
days inactive
596
days old
openldap-technical@openldap.org
14 comments
8 participants
participants (8)
-
Andreas Hasenack -
Dave Macias -
David Coutadeur -
Michael Ströder -
Paul B. Henson -
Quanah Gibson-Mount -
SATOH Fumiyasu -
Ulrich Windl