3:29 a.m.
I used to have some help with what and how I should add indexes, but it
looks like the new mdb backend is not telling me anymore. Is this
correct? Anyway to get a warning when some index needs to be set?
bdb_equality_candidates: (xxxx) not indexed
bdb_inequality_candidates: (createTimestamp) not indexed
etc
https://mishikal.wordpress.com/2013/05/16/openldap-a-comparison-of-back-m...
3:56 a.m.
On 8/17/19 12:29 PM, Marc Roos wrote:
I used to have some help with what and how I should add indexes, but
it
looks like the new mdb backend is not telling me anymore. Is this
correct? Anyway to get a warning when some index needs to be set?
bdb_equality_candidates: (xxxx) not indexed
bdb_inequality_candidates: (createTimestamp) not indexed
etc
This message is not specific to a particular backend.
But anyway this message is pretty useless without closely looking at the
search filters actually used:
https://www.openldap.org/its/index.cgi?findid=7796
Which OpenLDAP builds are you using on which platform?
Ciao, Michael.
4:04 a.m.
I am using CentOS6/CentOS7 and currently testing with CentOS7 default
openldap 2.4.44, but switched their default db to mdb.
I am using eg sendmail as a client. So I have no clue what the search
queries are. I can of course copy the indexes from an 'old' environment.
But if I am going to allow some other task to query the ldap in the
future, I will again not know if they are accessing keys that are maybe
not indexed or not properly indexed.
-----Original Message-----
From: Michael Ströder [mailto:michael@stroeder.com]
Sent: zaterdag 17 augustus 2019 12:56
To: Marc Roos; openldap-technical
Subject: Re: mdb index reporting available?
On 8/17/19 12:29 PM, Marc Roos wrote:
I used to have some help with what and how I should add indexes, but
it looks like the new mdb backend is not telling me anymore. Is this
correct? Anyway to get a warning when some index needs to be set?
bdb_equality_candidates: (xxxx) not indexed
bdb_inequality_candidates: (createTimestamp) not indexed etc
This message is not specific to a particular backend.
But anyway this message is pretty useless without closely looking at the
search filters actually used:
https://www.openldap.org/its/index.cgi?findid=7796
Which OpenLDAP builds are you using on which platform?
Ciao, Michael.
4:29 a.m.
On 8/17/19 1:04 PM, Marc Roos wrote:
I am using CentOS6/CentOS7 and currently testing with CentOS7 default
openldap 2.4.44, but switched their default db to mdb.
AFAIK they don't use my back-port patch for ITS#7796.
I am using eg sendmail as a client. So I have no clue what the search
queries are.
Look at your logs to see the LDAP filter strings (loglevel stats).
I can of course copy the indexes from an 'old' environment.
What do you mean by "copy"? The indexing is configured with slapd.conf
(aka static config) or cn=config (back-config).
But if I am going to allow some other task to query the ldap in the
future, I will again not know if they are accessing keys that are maybe
not indexed or not properly indexed.
Again: Simply look at the logs.
Ciao, Michael.
5:52 a.m.
Hi Michael
> I am using CentOS6/CentOS7 and currently testing with CentOS7
default
> openldap 2.4.44, but switched their default db to mdb.
AFAIK they don't use my back-port patch for ITS#7796.
> I am using eg sendmail as a client. So I have no clue what the
search
> queries are.
Look at your logs to see the LDAP filter strings (loglevel stats).
Was always seeing these with ber sync
> I can of course copy the indexes from an 'old'
environment.
What do you mean by "copy"? The indexing is configured with slapd.conf
(aka static config) or cn=config (back-config).
Over time I am adding indexes until there are no such messages (except
for
some incidental queries). And since I have slapd's for specific groups
I can easily determine the index settings by querying their oldDbIndex
and add those to the new setup
> But if I am going to allow some other task to query the ldap in
the
> future, I will again not know if they are accessing keys that are
maybe
> not indexed or not properly indexed.
Again: Simply look at the logs.
Yes will enable this stats
11:37 a.m.
On 8/17/19 2:52 PM, Marc Roos wrote:
Over time I am adding indexes until there are no such messages
(except for some incidental queries).
And that's exactly the wrong thing to
do!
That's why I requested to disable those messages in ITS#7796:
https://www.openldap.org/its/index.cgi?findid=7796
By adding random indexes just to get rid of not-indexed warnings you can
dramatically lower your search performance. You should only add an index
if you analyzed that clients send search requests with filters which can
make good use of the index.
Learn about search candidate sets:
https://unix.stackexchange.com/a/457561
LDAP filters used by clients can be seen with loglevel stats.
Ciao, Michael.
12:24 a.m.
New subject: Antw: Re: mdb index reporting available?
>> Michael Ströder <michael(a)stroeder.com> schrieb am
17.08.2019 um 20:37 in
Nachricht
<123c2421-b539-14cb-6c82-44235675b5cd(a)stroeder.com>:
On 8/17/19 2:52 PM, Marc Roos wrote:
> Over time I am adding indexes until there are no such messages
> (except for some incidental queries).
And that's exactly the wrong thing to do!
That's why I requested to disable those messages in ITS#7796:
https://www.openldap.org/its/index.cgi?findid=7796
By adding random indexes just to get rid of not-indexed warnings you can
dramatically lower your search performance. You should only add an index
if you analyzed that clients send search requests with filters which can
make good use of the index.
I agree that some thinking before adding an index isn't a bad idea, but still
I think these messages are important as they can make you start thinking.
Learn about search candidate sets:
https://unix.stackexchange.com/a/457561
LDAP filters used by clients can be seen with loglevel stats.
Ciao, Michael.
1:44 a.m.
On 8/19/19 9:24 AM, Ulrich Windl wrote:
>>> Michael Ströder <michael(a)stroeder.com> schrieb am
17.08.2019 um 20:37 in
Nachricht <123c2421-b539-14cb-6c82-44235675b5cd(a)stroeder.com>:
> On 8/17/19 2:52 PM, Marc Roos wrote:
>> Over time I am adding indexes until there are no such messages
>> (except for some incidental queries).
> And that's exactly the wrong thing to do!
> That's why I requested to disable those messages in ITS#7796:
>
> https://www.openldap.org/its/index.cgi?findid=7796
>
> By adding random indexes just to get rid of not-indexed warnings you can
> dramatically lower your search performance. You should only add an index
> if you analyzed that clients send search requests with filters which can
> make good use of the index.
I agree that some thinking before adding an index isn't a bad idea, but still
I think these messages are important as they can make you start thinking.
Experience shows that people do not analyse the LDAP filters but just
add an index to get rid of the message. You can find that many times in
various discussion / Q&A forums.
It's also bad that this message is written at any loglevel. Depending on
your config it can be written many times for the same attribute during
processing a single search operation, e.g. when processing the numerous
set-based ACLs in my Æ-DIR. 50+ senseless log lines for a single search
operation with deref control is not fun.
Ciao, Michael.
2:22 a.m.
New subject: Antw: Re: mdb index reporting available?
>> Michael Ströder <michael(a)stroeder.com> schrieb am
19.08.2019 um 10:44 in
Nachricht
<27f4384a-5d12-c6b3-3e1d-f85400e2a84a(a)stroeder.com>:
On 8/19/19 9:24 AM, Ulrich Windl wrote:
>>>> Michael Ströder <michael(a)stroeder.com> schrieb am 17.08.2019 um
20:37
in
> Nachricht
<123c2421-b539-14cb-6c82-44235675b5cd(a)stroeder.com>:
>> On 8/17/19 2:52 PM, Marc Roos wrote:
>>> Over time I am adding indexes until there are no such messages
>>> (except for some incidental queries).
>> And that's exactly the wrong thing to do!
>> That's why I requested to disable those messages in ITS#7796:
>>
>> https://www.openldap.org/its/index.cgi?findid=7796
>>
>> By adding random indexes just to get rid of not-indexed warnings you can
>> dramatically lower your search performance. You should only add an index
>> if you analyzed that clients send search requests with filters which can
>> make good use of the index.
>
> I agree that some thinking before adding an index isn't a bad idea, but
still
> I think these messages are important as they can make you start thinking.
Experience shows that people do not analyse the LDAP filters but just
add an index to get rid of the message. You can find that many times in
various discussion / Q&A forums.
It's also bad that this message is written at any loglevel. Depending on
your config it can be written many times for the same attribute during
processing a single search operation, e.g. when processing the numerous
set-based ACLs in my Æ-DIR. 50+ senseless log lines for a single search
operation with deref control is not fun.
I could imagine that either...
* writing each message only once since slapd started
* suppressing each message for a configurable time interval (like 1 hour, 1
day, 1 week, etc.)
... would be a solution people could live with.
As for the case: Can you present an example here where an index added as
suggested makes performance actually worse?
Ciao, Michael.
2:36 a.m.
On 8/19/19 11:22 AM, Ulrich Windl wrote:
Can you present an example here where an index added as
suggested makes performance actually worse?
I sent this link before:
https://unix.stackexchange.com/questions/451118/openldap-bdb-equality-can...
As you can see the guy asking did not accept my answer and his own
"solution" was to just add the index without further thinking.
Ciao, Michael.
7:28 a.m.
New subject: Anyone having tested indexes for the use sendmail
Is there maybe someone here that has tested the index use for sendmail,
taking into account the issue Michael addresses? :)
-----Original Message-----
Subject: Re: mdb index reporting available?
On 8/19/19 11:22 AM, Ulrich Windl wrote:
Can you present an example here where an index added as suggested
makes performance actually worse?
I sent this link before:
https://unix.stackexchange.com/questions/451118/openldap-bdb-equality-can...
As you can see the guy asking did not accept my answer and his own
"solution" was to just add the index without further thinking.
Ciao, Michael.
8:28 a.m.
New subject: Anyone having tested indexes for the use sendmail
On 8/19/19 4:28 PM, Marc Roos wrote:
Is there maybe someone here that has tested the index use for
sendmail,
taking into account the issue Michael addresses? :)
Given the fact that you can use different e-mail related schema it's not
possible to give general advice without knowing the config details and
queries sent by sendmail.
Again: Look at the filters in your log.
It's your own homework and actually not that hard to do.
Ciao, Michael.
8:25 a.m.
New subject: Antw: Re: mdb index reporting available?
--On Monday, August 19, 2019 12:22 PM +0200 Ulrich Windl
<Ulrich.Windl(a)rz.uni-regensburg.de> wrote:
I could imagine that either...
* writing each message only once since slapd started
slapd has no clue what search filters a client uses, therefore slapd has no
knowledge of what indices may or may not be necessary without a client
providing queries. Thus this is impossible for slapd to do.
* suppressing each message for a configurable time interval (like 1
hour,
1 day, 1 week, etc.)
So you would add a timer to slapd, which would require additional resource
overhead? What happens if slapd is restarted during that time period? etc.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
11:31 p.m.
New subject: Antw: Re: mdb index reporting available?
>> Quanah Gibson-Mount <quanah(a)symas.com> schrieb am
19.08.2019 um 17:25 in
Nachricht <958015EBB469A710414E5F48(a)[192.168.1.144]>:
‑‑On Monday, August 19, 2019 12:22 PM +0200 Ulrich Windl
<Ulrich.Windl(a)rz.xn--uniregensburg-dm6g.de> wrote:
> I could imagine that either...
>
> * writing each message only once since slapd started
slapd has no clue what search filters a client uses, therefore slapd has no
knowledge of what indices may or may not be necessary without a
client
providing queries. Thus this is impossible for slapd to do.
If slapd can create such a message now, you'd only have to set a flag in
memory that remembers the message had been written already. So that would be
one bit per index per database I guess.
> * suppressing each message for a configurable time interval (like 1 hour,
> 1 day, 1 week, etc.)
So you would add a timer to slapd, which would require additional resource
overhead? What happens if slapd is restarted during that time period? etc.
All in memory, of course: After a restart, the message would be logged again
despite of the interval configured. As a restart will log messages anyway, I
don't see a problem there.
‑‑Quanah
‑‑
Quanah Gibson‑Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5:55 a.m.
I do have to say I like this message at ber sync more. Enabling the
stats is not an option to keep in production on.
6:13 a.m.
Nevermind, I see the statement now with the ber sync logging, after
removing an index and then querying it
-----Original Message-----
Subject: RE: mdb index reporting available?
I do have to say I like this message at ber sync more. Enabling the
stats is not an option to keep in production on.
9:18 a.m.
--On Saturday, August 17, 2019 1:29 PM +0200 Marc Roos
<M.Roos(a)f1-outsourcing.eu> wrote:
I used to have some help with what and how I should add indexes, but it
looks like the new mdb backend is not telling me anymore. Is this
correct? Anyway to get a warning when some index needs to be set?
back-mdb logs whether or not indices are set the same way as back-bdb and
back-hdb, at any log level. You can trivially see this in the code:
if ( rc == LDAP_INAPPROPRIATE_MATCHING ) {
Debug( LDAP_DEBUG_ANY,
"<= mdb_equality_candidates: (%s) not indexed\n",
ava->aa_desc->ad_cname.bv_val, 0, 0 );
return 0;
}
Note the "LDAP_DEBUG_ANY". It is always important to understand that there
are situations in which indexing can negatively impact performance rather
than improve it. Thus why it is important to know what searches are being
performed that are generating those messages. This information can be
obtained by using the "stats" log level. Generally you will also find that
without the "stats" level of information, diagnosing problems that occur
will be next to impossible, regardless of backend. If replication is
involved, "stats sync" logging will generally be required, again regardless
of what backend is in use.
Furthermore, regardless of backend used, I'd advise using a more current
release than 2.4.44 which was released 3.5 years ago. The current release
is 2.4.48 from July 2019. You should likely read the change log for
OpenLDAP found at <https://www.openldap.org/software/release/changes.html>;.
You generally have 3 options in this:
a) You can use the builds from the LTB project found at
<https://ltb-project.org/documentation/openldap-rpm#yum_repository>;. These
install into /usr/local, so are isolated from the system libraries, etc.
Fixes released after 2.4.48 will not be included in these builds.
b) You can use the drop-in replacement builds from Symas:
<https://repo.symas.com/sofl/rhel7/>;. Any critical fixes post 2.4.48 will
be added to these builds.
c) You can build and maintain your own OpenLDAP packages.
Regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
1496
days inactive
1499
days old
openldap-technical@openldap.org
16 comments
4 participants
participants (4)
-
Marc Roos -
Michael Ströder -
Quanah Gibson-Mount -
Ulrich Windl