I used to have some help with what and how I should add indexes, but it looks like the new mdb backend is not telling me anymore. Is this correct? Anyway to get a warning when some index needs to be set?
bdb_equality_candidates: (xxxx) not indexed bdb_inequality_candidates: (createTimestamp) not indexed etc
https://mishikal.wordpress.com/2013/05/16/openldap-a-comparison-of-back-mdb-...
On 8/17/19 12:29 PM, Marc Roos wrote:
I used to have some help with what and how I should add indexes, but it looks like the new mdb backend is not telling me anymore. Is this correct? Anyway to get a warning when some index needs to be set?
bdb_equality_candidates: (xxxx) not indexed bdb_inequality_candidates: (createTimestamp) not indexed etc
This message is not specific to a particular backend.
But anyway this message is pretty useless without closely looking at the search filters actually used:
https://www.openldap.org/its/index.cgi?findid=7796
Which OpenLDAP builds are you using on which platform?
Ciao, Michael.
I am using CentOS6/CentOS7 and currently testing with CentOS7 default openldap 2.4.44, but switched their default db to mdb. I am using eg sendmail as a client. So I have no clue what the search queries are. I can of course copy the indexes from an 'old' environment. But if I am going to allow some other task to query the ldap in the future, I will again not know if they are accessing keys that are maybe not indexed or not properly indexed.
-----Original Message----- From: Michael Ströder [mailto:michael@stroeder.com] Sent: zaterdag 17 augustus 2019 12:56 To: Marc Roos; openldap-technical Subject: Re: mdb index reporting available?
On 8/17/19 12:29 PM, Marc Roos wrote:
I used to have some help with what and how I should add indexes, but it looks like the new mdb backend is not telling me anymore. Is this correct? Anyway to get a warning when some index needs to be set?
bdb_equality_candidates: (xxxx) not indexed bdb_inequality_candidates: (createTimestamp) not indexed etc
This message is not specific to a particular backend.
But anyway this message is pretty useless without closely looking at the search filters actually used:
https://www.openldap.org/its/index.cgi?findid=7796
Which OpenLDAP builds are you using on which platform?
Ciao, Michael.
On 8/17/19 1:04 PM, Marc Roos wrote:
I am using CentOS6/CentOS7 and currently testing with CentOS7 default openldap 2.4.44, but switched their default db to mdb.
AFAIK they don't use my back-port patch for ITS#7796.
I am using eg sendmail as a client. So I have no clue what the search queries are.
Look at your logs to see the LDAP filter strings (loglevel stats).
I can of course copy the indexes from an 'old' environment.
What do you mean by "copy"? The indexing is configured with slapd.conf (aka static config) or cn=config (back-config).
But if I am going to allow some other task to query the ldap in the future, I will again not know if they are accessing keys that are maybe not indexed or not properly indexed.
Again: Simply look at the logs.
Ciao, Michael.
Hi Michael
I am using CentOS6/CentOS7 and currently testing with CentOS7
default
openldap 2.4.44, but switched their default db to mdb.
AFAIK they don't use my back-port patch for ITS#7796.
I am using eg sendmail as a client. So I have no clue what the
search
queries are.
Look at your logs to see the LDAP filter strings (loglevel stats).
Was always seeing these with ber sync
I can of course copy the indexes from an 'old' environment.
What do you mean by "copy"? The indexing is configured with slapd.conf (aka static config) or cn=config (back-config).
Over time I am adding indexes until there are no such messages (except for some incidental queries). And since I have slapd's for specific groups I can easily determine the index settings by querying their oldDbIndex and add those to the new setup
But if I am going to allow some other task to query the ldap in the future, I will again not know if they are accessing keys that are
maybe
not indexed or not properly indexed.
Again: Simply look at the logs.
Yes will enable this stats
On 8/17/19 2:52 PM, Marc Roos wrote:
Over time I am adding indexes until there are no such messages (except for some incidental queries).
And that's exactly the wrong thing to do! That's why I requested to disable those messages in ITS#7796:
https://www.openldap.org/its/index.cgi?findid=7796
By adding random indexes just to get rid of not-indexed warnings you can dramatically lower your search performance. You should only add an index if you analyzed that clients send search requests with filters which can make good use of the index.
Learn about search candidate sets: https://unix.stackexchange.com/a/457561
LDAP filters used by clients can be seen with loglevel stats.
Ciao, Michael.
Michael Ströder michael@stroeder.com schrieb am 17.08.2019 um 20:37 in
Nachricht 123c2421-b539-14cb-6c82-44235675b5cd@stroeder.com:
On 8/17/19 2:52 PM, Marc Roos wrote:
Over time I am adding indexes until there are no such messages (except for some incidental queries).
And that's exactly the wrong thing to do! That's why I requested to disable those messages in ITS#7796:
https://www.openldap.org/its/index.cgi?findid=7796
By adding random indexes just to get rid of not-indexed warnings you can dramatically lower your search performance. You should only add an index if you analyzed that clients send search requests with filters which can make good use of the index.
I agree that some thinking before adding an index isn't a bad idea, but still I think these messages are important as they can make you start thinking.
Learn about search candidate sets: https://unix.stackexchange.com/a/457561
LDAP filters used by clients can be seen with loglevel stats.
Ciao, Michael.
On 8/19/19 9:24 AM, Ulrich Windl wrote:
Michael Ströder michael@stroeder.com schrieb am 17.08.2019 um 20:37 in
Nachricht 123c2421-b539-14cb-6c82-44235675b5cd@stroeder.com:
On 8/17/19 2:52 PM, Marc Roos wrote:
Over time I am adding indexes until there are no such messages (except for some incidental queries).
And that's exactly the wrong thing to do! That's why I requested to disable those messages in ITS#7796:
https://www.openldap.org/its/index.cgi?findid=7796
By adding random indexes just to get rid of not-indexed warnings you can dramatically lower your search performance. You should only add an index if you analyzed that clients send search requests with filters which can make good use of the index.
I agree that some thinking before adding an index isn't a bad idea, but still I think these messages are important as they can make you start thinking.
Experience shows that people do not analyse the LDAP filters but just add an index to get rid of the message. You can find that many times in various discussion / Q&A forums.
It's also bad that this message is written at any loglevel. Depending on your config it can be written many times for the same attribute during processing a single search operation, e.g. when processing the numerous set-based ACLs in my Æ-DIR. 50+ senseless log lines for a single search operation with deref control is not fun.
Ciao, Michael.
Michael Ströder michael@stroeder.com schrieb am 19.08.2019 um 10:44 in
Nachricht 27f4384a-5d12-c6b3-3e1d-f85400e2a84a@stroeder.com:
On 8/19/19 9:24 AM, Ulrich Windl wrote:
Michael Ströder michael@stroeder.com schrieb am 17.08.2019 um 20:37
in
Nachricht 123c2421-b539-14cb-6c82-44235675b5cd@stroeder.com:
On 8/17/19 2:52 PM, Marc Roos wrote:
Over time I am adding indexes until there are no such messages (except for some incidental queries).
And that's exactly the wrong thing to do! That's why I requested to disable those messages in ITS#7796:
https://www.openldap.org/its/index.cgi?findid=7796
By adding random indexes just to get rid of not-indexed warnings you can dramatically lower your search performance. You should only add an index if you analyzed that clients send search requests with filters which can make good use of the index.
I agree that some thinking before adding an index isn't a bad idea, but
still
I think these messages are important as they can make you start thinking.
Experience shows that people do not analyse the LDAP filters but just add an index to get rid of the message. You can find that many times in various discussion / Q&A forums.
It's also bad that this message is written at any loglevel. Depending on your config it can be written many times for the same attribute during processing a single search operation, e.g. when processing the numerous set-based ACLs in my Æ-DIR. 50+ senseless log lines for a single search operation with deref control is not fun.
I could imagine that either...
* writing each message only once since slapd started * suppressing each message for a configurable time interval (like 1 hour, 1 day, 1 week, etc.)
... would be a solution people could live with.
As for the case: Can you present an example here where an index added as suggested makes performance actually worse?
Ciao, Michael.
On 8/19/19 11:22 AM, Ulrich Windl wrote:
Can you present an example here where an index added as suggested makes performance actually worse?
I sent this link before:
https://unix.stackexchange.com/questions/451118/openldap-bdb-equality-candid...
As you can see the guy asking did not accept my answer and his own "solution" was to just add the index without further thinking.
Ciao, Michael.
Is there maybe someone here that has tested the index use for sendmail, taking into account the issue Michael addresses? :)
-----Original Message----- Subject: Re: mdb index reporting available?
On 8/19/19 11:22 AM, Ulrich Windl wrote:
Can you present an example here where an index added as suggested makes performance actually worse?
I sent this link before:
https://unix.stackexchange.com/questions/451118/openldap-bdb-equality-candid...
As you can see the guy asking did not accept my answer and his own "solution" was to just add the index without further thinking.
Ciao, Michael.
On 8/19/19 4:28 PM, Marc Roos wrote:
Is there maybe someone here that has tested the index use for sendmail, taking into account the issue Michael addresses? :)
Given the fact that you can use different e-mail related schema it's not possible to give general advice without knowing the config details and queries sent by sendmail.
Again: Look at the filters in your log.
It's your own homework and actually not that hard to do.
Ciao, Michael.
--On Monday, August 19, 2019 12:22 PM +0200 Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de wrote:
I could imagine that either...
- writing each message only once since slapd started
slapd has no clue what search filters a client uses, therefore slapd has no knowledge of what indices may or may not be necessary without a client providing queries. Thus this is impossible for slapd to do.
- suppressing each message for a configurable time interval (like 1 hour,
1 day, 1 week, etc.)
So you would add a timer to slapd, which would require additional resource overhead? What happens if slapd is restarted during that time period? etc.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Quanah Gibson-Mount quanah@symas.com schrieb am 19.08.2019 um 17:25 in
Nachricht <958015EBB469A710414E5F48@[192.168.1.144]>:
‑‑On Monday, August 19, 2019 12:22 PM +0200 Ulrich Windl <Ulrich.Windl@rz.uni‑regensburg.de> wrote:
I could imagine that either...
- writing each message only once since slapd started
slapd has no clue what search filters a client uses, therefore slapd has no
knowledge of what indices may or may not be necessary without a client providing queries. Thus this is impossible for slapd to do.
If slapd can create such a message now, you'd only have to set a flag in memory that remembers the message had been written already. So that would be one bit per index per database I guess.
- suppressing each message for a configurable time interval (like 1 hour,
1 day, 1 week, etc.)
So you would add a timer to slapd, which would require additional resource overhead? What happens if slapd is restarted during that time period? etc.
All in memory, of course: After a restart, the message would be logged again despite of the interval configured. As a restart will log messages anyway, I don't see a problem there.
‑‑Quanah
‑‑
Quanah Gibson‑Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
I do have to say I like this message at ber sync more. Enabling the stats is not an option to keep in production on.
Nevermind, I see the statement now with the ber sync logging, after removing an index and then querying it
-----Original Message-----
Subject: RE: mdb index reporting available?
I do have to say I like this message at ber sync more. Enabling the stats is not an option to keep in production on.
--On Saturday, August 17, 2019 1:29 PM +0200 Marc Roos M.Roos@f1-outsourcing.eu wrote:
I used to have some help with what and how I should add indexes, but it looks like the new mdb backend is not telling me anymore. Is this correct? Anyway to get a warning when some index needs to be set?
back-mdb logs whether or not indices are set the same way as back-bdb and back-hdb, at any log level. You can trivially see this in the code:
if ( rc == LDAP_INAPPROPRIATE_MATCHING ) { Debug( LDAP_DEBUG_ANY, "<= mdb_equality_candidates: (%s) not indexed\n", ava->aa_desc->ad_cname.bv_val, 0, 0 ); return 0; }
Note the "LDAP_DEBUG_ANY". It is always important to understand that there are situations in which indexing can negatively impact performance rather than improve it. Thus why it is important to know what searches are being performed that are generating those messages. This information can be obtained by using the "stats" log level. Generally you will also find that without the "stats" level of information, diagnosing problems that occur will be next to impossible, regardless of backend. If replication is involved, "stats sync" logging will generally be required, again regardless of what backend is in use.
Furthermore, regardless of backend used, I'd advise using a more current release than 2.4.44 which was released 3.5 years ago. The current release is 2.4.48 from July 2019. You should likely read the change log for OpenLDAP found at https://www.openldap.org/software/release/changes.html. You generally have 3 options in this:
a) You can use the builds from the LTB project found at https://ltb-project.org/documentation/openldap-rpm#yum_repository. These install into /usr/local, so are isolated from the system libraries, etc. Fixes released after 2.4.48 will not be included in these builds.
b) You can use the drop-in replacement builds from Symas: https://repo.symas.com/sofl/rhel7/. Any critical fixes post 2.4.48 will be added to these builds.
c) You can build and maintain your own OpenLDAP packages.
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
openldap-technical@openldap.org
-
Marc Roos -
Michael Ströder -
Quanah Gibson-Mount -
Ulrich Windl