On Thu, Nov 09, 2006 at 03:34:38PM +0100, Sylvain Amrani wrote:

Hi list,

I've got an attribute that could contains very long strings (more than 150 chr). It's a string made of small tokens separated by spaces and slashes :

departmentUID: BA/BAC ANDL/BAPZ IDF/GRPT YVLN/CIE GN ST GERM...

The attibute is indexed with pres,eq,sub

When I search for small substrings like (departmentUID=*/GRPT*) it's fast and ok.

When I search for a long subset of the string, the query is _very_ long (many seconds) : (departmentUID=BA/BAC ANDL/BAPZ IDF/GRPT YVLN/CIE GN ST*)

The substring index is not generated for long strings. Check these in slapd.conf(5): index_substr_if_minlen <integer> index_substr_if_maxlen <integer> index_substr_any_len <integer> index_substr_any_step <integer>

In the end, you may be better off spliting that big string in multivalued attributes perhaps or somethine else.

Using subinitial or subany in the index slap.conf parameter did not solve the problem.

I found only one post in the archives related to a server were any query string with more than 3 characters makes slow answers. He was told to adjust :

index_substr_if_minlen index_substr_if_maxlen index_substr_any_len index_substr_any_step

But theses parameters are only available with openldap 2.3 and it doesn't seem to be exactly my problem (I've no differences between 2,3,4 or 5 characters query strings)

Defaults may have changed. In earlier versions, this was a compile time definition.

Sylvain Amrani wrote:

Hi list,

I've got an attribute that could contains very long strings (more than 150 chr). It's a string made of small tokens separated by spaces and slashes :

departmentUID: BA/BAC ANDL/BAPZ IDF/GRPT YVLN/CIE GN ST GERM...

The attibute is indexed with pres,eq,sub

When I search for small substrings like (departmentUID=*/GRPT*) it's fast and ok.

When I search for a long subset of the string, the query is _very_ long (many seconds) : (departmentUID=BA/BAC ANDL/BAPZ IDF/GRPT YVLN/CIE GN ST*)

The more the string is long, the more the answer is slow.

I can use very complex filters (&((|()())(|()()))... and the answer is very fast, unless I use a long string in it.

The backend is BDB (berkeley 4.2) Openldap is 2.2.26

Is this a BDB or an Openldap related issue ? Using subinitial or subany in the index slap.conf parameter did not solve the problem.

That's a feature of OpenLDAP in 2.2 and older; subinitial and subfinal queries only index up to 4 characters of the string you provide. The rest are ignored. So indexing is not narrowing down the search candidates very much, and the full filter has to get evaluated on a lot of extraneous entries.

You should upgrade to the current 2.3, which uses the entire input string on index lookups.

I found only one post in the archives related to a server were any query string with more than 3 characters makes slow answers. He was told to adjust :

index_substr_if_minlen index_substr_if_maxlen index_substr_any_len index_substr_any_step

But theses parameters are only available with openldap 2.3 and it doesn't seem to be exactly my problem (I've no differences between 2,3,4 or 5 characters query strings)

Thanks, Sylvain.