Index seems to return wrong amount of candidate causing really poor search performance
by chrichardso27＠gmail.com
Hi,
Considering the following assumptions;
- OpenLDAP version 2.4.51
- attributes objectClass and abc are indexed based on equality
- the EQUALITY of attribute abc is based on distinguishedNameMatch
- The database contains roughly 2 million entries
- 2 entries have defined the attribute abc with a dn value cn=foo,dc=bar and objectClass=someClass
- 2 entries have defined the attribute abc with a dn value cn=bar,dc=baz and objectClass=someClass
Now, the issue started with really slow search performance using objectClass=someClass & abc=cn=foo,dc=bar as filter criteria. Debugging a while seems to indicate that the objectClass filter returns roughly 2 million entries as candidates. Now, one would expect that the second filter would return only the 2 potential candidates from the abc index, or a subset of the whole database but this is not the case. The second filter also returns nearly the whole database entries as potential candidates and causes really slow query performance. Interestingly, this only occurs when attribute abc has value cn=foo,dc=bar, but for some reason for the entry having attribute abc with value cn=bar,dc=baz the query returns immediately. In both cases, the actual entries matching the search return immediately but for the problematic search "(&(objectClass=someClass)(abc=cn=foo,dc=bar))", the completion of the search takes a long time (around 15 seconds to be precise).
The issue started suddenly and wasn't a degradation of query performance over time.
Few things I have tried
- Rebuilt the whole database again
- Reindex the existing database again
- Testing with bdb and mdb as backends
- Increased cache sizes for bdb to hold the whole database in cache
- For bdb adjust the page size of the indexes according to suggestion by db_tuner
- Change the order of the filters
None of these made any difference. At the moment, there does not seem to be any good options to try. Any ideas or help would be greatly appreciated!
1 year, 7 months
Use-case specific changes to LMDB
by martin＠urbackup.org
This post outlines a few changes to LMDB I had to do to make it work in a specific use case. I’d like to see those changes upstream, but I understand that they may be/are not relevant for e.g. OpenLDAP.
The use case is multiple databases on disks with long running large write transactions.
1. Option to not use custom memory allocator/page pool
LMDB has a custom malloc() implementation that re-uses pages (me_dpages). I understand that this improves the performance at bit (depending on the malloc implementation). But there should at least be the option to not do that (for many reasons). I would even make not using it the default.
2. Large transactions and spilling
In a large write transaction, it will use a lot of memory per default (512MiB) which won’t get freed when the transaction commits (see 1.). If one has a lot of databases it uses a lot of memory that never gets freed.
Alternatively, one can use MDB_WRITEMAP, but (i) per default Linux isn’t tuned to delay writing pages to disk and (ii) before commit LMDB has to remove a dirty bit, so each page is written twice.
Both problems would be fixed by making when pages get spilled configurable (mt_dirty_room as MDB_IDL_UM_MAX currently) and reducing the default non-spill memory amount for at least the MDB_WRITEMAP case. If this memory amount is low mt_spill_pgs gets sorted often so maybe this needs to be converted to a different data structure (e.g. red-black tree).
3. LMDB causes crashes if database is corrupted
If the database is corrupted it can cause the application to crash. I have fixed those cases when they (randomly) occurred. Properly fixing this would probably be best done with some fuzzing.
4. Allow LMDB to reside on a device
I used dm-cache to improve LMDB read performance. It needed a bit of adjustment to get the correct size of the device via ioctl BLKGETSIZE64.
--
I’ve fixed those issues w.r.t. my application. If there is interest in any of those application specific changes, I’ll clean them up and post them.
2 years, 2 months
How to clear a certificate from openldap cache - Client Side
by divyachauhan01.dc＠gmail.com
Divyanshi Chauhan <divyachauhan01.dc(a)gmail.com>
1:33 PM (3 hours ago)
to openldap-technical-owner, openldap-devel-owner, openldap-bugs-owner
Hello,
I have an ldaps client code which connects to the ldap server securely and does authentication.
I have set the global option for ca cert directory.
int res = ldap_set_option(0, LDAP_OPT_X_TLS_CACERTFILE, const_cast<char*>("path"));
Correct certificate is present in the path and hence connection to the ldap server and authentication is successful in first attempt.
Now, as per one of the requirements, the certificate is removed from the above client directory and authentication is attempted, we want it to fail as the certificate is deleted from the directory. But still the bind to ldap server and authentication is happening successfully. It should ideally fail as per my understanding.
I did try removing the certificate from memory using following option:
char * crt;
ldap_get_option(0, LDAP_OPT_X_TLS_CACERTFILE, (void*)&crt);
ldap_memfree(crt);
I am not sure if the above way is correct or not, please advise.
I also did try forcing to look for ca certificate using following option:
int reqcert = LDAP_OPT_X_TLS_HARD;
ldap_set_option(0, LDAP_OPT_X_TLS_REQUIRE_CERT, &reqcert);
But this also did not help.
Please suggest how a certificate, which is once loaded can be deleted from the openldap cache. Also please advise if I am doing something wrong in the above approach.
2 years, 2 months
Using supportedControl 1.3.6.1.4.1.4203.1.10.1 (Subentries) - critical extension is unavailable 12
by Gavin Henry
Hi all,
I'm doing this for a supportControl subentry delete:
https://metacpan.org/pod/Net::LDAP::Control
like so:
my $subentry_ctrl = Net::LDAP::Control->new(
type => '1.3.6.1.4.1.4203.1.10.1',
value => 'Subentries',
critical => 1
);
my $deleted = $c->model('LDAPContacts')->delete(
q{ou=Contacts,} . $user_dn,
control => [ $subentry_ctrl ]
);
if ( $deleted->code ) {
$c->error( qq{Failed to delete LDAP contact entries for: $user_dn}
. $deleted->error
. q{ Code: }
. $deleted->code );
return 0;
}
and I'm getting:
Dec 22 12:53:57 gabriel slapd[31511]: conn=1110022 op=2 SRCH attr=dn
Dec 22 12:53:57 gabriel slapd[31511]: conn=1110022 op=2 ENTRY dn="xxxx"
Dec 22 12:53:57 gabriel slapd[31511]: conn=1110022 op=2 SEARCH RESULT
tag=101 err=0 qtime=0.000010 etime=0.001015 nentries=1 text=
Dec 22 12:53:57 gabriel slapd[31511]: conn=1110023 fd=435 ACCEPT from
IP=xxx:51082 (IP=0.0.0.0:389)
Dec 22 12:53:57 gabriel slapd[31511]: conn=1110023 op=0 EXT
oid=1.3.6.1.4.1.1466.20037
Dec 22 12:53:57 gabriel slapd[31511]: conn=1110023 op=0 STARTTLS
Dec 22 12:53:57 gabriel slapd[31511]: conn=1110023 op=0 RESULT oid=
err=0 qtime=0.000007 etime=0.000038 text=
Dec 22 12:53:57 gabriel slapd[31511]: conn=1110023 fd=435 TLS
established tls_ssf=256 ssf=256 tls_proto=TLSv1.3
tls_cipher=TLS_AES_256_GCM_SHA384
Dec 22 12:53:57 gabriel slapd[31511]: conn=1110023 op=1 BIND dn="xxx" method=128
Dec 22 12:53:57 gabriel slapd[31511]: conn=1110023 op=1 BIND dn="xxx"
mech=SIMPLE ssf=0
Dec 22 12:53:57 gabriel slapd[31511]: conn=1110023 op=1 RESULT tag=97
err=0 qtime=0.000017 etime=0.000116 text=
Dec 22 12:53:58 gabriel slapd[31511]: conn=1110023 op=2 RESULT tag=107
err=12 qtime=0.000014 etime=0.000274 text=critical extension is
unavailable
Dec 22 12:53:58 gabriel slapd[31511]: conn=1110023 op=2 do_delete:
get_ctrls failed
Any ideas? Using ldapdelete with -r works as the same user (so not my
ACLs), but I note in the logs that it is doing a base search for
subentries and deleting each one.
What am I misunderstanding here?
Thanks,
Gavin.
2 years, 3 months
Bind using External/Client certificate not working - slapd returns binding in progress..
by dumitru s
Hi all,
I'm using openldap/slapd as a ldap server (using libsasl2-2 & related modules for sasl auth) on ubuntu and trying to get a client to authenticate/bind using external/client certificate.
I'm using two clients - one is a native C client using windows winldap native library and one is based on a different client ldap library (i.e. not using winldap or openldap native libraries). The client based on winldap works fine, but not the other one.
This is what I can see in the slapd logs for the two cases:
- the one which works fine via winldap
5fe876d4 conn=1000 op=0 do_bind
ber_scanf fmt ({imt) ber:
ber_scanf fmt ({m) ber:
ber_scanf fmt (m) ber:
ber_scanf fmt (}}) ber:
5fe876d4 >>> dnPrettyNormal: <>
5fe876d4 <<< dnPrettyNormal: <>, <>
5fe876d4 do_bind: dn () SASL mech EXTERNAL
5fe876d4 ==>slap_sasl2dn: converting SASL name email=test(a)test.com,cn=example,ou=example,o=example,st=anystate,c=us to a DN
5fe876d4 ==> rewrite_context_apply [depth=1] string='email=test(a)test.com,cn=example,ou=example,o=example,st=anystate,c=us'
5fe876d4 ==> rewrite_rule_apply rule='email=test(a)test.com,cn=example,ou=example,o=example,st=anystate,c=us' string='email=test(a)test.com,cn=example,ou=example,o=example,st=anystate,c=us' [1 pass(es)]
5fe876d4 ==> rewrite_context_apply [depth=1] res={0,'cn=test,dc=example,dc=com'}
5fe876d4 slap_parseURI: parsing cn=test,dc=example,dc=com
ldap_url_parse_ext(cn=test,dc=example,dc=com)
5fe876d4 >>> dnNormalize: <cn=test,dc=example,dc=com>
5fe876d4 <<< dnNormalize: <cn=test,dc=example,dc=com>
5fe876d4 <==slap_sasl2dn: Converted SASL name to cn=test,dc=example,dc=com
5fe876d4 slap_sasl_getdn: dn:id converted to cn=test,dc=example,dc=com
5fe876d4 SASL Authorize [conn=1000]: proxy authorization allowed authzDN=""
5fe876d4 send_ldap_sasl: err=0 len=-1
5fe876d4 do_bind: SASL/EXTERNAL bind: dn="cn=test,dc=example,dc=com" sasl_ssf=0
5fe876d4 send_ldap_response: msgid=1 tag=97 err=0
- the one which doesn't work
5fe87b50 conn=1001 op=0 do_bind
ber_scanf fmt ({imt) ber:
ber_scanf fmt ({m) ber:
ber_scanf fmt (}}) ber:
5fe87b50 >>> dnPrettyNormal: <>
5fe87b50 <<< dnPrettyNormal: <>, <>
5fe87b50 do_bind: dn () SASL mech EXTERNAL
5fe87b50 send_ldap_sasl: err=14 len=0
5fe87b50 send_ldap_response: msgid=1 tag=97 err=14
As can be seen, the second one stops at "do_bind: dn () SASL mech EXTERNAL" and slapd just returns the binding in progress result code.
Of course, the same client certificate is used in both cases. The fact that one client works fine suggests that the slapd configuration is correct.
Any idea what is wrong? Can I enable any additional logs (sasl one?) to be able to see more?
Thanks,
Dumitru
2 years, 3 months
Migrate HDB to MDB
by Pete Ashdown
I'm looking for some assistance in converting a legacy LDAP from an HDB
backend to MDB. I've been unable to find any resources in how this can
be executed and my attempts at using ldapmodify have failed. I'm
willing to pay consulting fees if someone is available to help, or
otherwise be educated if is documented somewhere.
Thanks in advance.
2 years, 3 months
ACL for authz-regexp
by Stefan Kania
Hello,
I try to figure out which ACL I need to get the rewriting of the
sasl-username working.
I have in my slapd.conf the following lines:
----------
authz-regexp
uid=(.+),cn=gssapi,cn=auth
ldap:///dc=example,dc=net??sub?(uid=$1)
-----------
If I do a "ldapwhoami" without any ACL I get what I want:
-----------
ptau@provider-stat:~$ ldapwhoami
SASL/GSSAPI authentication started
SASL username: ptau(a)EXAMPLE.NET
SASL SSF: 256
SASL data security layer installed.
dn:uid=ptau,ou=users,dc=example,dc=net
-----------
As soon as I include my ACLs I get:
------------
ptau@provider-stat:~$ ldapwhoami
SASL/GSSAPI authentication started
SASL username: ptau(a)EXAMPLE.NET
SASL SSF: 256
SASL data security layer installed.
dn:uid=ptau,cn=gssapi,cn=auth
------------
So the rewrite of the DN is not working anymore with ACLs. Here are the
relevant part of my ACLs. (It's at the beginning of the ACLs)
------------
limits dn.exact="uid=repl-user,ou=users,dc=example,dc=net"
size=unlimited time=unlimited
limits dn.exact="cn=kdc,ou=kerberos-adm,dc=example,dc=net"
size=unlimited time=unlimited
limits dn.exact="cn=kadmin,ou=kerberos-adm,dc=example,dc=net"
size=unlimited time=unlimited
access to dn.base="" by * read
access to dn.base="cn=subSchema" by * read
access to *
by dn.exact="uid=sssd-user,ou=users,dc=example,dc=net" read
by dn.exact="uid=ldap-admin,ou=users,dc=example,dc=net" write
by dn.exact="uid=repl-user,ou=users,dc=example,dc=net" read
by dn.exact="cn=kdc,ou=kerberos-adm,dc=example,dc=net" write
by dn.exact="cn=kadmin,ou=kerberos-adm,dc=example,dc=net" write
by dn.regex="(.+),cn=gssapi,cn=auth" auth
by * break
------------
As soon as I remove the line:
------------
by dn.regex="(.+),cn=gssapi,cn=auth" auth
------------
and change the last line to:
-------------
by * auth
-------------
But this will make all other ACLs, after this ACLs, useless.
I tried so far:
by ssf=256 auth
by sasl_ssf=256 auth
non of the above is working
So it's definitely an ACL issue, but how do I solve it? The log-File is
giving me the following output:
--------------
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 fd=20 ACCEPT from
IP=192.168.56.81:50728 (IP=0.0.0.0:636)
Dez 20 19:22:46 provider-stat ldapwhoami[2047]: GSSAPI client step 1
Dez 20 19:22:46 provider-stat ldapwhoami[2047]: GSSAPI client step 1
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 fd=20 TLS
established tls_ssf=256 ssf=256
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 op=0 BIND dn=""
method=163
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 op=0 RESULT tag=97
err=14 text=SASL(0): successful result:
Dez 20 19:22:46 provider-stat ldapwhoami[2047]: GSSAPI client step 1
Dez 20 19:22:46 provider-stat slapd[2038]: connection_input: conn=1004
deferring operation: binding
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 op=1 BIND dn=""
method=163
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 op=1 RESULT tag=97
err=14 text=SASL(0): successful result:
Dez 20 19:22:46 provider-stat ldapwhoami[2047]: GSSAPI client step 2
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 op=2 BIND dn=""
method=163
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 op=2 BIND
authcid="ptau" authzid="ptau"
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 op=2 BIND
dn="uid=ptau,cn=gssapi,cn=auth" mech=GSSAPI sasl_ssf=256 ssf=256
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 op=2 RESULT tag=97
err=0 text=
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 op=3 EXT
oid=1.3.6.1.4.1.4203.1.11.3
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 op=3 WHOAMI
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 op=3 RESULT oid=
err=0 text=
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 op=4 UNBIND
Dez 20 19:22:46 provider-stat slapd[2038]: conn=1004 fd=20 closed
Dez 20 19:22:46 provider-stat ldapwhoami[2047]: DIGEST-MD5 common mech free
--------------
I think it had something to do with what I found in the manpage:
-------------------
Some internal operations and some controls require specific
access privileges. The authzID mapping and the proxyAuthz control
require auth (=x) privileges on all the attributes that are present
in the
search filter of the URI regexp maps (the right-hand side of
the authz-regexp directives). Auth (=x) privileges are also required on
the authzTo attribute of the authorizing identity and/or on the au‐
thzFrom attribute of the authorized identity. In general, when
an internal lookup is performed for authentication or authorization
purposes, search-specific privileges (see the access requirements for the
search operation illustrated above) are relaxed to auth.
-------------------
But my problem here is, I'm not a native English speaker and it's hard
for me to figure out what the write is telling me here :-(
Any solution for the right ACL to solve the problem?
Greetings
Stefan
2 years, 3 months
How to set cipher
by Puranjay Pradhan
I am trying to set cipher list as like below :
std::string ciphers =
"AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA"
returnCode = ldap_set_option(NULL, LDAP_OPT_X_TLS_CIPHER_SUITE,
ciphers.c_str());
But, it returns an error : "TLS could not set cipher list ....."
Can anyone suggest what is the right way of doing this?
2 years, 3 months
verify openldap source
by A. Schulze
Hello,
I'm searching for a way to verify the integrity of downloads from openldap.org.
Many open source projects use to provide foo-$version.tar.gz.[asc|sha256sum] next to foo-$version.tar.gz
Is something similar available for openldap?
Andreas
2 years, 3 months
new index on consumer
by Stefan Kania
Hello
If I create a new index on a provider in slapd.conf NOT in cn=config
(using mdb). I have to do a slapindex on the provider. When I configure
the same index in consumer slapd.conf do I have to do the slapindex here
too?
Greeting
Stefan
--
2 years, 3 months
