Re: Large user bases cause huge number of roleOccupant in role organizationalRole
by Quanah Gibson-Mount
--On Monday, March 27, 2017 11:16 AM +0200 Huang Hongfu
<huang.hongfu(a)adnovum.ch> wrote:
> Dear Quanah,
>
> When will the OpenLDAP 2.5 be published? What is the maximum size of a
> large set?
Hi Hongfu,
There is no timeline for when 2.5 will be released at the moment.
Regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 12 months
Re: Large user bases cause huge number of roleOccupant in role organizationalRole
by Quanah Gibson-Mount
--On Friday, March 24, 2017 8:25 PM +0000 Howard Chu <hyc(a)symas.com> wrote:
> Meanwhile, for people with stupid data models, performance with large
> attributes in back-mdb is greatly improved in OpenLDAP 2.5.
I would note that Symas's OpenLDAP build includes support for attributes
with a large set of values as well, as it is undetermined when OpenLDAP 2.5
will release.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 12 months
Re: Large user bases cause huge number of roleOccupant in role organizationalRole
by Howard Chu
Huang Hongfu wrote:
> Hello,
>
> In a general design, we have a role, which is an object of organizationalRole.
> Within this role, let us say it will have more than 10 million users. Each
> user has a roleOccupant in the role object.
>
> You can imagine how huge this role object can be! This will cause a big
> performance problem. Especially with LMDB as backend, the "add new user" will
> become very slow at some point; and the replication will be very difficult
> even we setup a delta (with accesslog and session log) replication.
>
> From my point of view, organizationalRole is not designed for large user bases.
>
> Or someone has a better idea?
The same logic as used in indexing applies here - it is stupid to define a
case for something that matches the majority of your population. I.e., it's
stupid to define a presence index on an attribute if that attribute occurs in
the majority of your entries. It is stupid to define a group or role if the
majority of entries will be a member of it. A presence index is only useful if
the attribute occurs rarely. A group/role is only useful if the members are a
minority of the total population.
In this case you should define a group/role for all users who *aren't* part of
the majority.
Meanwhile, for people with stupid data models, performance with large
attributes in back-mdb is greatly improved in OpenLDAP 2.5.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
5 years, 12 months
Large user bases cause huge number of roleOccupant in role organizationalRole
by Huang Hongfu
Hello,
In a general design, we have a role, which is an object of
organizationalRole. Within this role, let us say it will have more than
10 million users. Each user has a roleOccupant in the role object.
You can imagine how huge this role object can be! This will cause a big
performance problem. Especially with LMDB as backend, the "add new user"
will become very slow at some point; and the replication will be very
difficult even we setup a delta (with accesslog and session log)
replication.
From my point of view, organizationalRole is not designed for large
user bases.
Or someone has a better idea?
Best regards
Hongfu
--
Hongfu Huang, Senior System Engineer
MSc in Computer Science
We're the tech in Fintech: www.adnovum.ch/fintech
AdNovum Informatik AG
Wengistrase 7, 8005 Zurich, Switzerland
phone +41 44 272 6111, Direct: +41 44 270 5266
huang.hongfu(a)adnovum.ch
www.adnovum.ch
Locations: Zurich (HQ), Bern, Budapest, Ho Chi Minh City, Singapore
5 years, 12 months
Re: 100.000 pr. second: "connection_read(64): no connection" from slapd
by Quanah Gibson-Mount
--On Thursday, March 23, 2017 2:37 PM +0000 Ole Nomann Thomsen
<ole.nomann(a)stil.dk> wrote:
>
>
> Hi all,
>
>
>
> my slapd (@(#) $OpenLDAP: slapd 2.4.44 (Feb 9 2017 13:38:13) $) has
> developed a tendency to become unresponsive for 10-30 minutes at a time.
>
> I run it with -dStats,Sync which gets me a fair amount of debugging info.
> This I pipe thru multilog to get the timestamps below.
Why are you using -d? This prevents slapd from forking. It is better to
set the loglevel to be stats+sync, and push out data to syslog.
> 2017-03-10 14:39:08.686058500 58c2ac7c slapd shutdown: waiting for
> 3643059 operations/tasks to finish
You need to determine why tasks are not finishing and instead are remaining
unfinished. What databse backend are you using? I would note that it
would appear that you have clients disconnecting w/o waiting for the server
to respond (thus the no connection messages).
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 12 months
Re: syncrepl and the frontend database
by Quanah Gibson-Mount
--On Tuesday, March 21, 2017 10:21 PM +0300 Andrey Klimentev
<andrei650816(a)gmail.com> wrote:
>
> Hello.
>
>
> I've got a question about replicating the frontend database. Should I
> send syncprov on it and replicate it with cn=config and other databases?
> Or is it somehow handled in automated manner?
You can replicate cn=config if you choose, but whether or not that is a
wise decision is debatable.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 12 months
100.000 pr. second: "connection_read(64): no connection" from slapd
by Ole Nomann Thomsen
Hi all,
my slapd (@(#) $OpenLDAP: slapd 2.4.44 (Feb 9 2017 13:38:13) $) has developed a tendency to become unresponsive for 10-30 minutes at a time.
I run it with -dStats,Sync which gets me a fair amount of debugging info. This I pipe thru multilog to get the timestamps below.
The two snippets below are from two different dates, due to the difficulty of reproducing the error.
What I'm seeing at the time, is that I am getting thousands of these (from an earlier run):
2017-03-06 14:15:11.641329500 58bd60df connection_read(64): no connection!
2017-03-06 14:15:11.641331500 58bd60df connection_read(64): no connection!
2017-03-06 14:15:11.641393500 58bd60df connection_read(64): no connection!
2017-03-06 14:15:11.641396500 58bd60df connection_read(64): no connection!
2017-03-06 14:15:11.641398500 58bd60df connection_read(64): no connection!
2017-03-06 14:15:11.641401500 58bd60df connection_read(64): no connection!
2017-03-06 14:15:11.641403500 58bd60df connection_read(64): no connection!
Yes: about 2 or 3 microseconds apart! About 100.000 pr. Second.
(
I tried post-filtering the "no connection" message away before it hit the log (possible with multilog),
theorizing that the disc simply got swamped by these, but slapd was still unresponsive.
I can't run the slapd without some debugging, as I need the logtrace, and the "no connection" seems to
be output on any debug flag:
>From openldap-2.4.44/servers/slapd/connection.c:
Debug( LDAP_DEBUG_ANY,
"connection_read(%ld): no connection!\n",
(long) s, 0, 0 );
)
The (64) varies, but there is always one number responsible for the 100.000/sec and a
few other numbers with less than 10/sec.
Sometimes it recovers on itself, other times I kill slapd and get this (from a later run);
2017-03-10 14:39:08.632744500 58c2ac7c daemon: shutdown requested and initiated.
2017-03-10 14:39:08.633288500 58c2ac7c conn=45284802 fd=19 closed (slapd shutdown)
2017-03-10 14:39:08.633348500 58c2ac7c conn=45284896 fd=20 closed (slapd shutdown)
2017-03-10 14:39:08.633468500 58c2ac7c conn=45284823 fd=24 closed (slapd shutdown)
2017-03-10 14:39:08.633519500 58c2ac7c conn=45284825 fd=25 closed (slapd shutdown)
2017-03-10 14:39:08.633558500 58c2ac7c conn=45284829 fd=26 closed (slapd shutdown)
2017-03-10 14:39:08.633596500 58c2ac7c conn=45284838 fd=27 closed (slapd shutdown)
2017-03-10 14:39:08.633684500 58c2ac7c conn=45284845 fd=28 closed (slapd shutdown)
2017-03-10 14:39:08.634018500 58c2ac7c conn=45284890 fd=29 closed (slapd shutdown)
2017-03-10 14:39:08.634823500 58c2ac7c conn=45266816 fd=33 closed (slapd shutdown)
2017-03-10 14:39:08.686058500 58c2ac7c slapd shutdown: waiting for 3643059 operations/tasks to finish
2017-03-10 14:39:17.320835500 58c2ac85 daemon: accept(-1) failed errno=9 (Bad file descriptor)
2017-03-10 14:39:34.750553500 58c2ac96 daemon: accept(-1) failed errno=9 (Bad file descriptor)
2017-03-10 14:40:55.413900500 58c2ace7 slapd stopped.
Note that: 3643059 operations/tasks
Does anybody have a hint what I am facing here, or where I should look?
I have not (yet) discovered a reliable way to reproduce the errorm normally I just wait for it to happend.
Regards, Ole.
5 years, 12 months
Dovecot can't connect to openldap over starttls
by info＠gwarband.de
Hello,
can anybody say something about my problem?
The mails in the bottom are from my discuss with the dovecot maillist.
Thanks,
Tobias
-------- Originalnachricht --------
Betreff: Re: Dovecot can't connect to openldap over starttls
Datum: 2017-03-18 14:22
Absender: info(a)gwarband.de
Empfänger: Tomas Habarta <lists+dovecot(a)tocc.cz>
Kopie: dovecot(a)dovecot.org
The serverlog of openldap with loglevel "any":
https://gwarband.de/openldap/openldap-connect.log
Note: openldap waits 1 Minute before he says "TLS negotiation failure"
after the connect.
and dovecot says direct "Connect error"
I've also delete the TLSCipherSuite from openldap.
Tobias
Am 2017-03-18 14:01, schrieb Tomas Habarta:
> Increase log level on server side as well to see what the server
> says...
> You may remove anything in TLSCipherSuite for the purpose of testing
> too.
> Hopefully anyone knowing OpenLDAP internals could help you analyse it
> more deeply.
> Tomas
> On 03/18/2017 01:31 PM, info(a)gwarband.de wrote:
>> I've replicate the settings from ldapsearch to dovecot but no
>> success.
>> To the certificate:
>> Yes it's a *.crt file but I have linked the *.pem file to it and
>> dovecot
>> has read access to that file.
>> I have enabled the debugging in dovecot and have uploaded the output:
>> https://gwarband.de/openldap/dovecot-connect.log
>> And the other site with ldapsearch:
>> https://gwarband.de/openldap/ldapsearch-connect.log
>> I'm pretty sure that there is a problem with the sslhandshaking
>> between
>> openldap and dovecot, but I can't find the source of the problem.
>> One of the steps in the sslhandshaking is not success but in the
>> debugging output I can't find any line with a hit to it.
>> Tobias
>> Am 2017-03-18 12:30, schrieb Tomas Habarta:
>>> Well, if ldapsearch works, try to replicate its settings for dovecot
>>> client.
>>> It's not obvious what settings ldapsearch uses, have a look at
>>> default
>>> client settings in /etc/openldap/ldap.conf, there may be something
>>> set a
>>> slightly different way.
>>> Also double check permissions for files used by dovecot, I mean
>>> mainly
>>> the file listed for tls_ca_cert_file as dovecot may not have an
>>> access
>>> for reading...
>>> I cannot see anything downright bad, just posted CA cert (which is
>>> ok,
>>> tested) is *.crt and your config mentions *.pem but I consider it's
>>> the
>>> same file.
>>> Finally, I would recommend to enable debug option for dovecot's
>>> client
>>> debug_level = -1 (which logs all available) in your
>>> dovecot-ldap.conf
>>> to see what the library reports and work further on that.
>>> You can compare with output from ldapsearch by adding -d-1 switch to
>>> it.
>>> Hard to tell more at the moment.
>>>
>>> Tomas
>>> On 03/18/2017 09:41 AM, info(a)gwarband.de wrote:
>>>> Hello,
>>>> I have also installed LE certs.
>>>> But nothing helps, I have double-checking all certs.
>>>> ldapsearch with -ZZ works see:
>>>> https://gwarband.de/openldap/ldapsearch.log
>>>> I have also uploaded the TLSCACertificateFile, maybe I have a
>>>> failure in
>>>> the merge of the two fiels:
>>>> https://gwarband.de/openldap/LetsEncrypt.crt
>>>> And also I have uploaded my complete openldap configuration:
>>>> https://gwarband.de/openldap/openldap.conf
>>>> All other components can work and communicate with my openldap
>>>> server.
>>>> The components are postfix, openxchange, apache (phpldapadmin).
>>>> My installated software is:
>>>> Debian 8
>>>> OpenLDAP 2.4.40
>>>> Dovecot 2.2.13
>>>> I hope you can find the issue.
>>>> Thanks,
>>>> Tobias
>>>> Am 2017-03-17 22:48, schrieb Tomas Habarta:
>>>>> Hi,
>>>>> been running Dovecot 2.2.27 against OpenLDAP 2.4.40 normally over
>>>>> the
>>>>> unix socket on the same machine, but tried over inet with STARTTLS
>>>>> and
>>>>> it's working ok...
>>>>> I would suggest double-checking key/certs setup on OpenLDAP side;
>>>>> for
>>>>> the test I have used LE certs, utilizing following cn=config
>>>>> attributes:
>>>>> olcTLSCertificateKeyFile contains private key
>>>>> olcTLSCertificateFile contains certificate
>>>>> olcTLSCACertificateFile contains both certs (DST Root CA X3
>>>>> and Let's Encrypt Authority X3)
>>>>> and used the same CA file in Dovecot's tls_ca_cert_file
>>>>> Is ldapsearch working ok (-ZZ) and only Dovecot has troubles or
>>>>> ... ?
>>>>>
>>>>>
>>>>> Hope that helps, good luck ;)
>>>>> Tomas
>>>>>
>>>>> On 03/17/2017 04:27 PM, info(a)gwarband.de wrote:
>>>>>> Hello guys,
>>>>>> actually I'm trying to configure dovecot to access openldap for
>>>>>> passwordcheck.
>>>>>> My openldap is only allow access over "secure ldap".
>>>>>> The dovecot can communicate with the openldap server but there is
>>>>>> maybe
>>>>>> a failure in the sslhandshake.
>>>>>> Additional information you can find in the logs or in the dump
>>>>>> below.
>>>>>> Also I have my ldap config from dovecot in the links below.
>>>>>> I have already created an bug reporting in the system of openldap
>>>>>> but
>>>>>> the answer was to get support from her.
>>>>>> All datalinks:
>>>>>> https://gwarband.de/openldap/dovecot.log
>>>>>> https://gwarband.de/openldap/dovecot-ldap.conf
>>>>>> https://gwarband.de/openldap/openldap.log
>>>>>> https://gwarband.de/openldap/trace.dump
>>>>>> The bugreportinglink from openldap:
>>>>>> http://www.openldap.org/its/index.cgi/Incoming?id=8615
>>>>>> I hope you can help me.
>>>>>> Regards.
>>>>>> Tobias Warband
6 years
Possible to rebind-as-user with chain overlay using TLS client certificates?
by Matthew Kemp
Hey all,
We've got a fairly straightforward producer/consumer setup with a single
producer and multiple syncrepl consumers configured with an updateref back
to the producer. The consumers are set up to use the chain overlay with the
ldap backend for transparent password updates. We use TLS client
certificates to authenticate the clients to the consumers and the consumers
to the producers with SASL EXTERNAL binds.
However, I've been struggling with configuring what I had hoped would be a
simple part of the setup. We would like the ability to have the user simple
bind as themselves to the consumer and then have the chain overlay
transparently follow the updateref back to the producer and rebind as the
user there to do the PASSMOD update. For whatever reason, I haven't been
successful at making that happen. I've tried several attempts at various
back_ldap configurations, to no avail. By playing with the
olcDbIdAssertBind and various mode/flags, I can get the backend to attempt
ProxyAuth (which we'd prefer not to use if possible), and alternatively I
am able to get the chain to anonymously bind, but I am not able to initiate
a rebind as the originating user.
We're running slapd 2.4.31 under Debian Wheezy on the producer and a
variety of 2.4.31 and 2.4.40 slapd consumers under Debian Wheezy and Debian
Jesse, respectively.
I was hoping you'd be able to shed some light whether this is even
possible? Attached are the most recent configuration that I believe should
work for my needs.
Thanks!
---
Current consumer config:
dn: olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config
objectClass: olcOverlayConfig
objectClass: olcChainConfig
olcOverlay: {0}chain
olcChainReturnError: TRUE
structuralObjectClass: olcChainConfig
dn: olcDatabase={0}ldap,olcOverlay={0}chain,olcDatabase={-1}
frontend,cn=config
objectClass: olcLDAPConfig
objectClass: olcChainDatabase
olcDatabase: {0}ldap
structuralObjectClass: olcLDAPConfig
olcDbURI: ldap://ldap-primary
olcDbRebindAsUser: TRUE
olcDbStartTLS: start tls_cert=/etc/ssl/certs/ldap-consumer.crt
tls_key=/etc/ssl/private/ldap-consumer.key tls_cacert=/usr/share/ca-certi
ficates/cosmos/ldap_producer_ca_pem.crt
--
Matt Kemp
Production Engineer
Braintree
6 years