openldap-technical

openldap-technical@openldap.org

9790 discussions

Re: Large user bases cause huge number of roleOccupant in role organizationalRole
by Howard Chu 27 Mar '17

27 Mar '17

Huang Hongfu wrote: > Hello, > > In a general design, we have a role, which is an object of organizationalRole. > Within this role, let us say it will have more than 10 million users. Each > user has a roleOccupant in the role object. > > You can imagine how huge this role object can be! This will cause a big > performance problem. Especially with LMDB as backend, the "add new user" will > become very slow at some point; and the replication will be very difficult > even we setup a delta (with accesslog and session log) replication. > > From my point of view, organizationalRole is not designed for large user bases. > > Or someone has a better idea? The same logic as used in indexing applies here - it is stupid to define a case for something that matches the majority of your population. I.e., it's stupid to define a presence index on an attribute if that attribute occurs in the majority of your entries. It is stupid to define a group or role if the majority of entries will be a member of it. A presence index is only useful if the attribute occurs rarely. A group/role is only useful if the members are a minority of the total population. In this case you should define a group/role for all users who *aren't* part of the majority. Meanwhile, for people with stupid data models, performance with large attributes in back-mdb is greatly improved in OpenLDAP 2.5. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

Large user bases cause huge number of roleOccupant in role organizationalRole
by Huang Hongfu 24 Mar '17

24 Mar '17

Hello, In a general design, we have a role, which is an object of organizationalRole. Within this role, let us say it will have more than 10 million users. Each user has a roleOccupant in the role object. You can imagine how huge this role object can be! This will cause a big performance problem. Especially with LMDB as backend, the "add new user" will become very slow at some point; and the replication will be very difficult even we setup a delta (with accesslog and session log) replication. From my point of view, organizationalRole is not designed for large user bases. Or someone has a better idea? Best regards Hongfu -- Hongfu Huang, Senior System Engineer MSc in Computer Science We're the tech in Fintech: www.adnovum.ch/fintech AdNovum Informatik AG Wengistrase 7, 8005 Zurich, Switzerland phone +41 44 272 6111, Direct: +41 44 270 5266 huang.hongfu(a)adnovum.ch www.adnovum.ch Locations: Zurich (HQ), Bern, Budapest, Ho Chi Minh City, Singapore

2 1

Re: 100.000 pr. second: "connection_read(64): no connection" from slapd
by Quanah Gibson-Mount 23 Mar '17

23 Mar '17

--On Thursday, March 23, 2017 2:37 PM +0000 Ole Nomann Thomsen <ole.nomann(a)stil.dk> wrote: > > > Hi all, > > > > my slapd (@(#) $OpenLDAP: slapd 2.4.44 (Feb 9 2017 13:38:13) $) has > developed a tendency to become unresponsive for 10-30 minutes at a time. > > I run it with -dStats,Sync which gets me a fair amount of debugging info. > This I pipe thru multilog to get the timestamps below. Why are you using -d? This prevents slapd from forking. It is better to set the loglevel to be stats+sync, and push out data to syslog. > 2017-03-10 14:39:08.686058500 58c2ac7c slapd shutdown: waiting for > 3643059 operations/tasks to finish You need to determine why tasks are not finishing and instead are remaining unfinished. What databse backend are you using? I would note that it would appear that you have clients disconnecting w/o waiting for the server to respond (thus the no connection messages). --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: syncrepl and the frontend database
by Quanah Gibson-Mount 23 Mar '17

23 Mar '17

--On Tuesday, March 21, 2017 10:21 PM +0300 Andrey Klimentev <andrei650816(a)gmail.com> wrote: > > Hello. > > > I've got a question about replicating the frontend database. Should I > send syncprov on it and replicate it with cn=config and other databases? > Or is it somehow handled in automated manner? You can replicate cn=config if you choose, but whether or not that is a wise decision is debatable. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

100.000 pr. second: "connection_read(64): no connection" from slapd
by Ole Nomann Thomsen 23 Mar '17

23 Mar '17

Hi all, my slapd (@(#) $OpenLDAP: slapd 2.4.44 (Feb 9 2017 13:38:13) $) has developed a tendency to become unresponsive for 10-30 minutes at a time. I run it with -dStats,Sync which gets me a fair amount of debugging info. This I pipe thru multilog to get the timestamps below. The two snippets below are from two different dates, due to the difficulty of reproducing the error. What I'm seeing at the time, is that I am getting thousands of these (from an earlier run): 2017-03-06 14:15:11.641329500 58bd60df connection_read(64): no connection! 2017-03-06 14:15:11.641331500 58bd60df connection_read(64): no connection! 2017-03-06 14:15:11.641393500 58bd60df connection_read(64): no connection! 2017-03-06 14:15:11.641396500 58bd60df connection_read(64): no connection! 2017-03-06 14:15:11.641398500 58bd60df connection_read(64): no connection! 2017-03-06 14:15:11.641401500 58bd60df connection_read(64): no connection! 2017-03-06 14:15:11.641403500 58bd60df connection_read(64): no connection! Yes: about 2 or 3 microseconds apart! About 100.000 pr. Second. ( I tried post-filtering the "no connection" message away before it hit the log (possible with multilog), theorizing that the disc simply got swamped by these, but slapd was still unresponsive. I can't run the slapd without some debugging, as I need the logtrace, and the "no connection" seems to be output on any debug flag: >From openldap-2.4.44/servers/slapd/connection.c: Debug( LDAP_DEBUG_ANY, "connection_read(%ld): no connection!\n", (long) s, 0, 0 ); ) The (64) varies, but there is always one number responsible for the 100.000/sec and a few other numbers with less than 10/sec. Sometimes it recovers on itself, other times I kill slapd and get this (from a later run); 2017-03-10 14:39:08.632744500 58c2ac7c daemon: shutdown requested and initiated. 2017-03-10 14:39:08.633288500 58c2ac7c conn=45284802 fd=19 closed (slapd shutdown) 2017-03-10 14:39:08.633348500 58c2ac7c conn=45284896 fd=20 closed (slapd shutdown) 2017-03-10 14:39:08.633468500 58c2ac7c conn=45284823 fd=24 closed (slapd shutdown) 2017-03-10 14:39:08.633519500 58c2ac7c conn=45284825 fd=25 closed (slapd shutdown) 2017-03-10 14:39:08.633558500 58c2ac7c conn=45284829 fd=26 closed (slapd shutdown) 2017-03-10 14:39:08.633596500 58c2ac7c conn=45284838 fd=27 closed (slapd shutdown) 2017-03-10 14:39:08.633684500 58c2ac7c conn=45284845 fd=28 closed (slapd shutdown) 2017-03-10 14:39:08.634018500 58c2ac7c conn=45284890 fd=29 closed (slapd shutdown) 2017-03-10 14:39:08.634823500 58c2ac7c conn=45266816 fd=33 closed (slapd shutdown) 2017-03-10 14:39:08.686058500 58c2ac7c slapd shutdown: waiting for 3643059 operations/tasks to finish 2017-03-10 14:39:17.320835500 58c2ac85 daemon: accept(-1) failed errno=9 (Bad file descriptor) 2017-03-10 14:39:34.750553500 58c2ac96 daemon: accept(-1) failed errno=9 (Bad file descriptor) 2017-03-10 14:40:55.413900500 58c2ace7 slapd stopped. Note that: 3643059 operations/tasks Does anybody have a hint what I am facing here, or where I should look? I have not (yet) discovered a reliable way to reproduce the errorm normally I just wait for it to happend. Regards, Ole.

1 0

Dovecot can't connect to openldap over starttls
by info＠gwarband.de 22 Mar '17

22 Mar '17

Hello, can anybody say something about my problem? The mails in the bottom are from my discuss with the dovecot maillist. Thanks, Tobias -------- Originalnachricht -------- Betreff: Re: Dovecot can't connect to openldap over starttls Datum: 2017-03-18 14:22 Absender: info(a)gwarband.de Empfänger: Tomas Habarta <lists+dovecot(a)tocc.cz> Kopie: dovecot(a)dovecot.org The serverlog of openldap with loglevel "any": https://gwarband.de/openldap/openldap-connect.log Note: openldap waits 1 Minute before he says "TLS negotiation failure" after the connect. and dovecot says direct "Connect error" I've also delete the TLSCipherSuite from openldap. Tobias Am 2017-03-18 14:01, schrieb Tomas Habarta: > Increase log level on server side as well to see what the server > says... > You may remove anything in TLSCipherSuite for the purpose of testing > too. > Hopefully anyone knowing OpenLDAP internals could help you analyse it > more deeply. > Tomas > On 03/18/2017 01:31 PM, info(a)gwarband.de wrote: >> I've replicate the settings from ldapsearch to dovecot but no >> success. >> To the certificate: >> Yes it's a *.crt file but I have linked the *.pem file to it and >> dovecot >> has read access to that file. >> I have enabled the debugging in dovecot and have uploaded the output: >> https://gwarband.de/openldap/dovecot-connect.log >> And the other site with ldapsearch: >> https://gwarband.de/openldap/ldapsearch-connect.log >> I'm pretty sure that there is a problem with the sslhandshaking >> between >> openldap and dovecot, but I can't find the source of the problem. >> One of the steps in the sslhandshaking is not success but in the >> debugging output I can't find any line with a hit to it. >> Tobias >> Am 2017-03-18 12:30, schrieb Tomas Habarta: >>> Well, if ldapsearch works, try to replicate its settings for dovecot >>> client. >>> It's not obvious what settings ldapsearch uses, have a look at >>> default >>> client settings in /etc/openldap/ldap.conf, there may be something >>> set a >>> slightly different way. >>> Also double check permissions for files used by dovecot, I mean >>> mainly >>> the file listed for tls_ca_cert_file as dovecot may not have an >>> access >>> for reading... >>> I cannot see anything downright bad, just posted CA cert (which is >>> ok, >>> tested) is *.crt and your config mentions *.pem but I consider it's >>> the >>> same file. >>> Finally, I would recommend to enable debug option for dovecot's >>> client >>> debug_level = -1 (which logs all available) in your >>> dovecot-ldap.conf >>> to see what the library reports and work further on that. >>> You can compare with output from ldapsearch by adding -d-1 switch to >>> it. >>> Hard to tell more at the moment. >>> >>> Tomas >>> On 03/18/2017 09:41 AM, info(a)gwarband.de wrote: >>>> Hello, >>>> I have also installed LE certs. >>>> But nothing helps, I have double-checking all certs. >>>> ldapsearch with -ZZ works see: >>>> https://gwarband.de/openldap/ldapsearch.log >>>> I have also uploaded the TLSCACertificateFile, maybe I have a >>>> failure in >>>> the merge of the two fiels: >>>> https://gwarband.de/openldap/LetsEncrypt.crt >>>> And also I have uploaded my complete openldap configuration: >>>> https://gwarband.de/openldap/openldap.conf >>>> All other components can work and communicate with my openldap >>>> server. >>>> The components are postfix, openxchange, apache (phpldapadmin). >>>> My installated software is: >>>> Debian 8 >>>> OpenLDAP 2.4.40 >>>> Dovecot 2.2.13 >>>> I hope you can find the issue. >>>> Thanks, >>>> Tobias >>>> Am 2017-03-17 22:48, schrieb Tomas Habarta: >>>>> Hi, >>>>> been running Dovecot 2.2.27 against OpenLDAP 2.4.40 normally over >>>>> the >>>>> unix socket on the same machine, but tried over inet with STARTTLS >>>>> and >>>>> it's working ok... >>>>> I would suggest double-checking key/certs setup on OpenLDAP side; >>>>> for >>>>> the test I have used LE certs, utilizing following cn=config >>>>> attributes: >>>>> olcTLSCertificateKeyFile contains private key >>>>> olcTLSCertificateFile contains certificate >>>>> olcTLSCACertificateFile contains both certs (DST Root CA X3 >>>>> and Let's Encrypt Authority X3) >>>>> and used the same CA file in Dovecot's tls_ca_cert_file >>>>> Is ldapsearch working ok (-ZZ) and only Dovecot has troubles or >>>>> ... ? >>>>> >>>>> >>>>> Hope that helps, good luck ;) >>>>> Tomas >>>>> >>>>> On 03/17/2017 04:27 PM, info(a)gwarband.de wrote: >>>>>> Hello guys, >>>>>> actually I'm trying to configure dovecot to access openldap for >>>>>> passwordcheck. >>>>>> My openldap is only allow access over "secure ldap". >>>>>> The dovecot can communicate with the openldap server but there is >>>>>> maybe >>>>>> a failure in the sslhandshake. >>>>>> Additional information you can find in the logs or in the dump >>>>>> below. >>>>>> Also I have my ldap config from dovecot in the links below. >>>>>> I have already created an bug reporting in the system of openldap >>>>>> but >>>>>> the answer was to get support from her. >>>>>> All datalinks: >>>>>> https://gwarband.de/openldap/dovecot.log >>>>>> https://gwarband.de/openldap/dovecot-ldap.conf >>>>>> https://gwarband.de/openldap/openldap.log >>>>>> https://gwarband.de/openldap/trace.dump >>>>>> The bugreportinglink from openldap: >>>>>> http://www.openldap.org/its/index.cgi/Incoming?id=8615 >>>>>> I hope you can help me. >>>>>> Regards. >>>>>> Tobias Warband

3 8

Re: [LMDB] getting MDB_CORRUPTED when deleting within a DUPSORT database
by Howard Chu 22 Mar '17

22 Mar '17

Klaus Malorny wrote: > Hi, > > I am using version 0.9.20 on Linux (Ubuntu derivates, uname see [1], [2]). Version 0.9.20 was never officially released. The release was withdrawn due to a corruption bug. I suggest you downgrade to 0.9.19. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

3 3

Possible to rebind-as-user with chain overlay using TLS client certificates?
by Matthew Kemp 21 Mar '17

21 Mar '17

Hey all, We've got a fairly straightforward producer/consumer setup with a single producer and multiple syncrepl consumers configured with an updateref back to the producer. The consumers are set up to use the chain overlay with the ldap backend for transparent password updates. We use TLS client certificates to authenticate the clients to the consumers and the consumers to the producers with SASL EXTERNAL binds. However, I've been struggling with configuring what I had hoped would be a simple part of the setup. We would like the ability to have the user simple bind as themselves to the consumer and then have the chain overlay transparently follow the updateref back to the producer and rebind as the user there to do the PASSMOD update. For whatever reason, I haven't been successful at making that happen. I've tried several attempts at various back_ldap configurations, to no avail. By playing with the olcDbIdAssertBind and various mode/flags, I can get the backend to attempt ProxyAuth (which we'd prefer not to use if possible), and alternatively I am able to get the chain to anonymously bind, but I am not able to initiate a rebind as the originating user. We're running slapd 2.4.31 under Debian Wheezy on the producer and a variety of 2.4.31 and 2.4.40 slapd consumers under Debian Wheezy and Debian Jesse, respectively. I was hoping you'd be able to shed some light whether this is even possible? Attached are the most recent configuration that I believe should work for my needs. Thanks! --- Current consumer config: dn: olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config objectClass: olcOverlayConfig objectClass: olcChainConfig olcOverlay: {0}chain olcChainReturnError: TRUE structuralObjectClass: olcChainConfig dn: olcDatabase={0}ldap,olcOverlay={0}chain,olcDatabase={-1} frontend,cn=config objectClass: olcLDAPConfig objectClass: olcChainDatabase olcDatabase: {0}ldap structuralObjectClass: olcLDAPConfig olcDbURI: ldap://ldap-primary olcDbRebindAsUser: TRUE olcDbStartTLS: start tls_cert=/etc/ssl/certs/ldap-consumer.crt tls_key=/etc/ssl/private/ldap-consumer.key tls_cacert=/usr/share/ca-certi ficates/cosmos/ldap_producer_ca_pem.crt -- Matt Kemp Production Engineer Braintree

1 0

syncrepl and the frontend database
by Andrey Klimentev 21 Mar '17

21 Mar '17

Hello. I've got a question about replicating the frontend database. Should I send syncprov on it and replicate it with cn=config and other databases? Or is it somehow handled in automated manner?

1 0

slapd-sock as overlay and PASSMOD
by Michael Ströder 20 Mar '17

20 Mar '17

HI! As said I'd like to intercept password changes (clear-text password) via back-sock used as overlay. Especially the client (Mac OS X) sends a Password Modify ext. op. Unfortunately back-sock does not send the PASSMOD itself to the external listener (which could parse the ASN.1) and therefore I have to look at the MODIFY messages. I'm using my package: https://pypi.python.org/pypi/slapdsock Problem: Everything works as expected with this configuration when rootdn is used with ldappasswd but not as normal user. Excerpt of slapd.conf: database mdb suffix "ou=realdb,dc=example,dc=org" rootdn "cn=root,ou=realdb,dc=example,dc=org" [..] overlay sock extensions binddn peername ssf connid socketpath sockoverlay-listener sockops modify Command: ldappasswd -H ldapi:// -s test uid=test1,ou=realdb,dc=example,dc=org slapd's log (LDAPI and SASL/EXTERNAL maps local user to rootdn): 58d03e84 conn=1000 fd=15 ACCEPT from PATH=/home/michael/Proj/slapd_sockd/examples/openldap/ldapi (PATH=/home/michael/Proj/slapd_sockd/examples/openldap/ldapi) 58d03e84 conn=1000 op=0 BIND dn="" method=163 58d03e84 conn=1000 op=0 BIND authcid="gidNumber=100+uidNumber=1000,cn=peercred,cn=external,cn=auth" authzid="gidNumber=100+uidNumber=1000,cn=peercred,cn=external,cn=auth" 58d03e84 conn=1000 op=0 BIND dn="cn=root,ou=realdb,dc=example,dc=org" mech=EXTERNAL sasl_ssf=0 ssf=71 58d03e84 conn=1000 op=0 RESULT tag=97 err=0 text= 58d03e84 conn=1000 op=1 EXT oid=1.3.6.1.4.1.4203.1.11.1 58d03e84 conn=1000 op=1 PASSMOD id="uid=test1,ou=realdb,dc=example,dc=org" new 58d03e84 sock search reading line (CONTINUE ) 58d03e84 conn=1000 op=1 RESULT oid= err=0 text= 58d03e84 conn=1000 op=2 UNBIND 58d03e84 conn=1000 fd=15 closed The log of my external listener: 2017-03-20 21:41:40,620 DEBUG 140544537579816 ----- incoming request via 'openldap/sockoverlay-listener' from pid=28285 uid=1000 gid=100 ----- 2017-03-20 21:41:40,620 DEBUG 140544537579816 request_data='MODIFY\nmsgid: 2\nbinddn: cn=root,ou=realdb,dc=example,dc=org\npeername: PATH=/home/michael/Proj/slapd_sockd/examples/openldap/ldapi\nssf: 71\nconnid: 1000\nsuffix: ou=realdb,dc=example,dc=org\ndn: uid=test1,ou=realdb,dc=example,dc=org\nreplace: userPassword\nuserPassword:: dGVzdA==\n-\n\n' 2017-03-20 21:41:40,620 DEBUG 140544537579816 reqtype='MODIFY' 2017-03-20 21:41:40,620 DEBUG 140544537579816 sock_req=<slapdsock.message.MODIFYRequest object at 0x7fd3133f10d0> // {'dn': 'uid=test1,ou=realdb,dc=example,dc=org', 'binddn': u'cn=root,ou=realdb,dc=example,dc=org', 'connid': 1000, 'suffix': u'ou=realdb,dc=example,dc=org', '_linecount': 7, 'msgid': 2, '_req_lines': ['MODIFY', 'msgid: 2', 'binddn: cn=root,ou=realdb,dc=example,dc=org', 'peername: PATH=/home/michael/Proj/slapd_sockd/examples/openldap/ldapi', 'ssf: 71', 'connid: 1000', 'suffix: ou=realdb,dc=example,dc=org', 'dn: uid=test1,ou=realdb,dc=example,dc=org', 'changetype: modify', 'replace: userPassword', 'userPassword:: dGVzdA==', '-', '', ''], 'peername': u'PATH=/home/michael/Proj/slapd_sockd/examples/openldap/ldapi', 'modops': [(2, 'userPassword', ['test'])], 'reqtype': 'MODIFY', 'ssf': 71} 2017-03-20 21:41:40,620 DEBUG 140544537579816 connid=1000 msgid=2 Request not cached: cache_key=None 2017-03-20 21:41:40,620 DEBUG 140544537579816 connid=1000 msgid=2 response_str='CONTINUE\n' 2017-03-20 21:41:40,620 DEBUG 140544537579816 connid=1000 msgid=2 response_delay=0.001 So far so good. But with this command (own password change) there's no MODIFY sent to the extenal listener at all: $ ldappasswd -H ldapi://%2Fhome%2Fmichael%2FProj%2Fslapd_sockd%2Fexamples%2Fopenldap%2Fldapi -D "uid=test1,ou=realdb,dc=example,dc=org" -w test -s test23 uid=test1,ou=realdb,dc=example,dc=org ldap_parse_extended_result: Bad parameter to an ldap routine (-9) slapd's log: 58d03f05 conn=1003 fd=15 ACCEPT from PATH=/home/michael/Proj/slapd_sockd/examples/openldap/ldapi (PATH=/home/michael/Proj/slapd_sockd/examples/openldap/ldapi) 58d03f05 conn=1003 op=0 BIND dn="uid=test1,ou=realdb,dc=example,dc=org" method=128 58d03f05 conn=1003 op=0 BIND dn="uid=test1,ou=realdb,dc=example,dc=org" mech=SIMPLE ssf=0 58d03f05 conn=1003 op=0 RESULT tag=97 err=0 text= 58d03f05 conn=1003 op=1 EXT oid=1.3.6.1.4.1.4203.1.11.1 58d03f05 conn=1003 op=1 PASSMOD id="uid=test1,ou=realdb,dc=example,dc=org" new 58d03f05 conn=1003 op=1 RESULT tag=103 err=50 text= 58d03f05 conn=1003 op=1 RESULT oid= err=50 text= 58d03f05 conn=1003 op=2 UNBIND 58d03f05 conn=1003 fd=15 closed Any clue what's going on? Ciao, Michael.

1 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical