April 2021 - openldap-technical

Re: performance tuning for n-way and heavy client load

by Zetan Drableg

> >> When I write LDIFs to one node like delete user or remove user from > >> group, we see spikes in authentication latency metrics (what's normally > >> .2 - .5 second response time goes up to 15-30 seconds) across all nodes > >> in the cluster at the same time. > > > > I ran mdb_copy -c to compact the LDAP databases. The size went from > > 2.9G to 140M and the latency problem during inserts went away. > > I've noticed the LDAP data.mdb is growing about 25M per day. What > > accounts for the growth of free pages? > > Do you have a lot of large groups that you frequently update? Yes we have several groups with ~40k users from which we frequently add/remove users based on upstream user provisioning workflows.

2 years, 1 month

2
1
0 / 0

performance tuning for n-way and heavy client load

by Zetan Drableg

openldap 2.4.57 on 16 core OracleLinux VMs with NVME disk. 8 nodes in n-way multi master configuration, MDB backend, 50k unique DNs. We see about 10,000 auths per minute per node. Under heavy client load, the log shows many "deferring operation: binding" messages in the same second. slapd is using only 400% cpu (of 1600 possible). [2021-04-13 19:15:58] connection_input: conn=150474 deferring operation: binding When I write LDIFs to one node like delete user or remove user from group, we see spikes in authentication latency metrics (what's normally .2 - .5 second response time goes up to 15-30 seconds) across all nodes in the cluster at the same time. What knobs can be adjusted to allow for more concurrency? It seems like writes are impacting reads. *slapd.conf: threads* default is 32, tried 64 and 128 with little improvement *slapd.conf: syncrepl* Should I increase sessionlog size? Should I increase checkpoint ops? How to determine optimum values? syncprov-checkpoint 100 5 syncprov-sessionlog 100 syncprov-reloadhint TRUE *mdb* maxsize 17179869184 *Indices* index objectClass eq,pres index cn,uid,mail,mobile eq,pres,sub index o,ou,dc,preferredLanguage eq,pres index member,memberUid eq,pres index uidNumber,gidNumber eq,pres index memberOf eq index entryUUID eq index entryCSN eq index uniqueMember eq index sAMAccountName eq *ulimit* bash-4.2$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 482391 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited *n-way config* serverID 1 ldap://XXXX:12389 syncrepl rid=1 provider=ldap://XXXXXX:12389 bindmethod=simple starttls=yes tls_cert=/opt/slapd/conf/cert.pem tls_cacert=/etc/pki/tls/cert.pem tls_key=/opt/slapd/conf/key.pem binddn="cn=replication_manager,dc=service-accounts,o=Root" credentials="YYYYYY" tls_reqcert=never searchbase="" schemachecking=on type=refreshAndPersist retry="60 +" (and 7 more) mirrormode on Any ideas? Thanks -Zetan

2 years, 1 month

3
5
0 / 0

Timeout values in search_ext(), ldap_result() and global

by varun mittal

I am using openldap-2.4.39 on CentOS 7, to query my AD server, with python-ldap wrapper I set the following scheme: ldap.set_option(ldap.OPT_NETWORK_TIMEOUT, 30) ldap.set_option(ldap.OPT_TIMEOUT, 120) conn = ldap.initialize(ldap://server-ip) Using 3 types of queries - synchronous search_s(), asynchronous with and without paging search_ext() I am not using any timeout in the _ext method or the result3() methods One of my python client LDAP searches(asynchronous with paging) took about 14 minutes to complete, in the customer environment. Eventually, the search was successful. Looking at the documentation, I am not sure which timeout value would be applicable here. I thought setting OPT_TIMEOUT should suffice for all kinds of searches. And the strange thing is that the similar query, but synchronous( ldap_search_ext_s) from my C client failed within 120 seconds. This is the default AD server timelimit. The C application didn't specify any timeouts What am I missing here?

2 years, 1 month

3
5
0 / 0

Problems setting up a proxy

by Hans van Zijst

Hi, After more than a day of fiddling with it, I turn to you, the gurus ;) I'm trying to create an OpenLDAP proxy that will talk to 2 OpenLDAP servers, doing MirrorMode replication and using a floating IP so that I can point all write queries to one and the same server. Those 2 MirrorMode servers are up and running and doing fine, but I can't figure out how to make that proxy. I'm running on Debian Bullseye (still "testing" at this moment), with OpenLDAP 2.4.57, both on the backend servers and the proxy I'm trying to make. I'm not using TLS yet, that's for later. After installation, there's an (empty, of course) mdb database. I think I should throw that away, but I'm not sure. The suffix in that database is different than the one I need to proxy, so it's probably not a problem to leave it there. I have loaded the extra schemas that I use on the MirrorMode machines, and loaded the backends ldap and meta, with LDIF files like this: dn: cn=module{0},cn=config changetype: modify add: olcModuleLoad olcModuleLoad: back_ldap.la And fed that to slapd with ldapmodify -Y EXTERNAL -h ldapi:/// -f <file> I checked with ldapvi and saw both modules loaded. So far, so good. Now I need to create the backend, and this is where I keep running into problems. Although the use of slapd.conf has fallen from grace a long time ago, every example I can find online only uses that. So I tried creating one and adding it to the configuration with slaptest. This is what I came up with: backend meta database meta suffix "dc=example,dc=com" rootdn "cn=admin,dc=example,dc=com" rootpw "super secret passwd" uri "ldap://172.16.7.6/dc=example,dc=com" readonly yes acl-authcDN "cn=admin,dc=example,dc=com" acl-passwd "super secret passwd" uri "ldap://172.16.7.7/dc=example,dc=com" readonly yes acl-authcDN "cn=admin,dc=example,dc=com" acl-passwd "super secret passwd" uri "ldap://172.16.7.8/dc=example,dc=com" readonly no acl-authcDN "cn=admin,dc=example,dc=com" acl-passwd "super secret passwd" But when I try to convert that, I get an error: # slaptest -f /root/proxybackend.conf -F /etc/ldap/slapd.d 6075bced /root/proxybackend.conf: line 1: <backend> failed init (meta)! slaptest: bad configuration directory! The information in the OpenLDAP Handbook is, well, lacking: https://openldap.org/doc/admin24/backends.html#Metadirectory I had hoped to find a way to create an LDIF file which I could add with ldapadd, but I never came much further than this: dn: olcDatabase=meta objectClass: olcDatabaseConfig objectClass: olcMetaConfig olcDatabase: meta olcSuffix: dc=example,dc=com olcRootDN: cn=admin,dc=example,dc=com olcRootPW: "super secret passwd" which results in: adding new entry "olcDatabase=meta" ldap_add: Server is unwilling to perform (53) additional info: no global superior knowledge I'm pretty sure I need more lines in that, to begin with the URI lines to point the proxy to the machines it needs to contact, but I couldn't find the olcSomeThing syntax for them. I'm pretty good at searching, but not so good at finding, unfortunately. Can somebody give me a few hints please? I'm pretty sure I'm missing something small here, but I'm stuck. Kind regards, Hans

2 years, 1 month

2
3
0 / 0

mdb_substring_candidates: (cn) not indexed

by Клеусов Владимир Сергеевич

Hi, In logs mdb_substring_candidates: (cn) not indexed But slapcat -b cn=config | grep olcDbIndex olcDbIndex: cn eq Please tell me what this message is about ?

2 years, 1 month

3
3
0 / 0

Getting MDB_MAP_RESIZED error, followed by slapd (appearing to) freeze

by Leonardo Bruno Lopes

Hello everyone. I could find (very) few results while searching the internet for MDB_MAP_RESIZED error in OpenLDAP, and they were of no help. I'll describe my setup and the circumstances of the error. I have OpenLDAP (version 2.4.47) and LMDB (version 0.9.22) installed from Debian 10 default packages in a amd64 box. Along with some usual settings (my slapd.conf file is attached), I set the main database up using the mdb backend with maxsize = 7516192768 (7GB). The on-disk base size (the data.mdb file) is 134MB. I also have loglevel = stats. Everything seems to work flawless and fast, so all of a sudden syslog prints this message: Apr 14 09:47:56 my-ldap-box slapd[20736]: mdb_opinfo_get: err MDB_MAP_RESIZED: Database contents grew beyond environment mapsize(-30785) and seconds later, the slapd daemon stop to answer requests, although the service appears to be still running. This has happened 7 times since April 7, when I moved the data from an old server to this one. I changed loglevel to 'stats trace args shell' and I realized that when the error occurs, there were always some ADD operations logged right before. I inspected the logs today, April 14, right after the occurrence of the error and attached a file containing: 1) the error message, which appears at 09:47:56, 2) the messages logged before, since 09:47:50, 3) the messages logged after, until 09:48:01. This time on, slapd logged no more messages and appeared to be stuck for the clients. At 09:53:41 I restarted the daemon. The log generated from this time until the first request answered is also attached. For the record, I initially suspected of the replica server which was running alongside the main server and which showed the same behavior. This was because at then I associated the error with syncrepl operations. But even after I stop the replica, the error keeps ocurring in the main server. Any clues? Thanks for your consideration. -- Atenciosamente, Leonardo Lopes INFRA-TI / DTI -- Esta mensagem foi verificada pelo sistema de antiv��rus e acredita-se estar livre de perigo.

2 years, 1 month

4
14
0 / 0

be_modify failed(16)

by Rallavagu Kon

Hello All, OpenLDAP server 2.4.49 Setup with replication Noticing following error messages in server logs, Apr 14 19:56:19 openldap-service-0 slapd[51]: syncrepl_null_callback : error code 0x10 Apr 14 19:56:19 openldap-service-0 slapd[51]: syncrepl_entry: rid=103 be_modify failed (16) As the error code “16” suggested, validated the schema across participating nodes and found no discrepancies in the schema configuration across nodes. However, we are noticing that one of the entries is missing from one node out of three participating replicas (multi master). Any pointers on where to start to debug to understand the issue and fix it. Thanks in advance.

2 years, 1 month

2
3
0 / 0

replication of cn=config

by Stefan Kania

Hello, I just try to set up the replication for cn=config using the example from the documentation: https://www.openldap.org/doc/admin24/replication.html I have 3 Provider in an MMR with delta-syncrpl. Delta-syncrepl is working fine I can add and change objects from all three providers. Then I set up the replication for cn=config. Here are the changes I made on all three providers: ----------- olcServerID: 1 ldap://hm-01.example.net olcServerID: 2 ldap://hm-02.example.net olcServerID: 3 ldap://hm-03.example.net ... # {0}config, config dn: olcDatabase={0}config,cn=config objectClass: olcDatabaseConfig olcDatabase: {0}config olcAccess: {0}to * by dn.exact=gidNumber=0+uidNumber=0,cn=peercred,cn=external ,cn=auth manage by * break olcRootDN: cn=admin,cn=config olcRootPW: {SSHA}Uyelb0kyqWuEqr4QmfvrpDaD7VYjeU8h olcSyncrepl: {0}rid=001 provider=ldap://hm-01.example.net binddn="cn=admin,cn= config" bindmethod=simple credentials=geheim searchbase="cn=config" type=ref reshAndPersist retry="5 5 300 5" timeout=1 starttls=yes olcSyncrepl: {1}rid=002 provider=ldap://hm-02.example.net binddn="cn=admin,cn= config" bindmethod=simple credentials=geheim searchbase="cn=config" type=refr eshAndPersist retry="5 5 300 5" timeout=1 starttls=yes olcSyncrepl: {2}rid=003 provider=ldap://hm-03.example.net binddn="cn=admin,cn= config" bindmethod=simple credentials=geheim searchbase="cn=config" type=refr eshAndPersist retry="5 5 300 5" timeout=1 starttls=yes olcMirrorMode: TRUE # {0}syncprov, {0}config, config dn: olcOverlay={0}syncprov,olcDatabase={0}config,cn=config objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: {0}syncprov ---------------- It's same on all three providers. When I do a change on the first provider (hm-01) the change will be replicat to hm-02 and hm-03. But if I try to change or add an ACL on one either hm-02 or hm-03 the change will not be replicated to the other providers. I'm using Debian 10 with the OpenLDAP-packages from debian-backport version 2.4.57. Before setting up the replication for cn=config I checked that the configuration on all three providers are the same. So what did I do wrong or did I miss something Stefan

2 years, 1 month

3
4
0 / 0

Enable support for TLS 1.2 and 1.3, and disable support for TLS 1.0.

by CLARKE, ED C

Hello, I am having trouble disabling TLS1.0 on my OpenLdap and enabling TLS 1.2 & 1.3, below are the scan results: * Enable support for TLS 1.2 and 1.3, and disable support for TLS 1.0. * "Consult the application's documentation to disable SSL 2.0 and 3.0. * Use TLS 1.2 (with approved cipher suites) or higher instead." * "Ports found: 389 * TLSv1 is enabled and the server supports at least one cipher." * Info for my LDAP * $ rpm -qa | grep ldap * openldap-clients-2.4.44-21.el7_6.s390x * sssd-ldap-1.16.2-13.el7_6.12.s390x * openldap-2.4.44-21.el7_6.s390x * openldap-servers-2.4.44-21.el7_6.s390x Any help in this matter would be greatly appreciated. Thanks, Ed Ed Clarke Senior Software Engineer Operations Transformation, Real-time Automation & Predictive Insights<http://mysolutions.dev.att.com/GNFO_Solutions/index.jsp> "RAPID" Certified Quality Eng. - ISO 9000/1 Six Sigma - Yellow Belt AT&T Veterans AT&T "ATO" 1010 Pine ST. Shared, St. Louis, MO. 63101 m 636.639.0713 | o 314.335.3158 | ec4397(a)att.com<mailto:ec4397@att.com>

2 years, 1 month

2
2
0 / 0

N-Way + Delta Sync Expected Behavior

by thomaswilliampritchard＠gmail.com

Hi Friends, I have setup N-Way multi provider with 3 providers all using delta sync as suggested to me previously on this forum. I'd like to report that I am seeing more expected results this time around when loading from a backup database and turning on replication so thank you for the suggestions. I would like to describe the behavior I am seeing in hopes to confirm this is expected and proper behavior in 3-Way Multi Master provider persistent delta sync configuration. I have provider1, provider2, provider3 all configured to replicate from each other with refresh and persist delta sync repl. When I issue a write to provider 1 I see it send the update to provider2 and provider3. I then see provider2 and provider3 "syncprov_qresp: set up a new syncres mode=1" for each consumer that is connected to them (including the other provider that did not send the write). Provider 2 then sends a message to provider 3 that ignores it because "CSN too old, ignoring 20210410001522.122666Z#000000#065#000000 (reqStart=20210410001522.000001Z,cn=accesslog)". At the same time provider 3 sends a message to Provider 2 that gets ignored in the same way. Since it is expected that all consumers know about all providers it is perplexing to me why the providers "forward" the update along to its own known consumers. It seems this could be expected given my amateur ldap knowledge but I wanted to confirm because I don't immediately understand why that behavior is necessary (results in ignored messages). Thank you for your consideration Thomas

2 years, 1 month

2
1
0 / 0

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical April 2021