December 2018 - openldap-technical

by Jean-Francois Malouin

Hi, I have those errors BER write errors that have crept up recently on 2 slapd providers in a mirror replication setup and I can't seem to find the cause of them. I must have made a change to the config but pea brain of me can't seem to find the root of it. They are now occurring a few times a day. I have turned on BER, conns, stats, and sync debugging but it's still unclear to me what is going on. Can anyone shed some light? IP=172.16.10.247 is a HAproxy server with the 2 mirrors as backends. There is also a Munin host that polls the providers every 5m or so to gather stats. Thanks and Happy Holiday! jf Here a anonymized excerpt of the logs of one server showing the error message: Dec 26 03:20:45 ldap-server slapd[4306]: daemon: listen=8, new connection on 18 Dec 26 03:20:45 ldap-server slapd[4306]: daemon: added 18r (active) listener=(nil) Dec 26 03:20:45 ldap-server slapd[4306]: conn=2799347 fd=18 ACCEPT from IP=172.16.10.247:33746 (IP=0.0.0.0:389) Dec 26 03:20:45 ldap-server slapd[4306]: daemon: activity on 2 descriptors Dec 26 03:20:45 ldap-server slapd[4306]: daemon: activity on: Dec 26 03:20:45 ldap-server slapd[4306]: 18r Dec 26 03:20:45 ldap-server slapd[4306]: Dec 26 03:20:45 ldap-server slapd[4306]: daemon: read active on 18 Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=8 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=9 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=10 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: conn=2799347 op=0 BIND dn="" method=128 Dec 26 03:20:45 ldap-server slapd[4306]: conn=2799347 op=0 RESULT tag=97 err=0 text= Dec 26 03:20:45 ldap-server slapd[4306]: daemon: activity on 1 descriptor Dec 26 03:20:45 ldap-server slapd[4306]: daemon: activity on: Dec 26 03:20:45 ldap-server slapd[4306]: Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=8 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=9 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=10 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: activity on 1 descriptor Dec 26 03:20:45 ldap-server slapd[4306]: daemon: activity on: Dec 26 03:20:45 ldap-server slapd[4306]: 18r Dec 26 03:20:45 ldap-server slapd[4306]: Dec 26 03:20:45 ldap-server slapd[4306]: daemon: read active on 18 Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=8 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=9 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=10 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: activity on 1 descriptor Dec 26 03:20:45 ldap-server slapd[4306]: daemon: activity on: Dec 26 03:20:45 ldap-server slapd[4306]: Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=8 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=9 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=10 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: activity on 1 descriptor Dec 26 03:20:45 ldap-server slapd[4306]: daemon: activity on: Dec 26 03:20:45 ldap-server slapd[4306]: Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=8 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=9 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=10 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: connection_read(18): input error=-2 id=2799347, closing. Dec 26 03:20:45 ldap-server slapd[4306]: connection_closing: readying conn=2799347 sd=18 for close Dec 26 03:20:45 ldap-server slapd[4306]: daemon: removing 18 Dec 26 03:20:45 ldap-server slapd[4306]: conn=2799347 fd=18 closed (connection lost) Dec 26 03:20:45 ldap-server slapd[4306]: daemon: activity on 1 descriptor Dec 26 03:20:45 ldap-server slapd[4306]: daemon: activity on: Dec 26 03:20:45 ldap-server slapd[4306]: Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=8 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=9 active_threads=0 tvp=zero Dec 26 03:20:45 ldap-server slapd[4306]: daemon: epoll: listen=10 active_threads=0 tvp=zero Dec 26 03:20:46 ldap-server slapd[4306]: daemon: activity on 1 descriptor Dec 26 03:20:46 ldap-server slapd[4306]: daemon: activity on: Dec 26 03:20:46 ldap-server slapd[4306]: 15r Dec 26 03:20:46 ldap-server slapd[4306]: Dec 26 03:20:46 ldap-server slapd[4306]: daemon: read active on 15 Dec 26 03:20:46 ldap-server slapd[4306]: daemon: epoll: listen=8 active_threads=0 tvp=zero Dec 26 03:20:46 ldap-server slapd[4306]: daemon: epoll: listen=9 active_threads=0 tvp=zero Dec 26 03:20:46 ldap-server slapd[4306]: daemon: epoll: listen=10 active_threads=0 tvp=zero Dec 26 03:20:46 ldap-server slapd[4306]: conn=2799050 op=3 SRCH base="dc=example,dc=com" scope=2 deref=0 filter="(&(objectClass=posixAccount)(uid=root))" Dec 26 03:20:46 ldap-server slapd[4306]: ber_flush2 failed errno=32 reason="Broken pipe" Dec 26 03:20:46 ldap-server slapd[4306]: connection_closing: readying conn=2799050 sd=15 for close Dec 26 03:20:46 ldap-server slapd[4306]: send_ldap_response: ber write failed Dec 26 03:20:46 ldap-server slapd[4306]: conn=2799050 op=3 SEARCH RESULT tag=101 err=0 nentries=0 text= Dec 26 03:20:46 ldap-server slapd[4306]: connection_resched: attempting closing conn=2799050 sd=15 Dec 26 03:20:46 ldap-server slapd[4306]: daemon: removing 15 Dec 26 03:20:46 ldap-server slapd[4306]: conn=2799050 fd=15 closed (connection lost on write)

4 years, 5 months

3
3
0 / 0

ITS#8928 - SOURCE_DATE_EPOCH (was: OpenLDAP 2.4.47 available)

by Michael Ströder

On 12/19/18 5:51 PM, OpenLDAP Project wrote: > Added slapd reproducible build support (ITS#8928) What would be an appropriate value for SOURCE_DATE_EPOCH to be chosen when packaging OpenLDAP? Ciao, Michael.

4 years, 5 months

2
2
0 / 0

RE24 testing call (2.4.47) LMDB RE0.9 testing call (0.9.23)

by Quanah Gibson-Mount

2.4.47 is now believed to be ready for release after tracking down a deadlock and a few other replication issues. Please test. :) Generally, get the code for RE24: <http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs...> Configure & build. Execute the test suite (via make test) after it is built. Optionally, cd tests && make its to run through the regression suite. Thanks! OpenLDAP 2.4.47 Engineering Added slapd-sock DN qualifier for subtrees to be processed (ITS#8051) Added slapd-sock ability to send extended operations to external listeners (ITS#8714) Fixed liblber to avoid incremental access to user-supplied bv in dupbv (ITS#8752) Fixed libldap dn to domain parsing with bad input (ITS#8842) Fixed slapd slapcat to correctly honor -g option (ITS#8667) Fixed slapd to correctly handle NO_SUCH_OBJECT with dynamic groups (ITS#8923) Fixed slapd to check status of rdnNormalize (ITS#8932) Fixed slapd cn=config when modifying slapo-syncprov config (ITS#8616) Fixed slapd sasl authz-policy "all" behavior (ITS#8909) Fixed slapd sasl minor typo (ITS#8918) Fixed slapd to correctly hide hidden DBs in the rootDSE (ITS#8912) Fixed slapd domainScope control to match Microsoft specification (ITS#8840) Fixed slapd-bdb/hdb/mdb to not convert certain IDLs to ranges (ITS#8868) Fixed slapo-accesslog deadlock during cleanup (ITS#8752) Fixed slapo-memberof cn=config modifications (ITS#8663) Fixed slapo-ppolicy with multimaster replication (ITS#8927) Fixed slapo-syncprov with NULL modlist (ITS#8843) Build Environment Fixed missing includes with OpenSSL 1.0.2 (ITS#8809) Contrib Fixed slapo-pbkdf2 hash generation (ITS#8878) Documentation admin24 fixed minor typo (ITS#8887) LMDB 0.9.23 Engineering ITS#8756 Fix loose pages in dirty list ITS#8831 Fix mdb_load flag init ITS#8844 Fix mdb_env_close in forked process Documentation ITS#8857 mdb_cursor_del doesn't invalidate cursor ITS#8908 GET_MULTIPLE etc don't change passed in key Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

4 years, 5 months

5
10
0 / 0

ldap user authentication, PAM and chsh (change shell): how to make it work?

by Jean-Francois Malouin

Hi, Please, bear with me! I know that this is not an openldap question per se, but I've been banging my head on the wall for a long time on this issue and maybe someone knows the quick answer: with user authentication coming from LDAP, what is the magic that has to inserted with the PAM stuff on a client to allow users to change their login shells using 'chsh'? I've been googling this for hours to no avail. I nice hint would just suffice. I'm using libnss-ldap along with pam-ldap on Ubuntu and Debian clients. I managed to make the 'passwd' command to work using the libnss-ldap configuration 'pam_password exop' directive but I'm clueless with chsh... Right now I'm getting messages chsh: user 'luser' does not exist in /etc/passwd and the system auth logs tells me: chsh[4638]: pam_unix(chsh:auth): authentication failure; logname=luser uid=1137 euid=0 tty= ruser= rhost= user=luser /etc/pam.d/chsh originally contained, once the @include included: auth required pam_shells.so auth sufficient pam_rootok.so auth [success=2 default=ignore] pam_unix.so nullok_secure auth [success=1 default=ignore] pam_ldap.so use_first_pass auth requisite pam_deny.so auth required pam_permit.so auth optional pam_cap.so account [success=2 new_authtok_reqd=done default=ignore] pam_unix.so account [success=1 default=ignore] pam_ldap.so account requisite pam_deny.so account required pam_permit.so session [default=1] pam_permit.so session requisite pam_deny.so session required pam_permit.so session optional pam_umask.so session required pam_unix.so session optional pam_ldap.so session optional pam_systemd.so I tried to trim it down -- removing the account and session entries but to no avail so far... thanks, jf

4 years, 5 months

5
11
0 / 0

Antw: Re: fast delete a lot of entries

by Ulrich Windl

>>> Derek Zhou <derek(a)shannon-data.com> schrieb am 14.12.2018 um 07:27 in Nachricht <4351992.5iqGPAtjck@speedy>: > On Thursday, December 13, 2018 05:24:53 PM Derek Zhou wrote: >> Howard and others: >> >> Again on deleting lots of entries. I have 2 experiments: >> >> 1, in a fresh db insert 10 million entries. let's call this state A. then > delete 9 million entries over night. let's call this state B >> 2, in a fresh db insert 1 million entries I call this state B' >> >> In B and B' even the 1 million entries are the same; so from user's > perspective B and B' are indistinguishable. However, deleting entries from B > is much slower than deleting entries from B', like 10x slower. It seems like > deleting speed depends on the peak db size, and how full the db currently is. > >> >> My question is: is this wide a deletion performance gap expected? >> > Further testing shows that this deletion speed drop only happens in writemap > mode. default mode is much better. When you are that deep into analyzing (assuming you use Linux), it wmight be interesting to run blocktrace to see how your filesystem is accessed. (Once I did that for our database server, and you could clearly see accesses to the redo logs as well as the effects of fsync) [...] Regards, Ulrich

4 years, 5 months

1
0
0 / 0

Q: "deferring operation: too many executing" / "deferring operation: pending operations"

by Ulrich Windl

Hi! I have a question for the following log messages: slapd[2215]: connection_input: conn=144871 deferring operation: too many executing slapd[2215]: connection_input: conn=144871 deferring operation: pending operations slapd[2215]: connection_input: conn=144871 deferring operation: pending operations What is "too many", i.e. where is that limit configured? Is it possible to tell how many "pending operations" there are? Finally: It seems slapd "drives against the wall hard", meaning it runs at full speed until the limit is reached, the refuses any more connection/operation. Is it possible to activate (configure) a "soft knee" behavior, where after reaching a specified limit of connections or operations rate slapd starts responding increasingly slower? That would help load balancers to select a different server in time... Regards, Ulrich

4 years, 5 months

3
3
0 / 0

ldap_oc_mappings | keycol of type varchar

by Thangavel, Parameswaran

Hi, While exploring the OpenLDAP's backend-sql package, I found below behavior. The ldap_oc_mappings expects the keycol to be of type integer. For example, if I change the datatype of the identifier (id) of persons table to of type varchar then I am not able to retrieve the persons record during LDAP requests. However, if the column "id" is of type Integer then I am able to view the persons record as part of LDAP search requests. insert into ldap_oc_mappings (id,name,keytbl,keycol,create_proc,delete_proc,expect_return) values (2,'inetOrgPerson','persons','id',NULL,NULL,0); insert into persons (id,name,surname,password) values ('a21qec','Mitya','Kovalev','mit'); Question: 1) Is my observation correct? i.e the keycol used to map the LDAP entries to SQL data expects the primary key (id) to be of type integer. 2) If #1 is correct, do we have a way to support VARCHAR datatype for the primary key? In my application, we use GUID (alpha-numeric characters) as the primary key. Please advise. Thanks Param

4 years, 5 months

2
1
0 / 0

fast delete a lot of entries

by Derek Zhou

dear list, I was experimenting with slapd by adding a lot of entries then deleting them. What I found is adding speed is not bad but deletion speed is lacking. With or without dbsync, delete speed is around half of that of adding. Naively, I thought deleting should be easier than adding, because in adding you actually need to pass and write whole entry of data, while in delete you can just mark the database page as free? The DB file did not shrink after massive deletion; which kind of suggest that deleting is only marking page as free, not really return them to the OS. I am using the latest git tree and the mdb backend. Another related idea is delete a whole branch from the DIT. LDAP is already hierarchical, to delete all entries under a branch, one would assume that there must be an better way than deleting entries with a client side script, like what I am doing? With SQL you can drop a table. With LDAP, can I delete a whole branch? -- Derek Zhou Shannon Systems http://www.shannon-sys.com

4 years, 5 months

3
6
0 / 0

Please help on Openldap issue

by Eggo why

Hi All, I have install openldap, openldap-servers and openldap-clients and systemctl start slapd without anyproblem but when try ti do an initial ldapadd I got error. My slapd.conf and oidbase.ldif as below. Please help and thank you very much Garry slapd.conf # # See slapd.conf(5) for details on configuration options. # This file should NOT be world readable. # include /etc/openldap/schema/core.schema include /etc/openldap/schema/cosine.schema include /etc/openldap/schema/inetorgperson.schema include /etc/openldap/schema/nis.schema include /etc/openldap/oidschema/oidbase.schema include /etc/openldap/oidschema/oidnet.schema include /etc/openldap/oidschema/oidrdbms.schema include /etc/openldap/oidschema/alias.schema # Allow LDAPv2 client connections. This is NOT the default. allow bind_v2 # Do not enable referrals until AFTER you have a working directory # service AND an understanding of referrals. #referral ldap://root.openldap.org pidfile /var/run/openldap/slapd.pid argsfile /var/run/openldap/slapd.args #default searchbase defaultsearchbase "dc=cornell,dc=edu" ####################################################################### # ldbm and/or bdb database definitions ####################################################################### database bdb suffix "dc=cornell,dc=edu" rootdn "cn=admin,dc=cornell,dc=edu" rootpw {SSHA}gZp2hnEA/xq/FfxMgPTjDmSPo78PuZAs # The database directory MUST exist prior to running slapd AND # should only be accessible by the slapd and slap tools. # Mode 700 recommended. directory /var/lib/ldap/cornell # Indices to maintain for this database index objectClass eq,pres index ou,cn,mail,surname,givenname eq,pres,sub index uidNumber,gidNumber,loginShell eq,pres index uid,memberUid eq,pres,sub index nisMapName,nisMapEntry eq,pres,sub ### for root oraclecontext # database bdb suffix "" rootdn "cn=admin" rootpw {SSHA}gZp2hnEA/xq/FfxMgPTjDmSPo78PuZAs # The database directory MUST exist prior to running slapd AND # should only be accessible by the slapd and slap tools. # Mode 700 recommended. directory /var/lib/ldap/rootoraclecontext # Indices to maintain for this database index objectClass eq,pres index ou,cn,mail,surname,givenname eq,pres,sub index uidNumber,gidNumber,loginShell eq,pres index uid,memberUid eq,pres,sub index nisMapName,nisMapEntry eq,pres,sub oidbase.ldif dn: dc=cornell,dc=edu objectClass: top objectClass: dcObject objectClass: organization o: cornell.edu dc: cornell dn: cn=OracleContext,dc=cornell,dc=edu objectclass: orclContext cn: OracleContext l$ ldapadd -vc -x -D “cn=admin,dc=cornell,dc=edu” -w $passwd -f oidbase.ldif ldap_initialize( <DEFAULT> ) ldap_bind: Invalid DN syntax (34) additional info: invalid DN

4 years, 5 months

3
6
0 / 0

speed up parallel add/delete

by Derek Zhou

all I have a patch that can speed up parallel ldap add/delete a lot; in my testing about 2.5X. The idea is described here: the mdb backend does not really support parallelism in rw transactions, so add/delete are serialized by a mutex inside the mdb backend as small transactions. In my patch, all add/delete are put in a queue and executed in a dedicated worker thread. It may seem pointless at first, but by doing them all in the same thread, now I can merge concurrent ops from different clients into larger transactions, and reduce the number of expensive txn_commit calls (fsyncs). my test results are ( your mileage may vary) 10 thread parallel add: 1, unpatched slapd + default config, ~ 2800 op/s (performance stable over time) 2, unpatched slapd + dbnosync ~9000 op/s (lots of fluctuation due to background dirty page flush) 3, unpatched slapd + dpnosync + checkpoint every minute ~8000 op/s, with reduced fluctuation 4, patched slapd ~7300 op/s (fluctuation nearly gone) 2 is not a recommended config, because crashing or power lost will cause massive data lost. 3 is better, but still up to 1 minutes of data can be lost at crash. With this patch, I can achieve >90% of the performance of 3, with data durability as good as 1; or 2.5X performance of 1 with same data durability guaranty. The patch still need some tidy up; is this email list the place to send patches? -- Derek Zhou Shannon Systems http://www.shannon-sys.com

4 years, 5 months

2
1
0 / 0

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical December 2018