openldap-technical

openldap-technical@openldap.org

22 participants
9789 discussions

Re: mdb fragmentation
by Quanah Gibson-Mount 25 Aug '17

25 Aug '17

Hi Geert, If I could, I would delete 8664 from the ITS system entirely as it was filed based on invalid information that was provided to me. It generally should be ignored. When a write operation is performed with LMDB, the freelist is scanned for available space to reuse if possible. The larger the size of the freelist, the longer amount of time it will take for the operation to complete successfully. When the database has gotten to a certain point of fragmentation (This differs based on any individual use case), it will be start taking a noticeable amount of time for those write operations to complete and the server processing the write operation does essentially come to a halt during this process. Once the write operation completes, things go back to normal. The only solution is to dump and reload the database (slapcat/slapadd or mdb_copy -c). Eventually, you will get back into the same situation and have to do this again. A recent option was added to the slapd-mdb configuration (rtxnsize) that can also help reduce the rate of fragmentation. There are some performance related issues you can find discussed on the -devel list from when it was added. Whether or not you are affected by them and whether or not the setting will help you in particular depends on whether or not your searches result in a large number of entries being returned. You can find some guidelines around tuning the parameter that I came up with in that thread. If you do not have an unlimited Zimbra License, the license check performed by the store servers will definitely affect this, since the result set is all active accounts which can be quite large. Additionally, I had at one point had a patch for the Zimbra build of OpenLDAP that made it very aggressive in finding freespace to reuse. I don't recall if it is still applied (I don't believe it currently is based on what I saw in github). It basically meant that in Zimbra, it would work extra hard to find reusable freespace, which would reduce the rate at which the database would fragment, but it also meant that once the DB was fragmented enough, it would amplify the amount of time it took for a write op to complete. I.e., it was a tradeoff of a longer time to reach a catastrophic state, but the state was more catastrophic once achieved. This is one area where LMDB differs significantly from back-hdb/bdb. You could have back-bdb/hdb databases that endured a high rate of write operations be in effect for years w/o needing maintenance. With LMDB, you get better read & write rates, but it requires periodic reloads. Hope this helps! --Quanah ----- Original Message ----- > From: "Geert Hendrickx" <geert(a)hendrickx.be> > To: "openldap-technical" <openldap-technical(a)openldap.org> > Sent: Thursday, August 24, 2017 4:53:32 AM > Subject: mdb fragmentation > Hi > > We have an OpenLDAP 2.4.44 based, 4-way MMR setup with 4 M entries, > which is fairly write intensive (Zimbra). > > Lately we've seen very frequent lockups of the master that receives > the updates (only 1 out of 4), whereas the replicas stay responsive. > According to -d stats logs, all threads suddenly take a long time to > answer any queries, and slapd can no longer accept new connections. > The issue always disappears again without intervention, but usually > hits a number of times in a row, on an almost daily basis. > > We tested a lot of things, but eventually "solved" the issue with a > slapcat and slapadd of the database - the master server has been > completely stable again since. The mdb was also reduced 50% in size. > > Looking at the old mdb (prior to the dump), mdb_stat -f shows it had > over 3.7 M free pages. Could it be an issue of database fragmentation > similar to ITS#8664? > > Is it natural that the freelist (and thus the mdb) gets this big over > time, I would expect those free pages to get reused constantly? > And in that case would it make sense to monitor the number of free > pages? Is there a threshold to look for, before things get problematic > again? (ITS#7770 would come handy here, as we already monitor/graph > various metrics from the monitor backend) > > > Geert > > > -- > geert.hendrickx.be :: geert(a)hendrickx.be :: PGP: 0xC4BB9E9F > This e-mail was composed using 100% recycled spam messages! -- -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

3 2

Re: lock.mdb taking lots of space
by Howard Chu 24 Aug '17

24 Aug '17

Muhammed Muneer wrote: > Hi, > > Is there any particular reason why my lock.mdb is taking above 600 MB. The > data.mdb size is around 4.5 GB. The lock.mdb size is proportional to the maxreaders config setting. > Is this normal? Depending on what you set maxreaders to, probably. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

lock.mdb taking lots of space
by Muhammed Muneer 24 Aug '17

24 Aug '17

Hi, Is there any particular reason why my lock.mdb is taking above 600 MB. The data.mdb size is around 4.5 GB. Is this normal?

1 0

OpenLDAP as Proxy
by Palacios, Christian 24 Aug '17

24 Aug '17

Hi there, We need to find out if OpenLDAP will allow us to use it as a proxy so it can retrieve users from three different Windows Active Directory Domains? These three domains do not have any similar users. The user retrieval process needs to work like this: - The application that needs this LDAP connection will point to the OpenLDAP server using an LDAP address such as ldap://server.example.com:389/OU=users...etc - This application will also need to retrieve the sAMAccountName from each user retrieved via the OpenLDAP server - The application's LDAP connection settings also need to specify an Administrator's DN and password, but I'm confused about this because I don't know what Administrator account to use. Like I said, each domain has their own set of users so they don't have any Administrator accounts in common. How would this work? If you need any more information, please let me know!! Thanks, - Christian This email and any accompanying attachments are confidential. If you received this email by mistake, please delete it from your system. Any review, disclosure, copying, distribution, or use of the email by others is strictly prohibited.

2 1

LDAPCon 2017 programme now online
by Andrew Findlay 24 Aug '17

24 Aug '17

The programme for the 2017 LDAP Conference has just been published: https://ldapcon.org/2017/conference-program/ It's looking good, so get your booking in quickly to get early-bird tickets and start thinking about where you want to stay in Brussels! The conference runs 19th and 20th October 2017. Andrew -- ----------------------------------------------------------------------- | From Andrew Findlay, Skills 1st Ltd | | Consultant in large-scale systems, networks, and directory services | | http://www.skills-1st.co.uk/ +44 1628 782565 | -----------------------------------------------------------------------

2 1

mdb fragmentation
by Geert Hendrickx 24 Aug '17

24 Aug '17

Hi We have an OpenLDAP 2.4.44 based, 4-way MMR setup with 4 M entries, which is fairly write intensive (Zimbra). Lately we've seen very frequent lockups of the master that receives the updates (only 1 out of 4), whereas the replicas stay responsive. According to -d stats logs, all threads suddenly take a long time to answer any queries, and slapd can no longer accept new connections. The issue always disappears again without intervention, but usually hits a number of times in a row, on an almost daily basis. We tested a lot of things, but eventually "solved" the issue with a slapcat and slapadd of the database - the master server has been completely stable again since. The mdb was also reduced 50% in size. Looking at the old mdb (prior to the dump), mdb_stat -f shows it had over 3.7 M free pages. Could it be an issue of database fragmentation similar to ITS#8664? Is it natural that the freelist (and thus the mdb) gets this big over time, I would expect those free pages to get reused constantly? And in that case would it make sense to monitor the number of free pages? Is there a threshold to look for, before things get problematic again? (ITS#7770 would come handy here, as we already monitor/graph various metrics from the monitor backend) Geert -- geert.hendrickx.be :: geert(a)hendrickx.be :: PGP: 0xC4BB9E9F This e-mail was composed using 100% recycled spam messages!

1 0

delta syncrepl errors
by Gerard Ranke 22 Aug '17

22 Aug '17

Hello all, I'm trying to set up delta-syncrepl on a test setup, consisting of a producer plus consumer running openldap 2.4.45. ( I'll put relevant parts of both producer and consumer config at the end of this post. ) I'm following the instructions from the openldap manual ( chapter 18.3.2 ) as much as possible. Both slapd's start up but replication isn't happening. I get the following messages in the logs: producer: 2017-08-18T14:54:39.153754+02:00 pro slapd[10098]: send_search_entry: conn 1278 ber write failed. 2017-08-18T14:54:44.157627+02:00 pro slapd[10098]: send_search_entry: conn 1279 ber write failed. 2017-08-18T14:54:49.368758+02:00 pro slapd[10098]: send_search_entry: conn 1281 ber write failed. 2017-08-18T14:55:09.390341+02:00 pro slapd[10098]: send_search_entry: conn 1283 ber write failed. 2017-08-18T14:55:19.395091+02:00 pro slapd[10098]: send_search_entry: conn 1284 ber write failed. consumer: 2017-08-18T14:54:39.153660+02:00 del slapd[25530]: do_syncrep2: rid=001 got search entry without Sync State control (reqStart=20170815125023.000001Z,cn=accesslog) 2017-08-18T14:54:39.154089+02:00 del slapd[25530]: do_syncrepl: rid=001 rc -1 retrying (1 retries left) 2017-08-18T14:54:44.157539+02:00 del slapd[25530]: do_syncrep2: rid=001 got search entry without Sync State control (reqStart=20170815125023.000001Z,cn=accesslog) 2017-08-18T14:54:44.158156+02:00 del slapd[25530]: do_syncrepl: rid=001 rc -1 retrying 2017-08-18T14:54:49.368843+02:00 del slapd[25530]: do_syncrep2: rid=001 got search entry without Sync State control (reqStart=20170815125023.000001Z,cn=accesslog) 2017-08-18T14:54:49.369446+02:00 del slapd[25530]: do_syncrepl: rid=001 rc -1 retrying 2017-08-18T14:54:59.383750+02:00 del slapd[25530]: do_syncrep2: rid=001 got search entry without Sync State control (reqStart=20170815125023.000001Z,cn=accesslog) 2017-08-18T14:54:59.384369+02:00 del slapd[25530]: do_syncrepl: rid=001 rc -1 retrying 2017-08-18T14:55:09.390382+02:00 del slapd[25530]: do_syncrep2: rid=001 got search entry without Sync State control (reqStart=20170815125023.000001Z,cn=accesslog) 2017-08-18T14:55:09.390971+02:00 del slapd[25530]: do_syncrepl: rid=001 rc -1 retrying 2017-08-18T14:55:19.395206+02:00 del slapd[25530]: do_syncrep2: rid=001 got search entry without Sync State control (reqStart=20170815125023.000001Z,cn=accesslog) When I do a search on the producer from the consumer I get results that look like I would expect to see: ldapsearch -x -h pro.hku.nl -b "cn=accesslog" -D cn=dsyncuser,dc=hku,dc=nl -w ****** "(&(objectClass=auditWriteObject)(reqResult=0))" (...) # 20170817170609.000001Z, accesslog dn: reqStart=20170817170609.000001Z,cn=accesslog objectClass: auditAdd reqStart: 20170817170609.000001Z reqEnd: 20170817170610.000000Z reqType: add reqSession: 1 reqAuthzID: cn=root,dc=hku,dc=nl reqDN: nlHkuID=77454,ou=People,dc=hku,dc=nl reqResult: 0 reqMod: objectClass:+ top reqMod: objectClass:+ posixAccount reqMod: objectClass:+ shadowAccount reqMod: objectClass:+ inetOrgPerson reqMod: objectClass:+ nlHkuPerson reqMod: objectClass:+ eduPerson reqMod: objectClass:+ apple-user reqMod: objectClass:+ sambaSamAccount reqMod: ou:+ People reqMod: authAuthority:+ ;basic; reqMod: nlHkuID:+ 77454 reqMod: loginShell:+ /bin/false reqMod: gidNumber:+ 300 (...) reqMod: structuralObjectClass:+ nlHkuPerson reqMod: entryUUID:+ 1b7405d0-17ba-1037-98e3-99bbae3c2a53 reqMod: creatorsName:+ cn=root,dc=hku,dc=nl reqMod: createTimestamp:+ 20170817170608Z reqMod: entryCSN:+ 20170817170608.603041Z#000000#000#000000 reqMod: modifiersName:+ cn=root,dc=hku,dc=nl reqMod: modifyTimestamp:+ 20170817170608Z reqEntryUUID: 1b7405d0-17ba-1037-98e3-99bbae3c2a53 (...) The producer config has: (...) dn: cn=module{0},cn=config objectClass: olcModuleList cn: module{0} olcModulePath: /usr/lib64/openldap olcModuleLoad: {0}back_mdb.so olcModuleLoad: {1}dynlist.so olcModuleLoad: {2}accesslog.so olcModuleLoad: {3}syncprov.so olcModuleLoad: {4}smbk5pwd.so structuralObjectClass: olcModuleList (...) dn: olcDatabase={1}mdb,cn=config objectClass: olcDatabaseConfig objectClass: olcMdbConfig olcDatabase: {1}mdb olcDbDirectory: /var/lib/ldap olcSuffix: dc=hku,dc=nl olcAccess: {0}to * by dn.base="cn=dsyncuser,dc=hku,dc=nl" read by * break (...) olcAccess: {15}to * by * read olcAddContentAcl: FALSE olcLastMod: TRUE olcLimits: {0}dn.exact="cn=root,dc=hku,dc=nl" size=unlimited time=unlimited olcLimits: {1}dn.exact="cn=dsyncuser,dc=hku,dc=nl" size=unlimited time=unlim ited olcMaxDerefDepth: 15 olcReadOnly: FALSE olcRootDN: cn=root,dc=hku,dc=nl olcRootPW:: *** olcSyncUseSubentry: FALSE olcMonitoring: TRUE (...) dn: olcOverlay={1}accesslog,olcDatabase={1}mdb,cn=config objectClass: olcAccessLogConfig objectClass: olcOverlayConfig olcOverlay: {1}accesslog olcAccessLogDB: cn=accesslog olcAccessLogOps: writes olcAccessLogPurge: 07+00:00 01+00:00 olcAccessLogSuccess: TRUE (...) dn: olcOverlay={2}syncprov,olcDatabase={1}mdb,cn=config objectClass: olcSyncProvConfig objectClass: olcOverlayConfig objectClass: olcConfig objectClass: top olcOverlay: {2}syncprov olcSpCheckpoint: 100 10 olcSpSessionlog: 100 olcSpNoPresent: TRUE olcSpReloadHint: TRUE structuralObjectClass: olcSyncProvConfig (...) dn: olcDatabase={2}mdb,cn=config objectClass: olcDatabaseConfig objectClass: olcMdbConfig olcDatabase: {2}mdb olcDbDirectory: /var/lib/ldap-accesslog olcSuffix: cn=accesslog olcAccess: {0}to * by dn.base="cn=dsyncuser,dc=hku,dc=nl" read olcLimits: {0}dn.exact="cn=dsyncuser,dc=hku,dc=nl" size=unlimited time=unlim ited olcRootDN: cn=accesslog olcDbIndex: default eq olcDbIndex: entryCSN,objectClass,reqEnd,reqResult,reqStart olcDbMaxSize: 1047483648 structuralObjectClass: olcMdbConfig (...) Parts from the consumer config: (...) dn: cn=module{0},cn=config objectClass: olcModuleList cn: module{0} olcModulePath: /usr/lib64/openldap olcModuleLoad: {0}back_mdb.so olcModuleLoad: {1}dynlist.so structuralObjectClass: olcModuleList (...) dn: olcDatabase={1}mdb,cn=config objectClass: olcDatabaseConfig objectClass: olcMdbConfig olcDatabase: {1}mdb olcDbDirectory: /var/lib/ldap olcSuffix: dc=hku,dc=nl olcAccess: {0}to * by dn.base="cn=dsyncuser,dc=hku,dc=nl" read by * break (...) olcAccess: {15}to * by * read olcAddContentAcl: FALSE olcLastMod: TRUE olcLimits: {0}dn.exact="cn=root,dc=hku,dc=nl" size=unlimited time=unlimited olcLimits: {1}dn.exact="cn=dsyncuser,dc=hku,dc=nl" size=unlimited time=unlim ited olcMaxDerefDepth: 15 olcReadOnly: FALSE olcRootDN: cn=replroot,dc=hku,dc=nl olcRootPW:: ***** olcSyncUseSubentry: FALSE olcSyncrepl: {0}rid=001 provider=ldap://pro.hku.nl bindmethod=simple binddn= "cn=dsyncuser,dc=hku,dc=nl" credentials="****" tls_cert="/etc/ssl/certs /del_cert.pem" tls_key="/etc/ssl/private/del_key.pem" tls_cacertdir="/etc/s sl/certs" tls_reqcert=demand tls_crlcheck=none logbase="cn=accesslog" logfi lter="(&(objectClass=auditWriteObject)(reqResult=0))" syncdata=accesslog se archbase="dc=hku,dc=nl" schemachecking=on type=refreshAndPersist retry="5 5 10 +" olcMonitoring: TRUE olcDbIndex: default pres,eq olcDbIndex: objectClass eq olcDbIndex: entryUUID eq olcDbIndex: entryCSN eq olcDbIndex: cn eq,sub (...) olcDbMaxReaders: 0 olcDbMaxSize: 12147483648 olcDbMode: 0600 olcDbSearchStack: 16 structuralObjectClass: olcMdbConfig Thank you for taking the time to read all this, any remarks on how to get this going will be very much appreciated! Best regards, gerard

1 0

How to enable memberOf overlay with posixGroup?
by MegaBrutal 16 Aug '17

16 Aug '17

Hi all, I've spent days trying to figure out how could I enable the memberOf overlay, and it doesn't seem to be easy for an LDAP-noob. I've read like 50+ tutorials which didn't help me get it working. Use case: I want to have 2 main groups which control access to different services on my network. A "unixusers" which is a minimum to log in to Linux servers (having a hostObject entry for the user is another requirement, which is irrelevant to this question, as I already solved that problem); and a "cloudusers" group which enables log in to my ownCloud instance. The groups should enforce the following rules: – Only users in "cloudusers" should be allowed to log in to ownCloud. – Users in "unixusers" are allowed to log in to a set of Linux servers controlled by "host" (hostObject) entries. – Users not in the "unixusers" group may not log in to any Linux systems, even if they have "host" entries. Problems: – ownCloud complains that the memberOf overlay is not enabled, hence it doesn't let me restrict access to the "cloudusers" group. It would allow any users regardless of any group memberships, which is not acceptable. – I have a similar problem on Linux with PAM: I can't really get it to consider "unixusers" membership for user logins, although I got the "host" entries working correctly, so at least I can already restrict access with that. My guess is that it all boils down to the lack of memberOf overlay. I also figured that memberOf would need groupOfNames groups, while I need posixGroup type groups. I evaluated the possibility to use groupOfNames, but it lacks the necessary gidNumber attribute which is a requirement for Unix groups. But anyway, I can't enable memberOf even for groupOfNames. I can't enable memberOf by any means. My OpenLDAP uses the new configuration method and it completely ignores slapd.conf, so the config must be injected with ldapadd to cn=config. Could you please help me with this? Regards, MegaBrutal

5 6

Re: syncprov
by Quanah Gibson-Mount 15 Aug '17

15 Aug '17

--On Tuesday, August 15, 2017 3:39 PM +0100 Philip Colmer <philip.colmer(a)linaro.org> wrote: > If I am correct and there only needs to be one, what is the best way > to remove the extraneous ones? You'll have to stop the server, slapcat the cn=config DB, delete the extra overlays, and then slapadd the cn=config DB back in. I would suggest adding logic to your process that checks for the existence of the syncprov overlay prior to attempting to add it to the stack. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

syncprov
by Philip Colmer 15 Aug '17

15 Aug '17

I'm adding a third server to an existing multi-server configuration and I think the LDIF files I'm using to configure the replication have some errors in them. dn: olcOverlay=syncprov,olcDatabase={1}mdb,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov The above is a snippet from the LDIF file and I'm concerned that it keeps on adding a new syncprov overlay when there only needs to be one. On the current three-server configuration, there are four instances of syncprov! If I am correct and there only needs to be one, what is the best way to remove the extraneous ones? Thanks. Philip

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical