openldap-technical November 2015

openldap-technical@openldap.org

64 participants
60 discussions

LMDB overflow pages - MDB_RESERVE
by Christian Sell 27 Nov '15

27 Nov '15

I forgot to mention: I am using MDB_RESERVE to avoid extra memcpy. Could it be that this is the cause for the extreme database bloat? > Hello, > > something wrong with the question below? > > I am trying to use LMDB to store large (huge) amounts of binary data which, > for > the reason of limiting memory footprint, are split into chunks. Each chunk ist > stored under a separate key, made up of [collectionId, chunkId], so that I can > later iterate the chunks using a LMDB cursor. Chunk size is configurable. > > During my tests, I encountered a strange scenario where, after inserting some > 2000 chunks consisting of 512KB each, the database size had grown to a value > that was roughly 135 times the calculated size of the data. I ran the stat > utility over the db and saw that there were > 12000 overflow pages vs. approx. > 2000 data pages. When I reduced the chunk size to 4060 bytes, the number of > overflow pages went down to 1000, and the database size went down to the > expected number (I experimented with different sizes, this was the best > result). > I did not find any documentation to explain this behaviour, or how to deal > with > it. Of course it makes me worry about database bloat and the consequences. Can > anyone shed light on this? > > thanks, > Christian > Christian Sell

1 1

Getting around the single-threaded syncrepl model?
by Bannister, Mark 25 Nov '15

25 Nov '15

I was speaking with Howard in person last week at LDAPCon about the problems I might hit if I wanted to run hundreds of replicas off a single master. If I heard right, Howard told me that OpenLDAP uses a single thread for replication, and therefore processes each replica in a serial fashion, one at a time. If any replica is going slowly (high network latency for example), it would have a knock-on effect to the time it takes to get changes replicated across the whole estate. We have an aggressive target to meet - updates out in 5 minutes. Updates are small, but alarm bells are now ringing in my head about OpenLDAP's ability to deliver this. This leads to a couple of questions: 1 - How easy would it be to patch OpenLDAP 2.4 to get the master to use multiple threads for replication? Is that a reasonably straightforward fix, or is this quite a sizeable architectural change? 2 - If my master server has 40 cores, would there be mileage in setting up a number of slapd processes (say 10) on the same host, bound to different sockets, all acting as first-level replicas, and then all the replicas fanning out across the WAN synchronise from these processes rather than from the master process? That would be a way, I suppose, of getting the master server to make better use of the available cores to get updates out quicker? Thanks, Mark. ________________________________ NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.

8 16

Sync replication halted
by espeake＠oreillyauto.com 25 Nov '15

25 Nov '15

We are run a 3 node MMR with Openldap 2.4.39 on Ubuntu 14.04 (trusty) Everything runs great with an occasional hiccup like this. I see the following in the logs that appear to be searches on the RID numbers that are used for config replicationL Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: syncprov_matchops: skipping relayed sid 003 Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: syncprov_matchops: skipping original sid 001 Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: syncprov_matchops: skipping relayed sid 003 Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: syncprov_matchops: skipping original sid 001 Imeadiately after this all changes show csn_too_old errors: Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 cookie=rid=004,sid=001,csn=20151125103121.701210Z#000000#001#000000 Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 CSN too old, ignoring 20151125103121.701210Z#000000#001#000000 (uid=JRIVAS21,ou=Users,dc=oreillyauto,dc=com) Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 cookie=rid=004,sid=001,csn=20151125103121.705110Z#000000#001#000000 Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 CSN too old, ignoring 20151125103121.705110Z#000000#001#000000 (uid=JRIVAS21,ou=Users,dc=oreillyauto,dc=com) Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 cookie=rid=004,sid=001,csn=20151125103121.713760Z#000000#001#000000 Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 CSN too old, ignoring 20151125103121.713760Z#000000#001#000000 (uid=ACERVANT4,ou=Users,dc=oreillyauto,dc=com) Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 cookie=rid=004,sid=001,csn=20151125103121.718018Z#000000#001#000000 Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 CSN too old, ignoring 20151125103121.718018Z#000000#001#000000 (uid=ACERVANT4,ou=Users,dc=oreillyauto,dc=com) Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 cookie=rid=004,sid=001,csn=20151125103121.726847Z#000000#001#000000 Nov 25 04:31:22 tn-ldap-a-2 slapd[5931]: do_syncrep2: rid=004 CSN too old, ignoring 20151125103121.726847Z#000000#001#000000 (uid=KCOOPER10,ou=Users,dc=oreillyauto,dc=com) This happened on one node. The third node had the same replication error for a few miliseconds and then recovered. I am attaching the database config file as well. Thank you, Eric Speake Senior Systems Administrator Information Systems O'Reilly Auto Parts (417) 862-2674 Ext. 1975 (See attached file: olcDatabase={1}mdb.ldif)This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS � 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you.

1 0

LMDB: successive fillups and drops break the DB after a while
by Dominik Taborsky 25 Nov '15

25 Nov '15

Hello, I've tried stress-testing LMDB in our use case and I've discovered something I consider a bug. In our use case we need to keep inserting to the DB until it's full, flush it and repeat. We've designed the insert function to fail if there are not enough headroom pages for safe delete / DB drop. During the tests the DB works correctly for some amount of repeats, but then it suddenly refuses to insert anything. It seems LMDB doesn't clear or reuse all of its pages. I would like to get anyone's opinion on this. I have attached the source file containing the test. The DB size is set to 20MB, the inserted values have by default 25kB. In the first run the DB accepts 365 inserts, which decreases over time and stabilizes at 280. After 18 repeats the DB does not accept any inserts at all: ok 1 - pass #1 fillup run (365 inserts) ok 2 - pass #2 fillup run (300 inserts) ok 3 - pass #3 fillup run (286 inserts) ok 4 - pass #4 fillup run (283 inserts) ok 5 - pass #5 fillup run (281 inserts) ok 6 - pass #6 fillup run (280 inserts) ok 7 - pass #7 fillup run (280 inserts) ok 8 - pass #8 fillup run (280 inserts) ok 9 - pass #9 fillup run (280 inserts) ok 10 - pass #10 fillup run (280 inserts) ok 11 - pass #11 fillup run (280 inserts) ok 12 - pass #12 fillup run (280 inserts) ok 13 - pass #13 fillup run (280 inserts) ok 14 - pass #14 fillup run (280 inserts) ok 15 - pass #15 fillup run (280 inserts) ok 16 - pass #16 fillup run (280 inserts) ok 17 - pass #17 fillup run (280 inserts) ok 18 - pass #18 fillup run (108 inserts) not ok 19 - pass #19 fillup run (0 inserts) not ok 20 - pass #20 fillup run (0 inserts) The test is run 4 times with different approaches that we thought could have had some impact. These approaches combine opening the DB per insert or per single fillup, and reopening the DB just for the DB drop. In this case these options have no effect and the results are the same, but originally a unit test that this originates from had some differences so I kept them in. Note that I also tried calling mdb_drop with "1" as the last parameter, but it had no effect on the result. I am linking against the latest LMDB from sources. I also tried adding a constant number of pages to the reserve before the insert function fails. This only changes the amount of inserted items but doesn't change the functionality. If you compile the program you can supply it with an argument to change the default size of the item value size. Lower values break the DB sooner (20kB requires only 12 repeated fillups and drops). If anyone has any idea what might be the issue, please let me know. I know deleting the DB from filesystem and creating a new one would work, but that's a hack, not a fix. Thanks. Dominik

2 4

rwm overlay
by BÖSCH Christian 25 Nov '15

25 Nov '15

Hi, I configured rwm overlay like the example in the man page to allow binds with the email address. dn: olcOverlay={0}rwm,olcDatabase={-1}frontend,cn=config objectClass: olcOverlayConfig objectClass: olcRwmConfig olcOverlay: {0}rwm olcRwmRewrite: {0}rwm-rewriteEngine "on" olcRwmRewrite: {1}rwm-rewriteMap "ldap" "attr2dn" "ldap://localhost/dc=abcd,dc =net?dn?sub" olcRwmRewrite: {2}rwm-rewriteContext "bindDN" olcRwmRewrite: {3}rwm-rewriteRule "^mail=[^,]+@[^,]+$" "${attr2dn($0)}" ":@I" olcRwmTFSupport: false olcRwmNormalizeMapped: FALSE But I get the error message the the DN is invalid: ldapsearch -x -D "cb(a)abcd.net" -W -b 'dc=abcd,dc=net' -H ldap://openldap1.abcd.net/ 'uid=cb' Enter LDAP Password: ldap_bind: Invalid DN syntax (34) additional info: invalid DN Is there something missing or wrong? Thanks, Chris

2 2

LMDB overflow pages
by Christian Sell 24 Nov '15

24 Nov '15

Hello, something wrong with the question below? I am trying to use LMDB to store large (huge) amounts of binary data which, for the reason of limiting memory footprint, are split into chunks. Each chunk ist stored under a separate key, made up of [collectionId, chunkId], so that I can later iterate the chunks using a LMDB cursor. Chunk size is configurable. During my tests, I encountered a strange scenario where, after inserting some 2000 chunks consisting of 512KB each, the database size had grown to a value that was roughly 135 times the calculated size of the data. I ran the stat utility over the db and saw that there were > 12000 overflow pages vs. approx. 2000 data pages. When I reduced the chunk size to 4060 bytes, the number of overflow pages went down to 1000, and the database size went down to the expected number (I experimented with different sizes, this was the best result). I did not find any documentation to explain this behaviour, or how to deal with it. Of course it makes me worry about database bloat and the consequences. Can anyone shed light on this? thanks, Christian

1 0

Re: ERR_employeeadd {'info': 'modifications require authentication', 'desc': 'Strong(er) authentication required'}
by Andrei Valoshyn 24 Nov '15

24 Nov '15

Le 19/11/2015 19:43, Andrei Valoshyn a écrit : Hello! I have slapd 2.4.39 and python 2.6 I tried to create an user via python when I tried do that with rootpermission - it's OK. But when I did this with config in slapd.conf"access to * bygroup.exact="cn=LDAP_admins,ou=Roles,ou=Groups,dc=exadel,dc=com" write"I have an error " ERR_employeeadd {'info': 'modifications requireauthentication', 'desc': 'Strong(er) authentication required'} " I tried to use " l.protocol_version = ldap.VERSION{2,3} " via 389 port My function for adding ldif is : l = ldap.initialize(server) l.simple_bind(username, ldapsrvpassword) def employeeadd(): ldif = modlist.addModlist(attrs) l.add_s(dn,ldif) Will be very appreciate for any help Hello Andrei, I suppose that the username you use is a member ofcn=LDAP_admins,ou=Roles,ou=Groups,dc=exadel,dc=com, but whichobjectClass did you use in your group? By default, the OpenLDAP ACLsystem will use groupOfNames, searching user in the member attribute. Ifyou have for example a groupOfUnixNames, you need to set your ACL likethis:access to * bygroup/groupOfUniqueNames/uniqueMember.exact="cn=LDAP_admins,ou=Roles,ou=Groups,dc=exadel,dc=com"write Hello *Clément*, Thank you for reply! I have groupOfNames and posixGroup objectClass. But at that moment this problem was fixed. The matter of fact Python had some problem. It was reinstalled. Thank you! -- CONFIDENTIALITY NOTICE: This email and files attached to it are confidential. If you are not the intended recipient you are hereby notified that using, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error please notify the sender and delete this email.

1 0

Multiple certificates in slapd
by Olivier Nicole 24 Nov '15

24 Nov '15

Hi, I am planing a transition of the certificate I use in OpenLDAP for LDAP over SSL (port 636). My selft signed certificate is quite old and has become obsolete/not recognized on some systems (for example Mac OS 10.11) so it is time to update. But I have many systems that use LDAP and updating all of them cannot be done at once. So I was wondering if it is possible to have one single slapd process running with several posut open over SSL, keeping port 636 with the old certificate and opening port 637 with the new certificate. That way, i can transition the clients at my onw pace, not needing to do all at same time. I know that I could set-up a slave server, but that would be not as transparent s0 I'd prefer my idea of havingslapd -h ldaps://192.168.10.1:636/ ldaps:/192.168.10.1:637/ each using a different certificate. Thanks in advance, Olivier --

2 1

Re: Trying to set up multimaster syncrepl, error attribute 'olcTLSCertificateFile' not allowed , why?
by Betsy Schwartz 24 Nov '15

24 Nov '15

On Sat, Nov 21, 2015 at 2:00 PM, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > > I would suggest using slapcat to export the config database and clean up > the invalid attribute values that were incorrectly added to the bdb > database. > > Thank you very much! I have made progress but am now in an entertaining new place (having *overwritten* my database with my overlay files). But I'm going to go RTFM and see if I can't dig myself out of that one. > a) Upgrading to a current openldap release > b) Switching to back-mdb, assuming a 64-bit OS. > Thank you! Sigh, 2.4.40 seems to be the latest in the Oracle repo (we are using Oracle's version of RHEL6). It looks like there have been some good bug fixes and I'll make the case for going outside the repo. Sometime in the next couple months these two new servers will become the primary production servers and then we'll be able to do what we want with them. The two old servers, current production, are running 2.4.39 and 2.4.23 (and not syncing with each other!) . I've been hoping that I could sync data between at least these two new servers and the 2.4.39 server, is that possible or a foolish hope? Again, once these new ones become primary I'll be able to keep them identical to each other, but I don't really 'own' the old ones and don't want to break them. I'm still working through the manual and the configuration settings. One more question - still trying to understand what was done on these old servers - on these servers the config database is 0, the monitor database is 1 and the bdb database is 2. I can't slapcat the monitor database, is that normal? I get slapcat -n 1 slapcat: database doesn't support necessary operations. I had a little trouble with this particular section of the manual which reads simply: 20.1. Monitor configuration via cn=config(5) This section has yet to be written. thanks again Betsy

2 1

questions about memberof-refint option
by M. P. 22 Nov '15

22 Nov '15

Hi, I'm playing with memberof overlay. For my tests, I use the default database (numbered 1) from slapd installation with suffix dc=nodomain. The tests are running on debian jessie 8.2 and slapd version 2.4.40+dfsg-1 Activating the module in cn=module entry and activating the overlay for the database, I have something that works like (I think) it should. I mean adding a user (attribute member) in a group creates an attribute memberOf for the user and deleting a user from the group deletes the user's memberOf attribute. That's great. There is nothing special configured. # Entrée 1: olcOverlay={0}memberof,olcDatabase={1}mdb,cn=config dn: olcOverlay={0}memberof,olcDatabase={1}mdb,cn=config objectclass: olcConfig objectclass: olcOverlayConfig objectclass: olcMemberOf objectclass: top olcoverlay: {0}memberof Reading the man page, I saw memberof-refint option. From what I understand, when set to true, you can alter the user's "is member of" attribute and that would be reflected in the group's "member" attribute. Right ? But, the member attribute is an operational attribute and can't be modified. So I started to search for an alternative and found the eduMember schema from here https://spaces.internet2.edu/display/macedir/OpenLDAP+eduMember. Once added to the installation I could use it for objects. It adds isMemberOf and hasMember attributes that can be setable for users and groups. But can't make it work with memberof overlay. When trying to add isMemberOf as memberof-memberof-ad it was rejected with member attribute=”isMemberOf” must either have DN (1.3.6.1.4.1.1466.115.121.1.12) or nameUID (1.3.6.1.4.1.1466.115.121.1.34) syntax And the same error was reported with hasMember as memberof-member-ad. To make it work together I modified the attribute's definitions and reimported them to openldap. So I can now set isMemberOf as memberof-memberof-ad and the same for hasMember as memberof-member-ad. The configuration then was like this # Entrée 1: olcOverlay={0}memberof,olcDatabase={1}mdb,cn=config dn: olcOverlay={0}memberof,olcDatabase={1}mdb,cn=config objectclass: olcConfig objectclass: olcOverlayConfig objectclass: olcMemberOf objectclass: top olcmemberofmemberofad: isMemberOf olcoverlay: {0}memberof Now that works like (I think) it should. I mean adding a user (attribute member) in a group creates an attribute isMemberOf for the user and deleting a user from the group deletes the user's isMemberOf attribute. That's great. isMemberOf is a modifiable attribute so it's time to test the memberof-refint and set it to TRUE # Entrée 1: olcOverlay={0}memberof,olcDatabase={1}mdb,cn=config dn: olcOverlay={0}memberof,olcDatabase={1}mdb,cn=config objectclass: olcConfig objectclass: olcOverlayConfig objectclass: olcMemberOf objectclass: top olcmemberofmemberofad: isMemberOf olcmemberofrefint: TRUE olcoverlay: {0}memberof And this is where things do not work. I mean what was working before is still working. If I add a member in a group an atttribute isMemberOf is created for the user. But if I add a second attribute isMemberOf with a second group, no new member is created on the second group. And if I delete the attribute isMemberOf from the user's entry, it is still visible on the group. Does anybody have any idea why the modifications made on the user (with the deletion of isMemberOf) are not applied to the group ? Is there something I'm doing wrong ? Thanks. -- ------------ M. P.

4 11

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical November 2015