openldap-technical

openldap-technical@openldap.org

14 participants
9795 discussions

mdb backup via slapcat
by Bastian Tweddell 27 Jun '16

27 Jun '16

Dear all, I am used to run slapcat to create backups of the backend database _while_ slapd is running. Recently I migrated from bdb to mdb. Now I read that using slapcat on a running slapd is only safe for bdb and hdb backends [1]. 1: http://www.openldap.org/faq/data/cache/287.html Question: - Is that still true? - In which situation could data corruption occur? - Is that supposed to change in the future? Many thanks, -- Bastian Tweddell Juelich Supercomputing Centre phone: +49 (2461) 61-6586 HPCNS, HPS ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

2 2

Re: LMDB data growth - overflow pages
by Christian Sell 26 Jun '16

26 Jun '16

Hi, as far as my scenario goes, I shortly afterward detected serious errors in my setup that were responsible to the buildup. There is an article on the symas website (https://symas.com/understanding-lmdb-database-file-sizes-and-memory-utiliza…) that adresses the question/issue of database growth, I assume you have read that. Crashes sould of course not happen, but what exactly do you mean by "crash" anyway? regards, Christian > salesman1 IIS Advertising <sales-iss(a)outlook.com> hat am 26. Juni 2016 um > 00:59 geschrieben: > > > Re: I HAVE THE SAME PROBLEMS, TOO ! > > > from your comments > > http://www.openldap.org/lists/openldap-technical/201511/msg00201.html > > > > I noticed the following using lmdb. Tested version is up-to-date 0.9.18, from > 2016. > > > Database grows much larger than expected in many use cases, and eventually > crashes, also was > > not a well-behaved database when simple, missing or misbehaved client code > executes it. > > . > > Do you think the same ? > > > Maybe lbmd is was fine-tuned for other especific needs, or is a toy database, > or a complicated > > case of application in our scenario. Unfortunatelly is not like Redis yet. > > We can not throw anything special on it without some effort. > > > > We really want do use lbmd, it's fast, but can't get it to work as expected. > Lmbd may be incomplete, > > buggy for (the common coder) or || (the experienced coder) who tries do use > it for other needs > > than designed. So it's now a dream but not easily realistic. Needs an expert. > > > > > 1. Database size grows 3x the data we put in, than stops growing - when > records (key+data) from 410 to 4010 bytes are added. > > -> and consumed much more space than expected for 5000*400 bytes = 2.000.000 > bytes > > > 2. Second, when we put key+data > 4096 bytes, database grows much larger and > crashes a lot of times, with a number > > of page overflows shown on 'mdb_dump ./testdb' > > > We made keys ("k:%d", i%5000) for (i=0; i < 1000000; i++) // so there are > 5000 keys, else is overwriting; > > Data is random ascii or fixed memset( 'x',400 ) > > > Program name: sample2 > > We are overwriting old values AND NOT appending, but it grows anyway. Larger > data than 4096 bytes per key-value > > seems impossible or non-sense at this time. Are we using wrong flags or > program structure ? > > > > trials - first run goes fine: #./sample2 > > > [root@ip-172-31-13-74 liblmdb]# ./sample2; ls ./testdb/* -l > Start > Init done > written 100000 pairs of len=400 bytes > -rw-r--r-- 1 root root 2584576 Jun 25 21:43 ./testdb/data.mdb > -rw-r--r-- 1 root root 8192 Jun 25 21:43 ./testdb/lock.mdb > [root@ip-172-31-13-74 liblmdb]# ./sample2; ls ./testdb/* -l > Start > Init done > written 100000 pairs of len=400 bytes > -rw-r--r-- 1 root root 5173248 Jun 25 21:43 ./testdb/data.mdb > -rw-r--r-- 1 root root 8192 Jun 25 21:43 ./testdb/lock.mdb > [root@ip-172-31-13-74 liblmdb]# ./sample2; ls ./testdb/* -l > Start > Init done > written 100000 pairs of len=400 bytes > -rw-r--r-- 1 root root 7761920 Jun 25 21:43 ./testdb/data.mdb > -rw-r--r-- 1 root root 8192 Jun 25 21:43 ./testdb/lock.mdb > [root@ip-172-31-13-74 liblmdb]# > > > > Looks like it's not replacing old keys and have a page/data size problem, > > > Any suggestions ? > > > REGARDS !! > > Fabio Martinez - Buenos Aires and São Paulo > > gcc developer > > Brazil > > > -- > > > > > Hello, > > I am trying to use LMDB to store large (huge) amounts of binary data which, > for the reason of limiting memory footprint, are split into chunks. Each chunk > ist stored under a separate key, made up of [collectionId, chunkId], so that I > can later iterate the chunks using a LMDB cursor. Chunk size is configurable. > > During my tests, I encountered a strange scenario where, after inserting some > 2000 chunks consisting of 512KB each, the database size had grown to a value > that was roughly 135 times the calculated size of the data. I ran stat over > the db and saw that there were > 12000 overflow pages vs. approx. 2000 data > pages. When I reduced the chunk size to 4060 bytes, the number of overflow > pages went down to 1000, and the database size went down to the expected > number (I experimented with different sizes, this was the best result). I did > not find any documentation that would explain this behavior, or how to deal > with it. Of course it makes me worry about database bloat and the > consequences. Can anyone shed light on this? > > thanks, > Christian >

1 0

Re: Antw: Re: openldap 2.4.44 - delta-syncrepl fails on auditContext
by Frank Swasey 24 Jun '16

24 Jun '16

Today at 6:09am, Ulrich Windl wrote: > I wonder why you configure accesslog on one node, but not on the other. Here we use same configuration on every node. I am continuing the practice of having a master and slaves. The masters are set up with MMR and do have accesslog configured on them. The slaves are set up to delta-syncrepl the main database from and force all updates to go to the MMR masters. I have a second set of slaves that are used for mail only, and they replicate just the mail related data from the MMR masters. -- Frank Swasey | http://www.uvm.edu/~fcs Sr Systems Administrator | Always remember: You are UNIQUE, University of Vermont | just like everyone else. "I am not young enough to know everything." - Oscar Wilde (1854-1900)

1 0

ldapsearch + read-only domain controller: cannot bind
by l＠avc.su 24 Jun '16

24 Jun '16

3 8

openldap 2.4.44 - delta-syncrepl fails on auditContext
by Frank Swasey 22 Jun '16

22 Jun '16

Delta-Syncrepl has started failing to actually replicate the consumer starting with an empty database, it fails with code 0x50 because of the auditContext attribute that is present in the suffix entry on the master server due to the accesslog overlay being used. I can make it work if I load the accesslog overlay in the replica's configuration (without actually configuring it). This appears to be new behavior since 2.4.42. Is this expected, and now required with 2.4.44, or should I open an ITS? -- Frank Swasey | http://www.uvm.edu/~fcs Sr Systems Administrator | Always remember: You are UNIQUE, University of Vermont | just like everyone else. "I am not young enough to know everything." - Oscar Wilde (1854-1900)

4 15

Schema's not importing correctly?
by Trent Dierking 22 Jun '16

22 Jun '16

Per the quickstart guide, I am trying to import an ldif file with the following: dn: dc=example,dc=com objectclass: dcObject objectclass: organization o: Example Company dc: example dn: cn=Manager,dc=example,dc=com objectclass: organizationalRole cn: Manager However, I receive errors indicating something is wrong about the object class. So I tried the first two lines only, and receive: ldap_add: Object class violation (65) additional info: no structural object class provided The common error faq indicates this is because my objectclass is not valid. Since this is the quickstart guide, and dcObject comes from core.schema (which is included by default), what the hell is going on? My best guess is that the schema is not being correctly imported, but I'm not sure why.

2 1

Re: Odd MMR behaviour with delta-syncrepl and refreshAndPersist
by Quanah Gibson-Mount 21 Jun '16

21 Jun '16

--On Wednesday, June 15, 2016 1:59 PM +0100 Mark Cairney <Mark.Cairney(a)ed.ac.uk> wrote: > Hi Quanah, > > I can confirm I still see the issue when deleting and adding user > objects and groups using 3-way delta-MMR. Please keep replies on the list. > From one of the servers receiving the change: > [snip] > I spotted the reference to "cn=marksgroup2" in the log above so decided > to try it with an objectclass that has no group memberships managed by > the memberOf overlay (simplesecurityobject) and it worked as expected: > > Again from the consumer I see the following logged: [snip] > So it looks like there's possibly an additional effect being caused by > the memberOf overlay but as about 90% of our LDAP writes are the > creation/modification/deletion of users and groups this could be a pain > on a production system :-) > > Is this enough for you to go on? If there's any additional logging or > details of my config I'm happy to pass them on. I dropped your replication log bits in case there was anything sensitive in there. I agree, it looks like the memberof overlay is breaking replication in your case. I would suggest filing an ITS with details on your setup, and the logging you provided, obfuscated as necessary. --Quanah -- Quanah Gibson-Mount Platform Architect Manager, Systems Team Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc

3 3

Give user only access to a few entries that he "owns"
by PenguinWhispererThe . 21 Jun '16

21 Jun '16

Hi all, I'm not very experienced with ldap. I've been looking into the access controls documentation but I'm unsure on what the proper way to handle this is. So let me expain what I want to accomplish: a user entry (posixAccount, password, givenName, ...) can update his own password by using the "self" keyword. All good there. But a user has some assets he owns. For example a host (in Common tree). I want the user to be able to update one attribute of this host. "self" keyword doesn't work here as the user doesn't bind to it. So I added an owner attribute to the host and with that attribute I reference to the user. Now I need some kind of "glue" to verify that the user is allowed to write to the attribute. Do I need a filter? Wouldn't this just filter out a specific attribute? Or will it only filter entries based on the filter match? In the latter case (which seems like a logical way for openldap to handle this) I would need: - attr: to select what attribute the user access is modified - filter: to only apply on the user his host - by: variable definition for this clause to only apply on the binded user I've read about dnattr but I'm unsure this is accomplishes what I want. Could anyone share an example? Thanks

2 3

cn=config database setup
by Trent Dierking 21 Jun '16

21 Jun '16

Following the quickstart guide at http://www.openldap.org/doc/admin24/quickstart.html and the following happens # ./sbin/slapadd -F etc/'cn=config' -l etc/openldap/slapd.ldif 57680d0f invalid config directory etc/cn=config, error 2 slapadd: bad configuration directory! # mkdir -p 'etc/cn=config' # ./sbin/slapadd -F etc/'cn=config' -l etc/openldap/slapd.ldif Available database(s) do not allow slapadd I've ensured the appropriate var/openldap-data directory exists. I've tried adding the maxsize key shown on the guide. I am not sure what else could cause this, as this is for the hardcoded cn=config database.

2 1

auditContext attribute ?
by Frank Swasey 20 Jun '16

20 Jun '16

I have discovered (on my test environment running 2.4.44 on RHEL7) that when I slapcat from either of the MMR syncing masters the suffix base entry has the auditContext attribute. If I take the generated LDIF to my replica server, it will not slapadd unless I delete the auditContext attribute. I assume this is something that I have accidentally turned on in the slapd configuration or is new in 2.4.44. I'm confused and not finding any documentation of this attribute and why it is getting included in the slapcat output - can anyone guide me to an answer? -- Frank Swasey | http://www.uvm.edu/~fcs Sr Systems Administrator | Always remember: You are UNIQUE, University of Vermont | just like everyone else. "I am not young enough to know everything." - Oscar Wilde (1854-1900)

2 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical