openldap-technical

openldap-technical@openldap.org

14 participants
9795 discussions

Re: Possible transaction issue with LMDB
by Howard Chu 17 Aug '18

17 Aug '18

William Brown wrote: > On Fri, 2018-08-17 at 07:33 +0100, Howard Chu wrote: >> William Brown wrote: >>> On Fri, 2018-08-17 at 07:06 +0100, Howard Chu wrote: >>>>> I'm quite aware that it is COW - this issue is specific to COW >>>>> trees. >>>>> Transactions must be removed in order, they can not be removed >>>>> out >>>>> of >>>>> order. It is about how pages are reclaimed for the freelist >>>> >>>> Incorrect. >>> >>> We may have to "agree to disagree" then. >> >> Pure, utter nonsense. You're simply wrong, and at this point you're >> spewing FUD. > > You know what - I made a mistake. I read "free list" when it should be > "pending free list". Which makes much more sense, but is poorly > signaled in the naming conventions. > > I also misread this comment about "incorrect". You meant "incorrect" > about the reclaiming of freelist not about the ordering. > > Instead of trying to have a constructive discussion, I believe your > responses here were quite hostile, and their "short" nature has led to > further miscommunication. My mistake was answering too quickly. I have > removed myself from the list because I really just don't want to be > involved in this style of discussion in the future. You claimed to have actual code showing a misbehavior in LMDB when the described misbehavior is impossible. That's not a misunderstanding or a mistake - that's pure fabrication. Dishonesty is absolutely not welcome. Bye. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Possible transaction issue with LMDB
by William Brown 17 Aug '18

17 Aug '18

Hi there, While doing some integration testing of LMDB I noticed that there may be an issue with out of order transaction handling. The scenario is: Open Read TXN A Open Write TXN X, and change values of the DB Commit X Open Read TXN B Open Write TXN Y, and change values of the DB Commit Y Open Read TXN C Abort/Close TXN B. At this point, because of the page touch between A -> B and B -> C, B now believes that the pages of A are the "last time" they are available as they were all subsequently copied for TXN C. The pages of A are then added to the freelists when B closes. When TXN A is read from the data may have been altered due to future writes as LMDB attempts to use previously allocated pages first. This situation is more likely to arise on large batch writes, but could manifest with smaller series of writes. This would be a silent issue, as the over-written pages may be valid, and could cause data to "silently vanish" inside of the read transaction A leading to unpredictable results. I hope that this report helps you to diagnose and resolve the issue. -- Sincerely, William

3 6

Re: [LMDB] What pointer is returned with combination of MDB_WRITEMAP and MDB_RESERVE?
by Victor Baybekov 16 Aug '18

16 Aug '18

Hello, Working with overflow pages directly via pointers outside write transactions works great and it helps that they do not move "by design" in current versions as discussed in this thread. I have two related scenarios that will give a substantial performance boost in my case. *The first one* is updating a value in-place via pointer from aborted write transaction. If I 1) use MDB_WRITEMAP, 2) from **write** transaction find a record (which is small and not in an overflow page), 3) modify a part of it's value (for duplicates this part is not used in the compare function) directly via the MDB_VAL data pointer (e.g. interlocked_increment or compare_and_swap), 4) and **abort** the transaction, then readers see the updated value via normal read transactions later. Since I do the direct updates from inside a write transaction all other writers should be locked until I exit the transaction (abort in this case), and no pages should move since the transaction is aborted. Is this correct? Does this work "by design" or "by accident" currently? *The second one* is about updating values in-place from read transactions. If I 1) use MDB_WRITEMAP, 2) open a database with MDB_INTEGERKEY (and could use a dedicated environment with a single DB if that changes the answer), 3) add values to the DB *only* using MDB_APPEND | MDB_NOOVERWRITE, 4) modify a part of a value directly via the MDB_VAL data pointer, is it possible that the page from the read transaction is replaced with a new one if there is a parallel write transaction? There is a quote from Howard on GitHub (https://github.com/lmdbjava/ benchmarks/issues/9#issuecomment-354184989): "When we do sequential inserts using MDB_APPEND, there is* no page split at all* - we just allocate a new page and fill it, sequentially." Does this mean that if there are no page splits then existing pages do not move as well, and it is "safe" to use pointers outside of write transactions as is the case with the overflow pages? In both cases I update values of a struct that indicate e.g. some lifecycle stage of an object the LMDB record refers to and stage transitions are idempotent. If a direct pointer write doesn't make to disk due to system failure subsequent readers (workers) will see an older stage and repeat stage transition. Therefore missed direct writes do not break application logic, I only care about physical corruption of the entire DB. If I update the values in-place inside read transactions and the page becomes stale this should not corrupt DB since the old page will go to the free list only after the read transaction is finished, so this "hack" should not break DB. But then missed writes will be a norm and not a special-case on OS failure. But if pages do not move, all these "soft" updates could be done in parallel and be very fast. Unfortunately I cannot answer this myself while trying to read the mbd.c file. In the second scenario I'm specifically concerned what is happening when DB becomes large and the tree needs rebalancing. At least in this case some pages need to move, but does the rebalancing replace/split existing pages? Thanks & best regards, Victor On Fri, Oct 30, 2015 at 9:26 PM, Howard Chu <hyc(a)symas.com> wrote: > Victor Baybekov wrote: > >> Thanks a lot! My proof-of-concept code works OK. >> >> I do not understand all subtle details of mmap reliability, could you >> please >> help with these two: >> >> If I write data to a pointer to an opaque blob as discussed above, and my >> process crashes before mdb_env_sync, but OS doesn't crash - will that >> data be >> secure in the mmap file? >> > > Of course. The OS owns the memory, it doesn't matter if your process > crashes. > > Also, am I correct that mdb_env_sync synchronizes all dirty pages in the >> mmap >> file as seen by a file system, regardless how they were modified - either >> via >> LMDB API or via a direct pointer writes? >> > > Yes. > > As for "you could at least set a callback to notify you that a block has >> moved" - if that is implemented, it would be nice to have a notification >> /before/ a block is moved (with old and new address, so that right after >> the >> callback it is OK to use the new address), otherwise this non-intended but >> convenient use of LMDB won't work anymore. >> > > "right after the callback it is OK to use the new address" - that's the > point of the callback, it's job is to make the new address valid. So yes, > when it returns, you use the new address. > >> >> >> Best regards, >> Victor >> >> >> >> On Sat, Oct 3, 2015 at 1:27 AM, Howard Chu <hyc(a)symas.com >> <mailto:hyc@symas.com>> wrote: >> >> Howard Chu wrote: >> >> Victor Baybekov wrote: >> >> Thank you! I understand this copy-on-write behavior, but am >> interested if I >> could control it a little. What if I use records that are >> always >> much bigger >> than a single page, e.g. 100 kb with 4kb pages, and make sure >> that >> a record is >> never updated (via LMDB means) during a lifetime of an >> environment, - is there >> any scenario that the location of such a big record could be >> changed during a >> lifetime of an environment, without updating the record? >> >> >> At this point in time, no, if you don't update a large record >> there is no >> reason that it will move. That is not to say that this won't >> change in the >> future. The documentation tells you what promises we are willing >> to make. >> Relying on any non-documented behavior is your own responsibility. >> >> >> Note that the relocation functions in LMDB are intended to accommodate >> blocks being moved around. The actual guts of that API haven't been >> implemented, but probably in 1.x we'll flesh them out. Given that >> support, >> you could at least set a callback to notify you that a block has >> moved. >> But currently, overflow pages don't move if they're not modified. >> >> >> >> >> On Fri, Oct 2, 2015 at 4:38 PM, Howard Chu <hyc(a)symas.com >> <mailto:hyc@symas.com> >> <mailto:hyc@symas.com <mailto:hyc@symas.com>>> wrote: >> >> Victor Baybekov wrote: >> >> Hi, >> >> Docs for MDB_RESERVE say that a returned pointer to >> the >> reserved >> space is >> valid "before the next update operation or the >> transaction ends." Docs >> for MDB_WRITEMAP say that it "writes directly to the >> mmap >> instead of >> using >> malloc for pages." Does combining the two options >> return >> a pointer >> directly to >> a place in a mmap >> >> >> Yes. >> >> so that this pointer could be used after a >> transaction ends >> or after the next update? >> >> >> No. >> >> Longer answer: maybe. >> >> Full answer: LMDB is copy-on-write. If you update another >> record on the >> same page, in a later transaction, the contents of that >> page >> will be >> copied to a new page and the original page will go onto >> the >> freelist. In >> that case, the pointer you got must not be used again. >> >> If you don't directly update that page and cause it to be >> copied, then you >> might get lucky and be able to use the pointer for a >> while. >> It all depends >> on what other modifications you do and how they affect >> that >> node or >> neighboring nodes. >> >> >> I have a use case where I want to somewhat abuse LMDB >> safety for >> convenience. >> If I could get a pointer to a place inside a mmap I >> could >> work with >> LMDB value >> as opaque blob or as a region inside the single big >> mmap. >> This could >> be more >> convenient than creating and opening hundreds of >> temporary memory >> mapped files >> and keeping open handles to them. For example, Aeron >> terms could be >> stored >> like this: a stream id per an LMDB db and a term id >> for a >> key in the >> db. >> >> >> Thanks! >> Victor >> >> > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ >

2 2

permissions replication
by Miroslav Misek 13 Aug '18

13 Aug '18

Hi, I am setting up master-slave replication for our off-site office, so it can use authentication against ldap even with internet connectivity issues. Replication itself is working without problems. But it replicates only data and not olcAccess attributes on database. So I have to set them manually. Please is there any way to replicate those attributes too? I found only one way, and it is master-master replication of cn=config database. And it is not usable in our environment. Off-site office don`t have public ip. And it is better for me to have this ldap instance read-only. Thank you, Miroslav Misek

4 4

Re: Search memberOf
by Arianna Milazzo 09 Aug '18

09 Aug '18

Ok, I understand that it isn't supported, but at the moment I can't try other solutions. And since that aside from that filter, the rest works, I don't want to give up like that. Infact if I look for the following values (then on the groups) Search base: cn=groupname,ou=group,dc=pigreco,dc=it Filter: (member=cn=Name Surname,ou=people,dc=pigreco,dc=it) I get if Name Surname is part of the groupname group If I search Search base: dc=pigreco,dc=it Filter: (member=cn=Name Surname,ou=people,dc=pigreco,dc=it) I get the list of which groups Name Surname belongs *But with this (then on the people)* Search base: dc=pigreco,dc=it Filter: (memberOf=cn=groupname,ou=group,dc=pigreco,dc=it) *I have no result and in the log I read:get_ava: illegal value for attributeType memberof* obviously same thing with: (& (uid=n.surname) (memberOf=cn=groupname,ou=group,dc=pigreco,dc=it) ) :( On the "groups" it is works! On the "people" it doesn't work (*get_ava: illegal value for attributeType memberof*) It's frustrating! 2018-08-07 21:35 GMT+02:00 Quanah Gibson-Mount <quanah(a)symas.com>: > --On Tuesday, August 07, 2018 1:30 PM -0700 Quanah Gibson-Mount < > quanah(a)symas.com> wrote: > > --On Tuesday, August 07, 2018 11:23 AM +0200 Arianna Milazzo >> <arianna(a)ariannamicrochip.it> wrote: >> Trying to force LDAP >> functionality with back-sql is going to work well as a path to pursue. >> > > *is not going to work well. > > --Qunah > > > > -- > > Quanah Gibson-Mount > Product Architect > Symas Corporation > Packaged, certified, and supported LDAP solutions powered by OpenLDAP: > <http://www.symas.com> > >

3 4

Re: Search memberOf
by Arianna Milazzo 07 Aug '18

07 Aug '18

Hello. Thank you for your reply. I have many applications using a db to manage access and profiles. Now I have to use even other application supporting ldap. So I’m hoping to use same db, so I can manage only a datasource for all Applications (same users!!!) But I need to use memberOf to manage the access to theese applications (profiling). (Excuse me for My English!) Thanks, Arianna Il giorno gio 2 ago 2018 alle 19:54 Quanah Gibson-Mount <quanah(a)symas.com> ha scritto: > --On Thursday, August 02, 2018 5:25 PM +0200 Arianna Milazzo > <arianna(a)ariannamicrochip.it> wrote: > > > > > > > Hello! > > I use OpenLDAP wuth MySQL backend. > > The MySQL backend is experimental and generally unsupported. It should > not > be used as a general purpose backend for an LDAP server. > > > Can someone help me? > > Is there a particular reason you're trying to wedge memberof support into > mysql, which is likely not designed to handle it? > > --Quanah > > > -- > > Quanah Gibson-Mount > Product Architect > Symas Corporation > Packaged, certified, and supported LDAP solutions powered by OpenLDAP: > <http://www.symas.com> > >

3 4

daemon: bind(11) failed errno=2 (No such file or directory)
by Alan Lukens 07 Aug '18

07 Aug '18

I encountered a problem with openldap-2.4.40 on debian when a vm crashed. Upon restore i could not start slapd service. But typing slapd on the command line would start slapd. I tried defining the urls, and tried defining the configuration directory with the slapd command, however that only resulted in the same error. After extensive searching i could only find one reference to the error "daemon: bind(11) failed errno=2 (No such file or directory)". and that was leading me in the direction of the directory /var/run/ldap didnt have the right permissions, but it was there and had openldap owner and group. I spent a number of hours double checking directories making sure everything was there and had the right permissions. I finally found /var/run/slapd was missing! A file called ldapi is created there on startup. What is /var/run/slapd/ldapi ?? And what does it do? For future reference what is bind(11) referring to? Where can i find the reference to error codes? Thanks for your help, Alan Lukens Identity Management Specialist University of Massachusetts Amherst, IT NSS voice: 413.545.1654 email: alan.lukens(a)umass.edu

2 1

ldapi and StartTLS
by Norman Gray 07 Aug '18

07 Aug '18

Greetings. I would have thought (possibly naively) that StartTLS was unnecessary when connecting to slapd through a unix socket -- the client and the server are on the same machine, and so don't need to be reassured about each other's identity. However this seems not to be be the case: % ldapsearch -LLL -H ldapi://%2Fvar%2Frun%2Fopenldap%2Fldapi '(uid=foo)' ldap_sasl_interactive_bind_s: Confidentiality required (13) additional info: stronger confidentiality required (same result with ldapi:///). What am I misunderstanding? In the slapd.ldif I have: dn: cn=config objectClass: olcGlobal cn: config olcArgsFile: /var/run/openldap/slapd.args olcPidFile: /var/run/openldap/slapd.pid olcSecurity: ssf=128 olcTLSCertificateFile: /usr/local/etc/openldap/certs/XXX.crt olcTLSCertificateKeyFile: /usr/local/etc/openldap/certs/XXX.key olcTLSCACertificateFile: /usr/local/etc/openldap/certs/FOO olcLogLevel: 0 The machine is also listening on ldap://0.0.0.0 and requiring TLS. I don't see anything in the documentation which seems to suggest I can have different TLS rules on different interfaces or protocols (ie, ldap: vs ldapi:) -- am I just missing that? The /usr/local/etc/ldap.conf doesn't mention TLS, so the TLS requirement isn't coming in from there. My practical problem is that I'm trying to get nslcd (on the same machine) to talk to OpenLDAP locally. If there's a certificate problem I can sort that out, but I can't help feeling that that ought to be unnecessary -- that I'm missing something simple. This is 2.4.45 on FreeBSD. Best wishes, Norman -- Norman Gray : https://nxg.me.uk

3 3

root server and subtree server replicate.
by Tian Zhiying 03 Aug '18

03 Aug '18

Dear all, I'd like to have a subtree managed by a second LDAP server and its contents replicated to the "upper" root server. server A(root server): suffix="dc=domain,dc=org" server B(subtree server): suffix="ou=people,dc=domain,dc=org" B's subtree should be replicated to A and should be searchable on A. Is there any solutions can fix this case? Thanks.

2 1

<olcMirrorMode> database is not a shadow
by admin＠genome.arizona.edu 02 Aug '18

02 Aug '18

Hello, Was setting up replication for our LDAP server, and was following the guide here, https://wiki.gentoo.org/wiki/Centralized_authentication_using_OpenLDAP#Sett… I had success with this guide but just a problem with authentication, I could see in the ldap debug log for node1 entries like this: Jul 20 16:21:22 node1 slapd[10218]: conn=3497 fd=38 ACCEPT from IP=<node1's IP>:34606 (IP=0.0.0.0:389) Jul 20 16:21:22 node1 slapd[10218]: conn=3497 op=0 BIND dn="cn=Manager,dc=genome,dc=arizona,dc=edu" method=128 Jul 20 16:21:22 node1 slapd[10218]: conn=3497 op=0 BIND dn="cn=Manager,dc=genome,dc=arizona,dc=edu" mech=SIMPLE ssf=0 Jul 20 16:21:22 node1 slapd[10218]: conn=3497 op=0 RESULT tag=97 err=0 text= Jul 20 16:21:22 node1 slapd[10218]: conn=3497 op=1 MOD dn="olcDatabase={1}bdb,cn=config" Jul 20 16:21:22 node1 slapd[10218]: conn=3497 op=1 MOD attr=olcSyncrepl Jul 20 16:21:22 node1 slapd[10218]: conn=3497 op=1 RESULT tag=103 err=0 text= Jul 20 16:21:22 node1 slapd[10218]: conn=3497 op=2 MOD dn="olcDatabase={1}bdb,cn=config" Jul 20 16:21:22 node1 slapd[10218]: conn=3497 op=2 MOD attr=olcMirrorMode Jul 20 16:21:22 node1 slapd[10218]: slap_client_connect: URI=ldap://node2.genome.arizona.edu DN="cn=ldapreader,dc=genome,dc=arizona,dc=edu" ldap_sasl_bind_s failed (49) Jul 20 16:21:22 node1 slapd[10218]: do_syncrepl: rid=001 rc 49 retrying Jul 20 16:21:22 node1 slapd[10218]: conn=3497 op=2 RESULT tag=103 err=0 text= Jul 20 16:21:22 node1 slapd[10218]: conn=3497 op=3 UNBIND Jul 20 16:21:22 node1 slapd[10218]: conn=3497 fd=38 closed and in the debug log for node2 entries like this: Jul 20 16:21:22 node2 slapd[25327]: conn=14036 fd=17 ACCEPT from IP=<node1's IP>:56460 (IP=0.0.0.0:389) Jul 20 16:21:22 node2 slapd[25327]: conn=14036 op=0 BIND dn="cn=ldapreader,dc=genome,dc=arizona,dc=edu" method=128 Jul 20 16:21:22 node2 slapd[25327]: conn=14036 op=0 RESULT tag=97 err=49 text= Jul 20 16:21:22 node2 slapd[25327]: conn=14036 op=1 UNBIND Jul 20 16:21:22 node2 slapd[25327]: conn=14036 fd=17 closed It turns out i had literally used credentials="secret" in the add-replication-node1/node2.ldif files! So I went back and used slappasswd to generate a new password and put it into the ldapreader.ldif and use ldapmodify instead this time with success on both nodes, [root@node1 openldap]# cat ldapreader.ldif dn: cn=ldapreader,dc=genome,dc=arizona,dc=edu changetype: modify replace: userPassword userPassword: <hash from slappasswd> [root@node1 openldap]# ldapmodify -x -W -D "cn=Manager,dc=genome,dc=arizona,dc=edu" -f ldapreader.ldif Enter LDAP Password: modifying entry "cn=ldapreader,dc=genome,dc=arizona,dc=edu" [root@node1 openldap]# [root@node2 openldap]# cat ldapreader.ldif dn: cn=ldapreader,dc=genome,dc=arizona,dc=edu changetype: modify replace: userPassword userPassword: <hash from slappwasswd> [root@node2 openldap]# ldapmodify -x -W -D "cn=Manager,dc=genome,dc=arizona,dc=edu" -f ldapreader.conf Enter LDAP Password: modifying entry "cn=ldapreader,dc=genome,dc=arizona,dc=edu" [root@node2 openldap]# Then I updated the add-replication-node1/node2.ldif to modify the entry with the actual password instead of "secret"... on node1 i got two success messages, [root@node1 openldap]# cat add-replication-node1.ldif dn: olcDatabase={1}bdb,cn=config changetype: modify replace: olcSyncrepl olcSyncrepl: rid=001 provider=ldap://node2.genome.arizona.edu binddn="cn=ldapreader,dc=genome,dc=arizona,dc=edu" bindmethod=simple credentials="<actual password>" searchbase="dc=genome,dc=arizona,dc=edu" type=refreshAndPersist timeout=0 network-timeout=0 retry="60 +" dn: olcDatabase={1}bdb,cn=config changetype: modify replace: olcMirrorMode olcMirrorMode: TRUE [root@node1 openldap]# ldapmodify -x -W -D "cn=Manager,dc=genome,dc=arizona,dc=edu" -f add-replication-node1.ldif Enter LDAP Password: modifying entry "olcDatabase={1}bdb,cn=config" modifying entry "olcDatabase={1}bdb,cn=config" [root@node1 openldap]# However when I went to modify the entries on node2, I now got the error <olcMirrorMode> database is not a shadow, [root@node2 openldap]# cat add-replication-node2.ldif dn: olcDatabase={1}bdb,cn=config changetype: modify replace: olcSyncrepl olcSyncrepl: rid=002 provider=ldap://node1.genome.arizona.edu binddn="cn=ldapreader,dc=genome,dc=arizona,dc=edu" bindmethod=simple credentials="<actual password>" searchbase="dc=genome,dc=arizona,dc=edu" type=refreshAndPersist timeout=0 network-timeout=0 retry="60 +" dn: olcDatabase={1}bdb,cn=config changetype: modify replace: olcMirrorMode olcMirrorMode: TRUE [root@node2 openldap]# ldapmodify -x -W -D "cn=Manager,dc=genome,dc=arizona,dc=edu" -f add-replication-node2.ldif Enter LDAP Password: modifying entry "olcDatabase={1}bdb,cn=config" modifying entry "olcDatabase={1}bdb,cn=config" ldap_modify: Other (e.g., implementation specific) error (80) additional info: <olcMirrorMode> database is not a shadow [root@node2 openldap]# Now the replication has stopped and there are no connection entries in the ldap debug logs. So what did i do wrong and how to get replication going again? Thanks, -- Chandler / Systems Administrator Arizona Genomics Institute www.genome.arizona.edu

2 4

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical