openldap-technical

openldap-technical@openldap.org

14 participants
9795 discussions

Re: ldapdelete recursive (-r) with syncrepl
by Frank Crow 13 Apr '16

13 Apr '16

OK, thanks for the clarification and correction. Doing so makes the restore process slightly more of pain but, on the bright side, I won't need to worry about SIDs. So that nice. Also on the bright side, working through this I managed to move to delta-syncrepl which appears to be a better solution. And yeah, I will move to mdb if I can get these guys to bite on upgrading. Thanks for all the info and advice, Frank On Wed, Apr 13, 2016 at 12:29 PM, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > --On Wednesday, April 13, 2016 10:20 AM -0700 Quanah Gibson-Mount < > quanah(a)zimbra.com> wrote: > > --On Wednesday, April 13, 2016 1:16 PM -0400 Frank Crow >> <fjcrow2008(a)gmail.com> wrote: >> >> >>> Then just slapcat the data first, do your destructive tests, and then >>>> restore from slapadd. Doing repetitive ldap deletes/adds will just >>>> causes massive database growth. >>>> >>> >>> >>> OK, but I just want to be clear - I'm still not. Does that mean that >>> I need to nuke /var/lib/ldap on every master replica prior to restoring >>> the DIT? >>> >> >> Yes. >> > > Or I should say, depends on what you've done. If all masters are in a > "bad" state (which would be likely given your test scenario), then yes, you > want to reload them all from the same slapcat dump that was taken, so their > DBs are identical after restore. > > > --Quanah > > > -- > > Quanah Gibson-Mount > Platform Architect > Zimbra, Inc. > -------------------- > Zimbra :: the leader in open source messaging and collaboration > A division of Synacor, Inc > -- Frank

1 0

Re: ldapdelete recursive (-r) with syncrepl
by Frank Crow 13 Apr '16

13 Apr '16

> Then just slapcat the data first, do your destructive tests, and then restore from slapadd. Doing repetitive ldap deletes/adds will just causes massive database growth. OK, but I just want to be clear - I'm still not. Does that mean that I need to nuke /var/lib/ldap on every master replica prior to restoring the DIT? Also, another question, if I do that will I have to know the server ID (for "-w -S <SID>") in order to properly reload? I'm asking because then my testers need to know which master is which - and there is no guaranteed order at the moment with our configuration. Thanks, Frank On Wed, Apr 13, 2016 at 11:51 AM, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > --On Wednesday, April 13, 2016 12:29 PM -0400 Frank Crow < > fjcrow2008(a)gmail.com> wrote: > > >> Well, OK. Our formal test procedures (created and executed by a >> separate team) will intentionally do all manner of destructive changes to >> the DIT after taking a snapshot. After which they want to clean and >> reload the entire DIT in order to: 1) be able to run the tests multiple >> times with the same data, and 2) restore the DIT for normal (non-OpenLDAP >> related) testing that does need the DIT intact. >> > > Then just slapcat the data first, do your destructive tests, and then > restore from slapadd. Doing repetitive ldap deletes/adds will just causes > massive database growth. > > During production use, we would not be doing that. For that, I think >> the individual node repair (slapadd/slapcat) would be exactly what we >> need. >> >> >> I took your suggestion and forwarded the list of changes since 2.4.40 >> (with the syncrepl fixes highlighted) to the important people up the >> "flag pole" this morning. I am "fighting the good fight" with them but >> they are hesitant to move without strenuous "encouragement!" ;-) >> > > I guess it just depends on whether or not they consider data loss > acceptable. If they do find it acceptable, by all means, stay on 2.4.40. ;) > > Also not sure which database backend you're using, but I'd strongly advise > back-mdb once you get to 2.4.44. > > > --Quanah > > > -- > > Quanah Gibson-Mount > Platform Architect > Zimbra, Inc. > -------------------- > Zimbra :: the leader in open source messaging and collaboration > A division of Synacor, Inc > -- Frank

2 2

Re: ldapdelete recursive (-r) with syncrepl
by Frank Crow 13 Apr '16

13 Apr '16

Well, OK. Our formal test procedures (created and executed by a separate team) will intentionally do all manner of destructive changes to the DIT after taking a snapshot. After which they want to clean and reload the entire DIT in order to: 1) be able to run the tests multiple times with the same data, and 2) restore the DIT for normal (non-OpenLDAP related) testing that does need the DIT intact. During production use, we would not be doing that. For that, I think the individual node repair (slapadd/slapcat) would be exactly what we need. I took your suggestion and forwarded the list of changes since 2.4.40 (with the syncrepl fixes highlighted) to the important people up the "flag pole" this morning. I am "fighting the good fight" with them but they are hesitant to move without strenuous "encouragement!" ;-) Thanks, Frank On Wed, Apr 13, 2016 at 11:17 AM, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > --On Wednesday, April 13, 2016 11:51 AM -0400 Frank Crow < > fjcrow2008(a)gmail.com> wrote: > > >> I did see that and do know how to do slapadd/slapcat backups. What >> I'm not clear on is how that works with N-Way MMR. Wouldn't I have >> to go and delete the the /var/lib/ldap on every master replica machine >> prior to loading the backup with slapadd? Or is that not necessary? >> > > Is your intent to drop the database and reload it on every master? That > would seem to indicate you've got significant issues with how you manage > your servers. > > You can trivially recover any one given server with the backup from > another MMR node. I.e., say you want to take down node 3... slapcat node1 > or node2, and load that onto node 3... > > Again, more information on what it is you're really trying to do here > would be useful. > > > In any case, I did update my syncrepl to use cn=accesslog and I no longer >> have any issues with bulk operations being propagated to the other master >> replicas. >> > > Good, delta-syncrepl's definitely better. But this doesn't resolve the > many replication problems that still exist in 2.4.40. > > > --Quanah > > -- > > Quanah Gibson-Mount > Platform Architect > Zimbra, Inc. > -------------------- > Zimbra :: the leader in open source messaging and collaboration > A division of Synacor, Inc > -- Frank

2 1

Re: ldapdelete recursive (-r) with syncrepl
by Frank Crow 13 Apr '16

13 Apr '16

I did see that and do know how to do slapadd/slapcat backups. What I'm not clear on is how that works with N-Way MMR. Wouldn't I have to go and delete the the /var/lib/ldap on every master replica machine prior to loading the backup with slapadd? Or is that not necessary? In any case, I did update my syncrepl to use cn=accesslog and I no longer have any issues with bulk operations being propagated to the other master replicas. Thanks, Frank On Wed, Apr 13, 2016 at 2:15 AM, Michael Ströder <michael(a)stroeder.com> wrote: > Please stay on the list so others can answer and learn as well. > > > Frank Crow wrote: > > OK, if I do a backup with slapcat, I still would want to wipe the > existing > > contents of the DIT first, right? > > That's why I wrote... > > > On Tue, Apr 12, 2016 at 5:47 PM, Michael Ströder <michael(a)stroeder.com> > > wrote: > > > >> Frank Crow wrote: > >>> I'm trying to create backup and restore scripts using LDAP command line > >>> tools. > >> > >> For various reasons backup and restore should be done with command-line > >> tools > >> slapcat and slapadd which operate directly on the database files. > >> > >> And yes, with recent backend modules like back-mdb and back-hdb you can > do > >> hot > >> backup while slapd is running. > >> > >> Of course, before a restore you have to stop slapd and remove the DB > files. > > ^^^^^^^^^^^^^^^^^^^ > ...to remove the database files. > > >> After using slapadd you should check whether ownership/permissions are > >> still > >> correct. > > Please read the responses carefully. > > Ciao, Michael. > > -- Frank

2 1

Re: ldapdelete recursive (-r) with syncrepl
by Frank Crow 13 Apr '16

13 Apr '16

Well, I can strongly suggest that we upgrade past that but I know that I'll get push back on it. I'll change the syncrepl to use refreshAndPersist. Thanks, Frank On Tue, Apr 12, 2016 at 7:04 PM, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > --On Tuesday, April 12, 2016 7:33 PM -0400 Frank Crow < > fjcrow2008(a)gmail.com> wrote: > > >> OpenLDAP 2.4.40 >> > > Upgrade. There are serious MMR issues in that release. > > > Syncrepl configuration: >> >> olcSyncUseSubentry: FALSE >> olcSyncrepl: {0}rid=101 provider=ldap://server1 >> searchbase="o=xxx,dc=yyy, >> dc=zzz" type=refreshOnly bindmethod=sasl saslmech=EXTERNAL >> > > I strongly advise against using refreshOnly. There's virtually no > instance where that is the correct option. > > > --Quanah > > > -- > > Quanah Gibson-Mount > Platform Architect > Zimbra, Inc. > -------------------- > Zimbra :: the leader in open source messaging and collaboration > A division of Synacor, Inc > -- Frank

2 1

Re: ldapdelete recursive (-r) with syncrepl
by Frank Crow 13 Apr '16

13 Apr '16

OpenLDAP 2.4.40 Syncrepl configuration: olcSyncUseSubentry: FALSE > olcSyncrepl: {0}rid=101 provider=ldap://server1 searchbase="o=xxx,dc=yyy, > dc=zzz" type=refreshOnly bindmethod=sasl saslmech=EXTERNAL > tls_cert=/etc/openldap/certs/xxxxx.crt > tls_key=/etc/openldap/certs/xxxxx.key > tls_cacert=/etc/openldap/certs/cacert.pem interval=00:00:00:10 > retry="5 10 10 10 30 +" timeout=1 starttls=critical > olcSyncrepl: {1}rid=102 provider=ldap://server2 > searchbase="o=xxx,dc=yyyy, > dc=zzz" type=refreshOnly bindmethod=sasl saslmech=EXTERNAL > tls_cert=/etc/openldap/certs/ldapadmin.crt > tls_key=/etc/openldap/certs/xxxxx.key > tls_cacert=/etc/openldap/certs/cacert.pem interval=00:00:00:10 > retry="5 10 10 10 30 +" timeout=1 starttls=critical > olcMirrorMode: TRUE BTW, I just tried addinging: dn: olcOverly={3}syncprov,olcDatabase={2},cn=config > changetype: modify > replace: olcSpCheckpoint > olcSpCheckpoint: 1024 > - > add: olcSpSessionlog > olcSpSessionlog: 1024 > - > add: olcSpReloadhint > olcSpReloadhint: TRUE And that seemed to fix it! Maybe it was just the checkpoint being "1 1" that was messing it up? Or maybe I needed the session log. I realize that this is the deprecated approach. I probably put in cn=changelog instead if there's a good reason to do so. -Frank On Tue, Apr 12, 2016 at 6:26 PM, Frank Crow <fjcrow2008(a)gmail.com> wrote: > OK, if I do a backup with slapcat, I still would want to wipe the existing > contents of the DIT first, right? > > Also, I just tried doing a list of deleted uid entries using "ldapdelete > -ZZ -f /file.ldif" and although the command did not complain, not all of > the entries in the file.ldif were deleted from all replicas. I really > think there is something wrong with my configuration! I suppose that I'll > try cn=changelog next. > > Thanks, > Frank > > > On Tue, Apr 12, 2016 at 5:47 PM, Michael Ströder <michael(a)stroeder.com> > wrote: > >> Frank Crow wrote: >> > I'm trying to create backup and restore scripts using LDAP command line >> > tools. >> >> For various reasons backup and restore should be done with command-line >> tools >> slapcat and slapadd which operate directly on the database files. >> >> And yes, with recent backend modules like back-mdb and back-hdb you can >> do hot >> backup while slapd is running. >> >> Of course, before a restore you have to stop slapd and remove the DB >> files. >> After using slapadd you should check whether ownership/permissions are >> still >> correct. >> >> Ciao, Michael. >> >> > > > -- > Frank > -- Frank

2 1

Re: ldapdelete recursive (-r) with syncrepl
by Frank Crow 13 Apr '16

13 Apr '16

Oops... sorry... had inadvertently used "reply" rather than "reply all"! Seems that if I used "ldapdelete -ZZ <dn>" on a single entry e.g., "uid=someuser,ou=Users,o=xxx,dc=yyy,dc=zzz" that propagates fine. But if I delete the parent in bulk "ldapdelete -ZZ ou=Users,o=xxx,dc=yyy,dc=zzz" then none of the "delete entry" actions get propagated to the other masters. They still have the entire tree. Is that the expected behavior? Should I only ever delete children nodes and not containers? Thanks, Frank On Tue, Apr 12, 2016 at 5:33 PM, Frank Crow <fjcrow2008(a)gmail.com> wrote: > The interesting thing to me, is that a single "ldapdelete -ZZ <dn>" works > and propagates no problem. That's really why I'm wondering if there is > something non-obvious about "ldapdelete -ZZ *-r* <dn>" (recursive). Or > is it that there is some issue in my configuration? > > Well, anyway, my test is going to do "higher up the tree" bulk deletes and > see if I have any different results. > > Thanks, > Frank > > > On Tue, Apr 12, 2016 at 4:49 PM, Frank Crow <fjcrow2008(a)gmail.com> wrote: > >> I'm trying to use ldapsearch and ldapadd for backup and restore with >> N-Way MMR on the backend. Maybe it's just my MMR config that's incorrect?? >> >> And I still need to have "delete entry" propagate to all masters. We >> have formal tests that will be exercising that capability with our users, >> roles, perms, etc. We have GUIs that add/change/delete entries for >> administration, etc. I can even expect "bulk" deletes from those GUIs. >> >> I'll be doing various experiments to change the replication parameters >> and probably add a session log, etc. I was just hoping that someone could >> point me to how/where I can better understand this. (I have and am >> reading the Admin Guide, man pages, and even "Mastering LDAP", etc., just >> BTW.) I must have missed something somewhere. >> >> Thanks, >> Frank >> >> >> >> On Tue, Apr 12, 2016 at 4:11 PM, Quanah Gibson-Mount <quanah(a)zimbra.com> >> wrote: >> >>> --On Tuesday, April 12, 2016 4:49 PM -0400 Frank Crow < >>> fjcrow2008(a)gmail.com> wrote: >>> >>> >>>> I'm trying to create backup and restore scripts using LDAP command line >>>> tools. Creating a suitable backup and loading it are no problem. >>>> However, if I attempt to: >>>> >>> >>> Generally one does backup and restore using slapcat & slapadd. What are >>> you trying to accomplish? >>> >>> --Quanah >>> >>> >>> -- >>> >>> Quanah Gibson-Mount >>> Platform Architect >>> Zimbra, Inc. >>> -------------------- >>> Zimbra :: the leader in open source messaging and collaboration >>> A division of Synacor, Inc >>> >> >> >> >> -- >> Frank >> > > > > -- > Frank > -- Frank

2 1

ldapdelete recursive (-r) with syncrepl
by Frank Crow 12 Apr '16

12 Apr '16

I'm trying to create backup and restore scripts using LDAP command line tools. Creating a suitable backup and loading it are no problem. However, if I attempt to: # ldapdelete -ZZ -r o=xxx,dc=yyyy,dc=zzz It appears to work fine on the master replica where the command was run, but the other (MMR) master still maintains the entire DIT fragment. It doesn't ever seem to delete any of the entries. Is there anything unique about the (-r) recursive ldapdelete? Is this behavior unique to "refreshOnly"? Should I use "refreshAndPersist"? Or do I need to implement a session log? Thanks for any help or advice, -- Frank

3 2

Re: slapd ACL - limit bind to employeeType=<various>
by Tim Watts 12 Apr '16

12 Apr '16

On 12/04/16 18:32, Michael Ströder wrote: > Tim Watts wrote: >> "Delete" - they vanish from libnss and other places, but we still hold their >> LDAP record for easy resurrection (this happens a lot - we have a class of >> nomadic users - they work on a project, go away, then come back on another >> project 2 years later). > > That state I'd call "inactive" or similar. But that's cosmetic. > > In my systems there's always exactly one status "active" for which I allow > "auth" on "userPassword". > > E.g. in Æ-DIR the attribute 'aeStatus' can have this Integer values: > > -1: requested > 0: active > 1: deactivated > 2: archived > > I suspect you're overloading the semantics of 'employeeType' by putting two > meaning into one attribute. > > Ciao, Michael. > Yes - I think I am. It was partly convenience (I'm using the LAM web front end to manage, with ldapvi where LAM cannot cope (odd problems with empty mail attributes). But also I did not think of your idea - it's pretty cool. I'd have to admit to being more of an LDAP n00b (we populated /etc/passwd directly from postgresql in my last place). -- Tim Watts Tel (VOIP): +44 (0)1580 848360 Systems Manager Kings Digital Lab (KDL), King's College London Systems Messages and Notifications: https://systemsblog.cch.kcl.ac.uk/

1 0

Re: slapd ACL - limit bind to employeeType=<various>
by Tim Watts 12 Apr '16

12 Apr '16

Sir, you are a genius :) On 11/04/16 07:31, Michael Ströder wrote: > # some entries matching filter > access to > attrs=userPassword > filter=(!(employeeType=Archive)(employeeType=Delete)) > by ..some who clauses for setting password > by * auth > > # all other entries > access to > attrs=userPassword > by * none Very slight tweak to the syntax (with huge thanks - I would not have guessed this was the required technique - I was concentrating on finding an "auth" ACL when I was googling.) This is the full ACL list for context, should anyone find this useful. The nice thing is that I can add new employeeTypes (eg "BA" for undergrads) without breaking the core logic. Also, this should be fundamentally more secure as we can now disable accounts in one place. In case anyone is wondering the difference between "Archive" and "Delete", they are as follows: "Archive" - keep the user in the libnss lists (eg getent passwd still returns them, so that file UIDs resolve). User cannot log in. "Delete" - they vanish from libnss and other places, but we still hold their LDAP record for easy resurrection (this happens a lot - we have a class of nomadic users - they work on a project, go away, then come back on another project 2 years later). =========================================== access to attrs=userPassword filter=(&(!(employeeType=Archive))(!(employeeType=Delete))) by peername.path="/var/run/slapd/ldapi" manage by set="user/uid & [cn=sysadmin,ou=groups,dc=dighum,dc=kcl,dc=ac,dc=uk ]/memberUid" manage by self write by * auth access to attrs=userPassword by * none # Certain attributes that should not be publicly readable access to attrs=bindTimestamp,modifyTimestamp,modifiersName,creatorsName,creat eTimestamp by peername.path="/var/run/slapd/ldapi" manage by set="user/uid & [cn=sysadmin,ou=groups,dc=dighum,dc=kcl,dc=ac,dc=uk ]/memberUid" manage by self read by * none # The admin dn has full write access, everyone else # can read everything. Local unix domain socket (root only) # Can do everything access to * by peername.path="/var/run/slapd/ldapi" manage by set="user/uid & [cn=sysadmin,ou=groups,dc=dighum,dc=kcl,dc=ac,dc=uk ]/memberUid" manage by * read =========================================== I'll do some more testing and have a look at that "set=" inefficiency too. Thanks very much for your help :) Tim -- Tim Watts Tel (VOIP): +44 (0)1580 848360 Systems Manager Kings Digital Lab (KDL), King's College London Systems Messages and Notifications: https://systemsblog.cch.kcl.ac.uk/

2 4

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical