openldap-technical

openldap-technical@openldap.org

14 participants
9795 discussions

fast delete a lot of entries
by Derek Zhou 14 Dec '18

14 Dec '18

dear list, I was experimenting with slapd by adding a lot of entries then deleting them. What I found is adding speed is not bad but deletion speed is lacking. With or without dbsync, delete speed is around half of that of adding. Naively, I thought deleting should be easier than adding, because in adding you actually need to pass and write whole entry of data, while in delete you can just mark the database page as free? The DB file did not shrink after massive deletion; which kind of suggest that deleting is only marking page as free, not really return them to the OS. I am using the latest git tree and the mdb backend. Another related idea is delete a whole branch from the DIT. LDAP is already hierarchical, to delete all entries under a branch, one would assume that there must be an better way than deleting entries with a client side script, like what I am doing? With SQL you can drop a table. With LDAP, can I delete a whole branch? -- Derek Zhou Shannon Systems http://www.shannon-sys.com

3 6

Please help on Openldap issue
by Eggo why 13 Dec '18

13 Dec '18

Hi All, I have install openldap, openldap-servers and openldap-clients and systemctl start slapd without anyproblem but when try ti do an initial ldapadd I got error. My slapd.conf and oidbase.ldif as below. Please help and thank you very much Garry slapd.conf # # See slapd.conf(5) for details on configuration options. # This file should NOT be world readable. # include /etc/openldap/schema/core.schema include /etc/openldap/schema/cosine.schema include /etc/openldap/schema/inetorgperson.schema include /etc/openldap/schema/nis.schema include /etc/openldap/oidschema/oidbase.schema include /etc/openldap/oidschema/oidnet.schema include /etc/openldap/oidschema/oidrdbms.schema include /etc/openldap/oidschema/alias.schema # Allow LDAPv2 client connections. This is NOT the default. allow bind_v2 # Do not enable referrals until AFTER you have a working directory # service AND an understanding of referrals. #referral ldap://root.openldap.org pidfile /var/run/openldap/slapd.pid argsfile /var/run/openldap/slapd.args #default searchbase defaultsearchbase "dc=cornell,dc=edu" ####################################################################### # ldbm and/or bdb database definitions ####################################################################### database bdb suffix "dc=cornell,dc=edu" rootdn "cn=admin,dc=cornell,dc=edu" rootpw {SSHA}gZp2hnEA/xq/FfxMgPTjDmSPo78PuZAs # The database directory MUST exist prior to running slapd AND # should only be accessible by the slapd and slap tools. # Mode 700 recommended. directory /var/lib/ldap/cornell # Indices to maintain for this database index objectClass eq,pres index ou,cn,mail,surname,givenname eq,pres,sub index uidNumber,gidNumber,loginShell eq,pres index uid,memberUid eq,pres,sub index nisMapName,nisMapEntry eq,pres,sub ### for root oraclecontext # database bdb suffix "" rootdn "cn=admin" rootpw {SSHA}gZp2hnEA/xq/FfxMgPTjDmSPo78PuZAs # The database directory MUST exist prior to running slapd AND # should only be accessible by the slapd and slap tools. # Mode 700 recommended. directory /var/lib/ldap/rootoraclecontext # Indices to maintain for this database index objectClass eq,pres index ou,cn,mail,surname,givenname eq,pres,sub index uidNumber,gidNumber,loginShell eq,pres index uid,memberUid eq,pres,sub index nisMapName,nisMapEntry eq,pres,sub oidbase.ldif dn: dc=cornell,dc=edu objectClass: top objectClass: dcObject objectClass: organization o: cornell.edu dc: cornell dn: cn=OracleContext,dc=cornell,dc=edu objectclass: orclContext cn: OracleContext l$ ldapadd -vc -x -D “cn=admin,dc=cornell,dc=edu” -w $passwd -f oidbase.ldif ldap_initialize( <DEFAULT> ) ldap_bind: Invalid DN syntax (34) additional info: invalid DN

3 6

speed up parallel add/delete
by Derek Zhou 11 Dec '18

11 Dec '18

all I have a patch that can speed up parallel ldap add/delete a lot; in my testing about 2.5X. The idea is described here: the mdb backend does not really support parallelism in rw transactions, so add/delete are serialized by a mutex inside the mdb backend as small transactions. In my patch, all add/delete are put in a queue and executed in a dedicated worker thread. It may seem pointless at first, but by doing them all in the same thread, now I can merge concurrent ops from different clients into larger transactions, and reduce the number of expensive txn_commit calls (fsyncs). my test results are ( your mileage may vary) 10 thread parallel add: 1, unpatched slapd + default config, ~ 2800 op/s (performance stable over time) 2, unpatched slapd + dbnosync ~9000 op/s (lots of fluctuation due to background dirty page flush) 3, unpatched slapd + dpnosync + checkpoint every minute ~8000 op/s, with reduced fluctuation 4, patched slapd ~7300 op/s (fluctuation nearly gone) 2 is not a recommended config, because crashing or power lost will cause massive data lost. 3 is better, but still up to 1 minutes of data can be lost at crash. With this patch, I can achieve >90% of the performance of 3, with data durability as good as 1; or 2.5X performance of 1 with same data durability guaranty. The patch still need some tidy up; is this email list the place to send patches? -- Derek Zhou Shannon Systems http://www.shannon-sys.com

2 1

not be able to initialize ldap
by Eggo why 05 Dec '18

05 Dec '18

3 3

Re: On removing a duplicated ppolicy overlay
by Derek Zhou 05 Dec '18

05 Dec '18

On Monday, December 03, 2018 02:11:27 PM Quanah Gibson-Mount wrote: > --On Monday, December 03, 2018 1:57 PM -0800 Daniel Howard > <dannyman(a)toldme.com> wrote: > > > > A potentially minor improvement along these lines could be a very nice > > feature enhancement for OpenLDAP. Thank you for your consideration. > > You're talking about two things: a) slapmodify, which is slated for > release with OpenLDAP 2.5, which allows offline modifications of cn=config, > and b) delete support in cn=config, also slated for release with OpenLDAP > 2.5. slapmodify is nice, however modifying schema is still hard. The old schema file format is still much better than the ldif format for viewing and editing. It would be a nice to have a tool that can parse and diff schemas in the old format and generate ldif modifications for slapmodify or ldapmodify to handle. -- Derek Zhou Shannon Systems http://www.shannon-sys.com

1 0

Re: On removing a duplicated ppolicy overlay
by Daniel Howard 04 Dec '18

04 Dec '18

On Wed, Nov 28, 2018 at 11:05 AM Quanah Gibson-Mount <quanah(a)symas.com> wrote: > --On Wednesday, November 28, 2018 10:16 AM -0800 Daniel Howard > <dannyman(a)toldme.com> wrote: > > ># This file MUST be edited with the 'visudo' command as root. > > > > Perhaps this is a consideration that is already on the roadmap? > > You mean like it already does? :) > > head -1 cn\=config.ldif > # AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify. > Quanah, (and Howard) That is certainly a warning. It would be more easily noticed with a bit more whitespace around it, as you find in the sudoers file. This warning is not as helpful as the visudo warning, as it does not give the user a way to edit the file. A wrapper script, or a link to a good document on managing configurations, would be more useful. For example, if I follow the advice in sudoers, and I type "visudo" then I get to edit /etc/sudoers, with a validation before the file can go live. This is fantastic! If I follow the advice here, and type "ldapmodify" then I get an error message. If I research the correct command line, then, in my experience, I manage to import multiple conflicting configs into the system that crash the server. I then research some more, and find that ldapmodify can not delete the conflicting configs. I research some more, and learn that I could just remove them from the filesystem. As I wish to be a good citizen, I share this knowledge, and I am told that this is wrong, and I need to use slapcat to export, delete my config files, then slapadd to import, using a different set of flags than ldapmodify. Perhaps, you can spare a moment of empathy to acknowledge how frustrating this must be for a user. I appreciate your warnings, but given the cumbersome and, in my experience, dangerous nature of managing config files through ldapmodify, I am inclined to very carefully tweak the config files in the config directory. If, however, there was a convenient, safe wrapper, like visudo, or a reference to a reassuring doc that explains the right way to do things, then I would preach the good news. A potentially minor improvement along these lines could be a very nice feature enhancement for OpenLDAP. Thank you for your consideration. Sincerely, -danny -- http://dannyman.toldme.com

3 2

Re: On removing a duplicated ppolicy overlay
by Daniel Howard 04 Dec '18

04 Dec '18

On Mon, Dec 3, 2018 at 2:11 PM Quanah Gibson-Mount <quanah(a)symas.com> wrote: > --On Monday, December 03, 2018 1:57 PM -0800 Daniel Howard > <dannyman(a)toldme.com> wrote: > > > > A potentially minor improvement along these lines could be a very nice > > feature enhancement for OpenLDAP. Thank you for your consideration. > > You're talking about two things: a) slapmodify, which is slated for > release with OpenLDAP 2.5, which allows offline modifications of > cn=config, > and b) delete support in cn=config, also slated for release with OpenLDAP > 2.5. > I love these features already. ;) -- http://dannyman.toldme.com

1 0

syncrepl + rid
by A. Schulze 28 Nov '18

28 Nov '18

Hello, I run a N-Way Multi-Master setup (N=4) and multiple read-only syncrepl. I like to make sure I understand the "rid" correct. According to http://www.openldap.org/doc/admin24/guide.html#olcSyncrepl, the "rid" has to be unique per "slapd" So I would configure on /every host/ (4x Multi-Master and multiple read-only syncrepl) rid=1..4 to identify the other servers. The 4 Multi-Master server get 4 additional "ServerID 1..4 ldaps://multimaster1..4.example" Is that right? Thanks for clarification. Andreas

2 2

Re: On removing a duplicated ppolicy overlay
by Daniel Howard 28 Nov '18

28 Nov '18

On Tue, Nov 27, 2018 at 3:17 PM Quanah Gibson-Mount <quanah(a)symas.com> wrote: > --On Tuesday, November 27, 2018 2:22 PM -0800 Daniel Howard > <dannyman(a)toldme.com> wrote: > > > I had been yearning for a config file, and it turns out I had them all > > along! > > It's a database, not configuration files. Removing files from underneath > a > database is generally not a good idea, although YMMV. > > > I am sharing my experience here, for the next person who finds themselves > > googling around, trying to figure out how to remove or tweak a config in > > OpenLDAP. It is nowhere near as complicated as what I had read. > > This is the wrong advice. It is also fairly trivial to do what you > avoided. > > a) slapcat -n 0 -l /tmp/config.ldif > b) Remove the duplicate entries from /tmp/config.ldif > c) mv /path/to/current/config /path/to/current/config.old;mkdir -p > /path/to/current/config > d) slapadd -n 0 -l /tmp/config.ldif > I can see how a naive sysadmin might interpret the various text files in /etc/ldap/slapd.d/cn=config/ as configuration files ... that could be carefully edited by hand ... or managed programatically through the local configuration management system. I appreciate your admonition that this interpretation is wrong, and that these would-be "config" files in the system configuration file hierarchy are in fact a software-managed database, so we should not edit what appear to be plain text configuration files, but simply export them to a text file, carefully edit the export of the database, delete the entire config file hierarchy, and then reimport the database. If I may make a minor feature suggestion: whenever I get a file into /etc that needs a special workflow, I like to put warnings in the comments at the top of such files, advising that the file(s) shouldn't be edited by hand, and explaining the proper workflow. (The visudo command is a golden standard in this regard.) djh@djh-p5510 ~> sudo head -3 /etc/sudoers # # This file MUST be edited with the 'visudo' command as root. # Perhaps this is a consideration that is already on the roadmap? Thanks, -danny -- http://dannyman.toldme.com

3 2

On removing a duplicated ppolicy overlay
by Daniel Howard 28 Nov '18

28 Nov '18

Hello, Back in April or May, I was trying to add and tweak a password policy, invoking a command like this multiple times: sudo ldapmodify -Q -Y EXTERNAL -H ldapi:/// -a -f ppolicy-overlay.ldif This created multiple password policy overlays, and the LDAP server started to crash with some frequency. Of course, you can not use this interface to DELETE a policy overlay, so I went about researching "hot to remove a ppolicy overlay" and go into some complicated process where you have to export the database, remove olc* entries, delete your database and re-import. My attention shifted to other priorities. Yesterday, I turned back to the question of how to remove the duplicated ppolicy overlays and started exporting the database, but I couldn't find the ppolicy stuff in my slapcat output. Another trip to the search mines and I discovered this gem: https://serverfault.com/a/280836/72839 While you are asked to configure stuff using an LDAP command that cannot delete duplicate policy overlays, the config data doesn't get written into the database, but just placed in plain-text files in a directory structure. Removing duplicated overlays is as simple as stop slapd, remove the files, start slapd. Similarly, you could tweak your ppolicy overlay or possibly even bootstrap new servers by merely editing the right config files in the right place. I had been yearning for a config file, and it turns out I had them all along! I am sharing my experience here, for the next person who finds themselves googling around, trying to figure out how to remove or tweak a config in OpenLDAP. It is nowhere near as complicated as what I had read. Thanks, -danny -- http://dannyman.toldme.com

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical