openldap-technical

openldap-technical@openldap.org

9937 discussions

contextCSN attribute update on replication
by Óscar Remírez de Ganuza Satrústegui 04 Sep '16

04 Sep '16

Good morning, I am writting from IT Services from Universidad de Navarra. We have recently upgraded our openldap servers from openldap 2.4.34 with BDB 5.3.21 to openldap 2.4.44 with MDB databases. We have got configured replication from the master server [1] to some slave servers [2] (syncrepl refreshAndPersist), and it is working ok. Usually, when a change is made on master server, I can see how it is propagated and applied on the slave server. Using Auditlog Overlay I can see on the slave server: # modify 1470723918 dc=base,dc=com cn=Admin,dc=base,dc=com conn=-1 dn: ... changetype: modify replace: [..] # end modify 1470723918 And just after that, the contextCSN gets updated too: # modify 1470723918 dc=base,dc=com cn=Admin,dc=base,dc=com conn=-1 dn: dc=base,dc=com changetype: modify replace: contextCSN contextCSN: 20160809062518.877725Z#000000#000#000000 - # end modify 1470723918 Is this the normal behaviour? I do not see the contextCSN update on the accesslog database on the master server, nor on his Auditlog. So I do not know if contextCSN has been replicated from the master server, or the slave database is updating it. But I am seeing some weird things from time to time: sometimes, somehow, the contextCSN attribute does not get updated after the modification. Checking its value in the master server, I can see that it has been updated correctly, but not on the slave server. The strange thing is that it happens just like once every tens of changes. Could it be some kind of bad configuration?? On the previous openldap version, we were checking contextCSN value on master and slave servers in order to check the replication status. But right now, although replication is working ok, sometimes the contextCSN does not get updated on the slave sever, so we can not use it in order to check the replication status. Thank you so much for your help. Regards, [1] Master: * Accesslog Database: database mdb maxsize 1073741824 suffix cn=log directory /../openldap/var/accesslog rootdn "cn=Admin,dc=base,dc=com" index objectClass eq index entryCSN eq index reqEnd eq index reqResult eq index reqStart eq index reqDN eq index default eq overlay syncprov syncprov-reloadhint true syncprov-nopresent true * Main Database overlays: overlay syncprov syncprov-checkpoint 1000 60 overlay accesslog logdb cn=log logops writes logsuccess true logpurge 14+00:00 01+00:00 [2] Slave: syncrepl rid=1 provider="ldap://ldap-master.base.com:389/" type=refreshAndPersist retry="60 10 300 +" searchbase="dc=base,dc=com" logbase="cn=log" syncdata=accesslog logfilter="(&(objectClass=auditWriteObject)(reqResult=0))" scope=sub schemachecking=off binddn=... *Oscar Remírez de Ganuza Satrústegui* IT Services Universidad de Navarra Tel. +34 948425600 x803130 http://www.unav.edu/web/it/

3 9

openldap stops responding after some time
by Daniel Betz 31 Aug '16

31 Aug '16

Hello list, i hope you can help me with my problem. To my setup: All servers are OpenLDAP 2.4.42 I have an master LDAP server, which replicates with standard syncrepl to an consumer ldap. On this consumer ldap server i have configured an standalone slapd proxy ldap with slapd-ldap which pushes changes to more than 6000 consumer ldaps. There are more ldap proxys running, with each 500 consumers to reduce startup time. The master and slave are connected via TCP, and the ldap proxys are on the slave via socket. Everything works fine and changes are replicated in realtime to the consumers behind the proxy, but after some time ( about 20 to 30 minutes ) the slave ldap just hangs and isnt responding anymore. A short time before it hangs the changes are pushed with an long delay, before it hangs fully. With debug on ( -d256 ) everything looks fine and no error is displayed, but it hangs. I have tested the standard syncrepl and delta syncrepl with the same result. When strace the process there are only many futex_wait() While i write this mail the error doesnt occur, so i am not able to paste an strace. So then. Has anyone an idea to this problem or an better solution for my setup ? Any hints to debug this, or some tips and tricks would be really nice. Here are the relevant configuration settings of all servers: ## all ldap servers are started with extended limits in systemd LimitCORE=0 LimitNPROC=5000000 LimitNOFILE=65535 LimitSTACK=81920 LimitDATA=infinity LimitMEMLOCK=infinity LimitRSS=infinity LimitAS=infinity and: echo 5000000 > /proc/sys/kernel/threads-max Cause limits in openldap itself i have patched it too: diff -rNu openldap-2.4.42.orig/libraries/libldap_r/tpool.c openldap-2.4.42/libraries/libldap_r/tpool.c --- openldap-2.4.42.orig/libraries/libldap_r/tpool.c 2015-08-31 08:26:55.000000000 +0200 +++ openldap-2.4.42/libraries/libldap_r/tpool.c 2015-08-31 07:39:25.000000000 +0200 @@ -42,10 +42,10 @@ /* Max number of thread-specific keys we store per thread. * We don't expect to use many... */ -#define MAXKEYS 32 +#define MAXKEYS 65535 /* Max number of threads */ -#define LDAP_MAXTHR 1024 /* must be a power of 2 */ +#define LDAP_MAXTHR 65535 /* must be a power of 2 */ /* (Theoretical) max number of pending requests */ #define MAX_PENDING (INT_MAX/2) /* INT_MAX - (room to avoid overflow) */ diff -rNu openldap-2.4.42.orig/servers/slapd/daemon.c openldap-2.4.42/servers/slapd/daemon.c --- openldap-2.4.42.orig/servers/slapd/daemon.c 2015-08-31 08:25:42.000000000 +0200 +++ openldap-2.4.42/servers/slapd/daemon.c 2015-08-31 07:42:02.000000000 +0200 @@ -1635,6 +1635,7 @@ #else /* ! HAVE_SYSCONF && ! HAVE_GETDTABLESIZE */ dtblsize = FD_SETSIZE; #endif /* ! HAVE_SYSCONF && ! HAVE_GETDTABLESIZE */ + dtblsize=8192; /* open a pipe (or something equivalent connected to itself). * we write a byte on this fd whenever we catch a signal. The main And raised the max integer numbers of syncrepl´s rid= ### Master ########### loglevel 0 sizelimit unlimited database mdb suffix "o=company, c=de" rootdn "cn=Manager,o=company,c=de" rootpw "xxxxxxxxxxxxxxxxxxxxxxxx" overlay syncprov syncprov-checkpoint 100 10 syncprov-sessionlog 1000 index DFan,DFname,uid,uidNumber,gidNumber,DFCronjobID eq index entryUUID,entryCSN eq index objectClass eq directory /var/lib/ldap/openldap-mdb maxsize 8500000000 #### Slave ################## loglevel 0 threads 2048 database mdb suffix "o=company, c=de" rootdn "cn=Manager,o=company,c=de" rootpw "xxxxxxxxxxxxxxxxxxxx" # here are all consumer ldap servers one by one access to dn.subtree="sid=240,sec=webhosting,o=company,c=de" by dn.exact="cn=replicator,sid=240,sec=webhosting,o=company,c=de" write by * auth access to dn.subtree="sid=241,sec=webhosting,o=company,c=de" by dn.exact="cn=replicator,sid=241,sec=webhosting,o=company,c=de" write by * auth access to dn.subtree="sid=242,sec=webhosting,o=company,c=de" by dn.exact="cn=replicator,sid=242,sec=webhosting,o=company,c=de" write by * auth ... ... index DFan,DFname,uid,uidNumber,gidNumber,DFCronjobID eq index entryUUID,entryCSN eq index objectClass eq directory /var/lib/ldap/openldap-mdb syncrepl rid=001 provider=ldaps://ldapmaster:636/ binddn="cn=Manager,o=company,c=de" bindmethod=simple credentials=xxxxxxxxxxxxxxxxxxxxxx searchbase="o=company,c=de" type=refreshAndPersist retry="5 5 300 5" overlay syncprov syncprov-checkpoint 1000 60 maxsize 8500000000 maxreaders 12000 ##### SLAPD Proxy ##################### database ldap hidden on suffix "sid=240,sec=webhosting,o=company,c=de" rootdn "cn=replicator,sid=240,sec=webhosting,o=company,c=de" uri ldaps://sid240.int.webslave.company.de:636 lastmod on restrict all acl-bind bindmethod=simple binddn="cn=replicator,sid=240,sec=webhosting,o=company,c=de" credentials="xxxxxxxxxxxxxxxxxxxxx" syncrepl rid=001 provider=ldapi:// binddn="cn=Manager,o=company,c=de" bindmethod=simple credentials=xxxxxxxxxxxxxxxxxxxxxxxxx searchbase="sid=240,sec=webhosting,o=company,c=de" type=refreshAndPersist retry="5 5 300 5" overlay syncprov # next one database ldap hidden on suffix "sid=241,sec=webhosting,o=company,c=de" rootdn "cn=replicator,sid=241,sec=webhosting,o=company,c=de" uri ldaps://sid241.int.webslave.company.de:636 lastmod on restrict all acl-bind bindmethod=simple binddn="cn=replicator,sid=241,sec=webhosting,o=company,c=de" credentials="xxxxxxxxxxxxxxxxxxxxx" syncrepl rid=001 provider=ldapi:// binddn="cn=Manager,o=company,c=de" bindmethod=simple credentials=xxxxxxxxxxxxxxxxxxxxxxxxx searchbase="sid=241,sec=webhosting,o=company,c=de" type=refreshAndPersist retry="5 5 300 5" overlay syncprov ... #### and the 6300 consumers on the end ############### database mdb suffix "sid=240,sec=webhosting,o=company,c=de" rootdn "cn=replicator,sid=240,sec=webhosting,o=company,c=de" rootpw {SSHA}xxxxxxxxxxxxxxxx index DFan,DFname,DFdnumber,sid,uid,uidNumber,gidNumber,DFCronjobID eq index objectClass eq index entryUUID,entryCSN eq directory /var/lib/ldap/openldap-mdb/sid240 updatedn "cn=replicator,sid=240,sec=webhosting,o=company,c=de" maxsize 1073741824 subordinate updateref ldaps://ldapmaster:636 database mdb suffix "o=company,c=de" rootdn "cn=Manager,o=company,c=de" rootpw {SSHA}xxxxxxxxxxxxxxxx index objectClass eq directory /var/lib/ldap/openldap-mdb/rest Regards, Daniel Betz System Design Engineer / Senior Systemadministration ___________________________________ domainfactory GmbH Oskar-Messter-Str. 33 85737 Ismaning Germany Telefon: +49 (0)89 / 55266-364 Telefax: +49 (0)89 / 55266-222 E-Mail: dbetz(a)df.eu Internet: www.df.eu Registergericht: Amtsgericht München HRB-Nummer 150294, Geschäftsführer: Tobias Mohr, Stephan Wolfram

2 2

nslcd listing users and groups twice
by John Lewis 28 Aug '16

28 Aug '16

This is surprisingly non-trivial especially when the nis schema for openldap is more documented than the samba one when I use to run samba-ad-dc. I have the nslcd.conf attatched.

2 5

rootDN problems with slapd-config
by Dave Schneider 28 Aug '16

28 Aug '16

I'm having problems getting the rootDN working when using slapd-config form of configuration, while the "exact" same configuration using slapd.conf works fine. Here are my stripped down test versions of the two configurations (hashed password is 'secret' from slappasswd): slapd.d/cn=e2config.ldif: ------------------------- dn: cn=config objectClass: olcGlobal cn: config olcArgsFile: /var/openldap/run/slapd.args olcPidFile: /var/openldap/run/slapd.pid dn: cn=schema,cn=config objectClass: olcSchemaConfig cn: schema include: file:///var/openldap/schema/core.ldif include: file:///var/openldap/schema/cosine.ldif dn: olcDatabase=frontend,cn=config objectClass: olcDatabaseConfig objectClass: olcFrontendConfig olcDatabase: frontend dn: olcDatabase=mdb,cn=config objectClass: olcDatabaseConfig objectClass: olcMdbConfig olcDatabase: mdb olcMaxsize: 1073741824 olcSuffix: dc=com olcRootDN: cn=reader,dc=com olcRootPW: {SSHA}RIC5hnBuWr4t857KR+dzTEOF/ekQaIVx olcDbDirectory: /var/openldap/data olcDbIndex: objectClass eq olcDbIndex: dc eq olcDbIndex: cn eq slapd.conf: ----------- include /var/openldap/schema/core.schema include /var/openldap/schema/cosine.schema backend mdb pidfile /var/openldap/run/slapd.pid argsfile /var/openldap/run/slapd.args database mdb maxsize 1073741824 suffix dc=com rootdn cn=reader,dc=com rootpw {SSHA}RIC5hnBuWr4t857KR+dzTEOF/ekQaIVx directory /var/openldap/data index objectClass eq index dc eq index cn eq When I run a simple search for slapd running with the slapd.conf configuration I get: $ ldapsearch -D cn=reader,dc=com -w secret -x -LLL -b dc=com -s base dc=* dn: dc=com objectClass: top objectClass: domain dc: com But when I run the same search with the slapd.d configuration I get: $ ldapsearch -D cn=reader,dc=com -w secret -x -LLL -b dc=com -s base dc=* ldap_bind: Invalid DN syntax (34) additional info: invalid DN Debug output on the server side isn't giving much info in addition to what's already displayed on the client: 57bf52df conn=1000 op=0 do_bind: invalid dn (cn=reader,dc=com) Any help on what I might be doing wrong is greatly appreciated. Oh yeah, I'm using version 2.4.44. Thanks, Dave

3 7

Re: libldap and libldap_r in the same process
by Howard Chu 28 Aug '16

28 Aug '16

Jan Engelhardt wrote: > > On Sunday 2016-08-28 15:03, Howard Chu wrote: >>> * Should Linux distributions make libldap and libldap_r be the same thing? >> >> I'm wary of doing that, but I suppose thread support on Linux today is a lot >> more stable than it was 7-15 years ago. > > The question really just is, will the openldap project combine ldap > and ldap_r, or do distributions have to copy the Debian-style solution? The OpenLDAP Project will not make such a change in the 2.x release family. What distributions decide to do is up to them. There are still embedded devices that use libldap, and have no thread support. We have no reason to make it harder for them to build their own projects. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: libldap and libldap_r in the same process
by Howard Chu 28 Aug '16

28 Aug '16

Jan Engelhardt wrote: > > A given (threaded) program makes use of libldap_r. It now also wants to > make use of, say, libcurl. On certain systems, libcurl happens to be > linked against libldap (not _r). > > Eventually, both libldap and libldap_r end up in the process image of my > program, and a symbol like "ldap_init" may point to the function of > either library file. The threaded program may end up unexpectedly been > given the non-threaded functions, depending on how the runtime linker > loaded said libraries and binds the symbols. (I am seeing crashes in a > large program's initialization because of this.) > > > There is a 7-year old post indicating libldap should get locking/ > be replaced by libldap_r. > http://www.openldap.org/lists/openldap-devel/200909/msg00018.html > To date, openldap still consists of the two libraries. > What are the preferred options to go forward here? > > * Killing non-threaded libldap in openldap for real > If not, > * Should curl have used -lldap_r instead of -lldap? Is curl already multithreaded? On Ubuntu curl is built with libldap_r already. If curl is a threaded program then yes, it should use libldap_r. > * Should Linux distributions make libldap and libldap_r be the same thing? I'm wary of doing that, but I suppose thread support on Linux today is a lot more stable than it was 7-15 years ago. > * Should libldap and libldap_r be augmented by ELF symbol versions? > * All of the above? :) -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

libldap and libldap_r in the same process
by Jan Engelhardt 27 Aug '16

27 Aug '16

A given (threaded) program makes use of libldap_r. It now also wants to make use of, say, libcurl. On certain systems, libcurl happens to be linked against libldap (not _r). Eventually, both libldap and libldap_r end up in the process image of my program, and a symbol like "ldap_init" may point to the function of either library file. The threaded program may end up unexpectedly been given the non-threaded functions, depending on how the runtime linker loaded said libraries and binds the symbols. (I am seeing crashes in a large program's initialization because of this.) There is a 7-year old post indicating libldap should get locking/ be replaced by libldap_r. http://www.openldap.org/lists/openldap-devel/200909/msg00018.html To date, openldap still consists of the two libraries. What are the preferred options to go forward here? * Killing non-threaded libldap in openldap for real If not, * Should curl have used -lldap_r instead of -lldap? * Should Linux distributions make libldap and libldap_r be the same thing? * Should libldap and libldap_r be augmented by ELF symbol versions? * All of the above? :)

1 0

Re: Using LMDB safely with fork() and exec()
by Howard Chu 23 Aug '16

23 Aug '16

Lorenz Bauer wrote: > If this works, I'd like to contribute the changes necessary to not > leak fds on exec, which are mentioned in the other thread. > > * Are there contribution guidelines somewhere? How do I submit a patch? Read http://www.openldap.org/ "Contributing" -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

Re: mirror mode vs multi master
by Howard Chu 23 Aug '16

23 Aug '16

Matwey V. Kornilov wrote: > Hello, > >>From Administrator's guide, it is not completely clear what is the > internal difference between mirror mode and n-way multi master. There is no internal difference. The difference is in how clients use the servers. In mirrormode you arrange for all write ops to only go to one provider. > 1) Can 2-way multi master configuration exist? If so, what should be > difference in slapd.conf between mirror mode and 2-way multi master? There is no difference in slapd configuration. > What is the difference in syncrepl behaviour? Or all difference is > that external proxy/balancer assumed when mirror mode is meant? All difference is in the external proxy. > 2) Is there quorum based consensus when 3-way multi master is used? If > not, how is the conflict resolution performed? No quorums needed. Conflict resolution is based on CSNs. In general, last writer wins. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

mirror mode vs multi master
by Matwey V. Kornilov 23 Aug '16

23 Aug '16

Hello, >From Administrator's guide, it is not completely clear what is the internal difference between mirror mode and n-way multi master. 1) Can 2-way multi master configuration exist? If so, what should be difference in slapd.conf between mirror mode and 2-way multi master? What is the difference in syncrepl behaviour? Or all difference is that external proxy/balancer assumed when mirror mode is meant? 2) Is there quorum based consensus when 3-way multi master is used? If not, how is the conflict resolution performed? -- With best regards, Matwey V. Kornilov http://blog.matwey.name xmpp://0x2207@jabber.ru

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical