syncrepl - when master restarted, slaves loose connection forever (and never reconnects)?
by Tomasz Chmielewski
I decided to use syncrepl (refreshAndPersist) instead of slurpd to
synchronize.
I made simple tests but something doesn't work right for me.
I started a totally empty OpenLDAP slave - it got replicated
from the master.
Any change made on the master is also replicated fine.
Unfortunately, when the connection between the slave and the master is
broken (i.e., master restarted), the slave never reconnects. According
to the fine slapd.conf manual, reconnection should be made quite fast:
"If the connection is lost, the consumer will attempt to reconnect
at an interval time (specified by interval parameter; 60 seconds by
default) until the session is re-established."
I waited for about 12 hours, and it didn't happen.
Restarting slave helped, and all pending changes were transferred from
the master.
Am I missing a setting or something?
The slave is running slapd from OpenLDAP 2.3.37, and here is its
slapd.conf part concerning syncrepl:
syncrepl rid=0
provider=ldap://master:389
# interval=00:00:00:30 <- also didn't help if it's 30 secs
type=refreshAndPersist
searchbase="dc=some,dc=domain"
bindmethod=simple
binddn="cn=replicationuser,dc=some,dc=domain"
credentials=secret
schemachecking=off
--
Tomasz Chmielewski
http://wpkg.org
14 years, 10 months
listening on localhost only
by Alex Girchenko
Hello.
I want to make slapd listen on localhost only. Is there such a
configuration option?
Thanks.
14 years, 10 months
slapd grows during SASL Binds
by Paul Turgyan
We've seen the slapd's on our mail slaves trying to grow greater
than the 2 gigabyte resident set size. I spent two days
w/ valgrind looking for memory leaks, but I failed to find anything.
I finally discovered that repeatedly binding and unbinding using a SASL/
GSSAPI bind would cause slapd's resident set size and vm size to
grow. So I wrote
a test program to SASL Bind, issue a search, and unbind. The test
program would
do this sequence 10000 times before it stopped. The test program
always issues
the same exact search trying to eliminate any interferance from a
growing entry cache
and/or idlcache.
I also tried this test w/ OpenLDAP 2.3.30 and got similar results.
Initial program size for slapd RSS = 8392 VMSIZE = 180180
"Bind, search, unbind" memory usage.
Searches Anonymous Bind SASL/GSSAPI Bind
Completed RSS VMSIZE RSS VMSIZE
10000 9320 182760 12560 184916
20000 9320 182760 15404 187988
30000 9376 187888 18268 196188
40000 9376 187888 21080 199260
50000 9380 187888 23896 201308
60000 9380 187888 26708 204380
70000 9380 187888 29524 207452
80000 9380 187888 32336 210524
90000 9380 187888 35148 212572
100000 9380 187888 37972 215644
And when I only bind once for every 10,00 searches, I get a
slapd memory usage like:
Searches Anonymous Bind SASL/GSSAPI Bind
Completed RSS VMSIZE RSS VMSIZE
10000 9308 181736 9788 181856
20000 9308 181736 9788 181856
30000 9308 181736 9788 181856
40000 9316 181736 9788 181856
50000 9316 181736 9788 181856
60000 9316 181736 9796 181856
70000 9316 181736 9796 181856
80000 9316 181736 9800 181856
90000 9316 181736 9800 181856
100000 9316 181736 9800 181856
Current Software Versions
Linux kernal 2.6.17.6
OpenLDAP 2.3.27 also tried 2.3.30 w/ similar results
cyrus SASL 2.1.21
Heimdal 0.7.2
Berkeley 4.2
slapd.conf
loglevel 768
include /etc/openldap/schema/core.schema
include /etc/openldap/schema/cosine.schema
include /etc/openldap/schema/inetorgperson.schema
include /etc/openldap/schema/nis.schema
include /etc/openldap/schema/local.schema
include /etc/openldap/slapd.acl.prod
# default settings
sizelimit 350
timelimit 60
idletimeout 300
defaultsearchbase "dc=umich,dc=edu"
pidfile /etc/openldap/slapd.slave.pid
argsfile /etc/openldap/slapd.slave.args
# Allow these features
allow bind_v2 bind_anon_cred bind_anon_dn
threads 32
include /etc/openldap/slapd.tls
# SASL / GSSAPI / Kerberos
sasl-realm UMICH.EDU
saslRegexp uid=simta/mx.umich.edu,cn=umich.edu,cn=gssapi,cn=auth
cn=simta,ou=mail agents,ou=security,dc=umich,dc=edu
saslRegexp uid=(.*),cn=umich.edu,cn=gssapi,cn=auth
uid=$1,ou=people,dc=umich,dc=edu
saslRegexp dc=edu,dc=umich,ou=security,ou=(.*),cn=(.*)
cn=$2,ou=$1,ou=security,dc=umich,dc=edu
# Proxy authorization checks saslAuthzFrom in destination entry
sasl-authz-policy from
include /etc/openldap/slapd.sasl.hostname
TLSCertificateFile /etc/openssl/private/ldap-dev_itd_umich_edu.crt
TLSCertificateKeyFile /etc/openssl/private/ldap-dev.itd.umich.edu.key
# database settings
database bdb
suffix "dc=umich,dc=edu"
rootdn xxxxxxxx
rootpw xxxxxxxx
limits dn.exact="cn=batch update,ou=security,dc=umich,dc=edu"
size.soft=unlimited size.hard=soft time.soft=unlimited
time.hard=soft
limits users size.soft=350 size.hard=soft size.unchecked=1000
time.soft=60 time.hard=soft
limits anonymous size.soft=50 size.hard=soft size.unchecked=1000
time.soft=5 time.hard=soft
directory /var/slapd/tmp/db/db.slave
cachesize 100000
checkpoint 2048 5
idlcachesize 20000
include /etc/openldap/slapd.index
# Monitor Backend
database monitor
BerkeleyDB DB_CONFIG file
set_lk_max_locks 25000
set_lk_max_objects 25000
set_cachesize 0 128000000 1
# Set transaction log buffer size to 2 megs
set_lg_bsize 2097152
Any ideas?
Thanks,
Paul Turgyan
University of Michigan
14 years, 10 months
howto create base dn??? postgres(8.1.4)+slapd(2.2.23)
by Axel Christiansen
Hello List,
i can not get the base dn into the postgres db.
The phpLDAPadmin tool complains about the missing
base dn. When i try to set one, the slapd segfalts.
The w32 LDAP Admin V0.9.9.2 does not complain about
the base dn. When i try to add an item, the slapd allso
segfalts.
Without the sql backend the LDAP where wokring fine. The slapd
seems to talk to postgres just fine.
I read a hole lot of howtos. Waht i really missed was
clear instructions in howto init the sql database and
brand it with the base dn.
What can i do next?
Thx a lot, Axel
The LDAP host:
Linux fn 2.6.7-1-686-smp #1 SMP Wed Jul 28 13:02:18 CEST 2004 i686
ii libiodbc2 3.52.2-3 iODBC Driver Manager
ii odbc-postgresq 07.03.0200-5 ODBC driver for PostgreSQL
ii odbcinst1 2.2.4-11 Support library and helper
ii odbc-postgresq 07.03.0200-5 ODBC driver for PostgreSQL
ii postgresql-8.1 8.1.4-6~bpo.1 object-relational SQL database
ii postgresql-cli 8.1.4-6~bpo.1 front-end programs for PostgreSQL 8.1
ii postgresql-cli 59~bpo.1 manager for multiple PostgreSQL
ii postgresql-com 59~bpo.1 manager for PostgreSQL database
ii postgresql-doc 8.1.4-6~bpo.1 documentation for the PostgreSL da
ii postgresql-ser 8.1.4-6~bpo.1 development files for PostgreSQL
ii ldap-utils 2.2.23-8 OpenLDAP utilities
ii libldap-2.2-7 2.2.23-8 OpenLDAP libraries
ii libldap2 2.1.30-8 OpenLDAP libraries
Down here you find:
1. slapd debug to stdout/err
2. postgres statements
######################################################################
1.
fn:/etc/ldap# slapd -d 160000 -f slapd.conf
@(#) $OpenLDAP: slapd 2.2.23 (May 30 2005 08:52:42) $
@pulsar:/home/torsten/packages/openldap/openldap2.2-2.2.23/debian/build/servers/slapd
slapd starting
conn=0 fd=10 ACCEPT from IP=217.86.159.169:4084 (IP=0.0.0.0:389)
conn=0 op=0 BIND dn="cn=admin,dc=c111,dc=org" method=128
conn=0 op=0 BIND dn="cn=admin,dc=c111,dc=org" mech=SIMPLE ssf=0
conn=0 op=0 RESULT tag=97 err=0 text=
conn=0 op=1 SRCH base="dc=c111,dc=org" scope=1 deref=0
filter="(objectClass=*)"
conn=0 op=1 SRCH attr=objectclass
conn=0 op=1 SEARCH RESULT tag=101 err=12 nentries=0 text=control
unavailable in context
conn=0 op=2 SRCH base="dc=c111,dc=org" scope=1 deref=0
filter="(objectClass=*)"
conn=0 op=2 SRCH attr=objectclass
conn=0 op=2 SEARCH RESULT tag=101 err=0 nentries=0 text=
conn=0 op=3 SRCH base="dc=c111,dc=org" scope=0 deref=0
filter="(objectClass=*)"
conn=0 op=3 SEARCH RESULT tag=101 err=0 nentries=0 text=
conn=0 op=4 ADD dn="ou=test,dc=c111,dc=org"
Segmentation fault
##########################################################################
2.
conn = 135496696, PGAPI_Connect(DSN='ldap', UID='ldap', PWD='xxxxx')
Global Options: Version='07.03.0200', fetch=100, socket=4096,
unknown_sizes=0, max_varchar_size=254, max_longvarchar_size=8190
disable_optimizer=1, ksqo=1, unique_index=1,
use_declarefetch=0
text_as_longvarchar=1, unknowns_as_longvarchar=0,
bools_as_char=1 NAMEDATALEN=64
extra_systable_prefixes='dd_;', conn_settings=''
conn_encoding='OTHER'
conn=135496696, query=' '
conn=135496696, query='select version()'
[ fetched 1 rows ]
[ PostgreSQL version string = 'PostgreSQL 8.1.4 on
i386-pc-linux-gnu, compiled by GCC cc (GCC) 3.3.5 (Debian 1:3.3.5-13)' ]
[ PostgreSQL version number = '8.1' ]
conn=135496696, query='set DateStyle to 'ISO''
conn=135496696, query='set geqo to 'OFF''
conn=135496696, query='set extra_float_digits to 2'
conn=135496696, query='select oid from pg_type where typname='lo''
[ fetched 0 rows ]
conn=135496696, query='select pg_client_encoding()'
[ fetched 1 rows ]
[ Client encoding = 'SQL_ASCII' (code = 0) ]
conn=135496696, query='SELECT
id,name,keytbl,keycol,create_proc,delete_proc,expect_return FROM
ldap_oc_mappings'
[ fetched 0 rows ]
conn=135496696, query='ROLLBACK'
conn=135496696, PGAPI_Disconnect
conn = 135495944, PGAPI_Connect(DSN='ldap', UID='ldap', PWD='xxxxx')
Global Options: Version='07.03.0200', fetch=100, socket=4096,
unknown_sizes=0, max_varchar_size=254, max_longvarchar_size=8190
disable_optimizer=1, ksqo=1, unique_index=1,
use_declarefetch=0
text_as_longvarchar=1, unknowns_as_longvarchar=0,
bools_as_char=1 NAMEDATALEN=64
extra_systable_prefixes='dd_;', conn_settings=''
conn_encoding='OTHER'
conn=135495944, query=' '
conn=135495944, query='select version()'
[ fetched 1 rows ]
[ PostgreSQL version string = 'PostgreSQL 8.1.4 on
i386-pc-linux-gnu, compiled by GCC cc (GCC) 3.3.5 (Debian 1:3.3.5-13)' ]
[ PostgreSQL version number = '8.1' ]
conn=135495944, query='set DateStyle to 'ISO''
conn=135495944, query='set geqo to 'OFF''
conn=135495944, query='set extra_float_digits to 2'
conn=135495944, query='select oid from pg_type where typname='lo''
[ fetched 0 rows ]
conn=135495944, query='select pg_client_encoding()'
[ fetched 1 rows ]
[ Client encoding = 'SQL_ASCII' (code = 0) ]
14 years, 10 months
RE: Disaster recovery question wrt replication
by Steven Harms (stharms)
Judging by the number of other comments about the miracles of 2.30,
maybe I'll try to talk to my admins and see if I can get that.
Steven
-----Original Message-----
From: openldap-software-bounces+stharms=cisco.com(a)openldap.org
[mailto:openldap-software-bounces+stharms=cisco.com@openldap.org] On
Behalf Of Tony Earnshaw
Sent: Thursday, December 14, 2006 2:04 AM
Cc: openldap-software(a)openldap.org
Subject: Re: Disaster recovery question wrt replication
Steven Harms (stharms) wrote:
[...]
> I suppose, more generally, I'm asking: How do a start replication all
> over. What files can / should I delete. Which should I not under any
> circumstance touch?
Others (Quanah) have mentioned dropping implementation of slurpd in
favor of i(delta-)syncrepl - requires upgrading to 2.3(.30 at the
moment).
This site used to use slurpd with OpenLDAP 2.2. Before we switched to
2.3, we tested syncrepl slave DB rebuilds from scratch (empty DB, valid
slapd.conf and DB_CONFIG in the DB directory). We simply started slapd
on the slave machines and the small (about 40MB) DB was automatically
rebuilt within seconds (100Mb Ethernet and fast SCSI RAID5),
high-quality iron.
Having previously had to deal with the same thing with slurpd earlier, I
was completely gob-smacked.
--Tonni
--
Tonni Earnshaw
tonni @ barlaeus.nl
14 years, 10 months
OpenLDAP Proxy Cache question
by Gessy
Hi ,
On OpenLDAP Proxy Cache, the pointer to backend directory server (uri) can
be multi-value? Like:
suffix "dc=example,dc=com"
uri ldap://host1/dc=example,dc=com
uri ldap://host2/dc=example,dc=com
#setting cache parameters
cacheparams 10000 15000 4 50 1000
Or
suffix "dc=example,dc=com"
uri ldap://host1/dc=example,dc=com ldap://host2/dc=example,dc=com
#setting cache parameters
cacheparams 10000 15000 4 50 1000
thanks a lot.
Gessy Jr.
--
(o< Avoid the Gates of hell!
//\ Use GNU Linux.
V_/_
14 years, 10 months
Re: slapd responds very slowly when cpu has 100% usage (but actually low load)
by Antonis Christofides
(My original message, which presents the problem, is at the bottom.)
Thank you for your responses, here is some more information:
> You didn't mention what version of slapd you're running.
That's right, sorry. I'm running Debian-packaged slapd 2.2.23-8,
using bdb 4.2.52-18 (everything on my system is the Debian sarge's
packages, except for the kernel, which is a recompiled Ubuntu 6.06
kernel, 2.6.12 SMP).
> I would expect the call to wait after forking to exec true to be treated
> as blocking IO, but, perhaps on your system, true is an sh builtin.
That's right, true is a shell builtin. So the two processes don't
fork or anything, they just run and run. Some tests I made right now
show that a certain ldapsearch completes in 4 seconds if the processes
have nice 19, 35-40 seconds if they have nice 10, and longer if nice
0.
And, yes, if I replace "while true" with "while /bin/true", slapd
responds instantly.
> nicing a process does not affect its time slice, just where it sits
> in the run queue when it is ready to run.
Your description of how the scheduler works might explain the cause of
the problem if slapd makes a huge number of blocking I/O requests:
each time it makes such a request, it goes to Sleep, and the next
process on the run queue (the niced shell in our case) is set to run,
and exhausts its time slice. Then, supposing slapd is ready again, it
is set to run, it makes a request, it sleeps again, etc. If it needs
to make hundreds of I/O requests, could it explain the delay?
Here is what ps shows while I'm waiting for slapd to respond:
anthony@acheloos:~$ ps u -m 16204
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 16204 0.0 0.5 30140 5596 ? - 11:45 0:00 /usr/sbin/slapd
root - 0.0 - - - - Ss 11:45 0:00 -
root - 0.0 - - - - Ss 11:45 0:00 -
root - 0.0 - - - - Rs 11:46 0:00 -
One of the threads is "Rs"; after slapd delivers its response, it goes
back to "Ss".
> Maybe you have the memory to let everything rest in memory. I don't know what
> your two looping shells do to your memory... If you had some control to never
> swap out ldap, this theory could be tested.
I think I have lots of spare memory:
top - 12:53:32 up 60 days, 1:35, 5 users, load average: 1.38, 0.77, 0.63
Tasks: 262 total, 3 running, 258 sleeping, 1 stopped, 0 zombie
Cpu0 : 0.7% us, 0.7% sy, 98.7% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu1 : 0.0% us, 0.0% sy, 100.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 1035680k total, 1016936k used, 18744k free, 39004k buffers
Swap: 2097144k total, 99888k used, 1997256k free, 577360k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8551 anthony 35 10 4032 1204 916 R 99.9 0.1 0:10.96 sh
8550 anthony 35 10 4032 1204 916 R 98.8 0.1 0:10.72 sh
16204 root 21 0 30140 5600 3528 S 0.0 0.5 0:00.22 slapd
I tried to play with cachesize and idlcachesize (set them to 10
thousand), but didn't see any difference, which hardly surprises me
given that my ldap database has only 257 records.
Finally, here is my DB_CONFIG:
set_cachesize 0 2097152 0
set_lg_bsize 524288
set_lk_max_objects 5000
set_lk_max_locks 5000
set_lk_max_lockers 5000
(My slapd.conf does not contain any db-related parameters).
My original message:
> Hi,
>
> At the almost idle Dual Core machine which runs slapd, I run:
>
> nice sh -c 'while true; do true; done' &
> nice sh -c 'while true; do true; done' &
>
> (i.e. I'm running this twice). Then each of the two CPUs always has
> some job to do, so both CPUs have 100% usage, but this is "nice".
>
> Then, slapd takes too long to respond to queries. It may take 10 or
> 20 seconds. If I kill or stop one of the two dummy processes, it
> replies instantly. If I continue both dummy processes, it's back to
> 10 or 20 seconds. Needless to say all machine resources seem ok; low
> disk usage, lots of spare memory; and slapd is not niced.
>
> If it's not something immediately obvious, could you help me debug it?
> I've run slapd with various "-d" options but it gives me results that
> I have trouble understanding.
>
> The OS is Debian 3.1 (Sarge), with a 2.6.12 SMP Linux kernel.
14 years, 10 months
dnsDomain2.schema and aRecord
by Jürgen Magin
Hello
I'm using dnsDomain2 schema with openldap 2.2.27 for dns server (pdns).
When I look into the logfiles i saw that some questions are not answered
by slapd.
When I ask:
ldapsearch -x -LLL "dc=hostname"
i get the follwing answer:
dn: dc=hostnanme,dc=example.local,ou=DNS,ou=Services,dc=example,dc=local
objectClass: top
objectClass: dNSDomain2
objectClass: domainRelatedObject
dc: hostname
aRecord: 192.168.1.1
pTRRecord: hostname.example.local
associatedDomain: hostname.example.local
ok, but
ldapsearch -x -LLL "aRecord=192.168.1.1"
or
ldapsearch -x -LLL "aRecord=192*"
returns nothing and
ldapsearch -x -LLL "aRecord=*"
returns all entries.
The entry 'aRecord' is of type caseIgnoreIA5Match.
What's wrong there? Any ideas?
--
Mit freundlichen Grüßen
Jürgen Magin
**************************************************************
# #
# OCTOGON Software Development GmbH #
# http://www.octo-soft.de #
# #
# Jürgen Magin, Einsteinstr. 11, D 68519 Viernheim #
# #
# Tel : +49 6204/738353 #
# Fax : +49 6204/914875 #
# EMail : gaston(a)octo-soft.de #
# #
**************************************************************
14 years, 10 months
syncrepl and slowness during updates
by Paul Main
Hello,
Here at UCI we have just deployed syncrepl to replicate our LDAP
directory. Unfortunately, we use an old directory structure (PH/QI)
for our master database, and ldap gets updates from that system in
batch. The batch process updates our master LDAP server, and
syncrepl is used to push the changes out to all the other LDAP
servers. This causes somewhere around 100 to 1000 entries to be
updated all at once (out of 80k + entries in the LDAP directory).
The problem we are experiencing is that when a syncrepl slave
receives a bunch of updates, queries to these slaves slow down
tremendously. We are talking going from a sub second query time for
a single LDAP entry, to, in some cases, over 20 seconds response time
for simple queries. This is causing all sorts of problems for us.
One thing to note: the master does basically the same updates, but
through an normal ldap client, rather than through syncrepl -- and it
does not experience this slowness.
We are using BDB, Openldap ver. 2.3.28 and our syncrepl entry looks
something like this:
syncrepl rid=11
provider=ldap://ldap3.nac.uci.edu:389
type=refreshAndPersist
interval=0:00:00:05
searchbase="OU=University of California Irvine,
O=University of
California, C=US"
filter="(objectClass=*)"
scope=sub
schemachecking=off
sizelimit=0
timelimit=0
updatedn="cn=root,OU=University of California
Irvine, O=Universi
ty of California, C=US"
bindmethod=simple
binddn="uid=nsp,OU=University of California Irvine,
O=University
of California, C=US"
credentials="REMOVED"
starttls=yes
retry=1,2,3,4,5,+
Any help would be appreciated,
-Paul Main
Network and Academic Computing Services
University of California, Irvine
14 years, 10 months
misc.schema bundled with OpenLDAP - recommended?
by Gavin Henry
Hi all,
I notice that misc.schema says at the top:
20 # Not recommended for production use!
21 # Use with extreme caution!
This definitely the case?, as I wanted to use 'nisMailAlias' and
'rfc822MailMember' from it, but not sure now.
I've done our own schema ages ago with
suretecMailAlias/suretecMailAliasMember, so might stick with that, but
wanted to use a bundled schema for most things.
Should we stick with our own for Mail Aliases, or use bundled?
Advice?
Thanks.
--
Kind Regards,
Gavin Henry.
Managing Director.
T +44 (0) 1224 279484
M +44 (0) 7930 323266
F +44 (0) 1224 824887
E ghenry(a)suretecsystems.com
Open Source. Open Solutions(tm).
http://www.suretecsystems.com/
14 years, 10 months
