Re: slow slapadd?
by kkalev
On Mon, May 18, 2009 at 7:56 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote:
> --On Monday, May 18, 2009 7:55 AM -0700 Bill MacAllister <whm(a)stanford.edu>
> wrote:
>
> However, on BDB 4.7 it seems the default behavior on Linux is also to
>>> do synchronous flushes of the cache. As such, one approach to getting
>>> consistent performance is to configure the backend to use shared
>>> memory for the BDB cache instead of mmap'd files. That way incidental
>>> page updates don't sync to anything, and the BDB library has full
>>> control over when pages get flushed back to disk.
>>>
>>
>> We can confirm that setting shm_key on 4.7 dramatically affects the load
>> time. In our initial tests of 4.7 we just pulled our old 4.2 BDB
>> parameters forward. We never saw the slapadd of a 4.6 gbyte database
>> complete. We killed it after 4 hours. After changing the shm_key
>> setting the load time dropped to the more normal 30 minutes.
>>
>
>
> To expand on this slightly. BDB has no difference in perf for small
> databases between disk and shm. It's only once your database is past some
> 6GB in size that shm vs disk starts to make a difference on Linux.
Could someone elaborate on the setting (shared memory instead of mmap) that
is best suited for each major OS? Like Linux, FreeBSD, Solaris?
>
>
> --Quanah
>
> --
>
> Quanah Gibson-Mount
> Principal Software Engineer
> Zimbra, Inc
> --------------------
> Zimbra :: the leader in open source messaging and collaboration
>
14 years, 6 months
slow slapadd?
by Diego Figueroa
Hi all,
I'm trying to load ldap with several ldif files. The biggest one
containing 500k records of approx 700 bytes in size each took slapadd a
little over 14 hours to load. Would this be considered normal given the
amount of records and size?
I am including my DB_CONFIG and slapd.conf files to see if anyone here can
help me figure out if I made a mistake while setting them up.
The command I use for loading is:
slapadd -q -f slapd.conf -l myfile.ldif
One thing I did notice is that using bdb instead of hdb took the load time
to 19 hours.
Anyone have any recommendations?
Thanks,
Diego.
---------
DB_CONFIG
---------
set_lg_max 209715200
set_lg_bsize 52428800
set_tmp_dir /data/ldap/tmp
set_cachesize 0 209715200 2
set_lk_max_locks 4000
set_lk_max_lockers 4000
set_lk_max_objects 4000
----------
slapd.conf
----------
allow bind_v2
include /etc/ldap/schema/core.schema
include /etc/ldap/schema/cosine.schema
include /etc/ldap/schema/nis.schema
include /etc/ldap/schema/inetorgperson.schema
include /etc/ldap/schema/my.schema
pidfile /var/run/slapd/build-slapd.pid
argsfile /var/run/slapd/build-slapd.args
modulepath /usr/lib/ldap
moduleload back_hdb
password-hash {SSHA}
disallow bind_anon
backend hdb
database hdb
suffix "dc=mydomain,dc=com"
directory "/var/lib/build-ldap"
lastmod on
14 years, 6 months
can't upload big CRL, no clear error message; what to set? bug?
by Barclay, Daniel
I'm having trouble uploading ("publishing"?) large CRLs file (any over
about 16.6 MB).
The client (OpenLDAP's ldapmodify) ends up saying:
ldapmodify: update failed: cn=...
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
even though it has contacted the server (OpenLDAP's slapd), as evidenced
by server log messages:
May 15 12:54:23 ... slapd[14659]: conn=0 fd=14 ACCEPT from IP=...:53557 (IP=0.0.0.0:389)
May 15 12:54:23 ... slapd[14659]: conn=0 op=0 BIND dn="... method=128
May 15 12:54:23 ... slapd[14659]: conn=0 op=0 BIND dn="..." mech=SIMPLE ssf=0
May 15 12:54:23 ... slapd[14659]: conn=0 op=0 RESULT tag=97 err=0 text=
May 15 12:54:23 ... slapd[14659]: conn=0 fd=14 closed (connection lost)
I wouldn't have been surprised if there were a server-side limit that I'm
hitting, but I'm not seeing any evidence of an intentional server-side limit
(e.g., an explicit error message).
I have found some references to slapd.conf settings sockbuf_max_incoming
and sockbuf_max_incoming_auth, but:
1) they're described in terms of LDAP PDUs, but I don't know whether a CRL (an
attribute value) needs to fit in a single PDU or not (does it?), and
2) the slapd.conf manual page says the default sockbuf_max_incoming_auth
value is 4194303, which make it seem less likely that it's related to the
limit I'm hitting around 16.6 MB .
Are they relevant or not?
Increasing the server logging level yields:
May 15 13:07:13 ... slapd[14691]: cber_get_next on fd 14 failed errno=34 (Numerical result out of range)
May 15 13:07:13 ... slapd[14691]: connection_read(14): input error=-2 id=0, closing.
May 15 13:07:13 ... slapd[14691]: connection_closing: readying conn=0 sd=14 for close
May 15 13:07:13 ... slapd[14691]: connection_close: conn=0 sd=14
May 15 13:07:13 ... slapd[14691]: daemon: removing 14
May 15 13:07:13 ... slapd[14691]: conn=0 fd=14 closed (connection lost)
Does this seem to a simple configuration problem or a bug?
(This is with Debian Lenny versions:
# slapd -V
@(#) $OpenLDAP: slapd 2.4.11 (Oct 12 2008 04:13:21) $
buildd@ninsei:/build/buildd/openldap-2.4.11/debian/build/servers/slapd
# ldapmodify -V
ldapmodify: @(#) $OpenLDAP: ldapmodify 2.4.11 (Oct 12 2008 04:12:41) $
buildd@ninsei:/build/buildd/openldap-2.4.11/debian/build/clients/tools
(LDAP library: OpenLDAP 20411)
)
Thanks,
Daniel
--
(Plain text sometimes corrupted to HTML "courtesy" of Microsoft Exchange.) [F]
14 years, 6 months
OpenLDAP lib & MinGW
by Eric Nichols
Jonathon Bowman wrote a great set of instructions on how to build the OpenLDAP
libs in MinGW. Apparently cygwin is no longer a valid option?
http://bowmansolutions.com/mingw-openldap/
I was able to get through all of the instructions and have now tried to run a
test program but I get the following error on compile:
gcc -Wall -lldap rootdse2.c
C:/DOCUME~1/ADMINI~1/LOCALS~1/Temp/ccMQuBSx.o:rootdse2.c:(.text+0x13):
undefined reference to `ldap_err2string'
C:/DOCUME~1/ADMINI~1/LOCALS~1/Temp/ccMQuBSx.o:rootdse2.c:(.text+0x8f):
undefined reference to `ldap_initialize'
collect2: ld returned 1 exit status
Can anyone lend a hand with trying to figure out why I can't use the library?
Also is there a simpler config option to build just the libs?
I'd be more than happy to put together a new set of instructions outlining the
configuration.
Thanks
Eric
14 years, 6 months
selfread access doesn't work as expected
by Christian Fischer
Hi all,
I'm running openldap-2.3.43 on gentoo amd64.
Shouldn't give the following access directive members of
ou=People,dc=foo,dc=bar selfread permissions to attrs=member and all others
(eg. the bind user cn=ldapbind,ou=dsa,dc=foo,dc=bar) unlimited read
permissions?
access to dn.subtree="ou=Group,dc=foo,dc=bar" attrs=member
by dn.children="ou=People,dc=foo,dc=bar" selfread
by * read
Selfread works only if i restrict * to none, but that's not what i want.
'by * read' is not what i want at least but it simplifies the example.
access to dn.subtree="ou=Group,dc=foo,dc=bar" attrs=member
by dn.children="ou=People,dc=foo,dc=bar" selfread
by * none
It should expand to
'by dn.children="ou=People,dc=foo,dc=bar" selfread stop'
but it seems to continue.
What's wrong?
Regards
Christian
--
"Without music to decorate it, time is just a bunch of boring production
deadlines or dates by which bills must be paid."
--- Frank Vincent Zappa
14 years, 6 months
Sync API beginner / Callback trigger grokking
by Sandra Snan
Dear all;
I'm using the ldap client sync api (ldap_sync(3)).
I'm trying to figure out how to use it.
I've added all the four possible callbacks - ls_search_entry,
ls_search_reference, ls_intermediate and ls_search_result. For
now, I only made them print out some data.
I call ldap_sync_init_refresh_only on that struct repeatedly.
ls_search_entry trigger once for each matching dn in the database
on the first call to ldap_sync_init_refresh_only.
ls_intermediate trigger once everytime a dn is deleted.
ls_search_result triggers once everytime ldap_sync_init_refresh_only is run.
ls_search_reference hasn't trigged at all so far.
What I want to do is find out when objects are added, deleted, or
attributes changed, added or deleted, and trigger things from
that. I'm probably doing a lot of things wrong. Any ideas?
So far, nothing seems to trigger from adding objects or changing
attributes.
S
14 years, 6 months
LDAP replication
by Mike
Folks,
I'm currently working on setting up replication between two LDAP
servers. I understand with OpenLDAP 2.4 this requires the use of
syncrepl. I have read that it supports push and pull replication (as
well as the wizzy-cool N-way) but I was wondering if anyone could advise
how this works at the transport layer.
From reading through the docs, I infer that with pull the slave
periodically makes a TCP connection to the master in order to check for
changes and with push, the slave makes a persistent TCP connection to
the master, over which updates are propogated from the master. Is this
correct?
I would like to know if there is a way to initiate the TCP connections
from the master to the slave as this would make life a little easier for
me?
Cheers,
Mike.
14 years, 6 months
Pool File: unknown
by Thierry Lacoste
I'm using openldap 2.4.16 with bdb 4.6 backend on one master and
one slave with syncrepl in refreshAndPersist mode.
Running 'db_stat -m' on the slave I have 9 entries like that:
Pool File: unknown
1024 Page size
0 Requested pages mapped into the process' address space
133 Requested pages found in the cache (99%)
1 Requested pages not found in the cache
2 Pages created in the cache
0 Pages read into the cache
0 Pages written from the cache to the backing file
I have no such entry on the master.
Can someone please explain?
Regards,
Thierry
14 years, 6 months
A question about entry and children pseudo-attributes
by Thierry Lacoste
Hello,
According to SLAPD.ACCESS(5) the access needed to the
'entry' pseudo-attribute always correspond to the access
needed on the 'children' pseudo-attribute of the entry's parent.
The only exception is the search operation.
So what is the purpose of children?
Are there examples of ACLs where the two pseudo-attributes
are treated differently.
Regards,
Thierry
14 years, 6 months
