Re: slapd syncrepl consumer having permanent high CPU load
by Rodrigo Costa
Folks,
I was preparing openLDAP with GDB symbols but looks like the issue was identified and solved in HEAD. Just to identify this issue; was created any sort of ITS for verification in a new load?
Sorry my late response but my baby daughter just born last week and I was having some work at home.
I will give a try in the HEAD load.
Best Regards,
Rodrigo.
PS-> Just some link from my daughter
http://sites.google.com/site/lauramenina/laura_english
--- On Wed, 3/18/09, Howard Chu <hyc(a)symas.com> wrote:
> From: Howard Chu <hyc(a)symas.com>
> Subject: Re: slapd syncrepl consumer having permanent high CPU load
> To: "John Morrissey" <jwm(a)horde.net>
> Cc: openldap-software(a)openldap.org
> Date: Wednesday, March 18, 2009, 5:21 AM
> John Morrissey wrote:
> > After ~16h uptime, slapd with this BDB had increased
> its DN cache to ~250k
> > entries after it previously appeared stable at the
> configured 20k entries,
> > and its entry cache had ballooned to ~480k entries.
> Its RSS was about 3.6GB
> > at this point, with a BDB cache size of 2GB.
>
> I was finally able to reproduce this (took several hours of
> searches. Fortunately I was at a St. Pat's party so I didn't
> have to wait around, just got home in time to see it start
> going bad...). A fix is now in HEAD.
>
> (And now we'll see if Guinness is Good For Your Code... ;)
> -- -- Howard Chu
> CTO, Symas Corp.
> http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
>
14 years, 8 months
slapadd performance
by Pete Giesin
I am trying to perform some benchmarking against OpenLDAP. So far I have ran
my tests against a 100K and 1Million entry database and I have had rather
decent numbers. My final set of tests were to be ran against a 10Million
entry database. Unfortunately, I am having difficult loading the database
with this many entries. I have generated 10 1Million LDIF files. I am using
"slapadd -c -q -v -l <file>" to import each file. The first 2 files took
approximately 15 minutes each to load. The remaining 8 files are taking
progressively longer and longer. So much longer that I anticipate the entire
proceess to take well over 24 hours. My question is is there anything I can
do to increase the performance of slapadd. I assume that since slapd is not
running at this point that the normal DB_CONFIG and slapd.conf settings do
not have much affect.
I currently have a single database configured to hold all 10 Million
entries. Is this an unrealtistic expectation? Should I instead be planning
on having multiple databases and using chaining/referrals to link them
together. What is the optimal configuration for handling this number of
entries?
Thanks,
Pete
14 years, 8 months
mirroring problem (unwilling to perform error)
by Hegedus Gabor
Hi all!
I have a serious problem!
I try to configure the ldap server mirroring mode, but something problem
with this.
Fist of all:
I tested the mirroring on two openldap 2.4.11-0ubuntu6.1. and it worked
fine.
I just configured slapd.conf file like this:
# Global section
serverID 1 (another ldap is 2)
moduleload syncprov
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100
syncrepl rid=001
provider=ldap://192.168.1.2
type=refreshAndPersist
interval=00:00:00:10
searchbase="dc=test,dc=hu"
filter="(objectClass=*)"
attrs="*"
scope=sub
schemachecking=off
bindmethod=simple
binddn="cn=admin,dc=test,dc=hu"
credentials="test"
retry="60 +"
mirrormode on
It was okay, when I modified the db1, the db2 modified too.
BUT!
I try to set up on version 2.3.30, I use the same configuration, the
ldap starts good, but when i try to modify the db, the phpldapadmin
shows this error:
Could not perform ldap_modify operation.
LDAP said: Server is unwilling to perform
Error number: 0x35 (LDAP_UNWILLING_TO_PERFORM)
Description: The LDAP server refused to perform the operation.
What is the problem, what is missconfigured?
please help,
thank you!
br Gabor
14 years, 8 months
The OpenLDAP Client and Multiple DNS Records
by Tim Gustafson
Hi,
How does the OpenLDAP client library handle multiple A records being returned for a DNS query for an LDAP server?
That is to say, if "host ldap" returns 1.1.1.1, 2.2.2.2 and 3.3.3.3, will the OpenLDAP client library only connect to 1.1.1.1? If a connection to 1.1.1.1 fails, will it try 2.2.2.2 and then 3.3.3.3?
Basically we're trying to achieve redundant servers and load balancing using a round-robin-style DNS entry.
Tim Gustafson
BSOE Webmaster
UC Santa Cruz
tjg(a)soe.ucsc.edu
831-459-5354
14 years, 8 months
Re: Mirror Mode, MMR and replicas
by Quanah Gibson-Mount
--On Wednesday, March 18, 2009 10:18 PM +0100 Rein Tollevik
<rein(a)OpenLDAP.org> wrote:
> Quanah Gibson-Mount wrote:
>> --On Wednesday, March 18, 2009 3:01 AM -0700 Howard Chu <hyc(a)symas.com>
>
>>>> I'll start with replication under MMR.
>>>
>>>> As I understand it, the replicas can only point at a single master.
>>>
>>> False.
>>
>> Last I checked, the provider parameter in the syncrepl configuration
>> block was single valued. Which means either it gets pointed at a load
>> balancer, or it only talks to a single master. If that master goes
>> down, it no longer talks to anything without reconfiguration.
>
> Each syncrepl engine (configuration block) can only refer to a single
> master, but you may configure many engines within the same database, each
> with its own master. If one master goes down the engine using that
> master will obviously not receive any updates until its master is up
> again, but the other will continue to receive updates from their master.
Ok, this was the point I didn't realize -- That syncrepl on a replica will
correctly handle looking at two masters providing the same changes to it.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
14 years, 8 months
Re: Mirror Mode, MMR and replicas
by Quanah Gibson-Mount
--On Wednesday, March 18, 2009 3:01 AM -0700 Howard Chu <hyc(a)symas.com>
wrote:
> Quanah Gibson-Mount wrote:
>> A number of our clients have requested "fail-over"/redundancy
>> capabilities for the LDAP master, and as I'm currently working on moving
>> our product to use OpenLDAP 2.4, this becomes a distinct possibility.
>> However, I have some questions about the
>> viability/reliability/effectiveness of using multiple masters combined
>> with replicas. I don't see these answered in the Admin Guide.
>
> You mean, setting up regular read-only replicas slaved to the masters?
Yes.
>> I'll start with replication under MMR.
>
>> As I understand it, the replicas can only point at a single master.
>
> False.
Last I checked, the provider parameter in the syncrepl configuration block
was single valued. Which means either it gets pointed at a load balancer,
or it only talks to a single master. If that master goes down, it no
longer talks to anything without reconfiguration.
>> So, if
>> I have a 2 master MMR setup, I assume I would want to point half my
>> replicas at master A and the other half at master B for their updates.
>> This leads to a problem in my mind, in that if master A goes down, then
>> half of my replica pool is now going to remain completely out of sync
>> with the remaining master until master A is recovered. Throwing a Load
>> balancer in front of the two masters, and pointing the replicas at that
>> instead, is not a viable option because the two masters may be getting
>> updates in a different sequence, so if a replica disconnects from the LB
>> and then reconnects, the updates it could get fed from whatever master
>> the LB is pointing at could lead to inconsistencies.
>
> What inconsistencies? Each master's changes are stamped with its own sid.
> Any consumer is going to know about the contextCSNs of each master it
> talks to.
The consumer can only talk to a single master, unless it is pointed at a
load balancer. See notes above. In the load balancer case then, I'm
assuming based on what you say, that if the replica gets disconnected
because master A went down, then when it gets reconnected to master B, that
its CSN will be completely different because of the SID values being
different, and do a full refresh. Is that correct?
>> Neither of these seem like a
>> good option. I don't see a good solution here to resolve this issue,
>> either, unless the replica could somehow know which master it had been
>> talking to,
>
> The replica always knows which master it's talking to...
>
>> and drop into refresh mode if it found itself talking to a new
>> master?
>
> Drop into refresh mode? Obviously in persist mode the consumer keeps a
> connection open to a specific master; a load balancer can't move an open
> connection. So obviously, if a particular master disappears, all of its
> clients are going to lose their connections and any consumers set up to
> retry are going to have to initiate new sessions. And every new
> replication session starts with a refresh phase. So this recovery is
> already automatic, it always has been.
Based off of the CSN value. See question above about CSN & SID.
>> I'm also not clear on what happens if your replicas are
>> delta-syncrepl based, rather than normal syncrepl, in the LB setup.
>
> Not possible. Current delta-sync requires all updates to be logged in
> order; in an MMR setup you can't guarantee order so *nobody* can use
> delta-sync in this scenario.
>
>> For Mirror Mode, I would assume you could point the replicas at the LB
>> fronting the two masters, since only one master is ever receiving
>> changes. I also assume delta-syncrepl would be a completely valid option
>> for replication to the replicas, again because only one master is
>> getting the updates, so all updates would be logged in the same sequence
>> on both servers. However, I don't know if this is correct or not, or if
>> there are limitations here I haven't considered. When I was first
>> pondering this on the #openldap-devel channel in IRC, Matt Backes made a
>> comment about delta-syncrepl not working with Mirror Mode.
>
> For MirrorMode, delta-sync should work since there is only ever one
> source of changes, and they will be logged in order. There is a window of
> vulnerability where a server crashes after committing changes to its
> accesslog, before it replicates them to the mirror. Those changes will be
> temporarily lost, and create a gap in the mirror's log. When the original
> server comes back up, the mirror will receive those lost changes, but the
> strict ordering of its log will be broken. In this case though, the
> delta-sync consumer will be fine - if the lost changes caused no
> conflicts, they will simply be committed. If they do cause a conflict,
> the consumer will just fallback to refresh mode and the conflicts will be
> erased.
Ok, good. Mirror Mode sounds like the way to go.
>> So, basically, I'm at a loss if my understanding things is correct, on
>> how I provide a consistent replicated environment for my customers,
>> while also providing master/master failover.
>
> This appears to have been a -software question, not a -devel question.
> Perhaps you should summarize back to the -software list and end this
> thread here.
Done.
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
14 years, 8 months
Re: root-only configuration
by Peter Mogensen
Mike Malsman wrote:
> On 11.Mar.2009, at 9:32 AM, Peter Mogensen wrote:
>> But limiting cn=config access to ldapi:/// ... no luck.
>>
>> Do someone have a working example of this?
>>
>> /Peter
>
> What does your 'access' directive look like?
access to dn.exact="cn=config"
by peername.path="/var/run/slapd/ldapi" auth
by * none
I've used this method before in "normal" databases, to control who can
become rootdn, but it just won't work for cn=config.
Of course, I have to add a "userPassword" attribute to cn=config.ldif,
but it seems to be ignored.
I've also tried to create a cn=root,cn=config object, but I have a
problem finding a schema which is loaded which allows me to set
userPassword.
If people on this list hadn't said that it was possible, I would
probably have concluded by now that it is simply not possible to limit
rootdn access to cn=config to ldapi:///.
/Peter
14 years, 8 months
slapd syncrepl consumer having permanent high CPU load
by Rodrigo Costa
openLDAP software community,
I would like to report some issue I'm having. Since I'm not sure if this is a problem then I would like some help/comment about this behavior.
I configure slapd.conf to have a single provider(master) and a single consumer(slave). Please see attached the slapd.conf for each one for comments.
These files were prepared in accordance with OpenLdap 2.4 Administration Guide and I do not believe this is related with configuration.
I'm using BDB4.7 with all patches available and using OpenLDAP 2.4.15 HEAD since I'm also testing ITS#5860 resolution I posted sometime ago.
In this way it is supposed I have one of the latest releases. My system is running Linux Kernel 2.6, more specifically :
Linux brtldp12 2.6.18-92.1.22.el5PAE #1 SMP Tue Dec 16 12:36:25 EST 2008 i686 i686 i386 GNU/Linux
The DB I have into system is around 4 million entrances.
The behavior is the following :
1) Start slapd in provider(master) node with command :
date; /usr/libexec/slapd -d 256 -h "ldap://10.142.15.41:389 ldap://10.142.15.171:389" -u ldap
2) Start slapd in consumer(slave) node with command :
date; /usr/libexec/slapd -d 256 -h "ldap://10.142.15.42:389 ldap://10.142.15.172:389" -u ldap
3) Since there are 2 DBs, CONTENT and INDEX, see the consumer starting a search in these 2 DBs(used to verify synchronicity):
conn=0 fd=16 ACCEPT from IP=10.142.15.42:52117 (IP=10.142.15.41:389)
conn=1 fd=17 ACCEPT from IP=10.142.15.42:52118 (IP=10.142.15.41:389)
conn=0 op=0 BIND dn="cn=admin,ou=content,o=alcatel,c=fr" method=128
conn=1 op=0 BIND dn="cn=admin,ou=indexes,o=alcatel,c=fr" method=128
conn=0 op=0 BIND dn="cn=admin,ou=content,o=alcatel,c=fr" mech=SIMPLE ssf=0
conn=1 op=0 BIND dn="cn=admin,ou=indexes,o=alcatel,c=fr" mech=SIMPLE ssf=0
conn=0 op=0 RESULT tag=97 err=0 text=
conn=1 op=0 RESULT tag=97 err=0 text=
conn=0 op=1 SRCH base="ou=content,o=alcatel,c=fr" scope=2 deref=0 filter="(objectClass=*)"
conn=0 op=1 SRCH attr=* +
conn=1 op=1 SRCH base="ou=indexes,o=alcatel,c=fr" scope=2 deref=0 filter="(objectClass=*)"
conn=1 op=1 SRCH attr=* +
4) Use monitor interface to verify cache size consumption. Seeing that boundaries are respected into provider. A ldapsearch consumes around 28 minutes to end in each DB.
5) Monitor the slapd process CPU and memory usage in provider(master) with top. Seeing many sync_p in the provider threads and CPU usage by slapd process always small, normally between 11% to 108%.
6) Monitor the slapd process CPU and memory usage consumer(slave) with top. Seeing almost no sync_p and then much more CPU usage since there are 2 search happening with provider. CPU usage normally around 48% to 200%.
The problem is exactly after sometime passes and then slapd into consumer has CPU utilization almost fixed in 200% and the slapd process in provider becomes iddle(CPU 0% utilized by slapd).
This appears to indicate the search ended but something in consumer is not working ok since too much CPU is being utilized.
Other strange thing is if I start a ldapsearch with consumer(slave) the responsiveness is very low indicating something is really wrong. I put a ldapsearch and after some yours and killing the slapd process I could see that only around 50 thousands entraces were passed. This is a really slow performance.
Also looks like consumer(slave) process never ends the search/sync since CPU utilization from the 2 searches(2 DBs), starts to consume 100% of CPU each performing 200% CPU utilization even after many hours passed.
Other strange behavior I was not expecting, is that even the search is from the consumer to provider I see the Cache information, using monitor interface, growing in consumer. So I see in consumer something like :
dn: cn=Database 2,cn=Databases,cn=Monitor
structuralObjectClass: monitoredObject
creatorsName:
modifiersName:
createTimestamp: 20090310002542Z
modifyTimestamp: 20090310002542Z
monitoredInfo: bdb
monitorIsShadow: TRUE
namingContexts: ou=INDEXES,o=domain,c=fr
readOnly: FALSE
monitorUpdateRef: ldap://10.142.15.41:389
olmBDBEntryCache: 9999
olmBDBDNCache: 1309605
olmBDBIDLCache: 4715
olmDbDirectory: /var/openldap-data/bdb2/
And I didn't make any search to consumer(manual) DB. This is being caused by syncrepl. Not sure why since I wasn't expecting it. The cache information can be seen in the files attached.
Please let me know if you think this could be a real problem or some configuration could solve this behavior.
Thanks,
Rodrigo Costa.
14 years, 8 months
Strange behavior with relay backend and ACLS
by David
With a relay backend, when i enable an ACL then i can only get back full
entries, but not specific attributes:
1. # extended LDIF
2. #
3. # LDAPv3
4. # base <dc=thunderbird> with scope subtree
5. # filter: uid=dmarkey
6. # requesting: ALL
7. #
8.
9. # dmarkey, user, thunderbird
10. dn: uid=dmarkey,ou=user,dc=thunderbird
11. objectClass: top
12. objectClass: person
13. objectClass: organizationalPerson
14. objectClass: inetOrgPerson
15. objectClass: posixAccount
16. objectClass: shadowAccount
17. objectClass: krb5Principal
18. objectClass: krb5KDCEntry
19. objectClass: sambaSamAccount
20. sn: Markey
21. givenName: David
22. uid: dmarkey
23. mail: dmarkey@xxxx
24. cn: David Markey Staff
25.
26. # search result
27. search: 2
28. result: 0 Success
29.
30. # numResponses: 2
31. # numEntries: 1
Thats normal, Now we'll try to just get the mail attribute:
1. Robinson:/opt/openldap/etc/openldap # ldapsearch -b dc=thunderbird -x
uid=dmarkey mail
2. # extended LDIF
3. #
4. # LDAPv3
5. # base <dc=thunderbird> with scope subtree
6. # filter: uid=dmarkey
7. # requesting: mail
8. #
9.
10. # search result
11. search: 2
12. result: 0 Success
13.
14. # numResponses: 1
Nothing is returned.
Here is the relay database definition
database relay
suffix "dc=thunderbird"
relay "dc=example,dc=ie"
overlay rwm
overlay memberof
rwm-rewriteEngine on
rwm-suffixmassage "dc=example,dc=ie"
map attribute cn gecos
map attribute mail *
map attribute uid *
map attribute sn *
map attribute givenname *
map attribute memberof *
map attribute *
access to filter="memberOf=cn=staff,ou=groupofnames,dc=thunderbird"
by * read
Anyone see what im doing wrong here?
Thanks.
14 years, 8 months
syncprov_db_open: invalid config, lastmod must be enabled
by Tim Gustafson
Hi,
I'm running slapd 2.4.15 on FreeBSD 7.1. I'm trying to work with the "chain" overlay. I've enabled the LDAP backend, and as soon as I put:
overlay chain
in slapd.conf, I start getting this in my error log:
syncprov_db_open: invalid config, lastmod must be enabled
Lastmod is enabled. I'm attaching my config file to this e-mail, as well as one restart worth of log file. Can someone please help me understand what I'm doing wrong? Thanks!
Tim Gustafson
BSOE Webmaster
UC Santa Cruz
tjg(a)soe.ucsc.edu
831-459-5354
14 years, 9 months
