openldap-software

openldap-software@openldap.org

2719 discussions

Re: slapd syncrepl consumer having permanent high CPU load
by Rodrigo Costa 25 Mar '09

25 Mar '09

Howard and all, I made more tests and looks like problem persists. I saw some changes but only in the memory consumption in "consumer(slave)" syncrepl. Let me try to explain better. I have a pair of provider/consumer machines where one machine will always receive all read/writes and the other is just for High Availability(HA) purposes, so it is better have the more close as possible the DBs. I start the provider(master) and then just after start the consumer(slave). The configuration doesn't appear to have problems since I have in my configuration 2 DBs, CONTENT and INDEX, and I see consumer doing 2 searches in these DBs when started(this is ok). After this both consumer and provider CPU usage increases so as memory allocation by slapd process. After the HEAD changes the memory consumption in consumer increases in a much more fast rate, something like 10:1. In this way to reproduce the issue I needed to reduce the dncachesize directive in consumer to 1/10 of the provider value, or from 4,000,000 to 400,000. This avoid the process to consume all memory before the issue arises. Let me try to summarize : 1) Start provider(mater) slapd process; 2) Start consumer(slave) slapd process; 3) Monitor memory and CPU usage in both provider and consumer; 4) Make sometimes a monitor check to see the cache information; 5) Before cache is full in provider(master) I made a gdb debug to check the consumer(slave) process threads; 6) Wait until the consumer(slave) process starts to use around 200% CPU and then collect again a gdb debug; 7) Wait a little more until the provider(master) CPU usage becomes 0% and then see that consumer(slave) CPU stay stable in 200%. Collect a gdb debug. 8) Wait some more time just for more gdb debug to see if something changed. I re-compile the HEAD with GDB symbols for debugging. In this way I created the file attached where more than once I collect the debug information from the consumer slapd(includes the syncrepl thread). Please see file attached for details. The item 7) is the issue I think is happening. The synchronization never ends, the responsiveness from consumer(slave) to queries is very slow, CPU usage becomes fixed in 200%, and then the logic appears never be working as expected, or in the end never synchronizing. In the end appears that syncrepl still with some issue to synchronize the DBs. Regards, Rodrigo. --- On Thu, 3/19/09, Howard Chu <hyc(a)symas.com> wrote: > From: Howard Chu <hyc(a)symas.com> > Subject: Re: slapd syncrepl consumer having permanent high CPU load > To: rlvcosta(a)yahoo.com > Cc: openldap-software(a)openldap.org, "John Morrissey" <jwm(a)horde.net> > Date: Thursday, March 19, 2009, 2:04 PM > Rodrigo Costa wrote: > > > > Folks, > > > > I was preparing openLDAP with GDB symbols but looks > like the issue was > identified and solved in HEAD. Just to identify this issue; > was created any > sort of ITS for verification in a new load? > > No, the further work was just associated with ITS#5860. > > > Sorry my late response but my baby daughter just born > last week and I was > having some work at home. > > Congratulations! > > > I will give a try in the HEAD load. > > Try RE24 now, that's the current release candidate. > > > > Best Regards, > > > > Rodrigo. > > > > PS-> Just some link from my daughter > > http://sites.google.com/site/lauramenina/laura_english > > > > --- On Wed, 3/18/09, Howard Chu<hyc(a)symas.com> > wrote: > > > >> From: Howard Chu<hyc(a)symas.com> > >> Subject: Re: slapd syncrepl consumer having > permanent high CPU load > >> To: "John Morrissey"<jwm(a)horde.net> > >> Cc: openldap-software(a)openldap.org > >> Date: Wednesday, March 18, 2009, 5:21 AM > >> John Morrissey wrote: > >>> After ~16h uptime, slapd with this BDB had > increased > >> its DN cache to ~250k > >>> entries after it previously appeared stable at > the > >> configured 20k entries, > >>> and its entry cache had ballooned to ~480k > entries. > >> Its RSS was about 3.6GB > >>> at this point, with a BDB cache size of 2GB. > >> > >> I was finally able to reproduce this (took several > hours of > >> searches. Fortunately I was at a St. Pat's party > so I didn't > >> have to wait around, just got home in time to see > it start > >> going bad...). A fix is now in HEAD. > >> > >> (And now we'll see if Guinness is Good For Your > Code... ;) > >> -- -- Howard Chu > >> CTO, Symas Corp. > >> http://www.symas.com > >> Director, Highland Sun > http://highlandsun.com/hyc/ > >> Chief Architect, OpenLDAP http://www.openldap.org/project/ > >> > > > > > > > > > > > > > -- > -- Howard Chu > CTO, Symas Corp. > http://www.symas.com > Director, Highland Sun > http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ >

2 1

Re: Mirrormode and cn=config
by Mathieu MILLET 25 Mar '09

25 Mar '09

On Wed, 25 Mar 2009 15:09:09 +0100, Peter Mogensen <apm(a)mutex.dk> wrote: > Hi, > [snip] > > Second: > I have problem with SASL/EXTERNAL and TLS. The server can't seem to find > the client certificate. I'm using slapd from Debian Lenny and Ubuntu > Hardy, and it's probably due to GnuTLS problems. > I get error from slapd like: > "TLS: can't accept: A TLS packet with unexpected length was received.." > "unable to get TLS client DN, error=-4 id=0" > > Are GnuTLS just completely broken on Debian Lenny or can this be made to > work? Which version of OpenLDAP are you using ? If using 2.4.15, the ldap "client" libs have broken SASL/EXTERNAL implementation. These libs are also used for consumer to connect to provider. Patch already submitted and to be available with 2.4.16. > /Peter Hope it can help, Sincerely yours, Mathieu.

2 1

Xenserver 5 + Centos 5.2 x64 DomU + DB 4.7.25 + Openldap 2.4.13 or 2.4.15 do not work
by Steven Truong 24 Mar '09

24 Mar '09

Dear, all. I googled and found some people mentioned that openldap does not work under Xen with db database backends and the information were from 2006 http://www.openldap.org/lists/openldap-software/200603/msg00199.html. I actually encountered this myself and I am not sure if this is still the same problem. Openldap and DB were compiled and installed from sources and everything was configure manually. My /etc/hosts include entry such as: 192.168.10.119 alfresco.mynetwork.com alfresco. [root@alfresco ~]# hostname alfresco.mynetwork.com [root@alfresco ~]# hostname -f alfresco.mynetwork.com Could some experts please look into this problem and suggest ways to fix it? Thank you. Attached is the output for strace for this problem and below is the part where the problem started. 2499 open("/etc/hosts", O_RDONLY) = 4 2499 fcntl(4, F_GETFD) = 0 2499 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 2499 fstat(4, {st_mode=S_IFREG|0644, st_size=294, ...}) = 0 2499 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b32b87fd000 2499 read(4, "# Do not remove the following li"..., 4096) = 294 2499 close(4) = 0 2499 munmap(0x2b32b87fd000, 4096) = 0 2499 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0 2499 pipe([4, 5]) = 0 2499 epoll_create(1024) = 6 2499 open("/etc/gai.conf", O_RDONLY) = -1 ENOENT (No such file or directory) 2499 futex(0x3f837509a8, FUTEX_WAKE, 2147483647) = 0 2499 socket(PF_NETLINK, SOCK_RAW, 0) = 7 2499 bind(7, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 2499 getsockname(7, {sa_family=AF_NETLINK, pid=2499, groups=00000000}, [17449945317507072012]) = 0 2499 time(NULL) = 1237846847 2499 sendto(7, "\24\0\0\0\26\0\1\3?\v\310I\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20 2499 connect(7, {sa_family=AF_INET6, sin6_port=htons(389), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0 2499 getsockname(7, {sa_family=AF_INET6, sin6_port=htons(42391), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [8589934620]) = 0 2499 close(7) = 0 2499 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 7 2499 connect(7, {sa_family=AF_INET, sin_port=htons(389), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 2499 getsockname(7, {sa_family=AF_INET, sin_port=htons(52357), sin_addr=inet_addr("127.0.0.1")}, [8589934608]) = 0 2499 close(7) = 0 2499 socket(PF_INET6, SOCK_STREAM, IPPROTO_IP) = 7 2499 setsockopt(7, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 2499 setsockopt(7, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 2499 bind(7, {sa_family=AF_INET6, sin6_port=htons(389), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0 2499 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 8 2499 setsockopt(8, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 2499 bind(8, {sa_family=AF_INET, sin_port=htons(389), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 2499 time([1237846847]) = 1237846847 2499 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0 2499 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0 2499 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0 2499 sendto(3, "<167>Mar 23 15:20:47 slapd[2499]"..., 72, MSG_NOSIGNAL, NULL, 0) = 72 2499 time([1237846847]) = 1237846847 2499 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0 2499 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0 2499 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0 2499 sendto(3, "<167>Mar 23 15:20:47 slapd[2499]"..., 49, MSG_NOSIGNAL, NULL, 0) = 49 2499 close(3) = 0 2499 time([1237846847]) = 1237846847 2499 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0 2499 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0 2499 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0 2499 socket(PF_FILE, SOCK_DGRAM, 0) = 3 2499 fcntl(3, F_SETFD, FD_CLOEXEC) = 0 2499 connect(3, {sa_family=AF_FILE, path="/dev/log"}, 110) = 0 2499 sendto(3, "<167>Mar 23 15:20:47 slapd[2499]"..., 75, MSG_NOSIGNAL, NULL, 0) = 75 2499 shutdown(5, 2 /* send and receive */) = -1 ENOTSOCK (Socket operation on non-socket) 2499 close(5) = 0 2499 shutdown(4, 2 /* send and receive */) = -1 ENOTSOCK (Socket operation on non-socket) 2499 close(4) = 0 2499 close(6) = 0 2499 exit_group(1) = ? 2498 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 2499 2498 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 2498 --- SIGCHLD (Child exited) @ 0 (0) --- 2498 wait4(-1, 0x7fffedefdfc4, WNOHANG, NULL) = -1 ECHILD (No child processes) 2498 rt_sigreturn(0xffffffffffffffff) = 0 2498 rt_sigaction(SIGINT, {SIG_IGN}, {SIG_IGN}, 8) = 0 2498 exit_group(1) = ? 2470 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 2498 2470 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 2470 --- SIGCHLD (Child exited) @ 0 (0) --- 2470 wait4(-1, 0x7fffec338194, WNOHANG, NULL) = -1 ECHILD (No child processes) 2470 rt_sigreturn(0xffffffffffffffff) = 0 2470 rt_sigaction(SIGINT, {SIG_IGN}, {SIG_IGN}, 8) = 0 2470 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 2470 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 2470 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 2470 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 2470 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 2470 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 2470 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 2470 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 2470 write(1, "\33[60G", 5) = 5 2470 write(1, "[", 1) = 1 2470 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 2470 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 2470 write(1, "\33[0;31m", 7) = 7 2470 write(1, "FAILED", 6) = 6 2470 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 2470 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 2470 write(1, "\33[0;39m", 7) = 7 2470 write(1, "]", 1) = 1 2470 write(1, "\r", 1) = 1 2470 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 2470 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 2470 stat("/usr/bin/rhgb-client", {st_mode=S_IFREG|0755, st_size=13704, ...}) = 0 2470 geteuid() = 0 2470 getegid() = 0 2470 getuid() = 0 2470 getgid() = 0 2470 access("/usr/bin/rhgb-client", X_OK) = 0 2470 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 2470 rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0 2470 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b1cbe77fe50) = 2500 2500 close(255 <unfinished ...> 2470 rt_sigprocmask(SIG_SETMASK, [], <unfinished ...> 2500 <... close resumed> ) = 0

2 2

Re: slapd syncrepl consumer having permanent high CPU load
by Rodrigo Costa 24 Mar '09

24 Mar '09

1 0

slapo-dynlist authentication
by Wes Rogers 23 Mar '09

23 Mar '09

I've got slapo-dynlist working : dn: cn=Dynamic Group,ou=Groups,dc=example,dc=com objectClass: groupOfURLs cn: Dynamic Group memberURL: ldap:///ou=Users,ou=Employees,dc=example,dc=com?uid?sub?(objectClass=*) memberURL: ldap:///ou=Users,ou=Customers,dc=example,dc=com?uid?sub?(objectClass=*) uid: employee1 uid: customer1 Although, I'm curious if anyone is using this to create dynamic groups to authenticate users with apache, etc. We can talk off list if desired, so it doesn't get extremely off topic. Thanks.

1 0

core dump with interactive password
by lacoste＠miage.univ-paris12.fr 22 Mar '09

22 Mar '09

I'm using openldap 2.4.15 on FreeBSD 6.4 ldapsearch with the -W switch gives me the expected output but ends with ldapsearch in free(): error: junk pointer, too high to make sense Abort trap: 6 (core dumped) ldapsearch with the -w switch does not core dump Regards, Thierry

3 3

slapo-autogroup?
by manu＠netbsd.org 22 Mar '09

22 Mar '09

Hello Anyone has been using slapo-autogroup? I tried this in slapd.conf: overlay autogroup autogroup-attrset groupOfURLs memberURL memberUid As soon as autogroup is configured, any search will return: result: 53 Server is unwilling to perform text: operation not supported within namingContext A quick look at the code shows tthat the search method is not implemented, but the same thing does not prevent other overlays from working. Did I misconfigured it, or is it broken? I would like to be sure I did not misunderstood how it works before starting hacking the code. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz manu(a)netbsd.org

1 1

Why Use accesslog overlay?
by Pete Giesin 20 Mar '09

20 Mar '09

I am working on setting up OpenLDAP for a web project. I had originally planned on using the accesslog overlay to track all access to the LDAP. I have discovered that the use of this overlay has a tremendous impact on performance. I started out by auditing all operations. With this configuration I was getting 40-50 reads/sec max. I then decided that I only needed to audit writes. Making this change sent the reads through the roof, but the writes were still only averaging 16-20 per second. I am in the process of retesting the writes with no accesslog. My assumption is that the writes per second will jump dramtically. So I have to question what is the purpose of the accesslog overlay? Is it really needed and if so is there are way to increase its performance. I have to admit that I have not paid attention to the settings for the accesslog backend. Do I need to tweak these settings just like I did for my primary backend? If so what are the optimal settings for a write intensive database? Thanks, Pete

4 5

delta-syncrepl missing changes
by Francis Swasey 20 Mar '09

20 Mar '09

I am running OpenLDAP 2.3.39 (locally built) on Red Hat Enterprise Linux 4 servers with several replicas. We use delta-syncrepl to keep the replicas in sync with the master server. We also use nagios and monitor the contextcsn value on the replica and alert if it gets too far out of sync with the master server. The issue we have now experienced a few times is that if there are a LOT of updates in the nightly batch update process, that not all of the updates make it to the replicas but the contextcsn stays in sync, so we see strange errors that eventually lead us to see that the replicas are not current even though they think they are. Is this a known issue? I haven't found a syslog entry on the server or the replicas that makes me think it is the flag of the root cause. I have downloaded and built the 2.3.43 release, having installed it on one replica. That replica is just as out of date this morning as the others -- so, if there was a solution between 2.3.39 and 2.3.43 -- it must have been on the provider side not the consumer side. Thanks for any insight. -- Frank Swasey | http://www.uvm.edu/~fcs Sr Systems Administrator | Always remember: You are UNIQUE, University of Vermont | just like everyone else. "I am not young enough to know everything." - Oscar Wilde (1854-1900)

6 28

Re: slapadd performance
by Pete Giesin 19 Mar '09

19 Mar '09

We are using Symas OpenLDAP 2.4.12.1 on a Red Hat EL 5.2 64-bit with 8GB of memory. My DB_CONFIG settings are: set_cachesize 3 0 2 set_lk_max_locks 3000 set_lk_max_objects 1500 set_lk_max_lockers 1500 set_lg_regionmax 262144 set_lg_bsize 2097152 set_lg_dir /opt/symas/var/openldap-logs/amwater set_flags DB_LOG_AUTOREMOVE set_tas_spins 1 The 10 million user load finished in about 18 hours. I believe the primary issue is the fact that we have a single disk on the system. I assume that if I were to move the database and log files off to separate disks I would get much better performance. Thanks, Pete On Wed, Mar 18, 2009 at 11:26 PM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote: > --On Wednesday, March 18, 2009 1:25 PM -0400 Pete Giesin < > pgiesin(a)hubcitymedia.com> wrote: > > I am trying to perform some benchmarking against OpenLDAP. So far I have >> ran my tests against a 100K and 1Million entry database and I have had >> rather decent numbers. My final set of tests were to be ran against a >> 10Million entry database. Unfortunately, I am having difficult loading >> the database with this many entries. I have generated 10 1Million LDIF >> files. I am using "slapadd -c -q -v -l <file>" to import each file. The >> first 2 files took approximately 15 minutes each to load. The remaining 8 >> files are taking progressively longer and longer. So much longer that I >> anticipate the entire proceess to take well over 24 hours. My question is >> is there anything I can do to increase the performance of slapadd. I >> assume that since slapd is not running at this point that the normal >> DB_CONFIG and slapd.conf settings do not have much affect. >> > > What are the settings in your DB_CONFIG file? It is absolutely critical to > the performance of slapadd. What version of OpenLDAP are you using? What > version of BDB? What operating system? > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration >

4 3

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-software