openldap-bugs October 2009

openldap-bugs@openldap.org

34 participants
223 discussions

Re: (ITS#6322) slapd suddenly stops working, and starts using 100% CPU
by hyc＠symas.com 02 Oct '09

02 Oct '09

helge(a)monsternett.no wrote: > Full_Name: Helge Milde > Version: slapd 2.4.11 > OS: Debian 5 > URL: > Submission from: (NULL) (81.175.2.164) > > > Hello, > > In the last two days, slapd has randomly stopped working on our server, about > one time every two or three hours. Before this, slapd had been running without > any problems for about two months. slapd.conf has not been touched since then. > > What happens is that slapd stops taking queries (postfix, dovecot, NSS etc. > starts getting timeouts), and it starts using exactly 100% of one core until we > restart it. > After restart, everything works again (until this happens again). > > It seems to only happen when slapd is under "stress" (we have an 8-core system > with 8G RAM, and so slapd is really never under much "stress"; the biggest CPU > load I've seen is about 40% on one core). The last ldap search before slapd > stops working is always the same: > Oct 2 10:57:38 oi-mail slapd[24550]: conn=1543 op=1 SRCH > base="ou=groups,dc=web,dc=xxxx,dc=net" scope=1 deref=0 > filter="(&(objectClass=posixGroup))" > .. which should return about 1800 entries. Different connections might query > this simultaneously, and it's a query that happens every ~10 seconds on > average. > > I've tried to reproduce this by doing .. > # while [ 1 ]; do ldapsearch -x -H ldaps://.. -b > 'ou=groups,dc=web,dc=xxxx,dc=net' '(&(objectClass=posixAccount))'; done > .. for two minutes without any "luck" :). slapd just uses a bit more CPU, but > never stops working. > > With "loglevel stats stats2", I get nothing out of the logs: > Oct 2 08:42:15 oi-mail slapd[15836]: conn=34798 op=1 ENTRY > dn="cn=xxxx,ou=groups,dc=web,dc=xxxx,dc=net" > ... time of "crash" (we're mid-search at this point); nothing gets logged until > we do a restart: ... > Oct 2 08:42:30 oi-mail slapd[15836]: daemon: shutdown requested and initiated. > Oct 2 08:42:30 oi-mail slapd[15836]: conn=34391 fd=31 closed (slapd shutdown) > ... many entires like the one above, and then I seem to be getting the next > entry from the same connection, right before it closes: ... > Oct 2 08:42:30 oi-mail slapd[15836]: conn=34798 op=1 ENTRY > dn="cn=xxxx,ou=groups,dc=web,dc=xxxx,dc=net" > Oct 2 08:42:30 oi-mail slapd[15836]: conn=34798 fd=81 closed (slapd shutdown) > > I've tried logging more than just stats, but the interesting loglevels also > causes a big performance hit (trace and args), and I'm not really sure which of > the other loglevels could be helpful. > This is a live ldap server, on a server that also hosts other services, so I > simply can't take the performance hit. > > I also did a strace -p<slapd pid>, which only gives "futex(0x41c499e0, > FUTEX_WAIT," until I restart slapd, but the "restarting routine" could be > useful, so here it is: http://www.pastie.org/639170 strace is useless. Use gdb and get a trace of all running threads when this occurs. http://www.openldap.org/faq/data/cache/59.html -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#6323) smbk5pwd: Makefile links module for a different directory than it's installed to
by jonathan＠phillipoux.net 02 Oct '09

02 Oct '09

Full_Name: Jonathan Clarke Version: RE24 OS: Linux URL: ftp://ftp.openldap.org/incoming/jonathan-clarke-smbk5pwd-rpath-20091002.pat… Submission from: (NULL) (82.67.204.30) Greetings, The makefile in the smbk5pwd contrib module runs the libtool link phase with rtool=$(libdir), but then installs the module to $(moduledir), which can be different, and thus causes install to bork: ../../../libtool --mode=install cp smbk5pwd.la /usr/lib/openldap libtool: install: error: cannot install `smbk5pwd.la' to a directory not ending in /usr/lib The patch above fixes this, removing libdir completely from the Makefile as it doesn't seem to be needed.

1 0

Re: (ITS#6320)
by j＠telepaths.org 02 Oct '09

02 Oct '09

OK I have a fully-functional accesslog configured on all applicable Provider servers. The respective DIT is now being watched for all WRITES (delete, add, modify and modrdn). I made a change a few minutes ago to a test record, and it DID replicate to ALL slapd servers, so we're looking good in the "short- term". I will check back with you guys later today with any developments, but we should know if there's still a problem today, and maybe now I can get some visibility into it. Thanks again to both of you, Howard and Quanah. Jeff

1 0

Re: (ITS#6320)
by j＠telepaths.org 02 Oct '09

02 Oct '09

Ah, I see. Then the accesslog overlay can be loaded and used on the Provider, but Consumers won't query the Provider any differently than with regular SyncRepl. I will promptly try this, thanks Howard. Jeff On Oct 1, 2009, at 23:39 , Howard Chu wrote: > j(a)telepaths.org wrote: >> Sorry about that .... resending to ITS address: >> >> Regarding use of the accesslog DB ..... is this still an option when >> using 'mirrormode' ? I understand at least in the recent past, this >> option was not compatible with the accesslog overlay ..... > > That's not correct. The accesslog overlay can always be used for > logging purposes. It is just that using it for multimaster delta- > syncrepl is not yet supported. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#6322) slapd suddenly stops working, and starts using 100% CPU
by helge＠monsternett.no 02 Oct '09

02 Oct '09

Full_Name: Helge Milde Version: slapd 2.4.11 OS: Debian 5 URL: Submission from: (NULL) (81.175.2.164) Hello, In the last two days, slapd has randomly stopped working on our server, about one time every two or three hours. Before this, slapd had been running without any problems for about two months. slapd.conf has not been touched since then. What happens is that slapd stops taking queries (postfix, dovecot, NSS etc. starts getting timeouts), and it starts using exactly 100% of one core until we restart it. After restart, everything works again (until this happens again). It seems to only happen when slapd is under "stress" (we have an 8-core system with 8G RAM, and so slapd is really never under much "stress"; the biggest CPU load I've seen is about 40% on one core). The last ldap search before slapd stops working is always the same: Oct 2 10:57:38 oi-mail slapd[24550]: conn=1543 op=1 SRCH base="ou=groups,dc=web,dc=xxxx,dc=net" scope=1 deref=0 filter="(&(objectClass=posixGroup))" .. which should return about 1800 entries. Different connections might query this simultaneously, and it's a query that happens every ~10 seconds on average. I've tried to reproduce this by doing .. # while [ 1 ]; do ldapsearch -x -H ldaps://.. -b 'ou=groups,dc=web,dc=xxxx,dc=net' '(&(objectClass=posixAccount))'; done .. for two minutes without any "luck" :). slapd just uses a bit more CPU, but never stops working. With "loglevel stats stats2", I get nothing out of the logs: Oct 2 08:42:15 oi-mail slapd[15836]: conn=34798 op=1 ENTRY dn="cn=xxxx,ou=groups,dc=web,dc=xxxx,dc=net" ... time of "crash" (we're mid-search at this point); nothing gets logged until we do a restart: ... Oct 2 08:42:30 oi-mail slapd[15836]: daemon: shutdown requested and initiated. Oct 2 08:42:30 oi-mail slapd[15836]: conn=34391 fd=31 closed (slapd shutdown) ... many entires like the one above, and then I seem to be getting the next entry from the same connection, right before it closes: ... Oct 2 08:42:30 oi-mail slapd[15836]: conn=34798 op=1 ENTRY dn="cn=xxxx,ou=groups,dc=web,dc=xxxx,dc=net" Oct 2 08:42:30 oi-mail slapd[15836]: conn=34798 fd=81 closed (slapd shutdown) I've tried logging more than just stats, but the interesting loglevels also causes a big performance hit (trace and args), and I'm not really sure which of the other loglevels could be helpful. This is a live ldap server, on a server that also hosts other services, so I simply can't take the performance hit. I also did a strace -p <slapd pid>, which only gives "futex(0x41c499e0, FUTEX_WAIT," until I restart slapd, but the "restarting routine" could be useful, so here it is: http://www.pastie.org/639170 I understand that this probably isn't enough data to see what the problem is, and so I'm hoping to get some help in debugging this properly. As I've said, args/trace loglevel takes too much of a performance hit, but if this is the only way, we might reconcider. Interesting parts of slapd.conf below. We use slapd.conf instead of cn=config, everything goes over TLS, we're AFAIK only using LDAPv3 binding, and we only have one database entry in slapd.conf. moduleload back_hdb sizelimit 99999 tool-threads 8 threads 16 concurrency 32 backend hdb database hdb cachesize 60000 dbconfig set_cachesize 0 52428800 0 dbconfig set_lk_max_objects 1500 dbconfig set_lk_max_locks 1500 dbconfig set_lk_max_lockers 1500 lastmod on checkpoint 512 30 Best regards, Helge Milde

1 0

Re: (ITS#6320)
by hyc＠symas.com 01 Oct '09

01 Oct '09

j(a)telepaths.org wrote: > Sorry about that .... resending to ITS address: > > Regarding use of the accesslog DB ..... is this still an option when > using 'mirrormode' ? I understand at least in the recent past, this > option was not compatible with the accesslog overlay ..... That's not correct. The accesslog overlay can always be used for logging purposes. It is just that using it for multimaster delta-syncrepl is not yet supported. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6217) proxycache not returning cached data
by masarati＠aero.polimi.it 01 Oct '09

01 Oct '09

> Sounds more like a schema config issue then. Most likely you don't have > the > matching schema in the proxy server, so it is unable to store the entries > it > has received. That was my first thought as well, but the proxy's slapd.conf he posted earlier included some samba-related schema file (I'm blindly assuming that file was defining sambaSamAccount). p.

1 0

(ITS#6320)
by j＠telepaths.org 01 Oct '09

01 Oct '09

Sorry about that .... resending to ITS address: Regarding use of the accesslog DB ..... is this still an option when using 'mirrormode' ? I understand at least in the recent past, this option was not compatible with the accesslog overlay ..... Jeff

1 0

Re: (ITS#6217) proxycache not returning cached data
by hyc＠symas.com 01 Oct '09

01 Oct '09

Jim van Keulen wrote: > Howard Chu wrote: >> w.van.keulen(a)few.vu.nl wrote: >>> masarati(a)aero.polimi.it wrote: >>>>> Here are slapd.conf and proxycache.conf as attachments. >>>> >>>> I just wanted to feed back before I leave for a few days. I don't see >>>> anything critical in your files. You explicitly give the rootdn write >>>> permissions, which is useless, but apart from this everything looks >>>> fine. >>>> >>>> I've tested a simple setup and it works as expected. I used HEAD and >>>> re24 >>>> code; the latter is nearly identical to 2.4.17. I also tested with TLS. >>>> >>>> At this point, I'm a bit clueless. There must be something we're >>>> overlooking. >> >>> Here is some more info, that I just discovered. >>> If my filter rule is >>> '(&(objectClass=sambaSamAccount)(uid=wvk))' >>> the data is cached the first time, but not returned >>> the second and later time. However when my filter rule is >>> (&(objectClass=posixAccount)(uid=wvk))' >>> it works as expected. >> >> Sounds like you may have already had the entry in the cache DB but with >> an incompatible objectclass before, and the attempt to update it with >> the new objectclass failed. This seems to be related to ITS#6242 which >> was recently fixed in HEAD. Can you please retest with the latest >> overlays/pcache.c? >> > > As you requested I retested with HEAD. However the cache is still not > working correctly. > > When my filter rule is: > '(&(objectClass=posixAccount)(uid=<who>))' the named attributes are > cached and correctly returned from the cache for every uid from the > existing user base > When my filter rule is: > '(&(objectClass=sambaSamAccount)(uid=<who>))' the named attributes are > cached and correctly returned from the cache for a subset of the > existing user base only. For all uid's the query is cached (according to > the log), but for some no data is returned. Sounds more like a schema config issue then. Most likely you don't have the matching schema in the proxy server, so it is unable to store the entries it has received. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6320) Replication Issues
by quanah＠zimbra.com 01 Oct '09

01 Oct '09

--On Thursday, October 01, 2009 12:20 PM -0700 Jeff Doyle <j(a)telepaths.org> wrote: > Yes, that is the first thing attempted (both via using LOCAL4 -> syslog > and via running slapd via cmd line) > > We see a brand-new server replicate all records (sids, csns, etc all look > fine) without incident. An existing server will receive, as I said, all > of the changes ... at least for awhile. > > Nothing in the sync activity looks out of the ordinary (and I have read > quite a few posts to be able to discern what the items inside mean) > > When a replication error occurs, often nothing at all is seen inside the > logs (no error mesg, no segfault, nothing) to indicate a failure has > occurred. Oftentimes, when SyncRepl fails to replicate, it is NOT during > testing, so it is difficult for us to find out exactly WHEN the error > occurred, as it could have happened at any point within the past 24 hrs, > and up to a week ago. > > I can try to replicate this error deliberately. Unfortunately it is very > difficult to make happen at-will. My concern is that someone will try to > close this ticket before I have witnessed the event. Hm, I would suggest creating an accesslog database on the master that records all of your changes. That way, you can look at the CSN stamp of what failed and look at the change in the accesslog DB. It sounds like however that the replica syncrepl is simply failing to see the change occur on the master if it isn't logging anything about the missing change. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs October 2009