Full_Name: Thomas Bopp
Version: 4.2.22
OS: Solaris 10
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (195.47.255.100)
After approximately one week our LDAP stops working and hangs (we had this
situation several times). We have the www.sunfreeware.com package of LDAP
installed.
It had a bunch of connection in CLOSE_WAIT state (indicating basically that a
close request has come in but slapd did not serve it).
Making a truss on the process showed all threads blocked on lwp_cond_wait.
I really need a solution for this problem.
Also, this is not the same as ITS#6242, because the patch for that issue (adding the manageDSAit control) already exists
in the versions of OpenLDAP I've tested with (2.4.21, 2.4.23).
ryans(a)aweber.com wrote:
> I am experiencing what appears to be the same issue, and opened a ticket -
> http://www.openldap.org/its/index.cgi?findid=6831 - because this one did not pop up when I searched for existing ITS's
> (perhaps my search terms did not match or I just overlooked it). As I mentioned in the ticket I opened, this is not a
> schema issue - all the schema match on every server, and if that were the problem, none of the users would be served
> properly. What I found is that the keys that proxycache was looking for did not exist in the bdb it was searching. Why
> exactly that is, I'm not sure. Hopefully the information I provided in that ITS (or, perhaps some additional debugging
> information - gdb output or copies of databases exhibiting the problem) will help shed some light. If it is indeed the
> same issue, I'm all for coalescing the two. I have tested this all the way up through 2.4.23, and there are no commits
> to pcache.c or back-ldap/search.c since 2.4.23 was released that addresses this problem.
Since you are not having the objectclass-related problem described in this
ticket it is not the same issue.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
I am experiencing what appears to be the same issue, and opened a ticket -
http://www.openldap.org/its/index.cgi?findid=6831 - because this one did not pop up when I searched for existing ITS's
(perhaps my search terms did not match or I just overlooked it). As I mentioned in the ticket I opened, this is not a
schema issue - all the schema match on every server, and if that were the problem, none of the users would be served
properly. What I found is that the keys that proxycache was looking for did not exist in the bdb it was searching. Why
exactly that is, I'm not sure. Hopefully the information I provided in that ITS (or, perhaps some additional debugging
information - gdb output or copies of databases exhibiting the problem) will help shed some light. If it is indeed the
same issue, I'm all for coalescing the two. I have tested this all the way up through 2.4.23, and there are no commits
to pcache.c or back-ldap/search.c since 2.4.23 was released that addresses this problem.
This may also be related to ITS#6217 (http://www.openldap.org/its/index.cgi?findid=6217). In fact, I'm pretty sure it
is. In response to the last question in that ITS posed by Howard Chu (Followup 18), the schema on the proxy servers
match the schema available on the primary servers exactly. If not, it wouldn't work for some users and not others
simultaneously (that is, users who have the same objectClasses and attributes, and differ only in the user-specific
attribute values like uid, gecos, phone extension, etc.). Hopefully the data and logs I was able to provide (and any
debugger output/cache databases that might be requested of me as the ITS moves forward) will help to fill in some of the
blanks.
Full_Name: Ryan Steele
Version: 2.4.23
OS: Ubuntu Server
URL: ftp://ftp.openldap.org/incoming/ryan-steele-110215.proxycache-failure.log
Submission from: (NULL) (207.106.239.81)
I use back-ldap + proxycache on many of my servers to reduce network traffic and
to alleviate load on the masters, as well as to maintain service continuity in
the event of a network failure. However, we have recently been noticing an
issue where the proxycache database claims that it has the data and that the
query is answerable, but fails to read data from any of the indices it thinks
the data is at. It happens randomly, and to random entries. We do not cache
negative search results, so the cache should never return nentries=0
authoritatively. I can temporarily fix it for some broken users by restarting
slapd and clearing the cache (i.e., pcachePersist is set to FALSE), but
inevitably others stop working (or sometimes, the same users stop working).
When this happens, most entries are still served from the cache just fine, but
the entries that aren't never will unless slapd is restarted. I have tested
this with 2.4.17, 2.4.21, and 2.4.23, using the amd64 architecture, and with
libdb4.6 and libdb4.7.
Included below is the slapd.conf I use on my back-ldap + proxycache nodes, an
example of the behavior using ldapsearch, and the log messages during a failed
search in which using log level 16383:
## Proxycache slapd configuration
# Schema
include /etc/ldap/schema/core.schema
include /etc/ldap/schema/collective.schema
include /etc/ldap/schema/corba.schema
include /etc/ldap/schema/cosine.schema
include /etc/ldap/schema/duaconf.schema
include /etc/ldap/schema/dyngroup.schema
include /etc/ldap/schema/inetorgperson.schema
include /etc/ldap/schema/java.schema
include /etc/ldap/schema/misc.schema
include /etc/ldap/schema/openldap.schema
include /etc/ldap/schema/ppolicy.schema
include /etc/ldap/schema/examplecom.schema
include /etc/ldap/schema/rfc2307bis.schema
include /etc/ldap/schema/samba.schema
include /etc/ldap/schema/apple_auxiliary.schema
include /etc/ldap/schema/apple.schema
# System
pidfile /var/run/slapd/slapd.pid
argsfile /var/run/slapd/slapd.args
loglevel stats
TLSCACertificateFile /etc/ldap/ssl/certs/cacert.pem
TLSCertificateFile /etc/ldap/ssl/certs/openldap.cert.pem
TLSCertificateKeyFile /etc/ldap/ssl/keys/openldap.key.pem
TLSVerifyClient never
# Modules
modulepath /usr/lib/ldap
moduleload back_ldap.la
moduleload back_hdb.la
moduleload pcache.la
# Back-LDAP
database ldap
uri "ldap://ldapmaster.example.com"
suffix "dc=example,dc=com"
rootdn "cn=admin,dc=example,dc=com"
rootpw SECRET
tls start
# ACLs
access to attrs=userPassword
by tls_ssf=128 ssf=128 self write
by tls_ssf=128 ssf=128 anonymous auth
by tls_ssf=128 ssf=128
group/groupOfURLs/Member="cn=ops,ou=Groups,dc=example,dc=com" write
by tls_ssf=128 ssf=128 * compare
access to *
by tls_ssf=128 ssf=128 self write
by tls_ssf=128 ssf=128
group/groupOfURLs/Member="cn=ops,ou=Groups,dc=example,dc=com" write
by tls_ssf=128 ssf=128 * read
# ProxyCache
overlay pcache
proxycache hdb 500000 1 5000 86400
directory /var/lib/ldap/proxycache
index cn eq
index departmentName eq
index entryCSN eq
index entryUUID eq
index gidNumber eq
index mail eq
index member eq
index memberUid eq
index objectClass eq
index pcacheQueryid eq
index uid eq
index uidNumber eq
index uniqueMember eq
proxycachequeries 1000000
proxyattrset 0 apple-user-homeDirectory blogCategory cn dateCreated
departmentName departmentNumber description
displayColor employeeNumber gecos getsPages gidNumber givenName homeDirectory
htaccessPasswd isAvailable isPhoneOperator
lastAdminVisit loginShell mail manager member memberUid mobile mobileEmail
numTickets objectClass ou phoneExtension sn
sortOrder uid uidNumber uniqueMember userPassword
proxytemplate (blogCategory=) 0 86400
proxytemplate (cn=) 0 86400
proxytemplate (dateCreated=) 0 86400
proxytemplate (departmentName=) 0 86400
proxytemplate (departmentNumber=) 0 86400
proxytemplate (description=) 0 86400
proxytemplate (displayColor=) 0 86400
proxytemplate (employeeNumber=) 0 86400
proxytemplate (gecos=) 0 86400
proxytemplate (getsPages=) 0 86400
proxytemplate (gidNumber=) 0 86400
proxytemplate (givenName=) 0 86400
proxytemplate (homeDirectory=) 0 86400
proxytemplate (apple-user-homeDirectory=) 0 86400
proxytemplate (htaccessPasswd=) 0 86400
proxytemplate (isAvailable=) 0 86400
proxytemplate (isPhoneOperator=) 0 86400
proxytemplate (lastAdminVisit=) 0 86400
proxytemplate (loginShell=) 0 86400
proxytemplate (mail=) 0 86400
proxytemplate (manager=) 0 86400
proxytemplate (member=) 0 86400
proxytemplate (memberUid=) 0 86400
proxytemplate (memberURL=) 0 86400
proxytemplate (mobile=) 0 86400
proxytemplate (mobileEmail=) 0 86400
proxytemplate (numTickets=) 0 86400
proxytemplate (objectClass=) 0 86400
proxytemplate (ou=) 0 86400
proxytemplate (phoneExtension=) 0 86400
proxytemplate (sn=) 0 86400
proxytemplate (sortOrder=) 0 86400
proxytemplate (uid=) 0 86400
proxytemplate (uidNumber=) 0 86400
proxytemplate (uniqueMember=) 0 86400
proxytemplate (|(memberUid=)(member=)) 0 86400
proxytemplate (|(memberUid=)(uniqueMember=)) 0 86400
proxytemplate (&(objectClass=)(uid=)) 0 86400
proxytemplate (&(objectClass=)(memberUid=)) 0 86400
proxytemplate (&(objectClass=)(uniqueMember=)) 0 86400
proxytemplate (&(objectClass=)(uidNumber=)) 0 86400
proxytemplate (&(objectClass=)(gidNumber=)) 0 86400
proxytemplate (&(objectClass=)(|(memberUid=)(member=))) 0 86400
proxytemplate (&(objectClass=)(|(memberUid=)(uniqueMember=))) 0 86400
proxytemplate (&(objectClass=)(member=)) 0 86400
proxytemplate (&(objectClass=)(cn=)) 0 86400
proxytemplate (&(|(objectClass=)(objectClass=))(uid=)) 0 86400
## example of the failures using ldapsearch
bash:~# for i in `seq 1 14`; do echo "PROCESSING jdoe$i"; ldapsearch -x -H
ldaps://localhost -LLL -b ou=Users,dc=example,dc=com
'(&(|(objectClass=examplecomEmployee)(objectClass=examplecomUtilityUser))(uid=jdoe'$i'))'
uid; sleep 1; done
PROCESSING jdoe1
dn: uid=jdoe1,ou=Users,dc=example,dc=com
uid: jdoe1
PROCESSING jdoe2
dn: uid=jdoe2,ou=Users,dc=example,dc=com
uid: jdoe2
PROCESSING jdoe3
dn: uid=jdoe3,ou=Users,dc=example,dc=com
uid: jdoe3
PROCESSING jdoe4
dn: uid=jdoe4,ou=Users,dc=example,dc=com
uid: jdoe4
PROCESSING jdoe5
PROCESSING jdoe6
PROCESSING jdoe7
PROCESSING jdoe8
PROCESSING jdoe9
PROCESSING jdoe10
PROCESSING jdoe11
PROCESSING jdoe12
PROCESSING jdoe13
PROCESSING jdoe14
bash:~#
A log file (with log level set to 16383) showing what happens when the cache is
queried and it responds with "QUERY ANSWERABLE", and then fails to read data
from any of the indices referenced, can be found at
ftp://ftp.openldap.org/incoming/ryan-steele-110215.proxycache-failure.log. It
seems similar to ITS#6242, but my version of pcache.c, at least in the 2.4.21
and 2.4.23 versions of OpenLDAP, definitely contain that patch, as I can see it
in the source (the manageDSAit control). Please let me know if you need any
other information to debug this problem (e.g., specific variables from a
debugger run, copy of a proxycache database experiencing the problem, etc.)
Howard Chu wrote:
> Yes, several objections. Post diffs that can be applied by the "patch"
> command, not the complete file.
See below.
> If you're going to add more rules, make
> them conform to the standard Makefiles. E.g. the correct macro is
> "prefix" not "PREFIX".
Not consequently used in contrib/slapd-modules/. Therefore I didn't know this.
Ciao, Michael.
Index: Makefile
===================================================================
RCS file: /repo/OpenLDAP/pkg/ldap/contrib/slapd-modules/lastbind/Makefile,v
retrieving revision 1.2.2.2
diff -u -r1.2.2.2 Makefile
--- Makefile 4 Feb 2011 23:39:17 -0000 1.2.2.2
+++ Makefile 15 Feb 2011 14:48:30 -0000
@@ -10,6 +10,8 @@
# top-level directory of the distribution or, alternatively, at
# <http://www.OpenLDAP.org/license.html>.
+prefix=/usr/local
+
CPPFLAGS+=-I../../../include -I../../../servers/slapd
CPPFLAGS+=-DSLAPD_OVER_LASTBIND=SLAPD_MOD_DYNAMIC
#LIBTOOL=libtool
@@ -23,7 +25,12 @@
lastbind.la: lastbind.lo
$(LIBTOOL) --mode=link $(CC) -version-info 0:0:0 \
- -rpath $(PREFIX)/lib -module -o $@ $?
+ -rpath $(prefix)/lib -module -o $@ $?
clean:
rm -rf lastbind.lo lastbind.la lastbind.o .libs/
+
+install: lastbind.la
+ mkdir -p $(prefix)/libexec/openldap
+ $(LIBTOOL) --mode=install cp lastbind.la $(prefix)/libexec/openldap
+ $(LIBTOOL) --finish $(prefix)/libexec/openldap
On Tue, Feb 15, 2011 at 05:02:52AM -0800, Howard Chu wrote:
> >slapo-ppolicy.5 incorrectly includes the NO-USER-MODIFICATION flag in the schema
> >fragments for pwdPolicySubentry and pwdAccountLockedTime.
>
> That's how they were defined in the IETF Draft. The schema fragments
> in the manpage were copied directly from the spec. The fact that the
> current implementation deviates from the spec is just out of
> necessity to make things work at all in our present code base.
Certainly the use of pwdPolicySubentry differs from the
intention of the draft (which I believe was intending to use
real X.500-style subentries).
The case of pwdAccountLockedTime is arguable.
draft-behera-ldap-password-policy-xx.txt says:
This attribute holds the time that the user's account was locked. A
locked account means that the password may no longer be used to
authenticate. A 000001010000Z value means that the account has been
locked permanently, and that only a password administrator can unlock
the account.
Unfortunately it says nothing about *how* a password
administrator should do that when the attribute is marked
NO-USER-MODIFICATION. I would argue that this is a
deficiency in the draft, and that the current OpenLDAP
behaviour is more useful.
> Things will not always work this way...
Indeed, but I would prefer the manpages to reflect the
reality of the current release!
Andrew
--
-----------------------------------------------------------------------
| From Andrew Findlay, Skills 1st Ltd |
| Consultant in large-scale systems, networks, and directory services |
| http://www.skills-1st.co.uk/ +44 1628 782565 |
-----------------------------------------------------------------------
andrew.findlay(a)skills-1st.co.uk wrote:
> Full_Name: Andrew Findlay
> Version: 2.4.24
> OS: OpenSuSE 11.3
> URL: ftp://ftp.openldap.org/incoming/afindlay-ppolicy-man-patch-20110215
> Submission from: (NULL) (88.97.25.132)
>
>
> slapo-ppolicy.5 incorrectly includes the NO-USER-MODIFICATION flag in the schema
> fragments for pwdPolicySubentry and pwdAccountLockedTime.
That's how they were defined in the IETF Draft. The schema fragments in the
manpage were copied directly from the spec. The fact that the current
implementation deviates from the spec is just out of necessity to make things
work at all in our present code base. Things will not always work this way...
> In addition, the description of pwdAccountLockedTime does not make it clear that
> this attribute can be changed by administrative action.
>
> The attached patch is a suggested clarification for the manpage.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/