- openldap-bugs - openldap.org

Re: (ITS#5460) unresolved symbol
by hyc＠symas.com 15 Apr '08

15 Apr '08

st-wong(a)cuhk.edu.hk wrote: > Full_Name: ST Wong > Version: 2.4.8 > OS: CentOS 5.1 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (137.189.29.184) > > > Got error when compiling Openldap 2.4.8 with BerkeleyDB 4.6: > > > ---------------------- cut here ----------------------- > ... > checking for db.h... yes > checking for Berkeley DB major version... 4 > checking for Berkeley DB minor version... 3 > checking for Berkeley DB link (-ldb-4.3)... (cached) no > checking for Berkeley DB link (-ldb43)... no > checking for Berkeley DB link (-ldb-43)... no > checking for Berkeley DB link (-ldb-4-3)... no > checking for Berkeley DB link (-ldb-4)... no > checking for Berkeley DB link (-ldb4)... no > checking for Berkeley DB link (-ldb)... yes > checking for Berkeley DB version match... yes > checking for Berkeley DB thread support... yes > checking Berkeley DB version for BDB/HDB backends... no > configure: error: BDB/HDB: BerkeleyDB version incompatible > ---------------------- cut here ----------------------- > > In config.log: > ---------------------- cut here ----------------------- > configure:17625: checking for resolver link (default) > configure:17691: cc -o conftest -g -O2 -I/usr/local/BerkeleyDB.4.6/include > -L/usr/local/BerkeleyDB.4.6/lib conftest.c>&5 > /tmp/ccczbyKW.o: In function `main': > /usr/local/src/openldap-2.4.8/conftest.c:113: undefined reference to > `__res_query' > /usr/local/src/openldap-2.4.8/conftest.c:132: undefined reference to > `__dn_expand' > collect2: ld returned 1 exit status > ---------------------- cut here ----------------------- Your config.log extract doesn't match the error text above. > Have to add "-lresolv" to LDFLAGS in order to resolve the problem. I suppose > configure will add all libraries required. Configure finds -lresolv automatically. That's certainly not the cause of your problem. Your configure session indicates that your system's default db.h came from BerkeleyDB 4.3 which is explicitly not supported. There's no indication of any software bug here; this ITS will be closed. Use the openldap-software mailing list if you need more help, these questions have been covered many times there and searching the list archives will give you the right answers. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5325) Build error: __lock_getlocker undefined symbol
by hyc＠symas.com 15 Apr '08

15 Apr '08

ando(a)sys-net.it wrote: > As Hallvard pointed out, probably that function was not meant to be > exposed, since it's not declared in Berkeley's headers, but > unfortunately OpenLDAP code seems to need it. At some point (soon) we'll have to remove that dependency and go back to using a per-thread read-only transaction for back-bdb's entry cache locking. The __lock_getlocker() function is no longer exported in BDB 4.7. Fortunately we still have the old read-only txn code in CVS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#5468) You can put irrevant thing on LDIF file "add" line
by patrick.ni＠strangeloopnetworks.com 14 Apr '08

14 Apr '08

Full_Name: Patrick Ni Version: 2.3.30-5 OS: Linux version 2.6.18-4-486 (Debian 2.6.18.dfsg.1-12etch2) (dannf(a)debian.org) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (76.77.66.100) Hello, I have a group entry for myself "patrick" in the LDAP database as follows: dn: cn=patrick,ou=Group,dc=strangeloop,dc=net objectClass: posixGroup objectClass: top cn: patrick userPassword:: e1NTSEF9eA== gidNumber: 500 memberUid: patrick Recently, I realized I could add my 2nd openLDAP account "dn:uid=patrick2,ou=Accounts,dc=strangeloop,dc=net" into my group "patrick" using the following LDIF file: ldap1:~/ldap# more group2_patrick.ldif dn: cn=patrick,ou=Group,dc=strangeloop,dc=net changetype:modify add:membership memberUid:patrick2 This should not have succeeded because the "add:membership" line is wrong; It should have been "add:memberUid". thanks -Patrick

1 0

Re: (ITS#4890) Default Values overlay (patch)
by ghenry＠OpenLDAP.org 14 Apr '08

14 Apr '08

>> People interested in the overlay are able to download it from >> the maintained location at: >> >> http://techpubs.spinlocksolutions.com/openldap/ > > Well, in that case you have to guarantee that the URL remains valid over > time. If you put it in contrib/ the OpenLDAP project guarantees (sort > of) it remains available. To the average potential user, I guess it's > the same level of guarantee :) > It could also be listed at: http://www.openldap.org/doc/admin24/appendix-contrib.html as it wasn't in contrib format when I wrote this section. -- Kind Regards, Gavin Henry. OpenLDAP Engineering Team. E ghenry(a)OpenLDAP.org Community developed LDAP software. http://www.openldap.org/project/

1 0

Re: (ITS#5467) aliases broke on hdb/bdb ?
by h.b.furuseth＠usit.uio.no 13 Apr '08

13 Apr '08

Not a bug. A search which dereferences aliases, checks the filter against the _aliased_ entry, not against the _alias_ entry. So, try ldapsearch -a always uid=wilfinge. -- Hallvard

1 0

(ITS#5467) aliases broke on hdb/bdb ?
by cowboy＠linux.vnet.ibm.com 13 Apr '08

13 Apr '08

Full_Name: Richard Nelson Version: 2.4.7-6.1 (Debian) OS: Linux (Debian) URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (66.245.249.50) Here's the alias ldif (modelled after the FAQ 1111 entry) $ ldapsearch -a never uid=wilfing dn: uid=wilfing,ou=Users,dc=cobpli,dc=svl,dc=ibm,dc=com uid: wilfing aliasedObjectName: uid=wilfinge,ou=Users,dc=cobpli,dc=svl,dc=ibm,dc=com objectClass: alias objectClass: extensibleObject And the aliased entry: $ ldapsearch -a never uid=wilfinge dn # wilfinge, Users, cobpli.svl.ibm.com dn: uid=wilfinge,ou=Users,dc=cobpli,dc=svl,dc=ibm,dc=com When I try to actually use the alias, it fails: $ ldapsearch -a always uid=wilfing # base <dc=cobpli,dc=svl,dc=ibm,dc=com> (default) with scope subtree # filter: uid=wilfing # requesting: ALL # # search result search: 4 result: 0 Success The logs seemed to imply that, rather than the expect DN replacement and search, that the slapd-hdb actually scanned the entire subtree (curiously missing the alias entry) and returned no data In case it matters, here's the index entry for uid: index uid pres,eq,sub I renamed the alias from 'wilfing' to 'alias' to make sure it wasn't due to the alias being a substring of the true name - but that didn't help either

1 0

(ITS#5466) slapo-dynlist attribute remapping
by manu＠netbsd.org 12 Apr '08

12 Apr '08

Full_Name: Emmanuel Dreyfus Version: HEAD OS: NetBSD URL: ftp://ftp.openldap.org/incoming/manu-20070412.patch Submission from: (NULL) (213.41.141.172) This patch adds attribute remapping capability to slapo-dynlist. If you configure it that way: overlay dynlist dynlist-attrset groupOfURLs memberURL bar:foo And if you have an object like this in your directory: dn: cn=list,o=home objectClass: groupOfURLs cn: list memberURL: ldap:///o=home?foo?sub?(objectClass=fooClass) Then searching for cn=list,o=home will display the expantion of the LDAP URI, with foo attributes been displayed as bar.

1 0

Re: (ITS#5465) Delta-Syncrepl cookie problems
by quanah＠zimbra.com 12 Apr '08

12 Apr '08

--On Saturday, April 12, 2008 10:14 AM -0700 Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > I realized last night that this is likely due to the race condition issue > between syncprov & syncrepl when check pointing the cookie. The > replica's run the syncprov overlay so that we can see the cookie as it's > a glue DB. They also have a check point line. I'll have the checkpoint > line removed on the replicas. Never mind, I was incorrect. The syncprov-checkpoint line is not present on the replicas, only "overlay syncprov", so this isn't it. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5465) Delta-Syncrepl cookie problems
by quanah＠zimbra.com 12 Apr '08

12 Apr '08

--On Friday, April 11, 2008 9:33 PM +0000 quanah(a)zimbra.com wrote: > So it's clear that the update took on the replica. I restarted slapd, > and it started replicating correctly. So it looks like there's a problem > where internally it thinks it is on a previous cookie to what it has > already committed. I realized last night that this is likely due to the race condition issue between syncprov & syncrepl when check pointing the cookie. The replica's run the syncprov overlay so that we can see the cookie as it's a glue DB. They also have a check point line. I'll have the checkpoint line removed on the replicas. Does this sound correct, as per the note Howard made recently about this issue? --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#4890) Default Values overlay (patch)
by ando＠sys-net.it 12 Apr '08

12 Apr '08

docelic(a)spinlocksolutions.com wrote: > The contrib section does not have usable makefiles (nor the system > to create them). Usually, building a contrib does not require much makefile stuff. > Putting this overlay in contrib, where it would be unmaintained > and practically broken right away from the start (due to missing > build integration) doesn't make any sense. It makes sense: if we integrate it in the mainstream source and it breaks because it's unmaintained, then the distribution is broken > People interested in the overlay are able to download it from > the maintained location at: > > http://techpubs.spinlocksolutions.com/openldap/ Well, in that case you have to guarantee that the URL remains valid over time. If you put it in contrib/ the OpenLDAP project guarantees (sort of) it remains available. To the average potential user, I guess it's the same level of guarantee :) p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5408) back-ldif bugs
by hyc＠symas.com 12 Apr '08

12 Apr '08

h.b.furuseth(a)usit.uio.no wrote: > Here are a bunch of back-ldif bugs some questions. I have code for most > of it, but need advice/discussion on functionality changes. > - When back-ldif uses OS-specific escaping, we can't move a directory > tree between Windows and Unix hosts if some RDN in the tree contains > characters with OS-specific escaping. This is never a priority. The only documented way to move databases is via slapcat/slapadd. I'm perfectly fine with a Windows-specific special case that simply maps all forbidden characters to control-characters, or some other equally obscure approach. (E.g. map all forbidden characters to the same character.) The typical Windows admin will never see the raw filenames anyway. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#5465) Delta-Syncrepl cookie problems
by quanah＠zimbra.com 11 Apr '08

11 Apr '08

Full_Name: Quanah Gibson-Mount Version: 2.3.41 OS: Linux 2.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (24.23.156.219) I have a replica spitting out the following error in its logs (master and replica running OpenLDAP 2.3.41): Apr 11 13:28:03 123-ldap-3 slapd[21846]: syncrepl_message_to_op: rid 100 be_modify uid=xxxxx,ou=people,dc=123,dc=abc,dc=edu (20) It's syncrepl cookie is: contextCSN: 20080410145621Z#000000#00#000000 Looking at the accesslog on the master, I see (for this time period): dn: reqStart=20080410145621.000001Z,cn=accesslog objectClass: auditModify structuralObjectClass: auditModify reqStart: 20080410145621.000001Z reqEnd: 20080410145621.000003Z reqType: modify reqSession: 190486 reqAuthzID: uid=zimbra,cn=admins,cn=zimbra reqDN: uid=xxxxx,ou=people,dc=123,dc=abc,dc=edu reqResult: 0 reqMod: zimbraPasswordLockoutFailureTime:+ 20080410145621Z reqMod: zimbraPasswordLockoutFailureTime:- 20080410145230Z reqMod: entryCSN:= 20080410145621Z#000000#00#000000 reqMod: modifiersName:= uid=zimbra,cn=admins,cn=zimbra reqMod: modifyTimestamp:= 20080410145621Z entryUUID: 06ac9202-9b5a-102c-8816-1596598f2bf0 creatorsName: cn=accesslog createTimestamp: 20080410145621Z entryCSN: 20080410145621Z#000000#00#000000 modifiersName: cn=accesslog modifyTimestamp: 20080410145621Z dn: reqStart=20080410145621.000027Z,cn=accesslog objectClass: auditModify structuralObjectClass: auditModify reqStart: 20080410145621.000027Z reqEnd: 20080410145621.000028Z reqType: modify reqSession: 190494 reqAuthzID: uid=zimbra,cn=admins,cn=zimbra reqDN: uid=yyyy_07,ou=people,dc=123,dc=abc,dc=edu reqResult: 0 reqMod: zimbraPasswordLockoutFailureTime:+ 20080410145621Z reqMod: entryCSN:= 20080410145621Z#000001#00#000000 reqMod: modifiersName:= uid=zimbra,cn=admins,cn=zimbra reqMod: modifyTimestamp:= 20080410145621Z entryUUID: 06ee59c6-9b5a-102c-8817-1596598f2bf0 creatorsName: cn=accesslog createTimestamp: 20080410145621Z entryCSN: 20080410145621Z#000001#00#000000 modifiersName: cn=accesslog modifyTimestamp: 20080410145621Z So as I understand (delta)syncrepl, the replica knows it already received the update for uid=xxxxx, and yet it is still trying to process that update, rather than the next one for uid=yyyy_07. The entry on the replica has: dn: uid=xxxxx,ou=people,dc=123,dc=abc,dc=edu zimbraPasswordLockoutFailureTime: 20080410145515Z zimbraPasswordLockoutFailureTime: 20080410145516Z zimbraPasswordLockoutFailureTime: 20080410145519Z zimbraPasswordLockoutFailureTime: 20080410145539Z zimbraPasswordLockoutFailureTime: 20080410145547Z zimbraPasswordLockoutFailureTime: 20080410145548Z zimbraPasswordLockoutFailureTime: 20080410145621Z So it's clear that the update took on the replica. I restarted slapd, and it started replicating correctly. So it looks like there's a problem where internally it thinks it is on a previous cookie to what it has already committed. --Quanah

1 0

Re: (ITS#4890) Default Values overlay (patch)
by docelic＠spinlocksolutions.com 11 Apr '08

11 Apr '08

The contrib section does not have usable makefiles (nor the system to create them). Putting this overlay in contrib, where it would be unmaintained and practically broken right away from the start (due to missing build integration) doesn't make any sense. People interested in the overlay are able to download it from the maintained location at: http://techpubs.spinlocksolutions.com/openldap/ Davor Ocelic Spinlock Solutions http://www.spinlocksolutions.com/

1 0

Re: (ITS#5463) syncrepl stops or pauses
by marian.eichholz＠freenet.ag 11 Apr '08

11 Apr '08

On Fri, Apr 11, 2008 at 11:34:33AM +0200, Rein Tollevik wrote: > This sounds a lot like the problems we had, see ITS#5454. Replication > has been running smootly since we implemented the patch included there. OK, we switched to the CVS version (openldap-20080411). The patch is rejected, as You might guess :-) The problem persists more or less, anyway. FYI: Whenever the replication stops, in the provider-log (sync, conns) there is a line: Apr 11 15:30:12 1 slapd[8994]: ber_flush2 failed errno=11 reason="Resource temporarily unavailable" I don't know if this is really meaningful. Just for those people not too familiar with CVS: How can I retrieve that very branch, where the hunk in ITS 5454 would succeed? Thank You very much in advance, anyway, and the help in the past! -- Mit freundlichen Gruessen / Yours sincerely Marian Eichholz Postmaster

1 0

Re: (ITS#5463) syncrepl stops or pauses
by hyc＠symas.com 11 Apr '08

11 Apr '08

rein(a)basefarm.no wrote: > marian.eichholz(a)freenet.ag skrev: >> Full_Name: >> Version: 2.4.8 >> OS: Linux 2.6.23.13 >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (194.97.7.65) >> >> The replication may suddenly stop, and we see no reason for that. Sometimes it >> recovers by itself, sometimes it recovers only when the consumer slapd is >> stopped and restarted. >> Is there a "golden path" to debug this issue? > > This sounds a lot like the problems we had, see ITS#5454. Replication > has been running smootly since we implemented the patch included there. > > An alternative version is implemented in the CVS head release, but that > lacks a change from trylock to real mutex lock before the race condition > is fully closed. > > We experienced a lot of replication problems after upgrading from a > slurpd based replication in 2.3.x to syncrepl in 2.4.8. We are > currently using the CVS head version (as it was about a week ago), and > most of our problems has been solved there :-) You should consider the > cvs version, as it has many syncrepl and syncprov fixes. At this point it would be better to test RE24, not HEAD. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5454) syncrepl refreshAndPersist stops receiving
by hyc＠symas.com 11 Apr '08

11 Apr '08

Hallvard B Furuseth wrote: > hyc(a)symas.com writes: >> OK, reverted the trylock (which was put there in rev 1.376). I >> unfortunately don't remember what prompted that commit; certainly >> there was no ITS associated with it. > > It was committed 9 minutes before this message: > http://www.openldap.org/lists/openldap-devel/200802/msg00032.html > Thanks for that reference. test050 is fine right now; I think rev 1.381 was the more correct fix for that problem. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5454) syncrepl refreshAndPersist stops receiving
by h.b.furuseth＠usit.uio.no 11 Apr '08

11 Apr '08

hyc(a)symas.com writes: > OK, reverted the trylock (which was put there in rev 1.376). I > unfortunately don't remember what prompted that commit; certainly > there was no ITS associated with it. It was committed 9 minutes before this message: http://www.openldap.org/lists/openldap-devel/200802/msg00032.html -- Hallvard

1 0

Re: (ITS#5454) syncrepl refreshAndPersist stops receiving
by hyc＠symas.com 11 Apr '08

11 Apr '08

hyc(a)symas.com wrote: >>> It also occurs to me that we probably don't even need to manipulate the >>> slapd runqueue in persist mode, when si->si_conn is already set. I.e., >>> in that case we can only have gotten here because of a listener event, >>> and not because of a runqueue schedule. >> That is probably true. > > Right, my patch avoids that now. Having looked at it again I think replacing > the trylock with a regular lock is fine. OK, reverted the trylock (which was put there in rev 1.376). I unfortunately don't remember what prompted that commit; certainly there was no ITS associated with it. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5463) syncrepl stops or pauses
by rein＠basefarm.no 11 Apr '08

11 Apr '08

marian.eichholz(a)freenet.ag skrev: > Full_Name: > Version: 2.4.8 > OS: Linux 2.6.23.13 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (194.97.7.65) > > The replication may suddenly stop, and we see no reason for that. Sometimes it > recovers by itself, sometimes it recovers only when the consumer slapd is > stopped and restarted. > Is there a "golden path" to debug this issue? This sounds a lot like the problems we had, see ITS#5454. Replication has been running smootly since we implemented the patch included there. An alternative version is implemented in the CVS head release, but that lacks a change from trylock to real mutex lock before the race condition is fully closed. We experienced a lot of replication problems after upgrading from a slurpd based replication in 2.3.x to syncrepl in 2.4.8. We are currently using the CVS head version (as it was about a week ago), and most of our problems has been solved there :-) You should consider the cvs version, as it has many syncrepl and syncprov fixes. -- Rein Tollevik Basefarm AS

1 0

(ITS#5464) ucred guarded by __USE_GNU in glibc 2.8
by rhafer＠suse.de 11 Apr '08

11 Apr '08

Full_Name: Ralf Haferkamp Version: Any OS: Linux (glibc 2.8) URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (85.8.71.17) Newer glibc Version guards the definition of struct ucred (e.g. used in liblutil/getpeereid.c) by __USE_GNU, hence it is only available when _GNU_SOURCE is defined. The matching glibc changelog only say "Cleanup namespace.". Here's the diff: http://sourceware.org/cgi-bin/cvsweb.cgi/libc/sysdeps/unix/sysv/linux/bits/…

1 0

(ITS#5463) syncrepl stops or pauses
by marian.eichholz＠freenet.ag 11 Apr '08

11 Apr '08

Full_Name: Version: 2.4.8 OS: Linux 2.6.23.13 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (194.97.7.65) Were still migrating von slurpd to syncrepl. We have a 2.3-slurpd-fed provider under OpenLDAP 2.4.8 used by currently two OpenLDAP 2.4.8 consumers with "refreshAndPersist" and a retry-interval of "180 +" (see below) The database is fairly large (8 Mio objects, 15 GB BDB files) and the consumers have a syncrepled database (pulled once, then cloned). There are ongoing updates "each second". We see two issues: When a consumer (re-) connects, the replication for the running consumer pauses until it continues on both nodes. This could become a problem, where a single consumer can be a SPOF for the whole cluster. The replication may suddenly stop, and we see no reason for that. Sometimes it recovers by itself, sometimes it recovers only when the consumer slapd is stopped and restarted. Is there a "golden path" to debug this issue? For the logs: We have "sync" as log level. Consumer 1 (rid 000) [...] Apr 10 16:15:18 0 slapd[27548]: connection_input: conn=192 deferring operation: binding [...] Apr 11 08:45:21 0 slapd[27548]: connection_input: conn=5182 deferring operation: binding Apr 11 08:50:11 0 slapd[27548]: connection_input: conn=5207 deferring operation: binding Apr 11 08:51:09 0 slapd[27548]: connection_input: conn=5212 deferring operation: binding Apr 11 08:53:05 0 slapd[27548]: connection_input: conn=5222 deferring operation: binding Apr 11 08:54:03 0 slapd[27548]: connection_input: conn=5228 deferring operation: binding Consumer 2 (rid 002): [...] Apr 11 02:00:16 2 slapd[22272]: connection_input: conn=3151 deferring operation: binding [...] Apr 11 08:52:09 2 slapd[22272]: connection_input: conn=5224 deferring operation: binding Apr 11 08:53:07 2 slapd[22272]: connection_input: conn=5229 deferring operation: binding Provider (filtered for last syncprov_sendresp entries per rid) Apr 10 16:06:43 1 slapd[27917]: syncprov_sendresp: cookie=rid=000,csn=2008041014 0643.069039Z#000000#000#000000 Apr 11 01:57:58 1 slapd[27917]: syncprov_sendresp: cookie=rid=002,csn=2008041023 5758.696324Z#000000#000#000000 I see no indicators for failing operations, missing resources or any other trouble, incept for the "connection_input" entries. Any idea how to get sync replication running smoothly and reliably? Thank You very much in advance, of course! OK, here are configuraiton snippets for provider and consumer, if it helps. To be honest: I am not quite shure about a reasonable size for syncprov-sessionlog, even after reading the slapo-syncprov. The actual number is meant as "big, but finite". Consumer: --------- syncrepl rid=2 provider="ldap://provider:389" type=refreshAndPersist bindmethod=simple binddn=... credentials=... searchbase=... retry="180 +" provider: --------- database bdb dbnosync cachesize 200000 suffix ... rootdn ... updatedn ... rootpw ... directory /var/ldap/db lastmod on index objectClass eq ... index entryCSN,entryUUID eq checkpoint 512 15 overlay syncprov syncprov-sessionlog 300000 database monitor

1 0

(ITS#5462) Allow setting of RANDOM_FILE for gnutls
by pogma＠thewrittenword.com 11 Apr '08

11 Apr '08

Full_Name: Peter O'Gorman Version: 2.4.8 OS: multiple URL: ftp://ftp.openldap.org/incoming/the-written-word-080410.patch Submission from: (NULL) (24.76.165.223) gcrypt-1.4.x and later allow the egd socket path to be set. This patch allows the conf file option to work when building openldap with gnutls on systems that do not have a /dev/random.

1 0

Re: (ITS#5451) syncprov_checkpoint deadlock bugfix?
by rein＠tollevik.no 10 Apr '08

10 Apr '08

On Wed, 9 Apr 2008, h.b.furuseth(a)usit.uio.no wrote: > Does this help? From my fiddling with ITS#5340 (REP_ENTRY_MODIFIABLE). > I do not understand syncprov's handling of REP_ENTRY_MUSTRELEASE though. > (For one thing it seems to assume that REP_ENTRY_MUSTRELEASE is set if > and only if rs.sr_entry->e_private != NULL. Which is possibly true with > back-bdb but seems a shaky assumption in general.) Almost, except that it triggers an abort.. Using be_entry_release_r() was not correct, as it ended up calling entry_free(). It works if I changes it to use overlay_entry_release_ov() instead, as in the alternative patch at the end. I have put a test script that shows this deadlock on: ftp://ftp.openldap.org/incoming/test053-syncprov-glue It deadlocks without this patch, it and the rest of the test suite succeed with it :-) I haven't tested it in production though.. Rein Index: OpenLDAP/servers/slapd/overlays/syncprov.c diff -u OpenLDAP/servers/slapd/overlays/syncprov.c:1.1.1.14 OpenLDAP/servers/slapd/overlays/syncprov.c:1.14 --- OpenLDAP/servers/slapd/overlays/syncprov.c:1.1.1.14 Thu Apr 10 17:31:24 2008 +++ OpenLDAP/servers/slapd/overlays/syncprov.c Thu Apr 10 17:34:36 2008 @@ -2389,7 +2396,14 @@ if ( !ap ) { if ( !(rs->sr_flags & REP_ENTRY_MODIFIABLE) ) { - rs->sr_entry = entry_dup( rs->sr_entry ); + Entry *new = entry_dup( rs->sr_entry ); + if ( rs->sr_flags & REP_ENTRY_MUSTRELEASE ) { + overlay_entry_release_ov( op, rs->sr_entry, 0, (slap_overinst *)op->o_bd->bd_info ); + rs->sr_flags ^= REP_ENTRY_MUSTRELEASE; + } else if ( rs->sr_flags & REP_ENTRY_MUSTBEFREED ) { + entry_free( rs->sr_entry ); + } + rs->sr_entry = new; rs->sr_flags |= REP_ENTRY_MODIFIABLE|REP_ENTRY_MUSTBEFREED; a = attr_find( rs->sr_entry->e_attrs,

1 0

Re: (ITS#5340) REP_ENTRY_MODIFIABLE bug in dynlist
by h.b.furuseth＠usit.uio.no 10 Apr '08

10 Apr '08

Howard Chu writes: > I have the feeling that MUSTBEFREED and MUSTRELEASE should be > consolidated into a single flag. be_entry_release_r() should simply > call entry_free() if e->e_private is NULL. Maybe. But consider translucent - it grabs incoming entries and remembers them. It can "steal" a MUSTBEFREED entry (though it doesn't now), but must duplicate a MUSTRELEASE entry so that it won't hold on to a lock in an underlying database. -- Hallvard

1 0

Re: (ITS#5451) syncprov_checkpoint deadlock bugfix?
by h.b.furuseth＠usit.uio.no 10 Apr '08

10 Apr '08

Howard Chu writes: > h.b.furuseth(a)usit.uio.no wrote: >> Does this help? From my fiddling with ITS#5340 (REP_ENTRY_MODIFIABLE). >> I do not understand syncprov's handling of REP_ENTRY_MUSTRELEASE though. >> (For one thing it seems to assume that REP_ENTRY_MUSTRELEASE is set if >> and only if rs.sr_entry->e_private != NULL. Which is possibly true with >> back-bdb but seems a shaky assumption in general.) > > That is a requirement, yes. If you implement a backend that has > cache/locking constraints on its entries and you don't provide a > pointer from the entry back to your cache state, then you're an idiot. OTOH you can send an entry without setting MUSTRELEASE, and release it after sending instead. In particular if you didn't about the undocumented sr_flags and their purpose. Or if you don't use be_release for unlocking but just to clean up e_private. ... have I gone blind? I don't see back-bdb setting that flag. Did I just read ITS#3671 and trust it to have updated bdb? -- Hallvard

1 0

Re: (ITS#5459) threw assertion in attr_dup
by richton＠nbcs.rutgers.edu 10 Apr '08

10 Apr '08

> Is this entry still in your database now? Clearly this condition should never > happen - if zero-length values are valid for the syntax, then it should have > been preserved in the (normalized) a_nvals. If not, then it should not have > been in the a_vals in the first place. No, it got deleted last night. But from the core I see: at_syntax_oid = 0x100495d80 "1.3.6.1.4.1.1466.115.121.1.26" IA5String, so empty string is valid. (And makes no logical sense in this situation, but that's for another day...)

1 0

Re: (ITS#5340) REP_ENTRY_MODIFIABLE bug in dynlist
by hyc＠symas.com 10 Apr '08

10 Apr '08

h.b.furuseth(a)usit.uio.no wrote: > - REP_ENTRY_MUSTRELEASE and REP_ENTRY_MUSTBEFREED: > > Callers that set these flags when sending an entry, *allow* the > callee to release/free rs->sr_entry. > Callers cannot depend on it being release/freed, but some do anyway. > > It should simplify things to make the send functions always obey > this flag, but I'm not sure how much cleanup that requires. > There is partial code for that in slap_send_search_entry(), along > with a FIXME note. > > Free MUSTBEFREED-entries with entry_free(), and MUSTRELEASE-entries > with be_entry_release_r(). There is no "use be_entry_release_w()" > flag, but no OpenLDAP code sends write-locked entries anyway. I have the feeling that MUSTBEFREED and MUSTRELEASE should be consolidated into a single flag. be_entry_release_r() should simply call entry_free() if e->e_private is NULL. > + REP_ENTRY_MUSTRELEASE: > > Created for ITS#3671 to ensure that network hangs don't delay > cache entry unlocks: > > Modules should release their cache entry locks before sending > anything. They can't do that with cached entries they send in > rs->sr_entry, so instead they should set REP_ENTRY_MUSTRELEASE. > Conversely, anything which does network I/O (and other potentially > slooow ops I guess) should first obey this flag. > > Several overlays don't obey, nor some other code for sending an > entry when it decides to send an error response instead. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5451) syncprov_checkpoint deadlock bugfix?
by hyc＠symas.com 10 Apr '08

10 Apr '08

h.b.furuseth(a)usit.uio.no wrote: > Does this help? From my fiddling with ITS#5340 (REP_ENTRY_MODIFIABLE). > I do not understand syncprov's handling of REP_ENTRY_MUSTRELEASE though. > (For one thing it seems to assume that REP_ENTRY_MUSTRELEASE is set if > and only if rs.sr_entry->e_private != NULL. Which is possibly true with > back-bdb but seems a shaky assumption in general.) That is a requirement, yes. If you implement a backend that has cache/locking constraints on its entries and you don't provide a pointer from the entry back to your cache state, then you're an idiot. That means every time you want to manipulate the cache state for that entry you'll have to do an expensive search for it, instead of simply dereferencing e_private. If you maintain private state for the entry, you point to it with e_private. If e_private is not set, then there is no private state. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5459) threw assertion in attr_dup
by hyc＠symas.com 10 Apr '08

10 Apr '08

Aaron Richton wrote: > Ergh, didn't realize there was more past that. > > (dbx) print a->a_vals[0] > a->a_vals[0] = { > bv_len = 28U > bv_val = 0x11925f250 "fixed-address 172.16.143.152" > } > (dbx) print a->a_vals[1] > a->a_vals[1] = { > bv_len = 0 > bv_val = 0x117e9a810 "" > } > (dbx) print a->a_vals[2] > a->a_vals[2] = { > bv_len = 0 > bv_val = (nil) > } > (dbx) print a->a_nvals[0] > a->a_nvals[0] = { > bv_len = 28U > bv_val = 0x11a48ff90 "fixed-address 172.16.143.152" > } > (dbx) print a->a_nvals[1] > a->a_nvals[1] = { > bv_len = 0 > bv_val = (nil) > } > Is this entry still in your database now? Clearly this condition should never happen - if zero-length values are valid for the syntax, then it should have been preserved in the (normalized) a_nvals. If not, then it should not have been in the a_vals in the first place. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5459) threw assertion in attr_dup
by richton＠nbcs.rutgers.edu 10 Apr '08

10 Apr '08

Ergh, didn't realize there was more past that. (dbx) print a->a_vals[0] a->a_vals[0] = { bv_len = 28U bv_val = 0x11925f250 "fixed-address 172.16.143.152" } (dbx) print a->a_vals[1] a->a_vals[1] = { bv_len = 0 bv_val = 0x117e9a810 "" } (dbx) print a->a_vals[2] a->a_vals[2] = { bv_len = 0 bv_val = (nil) } (dbx) print a->a_nvals[0] a->a_nvals[0] = { bv_len = 28U bv_val = 0x11a48ff90 "fixed-address 172.16.143.152" } (dbx) print a->a_nvals[1] a->a_nvals[1] = { bv_len = 0 bv_val = (nil) }

1 0

Re: (ITS#5459) threw assertion in attr_dup
by hyc＠symas.com 10 Apr '08

10 Apr '08

Aaron Richton wrote: >> Definitely an oddball. Can you print i, j, and a->a_vals[0] thru >> a->a_vals[i], >> plus a->a_nvals[0] thru a->a_nvals[j]. > > Just got back to my desk and started looking at this. So when the two > slaves that threw this were restarted, they went into an infinite loop > trying to replicate the entry that killed them: > > <= entry_decode: slap_str2undef_ad(^Zethernet 01:aa:aa:aa:aa:a1): AttributeDescription contains inappropriate characters > null_callback: error code 0x50 > syncrepl_entry: rid 110 be_add (80) > syncrepl_entry: rid 110 be_add failed (80) > do_syncrepl: rid 110 retrying > > and retrying, and retrying, and retrying...hence my thought that > the database was hosed...UNTIL... > > Apparently the entry was DEL'd a few hours later. I've now got perfect > sync from the master to the assert'd slaves. The entry_decode error definitely implies a corruption somewhere. It was looking at a 26 byte value as if it was the name of an attribute. I wonder if slapcat will run cleanly on it now, or if it will also encounter some decode errors. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5457) bdb_add() does not always release entries
by hyc＠symas.com 10 Apr '08

10 Apr '08

Hallvard B Furuseth wrote: > hyc(a)symas.com writes: >>> bdb_unlocked_cache_return_entry_r(&bdb->bi_cache, p ); >>> (...) >> Releasing entries is somewhat irrelevant, since all locks are released when >> the transaction commits. Note that this function is a no-op in proto-bdb.h. >> All of those statements are just relics from the first entry cache design, >> before we switched to using BDB locks for everything. > > OK. Does that apply to bdb_cache_return_entry_r( bdb, oe,&lock ); > after > bdb_cache_add( bdb, ei, op->ora_e,&nrdn, rlocker,&lock ); > failed TXN_COMMIT( ltid, 0 ); > too? bdb_cache_return_entry_r() is not a no-op. No, that call obviously matters. But I'm not sure that this commit can ever actually fail, or what to do about it if it does. Since all of these write ops use a nested transaction, and there is nothing outstanding in the outer transaction, and the inner transaction has already committed successfully, the only thing that could cause the outer commit to fail would be something like running out of disk for the transaction log. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5459) threw assertion in attr_dup
by richton＠nbcs.rutgers.edu 10 Apr '08

10 Apr '08

> Definitely an oddball. Can you print i, j, and a->a_vals[0] thru > a->a_vals[i], > plus a->a_nvals[0] thru a->a_nvals[j]. Just got back to my desk and started looking at this. So when the two slaves that threw this were restarted, they went into an infinite loop trying to replicate the entry that killed them: <= entry_decode: slap_str2undef_ad(^Zethernet 01:aa:aa:aa:aa:a1): AttributeDescription contains inappropriate characters null_callback: error code 0x50 syncrepl_entry: rid 110 be_add (80) syncrepl_entry: rid 110 be_add failed (80) do_syncrepl: rid 110 retrying and retrying, and retrying, and retrying...hence my thought that the database was hosed...UNTIL... Apparently the entry was DEL'd a few hours later. I've now got perfect sync from the master to the assert'd slaves. There's nothing secret in this particular backend; it's DHCP backing store, so I can give you full values. Both slaves died with an identical: i = 2 j = 1 *a->a_vals = { bv_len = 28U bv_val = 0x11925f250 "fixed-address 172.16.143.152" } *a->a_nvals = { bv_len = 28U bv_val = 0x11a48ff90 "fixed-address 172.16.143.152" }

1 0

Re: (ITS#5459) threw assertion in attr_dup
by hyc＠symas.com 10 Apr '08

10 Apr '08

richton(a)nbcs.rutgers.edu wrote: > Full_Name: Aaron Richton > Version: 2.3.41 > OS: Solaris 9 > URL: > Submission from: (NULL) (68.192.238.168) > Crazy part is that I hit this on two slaves during the same replication cycle > ... yet four slaves DIDN'T crash during said replication set. I think that > database is hosed on those two slaves; I'll check more thoroughly tomorrow (I'm > off site today). Provider is 2.3.39, so perhaps "it fed 2.3.41 garbage" -- but > still, 2.3.41 should be able to gracefully handle it? Or maybe an assert() is a > graceful handler, but I don't know about that, it kills my slave in the process? Definitely an oddball. Can you print i, j, and a->a_vals[0] thru a->a_vals[i], plus a->a_nvals[0] thru a->a_nvals[j]. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#5461) rs->sr_tag/sr_msgid uninitialized at abandon
by h.b.furuseth＠usit.uio.no 10 Apr '08

10 Apr '08

Full_Name: Hallvard B Furuseth Version: HEAD, RE23, RE24 OS: URL: Submission from: (NULL) (129.240.6.233) Submitted by: hallvard slap_send_ldap_result() does not initialize rs->sr_tag and rs->sr_msgid if rs->sr_err == SLAPD_ABANDON || op->o_abandon. It does call send_ldap_response() which calls slap_cleanup_play(). Some cleanup handlers dispatch on rs->sr_tag. If this is deliberate, it's a design bug: Any cleanup callbacks that depend on this suffer at least a race condition, since o_abandon could get set just after slap_send_ldap_result checks for abandon. The code is from slapd/result.c rev 1.252. The fix should be to first check that the cleanup callbacks handle abandon properly, then always initialize these SlapResponse fields.

1 0

Re: (ITS#5458) whitespace parsing in schema files
by mf＠14v.de 10 Apr '08

10 Apr '08

--On Donnerstag, April 10, 2008 11:29:13 +0200 Hallvard B Furuseth <h.b.furuseth(a)usit.uio.no> wrote: > Not a bug. The 'attributetype' format is contained inside the > slapd.conf format. The slapd.conf format is that each directive is > written in one conceptual line: one line plus any additional slapd.conf > continuation lines - i.e. lines which start with whitespace. Ok, thanks for replying, bug can be closed (if it isn't already). That was my fault, I had simply assumed from looking at the other schema files that schema definitions were in some context-free grammar like C or Pascal. The log message containing input from an earlier schema include is still somewhat confusing. Best regards Matthias Ferdinand

1 0

Re: (ITS#5458) whitespace parsing in schema files
by h.b.furuseth＠usit.uio.no 10 Apr '08

10 Apr '08

mf(a)14v.de writes: > attributetype ( 1.3.6.1.4.1.22558.2.1.1 > NAME 'o4vdescription' > DESC 'RFC2256: descriptive information' > EQUALITY caseIgnoreMatch > SUBSTR caseIgnoreSubstringsMatch > SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{1024} > ) > > If I leave out the blank preceding the closing paranthesis, so that > there is just the newline between the "{1024}" and the ")", slapd > refuses to start Not a bug. The 'attributetype' format is contained inside the slapd.conf format. The slapd.conf format is that each directive is written in one conceptual line: one line plus any additional slapd.conf continuation lines - i.e. lines which start with whitespace. Though if the documentation is unclear on this point, feel free to suggest how to improve it. -- Hallvard

1 0

(ITS#5460) unresolved symbol
by st-wong＠cuhk.edu.hk 10 Apr '08

10 Apr '08

Full_Name: ST Wong Version: 2.4.8 OS: CentOS 5.1 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (137.189.29.184) Got error when compiling Openldap 2.4.8 with BerkeleyDB 4.6: ---------------------- cut here ----------------------- ... checking for db.h... yes checking for Berkeley DB major version... 4 checking for Berkeley DB minor version... 3 checking for Berkeley DB link (-ldb-4.3)... (cached) no checking for Berkeley DB link (-ldb43)... no checking for Berkeley DB link (-ldb-43)... no checking for Berkeley DB link (-ldb-4-3)... no checking for Berkeley DB link (-ldb-4)... no checking for Berkeley DB link (-ldb4)... no checking for Berkeley DB link (-ldb)... yes checking for Berkeley DB version match... yes checking for Berkeley DB thread support... yes checking Berkeley DB version for BDB/HDB backends... no configure: error: BDB/HDB: BerkeleyDB version incompatible ---------------------- cut here ----------------------- In config.log: ---------------------- cut here ----------------------- configure:17625: checking for resolver link (default) configure:17691: cc -o conftest -g -O2 -I/usr/local/BerkeleyDB.4.6/include -L/usr/local/BerkeleyDB.4.6/lib conftest.c >&5 /tmp/ccczbyKW.o: In function `main': /usr/local/src/openldap-2.4.8/conftest.c:113: undefined reference to `__res_query' /usr/local/src/openldap-2.4.8/conftest.c:132: undefined reference to `__dn_expand' collect2: ld returned 1 exit status ---------------------- cut here ----------------------- Have to add "-lresolv" to LDFLAGS in order to resolve the problem. I suppose configure will add all libraries required. Thanks a lot.

1 0

(ITS#5459) threw assertion in attr_dup
by richton＠nbcs.rutgers.edu 10 Apr '08

10 Apr '08

Full_Name: Aaron Richton Version: 2.3.41 OS: Solaris 9 URL: Submission from: (NULL) (68.192.238.168) (dbx) where current thread: t@18 [1] __lwp_kill(0x0, 0x6, 0xffffffffffffffe6, 0x0, 0x0, 0x0), at 0xffffffff7f0a8d4c [2] raise(0x6, 0x0, 0xffffffff457fe390, 0x0, 0x0, 0x0), at 0xffffffff7f058dc0 [3] abort(0x30, 0x0, 0x30, 0x7efefeff, 0x81010100, 0xff0000), at 0xffffffff7f03e688 [4] __assert(0x10023d488, 0x10023d490, 0x8d, 0x1, 0xffffffff457fe758, 0x11b369c59), at 0xffffffff7f03e98c =>[5] attr_dup(a = 0x1174b0470), line 141 in "attr.c" [6] attrs_dup(a = 0x1174b0470), line 166 in "attr.c" [7] hdb_modify_internal(op = 0xffffffff457ff618, tid = 0x11b369b70, modlist = 0x118ea8bd0, e = 0xffffffff457feab8, text = 0xffffffff457fefd8, textbuf = 0xffffffff457feb4c "^X\xea\x8b\xd0", textlen = 256U), line 55 in "modify.c" [8] hdb_modify(op = 0xffffffff457ff618, rs = 0xffffffff457fefb8), line 480 in "modify.c" [9] syncrepl_entry(si = 0x1004b5050, op = 0xffffffff457ff618, entry = 0x118d17870, modlist = 0xffffffff457ff328, syncstate = 2, syncUUID = 0xffffffff457ff3c0, syncCookie_req = 0xffffffff457ff360, syncCSN = 0xffffffff457ff390), line 1987 in "syncrepl.c" [10] do_syncrep2(op = 0xffffffff457ff618, si = 0x1004b5050), line 735 in "syncrepl.c" [11] do_syncrepl(ctx = 0xffffffff457ffc30, arg = 0x1004b5230), line 1095 in "syncrepl.c" [12] ldap_int_thread_pool_wrapper(xpool = 0x10041e700), line 478 in "tpool.c" (dbx) list 130,150 130 131 if ( a->a_nvals != a->a_vals ) { 132 int j; 133 134 tmp->a_nvals = ch_malloc( (i + 1) * sizeof(struct berval) ); 135 for ( j = 0; !BER_BVISNULL( &a->a_nvals[j] ); j++ ) { 136 assert( j < i ); 137 ber_dupbv( &tmp->a_nvals[j], &a->a_nvals[j] ); 138 if ( BER_BVISNULL( &tmp->a_nvals[j] ) ) break; 139 /* FIXME: error? */ 140 } 141 assert( j == i ); 142 BER_BVZERO( &tmp->a_nvals[j] ); 143 144 } else { 145 tmp->a_nvals = tmp->a_vals; 146 } 147 148 } else { 149 tmp->a_vals = NULL; 150 tmp->a_nvals = NULL; Crazy part is that I hit this on two slaves during the same replication cycle ... yet four slaves DIDN'T crash during said replication set. I think that database is hosed on those two slaves; I'll check more thoroughly tomorrow (I'm off site today). Provider is 2.3.39, so perhaps "it fed 2.3.41 garbage" -- but still, 2.3.41 should be able to gracefully handle it? Or maybe an assert() is a graceful handler, but I don't know about that, it kills my slave in the process?

1 0

(ITS#5458)
by mf＠14v.de 10 Apr '08

10 Apr '08

Looks like newline does not count as whitespace. Regards Matthias

1 0

(ITS#5458) whitespace parsing in schema files
by mf＠14v.de 10 Apr '08

10 Apr '08

Full_Name: Matthias Ferdinand Version: 2.4.8 OS: Linux/SuSE10.0 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (194.102.180.118) Platform: SuSE Linux 10.0 OpenLDAP 2.4.8 from pkgsrc (http://www.netbsd.org/docs/pkgsrc/) gcc-4.2.1 (also with OpenLDAP 2.4.7 from Debian Testing) I tried the following simple attribute definition (located at /opt/pkgsrc/etc/openldap/schema/test2.schema): attributetype ( 1.3.6.1.4.1.22558.2.1.1 NAME 'o4vdescription' DESC 'RFC2256: descriptive information' EQUALITY caseIgnoreMatch SUBSTR caseIgnoreSubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{1024} ) If I leave out the blank preceding the closing paranthesis, so that there is just the newline between the "{1024}" and the ")", slapd refuses to start with some syntax error: AttributeTypeDescription = "(" whsp numericoid whsp ; AttributeType identifier [ "NAME" qdescrs ] ; name used in AttributeType ... and at loglevel 64 some strange syslog entries: Apr 10 00:45:08 vestatest slapd[25297]: line 6 (attributetype ( 1.3.6.1.4.1.22558.2.1.1 NAME 'o4vdescription' DESC 'RFC2256: descriptive information' EQUALITY caseIgnoreMatch SUBSTR caseIgnoreSubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{1024}) Apr 10 00:45:08 vestatest slapd[25297]: /opt/pkgsrc/etc/openldap/schema/test2.schema: line 6 attributetype: Missing closing parenthesis before amavisBannedFilesLover $ amavisBypassBannedChecks $ amavisBadHeaderLover $ amavisBypassHeaderChecks $ amavisSpamTagLevel $ amavisSpamTag2Level $ amavisSpamKillLevel $ amavisWhiteli Apr 10 00:45:08 vestatest slapd[25297]: /opt/pkgsrc/etc/openldap/slapd.conf: line 13: <include> handler exited with 1! (the amavis entries are from a schema included just before this one) Looks like the parser reads the attribute definition correctly, but is confused afterwards. Regards Matthias

1 0

Re: (ITS#5340) REP_ENTRY_MODIFIABLE bug in dynlist
by h.b.furuseth＠usit.uio.no 10 Apr '08

10 Apr '08

Below I'll summarize SlapReply.sr_flags as I now understand it after un-confusing myself a little, since this ITS is getting rather long. Mostly about how code which receives a filled SlapReply should use it, which I presume is primarily overlays, slap_callbacks, maybe back-relay. About discussion so far: The sr_flags are inconsistent about when which ones may or must be obeyed. E.g. REP_ENTRY_MUST<RELEASE/BEFREED> does not mean the entry will be released/freed, but REP_<MATCHED/REF> does in some functions. This needs to be normalized. We can state that overlays with a bi_entry_release_rw function must keep track of which module should release which entry, like glue does. At least for now. Never mind my worries about which overlay/backend should release an REP_ENTRY_MUSTRELEASE'd sr_entry. That doesn't matter with current OpenLDAP code. As far as I can see glue is the only overlay with a release function, and glue can dispatch on the entry DN. rs.sr_ref bugs, fixing now: - slap_send_ldap_result() forgets to free rs->sr_ref if it does not call send_ldap_response(). - send_ldap_response() can return with sr_ref set to a freed value. I'll make it return with sr_ref to NULL if REP_REF_MUSTBEFREED. ando(a)sys-net.it writes: > Similarly, the existence of REP_ENTRY_MUSTBEFREED is not totally clear: Looked a bit at cvs logs... Since you are the one who introduced MUSTBEFREED and MODIFIABLE (slap.h 1.503 for rwm) I think we can take it the intent of REP_ENTRY_MUSTBEFREED is indeed unclear/forgotten:-) Anyway, I think I've gotten the flags straight now: * REP_<ENTRY, MATCHED, REF>_MUST<RELEASE, BEFREED>: Overlays/callbacks may modify the rs->sr_<entry, text, ref> pointers. If they do, they must release/free the old value if and only if the corresponding MUST<RELEASE/BEFREED> flag is set. They should (re)set the relevant flags to describe the new value. Many overlays don't, or not thoroughly enough. - REP_<MATCHED, REF>_MUSTBEFREED: Callers setting these can depend on send_ldap_response() to free sr_<matched, ref>. That is, "send response/disconnect" calls. These flags do apparently not apply and should not be set when sending an entry or search continuation reference. Dunno why not REP_REF_MUSTBEFREED when sending continuation references. - REP_ENTRY_MUSTRELEASE and REP_ENTRY_MUSTBEFREED: Callers that set these flags when sending an entry, *allow* the callee to release/free rs->sr_entry. Callers cannot depend on it being release/freed, but some do anyway. It should simplify things to make the send functions always obey this flag, but I'm not sure how much cleanup that requires. There is partial code for that in slap_send_search_entry(), along with a FIXME note. Free MUSTBEFREED-entries with entry_free(), and MUSTRELEASE-entries with be_entry_release_r(). There is no "use be_entry_release_w()" flag, but no OpenLDAP code sends write-locked entries anyway. + REP_ENTRY_MUSTRELEASE: Created for ITS#3671 to ensure that network hangs don't delay cache entry unlocks: Modules should release their cache entry locks before sending anything. They can't do that with cached entries they send in rs->sr_entry, so instead they should set REP_ENTRY_MUSTRELEASE. Conversely, anything which does network I/O (and other potentially slooow ops I guess) should first obey this flag. Several overlays don't obey, nor some other code for sending an entry when it decides to send an error response instead. * REP_ENTRY_MODIFIABLE: Overlays and callbacks may modify the entry which rs->sr_entry points at if this flag is set. If they want to modify an incoming sr_entry when REP_ENTRY_MODIFIABLE is not set, they can use entry_dup(), release or free the old entry if sr_flags says so, put the new entry in sr_entry, and maybe set REP_ENTRY_MODIFIABLE | REP_ENTRY_MUSTBEFREED. Thus code which creates a temporary Entry struct and sends it as rs->sr_entry, should set REP_ENTRY_MODIFIABLE if the entry is of no further interest to it (and it has set no pointers into it, e.g. rs->sr_matched). That allows overlays that wish to modify the entry being sent, to use this entry instead of copying it. * REP_NO_<OPERATIONALS, ENTRYDN, SUBSCHEMA>: Set by the chain overlay to "tell the frontend not to add generated operational attributes". Only backend.c reacts to these flags. Dunno if overlays should reset these flags when replacing sr_entry with a "private" entry (with another DN) rather than a dup of the current one, or something. Some overlays set sr_flags to fresh values while others only update some bits in sr_flags, so the effect is likely somewhat ad-hoc. Leaving that worry to someone else. I've been fiddling with some functions to generalize this, but they seem to need a lot of flags or variants for different modes... Should maybe just commit some draft variants with warnings that they may change, and clean up later. After my changes don't cause slapd to crash, anyway:-) -- Hallvard

1 0

Re: (ITS#5408) back-ldif bugs
by h.b.furuseth＠usit.uio.no 09 Apr '08

09 Apr '08

hyc(a)symas.com writes: > h.b.furuseth(a)usit.uio.no wrote: >> - back-ldif escapes the directory separator as \<hex value>. But >> Windows uses "\" as directory separator, so that doesn't help. It >> needs another escape character. >> >> To avoid double hex-escaping of DNs that contain "\", we can pick >> another escape char, e.g. "^", escape that too, and translate "\" to >> it. Thus DN "cn=x^y\2Cz" gets filename "cn=x^5Ey^2Cz.ldif". > > That sounds fine. Is there any reason not to use %, which is already > used for URL encoding? Good point. The filenames I suggested will not be valid URLs but will resemble them, but that can be fixed by using proper URL-encoding of RDNs. Filenames for RDNs with \ in them get a bit longer though. >> - More characters should likely be hex-escaped: >> + ":" (as in "C:\") and "/" on Windows? I've seen programs use the >> latter as directory separator even on Windows. Others? >> + 8-bit chars in case the OS gets clever about charset handling. >> + Control chars. >> >> I don't use Windows myself though. Nor Mac... > > Again, the rules for URL encoding probably already cover any cases we > should worry about. I don't think so. At least "/" is not "unsafe", if I read it right. >> - RDNs ending with ".ldif" should be escaped. Currently "cn=foo.ldif" >> is the name of both RDN "cn=foo"'s entry file and RDN "cn=foo.ldif"'s >> non-leaf directory. > > Perhaps we should escape '.' Hm, true. "." is (was?) a special character on some filesystems. E.g. a file can only have one of them, or the 2nd is followed by a version number. >> - Sorted-value RDNs: No reply? :-) Ok, sort order can be adjusted separately anyway. One error in my previous attempt: bconfig.c mentions "pseudo-indexed" entries (cn=Include{xx}, cn=Module{xx}) which I presume must be sorted by {xx} too, so back-ldif can't treat only "attr={number}val" as indexed. I guess I'll either check both variants or allow {number} anywhere, depending on convenience. -- Hallvard

1 0

Re: (ITS#5408) back-ldif bugs
by hyc＠symas.com 09 Apr '08

09 Apr '08

h.b.furuseth(a)usit.uio.no wrote: > Full_Name: Hallvard B Furuseth > Version: HEAD, RE23, RE24 > OS: Linux > URL: http://folk.uio.no/hbf/OpenLDAP/back-ldif.c > Submission from: (NULL) (129.240.6.233) > Submitted by: hallvard > > > Here are a bunch of back-ldif bugs some questions. I have code for most > of it, but need advice/discussion on functionality changes. > > I've been sitting on it too long and would keep sitting if I waited > until I'd cleaned up everything, so, posting now instead. I enclose an > URL for a *draft* back-ldif/ldif.c. > > > Functionality changes. > > * Some LDIF files/directories with special chars need to be renamed: > > RE24 or RE25 change? If anyone uses back-ldif as a regular database, > they may need to slapcat/slapadd to upgrade past these changes. > > - back-ldif escapes the directory separator as \<hex value>. But > Windows uses "\" as directory separator, so that doesn't help. It > needs another escape character. > > To avoid double hex-escaping of DNs that contain "\", we can pick > another escape char, e.g. "^", escape that too, and translate "\" to > it. Thus DN "cn=x^y\2Cz" gets filename "cn=x^5Ey^2Cz.ldif". That sounds fine. Is there any reason not to use %, which is already used for URL encoding? > - More characters should likely be hex-escaped: > + ":" (as in "C:\") and "/" on Windows? I've seen programs use the > latter as directory separator even on Windows. Others? > + 8-bit chars in case the OS gets clever about charset handling. > + Control chars. > > I don't use Windows myself though. Nor Mac... Again, the rules for URL encoding probably already cover any cases we should worry about. > > - When back-ldif uses OS-specific escaping, we can't move a directory > tree between Windows and Unix hosts if some RDN in the tree contains > characters with OS-specific escaping. > > Should we special-case both Unix' and Windows' directory separators > on both OSes? Then instead there will be more directory trees which > can't be move between OpenLDAP versions, before and after the > change. Assuming anyone uses back-ldif in the first place:-) Might as well break it all at once. > > - The database suffix must be hex-escaped like the rest of the > filenames. > > - RDNs ending with ".ldif" should be escaped. Currently "cn=foo.ldif" > is the name of both RDN "cn=foo"'s entry file and RDN "cn=foo.ldif"'s > non-leaf directory. Perhaps we should escape '.' > > - Sorted-value RDNs: > > + In dn2path() when IX_FSL != IX_DNL (filename '{' != RDN '{'): > IX_FS[LR] must be hex-escaped. They are treated as equivalent to > '{' - '}', thus different RDNs can map to the same filename. > > The "{1}" in path "/foo/cn=x{,cn={1}y" is not recognized. > The IX_DN[LR] (i.e. '{', '}') to IX_FS[LR] translation is a bit > strange: It translates the first '{' it sees, then the next > '}', then the next '{', etc. Any reason not to just translate > all '{' and '}' chars? If so, we should at least reset at > each filename and '+'. > > + Sorting (in ldif_r_enum_tree()): > > Should "a={1}x" sort before or after "a=x"? Currently it sorts > like "a={". "a=" (normally before) or "a=<CHAR_MAX>" (normally > after) would be better. > > RDN "attr=foo{bar}baz" is treated as a sorted value. Can easily > check for '={'<successful strtol parse'> '}' instead. > "attr={0<octal>}val" and "attr={0x<hex>}val" are recognized as > sorted values, should they be? > > Some of this is from ITS#4627 (back-ldif issues). -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5457) bdb_add() does not always release entries
by h.b.furuseth＠usit.uio.no 09 Apr '08

09 Apr '08

hyc(a)symas.com writes: >> bdb_unlocked_cache_return_entry_r(&bdb->bi_cache, p ); >> (...) > Releasing entries is somewhat irrelevant, since all locks are released when > the transaction commits. Note that this function is a no-op in proto-bdb.h. > All of those statements are just relics from the first entry cache design, > before we switched to using BDB locks for everything. OK. Does that apply to bdb_cache_return_entry_r( bdb, oe, &lock ); after bdb_cache_add( bdb, ei, op->ora_e, &nrdn, rlocker, &lock ); failed TXN_COMMIT( ltid, 0 ); too? bdb_cache_return_entry_r() is not a no-op. -- Hallvard

1 0

Re: (ITS#5451) syncprov_checkpoint deadlock bugfix?
by rein＠tollevik.no 09 Apr '08

09 Apr '08

On Wed, 9 Apr 2008, h.b.furuseth(a)usit.uio.no wrote: > Does this help? From my fiddling with ITS#5340 (REP_ENTRY_MODIFIABLE). > I do not understand syncprov's handling of REP_ENTRY_MUSTRELEASE though. > (For one thing it seems to assume that REP_ENTRY_MUSTRELEASE is set if > and only if rs.sr_entry->e_private != NULL. Which is possibly true with > back-bdb but seems a shaky assumption in general.) I have found out how to reliably reproduce this bug in our master server, and it definitely involves syncprov. I'm now working on reproducing it in a minimalistic test configuration, I'll try your patch there and report back. Rein

1 0

Re: (ITS#5452) seg fault in syncprov_op abandon on master
by hyc＠symas.com 09 Apr '08

09 Apr '08

quanah(a)zimbra.com wrote: > --On Friday, April 04, 2008 1:23 PM +0200 Hallvard B Furuseth > <h.b.furuseth(a)usit.uio.no> wrote: > >> Since it crashes on >> if ( so->s_op->o_connid == op->o_connid&& >> so->s_op->o_msgid == op->orn_msgid ) >> please try >> frame 0 >> print *op >> print *so->s_op >> and maybe >> backtrace full >> while you are at it. So far there don't appear to be any invalid pointers anywhere, so there's no clue why you hit a SEGV here. Can you reproduce this situation using valgrind? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5451) syncprov_checkpoint deadlock bugfix?
by h.b.furuseth＠usit.uio.no 09 Apr '08

09 Apr '08

Does this help? From my fiddling with ITS#5340 (REP_ENTRY_MODIFIABLE). I do not understand syncprov's handling of REP_ENTRY_MUSTRELEASE though. (For one thing it seems to assume that REP_ENTRY_MUSTRELEASE is set if and only if rs.sr_entry->e_private != NULL. Which is possibly true with back-bdb but seems a shaky assumption in general.) Index: syncprov.c =================================================================== RCS file: /repo/OpenLDAP/pkg/ldap/servers/slapd/overlays/syncprov.c,v retrieving revision 1.227 diff -u -2 -r1.227 syncprov.c --- syncprov.c 3 Apr 2008 06:43:48 -0000 1.227 +++ syncprov.c 9 Apr 2008 10:29:40 -0000 @@ -2390,7 +2390,13 @@ if ( !ap ) { if ( !(rs->sr_flags & REP_ENTRY_MODIFIABLE) ) { - rs->sr_entry = entry_dup( rs->sr_entry ); - rs->sr_flags |= - REP_ENTRY_MODIFIABLE|REP_ENTRY_MUSTBEFREED; + Entry *new = entry_dup( rs->sr_entry ); + if ( rs->sr_flags & REP_ENTRY_MUSTRELEASE ) { + be_entry_release_r( op, rs->sr_entry ); + } else if ( rs->sr_flags & REP_ENTRY_MUSTBEFREED ) { + entry_free( rs->sr_entry ); + } + rs->sr_entry = new; + rs->sr_flags = (rs->sr_flags & ~REP_ENTRY_MASK) | + REP_ENTRY_MODIFIABLE | REP_ENTRY_MUSTBEFREED; a = attr_find( rs->sr_entry->e_attrs, slap_schema.si_ad_contextCSN ); -- Hallvard

1 0

Re: (ITS#5457) bdb_add() does not always release entries
by hyc＠symas.com 09 Apr '08

09 Apr '08

h.b.furuseth(a)usit.uio.no wrote: > Full_Name: Hallvard B Furuseth > Version: HEAD, 2.3, 2.4 > OS: > URL: > Submission from: (NULL) (129.240.6.233) > Submitted by: hallvard > > > back-bdb/add.c has several code paths where the parent entry > "p" is not released - it does "goto return_results;" without > first doing > > if ( p&& p !=&slap_entry_root ) > bdb_unlocked_cache_return_entry_r(&bdb->bi_cache, p ); > p = NULL; Releasing entries is somewhat irrelevant, since all locks are released when the transaction commits. Note that this function is a no-op in proto-bdb.h. All of those statements are just relics from the first entry cache design, before we switched to using BDB locks for everything. Probably could stand to do some cleanup. The diffs would be quite drastic though. Again, this may be something that's best deferred until 2.3 is dropped. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#5457) bdb_add() does not always release entries
by h.b.furuseth＠usit.uio.no 09 Apr '08

09 Apr '08

Full_Name: Hallvard B Furuseth Version: HEAD, 2.3, 2.4 OS: URL: Submission from: (NULL) (129.240.6.233) Submitted by: hallvard back-bdb/add.c has several code paths where the parent entry "p" is not released - it does "goto return_results;" without first doing if ( p && p != &slap_entry_root ) bdb_unlocked_cache_return_entry_r( &bdb->bi_cache, p ); p = NULL; The code looks like it expects that to be done below the return_results lablel, since it sets p to NULL before the goto. I don't know how the bdb cache code works though. Also I imagine the 'if ( !bvmatch( &pdn, &p->e_nname ) )' branch should check that p != &slap_entry_root before releasing p. Not sure when the parent needs to be freed. By ITS#3671 entries should be released before doing network I/O - but maybe the parent must be held until the slap_graduate_commit_csn()? I don't know what that does. Is it bdb_cache_add(,,op->ora_e,,,) or the following TXN_COMMIT which makes it necessary to bdb_cache_return_entry_r the entry? HEAD and RE24 only do bdb_cache_return_entry_r( bdb, oe, &lock ); if rs->sr_err == LDAP_SUCCESS at the return_results label, but rs->sr_err can change after the bdb_cache_add(). (A bit hard to read since the entry is referred to as both 'oe', op->ora_e and op->oq_add.rs_e. Not sure what the 'oe' variable is used for anyway.)

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs