Aaron Richton writes: The 2nd contextCSN in that entry is garbage: dn: cn=config objectClass: olcGlobal cn: config structuralObjectClass: olcGlobal creatorsName: cn=config createTimestamp: 20090211053745Z entryUUID: dade8f04-8c49-102d-935b-1f79305f5c76 olcServerID: 1 ldap://localhost:9011/ olcServerID: 2 ldap://localhost:9012/ olcServerID: 3 ldap://localhost:9013/ entryCSN: 20090211053747.167910Z#000000#001#000000 modifiersName: cn=config modifyTimestamp: 20090211053747Z contextCSN: 20090211053750.548808Z#000000#001#000000 contextCSN:: fH/zsAAkIZgAJAtgAEP0QABD8nAAAAAAAAAAAAAAAAAAAAAAAAAAAA== -- Hallvard

On Wed, 11 Feb 2009, Howard Chu wrote: RE24 changed to: pkg/ldap/servers/slapd/syncrepl.c,v 1.444 2009/02/11 21:00:00 pkg/ldap/servers/slapd/overlays/syncprov.c,v 1.257 2009/02/11 21:00:00 from HEAD. No asserts, and it looks like the same issue (although I could just be reading it wrong pre-coffee). This happened once during an overnight infinite loop (so figure maybe once in ~400-500 runs). testrun: https://www.nbcs.rutgers.edu/~richton/test050failed.2009022322.tar.bz2

Aaron Richton wrote: I've run about 600 iterations each on two separate servers all day today, no asserts. Tell me again what your system environment is? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

Aaron Richton wrote: Agreed. But at this point, 2.4.14 is such an improvement over .13 that I'd rather get it out instead of delay it further. We can revisit this for .15 when we get some better traces of the problem. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

Aaron Richton wrote: Added a few more asserts to HEAD syncprov.c, let's see if we can narrow this down any further. There aren't any other places that set si_si_ctxcsn, so if the closing assert triggers but not the earlier ones, then we probably have an overwrite problem. If the other asserts trigger instead, then we're being fed bad data from something further upstream. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

On Sun, 15 Feb 2009, Howard Chu wrote: I'll start a RE24 test050 loop overnight with syncrepl.c.1.444 syncprov.c.1.261 ctxcsn.c.1.53 We know that this is a pretty rare issue; hopefully some others on the list will join in so we can try and get the run count high...

On Sun, 15 Feb 2009, Aaron Richton wrote: Ran fine overnight here.

Dieter Kluenter wrote: Which is pretty squarely inside BerkeleyDB. But nobody else reported any such problem yet. Seems you'll have to check your BDB build. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

Howard Chu <hyc(a)symas.com&gt; writes: This patch had been applied already, as I have mentioned http://www.openldap.org/lists/openldap-devel/200902/msg00031.html still, one out of 5 testruns fails. -Dieter -- Dieter Klünter | Systemberatung http://www.dpunkt.de/buecher/2104.html sip: +49.180.1555.7770535 GPG Key ID:8EF7B6C6 53°08'09,95"N 10°08'02,42"E

Quanah Gibson-Mount <quanah(a)zimbra.com&gt; writes: And how many of those reported success is based on db-4.7 and amd64, with a multifold of iterations on test039? I have done some 200 iterations now, with a success/failure ratio of 5:1 -Dieter -- Dieter Klünter | Systemberatung http://www.dpunkt.de/buecher/2104.html sip: +49.180.1555.7770535 GPG Key ID:8EF7B6C6 53°08'09,95"N 10°08'02,42"E

Dieter Kluenter wrote: The email you reference above only says you have patch 4.7.25.1. As I said, you need patch #3. So, unless you made a mistake in your previous email, I see no evidence that you have the necessary patch. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

Howard Chu <hyc(a)symas.com&gt; writes: mea culpa, I wasn't aware that there were three patches already. -Dieter -- Dieter Klünter | Systemberatung http://www.dpunkt.de/buecher/2104.html sip: +49.180.1555.7770535 GPG Key ID:8EF7B6C6 53°08'09,95"N 10°08'02,42"E

With test050 and defined(HAVE_EPOLL), valgrind sometimes reports invalid reads/writes in slapd_daemon_task(). Maybe that clobbers some memory (like the garbage contextCSN mentioned previously). Turning SLAP_EPOLL_SOCK_SET(s, mode) into a function shows that s == 9 but SLAP_EPOLL_SOCK_IX(s) == -1, so slap_daemon.sd_epolls[] gets a negative index. I don't know where to go from there. Maybe hangup (below) doesn't clean up after itself. static void SLAP_EPOLL_SOCK_SET(int s, long mode) { long ix = SLAP_EPOLL_SOCK_IX(s); assert(ix >= 0); long ev = (slap_daemon.sd_epolls[ix]).events; if ( (ev & (mode)) != (mode) ) { SLAP_EPOLL_SOCK_EV(s) |= (mode); epoll_ctl( slap_daemon.sd_epfd, EPOLL_CTL_MOD, (s), &SLAP_EPOLL_SOCK_EP(s) ); } } (gdb) thread 3 [Switching to thread 3 (process 3915)]#3 0x0000000000419b9c in SLAP_EPOLL_SOCK_SET (s=9, mode=-2147483648) at daemon.c:181 181 assert(ix >= 0); (gdb) backtrace #0 0x000000350e230215 in raise () from /lib64/libc.so.6 #1 0x000000350e231cc0 in abort () from /lib64/libc.so.6 #2 0x000000350e229696 in __assert_fail () from /lib64/libc.so.6 #3 0x0000000000419b9c in SLAP_EPOLL_SOCK_SET (s=9, mode=-2147483648) at daemon.c:181 #4 0x000000000041ef40 in slapd_daemon_task (ptr=0x0) at daemon.c:2557 #5 0x000000350ee06367 in start_thread () from /lib64/libpthread.so.0 (gdb) frame 3 #3 0x0000000000419b9c in SLAP_EPOLL_SOCK_SET (s=9, mode=-2147483648) at daemon.c:181 181 assert(ix >= 0); (gdb) print ix $9 = -1 (gdb) print *slap_daemon.sd_index@128 $10 = {-1, -1, -1, -1, 0, -1, -1, 1, -1 <repeats 120 times>} (gdb) up #4 0x000000000041ef40 in slapd_daemon_task (ptr=0x0) at daemon.c:2557 2557 SLAP_EPOLL_SOCK_SET( fd, EPOLLET ); (gdb) list 2552 Debug( LDAP_DEBUG_CONNS, 2553 "daemon: hangup on %d\n", fd, 0, 0 ); 2554 #ifdef HAVE_EPOLL 2555 /* Don't keep reporting the hangup 2556 */ 2557 SLAP_EPOLL_SOCK_SET( fd, EPOLLET ); 2558 #endif -- Hallvard

Howard Chu writes: OS bug? Race condition? Valgrind bug, since it so far will only happen under Valgrind? Full testrun dir in <http://folk.uio.no/hbf/testrun.1.tbz> from GRIND=valgrind SLAPD_DEBUG=269 ./run test050 (I've done s/exec/exec $GRIND/ on last 'exec' in servers/slapd/slapd). Seems to usually be in slapd.2.log. It ends with: conn=0 op=1 SEARCH RESULT tag=101 err=0 nentries=4 text= daemon: activity on 1 descriptor daemon: activity on: 9r daemon: read active on 9 daemon: epoll: listen=7 active_threads=0 tvp=NULL connection_get(9) connection_get(9): got connid=0 connection_read(9): checking for input on id=0 ber_get_next daemon: activity on 1 descriptor daemon: activity on: daemon: hangup on 9 connection_get(9) ber_get_next: tag 0x30 len 5 contents: ber_get_next ber_get_next on fd 9 failed errno=0 (Success) connection_read(9): input error=-2 id=0, closing. connection_closing: readying conn=0 sd=9 for close connection_close: deferring conn=0 sd=9 conn=0 op=2 do_unbind conn=0 op=2 UNBIND connection_get(9): got connid=0 connection_close: deferring conn=0 sd=9 daemon: epoll: listen=7 active_threads=0 tvp=NULL connection_resched: attempting closing conn=0 sd=9 connection_close: conn=0 sd=9 daemon: removing 9 daemon: activity on 2 descriptors daemon: activity on: conn=0 fd=9 closed daemon: hangup on 9 lt-slapd: daemon.c:181: SLAP_EPOLL_SOCK_SET: Assertion `ix >= 0' failed. $ grep '^conn=' slapd.2.log conn=0 fd=9 ACCEPT from IP=127.0.0.1:46366 (IP=127.0.0.1:9012) conn=0 op=0 BIND dn="cn=config" method=128 conn=0 op=0 BIND dn="cn=config" mech=SIMPLE ssf=0 conn=0 op=0 RESULT tag=97 err=0 text= conn=0 op=1 do_search conn=0 op=1 SRCH base="cn=config" scope=2 deref=0 filter="(objectClass=*)" conn=0 op=1 SRCH attr=* + conn=0 op=1 SEARCH RESULT tag=101 err=0 nentries=4 text= conn=0 op=2 do_unbind conn=0 op=2 UNBIND conn=0 fd=9 closed -- Hallvard

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 All these configurations (and the normal full configuration) run through the tests without error on OS X 10.5.6/Intel. jens