Quanah Gibson-Mount quanah@zimbra.com writes:

Please test RE24 for possible 2.4.16 release.

OpenSUSE-11.1 x86_64 success

-Dieter

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Hi, Quanah,

Quanah Gibson-Mount wrote:

Please test RE24 for possible 2.4.16 release.

Thanks!

Any chance to have ITS 6014 fixed before the release? I can make a local fix in FreeBSD port but it doesn't seem to be good to free() stack content on any operating system...

Cheers, - -- Xin LI delphij@delphij.net http://www.delphij.net/ FreeBSD - The Power to Serve!

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Quanah Gibson-Mount wrote:

--On Wednesday, March 18, 2009 10:42 AM -0700 Xin LI delphij@delphij.net wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Hi, Quanah,

Quanah Gibson-Mount wrote:

...

Please test RE24 for possible 2.4.16 release.

Thanks!

Any chance to have ITS 6014 fixed before the release? I can make a local fix in FreeBSD port but it doesn't seem to be good to free() stack content on any operating system...

I suggest you check the status of the ITS. It is already noted to be fixed in RE24.

Oops, sorry for that, will checkout a fresh copy and test, thanks!

Cheers, - -- Xin LI delphij@delphij.net http://www.delphij.net/ FreeBSD - The Power to Serve!

Quanah Gibson-Mount wrote:

Please test RE24 for possible 2.4.16 release.

Hmm, I get failures in several syncrepl-related tests here and there...

Ciao, Michael.

Quanah Gibson-Mount wrote:

Please test RE24 for possible 2.4.16 release.

Ok (using db 4.7.25 patchlevel 3) on:

i686 RedHat3 x86_64 CentOS4 x86_64 Solaris10 sparcv9 Solaris10

Also around 1500 successful runs of test050 with head as of yesterday on the system where I earlier got a seg. fault.

Rein

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Quanah Gibson-Mount wrote:

Please test RE24 for possible 2.4.16 release.

All regression tests passed on FreeBSD/amd64 7.1.

Cheers, - -- Xin LI delphij@delphij.net http://www.delphij.net/ FreeBSD - The Power to Serve!

--On Thursday, March 19, 2009 12:04 AM +0100 Michael Ströder michael@stroeder.com wrote:

Quanah Gibson-Mount wrote:

...

--On Wednesday, March 18, 2009 9:22 PM +0100 Michael Ströder michael@stroeder.com wrote:

...

Quanah Gibson-Mount wrote:

...

Please test RE24 for possible 2.4.16 release.

Hmm, I get failures in several syncrepl-related tests here and there...

Can you be more specific please?

Saw failures in test033 and test058 every now and then, but no seg faults. Not easily reproducible.

You may need to up the SLEEP1 and SLEEP2 values in defines.sh. They work in general, but can need to be increased to have consistent passage, particularly if it is an older or busy system.

--Quanah

--

Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

Michael Ströder wrote:

Quanah Gibson-Mount wrote:

...

--On Wednesday, March 18, 2009 9:22 PM +0100 Michael Ströder michael@stroeder.com wrote:

...

Quanah Gibson-Mount wrote:

...

Please test RE24 for possible 2.4.16 release.

Hmm, I get failures in several syncrepl-related tests here and there...

Can you be more specific please?

Saw failures in test033 and test058 every now and then, but no seg faults. Not easily reproducible.

test058 is only in HEAD. The last test in RE24 is test055.

Are you testing the right code?

Is something improperly ordered during teardowns?

[6] ldap_pvt_thread_pool_destroy(tpool = 0x39c374, run_pending = 1), line 570 in "tpool.c"

https://www.nbcs.rutgers.edu/~richton/slapd.log.20090319.bz2

t@6 (l@6) terminated by signal SEGV (no mapping at the fault address) 0xff2b467c: strlen+0x0080: ld [%o1], %o2 Current function is lutil_debug 66 vsnprintf( buffer, sizeof(buffer), fmt, vl );

Current function is ldap_pvt_thread_join 197 return ERRVAL( pthread_join( thread, thread_return ) ); t@1 (l@1) stopped in __lwp_wait at 0xff31ff68 0xff31ff68: __lwp_wait+0x0008: bgeu,a __lwp_wait+0x1c ! 0xff31ff7c current thread: t@1 [1] __lwp_wait(0x4, 0xffbff55c, 0xff18f818, 0xff1524fc, 0x1, 0xffbff524), at 0xff31ff68 [2] lwp_wait(0x2, 0xffbff55c, 0x2c850, 0xff184e00, 0x5, 0xffbff554), at 0xff15d1cc [3] _thrp_join(0x2, 0x0, 0x0, 0x1, 0x81010100, 0xff00), at 0xff1590c4 =>[4] ldap_pvt_thread_join(thread = 2U, thread_return = (nil)), line 197 in "thr_posix.c" [5] slapd_daemon(), line 2665 in "daemon.c" [6] main(argc = 8, argv = 0xffbff77c), line 948 in "main.c" Current function is ldap_pvt_thread_cond_wait 277 return ERRVAL( pthread_cond_wait( cond, mutex ) ); t@2 (l@2) stopped in __lwp_park at 0xff1654b4 0xff1654b4: __lwp_park+0x0014: bgeu,a __lwp_park+0x28 ! 0xff1654c8 current thread: t@2 [1] __lwp_park(0x4, 0x0, 0x0, 0x1, 0xff178000, 0x0), at 0xff1654b4 [2] cond_wait_queue(0x3e81c8, 0xff178c08, 0x0, 0x0, 0xff140200, 0xff178000), at 0xff1626b8 [3] _cond_wait_cancel(0x3e81c8, 0x3e81b0, 0x3, 0xff370388, 0x75, 0xfe7fdcec), at 0xff162e74 [4] _pthread_cond_wait(0x3e81c8, 0x3e81b0, 0x2c838, 0xff184040, 0x5, 0xff00), at 0xff162eb0 =>[5] ldap_pvt_thread_cond_wait(cond = 0x3e81c8, mutex = 0x3e81b0), line 277 in "thr_posix.c" [6] ldap_pvt_thread_pool_destroy(tpool = 0x39c374, run_pending = 1), line 570 in "tpool.c" [7] slapd_daemon_task(ptr = (nil)), line 2600 in "daemon.c" Current function is ldap_pvt_thread_cond_wait 277 return ERRVAL( pthread_cond_wait( cond, mutex ) ); t@3 (l@3) stopped in __lwp_park at 0xff1654b4 0xff1654b4: __lwp_park+0x0014: bgeu,a __lwp_park+0x28 ! 0xff1654c8 current thread: t@3 [1] __lwp_park(0x4, 0x0, 0x0, 0x1, 0xff178000, 0x0), at 0xff1654b4 [2] cond_wait_queue(0x3e81c8, 0xff178c08, 0x0, 0x0, 0xff140400, 0xff178000), at 0xff1626b8 [3] _cond_wait_cancel(0x3e81c8, 0x3e81b0, 0x0, 0x44f3cc, 0x0, 0x0), at 0xff162e74 [4] _pthread_cond_wait(0x3e81c8, 0x3e81b0, 0xfdfffe0c, 0x1, 0x0, 0xfdfffd81), at 0xff162eb0 =>[5] ldap_pvt_thread_cond_wait(cond = 0x3e81c8, mutex = 0x3e81b0), line 277 in "thr_posix.c" [6] ldap_int_thread_pool_wrapper(xpool = 0x3e81a8), line 654 in "tpool.c" Current function is ldap_pvt_thread_cond_wait 277 return ERRVAL( pthread_cond_wait( cond, mutex ) ); t@4 (l@4) stopped in __lwp_park at 0xff1654b4 0xff1654b4: __lwp_park+0x0014: bgeu,a __lwp_park+0x28 ! 0xff1654c8 current thread: t@4 [1] __lwp_park(0x4, 0x0, 0x6, 0x1, 0xff178000, 0x0), at 0xff1654b4 [2] cond_wait_queue(0x3e81c8, 0xff178c08, 0x0, 0x0, 0xff140600, 0xff178000), at 0xff1626b8 [3] _cond_wait_cancel(0x3e81c8, 0x3e81b0, 0x3e81b0, 0x0, 0x0, 0xfd7ff978), at 0xff162e74 [4] _pthread_cond_wait(0x3e81c8, 0x3e81b0, 0xfd7ffe0c, 0x1, 0x0, 0xfd7ffd81), at 0xff162eb0 =>[5] ldap_pvt_thread_cond_wait(cond = 0x3e81c8, mutex = 0x3e81b0), line 277 in "thr_posix.c" [6] ldap_int_thread_pool_wrapper(xpool = 0x3e81a8), line 654 in "tpool.c" Current function is ldap_pvt_thread_cond_wait 277 return ERRVAL( pthread_cond_wait( cond, mutex ) ); t@5 (l@5) stopped in __lwp_park at 0xff1654b4 0xff1654b4: __lwp_park+0x0014: bgeu,a __lwp_park+0x28 ! 0xff1654c8 current thread: t@5 [1] __lwp_park(0x4, 0x0, 0x0, 0x1, 0xff178000, 0x0), at 0xff1654b4 [2] cond_wait_queue(0x3e81c8, 0xff178c08, 0x0, 0x0, 0xff140800, 0xff178000), at 0xff1626b8 [3] _cond_wait_cancel(0x3e81c8, 0x3e81b0, 0x443230, 0xfcfff978, 0x2, 0xfcfff978), at 0xff162e74 [4] _pthread_cond_wait(0x3e81c8, 0x3e81b0, 0xfcfffe0c, 0x1, 0x0, 0xfcfffd81), at 0xff162eb0 =>[5] ldap_pvt_thread_cond_wait(cond = 0x3e81c8, mutex = 0x3e81b0), line 277 in "thr_posix.c" [6] ldap_int_thread_pool_wrapper(xpool = 0x3e81a8), line 654 in "tpool.c" t@6 (l@6) stopped in strlen at 0xff2b467c 0xff2b467c: strlen+0x0080: ld [%o1], %o2 current thread: t@6 [1] strlen(0x0, 0x69643d30, 0xfc7fe2a0, 0x7efefeff, 0x81010100, 0x0), at 0xff2b467c [2] _doprnt(0x2ae010, 0x0, 0xfc7fe2a0, 0x0, 0x0, 0x0), at 0xff307580 [3] vsnprintf(0xfc7fe318, 0x1000, 0x2ae010, 0xfc7ff368, 0x0, 0x0), at 0xff3095f0 =>[4] lutil_debug(debug = 261, level = 1, fmt = 0x2ae010 ">>> slap_listener(%s)\n", ... = 0x69643d30, ...), line 66 in "debug.c" [5] slap_listener(sl = 0x39d708), line 1740 in "daemon.c" [6] slap_listener_thread(ctx = 0xfc7ffe0c, ptr = 0x39d708), line 1997 in "daemon.c" [7] ldap_int_thread_pool_wrapper(xpool = 0x3e81a8), line 663 in "tpool.c" Current function is lutil_debug 66 vsnprintf( buffer, sizeof(buffer), fmt, vl ); t@7 (l@7) stopped in _doprnt at 0xff3059bc 0xff3059bc: _doprnt+0x0028: st %i1, [%sp + 104] current thread: t@7 [1] _doprnt(0x2c5938, 0xfbfff938, 0xfbffe870, 0x0, 0x0, 0x0), at 0xff3059bc [2] vsnprintf(0xfbffe8e8, 0x1000, 0x2c5938, 0xfbfff938, 0x0, 0x0), at 0xff3095f0 =>[3] lutil_debug(debug = 261, level = 1, fmt = 0x2c5938 "=>do_syncrepl %s\n", ... = 0x44f3cc, ...), line 66 in "debug.c" [4] do_syncrepl(ctx = 0xfbfffe0c, arg = 0x447f28), line 1265 in "syncrepl.c" [5] ldap_int_thread_pool_wrapper(xpool = 0x3e81a8), line 663 in "tpool.c"

Quanah Gibson-Mount wrote:

Please test RE24 for possible 2.4.16 release.

Builds good on AIX 5.3 with BDB 4.6.21 and OpenSSL 0.9.8.8

Ran 300 test038 - OK Ran 500 test050 - 188 completed ok so far

Just wanted to give some feedback.

Cheers, Bill

Thanks!

--Quanah

--

Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc

---

Zimbra :: the leader in open source messaging and collaboration

--On Thursday, March 19, 2009 11:12 AM -0700 Howard Chu hyc@symas.com wrote:

Aaron Richton wrote:

...

Is something improperly ordered during teardowns?

...

[6] ldap_pvt_thread_pool_destroy(tpool = 0x39c374, run_pending = 1), line 570 in "tpool.c"

https://www.nbcs.rutgers.edu/~richton/slapd.log.20090319.bz2

Same issue we ran into earlier, the consumer info has been freed but there's still a consumer task queued in the thread pool. I had already changed things around to avoid this but obviously haven't totally prevented it. Should be fixed now in HEAD.

This is now in RE24 as well. Updated testing results welcomed.

--Quanah

--

Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

On Fri, 20 Mar 2009, Aaron Richton wrote:

Sure, I'll start a compile and a test loop.

Livelocked. Give me some time to write it up...

Aaron Richton wrote:

Please see https://www.nbcs.rutgers.edu/~richton/testrun.20090321.tgz (includes two backtraces too). I'm not sure if the slapd logs are enough to determine sufficient mutex details; if they're not, I think I can play around with dbx a bit.

I presume that the lockup is in server 3 / slapd.3.log ... ?

In backtrace.0 there are two threads t@6 and t@7 spinning on ldap_pvt_thread_yield, can you go up one frame and print si, *si in each thread?

Aaron Richton wrote:

On Sat, 21 Mar 2009, Howard Chu wrote:

...

I presume that the lockup is in server 3 / slapd.3.log ... ?

Hmm. Actually, what caught my eye is slapd.1 / slapd.2, which are both livelocked. But now that you mention it, slapd.3 appears deadlocked...

...

In backtrace.0 there are two threads t@6 and t@7 spinning on ldap_pvt_thread_yield, can you go up one frame and print si, *si in each thread?

I overwrote the tarball:

https://www.nbcs.rutgers.edu/~richton/testrun.20090321.tgz

and it now has backtrace.slapd{1,2,3,4} that correspond by server number. backtrace.0 is now backtrace.slapd1, and includes *si/si.

Thanks, fixed now in HEAD.

On Sat, 21 Mar 2009, Howard Chu wrote:

Thanks, fixed now in HEAD.

A couple more core dumps showed up. I haven't even opened them yet: might be the same, might be different...just a heads up at this point.

Which test were you running?

Do you have the tests/testrun directory? If so, preserve a tarball of that in case it's needed.

Run "threads" in dbx, and do a "where" in each thread:

Segmentation fault in noname.strlen [/usr/lib/libs.a] at 0xd0335680 ($t14) 0xd0335680 (strlen) 89030000 lbz r8,0x0(r3)

Strictly speaking, I can't rule out a bug in AIX's strlen(), but the odds that that's what you want us looking at are infinitesimally small.

William Jojo wrote:

Was running test050 500 times. Been watching threads on test038 and

Actually test039 is the problem, not 038.