RE24 testing

RE24 and BDB 4.7.25p1 hanging

test050-syncrepl-multimaster...

Quanah Gibson-Mount

Wednesday, 10 September 2008 Wed, 10 Sep '08

9:33 a.m.

Please test RE24. Thanks! --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

Show replies by date

Dieter Kluenter

Wednesday, 10 September Wed, 10 Sep

11:47 a.m.

Quanah Gibson-Mount <quanah(a)zimbra.com> writes:

...

Please test RE24.

success on x86_64 openSuse-11.0 success on x64 openSolaris snv_95 -Dieter -- Dieter Klünter | Systemberatung http://www.dpunkt.de/buecher/2104.html GPG Key ID:8EF7B6C6 53°08'09,95"N 10°08'02,42"E

Aaron Richton

3:24 p.m.

Would it be prudent to make these tests with BDB 4.7, or do you have reason to believe that's premature? On Wed, 10 Sep 2008, Quanah Gibson-Mount wrote:

...

Please test RE24. Thanks! --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

Howard Chu

3:32 p.m.

Aaron Richton wrote:

...

Would it be prudent to make these tests with BDB 4.7, or do you have reason to believe that's premature?

All of BDB 4.4 thru 4.7 should work equally well.

...

On Wed, 10 Sep 2008, Quanah Gibson-Mount wrote: > Please test RE24. > > Thanks! > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration >

-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

Gavin Henry

Monday, 15 September Mon, 15 Sep

2:46 a.m.

Aaron Richton wrote:

...

Would it be prudent to make these tests with BDB 4.7, or do you have reason to believe that's premature?

So 4.2 is dropped? checking db.h usability... yes checking db.h presence... yes checking for db.h... yes checking for Berkeley DB major version... 4 checking for Berkeley DB minor version... 2 checking for Berkeley DB link (-ldb-4.2)... yes checking for Berkeley DB version match... yes checking for Berkeley DB thread support... yes checking Berkeley DB version for BDB/HDB backends... no configure: error: BDB/HDB: BerkeleyDB version incompatible make: *** No rule to make target `depend'. Stop. make: *** No targets specified and no makefile found. Stop. make: *** No rule to make target `test'. Stop. If it is, the following should say no right? checking for Berkeley DB version match... yes checking for Berkeley DB thread support... yes -- Kind Regards, Gavin Henry. OpenLDAP Engineering Team. E ghenry(a)OpenLDAP.org Community developed LDAP software. http://www.openldap.org/project/

Hallvard B Furuseth

3:46 a.m.

Gavin Henry writes:

...

checking db.h usability... yes checking db.h presence... yes checking for db.h... yes checking for Berkeley DB major version... 4 checking for Berkeley DB minor version... 2 checking for Berkeley DB link (-ldb-4.2)... yes checking for Berkeley DB version match... yes checking for Berkeley DB thread support... yes checking Berkeley DB version for BDB/HDB backends... no configure: error: BDB/HDB: BerkeleyDB version incompatible make: *** No rule to make target `depend'. Stop. make: *** No targets specified and no makefile found. Stop. make: *** No rule to make target `test'. Stop. If it is, the following should say no right? checking for Berkeley DB version match... yes checking for Berkeley DB thread support... yes

Well, no. They are about the DB installation in themselves - the first is whether db.h version number matches the linked library version. But we could put the test for "BerkeleyDB version incompatible" just below finding major and minor version. Also I think the texts can be clarified: checking for Berkeley DB major version in db.h... 4 checking for Berkeley DB minor version in db.h... 2 checking Berkeley DB version supported by BDB/HDB backends... no configure: error: BerkeleyDB version 4.2 incompatible with BDB/HDB backends (or success above:) checking for Berkeley DB link (-ldb-4.2)... yes checking for Berkeley DB version match (header vs. library)... yes checking for Berkeley DB thread support... yes Could remove some cruft too. OL_BERKELEY_COMPAT_DB in build/openldap.m4 seems to be unused. And the code which searches for minor/major version numbers could be a for(version) loop. All this has been like this for years though, so I think we don't need to hold up next 2.4 release for this. -- Hallvard

Howard Chu

9:46 p.m.

Hallvard B Furuseth wrote:

...

But we could put the test for "BerkeleyDB version incompatible" just below finding major and minor version. Also I think the texts can be clarified: checking for Berkeley DB major version in db.h... 4 checking for Berkeley DB minor version in db.h... 2 checking Berkeley DB version supported by BDB/HDB backends... no configure: error: BerkeleyDB version 4.2 incompatible with BDB/HDB backends (or success above:) checking for Berkeley DB link (-ldb-4.2)... yes checking for Berkeley DB version match (header vs. library)... yes checking for Berkeley DB thread support... yes Could remove some cruft too. OL_BERKELEY_COMPAT_DB in build/openldap.m4 seems to be unused. And the code which searches for minor/major version numbers could be a for(version) loop. All this has been like this for years though, so I think we don't need to hold up next 2.4 release for this.

Well, now that we have new ITSs holding it up, probably no harm in making these changes. I'll take a look. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

Howard Chu

9:21 p.m.

Gavin Henry wrote:

...

Aaron Richton wrote: > Would it be prudent to make these tests with BDB 4.7, or do you have > reason to believe that's premature? So 4.2 is dropped?

Yes.

...

Wrong. The "checking Berkeley DB version" message checks the version. (gee, how strange...) -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

Jens Vagelpohl

Thursday, 11 September Thu, 11 Sep

1:21 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Having a failure on OS X 10.5/Intel:

...

>>>> Starting test019-syncreplication-cascade ...

running defines.sh Starting master slapd on TCP/IP port 9011... Using ldapsearch to check that master slapd (pid=44726) is running... Using ldapadd to create the context prefix entry in the master... Starting R1 slave slapd on TCP/IP port 9012... Using ldapsearch to check that R1 slave slapd (pid=44731) is running... Starting R2 slave slapd on TCP/IP port 9013... Using ldapsearch to check that R2 slave slapd (pid=44735) is running... Starting P1 slave slapd on TCP/IP port 9014... Using ldapsearch to check that P1 slave slapd (pid=44739) is running... Starting P2 slave slapd on TCP/IP port 9015... Using ldapsearch to check that P2 slave slapd (pid=44743) is running... Starting P3 slave slapd on TCP/IP port 9016... Using ldapsearch to check that P3 slave slapd (pid=44747) is running... Using ldapadd to populate the master directory... Waiting 25 seconds for syncrepl to receive changes... Using ldapmodify to modify master directory... Waiting 25 seconds for syncrepl to receive changes... Using ldapsearch to read all the entries from the master... Using ldapsearch to read all the entries from the R1 slave... Using ldapsearch to read all the entries from the R2 slave... Using ldapsearch to read all the entries from the P1 slave... Using ldapsearch to read all the entries from the P2 slave... Using ldapsearch to read all the entries from the P3 slave... Filtering master ldapsearch results... Filtering R1 slave ldapsearch results... Filtering R2 slave ldapsearch results... Filtering P1 slave ldapsearch results... Filtering P2 slave ldapsearch results... Filtering P3 slave ldapsearch results... Comparing retrieved entries from master and R1 slave... test failed - master and R1 slave databases differ

...

>>>> ./scripts/test019-syncreplication-cascade failed (exit 1)

make[2]: *** [bdb-yes] Error 1 make[1]: *** [test] Error 2 make: *** [test] Error 2 Wha data do you need? jens

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkjI1Q4ACgkQRAx5nvEhZLLT+ACfTcyR3uM5OWdgZumEuPX9R2bN iv4An1Rba+Ycb7LjvuNTU9zoqwEhBhpN =sH9N -----END PGP SIGNATURE-----

Quanah Gibson-Mount

Sunday, 14 September Sun, 14 Sep

4:27 p.m.

--On Thursday, September 11, 2008 10:21 AM +0200 Jens Vagelpohl <jens(a)dataflake.org> wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Having a failure on OS X 10.5/Intel: >>>>> Starting test019-syncreplication-cascade ... running defines.sh Starting master slapd on TCP/IP port 9011... Using ldapsearch to check that master slapd (pid=44726) is running... Using ldapadd to create the context prefix entry in the master... Starting R1 slave slapd on TCP/IP port 9012... Using ldapsearch to check that R1 slave slapd (pid=44731) is running... Starting R2 slave slapd on TCP/IP port 9013... Using ldapsearch to check that R2 slave slapd (pid=44735) is running... Starting P1 slave slapd on TCP/IP port 9014... Using ldapsearch to check that P1 slave slapd (pid=44739) is running... Starting P2 slave slapd on TCP/IP port 9015... Using ldapsearch to check that P2 slave slapd (pid=44743) is running... Starting P3 slave slapd on TCP/IP port 9016... Using ldapsearch to check that P3 slave slapd (pid=44747) is running... Using ldapadd to populate the master directory... Waiting 25 seconds for syncrepl to receive changes... Using ldapmodify to modify master directory... Waiting 25 seconds for syncrepl to receive changes... Using ldapsearch to read all the entries from the master... Using ldapsearch to read all the entries from the R1 slave... Using ldapsearch to read all the entries from the R2 slave... Using ldapsearch to read all the entries from the P1 slave... Using ldapsearch to read all the entries from the P2 slave... Using ldapsearch to read all the entries from the P3 slave... Filtering master ldapsearch results... Filtering R1 slave ldapsearch results... Filtering R2 slave ldapsearch results... Filtering P1 slave ldapsearch results... Filtering P2 slave ldapsearch results... Filtering P3 slave ldapsearch results... Comparing retrieved entries from master and R1 slave... test failed - master and R1 slave databases differ >>>>> ./scripts/test019-syncreplication-cascade failed (exit 1) make[2]: *** [bdb-yes] Error 1 make[1]: *** [test] Error 2 make: *** [test] Error 2 Wha data do you need?

What BDB version you are using would be useful to start. Also, manually run the diff the test is running, and see what the difference being reported is -- Missing entries, or something else? --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

Jens Vagelpohl

Thursday, 18 September Thu, 18 Sep

9:26 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sep 15, 2008, at 01:27 , Quanah Gibson-Mount wrote:

...

--On Thursday, September 11, 2008 10:21 AM +0200 Jens Vagelpohl <jens(a)dataflake.org > wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Having a failure on OS X 10.5/Intel: > > >>>>> Starting test019-syncreplication-cascade ... > running defines.sh > Starting master slapd on TCP/IP port 9011... > Using ldapsearch to check that master slapd (pid=44726) is running... > Using ldapadd to create the context prefix entry in the master... > Starting R1 slave slapd on TCP/IP port 9012... > Using ldapsearch to check that R1 slave slapd (pid=44731) is > running... > Starting R2 slave slapd on TCP/IP port 9013... > Using ldapsearch to check that R2 slave slapd (pid=44735) is > running... > Starting P1 slave slapd on TCP/IP port 9014... > Using ldapsearch to check that P1 slave slapd (pid=44739) is > running... > Starting P2 slave slapd on TCP/IP port 9015... > Using ldapsearch to check that P2 slave slapd (pid=44743) is > running... > Starting P3 slave slapd on TCP/IP port 9016... > Using ldapsearch to check that P3 slave slapd (pid=44747) is > running... > Using ldapadd to populate the master directory... > Waiting 25 seconds for syncrepl to receive changes... > Using ldapmodify to modify master directory... > Waiting 25 seconds for syncrepl to receive changes... > Using ldapsearch to read all the entries from the master... > Using ldapsearch to read all the entries from the R1 slave... > Using ldapsearch to read all the entries from the R2 slave... > Using ldapsearch to read all the entries from the P1 slave... > Using ldapsearch to read all the entries from the P2 slave... > Using ldapsearch to read all the entries from the P3 slave... > Filtering master ldapsearch results... > Filtering R1 slave ldapsearch results... > Filtering R2 slave ldapsearch results... > Filtering P1 slave ldapsearch results... > Filtering P2 slave ldapsearch results... > Filtering P3 slave ldapsearch results... > Comparing retrieved entries from master and R1 slave... > test failed - master and R1 slave databases differ > >>>>> ./scripts/test019-syncreplication-cascade failed (exit 1) > make[2]: *** [bdb-yes] Error 1 > make[1]: *** [test] Error 2 > make: *** [test] Error 2 > > Wha data do you need? What BDB version you are using would be useful to start. Also, manually run the diff the test is running, and see what the difference being reported is -- Missing entries, or something else?

The difference are missing entries. I compiled against BDB 4.6.21. Following list traffic I have cleaned out that testing sandbox and built BDB 4.7.25 to retry my test. However, now I'm getting a (non-BDB- related) failure during the configure stage: openldap-2.4-branch$ ./configure --enable-overlays --enable-ldap -- enable-meta Configuring OpenLDAP 2.4.X-Engineering (from CVS sources) ... checking build system type... i686-apple-darwin9.5.0 checking host system type... i686-apple-darwin9.5.0 < snip lots of stuff > config.status: creating servers/Makefile config.status: creating servers/slapd/Makefile config.status: creating servers/slapd/back-bdb/Makefile config.status: creating servers/slapd/back-dnssrv/Makefile config.status: creating servers/slapd/back-hdb/Makefile config.status: creating servers/slapd/back-ldap/Makefile config.status: creating servers/slapd/back-ldif/Makefile config.status: creating servers/slapd/back-meta/Makefile config.status: creating servers/slapd/back-monitor/Makefile config.status: creating servers/slapd/back-ndb/Makefile config.status: error: cannot find input file: servers/slapd/back-ndb/ Makefile.in ?? jens

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEUEARECAAYFAkjSgSUACgkQRAx5nvEhZLKQvQCgtnaVTRhgS8EskDhnGnys4/km RhYAmJyW/JRcdoOyST3xKU0kMkc9UZk= =MFmE -----END PGP SIGNATURE-----

Quanah Gibson-Mount

10:21 a.m.

--On Thursday, September 18, 2008 6:26 PM +0200 Jens Vagelpohl <jens(a)dataflake.org> wrote:

...

servers/slapd/back-ndb/Makefile.in

Do you have a servers/slapd/back-ndb/ directory? If not, you'll need to cvs upd it. I'd also suggest doing a --disable-ndb for now. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

Jens Vagelpohl

10:26 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sep 18, 2008, at 19:21 , Quanah Gibson-Mount wrote:

...

--On Thursday, September 18, 2008 6:26 PM +0200 Jens Vagelpohl <jens(a)dataflake.org > wrote: > servers/slapd/back-ndb/Makefile.in Do you have a servers/slapd/back-ndb/ directory? If not, you'll need to cvs upd it. I'd also suggest doing a --disable-ndb for now.

I have that folder and it's empty. I also did a "cvs up". I also tried "--disable-ndb", but the configure process still went into that folder trying to find a Makefile.in. jens

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkjSjzwACgkQRAx5nvEhZLL2GwCcCjaStnMUHLdiBSTIGF4Oe6h1 z7QAn0dEZ6vGH/cpq3KsrDjKgfPVmZlu =Wswe -----END PGP SIGNATURE-----

Quanah Gibson-Mount

11:11 a.m.

--On Thursday, September 18, 2008 7:26 PM +0200 Jens Vagelpohl <jens(a)dataflake.org> wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sep 18, 2008, at 19:21 , Quanah Gibson-Mount wrote: > --On Thursday, September 18, 2008 6:26 PM +0200 Jens Vagelpohl > <jens(a)dataflake.org > > wrote: > > >> servers/slapd/back-ndb/Makefile.in > > Do you have a servers/slapd/back-ndb/ directory? If not, you'll > need to cvs upd it. I'd also suggest doing a --disable-ndb for now. I have that folder and it's empty. I also did a "cvs up". I also tried "--disable-ndb", but the configure process still went into that folder trying to find a Makefile.in.

Did you go into that folder and do a cvs upd? It's definitely not empty in CVS or my checkout. ;) --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

Gavin Henry

12:41 p.m.

...

Did you go into that folder and do a cvs upd? It's definitely not empty in CVS or my checkout. ;)

Or do "cvs up -d" -- Kind Regards, Gavin Henry. OpenLDAP Engineering Team. E ghenry(a)OpenLDAP.org Community developed LDAP software. http://www.openldap.org/project/

Jens Vagelpohl

1:45 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sep 18, 2008, at 21:41 , Gavin Henry wrote:

...

> Did you go into that folder and do a cvs upd? It's definitely not > empty in > CVS or my checkout. ;) Or do "cvs up -d"

That was it. Sorry for being dense. This is the only project I am involved in which still uses CVS, all others moved to Subversion several years ago... jens

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkjSvc0ACgkQRAx5nvEhZLLVmACgoQBLnyd2HgFJoXNNvboGOKs4 FssAnjImvI2zAaH3AR+kkPnWJ/KxXyTz =q4oR -----END PGP SIGNATURE-----

Quanah Gibson-Mount

Friday, 19 September Fri, 19 Sep

4:19 p.m.

--On Thursday, September 18, 2008 10:45 PM +0200 Jens Vagelpohl <jens(a)dataflake.org> wrote:

...

> Or do "cvs up -d" That was it. Sorry for being dense. This is the only project I am involved in which still uses CVS, all others moved to Subversion several years ago...

Yes, I keep hoping someday we'll get there too. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

Jens Vagelpohl

Thursday, 18 September Thu, 18 Sep

2:30 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sep 18, 2008, at 21:41 , Gavin Henry wrote:

...

> Did you go into that folder and do a cvs upd? It's definitely not > empty in > CVS or my checkout. ;) Or do "cvs up -d"

OK, ran the tests now on OS X 10.5.5/Intel using BDB 4.7.25 and I get a failure in test050-syncrepl-multimaster, I believe someone else saw this as well: <snip> Using ldapsearch to read all the entries from the producer... Using ldapsearch to read all the entries from consumer1... Using ldapsearch to read all the entries from consumer2... Filtering producer results... Filtering consumer1 results... Filtering consumer2 results... Comparing retrieved entries from producer and consumer1... Comparing retrieved entries from producer and consumer2... test failed - producer and consumer2 databases differ

...

>>>> ./scripts/test050-syncrepl-multimaster failed (exit 1)

make[2]: *** [bdb-yes] Error 1 make[1]: *** [test] Error 2 make: *** [test] Error 2 jens

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkjSyH4ACgkQRAx5nvEhZLIlawCdFxeLFl76ITLztLuL1Yh1mFCR QKAAnj5Q+SSeGDyAqktzkpy6kt+tpWiI =vFHl -----END PGP SIGNATURE-----

Michael Ströder

Saturday, 20 September Sat, 20 Sep

4:51 a.m.

Jens Vagelpohl wrote:

...

OK, ran the tests now on OS X 10.5.5/Intel using BDB 4.7.25

BTW: There's already a patch available for BDB 4.7.25: http://www.oracle.com/technology/products/berkeley-db/db/update/4.7.25/pa... Does anybody know whether that's needed for OpenLDAP? Ciao, Michael.

Gavin Henry

12:56 p.m.

----- "Michael Ströder" <michael(a)stroeder.com> wrote:

...

Jens Vagelpohl wrote: > OK, ran the tests now on OS X 10.5.5/Intel using BDB 4.7.25 BTW: There's already a patch available for BDB 4.7.25: http://www.oracle.com/technology/products/berkeley-db/db/update/4.7.25/pa... Does anybody know whether that's needed for OpenLDAP?

Not sure, but I applied it when testing and failing on test50. -- Kind Regards, Gavin Henry. OpenLDAP Engineering Team. E ghenry(a)OpenLDAP.org Community developed LDAP software. http://www.openldap.org/project/

Jens Vagelpohl

Sunday, 21 September Sun, 21 Sep

2:48 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sep 20, 2008, at 21:56 , Gavin Henry wrote:

...

----- "Michael Ströder" <michael(a)stroeder.com> wrote: > Jens Vagelpohl wrote: >> OK, ran the tests now on OS X 10.5.5/Intel using BDB 4.7.25 > > BTW: There's already a patch available for BDB 4.7.25: > > http://www.oracle.com/technology/products/berkeley-db/db/update/4.7.25/pa... > > Does anybody know whether that's needed for OpenLDAP? > Not sure, but I applied it when testing and failing on test50.

I have applied the patch as well and reran the tests a few times. I've been seing intermittent failures on test019, test050 and now even on test048. This is on OS X 10.5.5/Intel. jens

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkjWGH8ACgkQRAx5nvEhZLJcLQCfbBPb8er8BfNYW2kfwrjiuAnP QuMAnR4i1aQ2RLmgbcFO0G0B24Asw/XM =+HJN -----END PGP SIGNATURE-----

Howard Chu

Monday, 22 September Mon, 22 Sep

2:22 p.m.

Michael Ströder wrote:

...

For the record - no, that patch is for a BDB subsystem that OpenLDAP never uses. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

Aaron Richton

Friday, 12 September Fri, 12 Sep

7:39 a.m.

test050 isn't replicating "cn=Consumer 2 Test,dc=example,dc=com" reproducibly. https://www.nbcs.rutgers.edu/~richton/test050fail.tar.gz

Aaron Richton

Monday, 15 September Mon, 15 Sep

1:37 p.m.

See ITS #5071. On Fri, 12 Sep 2008, Aaron Richton wrote:

...

test050 isn't replicating "cn=Consumer 2 Test,dc=example,dc=com" reproducibly. https://www.nbcs.rutgers.edu/~richton/test050fail.tar.gz

5371

days inactive

5383

days old

openldap-devel@openldap.org

Manage subscription

23 comments

8 participants

tags (0)

participants (8)

Aaron Richton
Dieter Kluenter
Gavin Henry
Hallvard B Furuseth
Howard Chu
Jens Vagelpohl
Michael Ströder
Quanah Gibson-Mount

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

RE24 testing