openldap-devel June 2009

openldap-devel@openldap.org

16 participants
22 discussions

RE24 testing call (2.4.17)
by Quanah Gibson-Mount 18 Jun '09

18 Jun '09

Please test RE24 as we prepare for 2.4.17. Thanks! --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

8 21

Re: commit: ldap/servers/slapd/overlays pcache.c
by Howard Chu 15 Jun '09

15 Jun '09

hyc(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/servers/slapd/overlays > > Modified Files: > pcache.c 1.168 -> 1.169 > > Log Message: > Partially revert 1.32; pcache must allow other callbacks to see its results Regarding the original reason for this change... It's clear that what we really want is for pcache to behave the same in the cached vs uncached case, wrt other overlays/callbacks. So wherever pcache takes control in the call stack for the uncached case, it should do the same in the cached case. What's not clear is how to do that in general. When "response-callback head" has been configured, it should just execute and leave the callback stack intact when answering from cache. When "tail" (the default) is in effect then it probably should bypass the stack. The above commit just leaves the stack intact; I'll fix it up now to depend on the response-callback setting. > * To: OpenLDAP Commit <openldap-commit2devel(a)OpenLDAP.org> > * Subject: Re: commit: ldap/servers/slapd/overlays pcache.c > * From: Pierangelo Masarati <ando(a)sys-net.it> > * Date: Tue, 24 Aug 2004 10:40:00 +0200 > ando(a)OpenLDAP.org wrote: > > > Update of /repo/OpenLDAP/pkg/ldap/servers/slapd/overlays > > > Modified Files: > pcache.c 1.31 -> 1.32 > > > Log Message: > - proxy cache erroneously returns the filtering attributes > and the objectClass right after caching, even if not requested, > while subsequent searches are fine; > > > I think this is a bug; it was also reflected in the test data, which has been > fixed accordingly > > - the response callback needs be apended at the end of the > callback list, otherwise the resulting entries are cached > > > I ran into this issue while developing an overlay that dynamically builds > entries; the raw entry was being cached instead of the completed one, > because pcache adds its callback only when required, while that of my > overlay is stacked by the overlay infrastructure. I think the "natural" > behavior of pcache is to cache the final result that is sent to the user. > I guess there might be applications where this is not true, so I made the > original behavior configurable by an (undocumented) directive. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: held locks after exit
by Benoit Donnette 11 Jun '09

11 Jun '09

Well, you've highlighted 4 unreleased locks, a solution quite like using a sledgehammer to crack a nut but predictably efficient would be to backtrace() every lock and unlock so as to match them and to identify which locks haven't been released (but your traces show enough detail to help us focus the watch on specific db4 code). If this can help. > After an otherwise "clean" exit of test058, the smc/db directory showed > the following, despite the fact that there was no more 9196 pid on the > system. Is there any facility in db4 to trace locking? > > I don't think slapd's own debugging would give the level of detail that > would be necessary to find a smoking gun, but if I'm just reading stuff > wrong, I have the testrun directory from this. > > ---------- begin output ---------- > > Default locking region information: > 138 Last allocated locker ID > 0x7fffffff Current maximum unused locker ID > 9 Number of lock modes > 1000 Maximum number of locks possible > 1000 Maximum number of lockers possible > 1000 Maximum number of lock objects possible > 20 Number of lock object partitions > 4 Number of current locks > 12 Maximum number of locks at any one time > 3 Maximum number of locks in any one bucket > 0 Maximum number of locks stolen by for an empty partition > 0 Maximum number of locks stolen for any one partition > 9 Number of current lockers > 15 Maximum number of lockers at any one time > 4 Number of current lock objects > 8 Maximum number of lock objects at any one time > 2 Maximum number of lock objects in any one bucket > 0 Maximum number of objects stolen by for an empty partition > 0 Maximum number of objects stolen for any one partition > 1009 Total number of locks requested > 1005 Total number of locks released > 0 Total number of locks upgraded > 46 Total number of locks downgraded > 4 Lock requests not available due to conflicts, for which we waited > 0 Lock requests not available due to conflicts, for which we did not wait > 0 Number of deadlocks > 0 Lock timeout value > 0 Number of locks that have timed out > 0 Transaction timeout value > 0 Number of transactions that have timed out > 488KB The size of the lock region > 4 The number of partition locks that required waiting (0%) > 4 The maximum number of times any partition lock was waited for (0%) > 0 The number of object queue operations that required waiting (0%) > 1 The number of locker allocations that required waiting (0%) > 0 The number of region locks that required waiting (0%) > 2 Maximum hash bucket length > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Lock REGINFO information: > Lock Region type > 5 Region ID > __db.005 Region name > 0xfecf0000 Original region address > 0xfecf0000 Region address > 0xfecf00c8 Region primary address > 0 Region maximum allocation > 0 Region allocated > Region allocations: 3006 allocations, 0 failures, 0 frees, 1 longest > Allocations by power-of-two sizes: > 1KB 3003 > 2KB 0 > 4KB 0 > 8KB 0 > 16KB 2 > 32KB 0 > 64KB 1 > 128KB 0 > 256KB 0 > 512KB 0 > 1024KB 0 > REGION_JOIN_OK Region flags > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Lock region parameters: > 104 Lock region region mutex [0/119 0% 11775/1] > 1031 locker table size > 1031 object table size > 616 obj_off > 63456 locker_off > 0 need_dd > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Lock conflict matrix: > 0 0 0 0 0 0 0 0 0 > 0 0 1 0 1 0 1 0 1 > 0 1 1 1 1 1 1 1 1 > 0 0 0 0 0 0 0 0 0 > 0 1 1 0 0 0 0 1 1 > 0 0 1 0 0 0 0 0 1 > 0 1 1 0 0 0 0 1 1 > 0 0 1 0 1 0 1 0 0 > 0 1 1 0 1 1 1 0 1 > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Locks grouped by lockers: > Locker Mode Count Status ----------------- Object --------------- > 85 dd= 0 locks held 1 write locks 0 pid/thread 9196/1 > 85 READ 1 HELD id2entry.bdb handle > 0 > 86 dd= 0 locks held 0 write locks 0 pid/thread 9196/1 > 87 dd= 0 locks held 1 write locks 0 pid/thread 9196/1 > 87 READ 1 HELD dn2id.bdb handle > 0 > 88 dd= 0 locks held 0 write locks 0 pid/thread 9196/1 > 89 dd= 0 locks held 0 write locks 0 pid/thread 9196/1 > 8a dd= 0 locks held 0 write locks 0 pid/thread 9196/1 > 8000013c dd= 0 locks held 0 write locks 0 pid/thread 9196/1 > 8000013d dd= 0 locks held 0 write locks 0 pid/thread 9196/1 > 80000145 dd= 0 locks held 2 write locks 2 pid/thread 9196/1 > 80000145 WRITE 1 HELD 0x39548 len: 5 data: 0000000x0100 > 80000145 WRITE 1 HELD id2entry.bdb page 1 > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Locks grouped by object: > Locker Mode Count Status ----------------- Object --------------- > 80000145 WRITE 1 HELD 0x39548 len: 5 data: 0000000x0100 > > 80000145 WRITE 1 HELD id2entry.bdb page 1 > > 85 READ 1 HELD id2entry.bdb handle > 0 > > 87 READ 1 HELD dn2id.bdb handle > 0 > > > >

3 2

Re: RE24 testing call (2.4.17)
by William Jojo 11 Jun '09

11 Jun '09

---- Original message ---- >Date: Mon, 08 Jun 2009 16:59:58 -0700 >From: Xin LI <delphij(a)delphij.net> >Subject: Re: RE24 testing call (2.4.17) >To: Quanah Gibson-Mount <quanah(a)zimbra.com> >Cc: openldap-devel(a)openldap.org > >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >Quanah Gibson-Mount wrote: >> Please test RE24 as we prepare for 2.4.17. Thanks! > >All regression tests gave 'completed OK' (test 058-syncrepl-asymmetric >says No race errors found after 10 iterations but Found 2 errors) on >FreeBSD/amd64. > Exact same for AIX 5.3. Cheers, Bill >Cheers, >- -- >Xin LI <delphij(a)delphij.net> http://www.delphij.net/ >FreeBSD - The Power to Serve! >-----BEGIN PGP SIGNATURE----- >Version: GnuPG v2.0.11 (FreeBSD) > >iEYEARECAAYFAkotpf4ACgkQi+vbBBjt66DRDwCgmJCG6IXHS62dBFbauXEuK9nG >c7sAnAkKs/9rVwGA1v0Nv0W54tCeinA3 >=Haue >-----END PGP SIGNATURE-----

1 0

Re: RE24 testing call (2.4.17)
by Hallvard B Furuseth 10 Jun '09

10 Jun '09

./configure --disable-backends --disable-overlays --enable-bdb --enable-dds \ CC=gcc CFLAGS="-g -O0" CPPFLAGS="-DLDAP_MEMORY_DEBUG -DSLAP_NO_SL_MALLOC ... ./run test046-dds ... Re-vitalizing the initial dynamic entry... Re-renaming the subordinate dynamic entry (new superior)... Deleting a dynamic entry... Refreshing the remaining dynamic entry... Waiting 15 seconds for remaining entry to expire... ./scripts/test046-dds: line 370: 16351 Segmentation fault (core dumped) $SLAPD -f $CONF1 -h $URI1 -d $LVL $TIMING > $LOG1 2>&1 gdb -q ../servers/slapd/slapd core.16351 #0 0x00000000004e45a7 in dds_expire (ctx=0x415e8d60, di=0x2a1e990) at dds.c:219 219 op->o_req_dn = de->de_ndn; (gdb) bt #0 0x00000000004e45a7 in dds_expire (ctx=0x415e8d60, di=0x2a1e990) at dds.c:219 #1 0x00000000004e4867 in dds_expire_fn (ctx=0x415e8d60, arg=0x2b97c70) at dds.c:274 #2 0x00000000004fff10 in ldap_int_thread_pool_wrapper (xpool=0x29f0a30) at tpool.c:663 #3 0x000000350ee06367 in start_thread () from /lib64/libpthread.so.0 #4 0x000000350e2d30ad in clone () from /lib64/libc.so.6 (gdb) p de $1 = (dds_expire_t *) 0xffffffffffffffff It only breaks if both of -DLDAP_MEMORY_DEBUG -DSLAP_NO_SL_MALLOC are given, then it crashes the same way every time. x86_64, Red Hat Enterprise Linux 5.3. C library: /lib64/libc.so.6 -> libc-2.5.so. -- Hallvard

1 0

Beware of Fedora 11's glibc
by Howard Chu 10 Jun '09

10 Jun '09

From this bug it appears there's a known history of problems with the experimental malloc being used in the new glibc. https://bugzilla.redhat.com/show_bug.cgi?id=494631 It appears that slapd is hitting some issues in this code still, even though the other symptoms in that bug report are marked as fixed. LD_PRELOADing a 3rd party malloc library shows no issues and likewise valgrind finds no invalid memory accesses in our code. I was seeing crashes on x86_64; the 32 bit code may actually be fixed by now. Just an fyi. I've already spent more than 2 weeks looking for a memory overwrite in slapd only to find that the malloc library is broken. I can't afford to spend any more time tracking this down. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 2

held locks after exit
by Aaron Richton 09 Jun '09

09 Jun '09

After an otherwise "clean" exit of test058, the smc/db directory showed the following, despite the fact that there was no more 9196 pid on the system. Is there any facility in db4 to trace locking? I don't think slapd's own debugging would give the level of detail that would be necessary to find a smoking gun, but if I'm just reading stuff wrong, I have the testrun directory from this. ---------- begin output ---------- Default locking region information: 138 Last allocated locker ID 0x7fffffff Current maximum unused locker ID 9 Number of lock modes 1000 Maximum number of locks possible 1000 Maximum number of lockers possible 1000 Maximum number of lock objects possible 20 Number of lock object partitions 4 Number of current locks 12 Maximum number of locks at any one time 3 Maximum number of locks in any one bucket 0 Maximum number of locks stolen by for an empty partition 0 Maximum number of locks stolen for any one partition 9 Number of current lockers 15 Maximum number of lockers at any one time 4 Number of current lock objects 8 Maximum number of lock objects at any one time 2 Maximum number of lock objects in any one bucket 0 Maximum number of objects stolen by for an empty partition 0 Maximum number of objects stolen for any one partition 1009 Total number of locks requested 1005 Total number of locks released 0 Total number of locks upgraded 46 Total number of locks downgraded 4 Lock requests not available due to conflicts, for which we waited 0 Lock requests not available due to conflicts, for which we did not wait 0 Number of deadlocks 0 Lock timeout value 0 Number of locks that have timed out 0 Transaction timeout value 0 Number of transactions that have timed out 488KB The size of the lock region 4 The number of partition locks that required waiting (0%) 4 The maximum number of times any partition lock was waited for (0%) 0 The number of object queue operations that required waiting (0%) 1 The number of locker allocations that required waiting (0%) 0 The number of region locks that required waiting (0%) 2 Maximum hash bucket length =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Lock REGINFO information: Lock Region type 5 Region ID __db.005 Region name 0xfecf0000 Original region address 0xfecf0000 Region address 0xfecf00c8 Region primary address 0 Region maximum allocation 0 Region allocated Region allocations: 3006 allocations, 0 failures, 0 frees, 1 longest Allocations by power-of-two sizes: 1KB 3003 2KB 0 4KB 0 8KB 0 16KB 2 32KB 0 64KB 1 128KB 0 256KB 0 512KB 0 1024KB 0 REGION_JOIN_OK Region flags =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Lock region parameters: 104 Lock region region mutex [0/119 0% 11775/1] 1031 locker table size 1031 object table size 616 obj_off 63456 locker_off 0 need_dd =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Lock conflict matrix: 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 1 0 1 0 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 1 0 0 1 0 0 0 0 0 1 0 1 1 0 0 0 0 1 1 0 0 1 0 1 0 1 0 0 0 1 1 0 1 1 1 0 1 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Locks grouped by lockers: Locker Mode Count Status ----------------- Object --------------- 85 dd= 0 locks held 1 write locks 0 pid/thread 9196/1 85 READ 1 HELD id2entry.bdb handle 0 86 dd= 0 locks held 0 write locks 0 pid/thread 9196/1 87 dd= 0 locks held 1 write locks 0 pid/thread 9196/1 87 READ 1 HELD dn2id.bdb handle 0 88 dd= 0 locks held 0 write locks 0 pid/thread 9196/1 89 dd= 0 locks held 0 write locks 0 pid/thread 9196/1 8a dd= 0 locks held 0 write locks 0 pid/thread 9196/1 8000013c dd= 0 locks held 0 write locks 0 pid/thread 9196/1 8000013d dd= 0 locks held 0 write locks 0 pid/thread 9196/1 80000145 dd= 0 locks held 2 write locks 2 pid/thread 9196/1 80000145 WRITE 1 HELD 0x39548 len: 5 data: 0000000x0100 80000145 WRITE 1 HELD id2entry.bdb page 1 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Locks grouped by object: Locker Mode Count Status ----------------- Object --------------- 80000145 WRITE 1 HELD 0x39548 len: 5 data: 0000000x0100 80000145 WRITE 1 HELD id2entry.bdb page 1 85 READ 1 HELD id2entry.bdb handle 0 87 READ 1 HELD dn2id.bdb handle 0

1 0

Re: RE24 testing call (2.4.17)
by ghenry＠OpenLDAP.org 09 Jun '09

09 Jun '09

Jaunty - 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:58:03 UTC 2009 x86_64 GNU/Linux all good. -- Kind Regards, Gavin Henry. OpenLDAP Engineering Team. E ghenry(a)OpenLDAP.org Community developed LDAP software. http://www.openldap.org/project/

1 0

BDB / HDB Implementaion Internals
by Lorenzo Pastrana 09 Jun '09

09 Jun '09

Hello, I'm diving into the BDB backend code, and I was wondering if there was any specification document accessible, ideally that (those) document(s) would describe two things : one would be the openldap engine -> backend interface the other the interface -> cache -> db4 rationnale. Thanks Lorenzo Pastrana R&D Head @ Happy End -------------------------- Web Shop Multimédia Design Visual Communication & Publishing -------------------------- Tél.: +33 1 42 47 83 09 Fax.: +33 1 47 70 70 19 E-mail : lorenzo.pastrana(a)happyend.fr

2 2

Re: RE24 testing call (2.4.17)
by Gavin Henry 08 Jun '09

08 Jun '09

2009/6/8 Quanah Gibson-Mount <quanah(a)zimbra.com>: > Please test RE24 as we prepare for 2.4.17. Thanks! All good here on i386, will test 64 bit shortly. Did get the expected: No race errors found after 10 iterations Found 2 errors >>>>>> Exiting with a false success status for now >>>>> ./scripts/test058-syncrepl-asymmetric completed OK. -- http://www.suretecsystems.com/services/openldap/ http://www.suretectelecom.com

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel June 2009