RE24 testing call (2.4.17)
by Quanah Gibson-Mount
Please test RE24 as we prepare for 2.4.17. Thanks!
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
13 years, 9 months
Re: commit: ldap/servers/slapd/overlays pcache.c
by Howard Chu
hyc(a)OpenLDAP.org wrote:
> Update of /repo/OpenLDAP/pkg/ldap/servers/slapd/overlays
>
> Modified Files:
> pcache.c 1.168 -> 1.169
>
> Log Message:
> Partially revert 1.32; pcache must allow other callbacks to see its results
Regarding the original reason for this change...
It's clear that what we really want is for pcache to behave the same in the
cached vs uncached case, wrt other overlays/callbacks. So wherever pcache
takes control in the call stack for the uncached case, it should do the same
in the cached case.
What's not clear is how to do that in general. When "response-callback head"
has been configured, it should just execute and leave the callback stack
intact when answering from cache. When "tail" (the default) is in effect then
it probably should bypass the stack. The above commit just leaves the stack
intact; I'll fix it up now to depend on the response-callback setting.
> * To: OpenLDAP Commit <openldap-commit2devel(a)OpenLDAP.org>
> * Subject: Re: commit: ldap/servers/slapd/overlays pcache.c
> * From: Pierangelo Masarati <ando(a)sys-net.it>
> * Date: Tue, 24 Aug 2004 10:40:00 +0200
> ando(a)OpenLDAP.org wrote:
>
>
> Update of /repo/OpenLDAP/pkg/ldap/servers/slapd/overlays
>
>
> Modified Files:
> pcache.c 1.31 -> 1.32
>
>
> Log Message:
> - proxy cache erroneously returns the filtering attributes
> and the objectClass right after caching, even if not requested,
> while subsequent searches are fine;
>
>
> I think this is a bug; it was also reflected in the test data, which has been
> fixed accordingly
>
> - the response callback needs be apended at the end of the
> callback list, otherwise the resulting entries are cached
>
>
> I ran into this issue while developing an overlay that dynamically builds
> entries; the raw entry was being cached instead of the completed one,
> because pcache adds its callback only when required, while that of my
> overlay is stacked by the overlay infrastructure. I think the "natural"
> behavior of pcache is to cache the final result that is sent to the user.
> I guess there might be applications where this is not true, so I made the
> original behavior configurable by an (undocumented) directive.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
13 years, 9 months
Re: held locks after exit
by Benoit Donnette
Well, you've highlighted 4 unreleased locks, a solution quite like using a
sledgehammer to crack a nut but predictably efficient would be to
backtrace() every lock and unlock so as to match them and to identify
which locks haven't been released (but your traces show enough detail to
help us focus the watch on specific db4 code).
If this can help.
> After an otherwise "clean" exit of test058, the smc/db directory showed
> the following, despite the fact that there was no more 9196 pid on the
> system. Is there any facility in db4 to trace locking?
>
> I don't think slapd's own debugging would give the level of detail that
> would be necessary to find a smoking gun, but if I'm just reading stuff
> wrong, I have the testrun directory from this.
>
> ---------- begin output ----------
>
> Default locking region information:
> 138 Last allocated locker ID
> 0x7fffffff Current maximum unused locker ID
> 9 Number of lock modes
> 1000 Maximum number of locks possible
> 1000 Maximum number of lockers possible
> 1000 Maximum number of lock objects possible
> 20 Number of lock object partitions
> 4 Number of current locks
> 12 Maximum number of locks at any one time
> 3 Maximum number of locks in any one bucket
> 0 Maximum number of locks stolen by for an empty partition
> 0 Maximum number of locks stolen for any one partition
> 9 Number of current lockers
> 15 Maximum number of lockers at any one time
> 4 Number of current lock objects
> 8 Maximum number of lock objects at any one time
> 2 Maximum number of lock objects in any one bucket
> 0 Maximum number of objects stolen by for an empty partition
> 0 Maximum number of objects stolen for any one partition
> 1009 Total number of locks requested
> 1005 Total number of locks released
> 0 Total number of locks upgraded
> 46 Total number of locks downgraded
> 4 Lock requests not available due to conflicts, for which we waited
> 0 Lock requests not available due to conflicts, for which we did not wait
> 0 Number of deadlocks
> 0 Lock timeout value
> 0 Number of locks that have timed out
> 0 Transaction timeout value
> 0 Number of transactions that have timed out
> 488KB The size of the lock region
> 4 The number of partition locks that required waiting (0%)
> 4 The maximum number of times any partition lock was waited for (0%)
> 0 The number of object queue operations that required waiting (0%)
> 1 The number of locker allocations that required waiting (0%)
> 0 The number of region locks that required waiting (0%)
> 2 Maximum hash bucket length
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Lock REGINFO information:
> Lock Region type
> 5 Region ID
> __db.005 Region name
> 0xfecf0000 Original region address
> 0xfecf0000 Region address
> 0xfecf00c8 Region primary address
> 0 Region maximum allocation
> 0 Region allocated
> Region allocations: 3006 allocations, 0 failures, 0 frees, 1 longest
> Allocations by power-of-two sizes:
> 1KB 3003
> 2KB 0
> 4KB 0
> 8KB 0
> 16KB 2
> 32KB 0
> 64KB 1
> 128KB 0
> 256KB 0
> 512KB 0
> 1024KB 0
> REGION_JOIN_OK Region flags
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Lock region parameters:
> 104 Lock region region mutex [0/119 0% 11775/1]
> 1031 locker table size
> 1031 object table size
> 616 obj_off
> 63456 locker_off
> 0 need_dd
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Lock conflict matrix:
> 0 0 0 0 0 0 0 0 0
> 0 0 1 0 1 0 1 0 1
> 0 1 1 1 1 1 1 1 1
> 0 0 0 0 0 0 0 0 0
> 0 1 1 0 0 0 0 1 1
> 0 0 1 0 0 0 0 0 1
> 0 1 1 0 0 0 0 1 1
> 0 0 1 0 1 0 1 0 0
> 0 1 1 0 1 1 1 0 1
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Locks grouped by lockers:
> Locker Mode Count Status ----------------- Object ---------------
> 85 dd= 0 locks held 1 write locks 0 pid/thread 9196/1
> 85 READ 1 HELD id2entry.bdb handle
> 0
> 86 dd= 0 locks held 0 write locks 0 pid/thread 9196/1
> 87 dd= 0 locks held 1 write locks 0 pid/thread 9196/1
> 87 READ 1 HELD dn2id.bdb handle
> 0
> 88 dd= 0 locks held 0 write locks 0 pid/thread 9196/1
> 89 dd= 0 locks held 0 write locks 0 pid/thread 9196/1
> 8a dd= 0 locks held 0 write locks 0 pid/thread 9196/1
> 8000013c dd= 0 locks held 0 write locks 0 pid/thread 9196/1
> 8000013d dd= 0 locks held 0 write locks 0 pid/thread 9196/1
> 80000145 dd= 0 locks held 2 write locks 2 pid/thread 9196/1
> 80000145 WRITE 1 HELD 0x39548 len: 5 data: 0000000x0100
> 80000145 WRITE 1 HELD id2entry.bdb page 1
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Locks grouped by object:
> Locker Mode Count Status ----------------- Object ---------------
> 80000145 WRITE 1 HELD 0x39548 len: 5 data: 0000000x0100
>
> 80000145 WRITE 1 HELD id2entry.bdb page 1
>
> 85 READ 1 HELD id2entry.bdb handle
> 0
>
> 87 READ 1 HELD dn2id.bdb handle
> 0
>
>
>
>
13 years, 9 months
Re: RE24 testing call (2.4.17)
by William Jojo
---- Original message ----
>Date: Mon, 08 Jun 2009 16:59:58 -0700
>From: Xin LI <delphij(a)delphij.net>
>Subject: Re: RE24 testing call (2.4.17)
>To: Quanah Gibson-Mount <quanah(a)zimbra.com>
>Cc: openldap-devel(a)openldap.org
>
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Quanah Gibson-Mount wrote:
>> Please test RE24 as we prepare for 2.4.17. Thanks!
>
>All regression tests gave 'completed OK' (test 058-syncrepl-asymmetric
>says No race errors found after 10 iterations but Found 2 errors) on
>FreeBSD/amd64.
>
Exact same for AIX 5.3.
Cheers,
Bill
>Cheers,
>- --
>Xin LI <delphij(a)delphij.net> http://www.delphij.net/
>FreeBSD - The Power to Serve!
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG v2.0.11 (FreeBSD)
>
>iEYEARECAAYFAkotpf4ACgkQi+vbBBjt66DRDwCgmJCG6IXHS62dBFbauXEuK9nG
>c7sAnAkKs/9rVwGA1v0Nv0W54tCeinA3
>=Haue
>-----END PGP SIGNATURE-----
13 years, 9 months
Re: RE24 testing call (2.4.17)
by Hallvard B Furuseth
./configure --disable-backends --disable-overlays --enable-bdb --enable-dds \
CC=gcc CFLAGS="-g -O0" CPPFLAGS="-DLDAP_MEMORY_DEBUG -DSLAP_NO_SL_MALLOC
...
./run test046-dds
...
Re-vitalizing the initial dynamic entry...
Re-renaming the subordinate dynamic entry (new superior)...
Deleting a dynamic entry...
Refreshing the remaining dynamic entry...
Waiting 15 seconds for remaining entry to expire...
./scripts/test046-dds: line 370: 16351 Segmentation fault (core dumped) $SLAPD -f $CONF1 -h $URI1 -d $LVL $TIMING > $LOG1 2>&1
gdb -q ../servers/slapd/slapd core.16351
#0 0x00000000004e45a7 in dds_expire (ctx=0x415e8d60, di=0x2a1e990) at dds.c:219
219 op->o_req_dn = de->de_ndn;
(gdb) bt
#0 0x00000000004e45a7 in dds_expire (ctx=0x415e8d60, di=0x2a1e990) at dds.c:219
#1 0x00000000004e4867 in dds_expire_fn (ctx=0x415e8d60, arg=0x2b97c70) at dds.c:274
#2 0x00000000004fff10 in ldap_int_thread_pool_wrapper (xpool=0x29f0a30) at tpool.c:663
#3 0x000000350ee06367 in start_thread () from /lib64/libpthread.so.0
#4 0x000000350e2d30ad in clone () from /lib64/libc.so.6
(gdb) p de
$1 = (dds_expire_t *) 0xffffffffffffffff
It only breaks if both of -DLDAP_MEMORY_DEBUG -DSLAP_NO_SL_MALLOC
are given, then it crashes the same way every time.
x86_64, Red Hat Enterprise Linux 5.3.
C library: /lib64/libc.so.6 -> libc-2.5.so.
--
Hallvard
13 years, 9 months
Beware of Fedora 11's glibc
by Howard Chu
From this bug it appears there's a known history of problems with the
experimental malloc being used in the new glibc.
https://bugzilla.redhat.com/show_bug.cgi?id=494631
It appears that slapd is hitting some issues in this code still, even though
the other symptoms in that bug report are marked as fixed. LD_PRELOADing a 3rd
party malloc library shows no issues and likewise valgrind finds no invalid
memory accesses in our code. I was seeing crashes on x86_64; the 32 bit code
may actually be fixed by now.
Just an fyi. I've already spent more than 2 weeks looking for a memory
overwrite in slapd only to find that the malloc library is broken. I can't
afford to spend any more time tracking this down.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
13 years, 9 months
held locks after exit
by Aaron Richton
After an otherwise "clean" exit of test058, the smc/db directory showed
the following, despite the fact that there was no more 9196 pid on the
system. Is there any facility in db4 to trace locking?
I don't think slapd's own debugging would give the level of detail that
would be necessary to find a smoking gun, but if I'm just reading stuff
wrong, I have the testrun directory from this.
---------- begin output ----------
Default locking region information:
138 Last allocated locker ID
0x7fffffff Current maximum unused locker ID
9 Number of lock modes
1000 Maximum number of locks possible
1000 Maximum number of lockers possible
1000 Maximum number of lock objects possible
20 Number of lock object partitions
4 Number of current locks
12 Maximum number of locks at any one time
3 Maximum number of locks in any one bucket
0 Maximum number of locks stolen by for an empty partition
0 Maximum number of locks stolen for any one partition
9 Number of current lockers
15 Maximum number of lockers at any one time
4 Number of current lock objects
8 Maximum number of lock objects at any one time
2 Maximum number of lock objects in any one bucket
0 Maximum number of objects stolen by for an empty partition
0 Maximum number of objects stolen for any one partition
1009 Total number of locks requested
1005 Total number of locks released
0 Total number of locks upgraded
46 Total number of locks downgraded
4 Lock requests not available due to conflicts, for which we waited
0 Lock requests not available due to conflicts, for which we did not wait
0 Number of deadlocks
0 Lock timeout value
0 Number of locks that have timed out
0 Transaction timeout value
0 Number of transactions that have timed out
488KB The size of the lock region
4 The number of partition locks that required waiting (0%)
4 The maximum number of times any partition lock was waited for (0%)
0 The number of object queue operations that required waiting (0%)
1 The number of locker allocations that required waiting (0%)
0 The number of region locks that required waiting (0%)
2 Maximum hash bucket length
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock REGINFO information:
Lock Region type
5 Region ID
__db.005 Region name
0xfecf0000 Original region address
0xfecf0000 Region address
0xfecf00c8 Region primary address
0 Region maximum allocation
0 Region allocated
Region allocations: 3006 allocations, 0 failures, 0 frees, 1 longest
Allocations by power-of-two sizes:
1KB 3003
2KB 0
4KB 0
8KB 0
16KB 2
32KB 0
64KB 1
128KB 0
256KB 0
512KB 0
1024KB 0
REGION_JOIN_OK Region flags
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock region parameters:
104 Lock region region mutex [0/119 0% 11775/1]
1031 locker table size
1031 object table size
616 obj_off
63456 locker_off
0 need_dd
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock conflict matrix:
0 0 0 0 0 0 0 0 0
0 0 1 0 1 0 1 0 1
0 1 1 1 1 1 1 1 1
0 0 0 0 0 0 0 0 0
0 1 1 0 0 0 0 1 1
0 0 1 0 0 0 0 0 1
0 1 1 0 0 0 0 1 1
0 0 1 0 1 0 1 0 0
0 1 1 0 1 1 1 0 1
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by lockers:
Locker Mode Count Status ----------------- Object ---------------
85 dd= 0 locks held 1 write locks 0 pid/thread 9196/1
85 READ 1 HELD id2entry.bdb handle 0
86 dd= 0 locks held 0 write locks 0 pid/thread 9196/1
87 dd= 0 locks held 1 write locks 0 pid/thread 9196/1
87 READ 1 HELD dn2id.bdb handle 0
88 dd= 0 locks held 0 write locks 0 pid/thread 9196/1
89 dd= 0 locks held 0 write locks 0 pid/thread 9196/1
8a dd= 0 locks held 0 write locks 0 pid/thread 9196/1
8000013c dd= 0 locks held 0 write locks 0 pid/thread 9196/1
8000013d dd= 0 locks held 0 write locks 0 pid/thread 9196/1
80000145 dd= 0 locks held 2 write locks 2 pid/thread 9196/1
80000145 WRITE 1 HELD 0x39548 len: 5 data: 0000000x0100
80000145 WRITE 1 HELD id2entry.bdb page 1
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by object:
Locker Mode Count Status ----------------- Object ---------------
80000145 WRITE 1 HELD 0x39548 len: 5 data: 0000000x0100
80000145 WRITE 1 HELD id2entry.bdb page 1
85 READ 1 HELD id2entry.bdb handle 0
87 READ 1 HELD dn2id.bdb handle 0
13 years, 9 months
BDB / HDB Implementaion Internals
by Lorenzo Pastrana
Hello,
I'm diving into the BDB backend code, and I was wondering if there was
any specification document accessible, ideally that (those) document(s)
would describe two things : one would be the openldap engine -> backend
interface the other the interface -> cache -> db4 rationnale.
Thanks
Lorenzo Pastrana
R&D Head @ Happy End
--------------------------
Web Shop
Multimédia Design
Visual Communication & Publishing
--------------------------
Tél.: +33 1 42 47 83 09
Fax.: +33 1 47 70 70 19
E-mail : lorenzo.pastrana(a)happyend.fr
13 years, 9 months
