Tested in HEAD again, on same machine with same cmd:
for i in `seq 1 100`; do ./run test020 >> test_results.log; done;
and grepped the log file for "Error" and -i "succeed".
Results:
100 Passes
0 Fails
Looks good to me.
Gavin.
--
Kind Regards,
Gavin Henry.
Managing Director.
T +44 (0) 1224 279484
M +44 (0) 7930 323266
F +44 (0) 1224 824887
E ghenry(a)suretecsystems.com
Open Source. Open Solutions(tm).
http://www.suretecsystems.com/
rhafer(a)suse.de wrote:
> On Donnerstag, 8. November 2007, you wrote:
>> <quote who="Ralf Haferkamp">
>>
>>> On Donnerstag, 8. November 2007, rhafer(a)suse.de wrote:
>>>> I think I found a way.
>>>> 1. The SLAPD_ABANDON/op->o_abandon part of the cleanup handler will now
>>>> only get executed if the final result for the search has not been
>>>> received
>>>> (caching_reason == PC_IGNORE).
>>>> 2. Additionally, to add the entries to the cache, I do no longer use the
>>>> original Connection Object but create a new one
>>>> (connection_fake_init()),
>>>> because the op->o_abandon will be set for the original connection as
>>>> soon
>>>> as the client closes it.
>>>>
>>>> I'll do some more testing locally an submit this to HEAD later for more
>>>> external testing.
>>> Fix submitted to HEAD. Please test (preferably on the machine that caused
>>> this
>>> test to fail most often ;) ).
>> I'll try this tonight.
>>
>> Just to clarify, this is a fix in the code, rather than the test, which I
>> thought Howard had changed?
> Yes, it is a fix in the pcache code. The fix in the testcase was only done
> temporarely in the RE24 branch (HEAD was not changed). I intend to remove it
> as soon as I get positive feedback.
>
Trying to test in HEAD, can't cant past test008-concurrency on bdb tests.
Using ldapsearch to retrieve all the entries...
Filtering ldapsearch results...
Filtering original ldif used to create database...
Comparing filter output...
comparison failed - database was not created correctly
>>>>> ./scripts/test008-concurrency failed (exit 1)
make[2]: *** [bdb-mod] Error 1
make[2]: Leaving directory `/anything/src/openldap/ldap/tests'
make[1]: *** [test] Error 2
make[1]: Leaving directory `/anything/src/openldap/ldap/tests'
make: *** [test] Error 2
testrun data:
ftp://ftp.openldap.org/incoming/test008.tar.gz
--
Kind Regards,
Gavin Henry.
OpenLDAP Engineering Team.
E ghenry(a)OpenLDAP.org
Community developed LDAP software.
http://www.openldap.org/project/
ghenry(a)OpenLDAP.org wrote:
> dhawes(a)vt.edu wrote:
>> On Friday 27 July 2007 05:20, ghenry(a)openldap.org wrote:
>>> Hi David,
>>>
>>> http://www.openldap.org/devel/cvsweb.cgi/contrib/slapd-modules/addpartial/
>>>
>>> Will be pushed out in the next 2.4 beta.
>>>
>>> Please bear in mind that you will be looking after this overlay, so test
>>> as often as you can and re-submit and changes during our release cycle.
>>>
>>> Thank you for your contribution.
>> Changes to work properly with 2.4.6 are at:
>>
>> ftp://ftp.openldap.org/incoming/david_hawes-addpartial-071109.tgz
>>
>> Thanks,
>>
>> dave
>>
>>
>
> Will take a look over the next week.
>
> Thanks.
>
Updated in RE24 and HEAD.
--
Kind Regards,
Gavin Henry.
OpenLDAP Engineering Team.
E ghenry(a)OpenLDAP.org
Community developed LDAP software.
http://www.openldap.org/project/
openldap2007(a)mnagl.de wrote:
> Full_Name: Matthias Nagl
> Version:
> OS: Linux
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (137.248.132.104)
> The current stable version of mit-krb5 (http://web.mit.edu/Kerberos/) seems to
> have a much better support for LDAP-Backends than Heimdal. Sadly the
> smbk5pwd-overlay currently won't support password synchronization with the new
> MIT-schema. It would be great if smbk5pwd could be extended to work with the new
> mit-krb5.
You're welcome to submit a patch to provide the necessary support.
I'll note that the MIT schema is deficient in a number of areas too; we're
looking at writing up an IETF Draft defining a more comprehensive schema that
can be used by both MIT and Heimdal going forward.
As a total aside, the MIT code's stability leaves a lot to be desired. I won't
deploy it on any of my networks because I've seen it crash too many times. In
contrast, I've deployed Heimdal at numerous sites and never had to fuss with
it, it just works. Your Mileage May Vary, just relating my personal experience
accumulated over several years.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
Full_Name: Matthias Nagl
Version:
OS: Linux
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (137.248.132.104)
The current stable version of mit-krb5 (http://web.mit.edu/Kerberos/) seems to
have a much better support for LDAP-Backends than Heimdal. Sadly the
smbk5pwd-overlay currently won't support password synchronization with the new
MIT-schema. It would be great if smbk5pwd could be extended to work with the new
mit-krb5.
<quote who="ghenry(a)OpenLDAP.org">
> <quote who="Sam Varshavchik">
>> No problem.
>>
>> Here are the fixes diff-ed against CVS HEAD:
>>
>> http://www.tldp.org/manpages/openldap-2.4.6-manpatch.txt
>>
Now patched in HEAD.
Gavin.
Full_Name: Sean Burford
Version: 2.3.32
OS: Linux
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (65.57.245.11)
Searches that span periods of BDB lock exhaustion may return truncated results
with a success error code. It should return an err=80 failure.
This was with a BDB 4.4.20 backend.
In the log below you can see:
a search starts at 11:00:01
the search returns success:240 entries at 11:00:03
a search starts at 11:15:01
bdb runs out of locks at 11:15:02, informs the mod operation that it failed
the search returns success:128 entries at 11:15:02
Both searches were identical.
Nov 9 11:00:01 conn=71131 op=2 SRCH base="dc=example,dc=com" scope=2 deref=0
filter="(objectClass=exampleClass)"
Nov 9 11:00:01 conn=71131 op=2 SRCH attr=* +
Nov 9 11:00:03 conn=71131 op=2 SEARCH RESULT tag=101 err=0 nentries=240 text=
...
Nov 9 11:15:01 conn=72711 op=2 SRCH base="dc=example,dc=com" scope=2 deref=0
filter="(objectClass=exampleClass)"
Nov 9 11:15:01 conn=72711 op=2 SRCH attr=* +
...
Nov 9 11:15:02 bdb(dc=example,dc=com): Lock table is out of available locks
Nov 9 11:15:02 => bdb_idl_delete_key: c_get id failed: Cannot allocate memory
(12)
Nov 9 11:15:02 bdb(dc=example,dc=com): Lock table is out of available locks
Nov 9 11:15:02 bdb(dc=example,dc=com): Lock table is out of available locks
Nov 9 11:15:02 bdb(dc=example,dc=com): Lock table is out of available locks
...
Nov 9 11:15:02 Attribute index delete failure
Nov 9 11:15:02 bdb(dc=example,dc=com): Lock table is out of available locks
Nov 9 11:15:02 bdb(dc=example,dc=com): Lock table is out of available locks
Nov 9 11:15:02 bdb(dc=example,dc=com): Lock table is out of available locks
Nov 9 11:15:02 conn=72642 op=10 RESULT tag=103 err=80 text=
Nov 9 11:15:02 conn=72131 op=144 RESULT tag=103 err=80 text=internal error
...
Nov 9 11:15:02 conn=72711 op=2 SEARCH RESULT tag=101 err=0 nentries=128 text=
Nov 9 11:15:05 conn=72711 op=3 UNBIND
Nov 9 11:15:05 conn=72711 fd=57 closed ()
The database definition is:
database bdb
suffix "dc=example,dc=com"
directory /var/lib/ldap
overlay auditlog
auditlog /var/lib/ldap/ldif/auditlog/audit.com.ldif
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 1000
overlay accesslog
logdb cn=accesslog
logops writes
logsuccess TRUE
logpurge 02+23:46 01+23:46
# This limits section applies to the user this bug is about...
limits dn.exact="uid=user1,ou=people,dc=example,dc=com"
time.soft=unlimited time.hard=unlimited size.soft=unlimited
size.hard=unlimited
cachesize 100000
idlcachesize 10000
sizelimit 200000
checkpoint 512 1
lastmod on
idletimeout 300
threads 64
mån 2007-11-12 klockan 23:17 +0000 skrev quanah(a)zimbra.com:
> --On Monday, November 12, 2007 7:02 AM +0000 hyc(a)symas.com wrote:
>
> > Dan.Oscarsson(a)tietoenator.com wrote:
> >> Full_Name: Dan Oscarsson
> >> Version: 2.3.32
> >> OS: SLES 10
> >> URL: ftp://ftp.openldap.org/incoming/
> >> Submission from: (NULL) (193.15.240.60)
> >
> Also, Just some general data on what it is you are doing that is a bit more
> explanative. For example, what does your tree layout look like? A single
> root with 20,000+ subtrees off of the root? A root with 10 subtrees, with
> thousands of subtrees off of those? How are you doing these modrdn's?
> Why, exactly? Anything that can help us to possibly come up with a
> progamatic generation of dummy data.
The tree is based on a company organisation structure, branching at each
organisational level util you get to a person. So the root has only a
few subtrees, and each subtree only a few (say 1-30) subtrees. Most
entries will probably a subtree for a unit with the persons beloning to
that unit. People will always be leafs but an entry may contain both
people and subtrees as children.
I have triggered the bug when doing major reorganisation of people. Then
very many of the people are moved around to a new place in the tree.
Many to newly created subtrees.
I have now used 2.3.38 compiled myself so I can add tracing to the code
if needed. Running a move of people, with logging set to any,
and then running my simple check program, the search at a subtree that
goes wrong (entries are missing) is logged as:
filter: (&(objectClass=enterprisePerson)(ba=telecom & media)(bu=telecom
solutions))
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: search_candidates: base=\"bu=telecom solutions,ba=telecom & media,cn=organisation,o=xx\" (0x0000513a) scope=2
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => hdb_dn2idl(\"bu=telecom solutions,ba=telecom & media,cn=organisation,o=xx\")
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => bdb_filter_candidates
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: AND
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => bdb_list_candidates 0xa0
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => bdb_filter_candidates
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: OR
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => bdb_list_candidates 0xa1
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => bdb_filter_candidates
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: EQUALITY
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => bdb_equality_candidates (objectClass)
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => key_read
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: bdb_idl_fetch_key: [b49d1940]
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_index_read: failed (-30989)
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_equality_candidates: id=0, first=0, last=0
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_filter_candidates: id=0 first=0 last=0
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => bdb_filter_candidates
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: AND
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => bdb_list_candidates 0xa0
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => bdb_filter_candidates
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: EQUALITY
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => bdb_equality_candidates (objectClass)
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => key_read
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: bdb_idl_fetch_key: [a98323a6]
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_index_read 20569 candidates
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_equality_candidates: id=20569, first=1483, last=26650
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_filter_candidates: id=20569 first=1483 last=26650
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => bdb_filter_candidates
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: EQUALITY
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => bdb_equality_candidates (ba)
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => key_read
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: bdb_idl_fetch_key: [410ca247]
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_index_read 6991 candidates
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_equality_candidates: id=6991, first=15620, last=26644
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_filter_candidates: id=6991 first=15620 last=26644
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => bdb_filter_candidates
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: EQUALITY
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => bdb_equality_candidates (bu)
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: => key_read
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: bdb_idl_fetch_key: [8b819570]
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_index_read 2195 candidates
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_equality_candidates: id=2195, first=30, last=26614
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_filter_candidates: id=2195 first=30 last=26614
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_list_candidates: id=2144 first=20813 last=26614
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_filter_candidates: id=2144 first=20813 last=26614
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_list_candidates: id=2144 first=20813 last=26614
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_filter_candidates: id=2144 first=20813 last=26614
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_list_candidates: id=2130 first=20813 last=26614
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: <= bdb_filter_candidates: id=2130 first=20813 last=26614
2007-11-13 15.19.05 ra [local4.debug] slapd[27292]: bdb_search_candidates: id=2130 first=20813 last=26614
Stopping and starting the server. The check now works. The above search is logged the same
except for the last part:
2007-11-13 15.19.41 ra [local4.debug] slapd[27460]: bdb_idl_fetch_key: [8b819570]
2007-11-13 15.19.41 ra [local4.debug] slapd[27460]: <= bdb_index_read 2195 candidates
2007-11-13 15.19.41 ra [local4.debug] slapd[27460]: <= bdb_equality_candidates: id=2195, first=30, last=26614
2007-11-13 15.19.41 ra [local4.debug] slapd[27460]: <= bdb_filter_candidates: id=2195 first=30 last=26614
2007-11-13 15.19.41 ra [local4.debug] slapd[27460]: <= bdb_list_candidates: id=2144 first=20813 last=26614
2007-11-13 15.19.41 ra [local4.debug] slapd[27460]: <= bdb_filter_candidates: id=2144 first=20813 last=26614
2007-11-13 15.19.41 ra [local4.debug] slapd[27460]: <= bdb_list_candidates: id=2144 first=20813 last=26614
2007-11-13 15.19.41 ra [local4.debug] slapd[27460]: <= bdb_filter_candidates: id=2144 first=20813 last=26614
2007-11-13 15.19.41 ra [local4.debug] slapd[27460]: <= bdb_list_candidates: id=2141 first=20813 last=26614
2007-11-13 15.19.41 ra [local4.debug] slapd[27460]: <= bdb_filter_candidates: id=2141 first=20813 last=26614
2007-11-13 15.19.41 ra [local4.debug] slapd[27460]: bdb_search_candidates: id=2141 first=20813 last=26614
In both above cases, with the same filter but search base
set to o=xx witch is the database prefix, it works as it should.
In this case the entries that were missing above is found.
I do not know it this might give you some clue to what the problem is.
Any suggestions on what I should look for or any suitable place
in the code where I could add debug logging, would be nice.
The logs get very large when running with logging set to any.
Dan
--
Dan Oscarsson
TietoEnator Email: Dan.Oscarsson(a)tietoenator.com
Box 85
201 20 Malmo, Sweden
--On Monday, November 12, 2007 7:02 AM +0000 hyc(a)symas.com wrote:
> Dan.Oscarsson(a)tietoenator.com wrote:
>> Full_Name: Dan Oscarsson
>> Version: 2.3.32
>> OS: SLES 10
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (193.15.240.60)
>
>> From this I suspect that it is the cache of which parent node belongs to
>> that gets corrupted. Or can it be something else?
>> What more could I do to trace down the bug?
>> I do not know if the above information is enough for you to find what is
>> wrong? Cannot include data as it contains company internal information
>> and a simple test program did not give the same error.
>> I have looked at the code, but it takes some time to understand it when
>> doing it for the first time. maybe som debugging code could be added
>> find the place where the bug is.
>
> This isn't a lot of information to go on. If you can create a test
> program that shows the problem occurring, using dummy data, that would
> help. --
Also, Just some general data on what it is you are doing that is a bit more
explanative. For example, what does your tree layout look like? A single
root with 20,000+ subtrees off of the root? A root with 10 subtrees, with
thousands of subtrees off of those? How are you doing these modrdn's?
Why, exactly? Anything that can help us to possibly come up with a
progamatic generation of dummy data.
And, you said you had a script to do this, can you include that?
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration