Re: commit: ldap/servers/slapd/back-bdb add.c back-bdb.h bind.c cache.c compare.c config.c delete.c dn2entry.c dn2id.c filterindex.c id2entry.c idl.c init.c key.c modify.c modrdn.c proto-bdb.h referral.c search.c tools.c
4:49 p.m.
hyc(a)OpenLDAP.org wrote:
Update of /repo/OpenLDAP/pkg/ldap/servers/slapd/back-bdb
Modified Files:
add.c 1.173 -> 1.174
back-bdb.h 1.173 -> 1.174
bind.c 1.54 -> 1.55
cache.c 1.180 -> 1.181
compare.c 1.57 -> 1.58
config.c 1.107 -> 1.108
delete.c 1.172 -> 1.173
dn2entry.c 1.35 -> 1.36
dn2id.c 1.156 -> 1.157
filterindex.c 1.75 -> 1.76
id2entry.c 1.86 -> 1.87
idl.c 1.134 -> 1.135
init.c 1.291 -> 1.292
key.c 1.24 -> 1.25
modify.c 1.176 -> 1.177
modrdn.c 1.200 -> 1.201
proto-bdb.h 1.159 -> 1.160
referral.c 1.50 -> 1.51
search.c 1.265 -> 1.266
tools.c 1.127 -> 1.128
Log Message:
Use read-only txn's instead of read lockers. Support BDB 4.4-4.7
Please test... At the moment I've only run it with BDB 4.7.25. I'll be testing
with older releases over the next couple of days. In particular, take note of
any LDAP_OTHER/Internal Error results in the test suite. Right now it passes
all tests for me on bdb and hdb.
If push comes to shove we can probably create a new BerkeleyDB4.2.patch file
to get 4.2 working again, but at this point most people should already be on
something newer, and people stuck on 4.2 probably won't compile their own anyway.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
1:06 p.m.
New subject: commit: ldap/servers/slapd/back-bdb add.c back-bdb.h bind.c cache.c compare.c config.c delete.c dn2entry.c dn2id.c filterindex.c id2entry.c idl.c init.c key.c modify.c modrdn.c proto-bdb.h referral.c search.c tools.c
Howard Chu wrote:
Please test... At the moment I've only run it with BDB 4.7.25.
I'll be testing
with older releases over the next couple of days. In particular, take note of
any LDAP_OTHER/Internal Error results in the test suite. Right now it passes
all tests for me on bdb and hdb.
I'm occasionally seeing the read-only transaction getting selected for
deadlock recovery, which makes the transaction unusable after that point. I'm
looking into how to configure the environment so that the reader transactions
never get selected for deadlock recovery, otherwise we need to add a lot more
abort/recovery code.
While investigating this, I noticed another problem - currently whenever the
search code gets a LOCK_DEADLOCK or LOCK_NOTGRANTED result, it just
immediately retries. That's only correct when the search is a standalone
operation. But, the search can also be invoked during ACL processing of a
write operation if an ACL specifies some group or other complex condition.
We've correctly propagated the write's TXN into the search using the
bdb_opinfo, but if we're acting in this situation and get a DEADLOCK, we need
to give up and return back to the caller.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
4:58 a.m.
New subject: commit: ldap/servers/slapd/back-bdb add.c back-bdb.h bind.c cache.c compare.c config.c delete.c dn2entry.c dn2id.c filterindex.c id2entry.c idl.c init.c key.c modify.c modrdn.c proto-bdb.h referral.c search.c tools.c
Howard Chu wrote:
Howard Chu wrote:
> Please test... At the moment I've only run it with BDB 4.7.25. I'll be
testing
> with older releases over the next couple of days. In particular, take note of
> any LDAP_OTHER/Internal Error results in the test suite. Right now it passes
> all tests for me on bdb and hdb.
I'm occasionally seeing the read-only transaction getting selected for
deadlock recovery, which makes the transaction unusable after that point. I'm
looking into how to configure the environment so that the reader transactions
never get selected for deadlock recovery, otherwise we need to add a lot more
abort/recovery code.
My approach for that was to set the NOWAIT flag on the read txns, but that
eats up too much CPU because most of our code immediately retries whenever it
sees the DB_LOCK_NOTGRANTED result. I've removed the NOWAIT flag again. Still
looking at the impact of this change.
While investigating this, I noticed another problem - currently
whenever the
search code gets a LOCK_DEADLOCK or LOCK_NOTGRANTED result, it just
immediately retries. That's only correct when the search is a standalone
operation. But, the search can also be invoked during ACL processing of a
write operation if an ACL specifies some group or other complex condition.
We've correctly propagated the write's TXN into the search using the
bdb_opinfo, but if we're acting in this situation and get a DEADLOCK, we need
to give up and return back to the caller.
This is now fixed in search.c. The most obvious case where it would cause
problems is evaluating an ACL set during a write operation; if the set caused
a search that hit a BDB deadlock, the search would retry once and get an
internal error on the retry, causing the ACL evaluation to fail. Whether that
caused the write op to be erroneously allowed or denied really depends on how
the set was defined... (For typical ACL group comparisons, we were already
doing the correct error recovery.)
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
5:23 a.m.
New subject: commit: ldap/servers/slapd/back-bdb add.c back-bdb.h bind.c cache.c compare.c config.c delete.c dn2entry.c dn2id.c filterindex.c id2entry.c idl.c init.c key.c modify.c modrdn.c proto-bdb.h referral.c search.c tools.c
Howard Chu writes:
My approach for that was to set the NOWAIT flag on the read txns,
but
that eats up too much CPU because most of our code immediately retries
whenever it sees the DB_LOCK_NOTGRANTED result. I've removed the
NOWAIT flag again. Still looking at the impact of this change.
Is this the kind of situation where it can help to retry at increasing
and somewhat randomized intervals until it succeeds?
(Randomized so two threads won't keep getting in each others' way.)
--
Hallvard
5:35 a.m.
New subject: commit: ldap/servers/slapd/back-bdb add.c back-bdb.h bind.c cache.c compare.c config.c delete.c dn2entry.c dn2id.c filterindex.c id2entry.c idl.c init.c key.c modify.c modrdn.c proto-bdb.h referral.c search.c tools.c
Hallvard B Furuseth wrote:
Howard Chu writes:
> My approach for that was to set the NOWAIT flag on the read txns, but
> that eats up too much CPU because most of our code immediately retries
> whenever it sees the DB_LOCK_NOTGRANTED result. I've removed the
> NOWAIT flag again. Still looking at the impact of this change.
Is this the kind of situation where it can help to retry at increasing
and somewhat randomized intervals until it succeeds?
(Randomized so two threads won't keep getting in each others' way.)
Perhaps. We already have a backoff function in here, I guess it wouldn't hurt
to use it. Until now, I've never actually seen retries occur on read operations...
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
5298
days inactive
5328
days old
4 comments
2 participants
participants (2)
-
Hallvard B Furuseth -
Howard Chu