Re: commit: ldap/servers/slapd/back-bdb add.c back-bdb.h bind.c cache.c compare.c config.c delete.c dn2entry.c dn2id.c filterindex.c id2entry.c idl.c init.c key.c modify.c modrdn.c proto-bdb.h referral.c search.c tools.c
hyc@OpenLDAP.org wrote:
Update of /repo/OpenLDAP/pkg/ldap/servers/slapd/back-bdb
Modified Files: add.c 1.173 -> 1.174 back-bdb.h 1.173 -> 1.174 bind.c 1.54 -> 1.55 cache.c 1.180 -> 1.181 compare.c 1.57 -> 1.58 config.c 1.107 -> 1.108 delete.c 1.172 -> 1.173 dn2entry.c 1.35 -> 1.36 dn2id.c 1.156 -> 1.157 filterindex.c 1.75 -> 1.76 id2entry.c 1.86 -> 1.87 idl.c 1.134 -> 1.135 init.c 1.291 -> 1.292 key.c 1.24 -> 1.25 modify.c 1.176 -> 1.177 modrdn.c 1.200 -> 1.201 proto-bdb.h 1.159 -> 1.160 referral.c 1.50 -> 1.51 search.c 1.265 -> 1.266 tools.c 1.127 -> 1.128
Log Message: Use read-only txn's instead of read lockers. Support BDB 4.4-4.7
Please test... At the moment I've only run it with BDB 4.7.25. I'll be testing with older releases over the next couple of days. In particular, take note of any LDAP_OTHER/Internal Error results in the test suite. Right now it passes all tests for me on bdb and hdb.
If push comes to shove we can probably create a new BerkeleyDB4.2.patch file to get 4.2 working again, but at this point most people should already be on something newer, and people stuck on 4.2 probably won't compile their own anyway.
Howard Chu wrote:
Please test... At the moment I've only run it with BDB 4.7.25. I'll be testing with older releases over the next couple of days. In particular, take note of any LDAP_OTHER/Internal Error results in the test suite. Right now it passes all tests for me on bdb and hdb.
I'm occasionally seeing the read-only transaction getting selected for deadlock recovery, which makes the transaction unusable after that point. I'm looking into how to configure the environment so that the reader transactions never get selected for deadlock recovery, otherwise we need to add a lot more abort/recovery code.
While investigating this, I noticed another problem - currently whenever the search code gets a LOCK_DEADLOCK or LOCK_NOTGRANTED result, it just immediately retries. That's only correct when the search is a standalone operation. But, the search can also be invoked during ACL processing of a write operation if an ACL specifies some group or other complex condition. We've correctly propagated the write's TXN into the search using the bdb_opinfo, but if we're acting in this situation and get a DEADLOCK, we need to give up and return back to the caller.
Howard Chu wrote:
Howard Chu wrote:
Please test... At the moment I've only run it with BDB 4.7.25. I'll be testing with older releases over the next couple of days. In particular, take note of any LDAP_OTHER/Internal Error results in the test suite. Right now it passes all tests for me on bdb and hdb.
I'm occasionally seeing the read-only transaction getting selected for deadlock recovery, which makes the transaction unusable after that point. I'm looking into how to configure the environment so that the reader transactions never get selected for deadlock recovery, otherwise we need to add a lot more abort/recovery code.
My approach for that was to set the NOWAIT flag on the read txns, but that eats up too much CPU because most of our code immediately retries whenever it sees the DB_LOCK_NOTGRANTED result. I've removed the NOWAIT flag again. Still looking at the impact of this change.
While investigating this, I noticed another problem - currently whenever the search code gets a LOCK_DEADLOCK or LOCK_NOTGRANTED result, it just immediately retries. That's only correct when the search is a standalone operation. But, the search can also be invoked during ACL processing of a write operation if an ACL specifies some group or other complex condition. We've correctly propagated the write's TXN into the search using the bdb_opinfo, but if we're acting in this situation and get a DEADLOCK, we need to give up and return back to the caller.
This is now fixed in search.c. The most obvious case where it would cause problems is evaluating an ACL set during a write operation; if the set caused a search that hit a BDB deadlock, the search would retry once and get an internal error on the retry, causing the ACL evaluation to fail. Whether that caused the write op to be erroneously allowed or denied really depends on how the set was defined... (For typical ACL group comparisons, we were already doing the correct error recovery.)
Howard Chu writes:
My approach for that was to set the NOWAIT flag on the read txns, but that eats up too much CPU because most of our code immediately retries whenever it sees the DB_LOCK_NOTGRANTED result. I've removed the NOWAIT flag again. Still looking at the impact of this change.
Is this the kind of situation where it can help to retry at increasing and somewhat randomized intervals until it succeeds? (Randomized so two threads won't keep getting in each others' way.)
Hallvard B Furuseth wrote:
Howard Chu writes:
My approach for that was to set the NOWAIT flag on the read txns, but that eats up too much CPU because most of our code immediately retries whenever it sees the DB_LOCK_NOTGRANTED result. I've removed the NOWAIT flag again. Still looking at the impact of this change.
Is this the kind of situation where it can help to retry at increasing and somewhat randomized intervals until it succeeds? (Randomized so two threads won't keep getting in each others' way.)
Perhaps. We already have a backoff function in here, I guess it wouldn't hurt to use it. Until now, I've never actually seen retries occur on read operations...
-
Hallvard B Furuseth -
Howard Chu