Just thinking aloud right now about what's needed here.
Basically all incoming requests participating in a transaction are queued until the Commit is received. Aside from basic parsing/validation, no other processing is performed until then.
If an Abort is received, the queued requests are simply discarded.
When the Commit request is received, all of the queued requests are performed sequentially by a single thread and the results are gathered. If any individual request fails, any preceding completed requests must be rolled back.
Since I'm considering this for both back-bdb and back-ndb, it appears we're going to need some transaction-specific hooks to be added to the BackendInfo structure. 1) start a TXN, get a TXN handle 2) end a TXN: Commit or Abort
For back-bdb the tricky part is exposing the updates atomically in the caches. I think the fact that entry caching uses BDB locks helps somewhat; we can keep entry cache items locked until the overall transaction completes. But for Aborts we'll either need to keep a copy of the previous cache info, or just discard it all. For now, discarding it all seems simpler.
For back-ndb things are currently easy since this backend does no caching of its own. As such, once the backend issues a Commit or Abort request, there's no further (backend) work to be done.
It's tempting to think about this for backglue, but we'd need a cross-database lock manager of some kind for detecting deadlocks. That implies that we really need an LDAP-level lock request, to handle distributed locking, and that the Transaction handling ought to be built on top of that. Currently the Transaction layer says nothing at all about locking, and it's up to the underying LDAP database to take care of it.
I guess another approach would just be to have backglue fully serialize all transactions; if only one is outstanding at any time there can be no deadlocks.
This brings up a question about whether slapd in general should fully serialize them. I was thinking, at the least, that we should only allow one active transaction per connection, though that was mainly a matter of convenience. Thoughts?
--On Wednesday, April 23, 2008 4:00 PM -0700 Howard Chu hyc@symas.com wrote:
I guess another approach would just be to have backglue fully serialize all transactions; if only one is outstanding at any time there can be no deadlocks.
This brings up a question about whether slapd in general should fully serialize them. I was thinking, at the least, that we should only allow one active transaction per connection, though that was mainly a matter of convenience. Thoughts?
Syncrepl was untenable with non-serialized updates (which moving to delta-syncrepl helped with). Even in a standalone setup, I find that serializing the update process provides faster write times (at least with back-(bh)db) than allowing multiple update threads.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On Donnerstag, 24. April 2008, Howard Chu wrote: [..]
It's tempting to think about this for backglue, but we'd need a cross-database lock manager of some kind for detecting deadlocks. That implies that we really need an LDAP-level lock request, to handle distributed locking, and that the Transaction handling ought to be built on top of that. Currently the Transaction layer says nothing at all about locking, and it's up to the underying LDAP database to take care of it.
It's not only tempting for backglue, I guess. It's also interesting for other multiple database setups. E.g. the setup that Samba4 uses currently. If I understand you correctly, slapo-memberof and -refint could be implemented to (optionally) work transaction based across multiple databases then. On the other hand, ignoring the multiple database setup for transactions for now seems to be quite a lot easier to implement and might the a good thing to do as a first step to see where we can go from there. Though I am probably overestimating the amount of work needed to get transactions working in a backglue config.
I guess another approach would just be to have backglue fully serialize all transactions; if only one is outstanding at any time there can be no deadlocks.
This brings up a question about whether slapd in general should fully serialize them. I was thinking, at the least, that we should only allow one active transaction per connection,
I guess you mean LDAP-level transactions here, not transactions on the backenddb-level? Yeah, I guess that's ok. AFAIK it's even already a restriction of the current (partly) implementation in HEAD.
though that was mainly a matter of convenience. Thoughts?
Quanah Gibson-Mount wrote:
Syncrepl was untenable with non-serialized updates (which moving to delta-syncrepl helped with). Even in a standalone setup, I find that serializing the update process provides faster write times (at least with back-(bh)db) than allowing multiple update threads.
But with transaction support there could be lots of modifications in one transaction which could prevent other LDAP clients from proceeding for an unacceptable amount of time if you serialize all transactions.
Ciao, Michael.
-
Howard Chu -
Michael Ströder -
Quanah Gibson-Mount -
Ralf Haferkamp