Just thinking aloud right now about what's needed here.
Basically all incoming requests participating in a transaction are queued until the Commit is received. Aside from basic parsing/validation, no other processing is performed until then.
If an Abort is received, the queued requests are simply discarded.
When the Commit request is received, all of the queued requests are performed sequentially by a single thread and the results are gathered. If any individual request fails, any preceding completed requests must be rolled back.
Since I'm considering this for both back-bdb and back-ndb, it appears we're going to need some transaction-specific hooks to be added to the BackendInfo structure. 1) start a TXN, get a TXN handle 2) end a TXN: Commit or Abort
For back-bdb the tricky part is exposing the updates atomically in the caches. I think the fact that entry caching uses BDB locks helps somewhat; we can keep entry cache items locked until the overall transaction completes. But for Aborts we'll either need to keep a copy of the previous cache info, or just discard it all. For now, discarding it all seems simpler.
For back-ndb things are currently easy since this backend does no caching of its own. As such, once the backend issues a Commit or Abort request, there's no further (backend) work to be done.
It's tempting to think about this for backglue, but we'd need a cross-database lock manager of some kind for detecting deadlocks. That implies that we really need an LDAP-level lock request, to handle distributed locking, and that the Transaction handling ought to be built on top of that. Currently the Transaction layer says nothing at all about locking, and it's up to the underying LDAP database to take care of it.
I guess another approach would just be to have backglue fully serialize all transactions; if only one is outstanding at any time there can be no deadlocks.
This brings up a question about whether slapd in general should fully serialize them. I was thinking, at the least, that we should only allow one active transaction per connection, though that was mainly a matter of convenience. Thoughts?