Over the past few weeks, I have been testing OpenLDAP as a backend for Samba4.
I've been working with the OpenLDAP team on my requirements, and there has been some really good outcomes - the memberOf module has been improved, as has the refint module.
However, I seem to have hit a brick wall, in the form of (internal) transaction support. I need an LDAP backend to support internal transactions - that is, when for example a 'member' modification is made, all the memberOf attributes must be updated before the call returns. Similarly, if a user or group moves, all the member/memberOf attributes that link the user to their groups must also move, before the modrdn returns.
The Samba4 test ldap.js tests this behaviour extensively, because I want to be sure it works.
As understand the discussion I've had with the OpenLDAP team, OpenLDAP does not support this, and will not support it for perhaps some time.
Similarly, from discussions with the Fedora DS team at the CIFS developer days, I understand that it is similarly very unlikely that Fedora DS will support internal transactions. (It also does not support subtree renames, which we also need).
The fact that LDAP does not expose a transaction API was always going to be a difficult part of having Samba4 use an LDAP backend, but I always assumed that if we pushed the really hard bit - updating linked attributes - into the LDAP server that we could at least always have a consistent DB. (It turns out this is one of the primary uses of transactions anyway.)
But without that consistency, and without knowing as a caller if all the updates succeed, I'm worried about how we can safely move forward.
This is especially disappointing because I was hoping that these free, replicating LDAP servers might solve the backed replication problem for me, without needing to use AD replication.
Does anybody have any ideas or suggestions on how I could get around this?
Should we drop the LDAP backend as a nice idea, but not going to work, and focus on DRS or some other form of replication?
Can someone imagine a sane way to reconstruct the DN links, including subtree renames, without the help of the LDAP backend? Could we ban subtree renames (as Fedora DS does), and try to handle this ourselves (with pre/post checks and a good deal of prayer)?
Thanks,
Andrew Bartlett
Andrew Bartlett wrote:
Over the past few weeks, I have been testing OpenLDAP as a backend for Samba4.
I've been working with the OpenLDAP team on my requirements, and there has been some really good outcomes - the memberOf module has been improved, as has the refint module.
However, I seem to have hit a brick wall, in the form of (internal) transaction support. I need an LDAP backend to support internal transactions - that is, when for example a 'member' modification is made, all the memberOf attributes must be updated before the call returns. Similarly, if a user or group moves, all the member/memberOf attributes that link the user to their groups must also move, before the modrdn returns.
The Samba4 test ldap.js tests this behaviour extensively, because I want to be sure it works.
As understand the discussion I've had with the OpenLDAP team, OpenLDAP does not support this, and will not support it for perhaps some time.
I may have overstated the problem in the previous discussion of our refint module. In fact, RE24 was already changed to work around any potential deadlock issues a long time ago. But to give some context: the refint module was originally written to operate synchronously (back in 2004). Some time later it was changed (2006) to asynchronous mode because users didn't want their clients to be stuck waiting for all the cascaded updates to complete. Most clients don't know or care that a particular change has side-effects. We could introduce a config keyword to select synch vs asynch behavior here, but I have a feeling that will still leave some group of users unhappy no matter how you set it.
Similarly, from discussions with the Fedora DS team at the CIFS developer days, I understand that it is similarly very unlikely that Fedora DS will support internal transactions. (It also does not support subtree renames, which we also need).
The fact that LDAP does not expose a transaction API
You mean draft-zeilenga-ldap-txn ?
was always going to be a difficult part of having Samba4 use an LDAP backend, but I always assumed that if we pushed the really hard bit - updating linked attributes - into the LDAP server that we could at least always have a consistent DB. (It turns out this is one of the primary uses of transactions anyway.)
But without that consistency, and without knowing as a caller if all the updates succeed, I'm worried about how we can safely move forward.
This is especially disappointing because I was hoping that these free, replicating LDAP servers might solve the backed replication problem for me, without needing to use AD replication.
Does anybody have any ideas or suggestions on how I could get around this?
There are other ways to guarantee consistency. The simplest approach is to just not store one end of the linked attributes, and always generate them dynamically when they're referenced.
In the old Symas Connexitor EMS product we used (the equivalent of) a UUIDAndOptionalName syntax for all references. In that case the DN was essentially just window-dressing; we always used the ID to actually reference entries and we updated the DNs whenever we found that they didn't match. As such, referential integrity was pretty simple - you never had anything pointing to the wrong entry; the worst that would happen is that you occasionally had dangling references to deleted entries stored in the DB but no one ever saw them because they were cleaned out whenever the entry containing the reference was read.
Should we drop the LDAP backend as a nice idea, but not going to work, and focus on DRS or some other form of replication?
Can someone imagine a sane way to reconstruct the DN links, including subtree renames, without the help of the LDAP backend? Could we ban subtree renames (as Fedora DS does), and try to handle this ourselves (with pre/post checks and a good deal of prayer)?
Banning subtree renames seems like a non-starter, and it only eliminates one case; the overall problem still remains.
On Wed, 2008-03-19 at 04:26 -0700, Howard Chu wrote:
Andrew Bartlett wrote:
Over the past few weeks, I have been testing OpenLDAP as a backend for Samba4.
I've been working with the OpenLDAP team on my requirements, and there has been some really good outcomes - the memberOf module has been improved, as has the refint module.
However, I seem to have hit a brick wall, in the form of (internal) transaction support. I need an LDAP backend to support internal transactions - that is, when for example a 'member' modification is made, all the memberOf attributes must be updated before the call returns. Similarly, if a user or group moves, all the member/memberOf attributes that link the user to their groups must also move, before the modrdn returns.
The Samba4 test ldap.js tests this behaviour extensively, because I want to be sure it works.
As understand the discussion I've had with the OpenLDAP team, OpenLDAP does not support this, and will not support it for perhaps some time.
I may have overstated the problem in the previous discussion of our refint module. In fact, RE24 was already changed to work around any potential deadlock issues a long time ago. But to give some context: the refint module was originally written to operate synchronously (back in 2004). Some time later it was changed (2006) to asynchronous mode because users didn't want their clients to be stuck waiting for all the cascaded updates to complete. Most clients don't know or care that a particular change has side-effects. We could introduce a config keyword to select synch vs asynch behavior here, but I have a feeling that will still leave some group of users unhappy no matter how you set it.
Great. If run sync, will it error out correctly if I make an invalid modification (say target not present etc).
Similarly, from discussions with the Fedora DS team at the CIFS developer days, I understand that it is similarly very unlikely that Fedora DS will support internal transactions. (It also does not support subtree renames, which we also need).
The fact that LDAP does not expose a transaction API
You mean draft-zeilenga-ldap-txn ?
I suppose I should have said 'The free LDAP implemetnations I'm looking at don't expose a transaction API'. What did end up happening with that draft?
was always going to be a difficult part of having Samba4 use an LDAP backend, but I always assumed that if we pushed the really hard bit - updating linked attributes - into the LDAP server that we could at least always have a consistent DB. (It turns out this is one of the primary uses of transactions anyway.)
But without that consistency, and without knowing as a caller if all the updates succeed, I'm worried about how we can safely move forward.
This is especially disappointing because I was hoping that these free, replicating LDAP servers might solve the backed replication problem for me, without needing to use AD replication.
Does anybody have any ideas or suggestions on how I could get around this?
There are other ways to guarantee consistency. The simplest approach is to just not store one end of the linked attributes, and always generate them dynamically when they're referenced.
In the old Symas Connexitor EMS product we used (the equivalent of) a UUIDAndOptionalName syntax for all references. In that case the DN was essentially just window-dressing; we always used the ID to actually reference entries and we updated the DNs whenever we found that they didn't match. As such, referential integrity was pretty simple - you never had anything pointing to the wrong entry; the worst that would happen is that you occasionally had dangling references to deleted entries stored in the DB but no one ever saw them because they were cleaned out whenever the entry containing the reference was read.
Do you think the LDAP backend could/should handle this, or will Samba4 have to do the GUID -> DN and DN -> GUID translations before passing things to the backend?
Should we drop the LDAP backend as a nice idea, but not going to work, and focus on DRS or some other form of replication?
Can someone imagine a sane way to reconstruct the DN links, including subtree renames, without the help of the LDAP backend? Could we ban subtree renames (as Fedora DS does), and try to handle this ourselves (with pre/post checks and a good deal of prayer)?
Banning subtree renames seems like a non-starter, and it only eliminates one case; the overall problem still remains.
Indeed.
Andrew Bartlett
Andrew Bartlett wrote:
On Wed, 2008-03-19 at 04:26 -0700, Howard Chu wrote:
Most clients don't know or care that a particular change has side-effects. We could introduce a config keyword to select synch vs asynch behavior here, but I have a feeling that will still leave some group of users unhappy no matter how you set it.
Great. If run sync, will it error out correctly if I make an invalid modification (say target not present etc).
No. We specifically don't validate the original modification here - the server's number one job is always to try to do exactly what the client specified. Also, you can't reliably detect an "invalid" name - it may simply be the name of a non-local entry.
The fact that LDAP does not expose a transaction API
You mean draft-zeilenga-ldap-txn ?
I suppose I should have said 'The free LDAP implemetnations I'm looking at don't expose a transaction API'. What did end up happening with that draft?
It's stalled, like a lot of other things... It is partially implemented in OpenLDAP. Eventually we'll have to complete the implementation, it's too important to leave dangling.
Does anybody have any ideas or suggestions on how I could get around this?
There are other ways to guarantee consistency. The simplest approach is to just not store one end of the linked attributes, and always generate them dynamically when they're referenced.
In the old Symas Connexitor EMS product we used (the equivalent of) a UUIDAndOptionalName syntax for all references. In that case the DN was essentially just window-dressing; we always used the ID to actually reference entries and we updated the DNs whenever we found that they didn't match. As such, referential integrity was pretty simple - you never had anything pointing to the wrong entry; the worst that would happen is that you occasionally had dangling references to deleted entries stored in the DB but no one ever saw them because they were cleaned out whenever the entry containing the reference was read.
Do you think the LDAP backend could/should handle this, or will Samba4 have to do the GUID -> DN and DN -> GUID translations before passing things to the backend?
That would only be practical if you had LDAP transaction support....
It strikes me that the best approach is to just dynamically generate the memberOf values instead of storing them statically. But that also depends on your usage patterns.
On Wed, 2008-03-19 at 17:09 -0700, Howard Chu wrote:
Andrew Bartlett wrote:
On Wed, 2008-03-19 at 04:26 -0700, Howard Chu wrote:
Most clients don't know or care that a particular change has side-effects. We could introduce a config keyword to select synch vs asynch behavior here, but I have a feeling that will still leave some group of users unhappy no matter how you set it.
Great. If run sync, will it error out correctly if I make an invalid modification (say target not present etc).
No. We specifically don't validate the original modification here - the server's number one job is always to try to do exactly what the client specified. Also, you can't reliably detect an "invalid" name - it may simply be the name of a non-local entry.
This is going to get painful then...
The fact that LDAP does not expose a transaction API
You mean draft-zeilenga-ldap-txn ?
I suppose I should have said 'The free LDAP implemetnations I'm looking at don't expose a transaction API'. What did end up happening with that draft?
It's stalled, like a lot of other things... It is partially implemented in OpenLDAP. Eventually we'll have to complete the implementation, it's too important to leave dangling.
OK.
Does anybody have any ideas or suggestions on how I could get around this?
There are other ways to guarantee consistency. The simplest approach is to just not store one end of the linked attributes, and always generate them dynamically when they're referenced.
In the old Symas Connexitor EMS product we used (the equivalent of) a UUIDAndOptionalName syntax for all references. In that case the DN was essentially just window-dressing; we always used the ID to actually reference entries and we updated the DNs whenever we found that they didn't match. As such, referential integrity was pretty simple - you never had anything pointing to the wrong entry; the worst that would happen is that you occasionally had dangling references to deleted entries stored in the DB but no one ever saw them because they were cleaned out whenever the entry containing the reference was read.
Do you think the LDAP backend could/should handle this, or will Samba4 have to do the GUID -> DN and DN -> GUID translations before passing things to the backend?
That would only be practical if you had LDAP transaction support....
It strikes me that the best approach is to just dynamically generate the memberOf values instead of storing them statically. But that also depends on your usage patterns.
Well, searches on member and memberOf are common, so it seems we would quickly find ourselves downloading the whole backend DB and filtering locally...
I would hate to give up on this area of work - if only because I've put a lot of time into it, and it seems like a really useful way to use Samba4, but it also seems that I've found a rat-hole, with far more to it than I ever imagined.
Any further thoughts on how we can resurrect this would be most appreciated.
Andrew Bartlett
Andrew Bartlett wrote:
On Wed, 2008-03-19 at 17:09 -0700, Howard Chu wrote:
It strikes me that the best approach is to just dynamically generate the memberOf values instead of storing them statically. But that also depends on your usage patterns.
Well, searches on member and memberOf are common, so it seems we would quickly find ourselves downloading the whole backend DB and filtering locally...
Searching on memberOf doesn't make a lot of sense to me, when you could simply read the group object directly. When is this actually a useful thing to do? An alternative would be to make the memberOf overlay intercept these filters and rewrite them in terms of member.
openldap-technical@openldap.org
-
Andrew Bartlett -
Howard Chu