The recent work on expanding dynamic group functionality in the dynlist overlay seems to have been a bad idea. It makes an already fairly complex overlay even more complicated, and it puts a lot more work into the read side of operations, which adds up to quite noticeable slowdowns in search performance on deployments that make heavy use of dynamic groups with large memberships.
It appears that in these situations, the approach used in the autogroup overlay (which has been in contrib since 2007) is better. It moves all of the membership management into the write side of operations, updating memberships whenever dynamic group definitions are modified, or when member entries are written/updated/etc. As such, it allows dynamically defined group memberships to be read/searched at full speed, as if they were static groups. The search performance difference between autogroup and the dynlist approach is pretty drastic when large groups and large numbers of groups are in use. As such I believe autogroup should be recommended, going forward, and we should move it from contrib into the core code.
For similar reasons, dynamically managing memberOf attributes in dynlist also has a major impact on search performance. And as alluded to before, adding that functionality to dynlist has made the code quite a bit more complex. Again for performance reasons, it's better to manage this attribute on the write side, updating a real attribute in the DB when groups and memberships are modified, instead of doing the lookup work on the read/search side. As such, we should reverse the decision to deprecate the memberof overlay. There were a couple problems that the overlay presented in replication environments before, that prompted us to deprecate it, but I believe those problems have now been resolved. The first one, referenced in ITS#7400, had to do with the actual memberof attributes getting replicated even though they weren't meant to be. The solution for that should have been to specify that memberof should be excluded from replication using "exattr" in a syncrepl consumer config. The exattr option wasn't behaving as intended in the past, and using it would cause data desyncs, but that problem was fixed long ago.
The other problem was simply due to the lack of ordering guarantees in syncrepl refreshes, which prevented the memberof overlay from updating an member's memberof attribute if the group entry got replicated before the member entry. That problem has been solved in ITS#10167, simply adding a check whenever new entries are added, to see if they're already members of any existing groups.
As such, the memberof overlay should be perfectly fine for use in replication scenarios now. More testing of those scenarios is welcome.
With statically managed member and memberof attributes, the major hits to search performance have been removed. We're still trying dynamically managed nesting of groups though, which is what the new nestgroup overlay (ITS#10161) is for. Again, we need testing to see how much performance impact there is from this much-reduced overhead in the read/search side of things. The config is also a lot easier to understand in its own overlay, instead of shoe-horned into the dynlist config.
Testing and feedback appreciated.
Thank you Howard (and openldap core team) for this synthesis of the current and future state of dynamic membership devs.
Indeed, some of my customers were suffering from dynlist performances, and the decision was taken to go back to more static features (like memberOf).
For now I did not test nestgroup overlay, but I'll give a look for next installations. Do you have any idea in which version of OpenLDAP nestgroup will be officially released?
Regards,
Le 26/02/2024 à 16:54, Howard Chu a écrit :
The recent work on expanding dynamic group functionality in the dynlist overlay seems to have been a bad idea. It makes an already fairly complex overlay even more complicated, and it puts a lot more work into the read side of operations, which adds up to quite noticeable slowdowns in search performance on deployments that make heavy use of dynamic groups with large memberships.
It appears that in these situations, the approach used in the autogroup overlay (which has been in contrib since 2007) is better. It moves all of the membership management into the write side of operations, updating memberships whenever dynamic group definitions are modified, or when member entries are written/updated/etc. As such, it allows dynamically defined group memberships to be read/searched at full speed, as if they were static groups. The search performance difference between autogroup and the dynlist approach is pretty drastic when large groups and large numbers of groups are in use. As such I believe autogroup should be recommended, going forward, and we should move it from contrib into the core code.
For similar reasons, dynamically managing memberOf attributes in dynlist also has a major impact on search performance. And as alluded to before, adding that functionality to dynlist has made the code quite a bit more complex. Again for performance reasons, it's better to manage this attribute on the write side, updating a real attribute in the DB when groups and memberships are modified, instead of doing the lookup work on the read/search side. As such, we should reverse the decision to deprecate the memberof overlay. There were a couple problems that the overlay presented in replication environments before, that prompted us to deprecate it, but I believe those problems have now been resolved. The first one, referenced in ITS#7400, had to do with the actual memberof attributes getting replicated even though they weren't meant to be. The solution for that should have been to specify that memberof should be excluded from replication using "exattr" in a syncrepl consumer config. The exattr option wasn't behaving as intended in the past, and using it would cause data desyncs, but that problem was fixed long ago.
The other problem was simply due to the lack of ordering guarantees in syncrepl refreshes, which prevented the memberof overlay from updating an member's memberof attribute if the group entry got replicated before the member entry. That problem has been solved in ITS#10167, simply adding a check whenever new entries are added, to see if they're already members of any existing groups.
As such, the memberof overlay should be perfectly fine for use in replication scenarios now. More testing of those scenarios is welcome.
With statically managed member and memberof attributes, the major hits to search performance have been removed. We're still trying dynamically managed nesting of groups though, which is what the new nestgroup overlay (ITS#10161) is for. Again, we need testing to see how much performance impact there is from this much-reduced overhead in the read/search side of things. The config is also a lot easier to understand in its own overlay, instead of shoe-horned into the dynlist config.
Testing and feedback appreciated.
David Coutadeur wrote:
Thank you Howard (and openldap core team) for this synthesis of the current and future state of dynamic membership devs.
Indeed, some of my customers were suffering from dynlist performances, and the decision was taken to go back to more static features (like memberOf).
For now I did not test nestgroup overlay, but I'll give a look for next installations. Do you have any idea in which version of OpenLDAP nestgroup will be officially released?
Most likely 2.7 later this year. But I see no reason why it wouldn't work in 2.5/2.6.
Regards,
Le 26/02/2024 à 16:54, Howard Chu a écrit :
The recent work on expanding dynamic group functionality in the dynlist overlay seems to have been a bad idea. It makes an already fairly complex overlay even more complicated, and it puts a lot more work into the read side of operations, which adds up to quite noticeable slowdowns in search performance on deployments that make heavy use of dynamic groups with large memberships.
-
David Coutadeur -
Howard Chu