Hi,
I was wondering whether OpenLDAP supports "LDAP ease modify restrictions" control (1.2.840.113556.1.4.1413). I was not able to find any information or documentation except for some vague clue that it might have been supported in OpenLDAP 2.2/2.3 times. The control is definitely not present in root DSE of my stock OpenLDAP 2.4.41. So I was wondering if there is some module/overlay that is needed to enable support for that.
Thanks
Radovan Semancik wrote:
I was wondering whether OpenLDAP supports "LDAP ease modify restrictions" control (1.2.840.113556.1.4.1413). I was not able to find any information or documentation except for some vague clue that it might have been supported in OpenLDAP 2.2/2.3 times. The control is definitely not present in root DSE of my stock OpenLDAP 2.4.41. So I was wondering if there is some module/overlay that is needed to enable support for that.
The control's OID is listed in my OpenLDAP 2.4.44 instance.
BTW: I'd always recommend to fix the client instead of using this control.
Ciao, Michael.
On 02/20/2016 04:17 PM, Michael Ströder wrote:
The control's OID is listed in my OpenLDAP 2.4.44 instance.
Thanks. I'll try that once I find Ubuntu packages of recent OpenLDAP version ... that'll take a bit of time :-)
BTW: I'd always recommend to fix the client instead of using this control.
Actually, that's not really a practical advice.
LDAP does not have ACID consistency. Adding a value that is already added may happen even if everything operates correctly and there is no bug in the client code.
E.g. imagine that two clients adds user to the same group. If the control is not present, one of the operations fails even if there is actually no logical error at all. Reading a value, filtering out the values and writing it again provides no guarantees, as the value might be changed in the meantime. Yes, the clients may have special handling for this error and evaluate the case and re-try the operation. In a way similar to handling of optimistic lock conditions. But that is at least one extra round-trip. Usually two. And this significantly complicates the client code as the operation may add a couple of values and remove other values. It is possible to write a correct code to handle that (we have done that in midPoint), but it is not easy to develop and (especially) test it. It is not something that one would expect from an ordinary LDAP client, is it?
Radovan Semancik wrote:
On 02/20/2016 04:17 PM, Michael Ströder wrote:
The control's OID is listed in my OpenLDAP 2.4.44 instance.
Thanks. I'll try that once I find Ubuntu packages of recent OpenLDAP version ... that'll take a bit of time :-)
BTW: I'd always recommend to fix the client instead of using this control.
Actually, that's not really a practical advice.
LDAP does not have ACID consistency. Adding a value that is already added may happen even if everything operates correctly and there is no bug in the client code.
E.g. imagine that two clients adds user to the same group.
What's the rationale for doing that?
If the control is not present, one of the operations fails even if there is actually no logical error at all. Reading a value, filtering out the values and writing it again provides no guarantees, as the value might be changed in the meantime. Yes, the clients may have special handling for this error and evaluate the case and re-try the operation. In a way similar to handling of optimistic lock conditions. But that is at least one extra round-trip. Usually two. And this significantly complicates the client code as the operation may add a couple of values and remove other values. It is possible to write a correct code to handle that (we have done that in midPoint), but it is not easy to develop and (especially) test it. It is not something that one would expect from an ordinary LDAP client, is it?
Sorry, I suspect that such a client would do things wrong. Bear in mind that maintaining group entries has a serious security impact!
Many years ago with brand new W2K there was a bug in MMC where the MMC client always replaced all values with a "new" set of values. That re-added group membership which was removed by another client instance before.
Therefore my web2ldap does two things:
1. Maintaining group membership is a separate use-case explicitly adding/removing distinct attribute values.
2. Modifying an entry provokes detecting write conflicts by - adding/removing distinct attribute values even when "replacing" values (and looking at subschema for determining whether attribute type has EQUALITY matching rule or not) - using special re-read filter and modify request with Assertion Control
BTW: Even when using the permissive modify control you would have to read the old entry for removing attribute values.
Ciao, Michael.
On 02/22/2016 12:53 PM, Michael Ströder wrote:
Radovan Semancik wrote:
E.g. imagine that two clients adds user to the same group.
What's the rationale for doing that?
There are two independent clients. Do you need an explicit rationale for each of them to do the same operation at the same time? I would rather say: what makes you think that the clients will NOT do that operation? Or would you expect that all the clients of the directory server must be explicitly synchronized by some "magic" external means?
But OK, let's get more specific. Consider this example: midPoint is an IDM system. It is processing several changes at once. E.g. user requests a set of roles. Some roles from this set are to be approved by approver A1, other roles by approver A2. We do not want to wait for both of them unless necessary (e.g. approver A2 may be on vacation), so So we split the request and the processing goes in parallel. What happens when both approvers approve the request at the same time? There may be two roles that add user to the same group (which is actually quite a common case). Yes, we can have complex coordination in midPoint that avoids concurrent writes to the server - even though that would be very difficult as midPoint can operate in multi-node cluster. But it is unnecessarily complex and even this complex thing does not entirely resolve the problem (see below). Currently in midPoint we are still doing the read-filter-write cycle with deltas just to lower the chance of this issue occuring. But that still does not guarantee success. E.g. both nodes read, both nodes decide that there is nothing to filter and both nodes write. Which results in error. So, in midPoint we had to implement quite complex error handling on top of that to make sure that we can handle all situations. Something tells me that other LDAP clients will not do that.
Please note that this case is not specific to midPoint. Any IDM has to do the same thing. E.g. the IDM will try to add a group, but the same group may be added at the same time using native administration tool in the application.
Anytime there is more than one client there is a risk of this happening. And as far as I know LDAP servers were designed especially to support many clients, weren't they?
Bear in mind that maintaining group entries has a serious security impact!
Agreed. But how exactly is "security impact" influencing consistency model of the LDAP sever?
Many years ago with brand new W2K there was a bug in MMC where the MMC client always replaced all values with a "new" set of values. That re-added group membership which was removed by another client instance before.
Well, that was really a bug in the client, wasn't it? And that's exactly what should be fixed in the client, right? That's the reason why add/delete should be used instead or replace. But I'm talking about add/delete case here.
BTW: Even when using the permissive modify control you would have to read the old entry for removing attribute values.
Why? I know the DN of the user and I know the DN of the group, why should I read the (potentially very long) list of all group members to make a simple operation?
Radovan Semancik wrote:
On 02/22/2016 12:53 PM, Michael Ströder wrote:
Radovan Semancik wrote:
E.g. imagine that two clients adds user to the same group.
What's the rationale for doing that?
There are two independent clients. Do you need an explicit rationale for each of them to do the same operation at the same time?
I need a rationale for both clients succeeding for this same operation.
BTW: You would have to further specify what "same operation" really means in detail.
But OK, let's get more specific. Consider this example: midPoint is an IDM system.
Yes, I saw your presentation at LDAPcon.
Which results in error. So, in midPoint we had to implement quite complex error handling on top of that to make sure that we can handle all situations.
There's no way around having decent error handling anyway. Permissive modify control won't help you there in general. And catching attributeOrValueExists and gracefully handle it is not a big deal.
Something tells me that other LDAP clients will not do that.
Yes for sure, there are many stupid LDAP clients out there.
Please note that this case is not specific to midPoint. Any IDM has to do the same thing.
Yes.
Bear in mind that maintaining group entries has a serious security impact!
Agreed. But how exactly is "security impact" influencing consistency model of the LDAP sever?
As said: I've decided to handle groups in web2ldap in specific way and to provoke failure for concurrent writes based on stale data in general.
Many years ago with brand new W2K there was a bug in MMC where the MMC client always replaced all values with a "new" set of values. That re-added group membership which was removed by another client instance before.
Well, that was really a bug in the client, wasn't it? And that's exactly what should be fixed in the client, right? That's the reason why add/delete should be used instead or replace. But I'm talking about add/delete case here.
You have to define more details of what you're really doing.
As said: It' sane to add/remove distinct attribute values and you can do that concurrently. You don't need permissive modify control for that.
BTW: Even when using the permissive modify control you would have to read the old entry for removing attribute values.
Why? I know the DN of the user and I know the DN of the group, why should I read the (potentially very long) list of all group members to make a simple operation?
I don't know the operations sent by your client. Either I have some doubts that you need permissive modify control at all or I have some doubts that your client does things right. Choose yourself.
But hey, you don't have to follow my advice. Just try and see yourself.
Ciao, Michael.
On 02/22/2016 02:13 PM, Michael Ströder wrote:
There are two independent clients. Do you need an explicit rationale for each of them to do the same operation at the same time? I need a rationale for both clients succeeding for this same operation.
BTW: You would have to further specify what "same operation" really means in detail.
As I've written before: two clients adding the same user to the same group. So, more specifically:
dn: cn=foogroup,ou=groups,dc=example,dc=com changetype: add add: member member: uid=bar,ou=people,dc=example,dc=com
Which results in error. So, in midPoint we had to implement quite complex error handling on top of that to make sure that we can handle all situations.
There's no way around having decent error handling anyway. Permissive modify control won't help you there in general. And catching attributeOrValueExists and gracefully handle it is not a big deal.
Right. Even though the situation is not that easy when going through abstractions such as JNDI or ConnId ... And there are still round-trips that are completely unnecessary.
But most importantly: I would rather like that error handling is trigged only if there is an actual error. MidPoint does a very good error handling. But having logfile full of errors that are in fact just a pretty normal operation is not a nice thing.
Something tells me that other LDAP clients will not do that.
Yes for sure, there are many stupid LDAP clients out there.
Well, maybe the reason for that is that creating a proper LDAP client is no easy task. Maybe part of the problem are subtle details such as those that we are now discussing?
Agreed. But how exactly is "security impact" influencing consistency model of the LDAP sever?
As said: I've decided to handle groups in web2ldap in specific way and to provoke failure for concurrent writes based on stale data in general.
Yes, but if you "provoke" a failure you have to be sure that other components in the system can handle that failure well. And given the arguments above I'm not entirely sure that this assumption holds ...
Anyway, what is actually the problem with operation that adds value that is already there? Why it should fail at all? It will not change the final state. It will not break anything. The same with the operation that deletes already deleted value. Even if two operations are executed at the same time, the result would be the same regardless of execution order. So what's the problem here?
The problem are operations that add and remove the same value at the same time. Or operations that replace the values. But the attributeOrValueExists error is not going to help here. The outcome will always depend on operation ordering. And the error cannot even be used as reliable detection of a conflict, because it will not happen in all the cases. So it only complicates client code without bringing any substantial benefit.
Radovan Semancik wrote:
On 02/22/2016 02:13 PM, Michael Ströder wrote:
There's no way around having decent error handling anyway. Permissive modify control won't help you there in general. And catching attributeOrValueExists and gracefully handle it is not a big deal.
[..] But most importantly: I would rather like that error handling is trigged only if there is an actual error. MidPoint does a very good error handling. But having logfile full of errors that are in fact just a pretty normal operation is not a nice thing.
Whether LDAP result code attributeOrValueExists is treated as an error or just logged as "nothing to be seen here" is up to your code, isn't it?
Especially it depends on whether you modify more than one attribute value. For the case get-the-change-out-of-the-door-now it's probably best approach to limit the group modifications to single value changes.
For various reasons bulk updates should be a different thing if you need to support concurrent operation.
BTW: If a single role change leads to write operations to more than entry you would need LDAP transaction support to really ensure atomicity of that single role change.
Something tells me that other LDAP clients will not do that.
Yes for sure, there are many stupid LDAP clients out there.
Well, maybe the reason for that is that creating a proper LDAP client is no easy task. Maybe part of the problem are subtle details such as those that we are now discussing?
Aren't subtle details an intrinsic part of all non-trivial systems? ;-)
As Howard said some years ago: "You always have to use your brain." It seems to me, you do.
As said: I've decided to handle groups in web2ldap in specific way and to provoke failure for concurrent writes based on stale data in general.
Yes, but if you "provoke" a failure you have to be sure that other components in the system can handle that failure well.
Yes. In case of web2ldap it even means handling the error in the UI.
And given the arguments above I'm not entirely sure that this assumption holds ...
I'm not able to comment on any existing system. ;-)
Anyway, what is actually the problem with operation that adds value that is already there? Why it should fail at all? It will not change the final state.
If you ensure that you're not re-adding old group membership relations then everything's fine.
Still I think you don't need it. And as developer of a general-purpose IDM you can't rely on this proprietary control to be supported by the LDAP server. Hence you need the error handling in your code anyway.
The problem are operations that add and remove the same value at the same time.
Of course a second user interacting with your UI could revert the changes made by a first user. There's nothing you could do about that.
Or operations that replace the values. But the attributeOrValueExists error is not going to help here.
We have to distinguish various write operations in detail: attributeOrValueExists (for MOD_ADD) and its counterpart noSuchAttribute (for MOD_DELETE) solely helps if your modify request only contains *single* attribute values.
The outcome will always depend on operation ordering. And the error cannot even be used as reliable detection of a conflict, because it will not happen in all the cases. So it only complicates client code without bringing any substantial benefit.
I think we mostly agree on the general issues.
But we agree to disagree whether permissive modify control is part of a solution or will mask serious security issues. Personally I prefer to let problems/error happen and then explicitly ignore them if I'm 100% sure it's ok. So personally I wouldn't use permissive modify control. YMMV.
Ciao, Michael.
On 02/22/2016 04:07 PM, Michael Ströder wrote:
Radovan Semancik wrote:
On 02/22/2016 02:13 PM, Michael Ströder wrote:
There's no way around having decent error handling anyway. Permissive modify control won't help you there in general. And catching attributeOrValueExists and gracefully handle it is not a big deal.
[..] But most importantly: I would rather like that error handling is trigged only if there is an actual error. MidPoint does a very good error handling. But having logfile full of errors that are in fact just a pretty normal operation is not a nice thing.
Whether LDAP result code attributeOrValueExists is treated as an error or just logged as "nothing to be seen here" is up to your code, isn't it?
Depends on a layer. The operation will always be logged as LDAP error response. The connector also does not know whether it is real error or not. We are keeping connectors very simple. So it also get logger in the ConnId as an error. Even if the situation is later handler correctly, the errors are there. Sometimes you cannot entirely control what gets logged if you are using libraries that you are not maintaining yourself. And who in the opensource world is using just their own stuff?
For various reasons bulk updates should be a different thing if you need to support concurrent operation.
I do not think so. The less exceptions are there the better.
BTW: If a single role change leads to write operations to more than entry you would need LDAP transaction support to really ensure atomicity of that single role change.
Not really. I do not need atomicity. I just need eventual consistency. I can get that without transactions.
As said: I've decided to handle groups in web2ldap in specific way and to provoke failure for concurrent writes based on stale data in general.
Yes, but if you "provoke" a failure you have to be sure that other components in the system can handle that failure well.
Yes. In case of web2ldap it even means handling the error in the UI.
Well, that might be enough for web2ldap. But we are a "self-healing" system. Even non-technical end users are using it (e.g. requesting and approving roles). Reporting this error to end user does not really help. And it won't work even if we report it to the admin. Bothering admins with any minor issue just won't scale.
Anyway, what is actually the problem with operation that adds value that is already there? Why it should fail at all? It will not change the final state.
If you ensure that you're not re-adding old group membership relations then everything's fine.
The thing is that you do not know. If you re-add old group membership you will actually NOT get the error. If you are adding existing group membership you will get the error. Strange, isn't it? Not very useful.
Still I think you don't need it. And as developer of a general-purpose IDM you can't rely on this proprietary control to be supported by the LDAP server. Hence you need the error handling in your code anyway.
We have it already. But I would like to avoid triggering the code unless really necessary. So, midPoint will look nicely and efficiently when working with nice LDAP servers. And when working with nasty LDAP servers the logfiles will be full of messages describing handled errors and there will be a lot of additional round-trips. I'm OK with that. The people that choose bad LDAP servers deserve to suffer :-) .... Anyway, this is much better than being equally nasty with all the LDAP servers.
The problem are operations that add and remove the same value at the same time.
Of course a second user interacting with your UI could revert the changes made by a first user. There's nothing you could do about that.
Exactly.
Or operations that replace the values. But the attributeOrValueExists error is not going to help here.
We have to distinguish various write operations in detail: attributeOrValueExists (for MOD_ADD) and its counterpart noSuchAttribute (for MOD_DELETE) solely helps if your modify request only contains *single* attribute values.
Not even in that case. E.g. see above. You will not get the error if you are re-adding a group that was deleted just a millisecond ago just because the network latencies haven't turned up in your favor.
So, the implication "error => something wrong happened" does not hold. And the implication "something wrong happened => error" does not hold either. So, what the error really says is:
"Hey there! Maybe something wrong happened. Or maybe not. It may all be OK. There is no way to be sure. So forget it. I just wanted to talk to you. Sorry to bother you. And, by the way, your operation failed. Just for fun. Try something else. I won't tell you what. Go figure. Bye."
How useful is that?
I think we mostly agree on the general issues.
But we agree to disagree whether permissive modify control is part of a solution or will mask serious security issues. Personally I prefer to let problems/error happen and then explicitly ignore them if I'm 100% sure it's ok. So personally I wouldn't use permissive modify control. YMMV.
Correct. But this specific thing will not help you. Because the error may happen when everything is OK. And even worse: it might NOT happen if there is a real problem. Relying on that error makes no sense. And in fact it might be even dangerous. This is a bad trade-off. Very bad.
Radovan Semancik wrote:
On 02/22/2016 04:07 PM, Michael Ströder wrote:
The problem are operations that add and remove the same value at the same time.
Of course a second user interacting with your UI could revert the changes made by a first user. There's nothing you could do about that.
Exactly.
Or operations that replace the values. But the attributeOrValueExists error is not going to help here.
We have to distinguish various write operations in detail: attributeOrValueExists (for MOD_ADD) and its counterpart noSuchAttribute (for MOD_DELETE) solely helps if your modify request only contains *single* attribute values.
Not even in that case. E.g. see above. You will not get the error if you are re-adding a group that was deleted just a millisecond ago just because the network latencies haven't turned up in your favor.
So, the implication "error => something wrong happened" does not hold. And the implication "something wrong happened => error" does not hold either. So, what the error really says is:
"Hey there! Maybe something wrong happened. Or maybe not. It may all be OK. There is no way to be sure. So forget it. I just wanted to talk to you. Sorry to bother you. And, by the way, your operation failed. Just for fun. Try something else. I won't tell you what. Go figure. Bye."
How useful is that?
I think we mostly agree on the general issues.
But we agree to disagree whether permissive modify control is part of a solution or will mask serious security issues. Personally I prefer to let problems/error happen and then explicitly ignore them if I'm 100% sure it's ok. So personally I wouldn't use permissive modify control. YMMV.
Correct. But this specific thing will not help you. Because the error may happen when everything is OK. And even worse: it might NOT happen if there is a real problem. Relying on that error makes no sense. And in fact it might be even dangerous. This is a bad trade-off. Very bad.
Your logic is flawed: "Just because you may not get an error message when something bad has happened, we want to *never* get an error message when something bad has happened."
Automated or not, large scale distributed or not, if two administrators are making overlapping changes to a single user's privileges at the same time, you have a broken system.
There's a relevant joke "A man with two watches never knows what time it is."
If you don't have distinctly delegated administration zones, and you allow multiple admins to independently operate on the same population of users, you can *never* know if your security definitions are correct. Error messages of this nature are a clear indication that your delegations are broken.
On 02/22/2016 06:29 PM, Howard Chu wrote:
Your logic is flawed: "Just because you may not get an error message when something bad has happened, we want to *never* get an error message when something bad has happened."
Yes and no. I do not care about the error. I just want my operation to proceed. It is actually very simple to do, isn't it? If the server really feels like it must indicate that it I'm doing a strange thing then I'm OK with that. I just want the operation to go on. I know what I'm doing.
Automated or not, large scale distributed or not, if two administrators are making overlapping changes to a single user's privileges at the same time, you have a broken system.
No. I do not have broken system. There are many use cases when that is valid. E.g. single admin adding user to a group manually during and emergency situation. And later make that change legit by following the proper processes, which tries to add the used to the group again. And it does not need to be a person doing that at all. When operation times out, you do not know if the request was processed or not. Most systems will re-try the operation. But if that operation was processed by the server then it will conflict with itself and end up with an error. Is that a broken system? That may happen even if there is no timeout. LDAP does not have distributed transactions, therefore the application may simply die between receiving an LDAP response and passing that information to upper layers. So the operation is likely to retry after restart. Or the information source and the LDAP may become inconsistent because someone simply screwed up. There are many ways how two databases on the network may become inconsistent. Too many. You say that if I consider inconsistency as a real possibility then my system is broken?
Yes, I can always read the entry first, compute changes and then modify it. But why do I need to do that? It takes two round trips and, client overhead and it does not guarantee a sucess anyway. Server can do that easily and reliably. Now, if my directory server is somewhere in the cloud tens of milliseconds away and I have millions of users to provision then each extra round-trip is a waste.
Yes, I do not need to read and entry. I just need to handle the resulting error. Then read the entry, modify the operation and retry. Maybe get the same error. Read entry again. And so on. Anyway, this is at least three round-trips instead of one. And it is strictly sequential. So, more waste. Yes, I do not need to do that for every entry. Only if I get a conflict. So if my data are fairly consistent this may a way to go. But that complicates the client. Good error handling is not easy to design and it is difficult to test properly. This is the hard way.
And now there is this simple and elegant control. It disables a "feature" that does not bring any benefit for me and it makes my life difficult. And the mere existence of that control means that obviously I'm not the only one having problem with this (in my opinion quite bad) part of LDAP spec. So why not use it?
If you don't have distinctly delegated administration zones, and you allow multiple admins to independently operate on the same population of users, you can *never* know if your security definitions are correct. Error messages of this nature are a clear indication that your delegations are broken.
Yes and no. Yes, you do not know the state of all systems exactly at every moment. But if you want to satisfy this requirement then LDAP is not the right tool for you anyway. To do that you will need strict ACID-like consistency, you will have to sacrifice availability, you will most likely need distributed transactions, you will need to coordinate backup and restore operations with a complex restore and harmonization ceremonies - all the things that LDAP servers are not really great at. We really cannot have that in practice. I'm quite sure you realize that. So now we are only discussing how well we can approximate the state of the systems. You are proposing one solution, I have another solution. But both solutions are approximating. You say that my solution is wrong. I know that it is not. My solution only accepts what's out there. In reality it is very common that there are many admins working independently over the same population. There are many operations (sometimes conflicting) coming from many independent sources. It is one of fundamental assumption of my design. I do not need an error to tell me that. I already know that. And that reality is not something that can be changed easily (if it can be changed at all). And even if we can somehow magically change that, the security and organizational policies are often a moving target anyway. They are changing faster than the data can adapt to them. In reality it is not always possible to make strictly consistent deterministic IDM system. We have learned that lesson very early. It looks like a much more practical solution is to make system that continuously approximates the data and makes sure that it eventually converges. That's what we have chosen to do with midPoint. It works well. Very well. It is not broken.
So, let's get back to the original question: does OpenLDAP support the control? Do I need to configure something to enable it? That's all I need.
Thanks.
Radovan Semancik wrote:
Yes, I can always read the entry first, compute changes and then modify it. But why do I need to do that? It takes two round trips and, client overhead and it does not guarantee a sucess anyway. Server can do that easily and reliably. Now, if my directory server is somewhere in the cloud tens of milliseconds away and I have millions of users to provision then each extra round-trip is a waste.
Maybe we have a different understanding of the semantics of the permissive modify control:
IMO using permissive modify control does not help getting rid of this extra round-trip because you have to read the target entry first anyway to determine whether you have to remove attributes or distinct attribute values.
Getting rid of the round-trip would require using something like the contrib addpartial overlay where the client application always sends add requests with the whole entry even for existing entries.
So, let's get back to the original question: does OpenLDAP support the control? Do I need to configure something to enable it? That's all I need.
As said in my *first* answer it's listed in the rootDSE of my installation.
And it seems to work:
test-permissive-control.ldif: ------------------------------------------------------ dn: uid=foobar42,ou=Testing,dc=stroeder,dc=de changetype: modify add: o o: Test -
------------------------------------------------------
$ ldapmodify -f test-permissive-control.ldif modifying entry "uid=foobar42,ou=Testing,dc=stroeder,dc=de"
$ ldapmodify -f test-permissive-control.ldif modifying entry "uid=foobar42,ou=Testing,dc=stroeder,dc=de" ldap_modify: Type or value exists (20) additional info: modify/add: o: value #0 already exists
$ ldapmodify -e 1.2.840.113556.1.4.1413 -f test-permissive-control.ldif modifying entry "uid=foobar42,ou=Testing,dc=stroeder,dc=de"
Ciao, Michael.
Radovan Semancik wrote:
On 02/22/2016 04:07 PM, Michael Ströder wrote:
Radovan Semancik wrote:
On 02/22/2016 02:13 PM, Michael Ströder wrote:
There's no way around having decent error handling anyway. Permissive modify control won't help you there in general. And catching attributeOrValueExists and gracefully handle it is not a big deal.
[..] But most importantly: I would rather like that error handling is trigged only if there is an actual error. MidPoint does a very good error handling. But having logfile full of errors that are in fact just a pretty normal operation is not a nice thing.
Whether LDAP result code attributeOrValueExists is treated as an error or just logged as "nothing to be seen here" is up to your code, isn't it?
Depends on a layer. The operation will always be logged as LDAP error response.
But it's up to your application whether a LDAP response code results in a warning/error log level message or a debug/info log level message.
The connector also does not know whether it is real error or not.
This sounds strange to me. Connectors I implement for my customers always know whether something is a real error or not and then it decides whether to abort or ignore.
BTW: If a single role change leads to write operations to more than entry you would need LDAP transaction support to really ensure atomicity of that single role change.
Not really. I do not need atomicity. I just need eventual consistency. I can get that without transactions.
That's ok.
As said: I've decided to handle groups in web2ldap in specific way and to provoke failure for concurrent writes based on stale data in general.
Yes, but if you "provoke" a failure you have to be sure that other components in the system can handle that failure well.
Yes. In case of web2ldap it even means handling the error in the UI.
Well, that might be enough for web2ldap. But we are a "self-healing" system.
Self-healing just means that if anything goes seriously wrong in a connector it will try to reach the target state by doing a full sync during next run. Nothing special in your product.
Or operations that replace the values. But the attributeOrValueExists error is not going to help here.
We have to distinguish various write operations in detail: attributeOrValueExists (for MOD_ADD) and its counterpart noSuchAttribute (for MOD_DELETE) solely helps if your modify request only contains *single* attribute values.
Not even in that case. E.g. see above. You will not get the error if you are re-adding a group that was deleted just a millisecond ago just because the network latencies haven't turned up in your favor.
Now this sounds suspicious to me. Are you using several MMR replicas to write to at the same time? If yes, here's some advice in good faith and without offense: DON'T DO THAT!
"Hey there! Maybe something wrong happened. Or maybe not. It may all be OK. There is no way to be sure. So forget it. I just wanted to talk to you. Sorry to bother you. And, by the way, your operation failed. Just for fun. Try something else. I won't tell you what. Go figure. Bye."
There's nothing wrong to log a debug/info message saying: "Ignored LDAP response code <foo> because of condition <bar>"
Who is informed about what message depends on your log level routing.
Ciao, Michael.
On 02/22/2016 06:43 PM, Michael Ströder wrote:
Whether LDAP result code attributeOrValueExists is treated as an error or just logged as "nothing to be seen here" is up to your code, isn't it?
Depends on a layer. The operation will always be logged as LDAP error response.
But it's up to your application whether a LDAP response code results in a warning/error log level message or a debug/info log level message.
Depends. It is possible to do as you suggest. But it may break code layering and encapsulation, ending up in a code that is difficult to maintain. But I understand that there are trade-offs to be made ...
The connector also does not know whether it is real error or not.
This sounds strange to me. Connectors I implement for my customers always know whether something is a real error or not and then it decides whether to abort or ignore.
The ConnId connectors are designed to be simple. There is a good reason for that: if most of the functionality is implemented in IDM it does not need to be reimplemented in each connector again and again. There is just one IDM, but there are many connectors (the connector API is in no way LDAP-specific). So the connectors are passing almost all errors to the upper layers to be handled by a common processing code.
Yes. I can do it entirely in the connector: handle the error, mute it, silently read the entry, recompute the change, modify the operation and retry it. But it is an ugly exception, it complicates the client code and it actually duplicates the code that is already implemented in the IDM. That is not good. Not good at all.
Well, that might be enough for web2ldap. But we are a "self-healing" system.
Self-healing just means that if anything goes seriously wrong in a connector it will try to reach the target state by doing a full sync during next run. Nothing special in your product.
That's not always feasible. Full re-synch of databases with millions of identities just takes too long. Even several hundred thousands can take a long time. And the state of the system changes during the resync. When one resync ends another may start again :-) Even a full sync of a single identity is not that easy. E.g. there are often attributes that are not managed by IDM, some are even source of authoritative data for IDM, some attributes even have several authoritative sources, some groups are assigned by IDM and some are not, etc. ... So there needs to be a continual and quite sophisticated process how to handle inconsistencies that are discovered anytime during normal operation. And midPoint has several mechanisms to handle that. E.g. if midPoint gets "object not found error" it consults the policies, re-creates the missing objects and then replays the operation. If midPoint gets "already exists" error it reads the entry, consults the policies and either links it to existing user or deletes it. Then restarts the operation. The handling of attributeOrValueExists error would be a good example of this "discovery" mechanism. Except that it is not. Because the error stops the operation that would fix the data if it is allowed to proceed. So the error just gets into the way without providing any benefit.
Not even in that case. E.g. see above. You will not get the error if you are re-adding a group that was deleted just a millisecond ago just because the network latencies haven't turned up in your favor.
Now this sounds suspicious to me. Are you using several MMR replicas to write to at the same time? If yes, here's some advice in good faith and without offense: DON'T DO THAT!
No, we are not writing to two masters at the same time. The same use case also applies to a single-server directory topology.
"Hey there! Maybe something wrong happened. Or maybe not. It may all be OK. There is no way to be sure. So forget it. I just wanted to talk to you. Sorry to bother you. And, by the way, your operation failed. Just for fun. Try something else. I won't tell you what. Go figure. Bye."
There's nothing wrong to log a debug/info message saying: "Ignored LDAP response code <foo> because of condition <bar>"
Once again: Ignoring the error is inconvenient, but it is not the primary problem. Primary problem is that the operation is stopped and unaffected changes are not executed. So it needs to be recomputed and retried. In my use case the error is useless and even harmful. It is great that there is a control to turn it off. And no, my client code is not broken.
--On Monday, February 22, 2016 3:29 PM +0100 Radovan Semancik radovan.semancik@evolveum.com wrote:
Right. Even though the situation is not that easy when going through abstractions such as JNDI
Wow, I'm surprised anyone is still using something as utterly broken as JNDI with LDAP. My bug about the fact JNDI can't even properly to 30 seconds is still open after over a decade. I'd strongly advise finding a different Java API for working with LDAP (We use UnboundID's LDAP API).
--Quanah
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
On 02/22/2016 05:57 PM, Quanah Gibson-Mount wrote:
Right. Even though the situation is not that easy when going through abstractions such as JNDI
Wow, I'm surprised anyone is still using something as utterly broken as JNDI with LDAP. My bug about the fact JNDI can't even properly to 30 seconds is still open after over a decade. I'd strongly advise finding a different Java API for working with LDAP (We use UnboundID's LDAP API).
Unfortunately yes. We have migrated to another API (Apache) almost completely. But there is still some legacy code from the old Sun times that is built on top of JNDI. It will take some time for this code to die out. And obviously JNDI is still somehow used in the wild. E.g. Spring LDAP library is built on top of JNDI.
--On Monday, February 22, 2016 10:00 PM +0100 Radovan Semancik radovan.semancik@evolveum.com wrote:
On 02/22/2016 05:57 PM, Quanah Gibson-Mount wrote:
Right. Even though the situation is not that easy when going through abstractions such as JNDI
Wow, I'm surprised anyone is still using something as utterly broken as JNDI with LDAP. My bug about the fact JNDI can't even properly to 30 seconds is still open after over a decade. I'd strongly advise finding a different Java API for working with LDAP (We use UnboundID's LDAP API).
Unfortunately yes. We have migrated to another API (Apache) almost completely. But there is still some legacy code from the old Sun times that is built on top of JNDI. It will take some time for this code to die out. And obviously JNDI is still somehow used in the wild. E.g. Spring LDAP library is built on top of JNDI.
Sounds like a good reason not to use Spring. ;)
--Quanah
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
openldap-technical@openldap.org
-
Howard Chu -
Michael Ströder -
Quanah Gibson-Mount -
Radovan Semancik