Hello,
I am writing a RFC4533 client implementation based on JLDAP. I have a question on how to interpret the rfc as a client, and secondly how the OpenLDAP server interprets it.
My question is: how can the client determine that it must delete content at the end of the refresh stage, when a refreshAndPersist is requested with an initial cookie?
The problem is that I can not defer from the rfc how I can differentiate between a server that sends an initial content poll and a server that sends a content update poll. The RFC specifies that the server may choose to ignore the initial cookie and the reloadHint flag, so either poll mode may be choosen by the server. In the case of a refreshOnly the RFC is clear: when refreshDeletes of syncDoneValue is FALSE, content that is not added, changed, or indicated as present must be deleted from the client copy (section 1.3.1 paragraph 8). However, in the case of a refreshAndPersist, there is no similar flag in the SyncInfoMessage that ends the refresh stage. SyncInfoMessage does have values named refreshDelete and refreshDeletes, but these are used for other purposes (see section 3.4.1 and section 3.3.2 paragraph 7 and 9).
Am I missing something?
What is the behavior of the OpenLDAP server in this matter? Does it ever ignore the reloadHint of the sync request control?
Regards, Erik.
-- Erik van Oosten http://2008.rubyenrails.nl/ http://www.day-to-day-stuff.blogspot.com/
Erik van Oosten wrote:
Hello,
I am writing a RFC4533 client implementation based on JLDAP. I have a question on how to interpret the rfc as a client, and secondly how the OpenLDAP server interprets it.
My question is: how can the client determine that it must delete content at the end of the refresh stage, when a refreshAndPersist is requested with an initial cookie?
The problem is that I can not defer from the rfc how I can differentiate between a server that sends an initial content poll and a server that sends a content update poll. The RFC specifies that the server may choose to ignore the initial cookie and the reloadHint flag, so either poll mode may be choosen by the server.
Your use of the words "client" and "server" seem inconsistent here. The above questions made no sense to me. Servers don't send polls.
In the case of a refreshOnly the RFC is clear: when refreshDeletes of syncDoneValue is FALSE, content that is not added, changed, or indicated as present must be deleted from the client copy (section 1.3.1 paragraph 8). However, in the case of a refreshAndPersist, there is no similar flag in the SyncInfoMessage that ends the refresh stage. SyncInfoMessage does have values named refreshDelete and refreshDeletes, but these are used for other purposes (see section 3.4.1 and section 3.3.2 paragraph 7 and 9).
Am I missing something?
If the server used a present phase, then it will send a SyncInfoMessage with refreshPresent. If it used a delete phase, it will send a SyncInfoMessage with refreshDelete. If the refresh is stage is complete, the message will have refreshDone set to TRUE.
If the client receives a SyncInfoMessage with refreshPresent and refreshDone set to TRUE that means there's only a present phase and no delete phase. Therefore, any entry that wasn't marked present or added must be deleted.
What is the behavior of the OpenLDAP server in this matter? Does it ever ignore the reloadHint of the sync request control?
In OpenLDAP 2.2 and early versions of 2.3 it always ignored the hint. In current versions you can configure it to use the hint.
Regards, Erik.
-- Erik van Oosten http://2008.rubyenrails.nl/ http://www.day-to-day-stuff.blogspot.com/
Thanks Howard,
Your use of the words "client" and "server" seem inconsistent here. The above questions made no sense to me. Servers don't send polls.
Sorry, my interpretation of RFC4533 terminology :) . What I read is that the client may select from 2 modes of operation (by providing an initial cookie). The first is described in the section called "initial content poll", the second is called "content update poll". The server may however ignore the request for a content update poll and do an "initial content poll".
I was writing about that the client can not reliably determine which mode was selected by the server. But I now see that it does not matter; at both the end of the "initial content poll" /and/ at the end of present phase (that is, without a delete phase) one should delete all unmentioned client content.
If the client receives a SyncInfoMessage with refreshPresent and refreshDone set to TRUE that means there's only a present phase and no delete phase. ...
Indeed. With the addition that this message is used to end both the "initial content poll" as the "content update poll".
... Therefore, any entry that wasn't marked present or added must be deleted.
Still, this interpretation can not be correct. When I run my sync client program against OpenLDAP (2.3.36), and then restart it after the refresh stage with the last received cookie (note: the DIT remains unchanged), the first message is exactly that SyncInfoMessage of refreshPresent with refreshDone set to TRUE. Using our interpretation I should now delete all client content! Since the DIT was not changed this is clearly wrong. (By the way the new server cookie is the same as the initial cookie, but depending on this is tricky; the client should never interpret the cookie.)
I am quite confused here. Where is the error in our reasoning?
Regards, Erik.
-- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
-- Erik van Oosten http://2008.rubyenrails.nl/ http://www.day-to-day-stuff.blogspot.com/
Erik van Oosten wrote:
Still, this interpretation can not be correct. When I run my sync client program against OpenLDAP (2.3.36), and then restart it after the refresh stage with the last received cookie (note: the DIT remains unchanged), the first message is exactly that SyncInfoMessage of refreshPresent with refreshDone set to TRUE. Using our interpretation I should now delete all client content! Since the DIT was not changed this is clearly wrong. (By the way the new server cookie is the same as the initial cookie, but depending on this is tricky; the client should never interpret the cookie.)
Are you still talking about refreshAndPersist mode, or refreshonly?
There appears to be a detail missing from the RFC - when no changes have occurred since the last refresh, for refreshOnly the provider will send a Sync Done refreshDeletes control with no cookie. I.e., if the state hasn't changed, the cookie would be identical, so there's no need to send it back to the consumer.
For refreshAndPersist it always sends the cookie back. I think this may be a bug, we probably shouldn't be sending the cookie in this case either. So, you *should* be able to assume that no changes have occurred at all when no cookie is returned.
I am quite confused here. Where is the error in our reasoning?
Regards, Erik.
Howard Chu wrote:
Erik van Oosten wrote:
Still, this interpretation can not be correct. When I run my sync client program against OpenLDAP (2.3.36), and then restart it after the refresh stage with the last received cookie (note: the DIT remains unchanged), the first message is exactly that SyncInfoMessage of refreshPresent with refreshDone set to TRUE. Using our interpretation I should now delete all client content! Since the DIT was not changed this is clearly wrong. (By the way the new server cookie is the same as the initial cookie, but depending on this is tricky; the client should never interpret the cookie.)
Are you still talking about refreshAndPersist mode, or refreshonly?
I am talking about a refreshAndPersist operation (otherwise I would not get a syncInfoMessage with refreshDone set to true). Mode refreshOnly is working as expected.
There appears to be a detail missing from the RFC - when no changes have occurred since the last refresh, for refreshOnly the provider will send a Sync Done refreshDeletes control with no cookie. I.e., if the state hasn't changed, the cookie would be identical, so there's no need to send it back to the consumer.
You say Sync Done refreshDeletes, that is indeed what you get with refreshOnly. However, with refreshAndPersist you get a single syncInfoMessage of refresh*Present*. Shall I report this as a bug?
My interpretation of 'no cookie' in the SyncDone/SyncInfoMessage was to use no initial cookie in the next synchronization. But your interpretation is probably better indeed.
For refreshAndPersist it always sends the cookie back. I think this may be a bug, we probably shouldn't be sending the cookie in this case either. So, you *should* be able to assume that no changes have occurred at all when no cookie is returned.
Okay thanks, I'll use that assumption. I'll also use the assumption that if the cookie is the same, no changes have occurred.
Thanks, Erik.
Erik van Oosten wrote:
There appears to be a detail missing from the RFC - when no changes have occurred since the last refresh, for refreshOnly the provider will send a Sync Done refreshDeletes control with no cookie. I.e., if the state hasn't changed, the cookie would be identical, so there's no need to send it back to the consumer.
You say Sync Done refreshDeletes, that is indeed what you get with refreshOnly. However, with refreshAndPersist you get a single syncInfoMessage of refresh*Present*. Shall I report this as a bug?
Please do, thanks.
My interpretation of 'no cookie' in the SyncDone/SyncInfoMessage was to use no initial cookie in the next synchronization. But your interpretation is probably better indeed.
For some reason I remember this being explicitly documented somewhere, but I can't find a reference now. Maybe Kurt will remember, if he's watching...
For refreshAndPersist it always sends the cookie back. I think this may be a bug, we probably shouldn't be sending the cookie in this case either. So, you *should* be able to assume that no changes have occurred at all when no cookie is returned.
Okay thanks, I'll use that assumption. I'll also use the assumption that if the cookie is the same, no changes have occurred.
Fair enough.
Howard,
Thanks for your support and the quick fix!
Regards, Erik.
PS. I filled a second, much smaller, syncrepl bug under ITS#5211.
Howard Chu wrote:
Erik van Oosten wrote:
Shall I report this as a bug?
Please do, thanks.
Erik van Oosten wrote:
Howard,
Thanks for your support and the quick fix!
You're welcome. And please post back here with more status on your syncrepl implementation, I believe you're one of the first outside the OpenLDAP Project.
Regards, Erik.
PS. I filled a second, much smaller, syncrepl bug under ITS#5211.
Thanks. Now fixed in HEAD.
Howard Chu wrote:
And please post back here with more status on your syncrepl implementation, I believe you're one of the first outside the OpenLDAP Project.
I am about done and quite happy with the result. I only need to write code to restart aborted syncs and some more code to manage many concurrent syncs in just a few threads.
I am afraid I am not allowed to open source this :( (I did ask). The effort was too large (about a week, but that excluded a lot of prior research), and the advantage compared to competitors in our market is just too large. I will however post a tutorial on extending JLDAP so that at least others can get a grasp on how to approach this. I'll post the link here when its done.
PS. I filled a second, much smaller, syncrepl bug under ITS#5211.
Thanks. Now fixed in HEAD.
Great!
Regards, Erik.
openldap-software@openldap.org
-
Erik van Oosten -
Howard Chu