Hello,
I have configured three mirrors and am using a python script (ldapSynchCheck.py) which check the sync status on retrieving the contextCSN from both Provider and consumer
I tried deleting a tree from directory and checked what is the output of the script and can still see "All of them are in sync". I believe this is happening because contextCSN is not getting updated on the respective directories and hence it will showing all in sync.
However sometimes I am getting below error and it seems one of the consumer changed its contextCSN while provider was synching the data and it also confirms that provider will not change the contextCSN until it completes the sync to the consumer.
2010-05-06 15:15:51,459 - ldapSynchCheck.py - INFO - Provider is: tardis03.nat.bt.com:389
2010-05-06 15:15:51,462 - ldapSynchCheck.py - DEBUG - Connecting to ldap://tardis03.nat.bt.com:389
2010-05-06 15:15:51,464 - ldapSynchCheck.py - DEBUG - LDAP protocol version 3
2010-05-06 15:15:51,465 - ldapSynchCheck.py - DEBUG - Binding with o=BT
2010-05-06 15:15:51,469 - ldapSynchCheck.py - INFO - Checking if consumer tardis02.nat.bt.com:9011 is in SYNCH with provider
2010-05-06 15:15:51,470 - ldapSynchCheck.py - DEBUG - Connecting to ldap://tardis02.nat.bt.com:9011
2010-05-06 15:15:51,471 - ldapSynchCheck.py - DEBUG - LDAP protocol version 3
2010-05-06 15:15:51,472 - ldapSynchCheck.py - DEBUG - Binding with o=BT
2010-05-06 15:15:51,475 - ldapSynchCheck.py - DEBUG - Retrieving Provider contextCSN
2010-05-06 15:15:51,482 - ldapSynchCheck.py - DEBUG - contextCSN = 20100506141551.044268Z#000000#003#000000
2010-05-06 15:15:51,483 - ldapSynchCheck.py - DEBUG - Retrieving Consumer contextCSN
2010-05-06 15:15:51,513 - ldapSynchCheck.py - DEBUG - contextCSN = 20100506141551.044268Z#000000#003#000000
2010-05-06 15:15:51,515 - ldapSynchCheck.py - INFO - Provider and consumer exactly in SYNCH
2010-05-06 15:15:51,516 - ldapSynchCheck.py - INFO - Checking if consumer tardis01.nat.bt.com:9022 is in SYNCH with provider
2010-05-06 15:15:51,517 - ldapSynchCheck.py - DEBUG - Connecting to ldap://tardis01.nat.bt.com:9022
2010-05-06 15:15:51,519 - ldapSynchCheck.py - DEBUG - LDAP protocol version 3
2010-05-06 15:15:51,520 - ldapSynchCheck.py - DEBUG - Binding with o=BT
2010-05-06 15:15:51,523 - ldapSynchCheck.py - DEBUG - Retrieving Provider contextCSN
2010-05-06 15:15:51,525 - ldapSynchCheck.py - DEBUG - contextCSN = 20100506141551.044268Z#000000#003#000000
2010-05-06 15:15:51,526 - ldapSynchCheck.py - DEBUG - Retrieving Consumer contextCSN
2010-05-06 15:15:51,573 - ldapSynchCheck.py - DEBUG - contextCSN = 20100506141549.755004Z#000000#003#000000 <here it goes different>
Traceback (most recent call last):
File "./ldapSynchCheck.py", line 258, in <module>
main()
File "./ldapSynchCheck.py", line 253, in main
is_insynch(ldapprov, ldapcons, options.basedn, options.threshold, logger)
File "./ldapSynchCheck.py", line 188, in is_insynch
delta = contextCSN_to_datetime(provcontextCSN) - contextCSN_to_datetime(conscontextCSN)
File "./ldapSynchCheck.py", line 155, in contextCSN_to_datetime
return datetime.datetime.fromtimestamp(time.mktime(time.strptime(gentime,"%Y%m% d%H%M%S")))
File "/software/python/lib/python2.6/_strptime.py", line 454, in _strptime_time
return _strptime(data_string, format)[0]
File "/software/python/lib/python2.6/_strptime.py", line 328, in _strptime
data_string[found.end():])
ValueError: unconverted data remains: .044268
As I know putting a frequent checkpoint will generate a large amount of checkpoint log file and will cause issues while database recovery during crash/corrupt situations.
Can someone please suggest how make this more precise? Is there any other way contextCSN keeps on changing frequently?
Please suggest.
Many Thanks in advance!!
Rahul Manchanda
GS Selfcare
Platform Build Team
Tel: +91 02066018100 Extn: 6178
SMTP: rahul.manchanda@bt.com
Office: Sharda Center ,6th Floor Annex
Off Karve Road, Erandwana ,Pune-04
--On Thursday, May 06, 2010 4:37 PM +0100 rahul.manchanda@bt.com wrote:
However sometimes I am getting below error and it seems one of the consumer changed its contextCSN while provider was synching the data and it also confirms that provider will not change the contextCSN until it completes the sync to the consumer.
What OpenLDAP release?
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Hi,
It is 2.4.19.
Rahul Manchanda
GS Selfcare Platform Build Team
Tel: +91 02066018100 Extn: 6178 SMTP: rahul.manchanda@bt.com
Office: Sharda Center ,6th Floor Annex Off Karve Road, Erandwana ,Pune-04
-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Thursday, May 06, 2010 6:03 PM To: Manchanda,RK,Rahul,DKE C; openldap-technical@openldap.org Subject: Re: OpenLDAP: contextCSN check across mirrors
--On Thursday, May 06, 2010 4:37 PM +0100 rahul.manchanda@bt.com wrote:
However sometimes I am getting below error and it seems one of the consumer changed its contextCSN while provider was synching the data
and
it also confirms that provider will not change the contextCSN until it completes the sync to the consumer.
What OpenLDAP release?
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
--On Friday, May 07, 2010 7:50 AM +0100 rahul.manchanda@bt.com wrote:
Hi,
It is 2.4.19.
Please update your servers to 2.4.22 and see if it still occurs. Numerous fixes have occurred since the 2.4.19 release. I've noted some specific ones below that are likely of interest.
OpenLDAP 2.4.20 Release (2009/11/27) Added slapd syncrepl contextCSN storing in subentry (ITS#6373) Fixed slapd syncrepl deletes in MirrorMode (ITS#6368) Fixed slapd syncrepl to use correct SID (ITS#6367) Fixed slapo-syncprov checkpoint conversion (ITS#6370) Fixed slapo-syncprov deadlock (ITS#6335) Fixed slapo-syncprov memory leak (ITS#6376) Fixed slapo-syncprov out of order changes (ITS#6346) Fixed slapo-syncprov psearch with stale cookie (ITS#6397)
OpenLDAP 2.4.21 Release (2009/12/20) Fixed slapd syncrepl freeing tasks from queue (ITS#6413) Fixed slapd syncrepl parsing of tls defaults (ITS#6419) Fixed slapd syncrepl uninitialized variables (ITS#6425)
OpenLDAP 2.4.22 Release (2010/04/24) Fixed slapd-bdb contextCSN updates from updatedn (ITS#6469)
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Thanks for the reply.
"Added slapd syncrepl contextCSN storing in subentry (ITS#6373)". In this point what I can understand is that contextCSN will also get stored in suffix entries as well as to sub entries as well. Please correct if wrong.
I have some doubts if you can please help in making it more clear:
1) Can the contextCSN entry differ at the sub entry level if compared with suffix(root) entry level? 2) If this is the case then a check will be required to each and every entry if the sync is there at the dn level across mirrors and scripts need to be modified so as to put a loop till all the entries has been checked in that particular LDAP tree. Please correct if wrong. 3) how will the contextCSN entries will get changed. Does checkpoint plays a role in changing the contextCSN value or not? 4) provider will not change the contextCSN until the whole data get synched but will consumer will still be able to do so. Is this still the scenario in the new version?
Many Thanks in advance!!
Rahul Manchanda
GS Selfcare Platform Build Team
Tel: +91 02066018100 Extn: 6178 SMTP: rahul.manchanda@bt.com
Office: Sharda Center ,6th Floor Annex Off Karve Road, Erandwana ,Pune-04
-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Friday, May 07, 2010 4:48 PM To: Manchanda,RK,Rahul,DKE C; openldap-technical@openldap.org Subject: RE: OpenLDAP: contextCSN check across mirrors
--On Friday, May 07, 2010 7:50 AM +0100 rahul.manchanda@bt.com wrote:
Hi,
It is 2.4.19.
Please update your servers to 2.4.22 and see if it still occurs. Numerous fixes have occurred since the 2.4.19 release. I've noted some specific ones below that are likely of interest.
OpenLDAP 2.4.20 Release (2009/11/27) Added slapd syncrepl contextCSN storing in subentry (ITS#6373) Fixed slapd syncrepl deletes in MirrorMode (ITS#6368) Fixed slapd syncrepl to use correct SID (ITS#6367) Fixed slapo-syncprov checkpoint conversion (ITS#6370) Fixed slapo-syncprov deadlock (ITS#6335) Fixed slapo-syncprov memory leak (ITS#6376) Fixed slapo-syncprov out of order changes (ITS#6346) Fixed slapo-syncprov psearch with stale cookie (ITS#6397)
OpenLDAP 2.4.21 Release (2009/12/20) Fixed slapd syncrepl freeing tasks from queue (ITS#6413) Fixed slapd syncrepl parsing of tls defaults (ITS#6419) Fixed slapd syncrepl uninitialized variables (ITS#6425)
OpenLDAP 2.4.22 Release (2010/04/24) Fixed slapd-bdb contextCSN updates from updatedn (ITS#6469)
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
rahul.manchanda@bt.com wrote:
Thanks for the reply.
"Added slapd syncrepl contextCSN storing in subentry (ITS#6373)". In this point what I can understand is that contextCSN will also get stored in suffix entries as well as to sub entries as well. Please correct if wrong.
That's wrong. Read the manpage and/or the ITS for the actual behavior.
I have some doubts if you can please help in making it more clear:
- Can the contextCSN entry differ at the sub entry level if compared
with suffix(root) entry level?
It will only be in one place or the other, not both.
- If this is the case then a check will be required to each and every
entry if the sync is there at the dn level across mirrors and scripts need to be modified so as to put a loop till all the entries has been checked in that particular LDAP tree. Please correct if wrong.
No.
- how will the contextCSN entries will get changed. Does checkpoint
plays a role in changing the contextCSN value or not?
No.
- provider will not change the contextCSN until the whole data get
synched
False.
but will consumer will still be able to do so. Is this still the
scenario in the new version?
I don't understand the question.
Many Thanks in advance!!
Rahul Manchanda
GS Selfcare Platform Build Team
Tel: +91 02066018100 Extn: 6178 SMTP: rahul.manchanda@bt.com
Office: Sharda Center ,6th Floor Annex Off Karve Road, Erandwana ,Pune-04
-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Friday, May 07, 2010 4:48 PM To: Manchanda,RK,Rahul,DKE C; openldap-technical@openldap.org Subject: RE: OpenLDAP: contextCSN check across mirrors
--On Friday, May 07, 2010 7:50 AM +0100 rahul.manchanda@bt.com wrote:
Hi,
It is 2.4.19.
Please update your servers to 2.4.22 and see if it still occurs. Numerous fixes have occurred since the 2.4.19 release. I've noted some specific ones below that are likely of interest.
OpenLDAP 2.4.20 Release (2009/11/27) Added slapd syncrepl contextCSN storing in subentry (ITS#6373) Fixed slapd syncrepl deletes in MirrorMode (ITS#6368) Fixed slapd syncrepl to use correct SID (ITS#6367) Fixed slapo-syncprov checkpoint conversion (ITS#6370) Fixed slapo-syncprov deadlock (ITS#6335) Fixed slapo-syncprov memory leak (ITS#6376) Fixed slapo-syncprov out of order changes (ITS#6346) Fixed slapo-syncprov psearch with stale cookie (ITS#6397)
OpenLDAP 2.4.21 Release (2009/12/20) Fixed slapd syncrepl freeing tasks from queue (ITS#6413) Fixed slapd syncrepl parsing of tls defaults (ITS#6419) Fixed slapd syncrepl uninitialized variables (ITS#6425)
OpenLDAP 2.4.22 Release (2010/04/24) Fixed slapd-bdb contextCSN updates from updatedn (ITS#6469)
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
rahul.manchanda@bt.com wrote:
[..] 2010-05-06 15:15:51,573 - ldapSynchCheck.py - DEBUG - contextCSN = *20100506141549.755004Z*#000000#003#000000 <*here it goes different*>
*Traceback (most recent call last): * [..] datetime.datetime.fromtimestamp(time.mktime(time.strptime(gentime,"%Y%m%d%H%M%S")))
File "/software/python/lib/python2.6/_strptime.py", line 454, in
_strptime_time *
return _strptime(data_string, format)[0] *
File "/software/python/lib/python2.6/_strptime.py", line 328, in
_strptime *
- data_string[found.end():]) *
*ValueError: unconverted data remains: .044268 *
I think is rather a question about whether Python's standard lib function time.strptime() can parse the input data with the format specifier "%Y%m%d%H%M%".
So I'd recommend to extract the input data in variable gendata and ask with such a test case in news:comp.lang.python
Ciao, Michael.
openldap-technical@openldap.org
-
Howard Chu -
Michael Ströder -
Quanah Gibson-Mount -
rahul.manchanda＠bt.com