Hi, I am facing a problem with OpenLDAP-2.4.23. That is I am trying to setup a delta replication based on accesslog. Although the accesslog database contains new entries and the log database is successfully read, no replication ever takes place, while a setup without accesslog replicates successfully. An excerpt from the systemlogs:
slapd[19170]: conn=1050 fd=20 ACCEPT from IP=192.168.0.8:41487 (IP=0.0.0.0:389) slapd[19170]: conn=1050 op=0 EXT oid=1.3.6.1.4.1.1466.20037 slapd[19170]: conn=1050 op=0 STARTTLS slapd[19170]: conn=1050 op=0 RESULT oid= err=0 text= slapd[19170]: conn=1050 fd=20 TLS established tls_ssf=256 ssf=256 slapd[19170]: conn=1050 op=1 BIND dn="" method=163 slapd[19170]: conn=1050 op=1 BIND authcid="cn=replicator,o=avci,c=de" authzid="cn=replicator,o=avci,c=de" slapd[19170]: conn=1050 op=1 BIND dn="cn=replicator,o=avci,c=de" mech=EXTERNAL sasl_ssf=0 ssf=256 slapd[19170]: conn=1050 op=1 RESULT tag=97 err=0 text= slapd[19170]: conn=1050 op=2 SRCH base="cn=log" scope=2 deref=0 filter="(&(objectClass=auditWriteObject)(reqResult=0))" slapd[19170]: conn=1050 op=2 SRCH attr=reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN slapd[19170]: conn=1050 op=2 SEARCH RESULT tag=101 err=0 nentries=7 text= slapd[19170]: conn=1050 op=3 UNBIND slapd[19170]: conn=1050 fd=20 closed
A ldapsearch -Y EXTERNAL -ZZ -H ldap://localhost -b cn=log -s one "(&(objectClass=auditWriteObject)(reqResult=0))" provides the expected results, an example
# 20101129144212.000003Z, log dn: reqStart=20101129144212.000003Z,cn=log objectClass: auditModify reqStart: 20101129144212.000003Z reqEnd: 20101129144212.000004Z reqType: modify reqSession: 1007 reqAuthzID: cn=admanager,o=avci,c=de reqControls: {0}{2.16.840.1.113730.3.4.18 controlValue "646O3K636O3N61646N616O 616765722M6P3N617663692M633N6465"} reqDN:: Y249U2Now6RmZXIgV29sZmdhbmcsb3U9YWRyZXNzYnVjaCxvPWF2Y2ksYz1kZQ== reqResult: 0 reqMod: sn:- Wolfgang reqMod: sn:+ Schaefer reqMod: entryCSN:= 20101129144212.353910Z#000000#000#000000 reqMod: modifiersName:= cn=admanager,o=avci,c=de reqMod: modifyTimestamp:= 20101129144212Z
The relevant config files can be found here: Provider slapd.conf: http://pastebin.de/12273 Consumer slapd.conf: http://pastebin.de/12275
-Dieter
--On Monday, November 29, 2010 6:31 PM +0100 Dieter Klünter dieter@dkluenter.de wrote:
Hi, I am facing a problem with OpenLDAP-2.4.23. That is I am trying to setup a delta replication based on accesslog. Although the accesslog database contains new entries and the log database is successfully read, no replication ever takes place, while a setup without accesslog replicates successfully. An excerpt from the systemlogs:
delta-sync replication works great for me with 2.4.23. I would suggest you define fewer things in your replica configuration. In specific, I would drop:
logfilter syncdata scope
I would also add a new line between the syncrepl and updateref lines.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Hi Quanah,
On Mon, Nov 29, 2010 at 10:38:16AM -0800, Quanah Gibson-Mount wrote:
--On Monday, November 29, 2010 6:31 PM +0100 Dieter Klünter dieter@dkluenter.de wrote:
Hi, I am facing a problem with OpenLDAP-2.4.23. That is I am trying to setup a delta replication based on accesslog. Although the accesslog database contains new entries and the log database is successfully read, no replication ever takes place, while a setup without accesslog replicates successfully. An excerpt from the systemlogs:
delta-sync replication works great for me with 2.4.23. I would suggest you define fewer things in your replica configuration. In specific, I would drop:
logfilter syncdata scope
After removing this paramaters, replication took place:
slapd[19170]: conn=3853 op=15 ADD dn="cn=xxxxxxxxxxxxxxx" slapd[19170]: slap_queue_csn: queing 0xb0b5789e 20101129201509.698643Z#000000#000#000000 slapd[19170]: syncprov_sendresp: cookie=rid=042,csn=20101129201509.698643Z#000000#000#000000 slapd[19170]: slap_queue_csn: queing 0xb0657370 20101129201509.698643Z#000000#000#000000 slapd[19170]: slap_graduate_commit_csn: removing 0xb7b024e0 20101129201509.698643Z#000000#000#000000
but I'm not sure, whether the consumer had read the accesslog database and replicated based on this information, because no access to cn=log has been logged.
-Dieter
--On Monday, November 29, 2010 9:31 PM +0100 Dieter Klünter dieter@dkluenter.de wrote:
Hi Quanah,
but I'm not sure, whether the consumer had read the accesslog database and replicated based on this information, because no access to cn=log has been logged.
If the replica has no data when it is started, then it will do a full refresh from the primary DB on the master, just like with normal syncrepl, and skip the log database entirely. Now that it is in sync, I would make a change and see if that gets replicated.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Dieter Klünter dieter@dkluenter.de writes:
Hi Quanah,
On Mon, Nov 29, 2010 at 10:38:16AM -0800, Quanah Gibson-Mount wrote:
--On Monday, November 29, 2010 6:31 PM +0100 Dieter Klünter dieter@dkluenter.de wrote:
Hi, I am facing a problem with OpenLDAP-2.4.23. That is I am trying to setup a delta replication based on accesslog. Although the accesslog database contains new entries and the log database is successfully read, no replication ever takes place, while a setup without accesslog replicates successfully. An excerpt from the systemlogs:
delta-sync replication works great for me with 2.4.23. I would suggest you define fewer things in your replica configuration. In specific, I would drop:
logfilter syncdata scope
After removing this paramaters, replication took place:
slapd[19170]: conn=3853 op=15 ADD dn="cn=xxxxxxxxxxxxxxx" slapd[19170]: slap_queue_csn: queing 0xb0b5789e 20101129201509.698643Z#000000#000#000000 slapd[19170]: syncprov_sendresp: cookie=rid=042,csn=20101129201509.698643Z#000000#000#000000 slapd[19170]: slap_queue_csn: queing 0xb0657370 20101129201509.698643Z#000000#000#000000 slapd[19170]: slap_graduate_commit_csn: removing 0xb7b024e0 20101129201509.698643Z#000000#000#000000
but I'm not sure, whether the consumer had read the accesslog database and replicated based on this information, because no access to cn=log has been logged.
It seems the consumer has not read the accesslog database, so without acesslog replication takes place, but with accesslog enabled, no replication. After adding the removed parameters, again no replication. The consumer logs (loglevel sync) are pested with:
slapd[2656]: do_syncrep2: rid=042 got search entry without Sync State control slapd[2656]: do_syncrepl: rid=042 rc -1 retrying (4 retries left) slapd[2656]: do_syncrep2: rid=042 got search entry without Sync State control slapd[2656]: do_syncrepl: rid=042 rc -1 retrying (4 retries left) slapd[2656]: do_syncrep2: rid=042 got search entry without Sync State control slapd[2656]: do_syncrepl: rid=042 rc -1 retrying (4 retries left)
Any idea what is going on?
-Dieter
--On Tuesday, November 30, 2010 6:04 PM +0100 Dieter Kluenter dieter@dkluenter.de wrote:
Any idea what is going on?
On the master, in the accesslog overlay section for the main DB, you're missing the following parameter:
logsuccess TRUE
On the master, in the accesslog database section, you are missing the syncprov overlay:
overlay syncprov syncprov-nopresent TRUE syncprov-reloadhint TRUE
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount quanah@zimbra.com writes:
--On Tuesday, November 30, 2010 6:04 PM +0100 Dieter Kluenter dieter@dkluenter.de wrote:
Any idea what is going on?
On the master, in the accesslog overlay section for the main DB, you're missing the following parameter:
logsuccess TRUE
I want to log positiv and negativ write operations
On the master, in the accesslog database section, you are missing the syncprov overlay:
overlay syncprov syncprov-nopresent TRUE syncprov-reloadhint TRUE
This I realised after running test043 too, but I wonder whether this is required at all, or is this just lack of documentation?
I am sorry for all this noise, mea culpa. As I mentioned, the OS on both systems is suse and as such, an administrator should take care of /etc/sysconfig/openldap, in particular the option OPENLDAP_CONFIG_BACKEND='ldap', that is, slapd is not reading slapd.conf but slapd.d
The system is working now as desired. Thank you Quanah.
-Dieter
--On Tuesday, November 30, 2010 8:33 PM +0100 Dieter Kluenter dieter@dkluenter.de wrote:
On the master, in the accesslog database section, you are missing the syncprov overlay:
overlay syncprov syncprov-nopresent TRUE syncprov-reloadhint TRUE
This I realised after running test043 too, but I wonder whether this is required at all, or is this just lack of documentation?
It is absolutely required, as the accesslog database is providing information to the syncrepl client, thus it must have the syncprov overlay in place.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount quanah@zimbra.com writes:
--On Tuesday, November 30, 2010 8:33 PM +0100 Dieter Kluenter dieter@dkluenter.de wrote:
On the master, in the accesslog database section, you are missing the syncprov overlay:
overlay syncprov syncprov-nopresent TRUE syncprov-reloadhint TRUE
This I realised after running test043 too, but I wonder whether this is required at all, or is this just lack of documentation?
It is absolutely required, as the accesslog database is providing information to the syncrepl client, thus it must have the syncprov overlay in place.
I am not convinced, as the consumer conducts a search operation on the log database and not a sync operation, unless the consumer maintains a local logdatabase. But others may clarify this.
-Dieter
--On Tuesday, November 30, 2010 9:23 PM +0100 Dieter Kluenter dieter@dkluenter.de wrote:
I am not convinced, as the consumer conducts a search operation on the log database and not a sync operation, unless the consumer maintains a local logdatabase. But others may clarify this.
There is replication information stored in the log database that is maintained by the syncprov overlay. Thus, it is required for it to be present. And that's why you were getting a very explicit error about the overlay not being present when it was searching the log database. But you can ignore your errors if you want.
Note the contextCSN stored in the accesslog DB:
# accesslog dn: cn=accesslog structuralObjectClass: auditContainer entryCSN: 20101123115530.669034Z#000000#000#000000 contextCSN: 20101130203425.474878Z#000000#000#000000 entryDN: cn=accesslog subschemaSubentry: cn=Subschema hasSubordinates: TRUE
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Hi Quanah,
Quanah Gibson-Mount quanah@zimbra.com writes:
--On Monday, November 29, 2010 6:31 PM +0100 Dieter Klünter dieter@dkluenter.de wrote:
Hi, I am facing a problem with OpenLDAP-2.4.23. That is I am trying to setup a delta replication based on accesslog. Although the accesslog database contains new entries and the log database is successfully read, no replication ever takes place, while a setup without accesslog replicates successfully. An excerpt from the systemlogs:
delta-sync replication works great for me with 2.4.23. I would suggest you define fewer things in your replica configuration. In specific, I would drop:
logfilter syncdata scope
Are you sure, that only a logbase statement as part of syncrepl statements as reference to delta synchronization is sufficient?
-Dieter
--On Monday, November 29, 2010 10:05 PM +0100 Dieter Kluenter dieter@dkluenter.de wrote:
Hi Quanah,
Are you sure, that only a logbase statement as part of syncrepl statements as reference to delta synchronization is sufficient?
Egh, no, you're right. I have:
syncrepl rid=${ldap_replica_rid} provider=${ldap_master_url} retry="60 +" type=refreshAndPersist schemachecking=off searchbase="" bindmethod=simple binddn=uid=zmreplica,cn=admins,cn=zimbra credentials=@@ldap_replication_password@@ logbase="cn=accesslog" logfilter="(&(objectClass=auditWriteObject)(reqResult=0))" syncdata="accesslog"
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
openldap-technical@openldap.org
-
Dieter Kluenter -
Dieter Klünter
-
Quanah Gibson-Mount