Hi!
I have a question preparing a transition of syncrepl to delta-syncrepl: Our database already has an accesslog overlay (olcAccessLogOps: writes) and a corresponding database configured for monitoring purposes. Now that delta syncreply also requires an accesslog overlay/database with somewhat different setting compared to the existing database, I wonder whether it's possible (and practically being used/proven to work) to have multiple accesslog overlays.
Thinking about it, I also wonder whether both needs for accesslog could be satisfied with one accesslog database, just using different LDAP filters. I guess the delta syncreply is using the acesslog database in a read-only manner (our monitoring also does), so it might actually work. Anybody went that way before?
Finally a question on the requirement to backup the acesslog database in addition to the corresponding database (18.2.1: "Note: since the database state is stored in both the changelog DB and the main DB on the provider, it is important to backup/restore both the changelog DB and the main DB using slapcat/slapadd when restoring a DB or copying it to another machine."): Is it actually a hard requirement needed for consistency, or will it just improve the performance (avoiding a full refresh)? Also: Isn't it tricky to backup both databases consistently while the database is writable?
Regards, Ulrich
--On Monday, August 19, 2019 1:21 PM +0200 Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de wrote:
Hi!
I have a question preparing a transition of syncrepl to delta-syncrepl: Our database already has an accesslog overlay (olcAccessLogOps: writes) and a corresponding database configured for monitoring purposes.
You haven't noted the value for olcAccessLogSuccess. Generally for delta-syncrepl *only* databases, this is set to TRUE. However, that's not mandatory (see the end for why).
Now that delta syncreply also requires an accesslog overlay/database with somewhat different setting compared to the existing database, I wonder whether it's possible (and practically being used/proven to work) to have multiple accesslog overlays.
Since each overlay instantiation specifically defines which accesslog database to log to, yes you can do this, but...
Thinking about it, I also wonder whether both needs for accesslog could be satisfied with one accesslog database, just using different LDAP filters. I guess the delta syncreply is using the acesslog database in a read-only manner (our monitoring also does), so it might actually work.
The accesslog DB is read-only by definition. But yes, as long as writes are being logged, then you could use it for delta-syncrepl. The filter on the consumer side already limits it to only consume successful write ops:
logfilter="(&(objectClass=auditWriteObject)(reqResult=0))"
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
On Mon, Aug 19, 2019 at 08:54:43 -0700, Quanah Gibson-Mount wrote:
--On Monday, August 19, 2019 1:21 PM +0200 Ulrich Windl wrote:
I have a question preparing a transition of syncrepl to delta-syncrepl: Our database already has an accesslog overlay (olcAccessLogOps: writes) and a corresponding database configured for monitoring purposes.
You haven't noted the value for olcAccessLogSuccess. Generally for delta-syncrepl *only* databases, this is set to TRUE. However, that's not mandatory (see the end for why).
Incidentally, we just toggled olcAccessLogSuccess to FALSE on a database last week, to debug a provisioning issue where the client did not log enough details. But our failed writes were still not logged.
Turns out that writes failed by an overlay (slapo-unique in our case) are only logged to the accesslog, if this overlay is defined in the config BEFORE the accesslog overlay (as overlays are loaded on a stack, see topic 12.18 in the admin guide). So we now put the accesslog overlay as last instead of first.
Geert
Geert Hendrickx geert@hendrickx.be schrieb am 22.08.2019 um 13:19 in
Nachricht 20190822111900.GA6110@vera.ghen.be:
On Mon, Aug 19, 2019 at 08:54:43 ‑0700, Quanah Gibson‑Mount wrote:
‑‑On Monday, August 19, 2019 1:21 PM +0200 Ulrich Windl wrote:
I have a question preparing a transition of syncrepl to delta‑syncrepl: Our database already has an accesslog overlay (olcAccessLogOps: writes) and a corresponding database configured for monitoring purposes.
You haven't noted the value for olcAccessLogSuccess. Generally for delta‑syncrepl *only* databases, this is set to TRUE. However, that's not mandatory (see the end for why).
Incidentally, we just toggled olcAccessLogSuccess to FALSE on a database last week, to debug a provisioning issue where the client did not log enough details. But our failed writes were still not logged.
Turns out that writes failed by an overlay (slapo‑unique in our case) are only logged to the accesslog, if this overlay is defined in the config BEFORE the accesslog overlay (as overlays are loaded on a stack, see topic 12.18 in the admin guide). So we now put the accesslog overlay as last instead of first.
Makes sense as the error has to "go through" the overlay, I guess. Anyway a reminder in the manual page would be a nice thing to have.
Geert
‑‑ geert.hendrickx.be :: geert@hendrickx.be :: PGP: 0xC4BB9E9F This e‑mail was composed using 100% recycled spam messages!
--On Thursday, August 22, 2019 2:19 PM +0200 Geert Hendrickx geert@hendrickx.be wrote:
On Mon, Aug 19, 2019 at 08:54:43 -0700, Quanah Gibson-Mount wrote:
--On Monday, August 19, 2019 1:21 PM +0200 Ulrich Windl wrote:
I have a question preparing a transition of syncrepl to delta-syncrepl: Our database already has an accesslog overlay (olcAccessLogOps: writes) and a corresponding database configured for monitoring purposes.
You haven't noted the value for olcAccessLogSuccess. Generally for delta-syncrepl *only* databases, this is set to TRUE. However, that's not mandatory (see the end for why).
Incidentally, we just toggled olcAccessLogSuccess to FALSE on a database last week, to debug a provisioning issue where the client did not log enough details. But our failed writes were still not logged.
Turns out that writes failed by an overlay (slapo-unique in our case) are only logged to the accesslog, if this overlay is defined in the config BEFORE the accesslog overlay (as overlays are loaded on a stack, see topic 12.18 in the admin guide). So we now put the accesslog overlay as last instead of first.
Hm, which is against best practices because it can cause other issues, IIRC (which is why it's advised to have syncprov followed by accesslog, as the first 2 overlays in the stack). This is something that needs redoing for 2.5.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
On Thu, Aug 22, 2019 at 06:32:53 -0700, Quanah Gibson-Mount wrote:
Turns out that writes failed by an overlay (slapo-unique in our case) are only logged to the accesslog, if this overlay is defined in the config BEFORE the accesslog overlay (as overlays are loaded on a stack, see topic 12.18 in the admin guide). So we now put the accesslog overlay as last instead of first.
Hm, which is against best practices because it can cause other issues, IIRC (which is why it's advised to have syncprov followed by accesslog, as the first 2 overlays in the stack). This is something that needs redoing for 2.5.
What sort of issues Quanah? Do you advise to rollback?
We currently have the following order: syncprov, unique, accesslog.
Before it was: syncprov, accesslog unique.
Are these best practices you mention documented somewhere?
Geert
--On Thursday, August 22, 2019 4:37 PM +0200 Geert Hendrickx geert@hendrickx.be wrote:
On Thu, Aug 22, 2019 at 06:32:53 -0700, Quanah Gibson-Mount wrote:
Turns out that writes failed by an overlay (slapo-unique in our case) are only logged to the accesslog, if this overlay is defined in the config BEFORE the accesslog overlay (as overlays are loaded on a stack, see topic 12.18 in the admin guide). So we now put the accesslog overlay as last instead of first.
Hm, which is against best practices because it can cause other issues, IIRC (which is why it's advised to have syncprov followed by accesslog, as the first 2 overlays in the stack). This is something that needs redoing for 2.5.
What sort of issues Quanah? Do you advise to rollback?
We currently have the following order: syncprov, unique, accesslog.
Before it was: syncprov, accesslog unique.
Are these best practices you mention documented somewhere?
As you noted, overlays can intercept write operations. The problem is when an overlay intercepts a write op, where the write op occurs in the primary DB, but returns before accesslog is called, meaning that accesslog does not record the write op. This then breaks consistency between the servers. Since you're logging failures, you hit a different case, which generally underscores why this entire processing stack needs rewriting for 2.5.
I don't believe this is currently documented as an issue (we don't have any document really addressing overlay ordering at all that I can find).
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
On Thu, Aug 22, 2019 at 08:07:53 -0700, Quanah Gibson-Mount wrote:
As you noted, overlays can intercept write operations. The problem is when an overlay intercepts a write op, where the write op occurs in the primary DB, but returns before accesslog is called, meaning that accesslog does not record the write op. This then breaks consistency between the servers. Since you're logging failures, you hit a different case, which generally underscores why this entire processing stack needs rewriting for 2.5.
Can that really happen between unique and accesslog overlays? The unique overlay doesn't write any data itself, it will only reject certain writes?
The actual write to the underlying mdb database will only occur after it passed through the entire overlay chain, or am I misunderstanding how it works?
Geert
--On Thursday, August 22, 2019 9:09 PM +0200 Geert Hendrickx geert@hendrickx.be wrote:
On Thu, Aug 22, 2019 at 08:07:53 -0700, Quanah Gibson-Mount wrote:
As you noted, overlays can intercept write operations. The problem is when an overlay intercepts a write op, where the write op occurs in the primary DB, but returns before accesslog is called, meaning that accesslog does not record the write op. This then breaks consistency between the servers. Since you're logging failures, you hit a different case, which generally underscores why this entire processing stack needs rewriting for 2.5.
Can that really happen between unique and accesslog overlays? The unique overlay doesn't write any data itself, it will only reject certain writes?
The actual write to the underlying mdb database will only occur after it passed through the entire overlay chain, or am I misunderstanding how it works?
I believe you're fine with unique, it's other ones like ppolicy and memberOf where they have to be after accesslog IIRC.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Quanah Gibson-Mount quanah@symas.com schrieb am 22.08.2019 um 20:22 in
Nachricht <A04BCE7561957BE4FFD75F7C@[192.168.1.144]>:
‑‑On Thursday, August 22, 2019 9:09 PM +0200 Geert Hendrickx geert@hendrickx.be wrote:
On Thu, Aug 22, 2019 at 08:07:53 ‑0700, Quanah Gibson‑Mount wrote:
As you noted, overlays can intercept write operations. The problem is when an overlay intercepts a write op, where the write op occurs in the primary DB, but returns before accesslog is called, meaning that accesslog does not record the write op. This then breaks consistency between the servers. Since you're logging failures, you hit a different case, which generally underscores why this entire processing stack needs rewriting for 2.5.
Can that really happen between unique and accesslog overlays? The unique overlay doesn't write any data itself, it will only reject certain writes?
The actual write to the underlying mdb database will only occur after it passed through the entire overlay chain, or am I misunderstanding how it works?
I believe you're fine with unique, it's other ones like ppolicy and memberOf where they have to be after accesslog IIRC.
So this would be the wrong order (this is not real syntax, but just the file names from a config dump, the LDIF files defining the corresponding overlay)? olcOverlay={0}syncprov.ldif olcOverlay={1}accesslog.ldif olcOverlay={2}ppolicy.ldif
Regards, Ulrich
‑‑Quanah
‑‑
Quanah Gibson‑Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
--On Monday, August 26, 2019 10:21 AM +0200 Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de wrote:
Quanah Gibson-Mount quanah@symas.com schrieb am 22.08.2019 um 20:22 in
Nachricht <A04BCE7561957BE4FFD75F7C@[192.168.1.144]>:
I believe you're fine with unique, it's other ones like ppolicy and memberOf where they have to be after accesslog IIRC.
So this would be the wrong order (this is not real syntax, but just the file names from a config dump, the LDIF files defining the corresponding overlay)? olcOverlay={0}syncprov.ldif olcOverlay={1}accesslog.ldif olcOverlay={2}ppolicy.ldif
In that case, you have ppolicy after accesslog, which would be correct.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
openldap-technical@openldap.org
-
Geert Hendrickx -
Quanah Gibson-Mount -
Ulrich Windl