3:21 a.m.
Hi!
I have a question preparing a transition of syncrepl to delta-syncrepl:
Our database already has an accesslog overlay (olcAccessLogOps: writes) and a
corresponding database configured for monitoring purposes.
Now that delta syncreply also requires an accesslog overlay/database with somewhat
different setting compared to the existing database, I wonder whether it's possible
(and practically being used/proven to work) to have multiple accesslog overlays.
Thinking about it, I also wonder whether both needs for accesslog could be satisfied with
one accesslog database, just using different LDAP filters. I guess the delta syncreply is
using the acesslog database in a read-only manner (our monitoring also does), so it might
actually work. Anybody went that way before?
Finally a question on the requirement to backup the acesslog database in addition to the
corresponding database (18.2.1: "Note: since the database state is stored in both the
changelog DB and the main DB on the provider, it is
important to backup/restore both the changelog DB and the main DB using slapcat/slapadd
when restoring a
DB or copying it to another machine."): Is it actually a hard requirement needed for
consistency, or will it just improve the performance (avoiding a full refresh)?
Also: Isn't it tricky to backup both databases consistently while the database is
writable?
Regards,
Ulrich
8:54 a.m.
--On Monday, August 19, 2019 1:21 PM +0200 Ulrich Windl
<Ulrich.Windl(a)rz.uni-regensburg.de> wrote:
Hi!
I have a question preparing a transition of syncrepl to delta-syncrepl:
Our database already has an accesslog overlay (olcAccessLogOps: writes)
and a corresponding database configured for monitoring purposes.
You haven't noted the value for olcAccessLogSuccess. Generally for
delta-syncrepl *only* databases, this is set to TRUE. However, that's not
mandatory (see the end for why).
Now that
delta syncreply also requires an accesslog overlay/database with somewhat
different setting compared to the existing database, I wonder whether
it's possible (and practically being used/proven to work) to have
multiple accesslog overlays.
Since each overlay instantiation specifically defines which accesslog
database to log to, yes you can do this, but...
Thinking about it, I also wonder whether both needs for accesslog
could
be satisfied with one accesslog database, just using different LDAP
filters. I guess the delta syncreply is using the acesslog database in a
read-only manner (our monitoring also does), so it might actually work.
The accesslog DB is read-only by definition. But yes, as long as writes
are being logged, then you could use it for delta-syncrepl. The filter on
the consumer side already limits it to only consume successful write ops:
logfilter="(&(objectClass=auditWriteObject)(reqResult=0))"
Regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
4:19 a.m.
On Mon, Aug 19, 2019 at 08:54:43 -0700, Quanah Gibson-Mount wrote:
--On Monday, August 19, 2019 1:21 PM +0200 Ulrich Windl wrote:
> I have a question preparing a transition of syncrepl to delta-syncrepl:
> Our database already has an accesslog overlay (olcAccessLogOps: writes)
> and a corresponding database configured for monitoring purposes.
You haven't noted the value for olcAccessLogSuccess. Generally for
delta-syncrepl *only* databases, this is set to TRUE. However, that's not
mandatory (see the end for why).
Incidentally, we just toggled olcAccessLogSuccess to FALSE on a database
last week, to debug a provisioning issue where the client did not log
enough details. But our failed writes were still not logged.
Turns out that writes failed by an overlay (slapo-unique in our case) are
only logged to the accesslog, if this overlay is defined in the config
BEFORE the accesslog overlay (as overlays are loaded on a stack, see topic
12.18 in the admin guide). So we now put the accesslog overlay as last
instead of first.
Geert
--
geert.hendrickx.be :: geert(a)hendrickx.be :: PGP: 0xC4BB9E9F
This e-mail was composed using 100% recycled spam messages!
6:27 a.m.
New subject: Antw: Re: Q: Multiple accesslog overlays?
>> Geert Hendrickx <geert(a)hendrickx.be> schrieb am
22.08.2019 um 13:19 in
Nachricht <20190822111900.GA6110(a)vera.ghen.be>:
On Mon, Aug 19, 2019 at 08:54:43 ‑0700, Quanah Gibson‑Mount wrote:
> ‑‑On Monday, August 19, 2019 1:21 PM +0200 Ulrich Windl wrote:
> > I have a question preparing a transition of syncrepl to delta‑syncrepl:
> > Our database already has an accesslog overlay (olcAccessLogOps: writes)
> > and a corresponding database configured for monitoring purposes.
>
> You haven't noted the value for olcAccessLogSuccess. Generally for
> delta‑syncrepl *only* databases, this is set to TRUE. However, that's not
> mandatory (see the end for why).
Incidentally, we just toggled olcAccessLogSuccess to FALSE on a database
last week, to debug a provisioning issue where the client did not log
enough details. But our failed writes were still not logged.
Turns out that writes failed by an overlay (slapo‑unique in our case) are
only logged to the accesslog, if this overlay is defined in the config
BEFORE the accesslog overlay (as overlays are loaded on a stack, see topic
12.18 in the admin guide). So we now put the accesslog overlay as last
instead of first.
Makes sense as the error has to "go through" the overlay, I guess. Anyway a
reminder in the manual page would be a nice thing to have.
Geert
‑‑
geert.hendrickx.be :: geert(a)hendrickx.be :: PGP: 0xC4BB9E9F
This e‑mail was composed using 100% recycled spam messages!
6:32 a.m.
--On Thursday, August 22, 2019 2:19 PM +0200 Geert Hendrickx
<geert(a)hendrickx.be> wrote:
On Mon, Aug 19, 2019 at 08:54:43 -0700, Quanah Gibson-Mount wrote:
> --On Monday, August 19, 2019 1:21 PM +0200 Ulrich Windl wrote:
> > I have a question preparing a transition of syncrepl to delta-syncrepl:
> > Our database already has an accesslog overlay (olcAccessLogOps: writes)
> > and a corresponding database configured for monitoring purposes.
>
> You haven't noted the value for olcAccessLogSuccess. Generally for
> delta-syncrepl *only* databases, this is set to TRUE. However, that's
> not mandatory (see the end for why).
Incidentally, we just toggled olcAccessLogSuccess to FALSE on a database
last week, to debug a provisioning issue where the client did not log
enough details. But our failed writes were still not logged.
Turns out that writes failed by an overlay (slapo-unique in our case) are
only logged to the accesslog, if this overlay is defined in the config
BEFORE the accesslog overlay (as overlays are loaded on a stack, see topic
12.18 in the admin guide). So we now put the accesslog overlay as last
instead of first.
Hm, which is against best practices because it can cause other issues, IIRC
(which is why it's advised to have syncprov followed by accesslog, as the
first 2 overlays in the stack). This is something that needs redoing for
2.5.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
6:37 a.m.
On Thu, Aug 22, 2019 at 06:32:53 -0700, Quanah Gibson-Mount wrote:
> Turns out that writes failed by an overlay (slapo-unique in our
case) are
> only logged to the accesslog, if this overlay is defined in the config
> BEFORE the accesslog overlay (as overlays are loaded on a stack, see topic
> 12.18 in the admin guide). So we now put the accesslog overlay as last
> instead of first.
Hm, which is against best practices because it can cause other issues, IIRC
(which is why it's advised to have syncprov followed by accesslog, as the
first 2 overlays in the stack). This is something that needs redoing for
2.5.
What sort of issues Quanah? Do you advise to rollback?
We currently have the following order: syncprov, unique, accesslog.
Before it was: syncprov, accesslog unique.
Are these best practices you mention documented somewhere?
Geert
--
geert.hendrickx.be :: geert(a)hendrickx.be :: PGP: 0xC4BB9E9F
This e-mail was composed using 100% recycled spam messages!
8:07 a.m.
--On Thursday, August 22, 2019 4:37 PM +0200 Geert Hendrickx
<geert(a)hendrickx.be> wrote:
On Thu, Aug 22, 2019 at 06:32:53 -0700, Quanah Gibson-Mount wrote:
> > Turns out that writes failed by an overlay (slapo-unique in our case)
> > are only logged to the accesslog, if this overlay is defined in the
> > config BEFORE the accesslog overlay (as overlays are loaded on a
> > stack, see topic 12.18 in the admin guide). So we now put the
> > accesslog overlay as last instead of first.
>
> Hm, which is against best practices because it can cause other issues,
> IIRC (which is why it's advised to have syncprov followed by accesslog,
> as the first 2 overlays in the stack). This is something that needs
> redoing for 2.5.
What sort of issues Quanah? Do you advise to rollback?
We currently have the following order: syncprov, unique, accesslog.
Before it was: syncprov, accesslog unique.
Are these best practices you mention documented somewhere?
As you noted, overlays can intercept write operations. The problem is when
an overlay intercepts a write op, where the write op occurs in the primary
DB, but returns before accesslog is called, meaning that accesslog does not
record the write op. This then breaks consistency between the servers.
Since you're logging failures, you hit a different case, which generally
underscores why this entire processing stack needs rewriting for 2.5.
I don't believe this is currently documented as an issue (we don't have any
document really addressing overlay ordering at all that I can find).
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
11:09 a.m.
On Thu, Aug 22, 2019 at 08:07:53 -0700, Quanah Gibson-Mount wrote:
As you noted, overlays can intercept write operations. The problem
is when
an overlay intercepts a write op, where the write op occurs in the primary
DB, but returns before accesslog is called, meaning that accesslog does not
record the write op. This then breaks consistency between the servers.
Since you're logging failures, you hit a different case, which generally
underscores why this entire processing stack needs rewriting for 2.5.
Can that really happen between unique and accesslog overlays? The unique
overlay doesn't write any data itself, it will only reject certain writes?
The actual write to the underlying mdb database will only occur after it
passed through the entire overlay chain, or am I misunderstanding how it
works?
Geert
--
geert.hendrickx.be :: geert(a)hendrickx.be :: PGP: 0xC4BB9E9F
This e-mail was composed using 100% recycled spam messages!
11:22 a.m.
--On Thursday, August 22, 2019 9:09 PM +0200 Geert Hendrickx
<geert(a)hendrickx.be> wrote:
On Thu, Aug 22, 2019 at 08:07:53 -0700, Quanah Gibson-Mount wrote:
> As you noted, overlays can intercept write operations. The problem is
> when an overlay intercepts a write op, where the write op occurs in the
> primary DB, but returns before accesslog is called, meaning that
> accesslog does not record the write op. This then breaks consistency
> between the servers. Since you're logging failures, you hit a different
> case, which generally underscores why this entire processing stack needs
> rewriting for 2.5.
Can that really happen between unique and accesslog overlays? The unique
overlay doesn't write any data itself, it will only reject certain writes?
The actual write to the underlying mdb database will only occur after it
passed through the entire overlay chain, or am I misunderstanding how it
works?
I believe you're fine with unique, it's other ones like ppolicy and
memberOf where they have to be after accesslog IIRC.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
12:21 a.m.
New subject: Antw: Re: Q: Multiple accesslog overlays?
>> Quanah Gibson-Mount <quanah(a)symas.com> schrieb am
22.08.2019 um 20:22 in
Nachricht <A04BCE7561957BE4FFD75F7C(a)[192.168.1.144]>:
‑‑On Thursday, August 22, 2019 9:09 PM +0200 Geert Hendrickx
<geert(a)hendrickx.be> wrote:
> On Thu, Aug 22, 2019 at 08:07:53 ‑0700, Quanah Gibson‑Mount wrote:
>> As you noted, overlays can intercept write operations. The problem is
>> when an overlay intercepts a write op, where the write op occurs in the
>> primary DB, but returns before accesslog is called, meaning that
>> accesslog does not record the write op. This then breaks consistency
>> between the servers. Since you're logging failures, you hit a different
>> case, which generally underscores why this entire processing stack needs
>> rewriting for 2.5.
>
>
> Can that really happen between unique and accesslog overlays? The unique
> overlay doesn't write any data itself, it will only reject certain writes?
>
> The actual write to the underlying mdb database will only occur after it
> passed through the entire overlay chain, or am I misunderstanding how it
> works?
I believe you're fine with unique, it's other ones like ppolicy and
memberOf where they have to be after accesslog IIRC.
So this would be the wrong order (this is not real syntax, but just the file
names from a config dump, the LDIF files defining the corresponding overlay)?
olcOverlay={0}syncprov.ldif
olcOverlay={1}accesslog.ldif
olcOverlay={2}ppolicy.ldif
Regards,
Ulrich
‑‑Quanah
‑‑
Quanah Gibson‑Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
6:53 a.m.
New subject: Antw: Re: Q: Multiple accesslog overlays?
--On Monday, August 26, 2019 10:21 AM +0200 Ulrich Windl
<Ulrich.Windl(a)rz.uni-regensburg.de> wrote:
>>> Quanah Gibson-Mount <quanah(a)symas.com> schrieb am
22.08.2019 um 20:22
>>> in
Nachricht <A04BCE7561957BE4FFD75F7C(a)[192.168.1.144]>:
> I believe you're fine with unique, it's other ones like ppolicy and
> memberOf where they have to be after accesslog IIRC.
So this would be the wrong order (this is not real syntax, but just the
file names from a config dump, the LDIF files defining the corresponding
overlay)? olcOverlay={0}syncprov.ldif
olcOverlay={1}accesslog.ldif
olcOverlay={2}ppolicy.ldif
In that case, you have ppolicy after accesslog, which would be correct.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
512
days inactive
520
days old
openldap-technical@openldap.org
10 comments
3 participants
participants (3)
-
Geert Hendrickx -
Quanah Gibson-Mount -
Ulrich Windl