Hi,
I have an OpenLDAP server I'm upgrading from 2.4.58 to 2.6.0. The server is configured with the syncprov plugin, with "olcSpNoPresent: TRUE".
Previously, on 2.4.58, we would never see any Present phase data originating from the server. However, post upgrade, we are seeing this, and it's causing problems with the consumers. I've tried looking at the code, but I must admit I got lost trying to trace through the propagation of this flag.
Are there undocumented circumstances where olcSpNoPresent will not be obeyed, or is this a regression?
Thanks!
Lisa
--On Tuesday, March 8, 2022 10:46 AM +0000 lisa.parratt@bjss.com wrote:
Hi,
I have an OpenLDAP server I'm upgrading from 2.4.58 to 2.6.0. The server is configured with the syncprov plugin, with "olcSpNoPresent: TRUE".
Hello,
You'd have to better describe your replication setup. The *only* time no-present should be set to TRUE is when it's on top of an accesslog database that's being used by delta-syncrepl, as explicitly stated in the man page.
Regards, Quanah
Hi Quanah,
It’s not a replication setup. The consumer is a relatively ancient proprietary implementation of the syncrepl protocol, over which I have very influence.
I believe the problem is syncprov.c lines 3335 and 3341, which appear to be setting `do_present = SS_PRESENT;` while simultaneously ignoring the value of si->si_nopres. Replacing the lines above with `if ( !si->si_nopres ) do_present = SS_PRESENT;` appears to fix the problem for my use case, and is closer to how the logic worked in previous versions of slapd.
Thanks!
Lisa
From: Quanah Gibson-Mount quanah@fast-mail.org Date: Tuesday, 8 March 2022 at 18:19 To: Lisa Parratt Lisa.Parratt@bjss.com, openldap-technical@openldap.org openldap-technical@openldap.org Subject: Re: syncprov olcSpNoPresent not being honoured
--On Tuesday, March 8, 2022 10:46 AM +0000 lisa.parratt@bjss.com wrote:
Hi,
I have an OpenLDAP server I'm upgrading from 2.4.58 to 2.6.0. The server is configured with the syncprov plugin, with "olcSpNoPresent: TRUE".
Hello,
You'd have to better describe your replication setup. The *only* time no-present should be set to TRUE is when it's on top of an accesslog database that's being used by delta-syncrepl, as explicitly stated in the man page.
Regards, Quanah
The information included in this email and any files transmitted with it may contain information that is confidential and it must not be used by, or its contents or attachments copied or disclosed to, persons other than the intended addressee. If you have received this email in error, please notify BJSS. In the absence of written agreement to the contrary BJSS' relevant standard terms of contract for any work to be undertaken will apply. Please carry out virus or such other checks as you consider appropriate in respect of this email. BJSS does not accept responsibility for any adverse effect upon your system or data in relation to this email or any files transmitted with it. BJSS Limited, a company registered in England and Wales (Company Number 2777575), VAT Registration Number 613295452, Registered Office Address, 1 Whitehall Quay, Leeds, LS1 4HR
On Wed, Mar 09, 2022 at 09:04:49AM +0000, Lisa Parratt wrote:
Hi Quanah,
It’s not a replication setup. The consumer is a relatively ancient proprietary implementation of the syncrepl protocol, over which I have very influence.
I believe the problem is syncprov.c lines 3335 and 3341, which appear to be setting `do_present = SS_PRESENT;` while simultaneously ignoring the value of si->si_nopres. Replacing the lines above with `if ( !si->si_nopres ) do_present = SS_PRESENT;` appears to fix the problem for my use case, and is closer to how the logic worked in previous versions of slapd.
Hi Lisa, I find it strange that you maintain a sessionlog (or use accesslog to host it) yet don't want to host a present/delete phase which is where they are useful. Could you explain what you're trying to achieve with your set up?
Hi Ondřej,
A byzantine java application connects to the LDAP server, and somehow uses the information presented to trigger API requests to business logic, eventually resulting in the data being replicated to a relational database. If olcSpNoPresent is honoured, this works. It it’s ignored, it dies with an error. I have, as yet, been unable to find an engineer who can explain this behaviour to me, and this code is largely forgotten.
It seems reasonable to me that if I configure slapd to not do a Present phase, then it shouldn’t do a Present phase? Am I wrong?
Thanks!
Lisa
From: Ondřej Kuzník ondra@mistotebe.net Date: Wednesday, 9 March 2022 at 16:32 To: Lisa Parratt Lisa.Parratt@bjss.com Cc: Quanah Gibson-Mount quanah@fast-mail.org, openldap-technical@openldap.org openldap-technical@openldap.org Subject: Re: syncprov olcSpNoPresent not being honoured On Wed, Mar 09, 2022 at 09:04:49AM +0000, Lisa Parratt wrote:
Hi Quanah,
It’s not a replication setup. The consumer is a relatively ancient proprietary implementation of the syncrepl protocol, over which I have very influence.
I believe the problem is syncprov.c lines 3335 and 3341, which appear to be setting `do_present = SS_PRESENT;` while simultaneously ignoring the value of si->si_nopres. Replacing the lines above with `if ( !si->si_nopres ) do_present = SS_PRESENT;` appears to fix the problem for my use case, and is closer to how the logic worked in previous versions of slapd.
Hi Lisa, I find it strange that you maintain a sessionlog (or use accesslog to host it) yet don't want to host a present/delete phase which is where they are useful. Could you explain what you're trying to achieve with your set up?
-- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP
The information included in this email and any files transmitted with it may contain information that is confidential and it must not be used by, or its contents or attachments copied or disclosed to, persons other than the intended addressee. If you have received this email in error, please notify BJSS. In the absence of written agreement to the contrary BJSS' relevant standard terms of contract for any work to be undertaken will apply. Please carry out virus or such other checks as you consider appropriate in respect of this email. BJSS does not accept responsibility for any adverse effect upon your system or data in relation to this email or any files transmitted with it. BJSS Limited, a company registered in England and Wales (Company Number 2777575), VAT Registration Number 613295452, Registered Office Address, 1 Whitehall Quay, Leeds, LS1 4HR
On Wed, Mar 09, 2022 at 04:42:17PM +0000, Lisa Parratt wrote:
Hi Ondřej,
A byzantine java application connects to the LDAP server, and somehow uses the information presented to trigger API requests to business logic, eventually resulting in the data being replicated to a relational database. If olcSpNoPresent is honoured, this works. It it’s ignored, it dies with an error. I have, as yet, been unable to find an engineer who can explain this behaviour to me, and this code is largely forgotten.
It seems reasonable to me that if I configure slapd to not do a Present phase, then it shouldn’t do a Present phase? Am I wrong?
The question was, is there a reason why you also configure a sessionlog when to me it seems you're saying you'd be perfectly happy without (ignoring any deletes)?
Hi Ondřej,
My understanding was a sessionlog was required to record delete events, to efficiently generate a Delete phase? The server has been configured with `olcSpSessionlog: 10000` since long before I inherited it, and the consumer seems to be processing the transmitted Delete phase as expected.
Lisa
From: Ondřej Kuzník ondra@mistotebe.net Date: Wednesday, 9 March 2022 at 19:23 To: Lisa Parratt Lisa.Parratt@bjss.com Cc: Quanah Gibson-Mount quanah@fast-mail.org, openldap-technical@openldap.org openldap-technical@openldap.org Subject: Re: syncprov olcSpNoPresent not being honoured On Wed, Mar 09, 2022 at 04:42:17PM +0000, Lisa Parratt wrote:
Hi Ondřej,
A byzantine java application connects to the LDAP server, and somehow uses the information presented to trigger API requests to business logic, eventually resulting in the data being replicated to a relational database. If olcSpNoPresent is honoured, this works. It it’s ignored, it dies with an error. I have, as yet, been unable to find an engineer who can explain this behaviour to me, and this code is largely forgotten.
It seems reasonable to me that if I configure slapd to not do a Present phase, then it shouldn’t do a Present phase? Am I wrong?
The question was, is there a reason why you also configure a sessionlog when to me it seems you're saying you'd be perfectly happy without (ignoring any deletes)?
-- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP
The information included in this email and any files transmitted with it may contain information that is confidential and it must not be used by, or its contents or attachments copied or disclosed to, persons other than the intended addressee. If you have received this email in error, please notify BJSS. In the absence of written agreement to the contrary BJSS' relevant standard terms of contract for any work to be undertaken will apply. Please carry out virus or such other checks as you consider appropriate in respect of this email. BJSS does not accept responsibility for any adverse effect upon your system or data in relation to this email or any files transmitted with it. BJSS Limited, a company registered in England and Wales (Company Number 2777575), VAT Registration Number 613295452, Registered Office Address, 1 Whitehall Quay, Leeds, LS1 4HR
On Thu, Mar 10, 2022 at 09:48:40AM +0000, Lisa Parratt wrote:
Hi Ondřej,
My understanding was a sessionlog was required to record delete events, to efficiently generate a Delete phase? The server has been configured with `olcSpSessionlog: 10000` since long before I inherited it, and the consumer seems to be processing the transmitted Delete phase as expected.
Hi Lisa, it is required for that, but the fact your sessionlog isn't enough means you had already been missing deletes and setting olcSpNoPresent just hid that from view.
It could be argued that olcSpNoPresent and sessionlog are mutually exclusive and we should reject that (or at least warn about it being very suspicious), feel free to open an issue for this.
You might want to use olcSpSessionlogSource instead instead of the in-memory implementation to make sure you don't get in this situation even after a server restart. Maybe also keep olcSpNoPresent: FALSE to alert you when you'd be losing information already and need to handle that manually.
Regards,
openldap-technical@openldap.org
-
Lisa Parratt -
lisa.parratt＠bjss.com
-
Ondřej Kuzník -
Quanah Gibson-Mount