asserts and manadatory build instructions (was ITS#8240)
by Michael Ströder
hyc(a)symas.com wrote in ITS#8240:
> Our patch response was too hasty. There is no OpenLDAP bug here, the real
> issue is production binaries being built with asserts enabled instead of
> compiling with -DNDEBUG. That's an issue for packagers and distros to resolve.
> Closing this ITS, not an OpenLDAP bug.
Maybe I missed something. But this is the first time I've heard about -DNDEBUG
being mandatory when compiling binary packages for production use. Does it
have other effects?
And what are general rules for assert statements in OpenLDAP code?
In my own (Python) code assert statements are supposed to be only triggered if
something goes wrong *internally* (type issues etc.). If somebody manages to
trigger an assert statement with invalid input from "outside" I always
consider this to be a serious bug revealing insufficient error handling even
though e.g. web2ldap just logs the exception but won't crash. YMMV, but please
clarify.
I also wonder whether there are more mandatory rules for building packages and
where I can find them.
Please don't get me wrong: My inquiry is in good faith to avoid unnecessary
ITS based on misunderstanding.
Ciao, Michael.
2 years, 1 month
Re: Persistent sessionlog
by Howard Chu
Ondřej Kuzník wrote:
> ITS#8486 suggests we use a more efficient structure to maintain the
> sessionlog in. If we're messing with sessionlog already, we might as
> well see if we can address another issue - it is always empty on slapd
> startup leading to unnecessary full refreshes happening.
>
> slapo-accesslog has most of the data we need to support that and is
> already sorted in CSN order (much like sessionlog).
>
> AFAIK, we can't use the accesslog database directly as the database as
> we can't efficiently search on a single serverID to get the serverID
> set and the oldest CSN for each.
We could tweak the overlay to always maintain these in the parent entry
(auditContainer). Currently the logpurge always sets the container's entryCSN
to the oldest remaining CSN.
> There are a few tasks that need to be done in order to achieve this:
> - configure syncprov with a suffix that contains the slapo-accesslog
> style logs for our DB
> - change struct sessionlog to use a more efficient structure that can be
> iterated over from any point (only tavl is available at the moment)
We've talked about this before, an in-memory B+tree would be better for all of
our AVL/TAVL uses.
> - on startup:
> - iterate through the *last* N entries (filtering on successful write
> ops that affect our suffix) and build slog_entry for each of those
> - for each entry, insert a new slog_entry and update sl_mincsn
> - add a control to hint the database that we require the database to
> iterate from the end backward (back-[mhb]db can support this)
> - update accesslog to log entryUUID for the entry that has just been
> written
> - update the test suite to exercise the new failure conditions
>
> There are some caveats to this still:
> - if we aren't guaranteed to receive the accesslog entries in reverse
> CSN order, the resulting sessionlog would be quite unsafe to use, we
> have to try and detect this and start with an empty sessionlog
> instead, resetting sl_mincsn set to match the database contextCSN
> - We might find an accesslog entry we can't use (modification that doesn't
> have enough information), we should still be able to use whatever we
> built until then, but can't continue
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
6 years, 1 month
ITS#6097 MMR problems with deletes
by Howard Chu
One of the RedHat talks at LDAPCon noted that OpenLDAP can get inconsistent
when conflicting Adds and Deletes are issued. We've had this issue outstanding
in ITS since 2009, it's probably time to address it.
http://www.openldap.org/its/index.cgi/Development?id=6097
The strategy I outlined in that ITS may not be the best one, we need to
evaluate if that approach still makes sense.
This also touches on a discussion of ITS#7052 back in 2012.
http://www.openldap.org/lists/openldap-devel/201202/msg00010.html
We committed a couple changes to fix this, but one of them subsequently had to
be reverted. One of the points raised in there was to fully serialize all
writes so they always complete in CSN order, and always propagate in CSN
order. I believe this is also the approach Leo Yuriev took in ReOpenLDAP. We
should look into doing the same. In practice we've already got serialization
occurring in accesslog and now in back-mdb too, so we're mostly there already.
The syncrepl consumer also needs to be fully serialized, to simplify conflict
resolution when conflicting writes are being received from multiple masters at
once.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
6 years, 1 month
Re: Persistent sessionlog
by Howard Chu
Ondřej Kuzník wrote:
> On Tue, Oct 24, 2017 at 01:43:21PM +0200, Ondřej Kuzník wrote:
>> ITS#8486 suggests we use a more efficient structure to maintain the
>> sessionlog in. If we're messing with sessionlog already, we might as
>> well see if we can address another issue - it is always empty on slapd
>> startup leading to unnecessary full refreshes happening.
>>
>> slapo-accesslog has most of the data we need to support that and is
>> already sorted in CSN order (much like sessionlog).
>>
>> AFAIK, we can't use the accesslog database directly as the database as
>> we can't efficiently search on a single serverID to get the serverID
>> set and the oldest CSN for each.
>>
>> There are a few tasks that need to be done in order to achieve this:
>> [...]
>> - update accesslog to log entryUUID for the entry that has just been
>> written
>> [...]
>
> Ah, slapo-accesslog already seems to do this, however it is not
> documented yet.
>
Yeah, looks like Ando added this in ITS#6656.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
6 years, 1 month
Persistent sessionlog
by Ondřej Kuzník
ITS#8486 suggests we use a more efficient structure to maintain the
sessionlog in. If we're messing with sessionlog already, we might as
well see if we can address another issue - it is always empty on slapd
startup leading to unnecessary full refreshes happening.
slapo-accesslog has most of the data we need to support that and is
already sorted in CSN order (much like sessionlog).
AFAIK, we can't use the accesslog database directly as the database as
we can't efficiently search on a single serverID to get the serverID
set and the oldest CSN for each.
There are a few tasks that need to be done in order to achieve this:
- configure syncprov with a suffix that contains the slapo-accesslog
style logs for our DB
- change struct sessionlog to use a more efficient structure that can be
iterated over from any point (only tavl is available at the moment)
- on startup:
- iterate through the *last* N entries (filtering on successful write
ops that affect our suffix) and build slog_entry for each of those
- for each entry, insert a new slog_entry and update sl_mincsn
- add a control to hint the database that we require the database to
iterate from the end backward (back-[mhb]db can support this)
- update accesslog to log entryUUID for the entry that has just been
written
- update the test suite to exercise the new failure conditions
There are some caveats to this still:
- if we aren't guaranteed to receive the accesslog entries in reverse
CSN order, the resulting sessionlog would be quite unsafe to use, we
have to try and detect this and start with an empty sessionlog
instead, resetting sl_mincsn set to match the database contextCSN
- We might find an accesslog entry we can't use (modification that doesn't
have enough information), we should still be able to use whatever we
built until then, but can't continue
--
Ondřej Kuzník
Senior Software Engineer
Symas Corporation http://www.symas.com
Packaged, certified, and supported LDAP solutions powered by OpenLDAP
6 years, 1 month
Re: kqueue in OpenLDAP for FreeBSD (fwd)
by Quanah Gibson-Mount
Looks like there's a little bit more work necessary for kqueue support to
function correctly.
--Quanah
------------ Forwarded Message ------------
Date: Monday, October 23, 2017 12:43 AM -0700
From: Xin Li <delphij(a)delphij.net>
To: Quanah Gibson-Mount <quanah(a)symas.com>, Xin LI <delphij(a)gmail.com>
Cc: d(a)delphij.net, Pietro Cerutti <gahr(a)gahr.ch>, Xin LI
<delphij(a)freebsd.org>
Subject: Re: kqueue in OpenLDAP for FreeBSD
Hi, Quanah,
I finally got some time to twiddle with OpenLDAP (again). It looks like
the EBADF is expected, because fork(2) says:
• The child process has its own copy of the parent's
descriptors, except for descriptors returned by
kqueue(2), which are not inherited from the parent
process.
And slapd does have a fork() after the initial kqueue(), which rendered
it invalid.
The kqueue() is part of SLAP_SOCK_INIT(), which is part of
slapd_daemon_init(). Then, after that, fork() happen in lutil_detach(),
and doing this hack would (incorrectly) make slapd to start:
diff --git a/libraries/liblutil/detach.c b/libraries/liblutil/detach.c
index c6464c038..f47d36548 100644
--- a/libraries/liblutil/detach.c
+++ b/libraries/liblutil/detach.c
@@ -73,7 +73,7 @@ lutil_detach( int debug, int do_close )
#ifdef HAVE_THR
pid = fork1();
#else
- pid = fork();
+ pid = rfork(RFPROC);
#endif
switch ( pid )
{
It is incorrect because the code in slapd/main.c seems to expect the
child to write a "1" to it before exiting with EXIT_SUCCESS() and
obviously if the two processes shares the same file table, the parent
would consider the start was failed because read() would fail.
I think the right fix would be to move the lutil_detach() to before
slapd_daemon_init(), see attachment, but it seems that some code after
the initial daemon initialization is still trying to output to stderr, etc.
What do you think?
Cheers,
---------- End Forwarded Message ----------
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
6 years, 1 month
Re: LMDB: issue with mdb_cursor_del
by Howard Chu
timur.kristof(a)gmail.com wrote:
> Hi,
>
> I have an app that uses LMDB, and I've experienced an interesting
> issue: when trying to delete a certain item with mdb_cursor_del, it
> crashed with the following backtrace: https://pastebin.com/7p9wtkj9
>
> It appears that it couldn't mark a page as dirty.
> Here is the relevant assertion from mdb_page_dirty:
> rc = insert(txn->mt_u.dirty_list, &mid);
> mdb_tassert(txn, rc == 0); // assertion failed
>
> What might I be doing wrong in my application that triggers this sort
> of error?
Take a look at the value of rc, then look in midl.c. Most likely the dirty
list is too big, which means you're trying to do too much in a single transaction.
> Also interesting:
> The database was about 60MB, and I now compacted it using mdb_copy -c.
> Now it is only ~6MB, and running the app with the compacted database,
> the above error also disappeared.
>
> I'm not sure why the compacting fixed the problem, could somebody offer
> an insight about this?
>
> Thank you guys in advance for your answers!
> Best regards,
> Timur
>
>
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
6 years, 1 month
LMDB: issue with mdb_cursor_del
by timur.kristof＠gmail.com
Hi,
I have an app that uses LMDB, and I've experienced an interesting
issue: when trying to delete a certain item with mdb_cursor_del, it
crashed with the following backtrace: https://pastebin.com/7p9wtkj9
It appears that it couldn't mark a page as dirty.
Here is the relevant assertion from mdb_page_dirty:
rc = insert(txn->mt_u.dirty_list, &mid);
mdb_tassert(txn, rc == 0); // assertion failed
What might I be doing wrong in my application that triggers this sort
of error?
Also interesting:
The database was about 60MB, and I now compacted it using mdb_copy -c.
Now it is only ~6MB, and running the app with the compacted database,
the above error also disappeared.
I'm not sure why the compacting fixed the problem, could somebody offer
an insight about this?
Thank you guys in advance for your answers!
Best regards,
Timur
6 years, 1 month
Re: ITS review 9/29/2017
by Quanah Gibson-Mount
--On Thursday, October 12, 2017 1:03 AM +0200 Michael Ströder
<michael(a)stroeder.com> wrote:
> Quanah Gibson-Mount wrote:
>> -----------------------
>> Purely for RE25:
>> -----------------------
>
> What about ITS#8714?
> RFE: Sendout EXTENDED operation message in back-sock
This is already in master. Anything currently in master will be part of
RE25, unless it is behind LDAP_DEVEL.
> Another one which really strikes me:
> (ITS#7796) LDAP_DEBUG_TRACE for "not indexed" log messages
> With Æ-DIR's set-based ACLs *lots* of stupid "not indexed" messages are
> sent to syslog.
I'll add it to my list of items to look at for 2.5. There has not been an
exhaustive pass through of the ITS system for items to add to 2.5. I'm
going through it in batches.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
6 years, 1 month
Re: ITS review 9/29/2017
by Quanah Gibson-Mount
--On Friday, October 06, 2017 5:43 PM -0700 Quanah Gibson-Mount
<quanah(a)symas.com> wrote:
>> This is debatable:
>>
>> 1. OpenLDAP 2.4.x accepts modify operations with LDAP_MOD_INCREMENT
>>
>> 2. OpenLDAP 2.4.x sends modify operations via back-sock to external
>> listeners
>>
>> Therefore I'd consider this rather a fix than a new feature.
>
> Convince Howard. ;)
This is now in RE24 for the 2.4.46 release.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
6 years, 1 month
