openldap-devel

openldap-devel@openldap.org

1418 discussions

Re: Persistent sessionlog
by Howard Chu 26 Oct '17

26 Oct '17

Ondřej Kuzník wrote: > ITS#8486 suggests we use a more efficient structure to maintain the > sessionlog in. If we're messing with sessionlog already, we might as > well see if we can address another issue - it is always empty on slapd > startup leading to unnecessary full refreshes happening. > > slapo-accesslog has most of the data we need to support that and is > already sorted in CSN order (much like sessionlog). > > AFAIK, we can't use the accesslog database directly as the database as > we can't efficiently search on a single serverID to get the serverID > set and the oldest CSN for each. We could tweak the overlay to always maintain these in the parent entry (auditContainer). Currently the logpurge always sets the container's entryCSN to the oldest remaining CSN. > There are a few tasks that need to be done in order to achieve this: > - configure syncprov with a suffix that contains the slapo-accesslog > style logs for our DB > - change struct sessionlog to use a more efficient structure that can be > iterated over from any point (only tavl is available at the moment) We've talked about this before, an in-memory B+tree would be better for all of our AVL/TAVL uses. > - on startup: > - iterate through the *last* N entries (filtering on successful write > ops that affect our suffix) and build slog_entry for each of those > - for each entry, insert a new slog_entry and update sl_mincsn > - add a control to hint the database that we require the database to > iterate from the end backward (back-[mhb]db can support this) > - update accesslog to log entryUUID for the entry that has just been > written > - update the test suite to exercise the new failure conditions > > There are some caveats to this still: > - if we aren't guaranteed to receive the accesslog entries in reverse > CSN order, the resulting sessionlog would be quite unsafe to use, we > have to try and detect this and start with an empty sessionlog > instead, resetting sl_mincsn set to match the database contextCSN > - We might find an accesslog entry we can't use (modification that doesn't > have enough information), we should still be able to use whatever we > built until then, but can't continue > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 2

ITS#6097 MMR problems with deletes
by Howard Chu 24 Oct '17

24 Oct '17

One of the RedHat talks at LDAPCon noted that OpenLDAP can get inconsistent when conflicting Adds and Deletes are issued. We've had this issue outstanding in ITS since 2009, it's probably time to address it. http://www.openldap.org/its/index.cgi/Development?id=6097 The strategy I outlined in that ITS may not be the best one, we need to evaluate if that approach still makes sense. This also touches on a discussion of ITS#7052 back in 2012. http://www.openldap.org/lists/openldap-devel/201202/msg00010.html We committed a couple changes to fix this, but one of them subsequently had to be reverted. One of the points raised in there was to fully serialize all writes so they always complete in CSN order, and always propagate in CSN order. I believe this is also the approach Leo Yuriev took in ReOpenLDAP. We should look into doing the same. In practice we've already got serialization occurring in accesslog and now in back-mdb too, so we're mostly there already. The syncrepl consumer also needs to be fully serialized, to simplify conflict resolution when conflicting writes are being received from multiple masters at once. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: Persistent sessionlog
by Howard Chu 24 Oct '17

24 Oct '17

Ondřej Kuzník wrote: > On Tue, Oct 24, 2017 at 01:43:21PM +0200, Ondřej Kuzník wrote: >> ITS#8486 suggests we use a more efficient structure to maintain the >> sessionlog in. If we're messing with sessionlog already, we might as >> well see if we can address another issue - it is always empty on slapd >> startup leading to unnecessary full refreshes happening. >> >> slapo-accesslog has most of the data we need to support that and is >> already sorted in CSN order (much like sessionlog). >> >> AFAIK, we can't use the accesslog database directly as the database as >> we can't efficiently search on a single serverID to get the serverID >> set and the oldest CSN for each. >> >> There are a few tasks that need to be done in order to achieve this: >> [...] >> - update accesslog to log entryUUID for the entry that has just been >> written >> [...] > > Ah, slapo-accesslog already seems to do this, however it is not > documented yet. > Yeah, looks like Ando added this in ITS#6656. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Persistent sessionlog
by Ondřej Kuzník 24 Oct '17

24 Oct '17

ITS#8486 suggests we use a more efficient structure to maintain the sessionlog in. If we're messing with sessionlog already, we might as well see if we can address another issue - it is always empty on slapd startup leading to unnecessary full refreshes happening. slapo-accesslog has most of the data we need to support that and is already sorted in CSN order (much like sessionlog). AFAIK, we can't use the accesslog database directly as the database as we can't efficiently search on a single serverID to get the serverID set and the oldest CSN for each. There are a few tasks that need to be done in order to achieve this: - configure syncprov with a suffix that contains the slapo-accesslog style logs for our DB - change struct sessionlog to use a more efficient structure that can be iterated over from any point (only tavl is available at the moment) - on startup: - iterate through the *last* N entries (filtering on successful write ops that affect our suffix) and build slog_entry for each of those - for each entry, insert a new slog_entry and update sl_mincsn - add a control to hint the database that we require the database to iterate from the end backward (back-[mhb]db can support this) - update accesslog to log entryUUID for the entry that has just been written - update the test suite to exercise the new failure conditions There are some caveats to this still: - if we aren't guaranteed to receive the accesslog entries in reverse CSN order, the resulting sessionlog would be quite unsafe to use, we have to try and detect this and start with an empty sessionlog instead, resetting sl_mincsn set to match the database contextCSN - We might find an accesslog entry we can't use (modification that doesn't have enough information), we should still be able to use whatever we built until then, but can't continue -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

2 2

Re: kqueue in OpenLDAP for FreeBSD (fwd)
by Quanah Gibson-Mount 23 Oct '17

23 Oct '17

Looks like there's a little bit more work necessary for kqueue support to function correctly. --Quanah ------------ Forwarded Message ------------ Date: Monday, October 23, 2017 12:43 AM -0700 From: Xin Li <delphij(a)delphij.net> To: Quanah Gibson-Mount <quanah(a)symas.com>, Xin LI <delphij(a)gmail.com> Cc: d(a)delphij.net, Pietro Cerutti <gahr(a)gahr.ch>, Xin LI <delphij(a)freebsd.org> Subject: Re: kqueue in OpenLDAP for FreeBSD Hi, Quanah, I finally got some time to twiddle with OpenLDAP (again). It looks like the EBADF is expected, because fork(2) says: • The child process has its own copy of the parent's descriptors, except for descriptors returned by kqueue(2), which are not inherited from the parent process. And slapd does have a fork() after the initial kqueue(), which rendered it invalid. The kqueue() is part of SLAP_SOCK_INIT(), which is part of slapd_daemon_init(). Then, after that, fork() happen in lutil_detach(), and doing this hack would (incorrectly) make slapd to start: diff --git a/libraries/liblutil/detach.c b/libraries/liblutil/detach.c index c6464c038..f47d36548 100644 --- a/libraries/liblutil/detach.c +++ b/libraries/liblutil/detach.c @@ -73,7 +73,7 @@ lutil_detach( int debug, int do_close ) #ifdef HAVE_THR pid = fork1(); #else - pid = fork(); + pid = rfork(RFPROC); #endif switch ( pid ) { It is incorrect because the code in slapd/main.c seems to expect the child to write a "1" to it before exiting with EXIT_SUCCESS() and obviously if the two processes shares the same file table, the parent would consider the start was failed because read() would fail. I think the right fix would be to move the lutil_detach() to before slapd_daemon_init(), see attachment, but it seems that some code after the initial daemon initialization is still trying to output to stderr, etc. What do you think? Cheers, ---------- End Forwarded Message ---------- -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: LMDB: issue with mdb_cursor_del
by Howard Chu 20 Oct '17

20 Oct '17

timur.kristof(a)gmail.com wrote: > Hi, > > I have an app that uses LMDB, and I've experienced an interesting > issue: when trying to delete a certain item with mdb_cursor_del, it > crashed with the following backtrace: https://pastebin.com/7p9wtkj9 > > It appears that it couldn't mark a page as dirty. > Here is the relevant assertion from mdb_page_dirty: > rc = insert(txn->mt_u.dirty_list, &mid); > mdb_tassert(txn, rc == 0); // assertion failed > > What might I be doing wrong in my application that triggers this sort > of error? Take a look at the value of rc, then look in midl.c. Most likely the dirty list is too big, which means you're trying to do too much in a single transaction. > Also interesting: > The database was about 60MB, and I now compacted it using mdb_copy -c. > Now it is only ~6MB, and running the app with the compacted database, > the above error also disappeared. > > I'm not sure why the compacting fixed the problem, could somebody offer > an insight about this? > > Thank you guys in advance for your answers! > Best regards, > Timur > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

4 14

LMDB: issue with mdb_cursor_del
by timur.kristof＠gmail.com 16 Oct '17

16 Oct '17

Hi, I have an app that uses LMDB, and I've experienced an interesting issue: when trying to delete a certain item with mdb_cursor_del, it crashed with the following backtrace: https://pastebin.com/7p9wtkj9 It appears that it couldn't mark a page as dirty. Here is the relevant assertion from mdb_page_dirty: rc = insert(txn->mt_u.dirty_list, &mid); mdb_tassert(txn, rc == 0); // assertion failed What might I be doing wrong in my application that triggers this sort of error? Also interesting: The database was about 60MB, and I now compacted it using mdb_copy -c. Now it is only ~6MB, and running the app with the compacted database, the above error also disappeared. I'm not sure why the compacting fixed the problem, could somebody offer an insight about this? Thank you guys in advance for your answers! Best regards, Timur

1 0

Re: ITS review 9/29/2017
by Quanah Gibson-Mount 12 Oct '17

12 Oct '17

--On Thursday, October 12, 2017 1:03 AM +0200 Michael Ströder <michael(a)stroeder.com> wrote: > Quanah Gibson-Mount wrote: >> ----------------------- >> Purely for RE25: >> ----------------------- > > What about ITS#8714? > RFE: Sendout EXTENDED operation message in back-sock This is already in master. Anything currently in master will be part of RE25, unless it is behind LDAP_DEVEL. > Another one which really strikes me: > (ITS#7796) LDAP_DEBUG_TRACE for "not indexed" log messages > With Æ-DIR's set-based ACLs *lots* of stupid "not indexed" messages are > sent to syslog. I'll add it to my list of items to look at for 2.5. There has not been an exhaustive pass through of the ITS system for items to add to 2.5. I'm going through it in batches. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: ITS review 9/29/2017
by Quanah Gibson-Mount 12 Oct '17

12 Oct '17

--On Friday, October 06, 2017 5:43 PM -0700 Quanah Gibson-Mount <quanah(a)symas.com> wrote: >> This is debatable: >> >> 1. OpenLDAP 2.4.x accepts modify operations with LDAP_MOD_INCREMENT >> >> 2. OpenLDAP 2.4.x sends modify operations via back-sock to external >> listeners >> >> Therefore I'd consider this rather a fix than a new feature. > > Convince Howard. ;) This is now in RE24 for the 2.4.46 release. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

2 1

ITS review 9/29/2017
by Quanah Gibson-Mount 12 Oct '17

12 Oct '17

Howard -- Looking for a thumbs up/down on these. The following ITSes were mostly supplied with patches and are IPR ok. I've imported them into my openldap-scratch repo for easy review. If they didn't have a supplied patch, I did the related work. Branch names are by ITS. ----------------------- Suggested for LMDB 0.9: ----------------------- its8612 - Fix LMDB builds on Solaris & derivative OSes <https://github.com/quanah/openldap-scratch/tree/its6035> ----------------------- Suggested for RE24: ----------------------- its5048 - Fix documentation for entryCSN with syncprov <https://github.com/quanah/openldap-scratch/tree/its5048> its7100 - Fix slapo-dds with entryTTL currently not decreasing <https://github.com/quanah/openldap-scratch/tree/its7100> its7373 - Fix tls_session reuse when hostname check fails <https://github.com/quanah/openldap-scratch/tree/its7373> its7374 - Fix MozNSS file matching for hashed CA cert directory (RE24 ONLY) <https://github.com/quanah/openldap-scratch/tree/its7374> its7389 - Fix MozNSS to fallback to PEM if cert not found in certdb (RE24 ONLY) <https://github.com/quanah/openldap-scratch/tree/its7389> its7442 - Add debug statements when index_intlen values are out of range <https://github.com/quanah/openldap-scratch/tree/its7442> its7520 - Omit unknown schema option for back-ldap <https://github.com/quanah/openldap-scratch/tree/its7520> its8037 - Fix delta-syncrepl with relax <https://github.com/quanah/openldap-scratch/tree/its8037> its8121 - Add LDAP_SASL_SIMPLE to ldap_bind(3) <https://github.com/quanah/openldap-scratch/tree/its8121> its8167 - Fix nonblocking TLS with referrals <https://github.com/quanah/openldap-scratch/tree/its8167> its8404 - Fix assertion with back-meta when olcDbRewrite is changed <https://github.com/quanah/openldap-scratch/tree/its8404> its8578 - Remove unused variables in RE24 <https://github.com/quanah/openldap-scratch/tree/its8578> its8583 - Fix C++ LDAPCtrl structure <https://github.com/quanah/openldap-scratch/tree/its8583> its8605 - Fix various spelling errors <https://github.com/quanah/openldap-scratch/tree/its8605> its8687 - OpenSSL 1.1 compatibility, fix build when cross-compiling <https://github.com/quanah/openldap-scratch/tree/its8687> ----------------------- Purely for RE25: ----------------------- its6035 - Fix slapd so it doesn't require restart after modifying olcAuthzRegexp <https://github.com/quanah/openldap-scratch/tree/its6035> its6300 - Add support for kqueue <https://github.com/quanah/openldap-scratch/tree/its6300> its6475 - Add documentation to slapd.conf(5) and slapd-config(5) for SASLDONTUSECOPY <https://github.com/quanah/openldap-scratch/compare/its6475> its7042 - Make it possible to unset TLS options with syncrepl <https://github.com/quanah/openldap-scratch/tree/its7042> its7532 - Add ldap_connection function for asynchronous clients <https://github.com/quanah/openldap-scratch/tree/its7532> its7721 - Allow authTimestamp to be forwarded via updateref <https://github.com/quanah/openldap-scratch/tree/its7721> its8037 - Fixes delta-sync replication when "relax" is used to modify the structural OC of an entry <https://github.com/quanah/openldap-scratch/tree/its8037> its8153 - olcTimeLimit should be SINGLE-VALUE <https://github.com/quanah/openldap-scratch/tree/its8153> its8291 - Fix slapmodify with BDB backends <https://github.com/quanah/openldap-scratch/tree/its8291> its8508 - ucgendat.c properly add title-case characters without upper-case equivalents (e.g. greek letters with iota subscript) <https://github.com/quanah/openldap-scratch/tree/its8508> its8511 - Fix documentation for multimaster, deprecate mirrormode <https://github.com/quanah/openldap-scratch/tree/its8511> its8527 - Improve SYNC debug output in certain situations <https://github.com/quanah/openldap-scratch/tree/its8527> its8573 - Add TLS options to ldap* tools <https://github.com/quanah/openldap-scratch/tree/its8573-tables> its8692 - Support LDAP_MOD_INCREMENT with back-sock <https://github.com/quanah/openldap-scratch/tree/its8692> --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

2 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel