openldap-devel

openldap-devel@openldap.org

1416 discussions

Re: Persistent sessionlog
by Howard Chu 24 Oct '17

24 Oct '17

Ondřej Kuzník wrote: > On Tue, Oct 24, 2017 at 01:43:21PM +0200, Ondřej Kuzník wrote: >> ITS#8486 suggests we use a more efficient structure to maintain the >> sessionlog in. If we're messing with sessionlog already, we might as >> well see if we can address another issue - it is always empty on slapd >> startup leading to unnecessary full refreshes happening. >> >> slapo-accesslog has most of the data we need to support that and is >> already sorted in CSN order (much like sessionlog). >> >> AFAIK, we can't use the accesslog database directly as the database as >> we can't efficiently search on a single serverID to get the serverID >> set and the oldest CSN for each. >> >> There are a few tasks that need to be done in order to achieve this: >> [...] >> - update accesslog to log entryUUID for the entry that has just been >> written >> [...] > > Ah, slapo-accesslog already seems to do this, however it is not > documented yet. > Yeah, looks like Ando added this in ITS#6656. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Persistent sessionlog
by Ondřej Kuzník 24 Oct '17

24 Oct '17

ITS#8486 suggests we use a more efficient structure to maintain the sessionlog in. If we're messing with sessionlog already, we might as well see if we can address another issue - it is always empty on slapd startup leading to unnecessary full refreshes happening. slapo-accesslog has most of the data we need to support that and is already sorted in CSN order (much like sessionlog). AFAIK, we can't use the accesslog database directly as the database as we can't efficiently search on a single serverID to get the serverID set and the oldest CSN for each. There are a few tasks that need to be done in order to achieve this: - configure syncprov with a suffix that contains the slapo-accesslog style logs for our DB - change struct sessionlog to use a more efficient structure that can be iterated over from any point (only tavl is available at the moment) - on startup: - iterate through the *last* N entries (filtering on successful write ops that affect our suffix) and build slog_entry for each of those - for each entry, insert a new slog_entry and update sl_mincsn - add a control to hint the database that we require the database to iterate from the end backward (back-[mhb]db can support this) - update accesslog to log entryUUID for the entry that has just been written - update the test suite to exercise the new failure conditions There are some caveats to this still: - if we aren't guaranteed to receive the accesslog entries in reverse CSN order, the resulting sessionlog would be quite unsafe to use, we have to try and detect this and start with an empty sessionlog instead, resetting sl_mincsn set to match the database contextCSN - We might find an accesslog entry we can't use (modification that doesn't have enough information), we should still be able to use whatever we built until then, but can't continue -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

2 2

Re: kqueue in OpenLDAP for FreeBSD (fwd)
by Quanah Gibson-Mount 23 Oct '17

23 Oct '17

Looks like there's a little bit more work necessary for kqueue support to function correctly. --Quanah ------------ Forwarded Message ------------ Date: Monday, October 23, 2017 12:43 AM -0700 From: Xin Li <delphij(a)delphij.net> To: Quanah Gibson-Mount <quanah(a)symas.com>, Xin LI <delphij(a)gmail.com> Cc: d(a)delphij.net, Pietro Cerutti <gahr(a)gahr.ch>, Xin LI <delphij(a)freebsd.org> Subject: Re: kqueue in OpenLDAP for FreeBSD Hi, Quanah, I finally got some time to twiddle with OpenLDAP (again). It looks like the EBADF is expected, because fork(2) says: • The child process has its own copy of the parent's descriptors, except for descriptors returned by kqueue(2), which are not inherited from the parent process. And slapd does have a fork() after the initial kqueue(), which rendered it invalid. The kqueue() is part of SLAP_SOCK_INIT(), which is part of slapd_daemon_init(). Then, after that, fork() happen in lutil_detach(), and doing this hack would (incorrectly) make slapd to start: diff --git a/libraries/liblutil/detach.c b/libraries/liblutil/detach.c index c6464c038..f47d36548 100644 --- a/libraries/liblutil/detach.c +++ b/libraries/liblutil/detach.c @@ -73,7 +73,7 @@ lutil_detach( int debug, int do_close ) #ifdef HAVE_THR pid = fork1(); #else - pid = fork(); + pid = rfork(RFPROC); #endif switch ( pid ) { It is incorrect because the code in slapd/main.c seems to expect the child to write a "1" to it before exiting with EXIT_SUCCESS() and obviously if the two processes shares the same file table, the parent would consider the start was failed because read() would fail. I think the right fix would be to move the lutil_detach() to before slapd_daemon_init(), see attachment, but it seems that some code after the initial daemon initialization is still trying to output to stderr, etc. What do you think? Cheers, ---------- End Forwarded Message ---------- -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: LMDB: issue with mdb_cursor_del
by Howard Chu 20 Oct '17

20 Oct '17

timur.kristof(a)gmail.com wrote: > Hi, > > I have an app that uses LMDB, and I've experienced an interesting > issue: when trying to delete a certain item with mdb_cursor_del, it > crashed with the following backtrace: https://pastebin.com/7p9wtkj9 > > It appears that it couldn't mark a page as dirty. > Here is the relevant assertion from mdb_page_dirty: > rc = insert(txn->mt_u.dirty_list, &mid); > mdb_tassert(txn, rc == 0); // assertion failed > > What might I be doing wrong in my application that triggers this sort > of error? Take a look at the value of rc, then look in midl.c. Most likely the dirty list is too big, which means you're trying to do too much in a single transaction. > Also interesting: > The database was about 60MB, and I now compacted it using mdb_copy -c. > Now it is only ~6MB, and running the app with the compacted database, > the above error also disappeared. > > I'm not sure why the compacting fixed the problem, could somebody offer > an insight about this? > > Thank you guys in advance for your answers! > Best regards, > Timur > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

4 14

LMDB: issue with mdb_cursor_del
by timur.kristof＠gmail.com 16 Oct '17

16 Oct '17

Hi, I have an app that uses LMDB, and I've experienced an interesting issue: when trying to delete a certain item with mdb_cursor_del, it crashed with the following backtrace: https://pastebin.com/7p9wtkj9 It appears that it couldn't mark a page as dirty. Here is the relevant assertion from mdb_page_dirty: rc = insert(txn->mt_u.dirty_list, &mid); mdb_tassert(txn, rc == 0); // assertion failed What might I be doing wrong in my application that triggers this sort of error? Also interesting: The database was about 60MB, and I now compacted it using mdb_copy -c. Now it is only ~6MB, and running the app with the compacted database, the above error also disappeared. I'm not sure why the compacting fixed the problem, could somebody offer an insight about this? Thank you guys in advance for your answers! Best regards, Timur

1 0

Re: ITS review 9/29/2017
by Quanah Gibson-Mount 11 Oct '17

11 Oct '17

--On Thursday, October 12, 2017 1:03 AM +0200 Michael Ströder <michael(a)stroeder.com> wrote: > Quanah Gibson-Mount wrote: >> ----------------------- >> Purely for RE25: >> ----------------------- > > What about ITS#8714? > RFE: Sendout EXTENDED operation message in back-sock This is already in master. Anything currently in master will be part of RE25, unless it is behind LDAP_DEVEL. > Another one which really strikes me: > (ITS#7796) LDAP_DEBUG_TRACE for "not indexed" log messages > With Æ-DIR's set-based ACLs *lots* of stupid "not indexed" messages are > sent to syslog. I'll add it to my list of items to look at for 2.5. There has not been an exhaustive pass through of the ITS system for items to add to 2.5. I'm going through it in batches. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: ITS review 9/29/2017
by Quanah Gibson-Mount 11 Oct '17

11 Oct '17

--On Friday, October 06, 2017 5:43 PM -0700 Quanah Gibson-Mount <quanah(a)symas.com> wrote: >> This is debatable: >> >> 1. OpenLDAP 2.4.x accepts modify operations with LDAP_MOD_INCREMENT >> >> 2. OpenLDAP 2.4.x sends modify operations via back-sock to external >> listeners >> >> Therefore I'd consider this rather a fix than a new feature. > > Convince Howard. ;) This is now in RE24 for the 2.4.46 release. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

2 1

ITS review 9/29/2017
by Quanah Gibson-Mount 11 Oct '17

11 Oct '17

Howard -- Looking for a thumbs up/down on these. The following ITSes were mostly supplied with patches and are IPR ok. I've imported them into my openldap-scratch repo for easy review. If they didn't have a supplied patch, I did the related work. Branch names are by ITS. ----------------------- Suggested for LMDB 0.9: ----------------------- its8612 - Fix LMDB builds on Solaris & derivative OSes <https://github.com/quanah/openldap-scratch/tree/its6035> ----------------------- Suggested for RE24: ----------------------- its5048 - Fix documentation for entryCSN with syncprov <https://github.com/quanah/openldap-scratch/tree/its5048> its7100 - Fix slapo-dds with entryTTL currently not decreasing <https://github.com/quanah/openldap-scratch/tree/its7100> its7373 - Fix tls_session reuse when hostname check fails <https://github.com/quanah/openldap-scratch/tree/its7373> its7374 - Fix MozNSS file matching for hashed CA cert directory (RE24 ONLY) <https://github.com/quanah/openldap-scratch/tree/its7374> its7389 - Fix MozNSS to fallback to PEM if cert not found in certdb (RE24 ONLY) <https://github.com/quanah/openldap-scratch/tree/its7389> its7442 - Add debug statements when index_intlen values are out of range <https://github.com/quanah/openldap-scratch/tree/its7442> its7520 - Omit unknown schema option for back-ldap <https://github.com/quanah/openldap-scratch/tree/its7520> its8037 - Fix delta-syncrepl with relax <https://github.com/quanah/openldap-scratch/tree/its8037> its8121 - Add LDAP_SASL_SIMPLE to ldap_bind(3) <https://github.com/quanah/openldap-scratch/tree/its8121> its8167 - Fix nonblocking TLS with referrals <https://github.com/quanah/openldap-scratch/tree/its8167> its8404 - Fix assertion with back-meta when olcDbRewrite is changed <https://github.com/quanah/openldap-scratch/tree/its8404> its8578 - Remove unused variables in RE24 <https://github.com/quanah/openldap-scratch/tree/its8578> its8583 - Fix C++ LDAPCtrl structure <https://github.com/quanah/openldap-scratch/tree/its8583> its8605 - Fix various spelling errors <https://github.com/quanah/openldap-scratch/tree/its8605> its8687 - OpenSSL 1.1 compatibility, fix build when cross-compiling <https://github.com/quanah/openldap-scratch/tree/its8687> ----------------------- Purely for RE25: ----------------------- its6035 - Fix slapd so it doesn't require restart after modifying olcAuthzRegexp <https://github.com/quanah/openldap-scratch/tree/its6035> its6300 - Add support for kqueue <https://github.com/quanah/openldap-scratch/tree/its6300> its6475 - Add documentation to slapd.conf(5) and slapd-config(5) for SASLDONTUSECOPY <https://github.com/quanah/openldap-scratch/compare/its6475> its7042 - Make it possible to unset TLS options with syncrepl <https://github.com/quanah/openldap-scratch/tree/its7042> its7532 - Add ldap_connection function for asynchronous clients <https://github.com/quanah/openldap-scratch/tree/its7532> its7721 - Allow authTimestamp to be forwarded via updateref <https://github.com/quanah/openldap-scratch/tree/its7721> its8037 - Fixes delta-sync replication when "relax" is used to modify the structural OC of an entry <https://github.com/quanah/openldap-scratch/tree/its8037> its8153 - olcTimeLimit should be SINGLE-VALUE <https://github.com/quanah/openldap-scratch/tree/its8153> its8291 - Fix slapmodify with BDB backends <https://github.com/quanah/openldap-scratch/tree/its8291> its8508 - ucgendat.c properly add title-case characters without upper-case equivalents (e.g. greek letters with iota subscript) <https://github.com/quanah/openldap-scratch/tree/its8508> its8511 - Fix documentation for multimaster, deprecate mirrormode <https://github.com/quanah/openldap-scratch/tree/its8511> its8527 - Improve SYNC debug output in certain situations <https://github.com/quanah/openldap-scratch/tree/its8527> its8573 - Add TLS options to ldap* tools <https://github.com/quanah/openldap-scratch/tree/its8573-tables> its8692 - Support LDAP_MOD_INCREMENT with back-sock <https://github.com/quanah/openldap-scratch/tree/its8692> --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

2 2

Re: recent commits review
by Howard Chu 11 Oct '17

11 Oct '17

Quanah Gibson-Mount wrote: > The following commits may be applicable for the next RE24 release > > 8678 - 375db33d13ab1fe414419a6f9cfb0b41c999862d > Note: Says it's a hack, and the underlying issue was resolved at the location > that stemmed this issue by using sortvals on the related attribute. Not sure > if we should push it to RE24 It tests for 64bit indexing, which isn't in RE24. So clearly no. > > 8226 - 221dd433995a480599d3038d109e63e0b570d84d > Further fix for an issue fixed in 2.4.44. Seems worthwhile Yeah it's fine. > > 8720 - 221dd433995a480599d3038d109e63e0b570d84d > Commit has wrong ITS. Fixes an observed issue in the field. Seems worthwhile What commit ID? This is the same as above. > > 8709 - 2c36a37f908cd2e7297b1bd5621fea2f0d32451a > contrib module fix for OpenSSL 1.1, which is supported in RE24. Seems worthwhile OK. > > 8717 - 738723866ed6858cb694b0a770300d781e09e333 > Fixes an observed issue in the field. Seems worthwhile OK > > 5adcdb7642dbafae484b8f3feb391490ff9be087 - Fix warnings issued by autoconf > 2.68+ > No related ITS. Not sure if this is relevant to RE24 or not, given we're > unlikely to regenerate configure for it. Seems unnecessary for RE24. > > --Quanah > > -- > > Quanah Gibson-Mount > Product Architect > Symas Corporation > Packaged, certified, and supported LDAP solutions powered by OpenLDAP: > <http://www.symas.com> > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 2

Re: ITS review 9/29/2017
by Howard Chu 06 Oct '17

06 Oct '17

Quanah Gibson-Mount wrote: > --On Friday, October 06, 2017 2:27 PM +0100 Howard Chu <hyc(a)symas.com> wrote: > >> Quanah Gibson-Mount wrote: >> >>> Suggested for RE24: >>> ----------------------- > >>> its7389 - Fix MozNSS to fallback to PEM if cert not found in certdb >>> (RE24 ONLY) <https://github.com/quanah/openldap-scratch/tree/its7389> >> >> Questionable, since the required PEM support module is 3rd party and not >> included in MozNSS. We have no way to test or support this patch. > > > This appears to be a fix for ITS#7276 (Added 2.4.32, 2012/07/31), which we > already accepted into RE24. So it seems a legitimate fix to include in RE24. OK. >>> its7442 - Add debug statements when index_intlen values are out of range >>> <https://github.com/quanah/openldap-scratch/tree/its7442> >> >> Looks pointless. > > Well, the man page is not clear on this point. I'm fine dropping the debug > statements, but what about the manpage updates which clarify the min/max > allowed values? Then we should also document all the other places where for example our integers accept a max value of 4294967295 or 18446744073709551615 too? It's stupid. Nobody is using 256 byte integers. Nobody is using integers bigger than 256 bytes. (Come on, 2^2048? really?) It's a limit that no one will ever hit in practice. >>> its8037 - Fix delta-syncrepl with relax >>> <https://github.com/quanah/openldap-scratch/tree/its8037> >> >> Looks like an enhancement, not a bugfix > > I included this for RE24 as the reporter hit this problem with RE24. If we > don't want to put it in RE24, are we OK for RE25/master? Already approved it for RE25/master. >>> its8167 - Fix nonblocking TLS with referrals >>> <https://github.com/quanah/openldap-scratch/tree/its8167> >> >> OK, but non-blocking TLS was LDAP_DEVEL, not supported in RE24. This >> patch should be master/RE25 only. > > I noted this for RE24 because the reporter was using the feature in RE24 > (I.e., they specifically enabled it). Is there any harm in including (but not > documenting via the changes file) it in RE24? OK, leave it undoc'd in RE24. >>> its8605 - Fix various spelling errors >>> <https://github.com/quanah/openldap-scratch/tree/its8605> >> >> Introduces trailing whitespace - kill that before committing. >> In general, this patch falls under the "do not improve" rule >> http://www.openldap.org/devel/programming.html and should be rejected for >> not fixing any actual bugs. Many of the typos being fixed are in comments >> that are never user-visible anyway. Pollutes git history for a large >> number of files without any significant benefit. >> >> Better leave it out of re24. > > > Is this ok for master/RE25 then? It's still a bunch of changes that don't actually fix anything. I guess we can take this in master. As a general rule, we should just reject patches like this in the future. >>> its8511 - Fix documentation for multimaster, deprecate mirrormode >>> <https://github.com/quanah/openldap-scratch/tree/its8511> >> >> Gratuitous change, existing docs and practices are already established. >> Hard enough to get people to update their docs, this is a bad idea. > > This change is not gratuitous in the least. The misinformation in our current > documentation leads to constant confusion among end users, who often do not > want to go to the lengths necessary to deploy the *concept* that is mirror > mode, and instead just want to do "multimaster", so they leave our current > misnamed 'mirrormode' parameter set to false. Fixing the documentation to > match the reality of what's being configured is a positive step to removing > confusion and to stop misleading end users on what is being done. I've > provided numerous links from the mailing list where this caused problems for > end users before. Our parameters should reflect what they actually do. You're talking about confusion for new users, meanwhile you're just creating confusion for existing users. Existing users tend to complain more because they have more invested into their running deployments. This is a bad idea. >>> its8573 - Add TLS options to ldap* tools >>> <https://github.com/quanah/openldap-scratch/tree/its8573-tables> >> >> The manpage updates are a bit excessive. Maybe we need a single manpage >> just for common options, that we can refer to from all of the individual >> commands' pages. > > Ok, I'll add that to my RE25 stack of rework. > > --Quanah > > -- > > Quanah Gibson-Mount > Product Architect > Symas Corporation > Packaged, certified, and supported LDAP solutions powered by OpenLDAP: > <http://www.symas.com> > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

3 2

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel