asserts and manadatory build instructions (was ITS#8240)
by Michael Ströder
hyc(a)symas.com wrote in ITS#8240:
> Our patch response was too hasty. There is no OpenLDAP bug here, the real
> issue is production binaries being built with asserts enabled instead of
> compiling with -DNDEBUG. That's an issue for packagers and distros to resolve.
> Closing this ITS, not an OpenLDAP bug.
Maybe I missed something. But this is the first time I've heard about -DNDEBUG
being mandatory when compiling binary packages for production use. Does it
have other effects?
And what are general rules for assert statements in OpenLDAP code?
In my own (Python) code assert statements are supposed to be only triggered if
something goes wrong *internally* (type issues etc.). If somebody manages to
trigger an assert statement with invalid input from "outside" I always
consider this to be a serious bug revealing insufficient error handling even
though e.g. web2ldap just logs the exception but won't crash. YMMV, but please
clarify.
I also wonder whether there are more mandatory rules for building packages and
where I can find them.
Please don't get me wrong: My inquiry is in good faith to avoid unnecessary
ITS based on misunderstanding.
Ciao, Michael.
2 years, 1 month
Config questions for back-ldap, back-meta, and back-asyncmeta
by Quanah Gibson-Mount
There are three options between back-ldap, back-meta, and back-asyncmeta
that seem to have an incorrect defintion for cn=config and/or a
documentation bug.
For back-ldap:
idle-timeout -> The man page says takes an integer, but is defined as a
string. However, I think the man page for this parameter is incorrect, and
in fact it takes a possible string as defined in the back-meta/async manual
pages for this same parameter. (I.e, it can have a format of something like
1d15h5s)
For back-ldap, back-meta, and back-asyncmeta:
network-timeout -> This takes an integer, but is defined as a string. The
back-ldap, back-meta, and back-asyncmeta man pages says it uses the same
format as idle-timeout, but the function that parses the value does not
agree with assertion. It appears to take only accept an integer.
For back-meta and back-asyncmeta:
bind-timeout -> This is clearly described in the man page as a taking an
integer value, but it is defined as a string. Any objection to me changing
it to be an integer type?
Thanks!
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 5 months
RE24 testing call #1 (2.4.46) LMDB RE0.9 testing call #1 (0.9.22)
by Quanah Gibson-Mount
Hello everyone,
At this point, I believe we're ready to being testing for a 2.4.46 release.
The primary focus on this release has been to fix several long standing
issues with replication, both for "standard" and "delta" based syncrepl.
These fixes have been tested against databases and workloads known to
trigger the problems that were encountered. Special thanks to Paul B.
Henson for doing additional validation for those issues that were
discovered in his deployment.
OpenLDAP 2.4.46 Engineering
Fixed libldap connection delete callbacks when TLS fails to start
(ITS#8717)
Fixed libldap to not reuse tls_session if TLS hostname check fails
(ITS#7373)
Fixed libldap cross-compiling with OpenSSL 1.1 (ITS#8687)
Fixed libldap OpenSSL 1.1.1 compatibility with BIO_method (ITS#8791)
Fixed libldap MozNSS CA certificate hash matching (ITS#7374)
Fixed libldap MozNSS with PEM certs when also using an NSS cert db
(ITS#7389)
Fixed libldap MozNSS initialization (ITS#8484)
Fixed libldap GnuTLS with GNUTLS_E_AGAIN (ITS#8650)
Fixed libldap memory leak with cancel operations (ITS#8782)
Fixed slapd Eventlog registry key creation on 64-bit Windows (ITS#8705)
Fixed slapd to maintain SSF across SASL binds (ITS#8796)
Fixed slapd syncrepl deadlock when updating cookie (ITS#8752)
Fixed slapd syncrepl callback to always be last in the stack (ITS#8752)
Fixed slapd telephoneNumberNormalize when the value is spaces and
hyphens (ITS#8778)
Fixed slapd CSN queue processing (ITS#8801)
Fixed slapd-ldap TLS connection timeout with high latency connections
(ITS#8720)
Fixed slapd-ldap to ignore unknown schema when omit-unknown-schema is
set (ITS#7520)
Fixed slapd-mdb with an optimization for long lived read transactions
(ITS#8226)
Fixed slapd-meta assert when olcDbRewrite is modified (ITS#8404)
Fixed slapd-sock with LDAP_MOD_INCREMENT operations (ITS#8692)
Fixed slapo-accesslog cleanup to only occur on failed operations
(ITS#8752)
Fixed slapo-accesslog to not expire the last entry in the database
(ITS#8100)
Fixed slapo-dds entryTTL to actually decrease as per RFC 2589 (ITS#7100)
Fixed slapo-syncprov memory leak with delete operations (ITS#8690)
Fixed slapo-syncprov to not clear pending operation when checkpointing
(ITS#8444)
Fixed slapo-syncprov to initialize an empty accesslog db if configured
(ITS#8100)
Fixed slapo-syncprov not to log checkpoints to accesslog db (ITS#8607)
Fixed slapo-syncprov to process changes from this SID on REFRESH
(ITS#8800)
Fixed slapo-syncprov session log parsing to not block other operations
(ITS#8486)
Build Environment
Fixed Windows build with newer MINGW version (ITS#8697)
Fixed compiler warnings and removed unused variables (ITS#8578)
Contrib
Fixed ldapc++ Control structure (ITS#8583)
Documentation
Delete stub manpage for back-ldbm (ITS#8713)
Fixed ldap_bind(3) to mention the LDAP_SASL_SIMPLE mechanism
(ITS#8121)
Fixed slapd-config(5) typo for olcTLSCipherSuite (ITS#8715)
Fixed slapo-syncprov(5) indexing requirements (ITS#5048)
LMDB 0.9.22 Engineering
Fix regression with new db from 0.9.19 (ITS#8760)
Fix liblmdb to build on Solaris (ITS#8612)
Fix delete behavior with DUPSORT DB (ITS#8622)
Fix mdb_cursor_get/mdb_cursor_del behavior (ITS#8722)
Thanks,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 5 months
[LMDB] how does reader thread read meta page without locking and avoid data race at the same time?
by Chuntao HONG
Background:
I am trying to modify the LMDB code so we can have multiple threads reading
from the same snapshot (same txn_id) at the same time. I am trying to do
this in a "fork txn" way. Basically I have a master thread with a read txn,
and then I try to create txns in the slave threads with the same txn_id. So
I modified the mdb_txn_renew0() function and provide it with the txn_id the
master thread is holding. With that I hope the slave transactions can read
the same meta page because we pick the meta pages with
meta = env->me_metas[txn->mt_txnid & 1];
But then I realized I might be doing it wrong. There are only two meta
pages used in LMDB. So what if there had been two write transactions
committed after the master thread held its transaction, i.e. the master
thread has txn_id==N and current txn_id==N+2? That means the meta page was
over-written and the slave thread may read different data from the meta
page than the master.
Then the question in the header popped into my mind. When reader threads
are created, they copy the meta-db infos with a memcpy like this:
memcpy(txn->mt_dbs, meta->mm_dbs, CORE_DBS * sizeof(MDB_db));
But if the meta page was written in the middle of the memcpy, we can get
corrupted data. I am sure there is some code that prevents this data race
from happening, since we have been using LMDB with multiple threads for
quite a while. Could someone point me to the code that prevents the data
race from happening?
cheers,
chuntao
5 years, 5 months
Deprecated options for slapd-(ldap|meta|asyncmeta)
by Quanah Gibson-Mount
There are several deprecated/obsolete options in the ldap, meta, and async
backends. Is there any reason not to remove these in git master? I.e., is
there a particular reason to keep them for the OpenLDAP 2.5 release?
They're clearly documented as obsolete in the 2.4 docs:
Example:
acl-passwd <password>
Formerly known as the bindpw, it is the password used with
the
above acl-authcDN directive. This directive is obsoleted by
the
credentials arg of acl-bind when bindmethod=simple, and will
be
dismissed in the future.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 5 months
Issue with mdb_cursor_put
by Brüns, Stefan
Hi,
I see a similar problem as reported in October 2017 for mdb_cursor_del, i.e.
pages ending up twice on the dirty list.
Backtrace:
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007ffff61c8da1 in __GI_abort () at abort.c:79
#2 0x00007ffff5a31052 in mdb_assert_fail (env=0x55555579d6e0,
expr_txt=expr_txt@entry=0x7ffff5a3294f "rc == 0",
func=func@entry=0x7ffff5a33268 <__func__.7016> "mdb_page_dirty",
line=line@entry=2127, file=0x7ffff5a32930 "mdb.c") at mdb.c:1542
#3 0x00007ffff5a25fe5 in mdb_page_dirty (txn=0x55555579eaa0, mp=<optimized
out>) at mdb.c:2127
#4 0x00007ffff5a2720b in mdb_page_alloc (num=num@entry=1,
mp=mp@entry=0x7fffffffd110, mc=<optimized out>) at mdb.c:2308
#5 0x00007ffff5a29114 in mdb_page_new (mc=mc@entry=0x7fffffffd2f0,
flags=flags@entry=4, num=1, mp=mp@entry=0x7fffffffd170) at mdb.c:7147
#6 0x00007ffff5a29519 in mdb_node_add (mc=mc@entry=0x7fffffffd2f0,
indx=<optimized out>, key=key@entry=0x7fffffffd6c0, data=0x7fffffffd6d0,
pgno=pgno@entry=0, flags=0) at mdb.c:7289
#7 0x00007ffff5a2ca59 in mdb_cursor_put (mc=0x7fffffffd2f0,
key=0x7fffffffd6c0, data=0x7fffffffd6d0, flags=<optimized out>) at mdb.c:6916
#8 0x00007ffff5a2ee4b in mdb_put (txn=0x55555579eaa0, dbi=3,
key=key@entry=0x7fffffffd6c0, data=data@entry=0x7fffffffd6d0,
flags=flags@entry=0) at mdb.c:8991
---
I have tracked down the moment where the duplicate pgno is added to the list:
---
#0 mdb_page_alloc (num=num@entry=6, mp=mp@entry=0x7fffffffd110, mc=<optimized
out>) at mdb.c:2277
#1 0x00007ffff5a29114 in mdb_page_new (mc=mc@entry=0x7fffffffd2f0,
flags=flags@entry=4, num=6, mp=mp@entry=0x7fffffffd170) at mdb.c:7147
#2 0x00007ffff5a29519 in mdb_node_add (mc=mc@entry=0x7fffffffd2f0,
indx=<optimized out>, key=key@entry=0x7fffffffd6c0, data=0x7fffffffd6d0,
pgno=pgno@entry=0, flags=0) at mdb.c:7289
#3 0x00007ffff5a2ca59 in mdb_cursor_put (mc=0x7fffffffd2f0,
key=0x7fffffffd6c0, data=0x7fffffffd6d0, flags=<optimized out>) at mdb.c:6916
#4 0x00007ffff5a2ee4b in mdb_put (txn=0x55555579eaa0, dbi=3,
key=key@entry=0x7fffffffd6c0, data=data@entry=0x7fffffffd6d0,
flags=flags@entry=0) at mdb.c:8991
---
(gdb) p idl[0]
$22 = 47
(gdb) x/18g &idl[ 30 ]
0x7fbff2d1ef20: 20234 20230
0x7fbff2d1ef30: 20229 20228
0x7fbff2d1ef40: 19230 19228
0x7fbff2d1ef50: 17181 17180
0x7fbff2d1ef60: 15736 15736 <- double entry
0x7fbff2d1ef70: 15274 8470
0x7fbff2d1ef80: 8438 7176
0x7fbff2d1ef90: 7174 4758
0x7fbff2d1efa0: 4719 1213
So the double page number is already stored in the freelist in the database,
i.e. the database itself is corrupt.
The database was initially created with lmdb 0.9.17, currently I am using
0.9.22.
Any idea how to deal with this issue?
Kind regards, Stefan
--
Stefan Brüns / Bergstraße 21 / 52062 Aachen
home: +49 241 53809034 mobile: +49 151 50412019
5 years, 5 months
Re: LMDB Fixed memory address mapping documentation and example(s)?
by Howard Chu
John Daly wrote:
> Hi Howard --
(cc'ing openldap-devel for general interest)
Hi John,
thanks for your interest. Unfortunately, this is a feature that we never
fully implemented. I can give you a rundown of how it was intended to be used,
though. You have the overall concept - you must make sure that any object
you want to store is marshalled into a single contiguous blob.
In OpenLDAP, the back-mdb backend already does this for us when storing LDAP
entries, but it only goes half way.
Follow along in slap.h:
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=servers/...
We have a struct Entry with a couple of struct berval names and then a linked
list of Attributes.
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=servers/...
Attributes point to an AttributeDescription and then have two arrays of struct
bervals for values, and a next pointer.
So to serialize all of this into an LMDB blob we would malloc a single blob to
hold an Entry struct contiguous with all of its Attribute structs and followed
by all of its value arrays. The various struct fields would then point to
addresses within this contiguous blob. This would be passed into mdb_put or
mdb_cursor_put. Laid out appropriately, the object could be used as-is with no
deserialization needed, on read.
The MDB_rel_func callback on the DB would have previously been set to a
function that knows the layout of this Entry blob, and would increment or
decrement all of these blob-internal pointers whenever the blob itself was
moved by an LMDB operation. The instances where this occurs are when copying a
read-only page to make a writable copy, when moving items in a page to fill
the gap from deleting a node, and when splitting a page to insert a new node.
This latter part has never been implemented, because it turned out we never
really needed it. Most LDAP entries are larger than half a page, so they wind
up being written to an overflow page. Once on an overflow page, they are never
moved or relocated.
In my experience, C++ doesn't give you much control over object storage
layout, particularly not if you're using standard templates and other such
stuff. I'm not sure how well you could leverage this capability from C++.
If you're looking at our implementation in back-mdb/id2entry.c to see how we
actually store the complete Entry's, you'll notice that it's more complicated
than I just described. The main reason here is that the AttributeDescription
is a pointer into the slapd schema, and schema elements aren't (currently)
stored in LMDB so we can't guarantee constant pointer values for those
objects. Instead we had to use a mapping table from 16-bit integers to schema
instances. Because of this, back-mdb still has to do some minor processing to
turn an on-disk entry into a slapd in-memory entry. Doing the complete
transition to persistent schema was deferred to OpenLDAP 2.6 or later.
> We're investigating database (persistence engines) for use in an embedded
> environment and stumbled across your LMDB database. The features of LMDB map
> quite nicely on to our needs and the fact that its very small, very fast, and
> highly configurable are also extremely attractive.
>
> Our current (home-grown) OO persistence solution/framework has a long list of
> problems (complicated, large, slow, etc.) that I won't bore you with. One
> feature/mode of your LMDB that piqued our interest is the use of 'fixed
> address memory mapping'. If we understand the feature correctly, we could
> effectively eliminate all our OO serialization/deserialization transformations
> by using a custom memory allocator to ensure objects-to-be-persisted are
> mapped into LMDB memory-mapped page space, thereby allowing LMDB (and the
> underlying OS virtual memory management system) to handle all our persistence
> needs. Is this correct?
>
> I've watched several talks you've given on LMDB, read thru the documentation,
> plus many articles & blogs on LMDB, but I haven't come across much information
> (or explicit examples) on using the 'fixed memory mapped feature', which leads
> me to my questions:
>
> Are we interpreting this feature correctly? Can LMDB be used as a persistence
> engine as described? If so, can you point us to documentation and/or examples
> that illustrate this use case? If not, any pointers on LMDBs application as
> an OO persistence engine would be much appreciated.
>
> Thanks in advance,
>
> -John
>
> P.S. Our application is currently written in a combination of C# and C++.
> We're in the process of rearchitecting to address a number of issues and will
> be moving to a completely native (C++) implementation in the future.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
5 years, 5 months