openldap-devel June 2018

openldap-devel@openldap.org

5 participants
8 discussions

asserts and manadatory build instructions (was ITS#8240)
by Michael Ströder 04 Nov '21

04 Nov '21

hyc(a)symas.com wrote in ITS#8240: > Our patch response was too hasty. There is no OpenLDAP bug here, the real > issue is production binaries being built with asserts enabled instead of > compiling with -DNDEBUG. That's an issue for packagers and distros to resolve. > Closing this ITS, not an OpenLDAP bug. Maybe I missed something. But this is the first time I've heard about -DNDEBUG being mandatory when compiling binary packages for production use. Does it have other effects? And what are general rules for assert statements in OpenLDAP code? In my own (Python) code assert statements are supposed to be only triggered if something goes wrong *internally* (type issues etc.). If somebody manages to trigger an assert statement with invalid input from "outside" I always consider this to be a serious bug revealing insufficient error handling even though e.g. web2ldap just logs the exception but won't crash. YMMV, but please clarify. I also wonder whether there are more mandatory rules for building packages and where I can find them. Please don't get me wrong: My inquiry is in good faith to avoid unnecessary ITS based on misunderstanding. Ciao, Michael.

1 1

Config questions for back-ldap, back-meta, and back-asyncmeta
by Quanah Gibson-Mount 27 Jun '18

27 Jun '18

There are three options between back-ldap, back-meta, and back-asyncmeta that seem to have an incorrect defintion for cn=config and/or a documentation bug. For back-ldap: idle-timeout -> The man page says takes an integer, but is defined as a string. However, I think the man page for this parameter is incorrect, and in fact it takes a possible string as defined in the back-meta/async manual pages for this same parameter. (I.e, it can have a format of something like 1d15h5s) For back-ldap, back-meta, and back-asyncmeta: network-timeout -> This takes an integer, but is defined as a string. The back-ldap, back-meta, and back-asyncmeta man pages says it uses the same format as idle-timeout, but the function that parses the value does not agree with assertion. It appears to take only accept an integer. For back-meta and back-asyncmeta: bind-timeout -> This is clearly described in the man page as a taking an integer value, but it is defined as a string. Any objection to me changing it to be an integer type? Thanks! --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

3 5

RE24 testing call #1 (2.4.46) LMDB RE0.9 testing call #1 (0.9.22)
by Quanah Gibson-Mount 15 Jun '18

15 Jun '18

Hello everyone, At this point, I believe we're ready to being testing for a 2.4.46 release. The primary focus on this release has been to fix several long standing issues with replication, both for "standard" and "delta" based syncrepl. These fixes have been tested against databases and workloads known to trigger the problems that were encountered. Special thanks to Paul B. Henson for doing additional validation for those issues that were discovered in his deployment. OpenLDAP 2.4.46 Engineering Fixed libldap connection delete callbacks when TLS fails to start (ITS#8717) Fixed libldap to not reuse tls_session if TLS hostname check fails (ITS#7373) Fixed libldap cross-compiling with OpenSSL 1.1 (ITS#8687) Fixed libldap OpenSSL 1.1.1 compatibility with BIO_method (ITS#8791) Fixed libldap MozNSS CA certificate hash matching (ITS#7374) Fixed libldap MozNSS with PEM certs when also using an NSS cert db (ITS#7389) Fixed libldap MozNSS initialization (ITS#8484) Fixed libldap GnuTLS with GNUTLS_E_AGAIN (ITS#8650) Fixed libldap memory leak with cancel operations (ITS#8782) Fixed slapd Eventlog registry key creation on 64-bit Windows (ITS#8705) Fixed slapd to maintain SSF across SASL binds (ITS#8796) Fixed slapd syncrepl deadlock when updating cookie (ITS#8752) Fixed slapd syncrepl callback to always be last in the stack (ITS#8752) Fixed slapd telephoneNumberNormalize when the value is spaces and hyphens (ITS#8778) Fixed slapd CSN queue processing (ITS#8801) Fixed slapd-ldap TLS connection timeout with high latency connections (ITS#8720) Fixed slapd-ldap to ignore unknown schema when omit-unknown-schema is set (ITS#7520) Fixed slapd-mdb with an optimization for long lived read transactions (ITS#8226) Fixed slapd-meta assert when olcDbRewrite is modified (ITS#8404) Fixed slapd-sock with LDAP_MOD_INCREMENT operations (ITS#8692) Fixed slapo-accesslog cleanup to only occur on failed operations (ITS#8752) Fixed slapo-accesslog to not expire the last entry in the database (ITS#8100) Fixed slapo-dds entryTTL to actually decrease as per RFC 2589 (ITS#7100) Fixed slapo-syncprov memory leak with delete operations (ITS#8690) Fixed slapo-syncprov to not clear pending operation when checkpointing (ITS#8444) Fixed slapo-syncprov to initialize an empty accesslog db if configured (ITS#8100) Fixed slapo-syncprov not to log checkpoints to accesslog db (ITS#8607) Fixed slapo-syncprov to process changes from this SID on REFRESH (ITS#8800) Fixed slapo-syncprov session log parsing to not block other operations (ITS#8486) Build Environment Fixed Windows build with newer MINGW version (ITS#8697) Fixed compiler warnings and removed unused variables (ITS#8578) Contrib Fixed ldapc++ Control structure (ITS#8583) Documentation Delete stub manpage for back-ldbm (ITS#8713) Fixed ldap_bind(3) to mention the LDAP_SASL_SIMPLE mechanism (ITS#8121) Fixed slapd-config(5) typo for olcTLSCipherSuite (ITS#8715) Fixed slapo-syncprov(5) indexing requirements (ITS#5048) LMDB 0.9.22 Engineering Fix regression with new db from 0.9.19 (ITS#8760) Fix liblmdb to build on Solaris (ITS#8612) Fix delete behavior with DUPSORT DB (ITS#8622) Fix mdb_cursor_get/mdb_cursor_del behavior (ITS#8722) Thanks, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

5 14

mmap MAP_FIXED, Linux 4.17
by Howard Chu 14 Jun '18

14 Jun '18

There's a new mmap option MAP_FIXED_NOREPLACE added in the Linux 4.17 kernel. This will check if the requested mapping overlaps any existing maps and fail if so, allowing fixed address mappings to be used more safely. Funny to see since we already worked around this danger in LMDB a long time ago. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?… As usual, adding more stuff to the kernel even though userland solutions already existed. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

[LMDB] how does reader thread read meta page without locking and avoid data race at the same time?
by Chuntao HONG 14 Jun '18

14 Jun '18

Background: I am trying to modify the LMDB code so we can have multiple threads reading from the same snapshot (same txn_id) at the same time. I am trying to do this in a "fork txn" way. Basically I have a master thread with a read txn, and then I try to create txns in the slave threads with the same txn_id. So I modified the mdb_txn_renew0() function and provide it with the txn_id the master thread is holding. With that I hope the slave transactions can read the same meta page because we pick the meta pages with meta = env->me_metas[txn->mt_txnid & 1]; But then I realized I might be doing it wrong. There are only two meta pages used in LMDB. So what if there had been two write transactions committed after the master thread held its transaction, i.e. the master thread has txn_id==N and current txn_id==N+2? That means the meta page was over-written and the slave thread may read different data from the meta page than the master. Then the question in the header popped into my mind. When reader threads are created, they copy the meta-db infos with a memcpy like this: memcpy(txn->mt_dbs, meta->mm_dbs, CORE_DBS * sizeof(MDB_db)); But if the meta page was written in the middle of the memcpy, we can get corrupted data. I am sure there is some code that prevents this data race from happening, since we have been using LMDB with multiple threads for quite a while. Could someone point me to the code that prevents the data race from happening? cheers, chuntao

2 2

Deprecated options for slapd-(ldap|meta|asyncmeta)
by Quanah Gibson-Mount 14 Jun '18

14 Jun '18

There are several deprecated/obsolete options in the ldap, meta, and async backends. Is there any reason not to remove these in git master? I.e., is there a particular reason to keep them for the OpenLDAP 2.5 release? They're clearly documented as obsolete in the 2.4 docs: Example: acl-passwd <password> Formerly known as the bindpw, it is the password used with the above acl-authcDN directive. This directive is obsoleted by the credentials arg of acl-bind when bindmethod=simple, and will be dismissed in the future. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

2 1

Issue with mdb_cursor_put
by Brüns, Stefan 14 Jun '18

14 Jun '18

Hi, I see a similar problem as reported in October 2017 for mdb_cursor_del, i.e. pages ending up twice on the dirty list. Backtrace: #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #1 0x00007ffff61c8da1 in __GI_abort () at abort.c:79 #2 0x00007ffff5a31052 in mdb_assert_fail (env=0x55555579d6e0, expr_txt=expr_txt@entry=0x7ffff5a3294f "rc == 0", func=func@entry=0x7ffff5a33268 <__func__.7016> "mdb_page_dirty", line=line@entry=2127, file=0x7ffff5a32930 "mdb.c") at mdb.c:1542 #3 0x00007ffff5a25fe5 in mdb_page_dirty (txn=0x55555579eaa0, mp=<optimized out>) at mdb.c:2127 #4 0x00007ffff5a2720b in mdb_page_alloc (num=num@entry=1, mp=mp@entry=0x7fffffffd110, mc=<optimized out>) at mdb.c:2308 #5 0x00007ffff5a29114 in mdb_page_new (mc=mc@entry=0x7fffffffd2f0, flags=flags@entry=4, num=1, mp=mp@entry=0x7fffffffd170) at mdb.c:7147 #6 0x00007ffff5a29519 in mdb_node_add (mc=mc@entry=0x7fffffffd2f0, indx=<optimized out>, key=key@entry=0x7fffffffd6c0, data=0x7fffffffd6d0, pgno=pgno@entry=0, flags=0) at mdb.c:7289 #7 0x00007ffff5a2ca59 in mdb_cursor_put (mc=0x7fffffffd2f0, key=0x7fffffffd6c0, data=0x7fffffffd6d0, flags=<optimized out>) at mdb.c:6916 #8 0x00007ffff5a2ee4b in mdb_put (txn=0x55555579eaa0, dbi=3, key=key@entry=0x7fffffffd6c0, data=data@entry=0x7fffffffd6d0, flags=flags@entry=0) at mdb.c:8991 --- I have tracked down the moment where the duplicate pgno is added to the list: --- #0 mdb_page_alloc (num=num@entry=6, mp=mp@entry=0x7fffffffd110, mc=<optimized out>) at mdb.c:2277 #1 0x00007ffff5a29114 in mdb_page_new (mc=mc@entry=0x7fffffffd2f0, flags=flags@entry=4, num=6, mp=mp@entry=0x7fffffffd170) at mdb.c:7147 #2 0x00007ffff5a29519 in mdb_node_add (mc=mc@entry=0x7fffffffd2f0, indx=<optimized out>, key=key@entry=0x7fffffffd6c0, data=0x7fffffffd6d0, pgno=pgno@entry=0, flags=0) at mdb.c:7289 #3 0x00007ffff5a2ca59 in mdb_cursor_put (mc=0x7fffffffd2f0, key=0x7fffffffd6c0, data=0x7fffffffd6d0, flags=<optimized out>) at mdb.c:6916 #4 0x00007ffff5a2ee4b in mdb_put (txn=0x55555579eaa0, dbi=3, key=key@entry=0x7fffffffd6c0, data=data@entry=0x7fffffffd6d0, flags=flags@entry=0) at mdb.c:8991 --- (gdb) p idl[0] $22 = 47 (gdb) x/18g &idl[ 30 ] 0x7fbff2d1ef20: 20234 20230 0x7fbff2d1ef30: 20229 20228 0x7fbff2d1ef40: 19230 19228 0x7fbff2d1ef50: 17181 17180 0x7fbff2d1ef60: 15736 15736 <- double entry 0x7fbff2d1ef70: 15274 8470 0x7fbff2d1ef80: 8438 7176 0x7fbff2d1ef90: 7174 4758 0x7fbff2d1efa0: 4719 1213 So the double page number is already stored in the freelist in the database, i.e. the database itself is corrupt. The database was initially created with lmdb 0.9.17, currently I am using 0.9.22. Any idea how to deal with this issue? Kind regards, Stefan -- Stefan Brüns / Bergstraße 21 / 52062 Aachen home: +49 241 53809034 mobile: +49 151 50412019

2 1

Re: LMDB Fixed memory address mapping documentation and example(s)?
by Howard Chu 13 Jun '18

13 Jun '18

John Daly wrote: > Hi Howard -- (cc'ing openldap-devel for general interest) Hi John, thanks for your interest. Unfortunately, this is a feature that we never fully implemented. I can give you a rundown of how it was intended to be used, though. You have the overall concept - you must make sure that any object you want to store is marshalled into a single contiguous blob. In OpenLDAP, the back-mdb backend already does this for us when storing LDAP entries, but it only goes half way. Follow along in slap.h: http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=servers/sl… We have a struct Entry with a couple of struct berval names and then a linked list of Attributes. http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=servers/sl… Attributes point to an AttributeDescription and then have two arrays of struct bervals for values, and a next pointer. So to serialize all of this into an LMDB blob we would malloc a single blob to hold an Entry struct contiguous with all of its Attribute structs and followed by all of its value arrays. The various struct fields would then point to addresses within this contiguous blob. This would be passed into mdb_put or mdb_cursor_put. Laid out appropriately, the object could be used as-is with no deserialization needed, on read. The MDB_rel_func callback on the DB would have previously been set to a function that knows the layout of this Entry blob, and would increment or decrement all of these blob-internal pointers whenever the blob itself was moved by an LMDB operation. The instances where this occurs are when copying a read-only page to make a writable copy, when moving items in a page to fill the gap from deleting a node, and when splitting a page to insert a new node. This latter part has never been implemented, because it turned out we never really needed it. Most LDAP entries are larger than half a page, so they wind up being written to an overflow page. Once on an overflow page, they are never moved or relocated. In my experience, C++ doesn't give you much control over object storage layout, particularly not if you're using standard templates and other such stuff. I'm not sure how well you could leverage this capability from C++. If you're looking at our implementation in back-mdb/id2entry.c to see how we actually store the complete Entry's, you'll notice that it's more complicated than I just described. The main reason here is that the AttributeDescription is a pointer into the slapd schema, and schema elements aren't (currently) stored in LMDB so we can't guarantee constant pointer values for those objects. Instead we had to use a mapping table from 16-bit integers to schema instances. Because of this, back-mdb still has to do some minor processing to turn an on-disk entry into a slapd in-memory entry. Doing the complete transition to persistent schema was deferred to OpenLDAP 2.6 or later. > We're investigating database (persistence engines) for use in an embedded > environment and stumbled across your LMDB database. The features of LMDB map > quite nicely on to our needs and the fact that its very small, very fast, and > highly configurable are also extremely attractive. > > Our current (home-grown) OO persistence solution/framework has a long list of > problems (complicated, large, slow, etc.) that I won't bore you with. One > feature/mode of your LMDB that piqued our interest is the use of 'fixed > address memory mapping'. If we understand the feature correctly, we could > effectively eliminate all our OO serialization/deserialization transformations > by using a custom memory allocator to ensure objects-to-be-persisted are > mapped into LMDB memory-mapped page space, thereby allowing LMDB (and the > underlying OS virtual memory management system) to handle all our persistence > needs. Is this correct? > > I've watched several talks you've given on LMDB, read thru the documentation, > plus many articles & blogs on LMDB, but I haven't come across much information > (or explicit examples) on using the 'fixed memory mapped feature', which leads > me to my questions: > > Are we interpreting this feature correctly? Can LMDB be used as a persistence > engine as described? If so, can you point us to documentation and/or examples > that illustrate this use case? If not, any pointers on LMDBs application as > an OO persistence engine would be much appreciated. > > Thanks in advance, > > -John > > P.S. Our application is currently written in a combination of C# and C++. > We're in the process of rearchitecting to address a number of issues and will > be moving to a completely native (C++) implementation in the future. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel June 2018