asserts and manadatory build instructions (was ITS#8240)
by Michael Ströder
hyc(a)symas.com wrote in ITS#8240:
> Our patch response was too hasty. There is no OpenLDAP bug here, the real
> issue is production binaries being built with asserts enabled instead of
> compiling with -DNDEBUG. That's an issue for packagers and distros to resolve.
> Closing this ITS, not an OpenLDAP bug.
Maybe I missed something. But this is the first time I've heard about -DNDEBUG
being mandatory when compiling binary packages for production use. Does it
have other effects?
And what are general rules for assert statements in OpenLDAP code?
In my own (Python) code assert statements are supposed to be only triggered if
something goes wrong *internally* (type issues etc.). If somebody manages to
trigger an assert statement with invalid input from "outside" I always
consider this to be a serious bug revealing insufficient error handling even
though e.g. web2ldap just logs the exception but won't crash. YMMV, but please
clarify.
I also wonder whether there are more mandatory rules for building packages and
where I can find them.
Please don't get me wrong: My inquiry is in good faith to avoid unnecessary
ITS based on misunderstanding.
Ciao, Michael.
1 year, 6 months
Re: LMDB encryption support
by Howard Chu
Greg Hudson wrote:
> On 08/10/2017 11:55 AM, Howard Chu wrote:
>> Thoughts? Hardcode 1 algorithm, or leave it pluggable?
>
> Some thoughts, without advocating for either option:
>
> * If support isn't built-in, then generic LMDB tools (including
> mdb_copy/dump/load/stat) can't operate on encrypted databases, if they
> need plaintext pages to work.
Yeah, already thought about that. We can add an option to the generic tools to
dynamically load a user-supplied module for such cases. I always wanted this
for BerkeleyDB as well, to safely operate on DBs with custom comparators.
> * Built-in support doesn't necessarily mean hardcoding an algorithm for
> all time, if the meta pages can include an algorithm selector. One of
> the selector values could even mean "use application callbacks".
>
> * Is the page size guaranteed to be a multiple of 16 bytes? 32 bytes?
> I would assume yes to both; documenting that would make it easier to use
> block ciphers since ciphertext expansion isn't allowed.
Yes, page sizes are always large powers of 2. 4096 bytes is typical (but on
the small side). SPARC uses 8192, some MIPS systems use 32768 or 65536.
> * Application writers are more likely to get encryption callbacks wrong
> than Howard is. They could ignore the IV (making it easy to detect
> duplicate initial blocks within a page) or even do pure ECB encryption
> (making it easy to detect duplicate blocks anywhere). Less egregiously,
> applications might not make the ideal choice of cipher mode. I would
> personally have to think about the best choice to use. If I were using
> a block cipher, CBC with the provided ivec seems like it should be okay,
> but assuming 128-bit cipher blocks, after around 2^64 blocks one would
> expect to experience a block collision which reveals the XOR of the
> plaintexts of the preceding two blocks[1]. Deriving a key with
> HKDF(key, ivec) and using counter mode might be safer, unless I'm
> missing something, which I easily could be. If I were using a stream
> cipher, I would have to do research to figure out how to incorporate the
> ivec.
The user-supplied IV is really just a seed, it will be hashed with some other
uniqifiers (pageID,txnID) before being passed to the cipher. I suppose we
could make some recommendations on ciphers and modes, but really I think it's
up to the user to determine what kind of strength/speed tradeoffs they'll accept.
I would expect stream ciphers to be used, in general.
> * Not wanting to depend on crypto libraries seems like a valid concern.
> Teaching the LMDB code how to dynamically load encryption plugins
> doesn't necessarily seem attractive either.
We'll probably do the dynamic loading anyway, as noted above.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
5 years, 9 months
None
by None
by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id v7AK4Rlv006874
(version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);
Thu, 10 Aug 2017 16:04:29 -0400
Subject: Re: LMDB encryption support
To: Howard Chu <hyc(a)symas.com>
References: <c4dab8cf-66b8-0873-f6a9-9c59a551a155(a)symas.com>
From: Greg Hudson <ghudson(a)mit.edu>
Message-ID: <e9dce464-9d5a-971a-bb8e-d4d95168bcb0(a)mit.edu>
Date: Thu, 10 Aug 2017 16:04:27 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
Thunderbird/45.8.0
MIME-Version: 1.0
In-Reply-To: <c4dab8cf-66b8-0873-f6a9-9c59a551a155(a)symas.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Brightmail-Tracker:
H4sIAAAAAAAAA+NgFnrFIsWRmVeSWpSXmKPExsUixG6nouu3pyfS4MtKU4vb236zWLxd8IDd
gcnjy9JvLB6NCy4zBzBFcdmkpOZklqUW6dslcGXcutfKVLCGv+L37MQGxhM8XYycHBICJhKn
Nv1m7mLk4hASWMwk8a39LyOEs5FRYs7UqSwQzlEmidU/bjCCtAgLqEhc7l7C3sXIwSEiICex
bbcpSFhIwEbiYdsEFhCbTUBZYv3+rSwgJcwC9hL758mChHkFrCR2LHnICmKzCKhKnNw9lwnE
FhWIkHjYuYsdokZQ4uTMJ2BjOAVsJU6u+gm2lVlAXeLPvEvMELa8xPa3c5gnMArMQtIyC0nZ
LCRlCxiZVzHKpuRW6eYmZuYUpybrFicn5uWlFuka6uVmluilppRuYgQFKackzw7GM2+8DjEK
cDAq8fAmiHZHCrEmlhVX5h5ilORgUhLlLf4EFOJLyk+pzEgszogvKs1JLT7EKMHBrCTC67K7
J1KINyWxsiq1KB8mJc3BoiTOK6HRGCEkkJ5YkpqdmlqQWgSTleHgUJLgfbQLqFGwKDU9tSIt
M6cEIc3EwQkynAdouDfY8OKCxNzizHSI/ClGRSlx3j6QZgGQREZpHlwvOImkcpx5xSgO9Iow
bx9IOw8wAcF1vwIazAQ0OMK3E2RwSSJCSqqB0fmdyYtHAlUfV9l5xtzetozfwO/8cbucal6G
sO3fNLNWbfaqYUzW7Jc6sYqnufvuis0mRzTTJkWvfLZl3crXs3/7ZN59evrXcYfpDzdYm+7M
FFHZsS3R62nEw+zN63wc8sS6vvhpLhA25NmcuZLj14FMt9enH1YsSqzXjl310U75+dGQRJX1
0UosxRmJhlrMRcWJANmcCPP9AgAA
X-Spam-Score: -1.5 (-)
X-Spam-Report: Spam detection software,
running on the system "gauss.openldap.net", has
identified this incoming email as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: On 08/10/2017 11:55 AM,
Howard Chu wrote: > Thoughts? Hardcode
1 algorithm, or leave it pluggable? Some thoughts, without advocating for
either option: * If support isn't built-in,
then generic LMDB tools (including
mdb_copy/dump/load/stat) can't operate on encrypted databases, if they need
plaintext pages to work. [...]
Content analysis details: (-1.5 points, 5.0 required)
pts rule name description
---- ----------------------
--------------------------------------------------
-2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/,
medium
trust [18.9.25.12 listed in list.dnswl.org]
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked.
See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information. [URIs: cryptographyengineering.com]
0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60%
[score: 0.4992]
0.0 UNPARSEABLE_RELAY Informational: message has unparseable relay lines
Cc: "OpenLDAP-devel(a)openldap.org" <OpenLDAP-devel(a)openldap.org>
X-BeenThere: openldap-devel(a)openldap.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: OpenLDAP development discussion list <openldap-devel.openldap.org>
List-Unsubscribe: <http://www.openldap.org/lists/mm/options/openldap-devel>,
<mailto:openldap-devel-request@openldap.org?subject=unsubscribe>
List-Archive: <http://www.openldap.org/lists/openldap-devel/>
List-Post: <mailto:openldap-devel@openldap.org>
List-Help: <mailto:openldap-devel-request@openldap.org?subject=help>
List-Subscribe: <http://www.openldap.org/lists/mm/listinfo/openldap-devel>,
<mailto:openldap-devel-request@openldap.org?subject=subscribe>
X-List-Received-Date: Thu, 10 Aug 2017 20:04:40 -0000
On 08/10/2017 11:55 AM, Howard Chu wrote:
> Thoughts? Hardcode 1 algorithm, or leave it pluggable?
Some thoughts, without advocating for either option:
* If support isn't built-in, then generic LMDB tools (including
mdb_copy/dump/load/stat) can't operate on encrypted databases, if they
need plaintext pages to work.
* Built-in support doesn't necessarily mean hardcoding an algorithm for
all time, if the meta pages can include an algorithm selector. One of
the selector values could even mean "use application callbacks".
* Is the page size guaranteed to be a multiple of 16 bytes? 32 bytes?
I would assume yes to both; documenting that would make it easier to use
block ciphers since ciphertext expansion isn't allowed.
* Application writers are more likely to get encryption callbacks wrong
than Howard is. They could ignore the IV (making it easy to detect
duplicate initial blocks within a page) or even do pure ECB encryption
(making it easy to detect duplicate blocks anywhere). Less egregiously,
applications might not make the ideal choice of cipher mode. I would
personally have to think about the best choice to use. If I were using
a block cipher, CBC with the provided ivec seems like it should be okay,
but assuming 128-bit cipher blocks, after around 2^64 blocks one would
expect to experience a block collision which reveals the XOR of the
plaintexts of the preceding two blocks[1]. Deriving a key with
HKDF(key, ivec) and using counter mode might be safer, unless I'm
missing something, which I easily could be. If I were using a stream
cipher, I would have to do research to figure out how to incorporate the
ivec.
* Not wanting to depend on crypto libraries seems like a valid concern.
Teaching the LMDB code how to dynamically load encryption plugins
doesn't necessarily seem attractive either.
[1]
https://blog.cryptographyengineering.com/2016/08/24/attack-of-week-64-bit...
5 years, 9 months
Re: LMDB encryption support
by Howard Chu
Timur Kristóf wrote:
> Hi,
>
>> I've recently added support for page-level encryption to LMDB 1.x
>> using user-supplied callbacks
>
> That does sound cool. :)
>
>> One question is whether we should actually make this pluggable like
>> this, or
>> we should just hardcode support for a specific algorithm and leave it
>> at that.
>
> I vote on keeping it pluggable, so every crypograpy nut out there can
> use their favourite mechanism.
Yeah, that's still my inclination as well. And yes, there's a reference
chacha20 implementation already, which I've been using for testing.
>> One
>> complication is that if the algorithm is actually user-selectable, we
>> need to
>> dynamically adjust DB page layouts to accommodate different nonce/IV
>> and
>> signature sizes. (Currently MDB_page metadata is a statically
>> defined
>> structure. A dynamic size element here will make processing slower.)
>
> What if page size would still be static, but that static size would be
> user-defined on a per-environment basis?
We sort of support that already, allowing page sizes larger than the OS
pagesize to be used. So I guess it's not too big of a change.
> Question: will this affect performance on non-encrypted databases?
Ideally, not. ;) It's a bit early to tell.
Anyway, the API I originally quoted needs to be tweaked to accomodate the
authentication signature support so this is all still in flux.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
5 years, 9 months
LMDB encryption support
by Howard Chu
I've recently added support for page-level encryption to LMDB 1.x using
user-supplied callbacks:
/** @brief A callback function used to encrypt/decrypt pages in the env.
*
* Encrypt or decrypt the data in src and store the result in dst using the
* provided key. The result must be the same number of bytes as the input.
* The input size will always be a multiple of the page size.
* @param[in] src The input data to be transformed.
* @param[out] dst Storage for the result.
* @param[in] key An array of two values: key[0] is the encryption key,
* and key[1] is the initialization vector.
* @param[in] encdec 1 to encrypt, 0 to decrypt.
*/
typedef void (MDB_enc_func)(const MDB_val *src, MDB_val *dst, const MDB_val
*key, int encdec);
/** @brief Set encryption on an environment.
*
* This must be called before #mdb_env_open().
* It implicitly sets #MDB_REMAP_CHUNKS on the env.
* @param[in] env An environment handle returned by #mdb_env_create().
* @param[in] func An #MDB_enc_func function.
* @param[in] key An array of two values: key[0] is the encryption key,
* and key[1] is the initialization vector.
* @return A non-zero error value on failure and 0 on success.
*/
int mdb_env_set_encrypt(MDB_env *env, MDB_enc_func *func, const MDB_val *key);
I intend to extend this a bit further to support authenticated encryption
instead, so that each page also yields a signature that can be used to detect
tampering or corruption.
One question is whether we should actually make this pluggable like this, or
we should just hardcode support for a specific algorithm and leave it at that.
For comparison, BerkeleyDB supported AES128 with HMAC-SHA1 for a 20 byte per
page signature. In some ways it simplifies use if the DB just takes care of
everything and there's only one hardcoded mechanism. Also, user-supplied
callback relies on the app developer to know what they're doing with the
crypto code.
From the LMDB maintenance perspective, it's simpler for us not to have
hardcoded dependencies on crypto libraries. Also, leaving it pluggable keeps
the door open for 3rd party hardware-accelerated crypto engines. One
complication is that if the algorithm is actually user-selectable, we need to
dynamically adjust DB page layouts to accommodate different nonce/IV and
signature sizes. (Currently MDB_page metadata is a statically defined
structure. A dynamic size element here will make processing slower.)
Thoughts? Hardcode 1 algorithm, or leave it pluggable?
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
5 years, 9 months
RE: ITS#8654 - Option for LDAP client to bind to a local address
by Quanah Gibson-Mount
Hi Daniel,
I would suggest sending in a patch against master to the -devel list for
review.
For final inclusion if it is approved, see:
<http://www.openldap.org/devel/contributing.html>
Regards,
Quanah
--On Monday, June 12, 2017 8:34 PM +0000 Daniel Le <daniel.le(a)exfo.com>
wrote:
> I've got a chance to write (and test) the code to add API support for
> socket binding addresses.
>
> Should I send the code diff to this openldap-devel email list for review?
> How to submit a patch request?
>
> Daniel
>
> -----Original Message-----
> From: Daniel Le
> Sent: Tuesday, May 16, 2017 6:02 PM
> To: 'openldap-devel(a)openldap.org' <openldap-devel(a)openldap.org>
> Subject: ITS#8654 - Option for LDAP client to bind to a local address
>
> Hello,
>
> In reference to the enhancement request ITS#865, please comment on the
> following to add support for binding a local IP address to client socket.
> This is just an outline of changes for one local address. I am not sure
> whether a list of local addresses is necessary. If it is, then a new
> function, similarly to ldap_url_parsehosts, may be written to parse the
> list of local addresses and store them into a linked list. In my use
> case, only one IPv4 or IPv6 local address is used for binding.
>
> - Modify ldap.h and ldap_set_option to handle the new option
> LDAP_OPT_LOCAL_ADDRESS. Should it be named LDAP_OPT_CLIENT_ADDRESS,
> LDAP_OPT_SOCKET_BIND_ADDRESS...?
>
> - Modify struct ldapoptions in ldap-int.h to add element "char
> *ldo_local_address" to hold client local address when
> ldap_set_option(LDAP_OPT_LOCAL_ADDRESS...) is executed. This can char
> pointer can point to an IPv4 address or IPv6 address.
>
> - ldap_connect_to_host() in os-ip.c
> After the connection socket is created (ldap_int_socket) and before it
> is connected (ldap_pvt_connect), extract the local IP address. If local
> address family (AF_INET/ AF_INET6) matches the one of the host, bind
> socket to the local address.
>
> Regards,
> Daniel
>
>
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 9 months