openldap-bugs August 2016

openldap-bugs@openldap.org

21 participants
50 discussions

Re: (ITS#8489) LMDB: C_EOF not cleared on cursor in mdb_cursor_put
by hyc＠symas.com 31 Aug '16

31 Aug '16

juerg.bircher(a)gmail.com wrote: > Full_Name: Juerg Bircher > Version: LMDB (master) > OS: MacOS / Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (178.82.36.179) > > > > The function mdb_cursor_put does not clear possibly set C_EOF flag on cursor. > Therefore an successive mdb_cursor_get call using MDB_NEXT may get an > MDB_NOTFOUND result even thought not having put the new value at the end of the > database. > > The following example should illustrate the issue: Thanks for the report, fixed now in mdb.master -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8489) LMDB: C_EOF not cleared on cursor in mdb_cursor_put
by juerg.bircher＠gmail.com 31 Aug '16

31 Aug '16

Full_Name: Juerg Bircher Version: LMDB (master) OS: MacOS / Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (178.82.36.179) The function mdb_cursor_put does not clear possibly set C_EOF flag on cursor. Therefore an successive mdb_cursor_get call using MDB_NEXT may get an MDB_NOTFOUND result even thought not having put the new value at the end of the database. The following example should illustrate the issue: Assume an database having two committed values "0" and "5". Begin a new transaction and open an cursor for the next steps: step 1 put 1 get next => returns 5 ==> ok step 2 put 6 get next 3E r returns MDB_NOTFOUND as at end of the database (sets C_EOF on cursor) ==> ok step 3 put 2 get next => should return 5 but due to the set C_EOF flag returns MDB_NOTFOUND ==> failure The fix could be to clear C_EOF on any successful mdb_cursor_put calls. The following code can be used to verify the issue. #include "lmdb.h" #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <string.h> static char *env_name = "/tmp/testdb"; static char *db_name = "tt"22; #define MAX_MAP_SIZ (1024 * 1024) #define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr) #define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0)) #define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf%2tdtderr, \ "%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort())) static MDB_env *env; static MDB_dbi dbi; static void createTestValue(int num, MDB_val *key, MDB_val *val) { static char buf[16]; sprintf(buf, "%03d", num); key->mv_data = (void *)buf; key->mv_size = strlen((const char *)buf); val->mv_data = key->mv_data; val->mv_size = key->mv_size; } static void setup() { int rc; MDB_txn *txn; MDB_cursor *cursor; MDB_val key, val; E(mdb_env_create(&env)); E(mdb_env_set_maxreaders(env, 2)); E(mdb_env_set_maxdbs(env, 10)); E(mdb_env_set_mapsize(env, MAX_MAP_SIZ)); E(mdb_env_open(env, env_name, 0, 0664)); E(mdb_txn_begin(env, NULL,%0, &txn)); E(mdb_dbi_open(txn, db_name, MDB_CREATE, &dbi)); E(mdb_cursor_open(txn, dbi, &cursor)); createTestValue(0, &key, &val); E(mdb_cursor_put(cursor, &key, &val, 0)); createTestValue(5, &key, &val); E(mdb_cursor_put(cursor, &key, &val, 0)); mdb_cursor_close(cursor); E(mdb_txn_commit(txn)); } static int step1(MDB_cursor *cursor) { int rc; MDB_val key, val, rkey, rval; createTestValue(1, &key, &val); E(mdb_cursor_put(cursor, &key, &val, MDB_NOOVERWRITE)); E(mdb_cursor_get(cursor, &rkey, &rval, MDB_NEXT)); // ==> returns 5 return 0; } static int step2(MDB_cursor *cursor) { int rc; MDB_val key, val, rkey, rval; createTestValue(6, &key, &val); E(mdb_cursor_put(cursor, &key, &val, MDB_NOOVERWRITE)); rc = mdb_cursor_get(cursor, &rkey, &rval, MDB_NEXT); // ==> return nothing as 6 is the last one if (rc != MDB_NOTFOUND) { fprintf(stderr, "step2 failed\n"); return -1; } return 0; } static int step3(MDB_cursor *cursor) { int rc; MDB_val key, val, rkey, rval; createTestValue(2, &key, &val); E(mdb_cursor_put(cursor, &key, &val, MDB_NOOVERWRITE)); E(mdb_cursor_get(cursor, &rkey, &rval, MDB_NEXT)); // ==> should return 5 return 0; } static void test() { int rc; MDB_txn *txn; MDB_cursor *cursor; E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); step1(cursor); step2(cursor); step3(cursor); mdb_cursor_close(cursor); E(mdb_txn_commit(txn)); } static void cleanup() { mdb_dbi_close(env, dbi); mdb_env_close(env); char name[1024]; sprintf(name% "22%s/data.mdb", db_name); unlink(name); sprintf(name, "%s/lock.mdb", db_name); unlink(name); } int main(int argc,char * argv[]) { setup(); test(); cleanup(); }

1 0

Re: (ITS#7682) LMDB: [PATCH] mdb_env_copy should retry open() if O_DIRECT fails
by h.b.furuseth＠usit.uio.no 31 Aug '16

31 Aug '16

Sorry, looks like I dropped this message. Yes, #ifdef F_NOCACHE / #elif defined O_DIRECT / #endif seems better. Included in branch "mdb/fopen" in my repo at UiO. On 24. des. 2013 18:18, Howard Chu wrote: > h.b.furuseth(a)usit.uio.no wrote: >> This was fixed in LMDB 0.9.9, except: If both O_DIRECT and >> F_NOCACHE are #defined, both are used. Looks a bit strange. >> Maybe just one should be used in that case. (Which?) Also I'm >> still wondering if env_open needs to fail if F_NOCACHE fails. >> >> Anyway, maybe it should be something like this. The "else {}" >> is in case O_DIRECT is an unsupported dummy definiton: > > Isn't that in the wrong order then? What you suggest here will use O_DIRECT > even if it's a dummy. > > I don't see any reason to fail the operation. They just want to make a > copy/backup; even if the optimization of avoiding pollution of the buffer > cache can't be done, they still want the copy to happen. > >> #ifdef O_DIRECT >> if ((rc = fcntl(newfd, F_GETFL)) != -1) >> (void) fcntl(newfd, F_SETFL, rc | O_DIRECT); >> else >> #endif >> { >> #ifdef F_NOCACHE /* __APPLE__ */ >> (void) fcntl(newfd, F_NOCACHE, 1); >> #endif >> } -- Hallvard

1 0

Re: (ITS#8488) OpenSSL 1.1.0
by hyc＠symas.com 30 Aug '16

30 Aug '16

norm.green(a)gemtalksystems.com wrote: > Full_Name: Norm Green > Version: 2.4.43 > OS: Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (50.240.19.49) > > > Please support OpenSSL 1.1.0. Current versions of OpenLDAP will not build with > OpenSSL 1.1.0 due to OpenSSL API changes. > > Already supported in git master. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8488) OpenSSL 1.1.0
by norm.green＠gemtalksystems.com 30 Aug '16

30 Aug '16

Full_Name: Norm Green Version: 2.4.43 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (50.240.19.49) Please support OpenSSL 1.1.0. Current versions of OpenLDAP will not build with OpenSSL 1.1.0 due to OpenSSL API changes.

1 0

Re: (ITS#8487) Properly handle realloc() failures
by hyc＠symas.com 29 Aug '16

29 Aug '16

cissp(a)bryongloden.com wrote: > Full_Name: Bryon Gloden, CISSP. > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (161.133.194.55) > > > Greetings. On line numbers 123 and 134 of liblmdb/midl.c > (https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/midl.c) there > are two common realloc() mistakes. Passing one pointer into realloc() and > assigning the result directly into that same pointer variable can cause a memory > leak if the reallocation fails, because the original allocation will still > exist. The correct way to do this is to use a temporary pointer variable. > > Credit: http://stackoverflow.com/a/11548901 > > Found by https://github.com/bryongloden/cppcheck > > This is all false. On line number 123 we are shrinking the memory region - this will never fail. At worst it can only return the original pointer unchanged. On line number 134 we are leaving the original pointer unchanged on failure, it is not leaked. The caller can free it if it wants to. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8487) Properly handle realloc() failures
by cissp＠bryongloden.com 29 Aug '16

29 Aug '16

Full_Name: Bryon Gloden, CISSP. Version: OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (161.133.194.55) Greetings. On line numbers 123 and 134 of liblmdb/midl.c (https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/midl.c) there are two common realloc() mistakes. Passing one pointer into realloc() and assigning the result directly into that same pointer variable can cause a memory leak if the reallocation fails, because the original allocation will still exist. The correct way to do this is to use a temporary pointer variable. Credit: http://stackoverflow.com/a/11548901 Found by https://github.com/bryongloden/cppcheck

1 0

(ITS#8486) Syncprov sessionlog is inefficient, kills perf
by quanah＠openldap.org 29 Aug '16

29 Aug '16

Full_Name: Quanah Gibson-Mount Version: 2.4.44 OS: Linux 2.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (75.111.52.177) After setting the sessionlog for syncprov to a large size so as to avoid ITS#8125. However, due to the sessionlog being a linear linked list, it can be vastly inefficient to iterate, and slows down the master to the point that searches can time out. Howard suggests converting it into a tree instead to help obviate the load this currently induces.

1 0

(ITS#8485) [PATCH] Adding support for encrypted server private keys
by ahnolds＠gmail.com 28 Aug '16

28 Aug '16

Full_Name: Alec Cooper Version: HEAD of master branch OS: Ubuntu Linux 16.04 URL: ftp://ftp.openldap.org/incoming/Alec-Cooper-160827.patch Submission from: (NULL) (73.134.243.211) Adding support for encrypted server private keys. The meat of this patch is changes to tls2.c, tls_g.c, tls_m.c, and tls_o.c to send a password to the underlying TLS library. Changes to ldap.h, init.c and ldap-int.h expose the new LDAP_OPT_X_TLS_KEYPASSWORD in the main API. Changes in the contrib/ldapc++ directory expose the corresponding TlsOptions::KEYPASSWORD option in the C++ API. Changes in the servers directory expose equivalent options for servers as configuration file entries or environment variables. Changes in the doc directory add documentation about the new options and remove statements that indicated that encrypted keyfiles are not supported. New files in the test directory are for testing TLS connections, both in general and with encrypted keyfiles. Tests pass for OpenSSL and GnuTLS using PEM formatted certs and keys, and for MozNSS using cert/key databases. The new unit test (test065-tls) has been written to detect when using NSS, and use the cert/key databases in this case. I have been unable to get a working version of libnsspem, so I cannot test MozNSS with (or without) encrypted keyfiles - testing for this case would be welcome! Notice of origin: The attached patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch were developed by Alec Cooper ahnolds(a)gmail.com. I have not assigned rights and/or interest in this work to any party. Rights statement: I, Alec Cooper, hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice. The patch has been uploaded to the OpenLDAP FTP server and can be found at ftp://ftp.openldap.org/incoming/Alec-Cooper-160827.patch

1 0

Re: (ITS#7992) lmdb: Windows: assume file paths are UTF-8, encode to UTF-16 for WinAPI and enable compiling when UNICODE is defined
by h.b.furuseth＠usit.uio.no 27 Aug '16

27 Aug '16

Seems the test for MultiByteToWideChar() == 0xFFFD can be dropped. The MultiByteToWideChar doc mentions U+FFFD under Return Value. But when I test bad UTF-8 characters, it inserts 0xFFFD for the bad character(s) in the output string. It doesn't return 0xFFFD. Or if we pass the MB_ERR_INVALID_CHARS flag, it returns 0 and sets GetLastError() as usual. (I wonder if I had a hand in this code - looks like my kind of thing to write a draft from the Windows doc but leave the Windows testing to someone else:-)

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs August 2016