openldap-bugs May 2014

openldap-bugs@openldap.org

26 participants
75 discussions

Re: (ITS#7844) LMDB Delete Cursor inconsistencies
by armon.dadgar＠gmail.com 16 May '14

16 May '14

--537675cf_2ae8944a_1704c Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline When I run the attached test case on my machine, I=E2=80=99m hitting the = failing case. Here is the test output:=C2=A0https://gist.github.com/armon/e529d7909fe30= 1126fc6 My steps: =24 clang 7844.c mdb.c midl.c =24 mkdir testdb =24 ./a.out =24 clang -v Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn) Target: x86=5F64-apple-darwin13.1.0 Thread model: posix =24 uname -a Darwin Armons-MacBook-Air.local 13.1.0 Darwin Kernel Version 13.1.0: Thu = Jan 16 19:40:37 PST 2014; root:xnu-2422.90.20=7E2/RELEASE=5FX86=5F64 x86=5F= 64 Best Regards, Armon Dadgar =46rom:=C2=A0Howard Chu hyc=40symas.com Reply:=C2=A0Howard Chu hyc=40symas.com Date:=C2=A0May 16, 2014 at 11:06:49 AM To:=C2=A0armon.dadgar=40gmail.com armon.dadgar=40gmail.com, openldap-its=40= openldap.org openldap-its=40openldap.org Subject:=C2=A0 Re: (ITS=237844) LMDB Delete Cursor inconsistencies =20 armon.dadgar=40gmail.com wrote: =20 > --5372ac85=5F8edbdab=5F1271 =20 > Content-Type: text/plain; charset=3D=22utf-8=22 =20 > Content-Transfer-Encoding: quoted-printable =20 > Content-Disposition: inline =20 > =20 > =3D46or now, we have application code to retry the delete until no furt= her =3D =20 > rows are removed. =20 > Still, it would be nice to have this resolved (and tested) in master=3D= 21 =20 Unable to reproduce the issue. I've attached my test program based on you= r =20 description. =20 -- =20 -- Howard Chu =20 CTO, Symas Corp. http://www.symas.com =20 Director, Highland Sun http://highlandsun.com/hyc/ =20 Chief Architect, OpenLDAP http://www.openldap.org/project/ =20 --537675cf_2ae8944a_1704c Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline <html><head><style>body=7Bfont-family:Helvetica,Arial;font-size:13px=7D</= style></head><body style=3D=22word-wrap: break-word; -webkit-nbsp-mode: s= pace; -webkit-line-break: after-white-space;=22><div id=3D=22bloop=5Fcust= omfont=22 style=3D=22font-family:Helvetica,Arial;font-size:13px; color: r= gba(0,0,0,1.0); margin: 0px; line-height: auto;=22>When I run the attache= d test case on my machine, I=E2=80=99m hitting the failing case.</div><di= v id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial;fo= nt-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22>= <br></div><div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helve= tica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-heig= ht: auto;=22>Here is the test output: https://gist.github.com/armon/= e529d7909fe301126fc6</div><div id=3D=22bloop=5Fcustomfont=22 style=3D=22f= ont-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin= : 0px; line-height: auto;=22><br></div><div id=3D=22bloop=5Fcustomfont=22= style=3D=22font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0= ,1.0); margin: 0px; line-height: auto;=22>My steps:</div><div id=3D=22blo= op=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial;font-size:13px;= color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22>=24 clang 784= 4.c mdb.c midl.c</div><div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-= family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0p= x; line-height: auto;=22>=24 mkdir testdb</div><div id=3D=22bloop=5Fcusto= mfont=22 style=3D=22font-family:Helvetica,Arial;font-size:13px; color: rg= ba(0,0,0,1.0); margin: 0px; line-height: auto;=22>=24 ./a.out</div><div i= d=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial;font-= size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22><br= ></div><div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetic= a,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height:= auto;=22><div id=3D=22bloop=5Fcustomfont=22 style=3D=22margin: 0px;=22>=24= clang -v</div><div id=3D=22bloop=5Fcustomfont=22 style=3D=22margin: 0px;= =22>Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)</div><= div id=3D=22bloop=5Fcustomfont=22 style=3D=22margin: 0px;=22>Target: x86=5F= 64-apple-darwin13.1.0</div><div id=3D=22bloop=5Fcustomfont=22 style=3D=22= margin: 0px;=22>Thread model: posix</div><div id=3D=22bloop=5Fcustomfont=22= style=3D=22margin: 0px;=22><br></div><div id=3D=22bloop=5Fcustomfont=22 = style=3D=22margin: 0px;=22><div id=3D=22bloop=5Fcustomfont=22 style=3D=22= margin: 0px;=22>=24 uname -a</div><div id=3D=22bloop=5Fcustomfont=22 styl= e=3D=22margin: 0px;=22>Darwin Armons-MacBook-Air.local 13.1.0 Darwin Kern= el Version 13.1.0: Thu Jan 16 19:40:37 PST 2014; root:xnu-2422.90.20=7E2/= RELEASE=5FX86=5F64 x86=5F64</div></div></div><div id=3D=22bloop=5Fcustomf= ont=22 style=3D=22font-family:Helvetica,Arial;font-size:13px; color: rgba= (0,0,0,1.0); margin: 0px; line-height: auto;=22><br></div> <div id=3D=22b= loop=5Fsign=5F1400272071967389952=22 class=3D=22bloop=5Fsign=22><div styl= e=3D=22font-family:helvetica,arial;font-size:13px=22>Best Regards,</div><= div style=3D=22font-family:helvetica,arial;font-size:13px=22>Armon Dadgar= <br></div></div> <div style=3D=22color:black=22><br>=46rom: <span st= yle=3D=22color:black=22>Howard Chu</span> <a href=3D=22mailto:hyc=40symas= .com=22>hyc=40symas.com</a><br>Reply: <span style=3D=22color:black=22= >Howard Chu</span> <a href=3D=22mailto:hyc=40symas.com=22>hyc=40symas.com= </a><br>Date: <span style=3D=22color:black=22>May 16, 2014 at 11:06:= 49 AM</span><br>To: <span style=3D=22color:black=22>armon.dadgar=40g= mail.com</span> <a href=3D=22mailto:armon.dadgar=40gmail.com=22>armon.dad= gar=40gmail.com</a>, <span style=3D=22color:black=22>openldap-its=40openl= dap.org</span> <a href=3D=22mailto:openldap-its=40openldap.org=22>openlda= p-its=40openldap.org</a><br>Subject: <span style=3D=22color:black=22= > Re: (ITS=237844) LMDB Delete Cursor inconsistencies <br></span></div><b= r> <blockquote type=3D=22cite=22 class=3D=22clean=5Fbq=22><span><div><div= ></div><div>armon.dadgar=40gmail.com wrote: <br>> --5372ac85=5F8edbdab=5F1271 <br>> Content-Type: text/plain; charset=3D=22utf-8=22 <br>> Content-Transfer-Encoding: quoted-printable <br>> Content-Disposition: inline <br>> <br>> =3D46or now, we have application code to retry the delete until = no further =3D <br>> rows are removed. <br>> Still, it would be nice to have this resolved (and tested) in ma= ster=3D21 <br> <br>Unable to reproduce the issue. I've attached my test program based on= your =20 <br>description. <br> <br>-- =20 <br> -- Howard Chu <br> CTO, Symas Corp. http://www.symas.com <br> Director, Highland Sun http://highlandsun.com/hyc/ <br> Chief Architect, OpenLDAP http://www.openldap.org/project/ <br><hr></div></div></span></blockquote></body></html> --537675cf_2ae8944a_1704c--

1 0

Re: (ITS#7844) LMDB Delete Cursor inconsistencies
by hyc＠symas.com 16 May '14

16 May '14

This is a multi-part message in MIME format. --------------010403060500020806080002 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit armon.dadgar(a)gmail.com wrote: > --5372ac85_8edbdab_1271 > Content-Type: text/plain; charset="utf-8" > Content-Transfer-Encoding: quoted-printable > Content-Disposition: inline > > =46or now, we have application code to retry the delete until no further = > rows are removed. > Still, it would be nice to have this resolved (and tested) in master=21 Unable to reproduce the issue. I've attached my test program based on your description. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ --------------010403060500020806080002 Content-Type: text/plain; charset=UTF-8; name="7844.c.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="7844.c.txt" /* 7844.c - memory-mapped database tester/toy */ /* * Copyright 2014 Howard Chu, Symas Corp. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted only as authorized by the OpenLDAP * Public License. * * A copy of this license is available in the file LICENSE in the * top-level directory of the distribution or, alternatively, at * <http://www.OpenLDAP.org/license.html>. */ #include <stdio.h> #include <stdlib.h> #include <string.h> #include <uuid/uuid.h> #include "lmdb.h" #define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr) #define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0)) #define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \ "%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort())) #define UUID_LEN 36 typedef char kval[UUID_LEN*2+2]; char sval[100]; int main(int argc,char * argv[]) { int i = 0, j = 0, rc; MDB_env *env; MDB_dbi kdbi, ddbi; MDB_val key, ikey, data; MDB_txn *txn; MDB_stat mst; MDB_cursor *cursor, *cur2; int count; unsigned long lastid = 0; kval *keys; char prefix[UUID_LEN+1], *ptr; uuid_t uuid; count = 128; keys = (kval *)malloc(count*sizeof(kval)); E(mdb_env_create(&env)); E(mdb_env_set_mapsize(env, 128*1024*1024)); E(mdb_env_set_maxdbs(env, 4)); E(mdb_env_open(env, "./testdb", MDB_NOSYNC, 0664)); E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_open(txn, "kvs", MDB_CREATE|MDB_INTEGERKEY, &ddbi)); E(mdb_open(txn, "kvs_id_idx", MDB_CREATE, &kdbi)); key.mv_size = UUID_LEN*2+1; data.mv_size = sizeof(sval); data.mv_data = sval; ikey.mv_size = sizeof(lastid); ikey.mv_data = &lastid; uuid_generate(uuid); uuid_unparse(uuid, prefix); printf("Adding %d values\n", count); for (i=0;i<count;i++) { uuid_generate(uuid); ptr = (char *)&keys[i]; memcpy(ptr, prefix, UUID_LEN); ptr[UUID_LEN] = '/'; uuid_unparse(uuid, ptr+UUID_LEN+1); ptr[UUID_LEN*2+1] = '\0'; printf("\t%s\n", ptr); key.mv_data = ptr; sprintf(sval, "Some stuff for key %s", ptr); lastid++; if (RES(MDB_KEYEXIST, mdb_put(txn, kdbi, &key, &ikey, MDB_NOOVERWRITE))) { j++; ikey.mv_size = sizeof(lastid); ikey.mv_data = &lastid; } E(mdb_put(txn, ddbi, &ikey, &data, MDB_NOOVERWRITE)); } if (j) printf("%d duplicates skipped\n", j); E(mdb_txn_commit(txn)); E(mdb_env_stat(env, &mst)); uuid_generate(uuid); uuid_unparse(uuid, prefix); printf("Adding %d values\n", count); E(mdb_txn_begin(env, NULL, 0, &txn)); for (i=0;i<count;i++) { uuid_generate(uuid); ptr = (char *)&keys[i]; memcpy(ptr, prefix, UUID_LEN); ptr[UUID_LEN] = '/'; uuid_unparse(uuid, ptr+UUID_LEN+1); ptr[UUID_LEN*2+1] = '\0'; printf("\t%s\n", ptr); key.mv_data = ptr; sprintf(sval, "Some stuff for key %s", ptr); lastid++; if (RES(MDB_KEYEXIST, mdb_put(txn, kdbi, &key, &ikey, MDB_NOOVERWRITE))) { j++; ikey.mv_size = sizeof(lastid); ikey.mv_data = &lastid; } E(mdb_put(txn, ddbi, &ikey, &data, MDB_NOOVERWRITE)); } if (j) printf("%d duplicates skipped\n", j); E(mdb_txn_commit(txn)); E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_cursor_open(txn, kdbi, &cursor)); E(mdb_cursor_open(txn, ddbi, &cur2)); key.mv_size = UUID_LEN; key.mv_data = prefix; j = 0; i = count; rc = mdb_cursor_get(cursor, &key, &ikey, MDB_SET_RANGE); while (rc == 0) { #if 0 printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, (int) key.mv_size, (char *) key.mv_data, data.mv_data, (int) data.mv_size, (char *) data.mv_data); #endif E(mdb_cursor_get(cur2, &ikey, &data, MDB_SET)); E(mdb_cursor_del(cur2, 0)); E(mdb_cursor_del(cursor, 0)); j++; i--; if (i == 0) break; rc = mdb_cursor_get(cursor, &key, &ikey, MDB_NEXT); } mdb_cursor_close(cursor); mdb_cursor_close(cur2); E(mdb_txn_commit(txn)); printf("Deleted %d values\n", j); mdb_env_close(env); return 0; } --------------010403060500020806080002--

1 0

(ITS#7856) TLS_REQCERT try is same as TLS_REQCERT hard?
by pguenther＠proofpoint.com 16 May '14

16 May '14

Full_Name: Philip Guenther Version: 2.4.39 OS: OpenBSD URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (76.253.0.176) The ldap.conf(5) manpage says this about TLS_REQCERT TLS_REQCERT <level> Specifies what checks to perform on server certificates in a TLS session, if any. The <level> can be specified as one of the following keywords: ... try The server certificate is requested. If no certificate is provided, the session proceeds normally. If a bad certificate is provided, the session is immediately terminated. demand | hard These keywords are equivalent. The server certificate is requested. If no certificate is provided, or a bad certificate is provided, the session is immediately terminated. This is the default setting. In testing, I can find no difference in behavior between the 'try' and 'hard' keywords. For the ldap* tools, both 'try' and 'hard' seem to place the same requirements on the server. What does "if no certificate is provided" *mean* in terms of server and/or client configuration?

1 0

Re: (ITS#7705) mdb back-end segfaults sporadically with paged searches
by hyc＠symas.com 15 May '14

15 May '14

openldap(a)semyon.org wrote: > On 13-09-23 10:16 PM, Howard Chu wrote: > >> If you backup the DB with slapcat and reload it on another server with >> slapadd, can you still reproduce the fault on the copy? > > Unfortunately, yes. It breaks trying to retrieve the same record as > before, on the same instruction in back_mdb: > > yesterday: > [126393.233615] slapd[19244]: segfault at 7f499f38d000 ip > 00007f4da0e9b6a1 sp 00007f499f1fa540 error 6 in > back_mdb-2.4.so.2.9.2[7f4da0e80000+34000] > > today: > [517860.953779] slapd[24871]: segfault at 7fdeb50a0000 ip > 00007fe2b63ad6a1 sp 00007fdeb4f0d540 error 6 in > back_mdb-2.4.so.2.9.2[7fe2b6392000+34000] > > Semyon A fix is now in git master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7855) Update to latest autotools to support new architecture ppc64le
by hyc＠symas.com 15 May '14

15 May '14

rajesh(a)linux.vnet.ibm.com wrote: > Full_Name: Rajeshkumar S > Version: 2.4.39 > OS: Ubuntu 14.04 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (122.248.161.59) > > > > I am trying to build the openldap package from the source following the > release tarball from > ftp://ftp.openldap.org/pub/OpenLDAP/openldap-release/openldap-2.4.39.tgz. in a > new architecture ppc64le ( IBM PowerPC Little endian ). As the config.guess and > libtool did not have the required patches to identify this new architecture, I > did autoreconf -f -i in my build system whose latest automake and libtool has > the patches > of ppc64le. > autoreconf fails with automake errors as shown below > > automake: error: no 'Makefile.am' found for any configure output > autoreconf: automake failed with exit status: 1 That's to be expected, since there certainly are no Makefile.am files in our source tree. We don't use automake. This is not a bug. > As discussed in the mailing list, Quanah Gibson-Mount suggested me to request an > update for auto-tools to support new architectures. > > The automake with version >= 1.13.4 has the correct config.guess with the > required ppc64le fixes. > Regarding the libtool, the last release is more than 2 years ago I believe, we > managed to get an alpha source release which has all the latest patches to > support the new architecture ppc64le > > An alpha version of libtool with ppc64le support is available at > ftp://alpha.gnu.org/gnu/libtool/libtool-2.4.2.418.tar.gz. Libtool is so perpetually problematic that we will certainly not integrate an alpha version into our source tree. > Hence I request for an update of the auto tools in your build system to support > these new architectures. > > Thanks and Regards > Rajeshkumar S > Linux Technology Center, IBM > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7855) Update to latest autotools to support new architecture ppc64le
by rajesh＠linux.vnet.ibm.com 15 May '14

15 May '14

Full_Name: Rajeshkumar S Version: 2.4.39 OS: Ubuntu 14.04 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (122.248.161.59) I am trying to build the openldap package from the source following the release tarball from ftp://ftp.openldap.org/pub/OpenLDAP/openldap-release/openldap-2.4.39.tgz. in a new architecture ppc64le ( IBM PowerPC Little endian ). As the config.guess and libtool did not have the required patches to identify this new architecture, I did autoreconf -f -i in my build system whose latest automake and libtool has the patches of ppc64le. autoreconf fails with automake errors as shown below automake: error: no 'Makefile.am' found for any configure output autoreconf: automake failed with exit status: 1 As discussed in the mailing list, Quanah Gibson-Mount suggested me to request an update for auto-tools to support new architectures. The automake with version >= 1.13.4 has the correct config.guess with the required ppc64le fixes. Regarding the libtool, the last release is more than 2 years ago I believe, we managed to get an alpha source release which has all the latest patches to support the new architecture ppc64le An alpha version of libtool with ppc64le support is available at ftp://alpha.gnu.org/gnu/libtool/libtool-2.4.2.418.tar.gz. Hence I request for an update of the auto tools in your build system to support these new architectures. Thanks and Regards Rajeshkumar S Linux Technology Center, IBM

1 0

Re: (ITS#7853) Improve safety of mdb_env_close
by hbf＠mail.uio.no 14 May '14

14 May '14

On 05/14/2014 01:18 AM, hyc(a)symas.com wrote: > dw(a)botanicus.net wrote: >> Currently if a user (wilfully or accidentally, say, through composition of >> third party libs) opens the same LMDB environment twice within a process, and >> maintains active read transactions at the time one mdb_env_close() is called, >> all reader slots will be deallocated in all environments due to the logic >> around line 4253 that unconditionally clears reader slots based on PID. >> >> I'd like to avoid this in py-lmdb, and seems there are a few alternatives to >> fix it: >> >> 1) Tell users not to do that (doc fix, nobody reads docs) > > This is already in the doc - don't open the same env twice within one process. 1.5) Use something else than fcntl locks on the lock-file descriptor to control whether to re-init the lockfile and its mutexes/semaphores. Like MDB_NOLOCK. Needed to get any other suggestions off the ground, see Caveats in lmdb.h. Once two MDB_env's in a process has the lockfile open, both lose their fcntl locks when either closes it - and the next mdb_env_open() will reset the lockfile. Then both readers and writers are screwed, and more care with reader slots won't help. Well, one MDB_env could leak its lockfile descriptor instead of closing it, or it could give the descriptor to the other MDB_env for a deferred close if it could find it using some global state. (Could work until someone wrote an open-close loop which would run into the max-open-FD limit.) Either way, no fun. Another way would be #ifdef ... more unportable code ... #endif:-( Surely some OSes have come up with sane way to address this problem, there's just no portable way. >> 2) Externally maintain a set of (st_dev, st_ino) keys representing already >> opened environments, and check any attempt to open an environment against this >> list, failing if data.mdb's key already appears. > > That would require global library state, which is definitely unwanted. Though in one sense we already have that, which is the problem. Global (well, per-process) lock state on open files and mutexes. >> 3) Disable the mdb_env_close() cleanup loop when MDB_NOTLS is enabled. Since no >> reader slot will ever exist in TLS in this mode, the existing mdb_env_close() >> doc "All transactions, databases, and cursors must already be closed before >> calling this function." ensures no readers exist at close time, and the loop >> need not run. > > Could do this, but since it's an incomplete solution, not sure there's any point. That's MDB_NOLOCK anyway. MDB_NOTLS does have reader table entries. > (...) -- Hallvard

1 0

(ITS#7854) minor patches to allow compile with VS2012
by louis＠steelbytes.com 14 May '14

14 May '14

Full_Name: Louis Solomon Version: 2.4.39 OS: Win32 & Win64 URL: Submission from: (NULL) (59.167.196.195) 1) in libraries/libldap/ldif.c move int ldif_debug = 0; to after _all_ the includes. Why? because the linker gets confused with it being declared as declspec(dllexport). Order matters. 2) in include/ac/socket.h EWOULDBLOCK, EINPROGRESS, ETIMEDOUT (lines 101-103) conflict with the windows header files. simply wrap with any rational choice of ifdef/ifndef 3) in include/ldap_utf8.h wrap prototypes for ldap_x_utf8_to_wc (line 61) through to ldap_x_mbs_utf8s (line 99) with #if SIZEOF_WCHAR>=4 to match the ifdef at the top of utf-8-conv.c. Why? because the linker tries to export them as they are defined in the header with declspec(dllexport) but are not in any .o/.obj 4) in include/ldap_cdefs.h the usage of __declspec(dllexport/dllimport) is not quite right ... should be like #if defined(_WIN32) ... # define LDAP_F(type) __declspec(dllimport) type # define LDAP_V(type) extern __declspec(dllimport) type #else # define LDAP_F(type) __declspec(dllexport) type # define LDAP_V(type) extern __declspec(dllexport) type #endif and of course the same for each of the other sets (LBER, LDAP, LDAP_AVL, LDAP_LDIF, etc) With the above patches and making the appropriate .h files from .hin files manually and then creating VS2012 project files, I was successfully able to create and use liblber, libldap, libldap_r as .dlls

1 0

Re: (ITS#7844) LMDB Delete Cursor inconsistencies
by armon.dadgar＠gmail.com 13 May '14

13 May '14

--5372ac85_8edbdab_1271 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline =46or now, we have application code to retry the delete until no further = rows are removed. Still, it would be nice to have this resolved (and tested) in master=21 Best Regards, Armon Dadgar =46rom:=C2=A0David Wilson dw=40hmmz.org Reply:=C2=A0David Wilson dw=40hmmz.org Date:=C2=A0May 13, 2014 at 3:41:08 PM To:=C2=A0armon.dadgar=40gmail.com armon.dadgar=40gmail.com Cc:=C2=A0openldap-its=40openldap.org openldap-its=40openldap.org Subject:=C2=A0 Re: (ITS=237844) LMDB Delete Cursor inconsistencies =20 This sounds very much like a known regression in mdb.master in the past =20 few months. If you are still affected, please retest against LMDB 0.9.11 = =20 which at least was not affected by the regression. =20 David =20 --5372ac85_8edbdab_1271 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline <html><head><style>body=7Bfont-family:Helvetica,Arial;font-size:13px=7D</= style></head><body style=3D=22word-wrap: break-word; -webkit-nbsp-mode: s= pace; -webkit-line-break: after-white-space;=22><div id=3D=22bloop=5Fcust= omfont=22 style=3D=22font-family:Helvetica,Arial;font-size:13px; color: r= gba(0,0,0,1.0); margin: 0px; line-height: auto;=22>=46or now, we have app= lication code to retry the delete until no further rows are removed.</div= ><div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Aria= l;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;= =22>Still, it would be nice to have this resolved (and tested) in master=21= </div><div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica= ,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: = auto;=22><br></div> <div id=3D=22bloop=5Fsign=5F1400024102388026112=22 cl= ass=3D=22bloop=5Fsign=22><div style=3D=22font-family:helvetica,arial;font= -size:13px=22>Best Regards,</div><div style=3D=22font-family:helvetica,ar= ial;font-size:13px=22>Armon Dadgar<br></div></div> <div style=3D=22color:= black=22><br>=46rom: <span style=3D=22color:black=22>David Wilson</s= pan> <a href=3D=22mailto:dw=40hmmz.org=22>dw=40hmmz.org</a><br>Reply:&nbs= p;<span style=3D=22color:black=22>David Wilson</span> <a href=3D=22mailto= :dw=40hmmz.org=22>dw=40hmmz.org</a><br>Date: <span style=3D=22color:= black=22>May 13, 2014 at 3:41:08 PM</span><br>To: <span style=3D=22c= olor:black=22>armon.dadgar=40gmail.com</span> <a href=3D=22mailto:armon.d= adgar=40gmail.com=22>armon.dadgar=40gmail.com</a><br>Cc: <span style= =3D=22color:black=22>openldap-its=40openldap.org</span> <a href=3D=22mail= to:openldap-its=40openldap.org=22>openldap-its=40openldap.org</a><br>Subj= ect: <span style=3D=22color:black=22> Re: (ITS=237844) LMDB Delete C= ursor inconsistencies <br></span></div><br> <blockquote type=3D=22cite=22= class=3D=22clean=5Fbq=22><span><div><div></div><div>This sounds very muc= h like a known regression in mdb.master in the past <br>few months. If you are still affected, please retest against LMDB 0.9= .11 <br>which at least was not affected by the regression. <br> <br> <br>David <br></div></div></span></blockquote></body></html> --5372ac85_8edbdab_1271--

1 0

Re: (ITS#7853) Improve safety of mdb_env_close
by hyc＠symas.com 13 May '14

13 May '14

dw(a)botanicus.net wrote: > Full_Name: David Wilson > Version: LMDB 0.9.11 > OS: > URL: https://raw.githubusercontent.com/dw/py-lmdb/master/misc/readers_mrb_env.pa… > Submission from: (NULL) (178.238.153.20) > > > Currently if a user (wilfully or accidentally, say, through composition of > third party libs) opens the same LMDB environment twice within a process, and > maintains active read transactions at the time one mdb_env_close() is called, > all reader slots will be deallocated in all environments due to the logic > around line 4253 that unconditionally clears reader slots based on PID. > > I'd like to avoid this in py-lmdb, and seems there are a few alternatives to > fix it: > > > 1) Tell users not to do that (doc fix, nobody reads docs) This is already in the doc - don't open the same env twice within one process. > 2) Externally maintain a set of (st_dev, st_ino) keys representing already > opened environments, and check any attempt to open an environment against this > list, failing if data.mdb's key already appears. That would require global library state, which is definitely unwanted. > 3) Disable the mdb_env_close() cleanup loop when MDB_NOTLS is enabled. Since no > reader slot will ever exist in TLS in this mode, the existing mdb_env_close() > doc "All transactions, databases, and cursors must already be closed before > calling this function." ensures no readers exist at close time, and the loop > need not run. Could do this, but since it's an incomplete solution, not sure there's any point. > 4) Modify lock.mdb to include MDB_env* address within the process, and update > mdb_env_close() to invalidate only readers associated with the environment > being closed. I dislike using the MDB_env* as an opaque publicly visible > cookie, but perhaps storing this field might be reused for other things in > future. Hmm. I guess this could work. I think storing the MDB_env* is harmless since the pointer is meaningless to any other process. > Option 3 lets the user wilfully mix py-lmdb Environment objects within the > process, since py-lmdb always enables NOTLS, but it does not fix the case where > the user is integrating a Python application with a C library with some opaque > interface, and both the application and the library independently have need to > access the environment. Allowing multiple independent libraries to open the same environment would consume redundant chunks of address space. For large enough DBs this may simply fail. Unfortunately the semantics of fcntl locks prevents us from even detecting that this problem has occurred. > This kind of scenario regularly crops up in huge software projects, the user > may not even be aware the library uses LMDB internally. In that case, the user > is again exposed to having their reader slots silently become deallocated > through no fault of their own. > > > Option 4 is the least aesthetic, but has all the benefits of option 3 in > addition to preventing the "accidental integration" scenario. I've attached a > patch for it (against 0.9.11), although I completely understand if it does not > get applied. :) > > One alternative would be instead of using MDB_env* address, using a > monotonically increasing counter based on a static global, say, "mrb_token", > but this might break in the case two copies of LMDB were somehow linked in the > application. (It's easy to argue the user is an idiot in this case ;) Global state is verboten. > > > David > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs May 2014